Sure independence screening for ultrahigh dimensional feature space

Jianqing Fan, Jinchi Lv

Research output: Contribution to journalArticlepeer-review

1820 Scopus citations

Abstract

Summary. Variable selection plays an important role in high dimensional statistical modelling which nowadays appears in many areas and is key to various scientific discoveries. For problems of large scale or dimensionality p, accuracy of estimation and computational cost are two top concerns. Recently, Candes and Tao have proposed the Dantzig selector using L1- regularization and showed that it achieves the ideal risk up to a logarithmic factor log (p). Their innovative procedure and remarkable result are challenged when the dimensionality is ultrahigh as the factor log (p) can be large and their uniform uncertainty principle can fail. Motivated by these concerns, we introduce the concept of sure screening and propose a sure screening method that is based on correlation learning, called sure independence screening, to reduce dimensionality from high to a moderate scale that is below the sample size. In a fairly general asymptotic framework, correlation learning is shown to have the sure screening property for even exponentially growing dimensionality. As a methodological extension, iterative sure independence screening is also proposed to enhance its finite sample performance. With dimension reduced accurately from high to below sample size, variable selection can be improved on both speed and accuracy, and can then be accomplished by a well-developed method such as smoothly clipped absolute deviation, the Dantzig selector, lasso or adaptive lasso. The connections between these penalized least squares methods are also elucidated.

Original languageEnglish (US)
Pages (from-to)849-911
Number of pages63
JournalJournal of the Royal Statistical Society. Series B: Statistical Methodology
Volume70
Issue number5
DOIs
StatePublished - Nov 2008

All Science Journal Classification (ASJC) codes

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Keywords

  • Adaptive lasso
  • Dantzig selector
  • Dimensionality reduction
  • Lasso
  • Oracle estimator
  • Smoothly clipped absolute deviation
  • Sure independence screening
  • Sure screening
  • Variable selection

Fingerprint

Dive into the research topics of 'Sure independence screening for ultrahigh dimensional feature space'. Together they form a unique fingerprint.

Cite this