Measuring Housing Vitality from Multi-Source Big Data and Machine Learning

Yang Zhou, Lirong Xue, Zhengyu Shi, Libo Wu, Jianqing Fan

Research output: Contribution to journalComment/debatepeer-review

2 Scopus citations


Measuring timely high-resolution socioeconomic outcomes is critical for policymaking and evaluation, but hard to reliably obtain. With the help of machine learning and cheaply available data such as social media and nightlight, it is now possible to predict such indices in fine granularity. This article demonstrates an adaptive way to measure the time trend and spatial distribution of housing vitality (number of occupied houses) with the help of multiple easily accessible datasets: energy, nightlight, and land-use data. We first identified the high-frequency housing occupancy status from energy consumption data and then matched it with the monthly nightlight data. We then introduced the Factor-Augmented Regularized Model for prediction (FarmPredict) to deal with the dependence and collinearity issue among predictors by effectively lifting the prediction space, which is suitable to most machine learning algorithms. The heterogeneity issue in big data analysis is mitigated through the land-use data. FarmPredict allows us to extend the regional results to the city level, with a 76% out-of-sample explanation of the spatial and timeliness variation in the house usage. Since energy is indispensable for life, our method is highly transferable with the only requirement of publicly accessible data. Our article provides an alternative approach with statistical machine learning to predict socioeconomic outcomes without the reliance on existing census and survey data. Supplementary materials for this article are available online.

Original languageEnglish (US)
Pages (from-to)1045-1059
Number of pages15
JournalJournal of the American Statistical Association
Issue number539
StatePublished - 2022
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Statistics and Probability
  • Statistics, Probability and Uncertainty


  • Computational social science
  • Factor model
  • FarmPredict
  • Housing vitality
  • Machine learning


Dive into the research topics of 'Measuring Housing Vitality from Multi-Source Big Data and Machine Learning'. Together they form a unique fingerprint.

Cite this