Massive data clustering by multi-scale psychological observations

Shusen Yang, Liwen Zhang, Chen Xu, Hanqiao Yu, Jianqing Fan, Zongben Xu

Research output: Contribution to journalArticlepeer-review

Abstract

Clustering is the discovery of latent group structure in data and is a fundamental problem in artificial intelligence, and a vital procedure in data-driven scientific research over all disciplines. Yet, existing methods have various limitations, especially weak cognitive interpretability and poor computational scalability, when it comes to clustering massive datasets that are increasingly available in all domains. Here, by simulating the multi-scale cognitive observation process of humans, we design a scalable algorithm to detect clusters hierarchically hidden in massive datasets. The observation scale changes, following the Weber-Fechner law to capture the gradually emerging meaningful grouping structure. We validated our approach in real datasets with up to a billion records and 2000 dimensions, including taxi trajectories, single-cell gene expressions, face images, computer logs and audios. Our approach outperformed popular methods in usability, efficiency, effectiveness and robustness across different domains.

Original languageEnglish (US)
Article numbernwab183
JournalNational Science Review
Volume9
Issue number2
DOIs
StatePublished - Feb 1 2022

All Science Journal Classification (ASJC) codes

  • General

Keywords

  • Clustering
  • Cognitive interpretability
  • Computational scalability
  • Massive data
  • Psychological observation
  • Weber-Fechner law

Fingerprint

Dive into the research topics of 'Massive data clustering by multi-scale psychological observations'. Together they form a unique fingerprint.

Cite this