Scaling up classifers to Cloud Computers

Christopher Moretti, Karsten Steinhaeuser, Douglas Thain, Nitesh V. Chawla

Research output: Chapter in Book/Report/Conference proceedingConference contribution

46 Scopus citations


As the size of available datasets has grown from Megabytes to Gigabytes and now into Terabytes, machine learning algorithms and computing infrastructures have continuously evolved in an effort to keep pace. But at large scales, mining for useful patterns still presents challenges in terms of data management as well as computation. These issues can be addressed by dividing both data and omputation to build ensembles of classi-ers in a distributed fashion, but trade-offs in cost, performance, and accuracy must be considered when designing or selecting an appropriate architecture. In this paper, we present an abstraction for scalable data mining that allows us to explore these tradeoffs. Data and computation are distributed to a computing cloud with minimal effort from the user, and multiple models for data management are available depending on the workload and system con-guration. We demonstrate the performance and scalability characteristics of our ensembles using a wide variety of datasets and algorithms on a Condor-based pool with Chirp to handle the storage.

Original languageEnglish (US)
Title of host publicationProceedings - 8th IEEE International Conference on Data Mining, ICDM 2008
Number of pages10
StatePublished - 2008
Externally publishedYes
Event8th IEEE International Conference on Data Mining, ICDM 2008 - Pisa, Italy
Duration: Dec 15 2008Dec 19 2008

Publication series

NameProceedings - IEEE International Conference on Data Mining, ICDM
ISSN (Print)1550-4786


Conference8th IEEE International Conference on Data Mining, ICDM 2008

All Science Journal Classification (ASJC) codes

  • General Engineering


Dive into the research topics of 'Scaling up classifers to Cloud Computers'. Together they form a unique fingerprint.

Cite this