Analyzing large data sets from XGC1 magnetic fusion simulations using apache spark

R. Michael Churchill

doi:10.1109/NYSDS.2016.7747820

Analyzing large data sets from XGC1 magnetic fusion simulations using apache spark

R. Michael Churchill

PPPL Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Scopus citations

Abstract

Apache Spark is explored as a tool for analyzing large data sets from the magnetic fusion simulation code XGC1. Implementation details of Apache Spark on the NERSC Edison supercomputer are discussed, including binary file reading, and parameter setup. An unsupervised machine learning algorithm, k-means clustering, is applied to XGC1 particle distribution function data, showing that highly turbulent spatial regions do not have common coherent structures, but rather broad, ringlike structures in velocity space.

Original language	English (US)
Title of host publication	2016 New York Scientific Data Summit, NYSDS 2016 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781467390514
DOIs	https://doi.org/10.1109/NYSDS.2016.7747820
State	Published - Nov 17 2016
Event	2016 New York Scientific Data Summit, NYSDS 2016 - New York, United States Duration: Aug 14 2016 → Aug 17 2016

Publication series

Name	2016 New York Scientific Data Summit, NYSDS 2016 - Proceedings

Conference

Conference	2016 New York Scientific Data Summit, NYSDS 2016
Country/Territory	United States
City	New York
Period	8/14/16 → 8/17/16

All Science Journal Classification (ASJC) codes

Information Systems
Computer Networks and Communications
Computer Science Applications

Keywords

distributed computing
k-means clustering
machine-learning
magnetic fusion
simulation
spark

Access to Document

10.1109/NYSDS.2016.7747820

Cite this

@inproceedings{698a04a67c2d4dc9881eae587f3cfb6e,

title = "Analyzing large data sets from XGC1 magnetic fusion simulations using apache spark",

abstract = "Apache Spark is explored as a tool for analyzing large data sets from the magnetic fusion simulation code XGC1. Implementation details of Apache Spark on the NERSC Edison supercomputer are discussed, including binary file reading, and parameter setup. An unsupervised machine learning algorithm, k-means clustering, is applied to XGC1 particle distribution function data, showing that highly turbulent spatial regions do not have common coherent structures, but rather broad, ringlike structures in velocity space.",

keywords = "distributed computing, k-means clustering, machine-learning, magnetic fusion, simulation, spark",

author = "Churchill, \{R. Michael\}",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 2016 New York Scientific Data Summit, NYSDS 2016 ; Conference date: 14-08-2016 Through 17-08-2016",

year = "2016",

month = nov,

day = "17",

doi = "10.1109/NYSDS.2016.7747820",

language = "English (US)",

series = "2016 New York Scientific Data Summit, NYSDS 2016 - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2016 New York Scientific Data Summit, NYSDS 2016 - Proceedings",

address = "United States",

}

Churchill, RM 2016, Analyzing large data sets from XGC1 magnetic fusion simulations using apache spark. in 2016 New York Scientific Data Summit, NYSDS 2016 - Proceedings., 7747820, 2016 New York Scientific Data Summit, NYSDS 2016 - Proceedings, Institute of Electrical and Electronics Engineers Inc., 2016 New York Scientific Data Summit, NYSDS 2016, New York, United States, 8/14/16. https://doi.org/10.1109/NYSDS.2016.7747820

Analyzing large data sets from XGC1 magnetic fusion simulations using apache spark. / Churchill, R. Michael.
2016 New York Scientific Data Summit, NYSDS 2016 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2016. 7747820 (2016 New York Scientific Data Summit, NYSDS 2016 - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Analyzing large data sets from XGC1 magnetic fusion simulations using apache spark

AU - Churchill, R. Michael

PY - 2016/11/17

Y1 - 2016/11/17

N2 - Apache Spark is explored as a tool for analyzing large data sets from the magnetic fusion simulation code XGC1. Implementation details of Apache Spark on the NERSC Edison supercomputer are discussed, including binary file reading, and parameter setup. An unsupervised machine learning algorithm, k-means clustering, is applied to XGC1 particle distribution function data, showing that highly turbulent spatial regions do not have common coherent structures, but rather broad, ringlike structures in velocity space.

AB - Apache Spark is explored as a tool for analyzing large data sets from the magnetic fusion simulation code XGC1. Implementation details of Apache Spark on the NERSC Edison supercomputer are discussed, including binary file reading, and parameter setup. An unsupervised machine learning algorithm, k-means clustering, is applied to XGC1 particle distribution function data, showing that highly turbulent spatial regions do not have common coherent structures, but rather broad, ringlike structures in velocity space.

KW - distributed computing

KW - k-means clustering

KW - machine-learning

KW - magnetic fusion

KW - simulation

KW - spark

UR - https://www.scopus.com/pages/publications/85006955775

UR - https://www.scopus.com/inward/citedby.url?scp=85006955775&partnerID=8YFLogxK

U2 - 10.1109/NYSDS.2016.7747820

DO - 10.1109/NYSDS.2016.7747820

M3 - Conference contribution

AN - SCOPUS:85006955775

T3 - 2016 New York Scientific Data Summit, NYSDS 2016 - Proceedings

BT - 2016 New York Scientific Data Summit, NYSDS 2016 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2016 New York Scientific Data Summit, NYSDS 2016

Y2 - 14 August 2016 through 17 August 2016

ER -

Analyzing large data sets from XGC1 magnetic fusion simulations using apache spark

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Keywords

Access to Document

Other files and links

Fingerprint

Cite this