Sparkmach: A Distributed Data Processing System Based on Automated Machine Learning for Big Data

No hay miniatura disponible
Asner, Gregory P.
Bentley, Lisa Patrick
Blonder, Benjamin
Boza Espinoza, Tatiana Erika
Chambi Porroa, Percy Orlando
Díaz, Sandra
Enquist, Brian J.
Enrico, Lucas
Goldsmith, Gregory R.
Malhi, Yadvinder
Título de la revista
Revista ISSN
Título del volumen
Springer International Publishing
Proyectos de investigación
Unidades organizativas
Número de la revista
This work proposes a semi-automated analysis and modeling package for Machine Learning related problems. The library goal is to reduce the steps involved in a traditional data science roadmap. To do so, Sparkmach takes advantage of Machine Learning techniques to build base models for both classification and regression problems. These models include exploratory data analysis, data preprocessing, feature engineering and modeling. The project has its basis in Pymach, a similar library that faces those steps for small and medium-sized datasets (about ten millions of rows and a few columns). Sparkmach central labor is to scale Pymach to overcome big datasets by using Apache Spark distributed computing, a distributed engine for large-scale data processing, that tackle several data science related problems in a cluster environment. Despite the software nature, Sparkmach can be of use for local environments, getting the most benefits from the distributed processing tools.
Palabras clave
Statistics, Semi-automated machine learning, Data Science, Data mining, Data engineering, Big data