Enhancing Big Data Processing Performance Using Distributed AI Techniques on High-Performance Computing Systems

Ahmed Nafea Ayesh

Authors

Ahmed Nafea Ayesh Al Iraqia University , Baghdad, Iraq

Keywords:

Distributed AI, High-Performance Computing (HPC), Big Data processing, Apache Spark, GPU acceleration, Random Forest, Deep Neural Networks, energy consumption, scalability

Abstract

Big Data processing requires high-performance solutions in today's industries with the increasing growth of data. Traditional computing techniques are not efficient to deal with huge datasets based on process and memory constraints . Distributed AI algorithms on HPC platforms are utilized in this work to enhance Big Data processing performance. Distributed Random Forest and Deep Neural Networks were experimented with multi-core CPUs and GPU clusters. Memory optimization and cache reuse were employed to minimize data access latency. Experiments based on synthetic health-care and financial data sets show remarkable improvement in processing time, prediction accuracy, and power consumption. Experiments prove the efficacy of distributed AI strategies along with HPC for scalable Big Data analysis with high performance.

Downloads

Download data is not yet available.

References

J. Dean and S. Ghemawat, “MapReduce: Simplified data processing on large clusters,” Communications of the ACM, vol. 51, no. 1, pp. 107–113, 2008, doi: 10.1145/1327452.1327492.

M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, “Spark: Cluster computing with working sets,” in Proc. 2nd USENIX Conf. Hot Topics in Cloud Computing, Boston, MA, USA, 2010, pp. 10–10.

M. Li et al., “Scaling distributed machine learning with the parameter server,” in Proc. 11th USENIX Symp. Operating Systems Design and Implementation (OSDI), Broomfield, CO, USA, 2014, pp. 583–598.

M. Abadi et al., “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2016. [Online]. Available: https://www.tensorflow.org

N. Ahmed, “A comprehensive performance analysis of Apache Hadoop and Apache Spark for big data processing,” Journal of Big Data, vol. 7, no. 1, pp. 1–15, 2020, doi: 10.1186/s40537-020-00388-5.

S. Wang, H. Zheng, X. Wen, and F. Shang, “Distributed high-performance computing methods for accelerating deep learning training,” JKLST Journal of Computer Science and Technology, vol. 10, no. 1, pp. 1–15, 2024.

M. Priyadi, Migunani, and Sasmoko, “Enhancing big data processing efficiency in AI-based healthcare systems: A comparative analysis of Random Forest and deep learning algorithms,” Journal of Technology Informatics and Engineering, vol. 3, no. 3, pp. 263–278, 2024.

M. Zaharia et al., “Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing,” in Proc. 9th USENIX Symp. Networked Systems Design and Implementation (NSDI), San Jose, CA, USA, 2012.

J. Dean et al., “Large scale distributed deep networks,” in Advances in Neural Information Processing Systems (NeurIPS), 2012.

J. Shi, X. Chu, and B. Li, “Benchmarking state-of-the-art deep learning software tools,” in Proc. IEEE Int. Conf. Big Data, Washington, DC, USA, 2016.

Q. Chen, M. Guo, and L. Xiao, “Optimizing data-intensive workloads in high-performance computing systems,” IEEE Transactions on Parallel and Distributed Systems, vol. 29, no. 4, pp. 825–838, 2018.

X. Meng et al., “MLlib: Machine learning in Apache Spark,” Journal of Machine Learning Research, vol. 17, no. 34, pp. 1–7, 2016.

M. Armbrust et al., “Spark SQL: Relational data processing in Spark,” in Proc. ACM SIGMOD Int. Conf. Management of Data, Melbourne, Australia, 2015, pp. 1383–1394.

L. A. Barroso, J. Clidaras, and U. Hölzle, The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, 2nd ed. San Rafael, CA, USA: Morgan & Claypool, 2013.

Y. Chen, A. Ganapathi, R. Griffith, and R. Katz, “The case for evaluating MapReduce performance using workload suites,” in Proc. IEEE Int. Symp. Modeling, Analysis & Simulation of Computer and Telecommunication Systems (MASCOTS), Singapore, 2011.