Big Data Engineering on Cloud Platforms
  IJCTT-book-cover
 		          
 
© 2024 by IJCTT Journal
Volume-72 Issue-12
Year of Publication : 2024
Authors : Shrikaa Jadiga
DOI :  10.14445/22312803/IJCTT-V72I12P113

How to Cite?

Shrikaa Jadiga, "Big Data Engineering on Cloud Platforms," International Journal of Computer Trends and Technology, vol. 72, no. 12, pp. 108-120, 2024. Crossref, https://doi.org/10.14445/22312803/IJCTT-V72I12P113

Abstract
With the rapid proliferation of data within the digital landscape, a significant amount of dossier is available, ushering in “Big Data” concepts, embracing massive volume, velocity, and variety of information generated daily. The Phenomenon of Big Data presents significant challenges and opportunities for individuals, business organizations, and all entities that can use the concept to achieve strategic advantages. Extensive data engineering is a critical discipline focusing on designing, developing, and managing systems and architectures that enable effective data handling and analysis. The paper's objective is to explore Big Data evolution and the role of Big Data in informed decision-making in the real world. By integrating enhanced technologies such as Apache, Hadoop, and Spark, cloud computing platforms have changed the data processing landscape, allowing individuals, organizations, and entities to extract meaningful data for decision-making. The paper has examined the current trends in big data, including machine learning and artificial intelligence, and the increasing concerns about data security, privacy and availability, and the rising edge of computing. Additionally, the journal discusses challenges such as managing diverse data sources, ensuring data quality, and addressing the skills gap in the workforce. From such a multilevel lens, the paper offers a nuanced understanding of Big Data Engineering, OOP, and Cloud Platforms and their significant impact on organizational strategic decisions and performance. The paper reveals that integrating Big Data concepts supports an organization's innovation, enhances decision-making processes, and gains a competitive edge, ultimately reshaping the future of data management and analysis. Accordingly, the study's findings underscore notable significant data engineering investment, unlocking capabilities for valuable data as an asset in an increasingly complex world.

Keywords
Analytics, Big data, Cloud platforms, Data engineering and Hadoop.

Reference

[1] Apache Software Foundation, Hadoop, 2020. [Online]. Available: https://hadoop.apache.org/
[2] Amir Gandomi, and Murtaza Haider, “Beyond the Hype: Big Data Concepts, Methods, And Analytics,” International Journal of Information Management, vol. 35, no. 2, pp. 137-144, 2015.
[CrossRef] [Google Scholar] [Publisher Link]
[3] Ibrahim Abaker Targio Hashem et al., “The Role of Big Data in Smart City, International Journal of Information Management, vol. 35, no. 5, pp. 155-157, 2015.
[CrossRef] [Google Scholar] [Publisher Link]
[4] IBM, Big Data and Analytics, 2021. [Online]. Available: https://www.ibm.com/think/topics/big-data-analytics
[5] Avita Katal, Mohammad Wazid, and R. H. Goudar, “Big Data: Issues, Challenges, Tools, and Good Practices,” 2013 Sixth International Conference on Contemporary Computing (IC3), Noida, India, pp. 404-409, 2013.
[CrossRef] [Google Scholar] [Publisher Link]
[6] Doug Laney, “3D Data Management: Controlling Data Volume, Velocity, and Variety,” META Group Research Note, vol. 6, no. 70, 2001.
[Google Scholar]
[7] Hui Luan et al., “Challenges and Future Directions of Big Data and Artificial Intelligence in Education,” Frontiers in Psychology, vol. 11, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[8] Mohamed Dhiaeddine Messaoudi, Bob-Antoine J. Menelas, and Hamid Mcheick, “Integration of Smart Cane with Social Media: Design of a New Step Counter Algorithm for Cane,” IoT, vol. 5, no. 1, pp. 168-186, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[9] Michael Minelli, Michele Chambers, and Ambiga Dhiraj, Big Data, Big Analytics: Emerging Business Intelligence and Analytic Trends for Today's Businesses, Wiley, 1st ed., 2012.
[Google Scholar] [Publisher Link]
[10] Statista, Number of connected devices worldwide from 2019 to 2030, Statista, 2021. [Online]. Available: https://www.statista.com/statistics/1194682/iot-connected-devices vertically/#:~:text=Number%20of%20IoT%20connected%20devices%20worldwide%202019%2D2033%2C%20by%20vertical&text=T he%20consumer%20sector%20is%20anticipated,24%20billion%20connected%20devices%20worldwide.
[11] Tom White, Hadoop: The Definitive Guide, O'Reilly Media, 4th ed., 2015.
[Google Scholar] [Publisher Link]
[12] Paul Zikopoulos, and Chris Eaton, Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data, McGraw-Hill Osborne Media, 2011.
[Google Scholar]