IJBTT

Data Virtualization Architecture Framework using Multi-Engine Data Platforms for Big Data Analytics and Machine Learning
	© 2024 by IJCTT Journal
	Volume-72 Issue-2
	Year of Publication : 2024
	Authors : Anandaganesh Balakrishnan
	DOI :  10.14445/22312803/IJCTT-V72I2P115

How to Cite?

Anandaganesh Balakrishnan, "Data Virtualization Architecture Framework using Multi-Engine Data Platforms for Big Data Analytics and Machine Learning," International Journal of Computer Trends and Technology, vol. 72, no. 2, pp. 82-91, 2024. Crossref, https://doi.org/10.14445/22312803/IJCTT-V72I2P115

Abstract
In the current landscape of Artificial Intelligence and Machine Learning innovation, the imperative to aggregate data from varied sources and derive real-time insights is more pronounced than ever. This necessity gives rise to a Multi-Engine Data Virtualization Framework, a novel approach designed to refine data virtualization and management strategies. Distinct from conventional data virtualization systems, which often falter in processing complex and voluminous data, this innovative framework aims to capitalize on the diverse strengths of various data platforms, thereby elevating efficiency and efficacy in data virtualization. The framework effectively tackles prevalent data management and access obstacles by facilitating the seamless amalgamation of federated queries with multiple data engines. It delves into advanced caching databases, Massively Parallel Processing (MPP) engines, and vector databases to support real-time big data analytics and machine learning endeavors. The necessity of this framework underscores the inadequacies of current data virtualization solutions in fulfilling the multifaceted demands of contemporary data management, which include costeffective caching, vector embeddings for machine learning, and the distributed processing of large data volumes. The paper also emphasizes future research avenues such as evaluating performance, optimizing queries adaptively, augmenting caching strategies, ensuring scalability and fault tolerance, addressing security and privacy, and incorporating emerging technologies. This research marks a pivotal advancement towards attaining unparalleled data management efficiency and flexibility, poised to transform organizational practices in managing, accessing, and leveraging data for insights.

Keywords
Big data analytics, Caching, Data virtualization, Massive parallel processing, Vector databases.

Reference

[1] Sasidhar Duggineni, “Data Analytics in Modern Business Intelligence,” Journal of Marketing & Supply Chain Management, vol. 1, no. 2, pp. 1-4, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[2] Data virtualization, Wikipedia. [Online]. Available: https://en.wikipedia.org/wiki/Data_virtualization
[3] Ibis-Project, Ibis-Framework. [Online]. Available: https://ibis-project.org/
[4] Alexander Bogdanov et al., “Big Data Virtualization: Why and How?,” CEUR Workshop Proceedings, vol. 2679, pp. 11-21, 2020.
[Google Scholar]
[5] Denodo Platform 8.0, Denodo Embedded MPP, Denodo. [Online]. Available: https://community.denodo.com/docs/html/browse/latest/en/vdp/administration/embedded_parallel_processing/embedded_parallel_processing
[6] Manoj Muniswamaiah, Tilak Agerwala, and Charles Tapper, “Data Virtualization for Analytics and Business Intelligence in Big Data,” Seidenberg School of CSIS, Pace University, White Plains, New York, pp. 297-302, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[7] Laijo John Pullokkaran, “Analysis of Data Virtualization & Enterprise Data Standardization in Business Intelligence,” M.S. Thesis, Massachusetts Institute of Technology, USA, pp. 1-59, 2013.
[Google Scholar] [Publisher Link]
[8] Redis, Wikipedia. [Online]. Available: https://en.wikipedia.org/wiki/Redis
[9] Memcached, Wikipedia. [Online]. Available: https://en.wikipedia.org/wiki/Memcached
[10] Apache Ignite, Wikipedia. [Online]. Available: https://en.wikipedia.org/wiki/Apache_Ignite
[11] Ehcache, Wikipedia. [Online]. Available: https://en.wikipedia.org/wiki/Ehcache
[12] Naresh Kumar Miryala, and Divit Gupta, “Big Data Analytics in Cloud – Comparative Study,” International Journal of Computer Trends and Technology, vol. 71, no. 12, pp. 30-34, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[13] Performance in Logical Architectures and Data Virtualization with the Denodo Platform and Presto MPP, Denodo. [Online]. Available: https://denodo.medium.com/performance-in-logical-architectures-and-data-virtualization-with-the-denodo-platform-and-presto-e689762f912b
[14] System Properties Comparison Spark SQL vs Trino, Db-Engines. [Online]. Available: https://db-engines.com/en/system/Spark+SQL%3BTrino
[15] System Properties Comparison Ignite vs. Memcached vs. Redis, Db-Engines. [Online]. Available: https://db-engines.com/en/system/Ignite%3BMemcached%3BRedis
[16] Balancing the Challenges and Opportunities of Multiplatform Data Architectures, Tdwi. [Online]. Available: https://tdwi.org/articles/2018/04/23/ARCH-ALL-Challenges-Opportunities-of-Multiplatform-Data-Architectures.aspx
[17] Vijaynath Viswanathan, Caching Strategies and Cache Eviction Policies, Medium, 2023. [Online]. Available: https://medium.com/@vijaynathv/caching-strategies-and-cache-eviction-policies-768351e25f1f
[18] Shirish Joshi, The 5 Data Consolidation Patterns — Data Lakes, Data Hubs, Data Virtualization/Data Federation, Data Warehouse, and Operational Data Stores, Medium, 2020. [Online]. Available: https://medium.com/swlh/the-5-data-store-patterns-data-lakes-data-hubs-data-virtualization-data-federation-data-27fd75486e2c
[19] Maggy Hu, MPP: The Transformation on Big Data Analytics, Medium, 2019. [Online]. Available: https://medium.com/slalom-technology/mpp-the-transformation-on-big-data-analytics-684082067841
[20] Jagadesh Jamjala, 5 Key Concepts of Massively Parallel Processing, Medium, 2023. [Online]. Available: https://medium.com/@jagadeshjamjalanarayanan/5-key-concepts-of-massively-parallel-processing-86d993552f8c
[21] Apache Spark, Wikipedia. [Online]. Available: https://en.wikipedia.org/wiki/Apache_Spark
[22] SeattleDataGuy, What Is Trino And Why Is It Great At Processing Big Data, Dev, 2021. [Online]. Available: https://dev.to/seattledataguy/what-is-trino-and-why-is-it-great-at-processing-big-data-8pc
[23] Everton Gomede, Vector Databases: Revolutionizing Data Management in the Age of AI, Medium, 2023. [Online]. Available: https://medium.com/@evertongomede/vector-databases-revolutionizing-data-management-in-the-age-of-ai-ba5a14444ab5
[24] Mahalakshmi Hariharan, Vector Databases in Action: Real-World Use Cases and Benefits, Medium, 2023. [Online]. Available: https://medium.com/@mahalakshmi1117/vector-databases-in-action-real-world-use-cases-and-benefits-549c395794a8
[25] Christoph Bussler, Vector Databases (are All The Rage), Medium, 2023. [Online]. Available: https://medium.com/google-cloud/vectordatabases-are-all-the-rage-872c888fa348