Enhancing Cybersecurity Through the Unification of Data Analytics, Artificial Intelligence, and Machine Learning in Big Data Cloud Environments: A Databricks Lakehouse Approach
|© 2023 by IJCTT Journal|
|Year of Publication : 2023|
|Authors : Mayur Katariya, Digan Parikh|
|DOI : 10.14445/22312803/IJCTT-V71I6P104|
How to Cite?
Mayur Katariya, Digan Parikh, "Enhancing Cybersecurity Through the Unification of Data Analytics, Artificial Intelligence, and Machine Learning in Big Data Cloud Environments: A Databricks Lakehouse Approach," International Journal of Computer Trends and Technology, vol. 71, no. 6, pp. 25-29, 2023. Crossref, https://doi.org/10.14445/22312803/IJCTT-V71I6P104
“Data without analytics is dormant, the cloud without security is vulnerable, and AI without data is blind. Together, they form an unstoppable force in the realm of cybersecurity.” The cybersecurity industry faces numerous challenges in combating the ever-evolving threat landscape. This scholarly article aims to shed light on these challenges and explores how Databricks' lakehouse cloud architecture offers potential solutions. The article provides a comprehensive detail of the current state of the cybersecurity industry and maturity curve and highlights the pressing issues that organizations encounter. The article focuses on the utilization of Databricks' lakehouse architecture as a means to address these challenges. Furthermore, the paper delves into the key features and capabilities of Databricks' lakehouse architecture that enable organizations to efficiently analyze and govern vast amounts of security data, detect anomalies, and identify potential threats in real time. The paper concludes by drawing on real-world examples and industry case studies. This paper offers a detailed overview of the real-world cybersecurity architecture examples, data flow and how different companies are embracing the lakehouse architecture to power their use cases. By unifying the combined power of data analytics, artificial intelligence (AI), and machine learning (ML), the cloud lakehouse architecture helps organizations to break silos, derive actionable insights, offer a promising approach for bolstering cybersecurity defenses, and enhance their proactive cybersecurity strategies at scale on cloud.
Databricks lakehouse, Cloud, Bigdata, Analytics, Artificial Intelligence(AI), Machine Learning(ML), Cybersecurity.
 Aaron Drapkin, Data Breaches That Have Happened in 2022 and 2023 so far, 2023. [Online]. Available: https://tech.co/news/data-breaches-updated-list
 Jade Artry, Average Data Breach Cost for US Businesses Almost $10 Million, 2023. [Online]. Available: https://tech.co/news/average-us-company-data-breach-cost-10-million
 Databricks Website. [Online]. Available: https://www.databricks.com/blog/announcing-general-availability-databricks-sql-serverless
 Databricks Website, what is Delta Lake?, 2023. [Online]. Available: https://docs.databricks.com/delta/index.html
 Databricks Website, GDPR & CCPA. [Online]. Available: https://docs.databricks.com/security/privacy/gdpr-delta.html
 Databricks Website, Compliance Security and Trust Center. [Online]. Available: https://www.databricks.com/trust#compliance
 Databricks Website, MLOps. [Online]. Available: https://www.databricks.com/resources/ebook/the-big-book-of-mlops
 Databricks Website, Unity Catalog. [Online]. Available: https://docs.databricks.com/data-governance/unity-catalog/index.html
 Databricks Website, Medallion-Architecture. [Online]. Available: https://www.databricks.com/glossary/medallion-architecture
 Spark + AI Summit, 2018 Vimeo Website. [Online]. Available: https://vimeo.com/274267634
 Data and AI Summit, Accidentally Building a Petabyte-Scale Cybersecurity Data Mesh in Azure With Delta Lake at HSBC, YouTube Website, 2022. [Online]. Available: https://www.youtube.com/watch?v=G9x-1s-1TJI
 Data and AI Summit, Akamai Interactive Analytics on a Massive Scale Using Delta Lake, YouTube Website, 2022. [Online]. Available: https://www.youtube.com/watch?v=_K9gIOsP4Os
 Databricks Website Customers – Akamai. [Online]. Available: https://www.databricks.com/customers/akamai
 Databricks Website Customers - Abnormal Security. [Online]. Available: https://www.databricks.com/customers/abnorma