Developing a Data Quality Framework on Azure Cloud : Ensuring Accuracy, Completeness, and Consistency |
||
 |
 |
|
| © 2023 by IJCTT Journal | ||
| Volume-71 Issue-5 |
||
| Year of Publication : 2023 | ||
| Authors : Dinesh Eswararaj | ||
| DOI : 10.14445/22312803/IJCTT-V71I5P111 | ||
How to Cite?
Dinesh Eswararaj, "Developing a Data Quality Framework on Azure Cloud : Ensuring Accuracy, Completeness, and Consistency," International Journal of Computer Trends and Technology, vol. 71, no. 5, pp. 62-72, 2023. Crossref, https://doi.org/10.14445/22312803/IJCTT-V71I5P111
Abstract
This article outlines the development of a data quality framework on Azure Cloud aimed at ensuring data accuracy, completeness, and consistency. The framework is designed to address common data quality issues such as missing data, inconsistent data formats, and inaccurate data. It utilizes various Azure Cloud services such as Azure Data Factory, Azure Databricks, and Azure SQL Database to automate data quality checks and ensure data integrity. The article also discusses the benefits of using the framework, including improved decision-making, compliance with regulatory requirements, and enhanced customer experience. Overall, the data quality framework on Azure Cloud provides a comprehensive solution for ensuring data quality in business operations.
Keywords
Data Quality, Azure Cloud, Framework, Accuracy, Consistency.
Reference
[1] Carlo Batini, and Monica Scannapieca, Data Quality: Concepts, Methodologies and Techniques. Springer, 2006.
[Google Scholar] [Publisher Link]
[2] Mohammad Abdallah et al., “Big Data Quality: Factors, Frameworks, and Challenges,” An International Journal of Advanced Computer Technology, vol. 9, no. 8, p. 3785-3790, 2020.
[Google Scholar] [Publisher Link]
[3] Patrícia Alves de Freitas et al., "Information Governance, Big Data and Data Quality," 2013 IEEE 16th International Conference on Computational Science and Engineering, pp. 1142-1143, 2013.
[CrossRef] [Google Scholar] [Publisher Link]
[4] Majid Al-Ruithe, Elhadj Benkhelifa, and Khawar Hameed, "Key Dimensions for Cloud Data Governance," 2016 IEEE 4th International Conference on Future Internet of Things and Cloud (FiCloud), pp. 379-386, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[5] Imran Quadri Syed, Implementing a Data Quality Monitoring Framework. Redgate, 2020. [Online]. Available: https://www.red-gate.com/simple-talk/databases/sql-server/bi-sql-server/implementing-a-data-quality-monitoring-framework
[6] Amber Lee Dennis, Data Quality, Data Stewardship, Data Governance: Three Keys, Dataversity, 2020. [Online]. Available: https://www.dataversity.net/data-quality-data-stewardship-data-governance-three-keys/
[7] Ehsan Elahi, How to Implement a Data Quality Framework, Dataversity, 2022. [Online]. Available: https://www.dataversity.net/how-to-implement-a-data-quality-framework/
[8] Ankur, G, The 6 Dimensions of Data Quality, 2022. [Online]. Available: https://www.collibra.com/us/en/blog/the-6-dimensions-of-data-quality
[9] Fernandez, R, An Introduction to Data Quality Management Frameworks, Big Data, Techrepublic, 2022. [Online]. Available: https://www.techrepublic.com/article/what-is-a-data-quality-management-framework/
[10] Microsoft. (N.D.). Azure Purview. [Online]. Available: https://azure.microsoft.com/en-us/services/purview/
[11] Microsoft. (N.D.). Azure Synapse Analytics. [Online]. Available: https://azure.microsoft.com/en-us/services/synapse-analytics/
[12] Microsoft. (N.D.). Azure Data Factory. [Online]. Available: https://azure.microsoft.com/en-us/services/data-factory/
[13] Microsoft. (N.D.). Azure Data Lake Storage. [Online]. Available: https://azure.microsoft.com/en-us/services/data-lake-storage/
[14] Microsoft. (N.D.). Azure Databricks. [Online]. Available: https://azure.microsoft.com/en-us/services/databricks/
[15] Microsoft. (N.D.). Cloud Adaption Framework. [Online]. Available: https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/cloud-scale-analytics/govern-data-quality
[16] Lbarrera, Designing a Framework for Data Quality Management. Data Ladder, 2022. [Online]. Available: https://dataladder.com/designing-a-framework-for-data-quality-management/
[17] Farrell, B, What Is Microsoft Purview? Data Driven Daily, 2023. [Online]. Available: https://datadrivendaily.com/what-is-microsoft-purview/
[18] Mike, Data Quality ELT with Azure Data Factory, SQL of the North, 2022. [Online]. Available: https://sqlofthenorth.blog/2022/08/12/data-quality-elt-with-azure-data-factory/
[19] Ikbal Taleb et al., “Big Data Quality Framework: A Holistic Approach to Continuous Quality Management,” Journal of Big Data, vol. 8, no. 76, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[20] The DGI Data Governance Framework Components, Data Governance Institute. [Online]. Available: https://datagovernance.com/the-dgi-data-governance-framework/dgi-data-governance-framework-components/
[21] Steve Young, Data Governance Process, 5 Minute BI, 2023. [Online]. Available: https://5minutebi.com/2021/08/18/data-governance-process
[22] Kevin Booth, How to Implement a Data Governance Program with Azure Purview, EPC Group, 2022. [Online]. Available: https://www.epcgroup.net/how-to-implement-a-data-governance-program-with-azure-purview/
[23] Chau Vinh Loi, A Comprehensive Framework for Data Quality Management: How to Monitor and Maintain Data Quality to Make Sure the Data Meets Certain Standards for Specific Business Use-Cases, 2021. [Online]. Available: https://towardsdatascience.com/a-comprehensive-framework-for-data-quality-management-b110a0465e83
[24] Balvinder Khurana, How to Create and Implement a Robust Data Quality Framework (Part One), Data Strategy Blog, 2022. [Online]. Available: https://www.thoughtworks.com/en-us/insights/blog/data-strategy/enterprises-data-quality-part-one
[25] Balvinder Khurana, How to Create and Implement a Robust Data Quality Framework (Part Two), Data Strategy Blog, 2022. [Online]. Available: https://www.thoughtworks.com/en-us/insights/blog/data-strategy/data-quality-framework-part-two