Dalarna University's logo and link to the university's website

In data-driven projects, high-quality data is crucial for informed decision-making, operational efficiency, and the reliability of products and services. Traditional data quality assessment methods often struggle with the rapid pace of data changes and can be resource-intensive to maintain. This research explores the potential of Large Language Models (LLMs), such as OpenAI’s GPT-3 and GPT-4, in assessing data quality within asset management systems. The study compares the performance of LLMs against traditional hard-coded rule-based systems, focusing on structured and semi-structured data types, evaluating them in terms of accuracy and response time. The methodology involves developing plain text rules for key data quality dimensions—accuracy, completeness, and consistency—and testing the performance of different LLMs (GPT-3.5-turbo, GPT-4, and GPT-4-turbo) using these rules. The results show that while LLMs, especially GPT-4-turbo, achieve high accuracy (meanaccuracy of 0.86 for semi-structured files and 0.759 for structured files) for straightforward rules, their performance diminishes with increased rule complexity. For instance, the accuracy of GPT-4-turbo drops significantly for advanced rules, with a 5th percentile accuracy of 0.17. Moreover, GPT-4 demonstrated faster response times compared to its turbo counterparts, with an average time advantage of 40 seconds for semi-structured files. However, substantial variability in response times (e.g., a standard deviation of 50.4 seconds for GPT-4 on semi-structured files, with an average response time difference of 165.0 seconds) and inconsistent accuracy (MCC values around zero) underscore the limitations of LLMs in data quality assessment. This study contributes to the field by providing a comparative analysis of LLMs and traditional systems using a structured methodology for rule-based assessments and identifying areas for future enhancement in data quality assessment using LLMs. It discusses several limitations, including high financial costs, response time variability ,and challenges in processing large datasets and specific formatting instructions. Future research should explore advanced prompt optimization, domain-specific GPT training, and the development of custom LLMs tailored to the needs of data quality assessment.