Data quality: the foundation for effective data-driven work

How important is data quality?

To determine the importance of having your data quality in order, you must view your data in the context of your business processes:

• Why do we need this data?
• What do we use it for?
• Which processes rely on it?
• What value does it provide?

In short, what value does it bring? From this starting point, you can consider the potential consequences of having errors in your data, such as missing values and inaccurate data. When you see your data in that context, the question is no longer 'Should I think about data quality?' but 'Can I afford not to?'

A recent study has shown that the average number of data quality incidents, the time required to detect them, and the time to resolve them have all increased compared to the previous year. At the same time, the estimated impact of data quality issues on both business stakeholders and revenue has increased due to the growth of data-driven working.

What are the risks and consequences of poor data quality?

It is complex to assess the risks of poor data quality. This requires good communication between the business and technical sides of your organisation. The business knows the value of certain data and how processes run, while technical experts often have better insights into data quality. Together, they can determine which data is critical and where data quality must be impeccable.

By having these discussions, you can identify areas for improvement and also discover where you can save time and money by not investing in higher data quality. For data used in critical decisions or processes, you must be certain that it is accurate. If a dataset is primarily used for trend analysis, minor data quality issues may be less concerning. For example, consider the difference between critical data used by governments, pension funds, and banks versus data like retail sales and customer service data.

Assess the risks and consequences by asking a few questions:

• What could go wrong?
• What are the consequences if things go wrong?
• How likely is it to go wrong?

Always adapt your data quality strategy based on the type of data and the risks involved. The investment to get data quality in order and ensure its consistency should be proportionate. Additionally, make sure to document your decisions regarding data quality.

Data quality in order: what opportunities does it bring?

When decisions are based on accurate data, there's greater confidence in the choices made. The risk of making incorrect decisions based on erroneous information is reduced. Furthermore, relying on demonstrably correct data enhances support and trust for data-driven decisions.

With effective monitoring of data quality, you can detect issues earlier. Knowing exactly which process or source produces low-quality data allows you to address the problem at its root cause, improving both data quality and the underlying processes.

Other benefits of implementing data quality management include:

• Cost savings by preventing time-consuming data issues
• Prevention of financial and reputational damage
• More accurate and efficient analyses
• Scalable and robust systems
• Increased revenue or profit through better decisions
• Compliance with data management laws and regulations

How to develop a data quality strategy?

Step 1 - Start with an assessment.

Before determining your strategy, assess where your organization stands. Begin by answering questions such as:

• What motivates us to work on data quality?
• What is our maturity level in terms of data quality?
• How transparent is our data landscape and its data lineage?
• Is there an overview of available datasets?
• Who uses them?
• Who is responsible for specific datasets?
• Do existing guidelines on data quality exist, and are they followed?
• Are there processes to monitor data quality and address problems at the source?

Several models can assist in this process. Look for one that suits your organisation and has logical steps within your context. If you're looking for a framework, the Data Management Body of Knowledge by the Data Management Association (DAMA-DMBOK2) is the most widely used framework worldwide. In the chapter on data quality, it outlines the essential dimensions of data quality and provides a list of actions to define and implement a data quality strategy.

Step 2 - An overview of your as-is and to-be.

Determine the current state of data management (as-is) within your organisation, where you want to go (to-be), and why. Create a business case that explains the value of having data quality in order. Provide insight into the risks and opportunities. You can develop new business cases or explore where higher data quality can add value to existing ones.

Step 3 - Set the budget and the extent of improvement.

Data quality doesn't need to go from 0 to 100 immediately; incremental progress is possible. Critically assess the investment and potential returns. View improving data quality as a continuous process. Ideally, considering data quality should always be part of designing new business processes, systems, and solutions. Having a data strategy in place helps identify the necessary steps for data quality.

How to find time to address data quality?

Skilled data professionals are scarce, and data teams are often busy. To optimise the technical capacity of your team, consider the following tips:

Tip 1: Appoint responsible individuals for datasets within your organisation.

Currently, data quality often falls under the responsibility of Data Engineers. However, ideally, a technical person should not be solely responsible for a dataset. The content must align with the business process it supports. Technical colleagues can build checks and monitor processes, but they may not always be best positioned to assess the required data quality level. Business stakeholders who are intimately familiar with the business processes that this data supports are in a better position to evaluate this.

By doing so, technical individuals, such as Data Engineers, can focus on their expertise: designing and implementing data flows and ensuring that defined quality checks are correctly implemented in the system.

Tip 2: Utilize helpful tools and services.

To make the work of scarce technical professionals more efficient, various tools and services are being developed to set up data quality checks relatively easily. These tools can be used to check data quality at specific moments as well as track quality over time. Are there missing values? Are the correct data types used? Where does the data come from (data lineage), and what is available (data catalogs)?

Examples of tools widely used for data quality implementation are Soda, Great Expectations, and dbt. The right tool choice depends on the design of your data applications, the technologies used, and your data team's expertise. For introducing and maintaining data quality checks, the crucial tools must align well with the current systems and workflow.

Another significant development is the use of generative AI. Several data quality services are starting to adopt this approach. With AI, you can define checks in regular language, and the tool generates the check for you. This simplifies programmatic tasks.

However, someone with technical knowledge must verify that the correct tests are implemented and that a test performs its intended function. We foresee this expertise becoming even more valuable as data quality is more widely implemented.

Should improving data quality be a higher priority?

Improving data quality within your organisation might not have been at the top of your priority list before you read this article. After all, your data team is already quite busy! However, it's essential to realize that having incorrect data can be time-consuming.

As your pipelines and data landscape become more complex and extensive, it becomes increasingly challenging and time-consuming to detect and resolve data quality issues. All this time spent on data quality issues takes away from new developments that could have a more significant impact and business value.

By getting your data quality processes in order, you can minimize this effort and develop a more robust, scalable, and easily maintainable system. Improving your data quality requires an investment of time, but it can ultimately save you a lot of time in the long run.

We're here to help!

Is data quality high on your agenda? Our data consultants can assist you at every step of the process. This includes designing your data strategy, analysing your data flows, and determining the data quality requirements that suit your needs.

We can also handle the technical implementation for you. We'll conduct checks on your existing data warehouse and establish data quality and governance within your current environment or set up an entirely new data infrastructure for you. Additionally, we're more than happy to help you choose the right tools or services to achieve your desired level of data quality.

Receive data insights, use cases and behind-the-scenes peeks once a month?

Sign up for our email list and stay 'up to data':