Zensations
All articles

Artificial Intelligence

Data Quality Before AI Projects: The Underestimated Preparatory Work

Zensations

No model compensates for poor data. Those who organise the fundamentals first achieve better results with simple means than with expensive technology.

The four most common data problems

In projects, we repeatedly encounter the same patterns: duplicate data sets, outdated information, inconsistent terminology, and missing responsibilities. Each of these problems noticeably reduces a system's hit rate.

The effort required for clean-up is usually smaller than feared, because errors often concentrate on a few sources.

A pragmatic approach

Here’s how to start without a big programme:

  • Name the ten most important data fields for the use case.
  • Define a binding source for each field.
  • Define rules for spelling and mandatory entries.
  • Record the responsible person and review cycle.

Clarify legal foundations

Before use, check the basis on which data is processed and how long it may be stored. This clarification should happen at the outset, as it determines the permissible scope of the project.

The benefits are quickly apparent

Within just a few weeks, rework and queries decrease, analyses become more reliable, and automations function more stably. This improvement works regardless of which tool is subsequently used.

Clean data is the most cost-effective way to achieve better results.

Why data quality is the foundation for every successful AI project

Artificial intelligence is not a magic word that automatically turns bad data into gold. On the contrary: the quality of an AI system's results is directly proportional to the quality of the data with which it is trained and operated. Those who ignore this connection invest a lot of money in technology that ultimately delivers incorrect or useless analyses. A typical example is an AI-powered recommendation system that suggests products customers have already purchased because their purchase history has not been correctly updated. Or a chatbot that provides incorrect opening hours because it is fed with outdated data. User frustration is high, and the loss of trust can be lasting.

Imagine a logistics company in Austria wants to optimise delivery routes with AI. If data on customer locations is inaccurate, house numbers are missing, or street names are misspelled, the AI cannot calculate efficient routes. The result is longer journey times, higher fuel costs, and dissatisfied customers waiting for their delivery. An investment in cleaning up the address database beforehand, for example, through automatic address validation and manual checking of unusual entries, would quickly pay for itself here. In one case, a medium-sized transport company in Styria reduced its delivery costs by 8 per cent after the quality of its address database was improved by 15 per cent through an initial clean-up project.

What costs do poor data incur?

Poor data quality is a hidden cost factor that manifests itself in various ways within a company. These costs are often not immediately apparent but quickly add up to significant amounts. Direct costs, for example, include higher personnel expenses for manual corrections, the creation of new data records due to duplicates, or wasted marketing spend on undeliverable mailings. Indirect costs are harder to quantify but are at least as relevant: lost customers due to faulty communication, missed sales opportunities due to inaccurate market analyses, or strategic misjudgements based on incorrect key figures. The damage to reputation from publicly visible errors, such as an incorrect product description on a website, should also not be underestimated.

A study by IBM estimated the annual cost of poor data quality in the USA at 3.1 trillion dollars. For an average company, this means that 15 to 25 per cent of resources must be spent managing data problems. An Austrian e-commerce company found that 12 per cent of its customer enquiries were due to erroneous product information. Rectifying these errors and handling complaints cost the company over 50,000 Euros annually before a targeted data quality initiative was launched. By improving product master data, these costs were reduced by 70 per cent within six months.

The most common sources of data problems and how to identify them

Data problems rarely arise from malicious intent but are often the result of inadequate processes, missing standards, and human errors during data entry. Identifying these sources is the first step towards sustainable data quality improvement.

  • Manual Data Entry: Typos, different spellings, or omissions of information are common problems. One sales representative enters "Muster GmbH", another "Muster GesmbH", a third "Muster Gmbh & Co KG". You then have three entries for the same company.
  • System Integrations: Inconsistencies can arise when data is transferred between different systems, e.g., from CRM to ERP or from the online shop to the merchandise management system. Fields are incorrectly mapped, data types do not match, or updates are not synchronised.
  • Missing or Unclear Definitions: If it's not clear what exactly belongs in a specific field (e.g., "customer type"), this leads to inconsistent entries. Is "regular customer" a customer type or a description?
  • Old Data Migrations: When transferring legacy data to new systems, existing errors are often carried over, or new ones are created during migration, without any clean-up taking place.
  • Missing Data Validation: If systems do not perform plausibility checks (e.g., email format, postcode to location), any erroneous data can be entered.
  • External Data Sources: Data obtained from third-party providers or collected by external service providers can also have quality deficiencies that are not always immediately apparent.

To identify these sources, data profiling software can be used to detect patterns and anomalies. A manual approach is to interview employees who work with the data daily. They can provide valuable insights into "pain points" and recurring errors. Conduct workshops to collect and prioritise these problems. A medium-sized service company in Vienna identified six critical data fields in such workshops that were erroneous in 70 per cent of cases. The systematic correction and definition of input rules for these fields led to a significant increase in data quality in the customer database.

How can a company systematically improve data quality?

Improving data quality is not a one-off project but a continuous process. A systematic approach is crucial to achieve sustainable success.

1. Define a data quality strategy

  • Determine which data is critical for your business processes and AI applications. Not every data row needs to be perfect, but core information must be correct.
  • Define clear quality objectives, e.g., "98 per cent of all customer names are unique and correct" or "A maximum of 1 per cent of product descriptions contain errors".
  • Link these objectives to business goals. If sales are to be increased through personalised offers, the data quality of customer segments must be very high.

2. Collect and analyse data

  • Conduct a data audit. What data exists? Where does it come from? Who is responsible?
  • Use data profiling tools to measure metrics such as completeness, consistency, uniqueness, and timeliness. A tool shows that 25% of email addresses in the database are invalid or have an incorrect format.
  • Visualise the results to quickly identify problem areas. A dashboard displaying the percentage of missing values in important fields is helpful here.

3. Cleanse and enrich data

  • Prioritise cleansing measures based on identified costs and risks. Start with the data that has the greatest business impact.
  • Use automation for simple tasks, e.g., duplicate removal or address standardisation. However, manual corrections may also be necessary for unclear entries.
  • Supplement missing data with external sources where sensible and permissible. Enriching B2B customer data with company size or industry can provide valuable insights for marketing and sales. Always consider Entity Clarity for AI systems.

4. Ensure and monitor data quality

  • Implement data quality rules directly into your data entry systems, e.g., through validation checks during form input.
  • Establish continuous monitoring processes and dashboards to keep an eye on data quality.
  • Regularly train employees on data handling and defined quality standards.
  • Appoint a "Data Steward" or a team responsible for data quality and serving as a contact point. A good AI Content Operations strategy also includes data maintenance.

A leading Swiss banking institution applied these steps to optimise its customer database. After one year, the institution was able to reduce the number of erroneous data records by 60 per cent, leading to a 15 per cent improvement in marketing campaign results and a 20 per cent reduction in manual rework.

The value of a Data Governance Framework for AI projects

A Data Governance Framework is the organisational structure that defines rules, processes, roles, and responsibilities for handling data within a company. It is the backbone of any serious data quality initiative and therefore indispensable for successful AI projects.

Why is Data Governance so important?

  • Clarity on Data Origin: It defines where data comes from, who created it, and who is the primary source. This is crucial for data trustworthiness.
  • Standardisation: It establishes uniform definitions, formats, and quality standards for all relevant data. Without these standards, AI cannot recognise consistent patterns.
  • Compliance and Legal Certainty: It ensures that data is processed and stored in accordance with legal requirements such as the GDPR. This is particularly important when AI systems work with personal data. Early clarification of the legal basis is essential and can be done, for example, through digital consulting, see Digital Consulting.
  • Risk Management: It helps minimise risks associated with data leaks, errors, or misuse.
  • Decision-making: High-quality, trustworthy data enables more informed business decisions.

A Data Governance Framework often includes the following elements:

  • Data Policy: An overarching policy that defines the company's general stance on data.
  • Data Stewardship: The designation of individuals or teams responsible for the quality, security, and compliance with policies for specific data areas.
  • Data Catalogue: A kind of "data map" that describes all relevant data assets, their meaning, and their metadata.
  • Processes and Workflows: Defined procedures for data collection, storage, processing, and deletion.
  • Technologies: Tools and systems that support the implementation of the framework (e.g., Master Data Management systems, data quality tools).

In a large Austrian financial services provider, a comprehensive Data Governance Framework was implemented to create the data basis for developing new AI-based fraud detection systems. The introduction took 18 months and involved training over 100 employees. As a result, the accuracy of fraud detection increased from 75% to 92%, saving over 2 million Euros annually that were previously lost due to fraud cases.

The role of Master Data Management (MDM) for consistent data

Master Data Management (MDM) is a central component of a comprehensive data quality strategy, especially when it comes to ensuring consistent data across various systems and departments. Master data is a company's most critical data, such as customer information, product data, supplier information, or financial accounts. It forms the basis for operational and analytical processes.

What does MDM achieve?

  • Single Source of Truth: MDM creates and maintains a single, consistent, and reliable version of master data. Instead of different departments having their own, potentially contradictory versions of a customer base, there is a central source.
  • Duplicate Prevention: MDM systems detect and merge duplicate data records, thereby avoiding inconsistencies and confusion.
  • Data Standardisation: It ensures uniform formats and values by enforcing rules for data entry and automatically converting data into the desired format.
  • Data Distribution: The MDM system distributes the "clean" master data to all connected operational and analytical systems within the company, e.g., CRM, ERP, e-commerce platforms.
  • Data Lifecycle Management: It manages the entire lifecycle of master data, from creation and updating to archiving or deletion.

For AI projects, MDM is invaluable. An AI model that aims to analyse customer behaviour needs a consistent view of the customer, regardless of whether the data comes from the online shop, the call centre, or sales. Without MDM, the AI system could identify the same person as multiple different customers, leading to failed analyses and incorrect recommendations.

A leading building materials dealer in Germany implemented an MDM system to consolidate its more than 100,000 item master data records. Previously, there were significant inconsistencies in product names, dimensions, and prices across various warehouse locations and sales channels. Following the introduction of MDM and a nine-month clean-up phase, errors in order processing were reduced by 20 per cent, and logistics efficiency increased by 10 per cent. This laid the foundation for the successful implementation of an AI-powered price optimisation system.

Checklist: Your first steps towards better data quality for AI

Do you want to start improving your data quality but are unsure where and how to begin? This checklist will help you plan and implement the first important steps.

Phase 1: Inventory and Prioritisation

  • Name the AI project you want to implement first (e.g., chatbot for customer service, personalised recommendations, automated financial analysis).
  • Identify the 5 to 10 most important data fields that are absolutely critical for this AI project.
  • Determine the primary data source for each of these fields (e.g., CRM, ERP, website analytics).
  • Assess the current quality of these critical data fields: How many missing values are there? How many duplicates? What is the proportion of erroneous or inconsistent entries? A simple sample of 100 data records can provide initial insights.
  • Document the potential impact of poor data quality on your planned AI project (e.g., "Chatbot provides incorrect product information if item descriptions are outdated").

Phase 2: Definition and Preparation

  • Define clear standards for the critical data fields: Which spellings are allowed? Which formats must be adhered to? (e.g., "Date always in DD.MM.YYYY format", "Product numbers always 8-digit alphanumeric").
  • Appoint a responsible person (the "Data Steward") for each critical data field, who will ensure compliance with standards.
  • Clarify the legal framework for using data in the AI project, particularly regarding data protection and storage duration. An initial Free GEO-Check or digital consulting can be helpful here.
  • Plan a review cycle for the data quality of these fields (e.g., weekly sample checks, monthly automated reports).

Phase 3: Implementation and Monitoring

  • Start by cleaning up a small but critical data area. This is often more manageable and delivers quick successes that motivate.
  • Implement initial validation rules directly into the data entry systems to prevent new errors.
  • Train affected employees on the new standards and processes.
  • Measure the improvement in data quality in the selected fields and communicate successes internally.
  • Gradually scale the approach to further data fields and business areas.

A practical example: An Austrian hotel company wanted to implement an AI chatbot for booking enquiries. The checklist helped them identify room category data, price lists, and availabilities as critical. They defined uniform designations for room categories ("Single Room", "Double Room Standard", "Suite Deluxe") and ensured that price data was always up to date. Within four weeks, the data quality of the relevant fields was improved to such an extent that the chatbot could provide information and mediate bookings without errors, leading to a 15% reduction in telephone enquiries.

Looking to the future: Establishing a data culture

Long-term success with AI requires more than just one-off data cleansing; it requires a deeply ingrained data culture. A data culture means that everyone in the company understands the value of data and feels responsible for its quality. It is a shift in mindset that views data not just as a tedious necessity but as a valuable asset that needs nurturing. This goes hand in hand with strategic digitisation and the right technological decisions, such as the question of Headless CMS or traditional systems for content data storage.

A lived data culture is reflected in the following points:

  • Data Literacy: Employees from all departments are trained in data literacy. They understand how their daily work affects data quality and what impact that has on other areas or AI systems.
  • Data-Driven Decisions: Decisions are no longer made instinctively but are based on sound data analyses. AI systems become an integral part of decision-making.
  • Transparency: Access to data and metadata is made transparent. Everyone knows what data is where, who is responsible for it, and what its quality is.
  • Continuous Improvement: Data quality is understood as a continuous process that requires constant adaptation and optimisation. There are established processes for feedback and improvements.
  • Accountability: Clear responsibilities for data are established, from senior management to individual employees.

Establishing a data culture is a long-term project that offers immense advantages for a company's future viability. It not only enables the successful use of AI but also increases efficiency, minimises risks, and fosters innovation. Those who take their data seriously create the foundation for a digital transformation that creates added value not just on paper, but in reality. This is also closely linked to how a company formulates its messages; even the brand voice in times of AI benefits from clear, consistent data.

Your next step

Data quality is the undisputed foundation for every successful AI project. Do not underestimate this preparatory work. A well-thought-out plan and step-by-step implementation will lead you to reliable AI results and real competitive advantages. At Zensations, we understand these challenges. As a digital agency focusing on strategy and implementation, we are happy to help you prepare your data for AI.

Would you like to implement this topic for your company? We will examine your starting position, identify the three most effective measures, and provide you with an estimate of effort and timeframe. Write to us at office@zensations.at or start with the free GEO-Check. If you would like to discuss a project with our team, you can find contact information here.

Further reading from Zensations

Sources and Standards

  • General Data Protection Regulation
  • ISO 8000 Data Quality

Frequently Asked Questions

How can you recognise poor data quality?

By high correction effort, contradictory analyses, and many queries within the team.

How long does a clean-up take?

For a clearly defined area, usually two to six weeks.

Is a dedicated role for this worthwhile?

From medium-sized companies upwards, yes, at least as a partial task with a fixed time budget.

Your next step

Would you like to implement this topic for your company? We will examine your starting position, identify the three most effective measures, and provide you with an estimate of effort and timeframe. Write to us at office@zensations.at or start with the free GEO-Check.

Further reading from Zensations

Sources and Standards

Share

More on this topic