Data quality as a competitive advantage – an example from the pharmaceutical industry

In this era of data overload, it is vitally important for companies to manage their data effectively in order to remain competitive. Especially when it comes to vendor & service management, incorrect or duplicate data records (duplicates) pose an enormous risk. If, for example, transfers are made more than once or go to the wrong addressees, this results in unnecessary costs and delays. We know from projects with customers from the pharmaceutical industry that this can even delay their market entry. It may also lead to legal consequences if, for example, obligations to provide evidence of speakers are not fulfilled. Clean data management is therefore essential for competitiveness.

Duplicates as a data management challenge

Data management resembles a challenging team sport that requires coordinated interaction and perfect alignment between the stakeholders involved. Gigantic amounts of data generated in different regions and used by different systems frequently lead to a confusing data environment – full of incoherent and inconsistent information. Here’s a fictitious example: Prof. Dr. hc. Gertrude Belle Elion is listed as a duplicate by a pharmaceutical company in various systems and roles, including.

• hc. G. B. Elion, physician at the National Cancer Institute
• G. Belle Elion, professor at Duke University
• Trudy Elion, speaker at a HIV conference

In total, up to 10 different entries for different roles, organizational affiliations and addresses accumulate rapidly per person. When pharmaceutical companies have 500,000 or more system entries, you can imagine the chaos.

So how can this data landscape, which has often evolved over time, be cleaned up quickly and efficiently? And what to do to keep it then properly maintained?

The solution for efficient deduplication: AI & teamwork

Traditionally, deduplication is performed to clean up data via rule-based approaches that compare entries in the system on a pair-by-pair basis. These rules for comparison usually have to be created manually and are quite rigid. It becomes easier when the rule sets are created using an AI-based active learning approach. Here, an algorithm uses confirmed duplicates to learn which records are duplicates and can then identify them automatically. Even business logic is taken into account, so that “desired duplicates” can be distinguished from unnecessary ones. This approach also makes it possible not only to find duplicates and inactive data points, but also to make transparent partner networks and organizational structures in which data is created and used – the basis for continuously clean data management.

Since every organization’s data situation and business logic are different, there are no ready-made, off-the-shelf solutions for this AI-based deduplication. The key component in such projects, in addition to data science expertise, is therefore close collaboration with those who actually work with the data. The experts from vendor management and the servicing department check the quality of the results found with the help of AI at the start of the project and later in regular reviews. In this way, the ML-based solution can be aligned with the company’s individual processes and successively improved with each additional verified case. This ensures that the conditions of the company’s own data landscape are mapped and that local special cases are also correctly identified.