AI for Audit Findings Classification

Our client is one of the largest contract research organisations (CRO) providing clinical trial services, with annual revenues of over $2 billion and over 11,000 employees in more than 60 countries.

Good Clinical Practice (GCP) requires conducting site audits, i.e. formal examinations of research sites' GCP compliance. The result of such audit is a report containing findings—statements indicating some risks or non-compliances requiring action. For each finding, the auditor draws the root cause, i.e. the underlying reason of each problem or issue, and recommends Corrective and Preventive Actions (CAPAs).

The root causes, grouped into topics and subtopics (eg, Documents, Personnel training, Logistics etc.), can also be used for statistical analysis in order to find the riskiest areas, improve internal procedures and evaluate effectiveness of the procedure changes.

However, the client came to the conclusion that one finding may be related to several root causes. Still, the data schema had only one root case + subtopic + topic assigned to each finding. This led to lack of corrective actions, incomplete statistics and repeat audit findings, as each time only one issue for each finding was recorded and corrected.

Having decided to add several root causes to each finding, the client needed not only to change the database scheme, but also to update all the data extracted from audit reports that were made in the past. Such update would require thousands of man-hours.

The client asked our team to create an AI instrument to solve the challenge.

Parsing finding details

The team used LLM (Large Language Model) to parse the details of each audit finding, uncover root causes and extract all additional extracted root causes. As the project required multiple calling of LLM to analyse each report, we used LangChain* to build pipelines for processing big amounts of data.

Assigning compliance subtopic to extracted root causes

This was done based on business rules and training data.

Assigning compliance topic

The topic is assigned based on the ready topic-subtopic relations data revealed from assignments made by auditors.

Clustering compliance topics and trend analysis

Clustering is a bonus step to reveal trends and patterns of discovered root causes (compliance subtopics). Several topics / subtopics can be assigned to a larger cluster. When all root causes are assigned and grouped, the company gets statistical data that can be used for analysis and revealing temporal trends in numbers of occurrences of certain topics.

The analysis is done using a tool that our developers created for the client. It allows revealing and observing temporal trends for any dataset (data column + date column) and can be used for various tasks.

After training on the existing data about audit findings and their root causes, the AI solution performed a thorough review of the existing 1 GB database of clinical investigator site audit findings and assigned additional root causes with topics and subtopics. The review by AI allowed to save months of time and thousands of man-hours.

The client is going to use the AI as an assistant to site auditors for cross-checking the information received from new audits.

Due to full coverage of topics on each finding, it is possible to get full statistics on the most common topics and subtopics arising during audits. This will lead to better managerial decisions on improvement of the standard operational procedures for future clinical trials.

It is also expected that such enhancement will lead to better coverage of root causes and more comprehensive lists of CAPAs, thus leading to less repeat audit findings and reducing costs of the clinical trials.