Garbage In – Garbage Out: Don’t Skimp On Data Quality

The process of training an ML model involves providing an ML algorithm with training data to learn from. The training data must contain the correct answer, which is known as a target or target attribute. The learning algorithm finds patterns in the training data that map the input data attributes to the target (the answer that you want to predict), and it outputs an ML model that captures these patterns. There is one important aspect that can truly make or break an ML project: data annotation. Today we will take a look at the importance of data annotation, how it affects the data quality, and, ultimately, the impact it will have on your project. Let’s start by taking a look at the ML training process.

What Does the ML Training Process Look Like? 

To train an ML model, you need to specify the following:

• Input training data source
• Name of the data attribute that contains the target to be predicted
• Required data transformation instructions
• Training parameters to control the learning algorithm

As we can see, the first step in the process is to input training data, but this alone can be very difficult to find. There are some open-source datasets such as CIFAR, ImageNet, COCO, and many others. However, these training datasets offer more of a one-size-fits-all approach, which may not be suitable for certain projects. In this case, companies may have to generate the needed data sets themselves. 

A great example of this is a recent client of ours that was working on an AI chatbot for customer service purposes and they needed to generate many different dialogues on many different topics that could be used across industries. We hired 100 agents that created more than 20,000 dialogues on 120 inquiry topics across five different industries. You can read more about this in our case study. This is just a great example of the volume of data you need to create an AI product and how you can generate the training data you need. 

Having said this, while simply obtaining the needed training data sets is important, you also need to annotate this raw data to ensure that the system is able to learn what it needs to. We explore this in the next section. 

How Does Data Annotation Help You Improve Data Quality?

Usually, organizations that are creating high-quality training data sets use three standard methods for ensuring accuracy and consistency: 

• Ground truth – this is used to measure accuracy by comparing annotations (or annotators) to a “gold set” or vetted example. This helps to measure how well a set of annotations from a group or individual matches the benchmark.
• Consensus – measures consistency and agreement amongst a group, and does so by dividing the sum of agreeing data annotations by the total number of annotations.
• QA Process – measures both accuracy and consistency by having an expert review the labels, either by spot-checking or reviewing them all. 

So, how does this help improve your data quality? Well, first of all, it helps make sure that the system is able to identify the needed patterns and perform the desired outcome. This can be something like detecting tumors in medical images, identifying street and road signs for autonomous vehicles, and many other things. Also, it helps you reduce hidden biases that may exist in your data. High-quality training data ensures more accurate algorithms, and it can also help mitigate the potential bias in many AI projects. Bias can manifest as uneven voice or facial recognition performance for different genders, accents, or ethnicities. Fighting bias during your data annotation process is another way to infuse your training data set with quality.

Trust Mindy Support With All of Your Data Annotation Needs

Regardless of the volume of data, you need to be annotated or the complexity of your project, Mindy Support will be able to assemble a team for you to actualize your project and meet deadlines. We are the largest data annotation company in Eastern Europe with more than 2,000 employees in six locations all over Ukraine and in other geographies globally. Our size and location allow us to source and recruit the needed number of candidates within a short time frame and we can scale your team without sacrificing the quality of the work provided. Contact us today to learn more about how we can help you.