Ensuring Data Annotation Quality to Help Self-Driving Cars Accurately Navigate Traffic

Quality validation, QA for data annotation

The client is involved in developing software for self-driving cars and the idea of the project was to build software to recognize different types and modes of traffic lights. This would help self-driving cars accurately navigate traffic. They hired one of the data labeling providers to annotate traffic lights and other images, but they required another data annotation vendor for both quality assurance and quality validation services to ensure that all work was done correctly. On the quality assurance side, the prospective vendor needed to find errors and correct them. In terms of quality validation, the potential vendor would need to report on the number of errors for each indicator (geometry accuracy, label correctness) and add more screens with error trends. 

Since the client has already been working with Mindy Support for many years and knows that we have a dedicated QA team with good standing QA processes and procedures and have a highly skilled team to maintain the work described above, they felt comfortable reaching out to us with such a request. Also, It was critical for the client that the quality had to be on a high level after our approval, and no other reworks were acceptable to not delay the project deadline further, which is another reason why they chose to partner with a proven vendor like Mindy Support.

Mindy Support assembled a team of QA professionals who would work with the client’s tool while providing only the services of QA without any extra pre-\post-processing of metadata. We quickly realized that the quality of the annotation work was very low and decided to scale up our team to meet the strict deadlines of the client. 

In addition to the time crunch, we also faced additional quality challenges since the annotations contained an alarming number of mistakes (3-4 times more than after our internal annotation phase) and we had to be very focused to detect and correct mistakes to meet expectations. Correction of mistakes was also crossed with the preparation of statistics for the client on a number of mistakes in each category (geometry and label accuracy), so the team had to cover both needs (correction and statistics accumulation) at the same time.

The client was happy with our speed and the quality. After the approval step, we increased the quality from <80% to >95%, so we met the requirements of the project within the set-up deadline. Also, we helped the client’s team to prepare reports on the quality for another vendor. After those projects, the client was providing all projects on traffic lights annotation only to Mindy. Such projects included annotations with 2D and 3D data.