Categorizing Toxic Language for a GAFAM Client to Improve Customer Experiences

Content Evaluation and Categorization

This project focused on evaluating behaviors in text and audio communications by systematically labeling content according to specific toxicity categories. The objective was to enhance the customer’s content evaluation solutions by providing detailed and accurate categorization of toxic behaviors. By identifying and labeling various types of toxic content, the client can better measure the performance of their existing evaluation tools and implement improvements. This process aims to ensure a healthier digital environment, improve user experience, and uphold platform integrity by effectively managing and mitigating toxic behavior in communications. 

Online bullying and other inappropriate behavior are big problems on the internet, and our client was looking to tackle online toxicity across various languages by developing a comprehensive and robust evaluation system. To achieve this, they aimed to implement a three-worker consensus model to ensure accuracy and reliability in categorizing toxicity. This model focused on several key categories of inappropriate and harmful content. The project encompasses both text and audio communications, requiring meticulous labeling and analysis to address toxic behavior effectively. By leveraging the expertise of multiple evaluators and covering diverse languages, the client aimed to create a more inclusive and effective solution for mitigating online toxicity.

Mindy Support provided the client with an exceptional blend of cost-efficiency and quality retention, making it an ideal partner for the project. The team at Mindy Support swiftly sourced and recruited candidates, ensuring a seamless integration into the client’s workflow. We maintained high standards by implementing a thorough training process overseen by native language speakers and English-speaking supervisors, ensuring that recruits were proficient in the client’s tools and project specifics. This meticulous approach allowed Mindy Support to deliver outstanding results, meeting the client’s needs for accurate and effective toxicity categorization while keeping costs manageable.

Mindy Support assembled an international team of data annotators to categorize text and audio content in Spanish and Portuguese on several key categories of inappropriate and harmful content. The most challenging aspect of this project was achieving the consensus between 3 evaluators, where 8 from 10 statements (80%) must have the same judgment opinion from 3 of these persons, a statistical measure of inter-rater agreement. To meet this goal, we developed numerous training manuals that provided detailed descriptions for each toxicity category, accompanied by clear examples. This foundational training was supplemented with additional calibration sessions to refine our team’s understanding and application of the categories.

Furthermore, we held regular daily sync sessions with the team to ensure a unified understanding of the categorization criteria and to promptly address any discrepancies or uncertainties. These collaborative efforts and constant communication were crucial in enhancing the team’s consistency and accuracy. As a result, we not only achieved but exceeded the consensus level target, reaching a 0.9 coefficient. This stable delivery and high performance have paved the way for the project’s successful continuation and the inclusion of new locales for production, marking a significant milestone in our partnership with the client.

• Sourced and recruited a team of Spanish and Portuguese native speakers 
• Achieved consensus between 8 from 10 statements (80%)