Training AI to Understand Relationships Between Objects

While machines have become fairly proficient in recognizing objects in the real world, they still need to make connections between them in order to get a step closer to replicating the human thought process. This includes very simple things like a kitchen table with some cups, plates, and utensils on them. While machines would be able to recognize each individual object, they would not be able to perform a command like “Put the fork next to the plate”. 

Researchers at MIT are trying to change all of this and give AI new capabilities that would be very useful in simplifying tasks for humans. Let’s take a closer look at what the MIT researchers are working on and how they are training the AI model.

Teaching Machines to Spot Relationships

The MIT researchers have taken a very interesting approach to train the AI model. They are using text descriptions of various scenes as training data and the AI system would have to generate an image based on the text. For example, let’s say we have something like “The notebook is to the right of the phone.” The system would model all of the scenarios in the training data and all of these pieces would then be combined through an optimization process that generates an image of the scene.

This is called energy-based modeling. With this method, researchers are able to use one energy-based model to encode each relational description, and then compose them together in a way that infers all objects and relationships.

How are Such Models Trained? 

One of the main factors in training AI models is the data itself. There may be situations where the training data is hard to come by, which means that it would need to be generated in-house, which could be very costly and time-consuming. In our case, the researchers would need thousands of scenarios to train the system to properly recognize the objects. They could either perform such tasks themselves or outsource this work to a service provider. 

Mindy Support has experience generating data for AI and ML projects. For example, one of our clients needed to generate dialogues to train their AI chatbot to converse with customers in the insurance, banking, logistics, e-commerce industries, and general customer support. We hired a team of 100 data annotators who generated more than 20,000 dialogues on 120 different inquiry topics across 5 industries. You can read more about the services we provided in our case study. 

Now, after you have the needed data, you need to get it annotated. In our case, text annotation would be needed. To better understand what this involves, let’s look at the image below: 

Image Source

In the image above we see that that part of speech (POS) tagging was done which annotates the functions of the various words within a text. However, if we were to do the type of text annotation required by the AI system that MIT researchers are working on, we would need much more advanced types of data annotation. This includes things like named entity recognition, which annotates entities with proper names. 

We would also need end-to-end entity linking, which is the joint process of first analyzing and annotating entities within a text (named entity recognition), and engaging in entity disambiguation. The latter is when you link named entities to knowledge databases about them. 

What are the Practical Applications of Such Technology? 

Let’s imagine that we have a warehouse with many different boxes that need to be organized on various shelves and other locations. In order to have the machine do the heavy lifting, we would need to give it a command such as “Put the red box next to the yellow box”. Also, if we would like AI to assist humans in the manufacturing process, the machine would need to know the exact position of a certain part, in reaction to other parts. More exact use cases would need specific training data and, consequently, more data annotation. 

Trust Mindy Support With All of Your Data Annotation Needs

If you need to annotate an existing dataset or generate a new one to match your product requirements, consider hiring Mindy Support to do this work for you. We are the largest data annotation company in Eastern Europe with more than 2,000 professionals in eight locations all over Ukraine and in other geographies globally.  Contact us today to learn more about how we can help you.