Major Challenges for Machine Learning Projects

While many researchers and experts alike agree that we are living in the prime years of artificial intelligence, there are still a lot of obstacles and challenges that will have to be overcome when developing your project. Groundbreaking developments in machine learning algorithms, such as the ones in AlphaGo, are conquering new frontiers and proving once and for all that machines are capable of thinking and planning out their next moves. Even though AlphaGo and its successors are very advanced and niche technologies, machine learning has a lot of more practical applications such as video suggestions, predictive maintenance, driverless cars, and many others. 

With all of this in mind, let’s take a look at some of the obstacles companies are dealing with on their way towards developing machine learning technology.

Talent Deficit

Machine learning is an exciting and evolving field, but there are not a lot of specialists who can develop such technology. Even a data scientist who has a solid grasp of machine learning processes very rarely has enough software engineering skills. While such a skills gap shortage poses some problems for companies, the demand for the few available specialists on the market who can develop such technology is skyrocketing as are the salaries of such experts. According to a recent New York Time’s report, people with only a few years of AI development experience earned as much as half a million dollars per year, with the most experienced one earning as much as some NBA superstars. Mindy Support can assemble a team for you that will be a lot cheaper than hiring one in your local market.

In addition to the development deficit, there is a deficit in the people who can perform the data annotation. If we take a look at the healthcare industry, for example, there are only about 30,000 cardiologists in the US and somewhere between 25 and 40,000 radiologists. Since there are so few radiologists and cardiologists, they do not have time to sit and annotate thousands of x-rays and scans. This is why a lot of companies are looking abroad to outsource this activity given the availability of talent at an affordable price. Speaking of costs, this is another problem companies are grappling with. Let’s take a look.

High Costs of Development

While we already mentioned the high costs of attracting AI talent, there are additional costs of training the machine learning algorithms. This process involves lots of hours of data annotation and the high costs incurred could potentially derail projects. This is why a lot of companies are opting to outsource the data annotation services, thus allowing them to focus more attention on developing their products. This is especially popular in the automotive, healthcare and agricultural industries, but can be applied to others as well. Therefore, in order to mitigate some of the development costs, outsourcing is becoming a go-to solution for businesses worldwide.

Obtaining Data

As we know, data is absolutely essential to train machine learning algorithms, but you have to obtain this data from somewhere and it is not cheap. Creating a data collection mechanism that adheres to all of the rules and standards imposed by governments is a difficult and time-consuming task. You need to plan out in advance how you will be classifying the data, ranking, cluster regression and many other factors. Even when the data is obtained, not all of it will be useable. In order to refine the raw data, you will have to perform attribute and record sampling, in addition to data decomposition and rescaling. Even if you have a lot of room to store the data, this is a very complicated, time-consuming and expensive process.

Furthermore, even the raw data must be reliable. If the data being fed into the algorithms is “poisoned” then the results could be catastrophic. For example, one time Microsoft released chatbot and taught it by letting it communicate with users on Twitter. the project was a complete disaster because people quickly taught it to curse and use phrases from Mein Kampf which cause Microsoft to abandon the project within 24 hours.

While this might be an extreme example, it further underscores the need to obtain reliable data because the success of the project depends on it. Therefore, it is important to have a human factor in place to monitor what the machine is doing.

If you are having issues finding the data you need, there is another option of hiring an outsourcing company to help you create it. For example, at Mindy Support, we recently worked on a project where the client was creating a chatbot for customer service purposes and they needed all kinds of possible dialogues that could exist between customer and customer service agents. We were able to generate 120,000 dialogues on 120 topics across 5 industries. While this was a challenging project, we had the necessary human resources, project management and QA professionals in place to make sure the job was done right. 

Finding Quality Data Annotators

Simply finding the data is not enough since it needs to be prepared for processing through various data annotation methods. This is a very important aspect of the project since the accuracy of the end product will depend on the quality of work done by the data annotators. Even though data annotation is very important, it is very tedious and time consuming work, which is why a lot of companies choose to outsource such work to a service provider. 

Mindy Support has extensive experience annotating all kinds of data for machine learning projects in the automotive, agricultural, healthcare and many other industries. Some of our interesting use cases include: 

• Detecting and marking traffic signs and traffic lights to train machine learning algorithms in self-driving cars
• Labeling the position of farm animals to allow the computer vision camera to determine whether or not an animal is sick
• Annotating of red and blue cells in immunological videos to track the movement of these cells  to assist further research in immunology

Working with Young Technology

While it may seem that all of the developments in AI and machine learning are something out of a sci-fi movie, the reality is that the technology is not all that mature. Even if we take environments such as TensorFlow from Google or the Open Neural Network Exchange offered by the joint efforts of Facebook and Microsoft, they are being advanced, but still very young. To put all of this in perspective, the first TensorFlow was released a couple of years ago in 2017. Web application frameworks have a lot more history to them since they are around 15 years old. These include frameworks such as Django, Python, Ruby-on-Rails and many others.

Young technology is a double-edged sword. In one hand, it incorporates the latest technology and developments, but on the other hand, it is not production-ready. Still, companies realize the potential benefits of AI and machine learning and want to integrate it into their business offering. This means that businesses will have to make adjustments, upgrades, and patches as the technology becomes more developed to make sure that they are getting the best return on their investment.

Patience is a Virtue

Regular enterprise software development takes months to create given all of the processes involved in the SDLC. These include identifying business goals, determining functionality,  technology selection, testing, and many other processes. machine learning is much more complicated and includes additional layers to it. The most notable difference is the need to collect the data and train the algorithms. In a traditional software development environment, an experienced team can provide you with a fairly specific timeline in terms of when the project will be completed. In a machine learning environment, they’re a lot more uncertainties, which makes such forecasting difficult and the project itself could take longer to complete.

The reason is that even the best machine learning experts have no idea in terms of how the deep learning algorithms will act when analyzing all of the data sets. This also means that they can not guarantee that the training model they use can be repeated with the same success.

Ethical Implications

Computers themselves have no ethical reasoning to them. A machine learning algorithm can fulfill any task you give it, but without taking into account the ethical ramification. For example, if you give it a task of creating a budget for your company. It could put more emphasis on business development and not put enough on employee retention efforts, insurance and other things that do not grow your business.

Furthermore, the opinion on what is ethical and what is not to change over time. For example, machine learning technology is being used by governments for surveillance purposes. While this might be acceptable in one country, it might not be somewhere else. The same is true for more widely used techniques such as personalized recommendations. When you shop online, browse through items and make a purchase the system will recommend you additional, similar items to view. While some people might think that such a service is great, others might view it as an invasion of privacy.

It is clear that as time goes on we will be able to better hone machine learning technology to the point where it will be able to perform both mundane and complicated tasks better than people. Therefore, it is important to put all of these issues in perspective. The technology is still very young and all of these problems can be fixed in the near future.

Mindy Support Provides Comprehensive Data Annotation Services

Mindy Support has extensive experience actualizing data annotation projects in a wide variety of industries and complexities. We are one of the largest BPO providers in Eastern Europe with more than 2,000 employees in six locations all over Ukraine. Our size and location allow us to source and recruit the necessary number of candidates quickly and we can also scale your team quickly without sacrificing the quality of the annotation. Contact us today to find out how we can help you.