Major Challenges for Machine Learning Projects
While many researchers and experts alike agree that we are living in the prime years of artificial intelligence, there are still a lot of obstacles and challenges that will have to be overcome when developing your project. Groundbreaking developments in machine learning algorithms, such as the ones in AlphaGo, are conquering new frontiers and proving once and for all that machines are capable of thinkings and planning out their next moves. Even though AlphaGo and its successors are very advanced and niche technologies, machine learning has a lot of more practical applications such as video suggestions, predictive maintenance, driverless cars, and many others.
With all of this in mind, let’s take a look at some of the obstacles companies are dealing with on their way towards developing machine learning technology.
Machine learning is an exciting and evolving field, but there are not a lot of specialists who can develop such technology. Even a data scientist who has a solid grasp of machine learning processes very rarely has enough software engineering skills. While such a skills gap shortage poses some problems for companies, the demand for the few available specialists on the market who can develop such technology is skyrocketing as are the salaries of such experts. According to a recent New York Time’s report, people with only a few years of AI development experience earned as much as half a million dollars per year, with the most experienced one earning as much as some NBA superstars.
In addition to the development deficit, there is a deficit in the people who can perform the data annotation. If we take a look at the healthcare industry, for example, there are only about 30,000 cardiologists in the US and somewhere between 25 and 40,000 radiologists. Since there are so few radiologists and cardiologists, they do not have time to sit and annotate thousands of x-rays and scans. This is why a lot of companies are looking abroad to outsource this activity given the availability of talent at an affordable price. Speaking of costs, this is another problem companies are grappling with. Let’s take a look.
High Costs of Development
While we already mentioned the high costs of attracting AI talent, there are additional costs of training the machine learning algorithms. This process involves lots of hours of data annotation and the high costs incurred could potentially derail projects. This is why a lot of companies are opting to outsource the data annotation services, thus allowing them to focus more attention on developing their products. This is especially popular in the automotive, healthcare and agricultural industries, but can be applied to others as well. Therefore, in order to mitigate some of the development costs, outsourcing is becoming a go-to solution for businesses worldwide.
As we know, data is absolutely essential to train machine learning algorithms, but you have to obtain this data from somewhere and it is not cheap. Creating a data collection mechanism that adheres to all of the rules and standards imposed by governments is a difficult and time-consuming task. You need to plan out in advance how you will be classifying the data, ranking, cluster regression and many other factors. Even when the data is obtained, not all of it will be useable. In order to refine the raw data, you will have to perform attribute and record sampling, in addition to data decomposition and rescaling. Even if you have a lot of room to store the data, this is a very complicated, time-consuming and expensive process.
Furthermore, even the raw data must be reliable. If the data being fed into the algorithms is “poisoned” then the results could be catastrophic. For example, one time Microsoft released chatbot and taught it by letting it communicate with users on Twitter. the project was a complete disaster because people quickly taught it to curse and use phrases from Mein Kampf which cause Microsoft to abandon the project within 24 hours.
While this might be an extreme example, it further underscores the need to obtain reliable data because the success of the project depends on it. Therefore, it is important to have a human factor in place to monitor what the machine is doing.
Working with Young Technology
While it may seem that all of the developments in AI and machine learning are something out of a sci-fi movie, the reality is that the technology is not all that mature. Even if we take environments such as TensorFlow from Google or the Open Neural Network Exchange offered by the joint efforts of Facebook and Microsoft, they are being advanced, but still very young. To put all of this in perspective, the first TensorFlow was released a couple of years ago in 2017. Web application frameworks have a lot more history to them since they are around 15 years old. These include frameworks such as Django, Python, Ruby-on-Rails and many others.
Young technology is a double-edged sword. In one hand, it incorporates the latest technology and developments, but on the other hand, it is not production-ready. Still, companies realize the potential benefits of AI and machine learning and want to integrate it into their business offering. This means that businesses will have to make adjustments, upgrades, and patches as the technology becomes more developed to make sure that they are getting the best return on their investment.
Patience is a Virtue
Regular enterprise software development takes months to create given all of the processes involved in the SDLC. These include identifying business goals, determining functionality, technology selection, testing, and many other processes. machine learning is much more complicated and includes additional layers to it. The most notable difference is the need to collect the data and train the algorithms. In a traditional software development environment, an experienced team can provide you with a fairly specific timeline in terms of when the project will be completed. In a machine learning environment, they’re a lot more uncertainties, which makes such forecasting difficult and the project itself could take longer to complete.
The reason is that even the best machine learning experts have no idea in terms of how the deep learning algorithms will act when analyzing all of the data sets. This also means that they can not guarantee that the training model they use can be repeated with the same success.
Computers themselves have no ethical reasoning to them. A machine learning algorithm can fulfill any task you give it, but without taking into account the ethical ramification. For example, if you give it a task of creating a budget for your company. It could put more emphasis on business development and not put enough on employee retention efforts, insurance and other things that do not grow your business.
Furthermore, the opinion on what is ethical and what is not to change over time. For example, machine learning technology is being used by governments for surveillance purposes. While this might be acceptable in one country, it might not be somewhere else. The same is true for more widely used techniques such as personalized recommendations. When you shop online, browse through items and make a purchase the system will recommend you additional, similar items to view. While some people might think that such a service is great, others might view it as an invasion of privacy.
It is clear that as time goes on we will be able to better hone machine learning technology to the point where it will be able to perform both mundane and complicated tasks better than people. Therefore, it is important to put all of these issues in perspective. The technology is still very young and all of these problems can be fixed in the near future.
February 12th, 2020Mindy News Blog