Machine Learning Implementation and Project Management: A How-To Guide

Meet our machine learning experts

Łukasz Grzybowski is one of us, a seasoned professional who’s currently the Head of Machine Learning and Data Engineering at STX Next. Łukasz has over 9 years of experience building solutions and managing a wide range of products for clients in banking, ecommerce, and other notable industries.

Willy Picard is the Head of the Machine Learning Team at Egnyte. He leads their team to improve products with machine learning, using machine learning to analyze file content and detect user behavior anomalies. Before joining Egnyte, Willy was actively involved in academic research for about 15 years at the Poznań University of Economics and worked in a machine learning-related role at Samsung for 4 years.

Here are the recordings from our two live sessions with Łukasz and Willy:

Part I

Part II

But if you’d rather read what we’ve learned from our conversations with them, you’ll find all that precious knowledge collected below!

When should you use machine learning? 4 key questions to consider

Without a doubt, machine learning can be a major growth driver for many organizations and help you automate certain processes. However, it isn’t always the right solution. The machine learning life cycle can be long, complex, and tedious. It may also not be a budget-friendly solution.

Therefore, it’s important to consider if machine learning really is the best solution for you before building software with it.

While there is no blueprint for figuring this out, answering the following questions will help:

1. Are there alternative ways of solving the problem?

Machine learning is often costly and time-consuming. Thus, if it’s possible to program the solution via traditional coding or use other alternatives, you should consider those.

2. Is the problem well-defined?

If the problem you’re trying to solve is not well-defined, you’re less likely to get the best results from machine learning. A good understanding of the problem is essential to determine the inputs, outputs, and model specifications.

3. Is the solution data-dependent?

Machine learning is best used when the problem is data-driven. Using a more traditional, non-ML, rule-based system may be challenging and inefficient when working with large data sets. This makes machine learning a viable alternative when your problem can only be solved with lots of data.

4. Do the benefits outweigh the costs?

It’s essential to weigh the cost and effort machine learning projects require against the benefits. If you don’t have enough resources to start and complete machine learning projects, you should consider alternatives. Machine learning models are ideal for large-scale problems that justify the cost.

What is the machine learning life cycle?

To guide your team toward building top-performing models, you must first grasp the machine learning life cycle. The machine learning life cycle means all the steps involved from the moment a request for a model is made to deploying and improving the model.

Here’s a breakdown of the machine learning life cycle.

Stage #1: Understanding the business needs

The first step in machine learning implementation is understanding and defining the business needs. You need a clear understanding of the business requirements to build self-learning models that will solve future problems.

If you’ve been assigned to manage a machine learning project, the following tips will help you understand the business needs:

Define the objectives

Objectives differ for each business or user. For example, it could be maximizing profit for one organization and increasing customer retention for another. You need to identify each client’s or user’s goals. Don’t hesitate to ask the decision-makers of the organization or the potential end users for input.

Identify the problem

Most times, the decision to adopt machine learning is based on a problem. For example, an organization may need models to determine its operational expenses for the next year, automatically generate optimal pricing for their products, or predict employee attrition within a period. Hence, you need to identify the problem to build a model that works.

Set quantifiable goals

Machine learning projects may run indefinitely if you fail to set measurable goals. You may spend donkey’s years on research and development and tens of thousands of dollars if the project goals aren’t defined early on. It’s best to establish metrics for evaluating and measuring performance. You should note the immediate results the client or user is expecting from the models and work toward achieving those results.

Stage #2: Working with data

Working with data sets is unavoidable when building machine learning models. You’ll need data to develop, train, and test the model. The type of data available will also determine the model specifications.

Identifying the data you need and preparing the data for machine learning models is critical to the success of machine learning implementation. Lucky for you, we’ve identified the essential data preparation steps below.

1. Data gathering

Contrary to popular belief, you don’t need complex data to build apps with machine learning. You can also gather data from different sources. Most machine learning models are built using historical data collected from the business or user. However, if you don’t have historical data to work with, you can use artificial or public data.

Regardless of your data source, make sure that you obtain the necessary permissions to access the data. If the data contains sensitive or personal information, double-check that you’re processing data in compliance with data protection regulations.

Finally, the data should be vetted to ensure that it’s accurate and secure.

2. Data cleansing and transformation

Data cleansing and transformation involves inspecting the collected data for common mistakes such as missing values, duplicate entries, or redundant variables. You also need to confirm that the data is relevant for building the desired application.

Data cleansing requires an understanding of the problem and user’s journey to be done effectively. You will also gain key insights into the type of data in your possession and this knowledge will come in handy for future projects.

You may need to use distributed processing and dedicated frameworks if you’re processing a large volume of data. The time taken to build these processes and frameworks is quite a lot. However, it’s important to do a thorough data analysis. Don’t rush the process, since you’ll need quality data to build, test, and train your models.

3. Data annotation

Models can be likened to children learning a language. You’ll need to correct them when they mispronounce words or use the wrong words. In the data preparation process, data annotation can be described as the data learning process.

Models need to be able to recognize data to be properly trained. For instance, if you’re building a model that’s supposed to discern bees from trees, you’ll need to supply it with relevant data, such as images of bees and trees.

Annotating the data means manually going through the pictures and tagging each of them as either a bee or a tree—or a third category of "neither a bee nor a tree" if you want the model to be able to predict that, too.

Annotated data helps the models interpret the data quickly and improves the model’s accuracy in the long run. But the annotation process can be tedious if you’re working with a large volume of data. You can save time and limit the cost by outsourcing this task via Amazon Mechanical Turk or Appen. You can also crowdsource it.

Stage #3: Building the model

Once you’ve prepared your data, it’s time to build the model. Building the model is simply matching the business needs with the available data. Of course, it’s more technical than that in reality and involves actual programming. But, ultimately, the tool developed should be able to solve the problem with the available data sets.

Here are the key steps in building machine learning models:

1. Finding the best-performing algorithm for the problem

There are several algorithms for building machine learning models. However, you should use the algorithm that best solves the problem and is compatible with your data sets. There may be a pre-trained model that can solve the problem.

2. Establishing boundaries for the product

It’s best to establish boundaries at this point to avoid common problems such as overfitting. Important boundaries to establish include the timeline and the budget. If your team has been briefed on the expected turnaround time for the product and the resources available, they’ll be able to adopt a plan to work within these boundaries.

3. Model training

Model training is essential for determining the accuracy of your model. If you don’t train the model with the right data sets, it may not meet expectations when processing real data.

Again, it’s important to set boundaries on the time and resources for model training. A single epoch may run for a long time depending on the volume of data. Thus, it’s important to determine how many epochs are needed to train the model.

Besides, model training is a continuous process. Data sets, user needs, and the market may change after your model has been deployed into production. When that happens, you may need to retrain your model after deployment to keep up.

4. Model validation and testing

After training your model, you should test it to make sure that it works. This can be done by testing different data sets on the model. Model validation is vital to ensure that the model is working and ready for deployment.

An understanding of the business and clearly defined metrics makes the model testing process more efficient. You need to know the goals for the project to find out whether the model is working or not. However, no model is 100% accurate, so try not to cherry-pick during testing.

Test with as many data sets as possible before you’re done. The goal should be to determine if the model works correctly enough to solve the problem and not if it’s always correct.

Model testing can be quite repetitive and time-consuming. You’ll need to keep records as you test with different data sets. Tools like DVC or Weights & Biases are great for keeping track of your tests.

Stage #4: Model deployment

After training and testing the model, the next step is to deploy the model into production. Model deployment is essentially introducing your model to the market. It’s the process of making your technology available to users.

Before deploying models, consider what additional tasks need to be done. You may need to engage extra hands to deploy the model into production or perform certain pre-processing activities. Also, you’ll need to minimize the cost of deploying the model, which, if you let it spin out of control, may far exceed the production cost.

Here are some tips for deploying models into production:

1. Build a cost model

Depending on the scale of data to be processed by the model, models in production can cost from tens of thousands to millions of dollars. Building a cost model will help you figure out the cost-effective way to deploy the model.

The cost model doesn’t have to be fancy. It can be a basic Microsoft Excel sheet showing the type and number of machines you can use to deploy the model, and the infrastructure such as CPUs and GPUs needed to deploy the model. With this analysis, you’ll be able to arrive at the most cost-effective way to deploy the model for optimal results.

2. Build a mock version

Machine learning models can be retrained as many times as possible to improve accuracy. So, instead of trying to build the perfect product, you should rather create a mock product to test and monitor your model’s performance.

You can test this internally before building the final product. This will reduce the product’s time-to-market and lower the cost.

3. Deploy the model in a simple format

Machine learning models can be deployed in different formats such as:

Web servers

Users can submit their data to the web server, which can then be processed.

Custom functions

You can extract the rules and create custom functions that can be integrated on different platforms.

APIs

You can deploy the machine learning model via an API.

It’s advisable to deploy with the format that is suitable for the users as you can always use a more complex format if the need arises.

4. Collaborate with other teams and users

Deploying models into production involves processes that are best managed by DevOps, which means you’ll need to talk with DevOps software engineers. You may also need to engage the UI/UX team to work on a more user-friendly interface for the model.

You’ll also need to collaborate with the clients or users. It’s essential to ensure that the users understand what the technology can do. In addition, they may have suggestions that can help improve its effectiveness. If you’re building the machine learning model for a client, they may want to see preliminary results before signing off.

Stage #5: Model maintenance

The final stage after deployment is maintaining models in production. Models in production interact with the real world and real data sets. Hence, skipping this stage may be costly. Many issues may come up after the model is deployed that can only be spotted with proper maintenance.

Sometimes, models may not be performing their intended functions or delivering the expected answers. Therefore, you should always have a plan for maintaining your models after deployment.

Models can be maintained through monitoring and extension as discussed below.

1. Model monitoring

Monitoring models help you spot issues with the models and enable you to intervene on time. This way, you’ll know when your model is not performing as expected and retrain your model when necessary.

So, what exactly should you monitor?

Inputs

You should compare the input data with the training data. Take note of slight differences, such as brighter images or plain backgrounds. This will help you justify the outputs.

Performance

You’ll need to make sure that the models work. You can use data sets similar to the ideal data sets to test the models and ensure that they work.

Accuracy

You’ll need to make sure that your models can actually predict observations correctly. You don’t want to leave a model that gives incorrect answers in production.

2. Techniques for monitoring models

The essence of machine learning is to build a model that works with little human intervention. Thus, monitoring may seem like a tedious task for machine learning projects, but you can make it a lot easier with these techniques:

Create an alert or monitoring system

There are many helpful ways to track model performance that will require collaboration with the UI/UX team and DevOps. For instance, you may need to create dashboards or alert systems to notify the users of failures or inaccuracy of the models in production. You’ll surely learn that the Product Design department is vital to achieving this goal.

Use feedback from the users

Users are often interested in improving the models they use. Hence, you can explore the users’ feedback to figure out areas of improvement on the models. You may request feedback from users through websites and social media pages by incorporating a feedback request mechanism on these platforms.

Make use of available software

Most of the techniques described above may be suitable for small models and ideal when there’s enough time and budget to keep monitoring and retraining. However, if you’re pressed for time and need to work with a lean budget, you can make use of open-source monitoring software.

3. Model modification

Every organization or user hopes for a model that can keep up with the changes in the users’ needs, new situations, and new environments. However, machine learning models have been programmed to solve problems using predefined data sets and give predictions or answers based on these data sets.

Models often make mistakes. They may process data similar to the predefined data sets. For instance, a model programmed to identify CVs may mistakenly identify the title page of a book as a CV. Clients may also request that new features be added to the model.

In such cases, you may need to modify the models after deployment to cover broader data sets, improve accuracy, and eliminate false positives/negatives.

If you find yourself in this situation, consider doing the following:

Identify the problem

Locate the issues with the model or the new features to be added to the model.

Decide on the best approach to solve the problem

You should decide if the model needs to be retrained or extended to solve the problem or meet the user’s needs.

Fix it

Using the best approach that you’ve identified, fix the problem.

Final thoughts on implementing and managing machine learning projects

Machine learning is more complex than traditional software development methods. Hence, managing a machine learning project can be quite challenging. But if you have a solid guide to follow, there are ways to make it a walk in the park.

We hope that the roadmap we’ve provided in this article will make machine learning implementation and management a lot easier for you. However, you can always reach out to us if you have any questions—we’ll be happy to answer them.

You’ll find plenty of other useful technical guides on our website. Here are a couple of examples we recommend:

And if you’re looking to extend your machine learning team or need any support building your next machine learning project, we’re here to help.