Learn how to test & monitor production machine learning models.
What is model testing?
You’ve taken your model from a Jupyter notebook and rewritten it in your production system. Are you sure there weren’t any mistakes when you moved from the research environment to the production system? How can you control the risk before your deployment? ML-specific unit, integration and differential tests can help you to minimize the risk.
What is model monitoring?
You’ve deployed your model to production. OK now what? Is it working as you expect? How do you know? By monitoring models, we can check for unexpected changes in:
- Incoming data
- Model quality
- System operations
When we think about data science, we think about how to build machine learning models, which algorithm will be more predictive, how to engineer our features and which variables to use to make the models more accurate. However, how we are going to actually test & monitor these models in a production system is often neglected, . Only when we can effectively monitor our production models can we determine if they are performing as we expect.
Why take this course?
This is the first and only online course where you can learn how to test & monitor machine learning models. The course is comprehensive, and yet easy to follow. Throughout this course you will learn all the steps and techniques required to effectively test & monitor machine learning models professionally.
In this course, you will have at your fingertips the sequence of steps that you need to follow to test & monitor a machine learning model, plus a project template with full code, that you can adapt to your own models.
What is the course structure?
Part 1: Testing
The course begins from the most common starting point for the majority of data scientists: a Jupyter notebook with a machine learning model trained in it. We gradually build up the complexity, testing the model first in the Juyter notebook and then in a realistic production code base. Hands-on exercises are interspaced with relevant and actionable theory.
Part 2: Shadow Mode
We explain the theory & purpose of deploying a model in shadow mode to minimize your risk, and walk you through an example project setup.
Part 3: Monitoring
We take you through the theory & practical application of monitoring metrics & logs for ML systems.
- This course does not cover model deployment (we have a separate course dedicated to that topic)
Who are the instructors?
We have gathered a fantastic team to teach this course. Sole is a leading data scientist in finance and insurance, with 3+ years of experience in building and implementing machine learning models in the field, and multiple IT awards and nominations. Chris is a tech lead & ML software engineer with enormous experience in building APIs and deploying machine learning models, allowing business to extract full benefit from their implementation and decisions.
Who is this course for?
- Data Scientists who want to know how to test & monitor their models beyond in production
- Software engineers who want to learn about Machine Learning engineering
- Machine Learning engineers who want to improve their testing & monitoring skills
- Data Engineers looking to transition to ML engineering
- Lovers of open source technologies
How advanced is this course?
This is an advanced level course, and it requires you to have experience with Python programming and git. How much experience? It depends on how much time you would like to set aside to go ahead and learn those concepts that are new to you. To give you an example, we will work with Python environments, we will work with object oriented programming, we will work with the command line to run our scripts, and we will checkout code at different stages with git. You don’t need to be an expert in all of these topics, but you need a reasonable working knowledge. We also work with Docker a lot, though we will provide a recap of this tool.
For those relatively new to software engineering, the course will be challenging. We have added detailed lecture notes and references, so we believe that those missing some of the prerequisites can take the course, but keep in mind that you will need to put in the hours to read up on unfamiliar concepts. On this point, the course slowly increases in complexity, so you can see how we pass, gradually, from the familiar Jupyter notebook, to the less familiar production code, using a project-based approach which we believe is optimal for learning. It is important that you follow the code, as we gradually build it up.
Still not sure if this is the right course for you?
Here are some rough guidelines:
Never written a line of code before: This course is unsuitable
Never written a line of Python before: This course is unsuitable
Never trained a machine learning model before: This course is unsuitable. Ideally, you have already built a few machine learning models, either at work, or for competitions or as a hobby.
Never used docker before: The second part of the course will be very challenging. You need to be ready to read up on lecture notes & references.
Have only ever operated in the research environment: This course will be challenging, but if you are ready to read up on some of the concepts we will show you, the course will offer you a great deal of value.
Have a little experience writing production code: There may be some unfamiliar tools which we will show you, but generally you should get a lot from the course.
Non-technical: You may get a lot from just the theory lectures, so that you get a feel for the challenges of ML testing & monitoring, as well as the lifecycle of ML models. The rest of the course will be a stretch.
To sum up:
With more than 70 lectures and 8 hours of video this comprehensive course covers every aspect of model testing & monitoring. Throughout the course you will use Python as your main language and other open source technologies that will allow you to host and make calls to your machine learning models.
We hope you enjoy it and we look forward to seeing you on board!
- Deploying a Model to Production (8:31)
- Course Scenario - Predicting House Prices (9:27)
- Setup A - Python Install (Do Not Skip) (3:47)
- Setup B: Git Installation (Advanced Users Can Skip) (3:02)
- Course Github Repo & Data (2:38)
- Download Data and Github Link
- Setup C: Install Jupyter (Advanced users can skip) (2:13)
- Setup D: Install Initial Dependencies (Advanced users can skip) (2:19)
- Introduction to the Dataset & Model Pipeline (13:21)
- ML System Lifecycle (5:51)
- Overview (0:48)
- Testing Focus in this Course (1:25)
- Why Test? (3:44)
- The Value of Testing
- Testing Theory (3:47)
- Testing ML Systems (Important) (6:31)
- Testing Concepts - Exercise 1 Instructions (12:07)
- Testing Concepts Exercise 1 - Solution (3:13)
- Unit Testing Concepts - Exercise 2 Instructions (4:24)
- Testing Concepts Exercise 2 - Solution (4:04)
- Testing Concepts - Exercise 3 Instructions (5:11)
- Testing Concepts Exercise 3 - Solution (5:04)
- Testing Concepts Exercise 4 - Instructions (3:23)
- Testing Concepts Exercise 4 - Solution (1:27)
- Summary (0:26)
- Overview (0:45)
- Python Code Conventions (2:26)
- Intro to Pytest (11:49)
- Setup - Download Dataset from Kaggle (3:22)
- Using Tox (5:47)
- Codebase Overview (13:41)
- Preprocessing & Feature Engineering Testing Theory (3:24)
- Unit Testing Preprocessing & Feature Engineering Code (11:06)
- Git Hygiene
- Config Tests Theory (3:00)
- Unit Testing Config Code (9:57)
- Testing Input Data Theory (3:06)
- Unit Testing Input Data Code (8:35)
- Testing Model Quality Theory (2:19)
- Unit Testing Model Quality Code (10:10)
- Repo Tooling (2:41)
- Wrap Up (1:41)
- Shadow Mode Overview (0:44)
- Shadow Mode Theory (4:23)
- Testing Models in Production (9:32)
- Tests in Shadow Deployments (15:08)
- Shadow Mode Code Overview - DB Setup (13:13)
- Shadow Mode - Setup Tests (11:40)
- Shadow Mode - Asynchronous Implementation (4:25)
- Populate Database with Shadow Predictions (5:22)
- Jupyter Demo - Setup (5:02)
- Jupyter Demo - Tests in Shadow Mode (14:18)
- Overview (1:36)
- Why Monitor ML Models? (5:34)
- Monitoring Theory (8:29)
- Metrics for ML Systems (6:03)
- Prometheus & Grafana Overview (6:42)
- WINDOWS Setup - More Port Mapping (2:27)
- Basic Prometheus Setup (5:33)
- Adding Prometheus Metrics (8:22)
- Setup Grafana (7:21)
- Infrastructure Level Metrics (6:44)
- Adding Metrics Monitoring to Our Example Project (7:30)
- Creating an ML System Grafana Dashboard (15:44)
Chris Samiullah - Tech Lead & ML Engineer
My name is Chris. I'm a professional software engineer from the UK. I've been writing code for 8 years, and for the past three years, I've focused on scaling machine learning applications. I've done this at fintech and healthtech companies in London, where I've worked on and grown production machine learning applications used by hundreds of thousands of people. I've built and maintained machine learning systems which make credit-risk and fraud detection judgements on over a billion dollars of personal loans per year for the challenger bank Zopa. I currently work on systems for predicting health risks for patients around the world at Babylon Health.
Soledad Galli - Lead Data Scientist.
Soledad Galli is a lead data scientist with experience in finance and insurance. She received a Data Science Leaders Award in 2018 and was selected “LinkedIn’s voice” in data science and analytics in 2019. Soledad is passionate about sharing knowledge and helping others succeed in data science.
As a data scientist in Finance and Insurance companies, Sole researched, developed and put in production machine learning models to assess Credit Risk, Insurance Claims and to prevent Fraud, leading in the adoption of machine learning in the organizations.
Sole is passionate about enabling people to step into and excel in data science, that is why she mentors data scientists and speaks at data science meetings regularly, and teaches online courses on machine learning.
Questions? Send me an email: [email protected]