Introduction
Course Scenario & Model Lifecycle
-
Deploying a Model to Production (8:31)
-
Course Scenario - Predicting House Prices (9:27)
-
Setup A - Python Install (Do Not Skip) (3:47)
-
Setup B: Git Installation (Advanced Users Can Skip) (3:02)
-
Course Github Repo & Data (2:38)
-
Download Data and Github Link
-
Setup C: Install Jupyter (Advanced users can skip) (2:13)
-
Setup D: Install Initial Dependencies (Advanced users can skip) (2:19)
-
Introduction to the Dataset & Model Pipeline (13:21)
-
ML System Lifecycle (5:51)
Testing Concepts for ML Systems
-
Overview (0:48)
-
Testing Focus in this Course (1:25)
-
Why Test? (3:44)
-
The Value of Testing
-
Testing Theory (3:47)
-
Testing ML Systems (Important) (6:31)
-
Testing Concepts - Exercise 1 Instructions (12:07)
-
Testing Concepts Exercise 1 - Solution (3:13)
-
Unit Testing Concepts - Exercise 2 Instructions (4:24)
-
Testing Concepts Exercise 2 - Solution (4:04)
-
Testing Concepts - Exercise 3 Instructions (5:11)
-
Testing Concepts Exercise 3 - Solution (5:04)
-
Testing Concepts Exercise 4 - Instructions (3:23)
-
Testing Concepts Exercise 4 - Solution (1:27)
-
Summary (0:26)
Unit Testing ML Systems
-
Overview (0:45)
-
Python Code Conventions (2:26)
-
Intro to Pytest (11:49)
-
Setup - Download Dataset from Kaggle (3:22)
-
Using Tox (5:47)
-
Codebase Overview (13:41)
-
Preprocessing & Feature Engineering Testing Theory (3:24)
-
Unit Testing Preprocessing & Feature Engineering Code (11:06)
-
Git Hygiene
-
Config Tests Theory (3:00)
-
Unit Testing Config Code (9:57)
-
Testing Input Data Theory (3:06)
-
Unit Testing Input Data Code (8:35)
-
Testing Model Quality Theory (2:19)
-
Unit Testing Model Quality Code (10:10)
-
Repo Tooling (2:41)
-
Wrap Up (1:41)
Docker Refresher
Integration Testing ML Systems
Differential Testing
Shadow Mode
-
Shadow Mode Overview (0:44)
-
Shadow Mode Theory (4:23)
-
Testing Models in Production (9:32)
-
Tests in Shadow Deployments (15:08)
-
Shadow Mode Code Overview - DB Setup (13:13)
-
Shadow Mode - Setup Tests (11:40)
-
Shadow Mode - Asynchronous Implementation (4:25)
-
Populate Database with Shadow Predictions (5:22)
-
Jupyter Demo - Setup (5:02)
-
Jupyter Demo - Tests in Shadow Mode (14:18)
Monitoring Metrics with Prometheus
-
Overview (1:36)
-
Why Monitor ML Models? (5:34)
-
Monitoring Theory (8:29)
-
Metrics for ML Systems (6:03)
-
Prometheus & Grafana Overview (6:42)
-
WINDOWS Setup - More Port Mapping (2:27)
-
Basic Prometheus Setup (5:33)
-
Adding Prometheus Metrics (8:22)
-
Setup Grafana (7:21)
-
Infrastructure Level Metrics (6:44)
-
Adding Metrics Monitoring to Our Example Project (7:30)
-
Creating an ML System Grafana Dashboard (15:44)
Monitoring Logs with Kibana
Conclusion
