Tech talk: Visibility and monitoring in deployed ML systems

Machine learning (ML) allows us to create business logic of unprecedented complexity, enabling everything from self-driving cars to hospital-level forecasting of the resources needed to fight COVID-19. This complexity comes at a cost, making it harder to understand and monitor the behaviour of live ML systems.

In this talk, Faculty Lead Data Scientist Scott Stevenson and ML Engineer Victor Zabalza will share insights from their experience building tools and workflows to monitor ML systems, including:

• Why ML systems require us to re-imagine how we monitor software.
• What to monitor and how to monitor it, from population drift or domain shift to historical backtests.
• The importance of integrating data scientists into the monitoring process.

We will assume the audience has a basic understanding of machine learning, but no experience around deployment is necessary.