mly s00 table of contents

Why Machine Learning Strategy

How to use this book to help your team

Prerequisites and Notation

Scale drives machine learning progress

Your development and test sets

Your dev and test sets should come from the same distribution

How large do the dev/test sets need to be?

Establish a single-number evaluation metric for your team to optimize

Optimizing and satisficing metrics

Having a dev set and metric speeds up iterations

When to change dev/test sets and metrics

Takeaways: Setting up development and test sets

Build your first system quickly, then iterate

Error analysis: Look at dev set examples to evaluate ideas

Evaluate multiple ideas in parallel during error analysis

If you have a large dev set, split it into two subsets, only one of which you look at

How big should the Eyeball and Blackbox dev sets be?

Takeaways: Basic error analysis

Bias and Variance: The two big sources of error

Examples of Bias and Variance

Comparing to the optimal error rate

Addressing Bias and Variance

Bias vs

Techniques for reducing avoidable bias

Techniques for reducing Variance

Error analysis on the training set

Diagnosing bias and variance: Learning curves

Plotting training error

Interpreting learning curves: High bias

Interpreting learning curves: Other cases

Plotting learning curves

Why we compare to human-level performance

How to define human-level performance

Surpassing human-level performance

Why train and test on different distributions

Whether to use all your data

Whether to include inconsistent data

Weighting data

Generalizing from the training set to the dev set

Addressing Bias and Variance

Addressing data mismatch

Artificial data synthesis

The Optimization Verification test

General form of Optimization Verification test

Reinforcement learning example

The rise of end-to-end learning

More end-to-end learning examples

Pros and cons of end-to-end learning

Learned sub-components

Directly learning rich outputs

Error Analysis by Parts

Beyond supervised learning: What’s next?

Building a superhero team - Get your teammates to read this

Big picture

Credits