Human-in-the-Loop Machine Learning
August 23, 2020
- Outline
- welcome
- 1 Introduction to Human-in-the-Loop Machine Learning
- 1.1 The Basic Principles of Human-in-the-Loop Machine Learning
- 1.2 Introducing Annotation
- 1.2.1 Simple and more complicated annotation strategies
- 1.2.2 Plugging the gap in data science knowledge
- 1.2.3 Quality human annotations: why is it hard?
- 1.3 Introducing Active Learning: improving the speed and reducing the cost of training data
- 1.3.1 Three broad Active Learning sampling strategies: uncertainty, diversity, and random
- 1.3.2 What is a random selection of evaluation data?
- 1.3.3 When to use Active Learning?
- 1.4 Machine Learning and Human-Computer Interaction
- 1.4.1 User interfaces: how do you create training data?
- 1.4.2 Priming: what can influence human perception?
- 1.4.3 The pros and cons of creating labels by evaluating Machine Learning predictions
- 1.4.4 Basic principles for designing annotation interfaces
- 1.5 Machine Learning-Assisted Humans vs Human-Assisted Machine Learning
- 1.6 Transfer learning to kick-start your models
- 1.7 What to expect in this text
- 1.8 Summary
- 2 Getting Started with Human-in-the-Loop Machine Learning
- ship the Minimum Viable Product (MVP) and then iterate on that product.
- 2.1 Beyond “Hack-tive Learning:” your first Active Learning algorithm
- 2.1.1 The architecture of your first HuML system
- 2.2 Interpreting model predictions and data to support Active Learning
- 2.2.1 Confidence ranking
- 2.2.2 Identifying outliers
- 2.2.3 What to expect as you iterate
- 2.3 Building an interface to get human labels
- 2.3.1 A simple interface for labeling text
- 2.3.2 Managing Machine Learning data
- 2.4 Deploying your first Human-in-the-Loop Machine Learning system
- implementation