Active Learning Strategies & DISTIL


In this Article

  1. Introduction
  2. Deep dIverSified inTeractIve Learning (DISTIL)
  3. Various Active Learning Strategies
  • Coreset
  • FASS
  • Adversarial Techniques
  • BALD


Deep learning models, which are often deemed to be the state of the art, are specially equipped to find hidden patterns from large datasets as they learn to craft features. However, training these deep learning models is very demanding both in terms of computational resources and large training data. The deeper the model, the more are parameters to be learnt. This makes models more and more data-hungry to achieve good generalization. This begs the question what is the cost of acquiring the data? Are the datasets always labelled and if not, what is the cost incurred in getting unlabeled datasets labelled?

Price for labelling 1000 data points. (Source: Similar rates can be found at

2. Deep dIverSified inTeractIve Learning (DISTIL)

Active learning can be easily incorporated with the new DISTIL Toolkit. DISTIL is a library that features many state-of-the-art active learning algorithms. Implemented in PyTorch, it gives fast and efficient implementations of these active learning algorithms. It has the most state-of-the-art active learning strategies. DISTIL employs mini-batch adaptive active learning, which is more appropriate for deep neural networks. Thus, in each round DISTIL strategies select k sized mini-batch for n rounds. Now let’s see the various strategies present in DISTIL.

3. Various Active Learning Strategies

1. Uncertainty Sampling

4 centres chosen. Source: Core-Set Paper
Choosing samples in adversarial Setting. Paper: Adversarial active learning for deep networks: a margin based approach
The intuition behind BALD. Areas in grey contribute to the BALD score. Paper: Efficient and diverse batch acquisition for deep Bayesian active learning

4. Video Explanation

5. Resources

More about Active Learning & DISTIL:

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store