8.2.4 Other

1. [M] An autoencoder is a neural network that learns to copy its input to its output. When would this be useful?
2. Self-attention.
   1. [E] What’s the motivation for self-attention?
   2. [E] Why would you choose a self-attention architecture over RNNs or CNNs?
   3. [M] Why would you need multi-headed attention instead of just one head for attention?
   4. [M] How would changing the number of heads in multi-headed attention affect the model’s performance?
3. Transfer learning
   1. [E] You want to build a classifier to predict sentiment in tweets but you have very little labeled data (say 1000). What do you do?
   2. [M] What’s gradual unfreezing? How might it help with transfer learning?
4. Bayesian methods.
   1. [M] How do Bayesian methods differ from the mainstream deep learning approach?
   2. [M] How are the pros and cons of Bayesian neural networks compared to the mainstream neural networks?
   3. [M] Why do we say that Bayesian neural networks are natural ensembles?
5. GANs.
   1. [E] What do GANs converge to?
   2. [M] Why are GANs so hard to train?