Member-only story
What the Heck Is Foundation Models (VS. Specialized Models)?
What are foundation models?
This is best explained by Dr. Percy Liang from Stanford:
Foundation models are models which are trained on broad data usually at immense scale in a self-supervised way and that can be quickly adapted to a wide range of downstream tasks.
That makes sense! If you want your model (for instance, in the NLP setting) to understand finance (downstream and specialized), why not having it understand plain English first (foundation)? This IS the general and brilliant idea of transfer learning (we will cover this in a later post!).
What about examples?
- NLP: Large language models like BERT, RoBERTa, and GPT-3
- Vision: Image representation learning model
- Speech: Generative Spoken Language Model (GSLM)
- Music: Jukebox
- Reinforcement Learning: Decision Transformer
- Protein Folding: Protein Sequence Learning
What’s so GOOD about foundation models?
What is really significant about foundation models is that it changes the paradigm in which AI systems are built. Rather than having bespoke models for each individual task now you train a general purpose foundation model that can be adapted to a wide variety of different scenarios.
