Report · Page 5

On the Opportunities and Risks of Foundation Models
5
There had been considerable progress in self-supervised learning dating back to word embeddings
[Turian et al. 2010; Mikolov et al. 2013; Pennington et al. 2014], which associated each word with a
context-independent vector, provided the basis for a wide range of NLP models. Shortly thereafter,
self-supervised learning based on autoregressive language modeling (predict the next word given
the previous words) [Dai and Le 2015] became popular. This produced models that represented
words in context, such as GPT [Radford et al. 2018], ELMo [Peters et al. 2018], and ULMFiT [Howard
and Ruder 2018].4
The next wave of developments in self-supervised learning — BERT [Devlin et al. 2019] GPT-2
[Radford et al. 2019], RoBERTa [Liu et al. 2019], T5 [Raffel et al. 2019], BART [Lewis et al. 2020a] —
quickly followed, embracing the Transformer architecture, incorporating more powerful deep
bidirectional encoders of sentences, and scaling up to larger models and datasets.
While one can view this last wave of technical developments purely through the lens of self-
supervised learning, there was a sociological inflection point around the introduction of BERT.
Before 2019, self-supervised learning with language models was essentially a subarea in NLP, which
progressed in parallel to other developments in NLP. After 2019, self-supervised learning with
language models became more of a substrate of NLP, as using BERT has become the norm. The
acceptance that a single model could be useful for such a wide range of tasks marks the beginning
of the era of foundation models.
Foundation models have led to an unprecedented level of homogenization: Almost all state-of-
the-art NLP models are now adapted from one of a few foundation models, such as BERT, RoBERTa,
BART, T5, etc. While this homogenization produces extremely high leverage (any improvements in
the foundation models can lead to immediate benefits across all of NLP), it is also a liability; all AI
systems might inherit the same problematic biases of a few foundation models [Bolukbasi et al.
2016; Caliskan et al. 2017; Abid et al. 2021, inter alia]) — see §5.1: fairness, §5.6: ethics for further
discussion.
We are also beginning to see a homogenization across research communities. For example, similar
Transformer-based sequence modeling approaches are now applied to text [Devlin et al. 2019;
Radford et al. 2019; Raffel et al. 2019], images [Dosovitskiy et al. 2020; Chen et al. 2020d], speech [Liu
et al. 2020d], tabular data [Yin et al. 2020], protein sequences [Rives et al. 2021], organic molecules
[Rothchild et al. 2021], and reinforcement learning [Chen et al. 2021b; Janner et al. 2021]. These
examples point to a possible future where we have a unified set of tools for developing foundation
models across a wide range of modalities [Tamkin et al. 2021b].
Besides the homogenization of approaches, we also see the homogenization of actual models
across research communities in the form of multimodal models — e.g., foundation models trained
on language and vision data [Luo et al. 2020; Kim et al. 2021a; Cho et al. 2021; Ramesh et al. 2021;
Radford et al. 2021]. Data is naturally multimodal in some domains—e.g., medical images, structured
data, clinical text in healthcare (§3.1: healthcare). Thus, multimodal foundation models are a
natural way of fusing all the relevant information about a domain, and adapting to tasks that also
span multiple modes (Figure 2).