Report · Page 24

24
Center for Research on Foundation Models (CRFM)
models has also led to a flowering of research for language generation tasks like summarization
and dialogue generation. The rise of the foundation model paradigm has begun to play a similar
role in spoken language as well as written. Modern automatic speech recognition (ASR) models
like wav2vec 2.0 are trained on large datasets of speech audio alone, and then adapted on audio
with associated transcriptions for the task of ASR [Baevski et al. 2020].
Due to the changes brought about by the foundation model paradigm, the focus of research and
practice in NLP has shifted from making bespoke architectures for different tasks to exploring
how to best leverage foundation models. Research into adaptation methods has blossomed (see
§4.3: adaptation for a detailed look at adaptation), and the surprising successes of foundation
models have also caused a shift in research interest towards analyzing and understanding foundation
models (see §4.11: interpretability for interpretability and analysis of foundation models).
2.1.3
Language variation and multilinguality.
Though foundation models are surprisingly versatile with the linguistic knowledge they obtain from
pretraining, there are limits to this adaptability: it is not clear how successful current foundation
models are at handling language variation. Language varies greatly. Apart from the fact that there
are thousands of different languages in the world, language varies even within one language or
within one speaker. To point out a few examples, informal conversation manifests differently from
written language, the grammatical constructions that people reach for when speaking to friends
are very different from those used when speaking to someone with authority, and communities of
speakers within a language use different dialects. Social and political factors are embedded in how
language variation is viewed and valued, and in how much different varieties are represented in NLP
research (see for example Blodgett and O’Connor [2017] on the failures of NLP for African American
English, and §5.1: fairness for a deeper discussion on inequities in foundation models). Due to their
large capacity for learning linguistic information and flexibly adapting that knowledge, foundation
models hold promise for expanding NLP to encompass more linguistic diversity. It remains an open
research question to understand whether it is possible to make foundation models that robustly and
equitably represent language with both its major and subtle variations, giving equal weight and
acuity to what makes each linguistic variety distinct [research posing and addressing this question
includes Ponti et al. 2019; Bender 2011; Joshi et al. 2020].
Following the success of foundation models for English, multilingual foundation models have
been released to extend that success to non-English languages. For most of the over 6,000 languages
in the world, the text data available is not enough to train a large-scale foundation model. To give
one example, there are over 65 million speakers of Fula, a West African language, but few if any
resources available for NLP in Fula [Nguer et al. 2020]. Multilingual foundation models address this
by jointly training on multiple languages simultaneously. The multilingual foundation models to
date (mBERT, mT5, XLM-R) are each trained on around 100 languages [Devlin et al. 2019; Goyal et al.
2021; Xue et al. 2020]. Joint multilingual training reli