Report · Page 88

88
Center for Research on Foundation Models (CRFM)
Temporal adaptation. Ideally, foundation models store knowledge that closely represents the
state of the world, independent of modality. However, the world is constantly changing; new heads
of state are elected, clothing styles change, social norms and beliefs shift (§5.6: ethics), and the use
of language evolves, causing a shift in the input distribution, target predictive distribution, or both.
This temporal shift presents a challenging statistical problem due to the induced distribution shift,
as discussed in §4.8: robustness. For foundation models, temporal shift also presents a particu-
larly difficult computational problem; due to the computationally demanding nature of training
foundation models [Shoeybi et al. 2019; Brown et al. 2020], frequent re-training from scratch might
carry unacceptable financial or environmental impacts [Bender et al. 2021] (§5.3: environment),
or simply take too long to be a viable method for keeping models up to date. In visual domains,
gradual self-training on unlabeled data across intermediate time points can bridge the temporal
shift across a long time period, but remains an expensive retraining procedure [Kumar et al. 2020a].
In the context of language models, temporally-partitioned diagnostic datasets help quantify the
rate at which large language models become outdated [Lazaridou et al. 2021; Hombaiah et al. 2021;
Dhingra et al. 2021], showing that classic techniques like re-weighting training data and dynamic
evaluation (updating model parameters with new data at production time [Mikolov et al. 2010])
can partially alleviate, but not fully solve, this problem. Explicitly conditioning a language model
on the time period it is intended to model is another technique that has shown some promise
[Dhingra et al. 2021]. An alternative strategy to addressing temporal shift is to design retrieval-
based (semi-parametric) models, which augment the model input with additional context retrieved
from a large, human-interpretable database (e.g., Wikipedia articles) [Karpukhin et al. 2020; Lewis
et al. 2020b; Guu et al. 2020; Khandelwal et al. 2020; Khattab et al. 2020]. For retrieval-based models,
adaptation corresponds to updating individual units of information in the database (e.g., single
paragraphs of encyclopedia articles) without re-training the model. While promising, challenges
for retrieval-based approaches remain both in training an accurate retrieval mechanism and in
accurately conditioning on the retrieved information [Lewis et al. 2020b]. We revisit the problem
of temporal adaptation in the more general context of continual learning later in the section.
Domain specialization. In addition to task specialization, it is often necessary to specialize a
foundation model to a particular domain (such as legal documents), without limiting the breadth of
tasks the foundation model can accomplish. This specialization induces a mismatch between the
foundation model training and adaptation data distributions (§4.8: robustness) which may require
new adaptation methods to handle. Prior works have found that diverse and general pretraining
data can cause negative transfer for current adaptation methods. For example, Cole et al. [2021]
shows that fine-tuning a model pretrained only on the iNaturalist animal classification dataset
provides better downstream performance than fine-tuning a model pretrained on iNaturalist along
with 750K other images; similarly Le