Report · Page 127

On the Opportunities and Risks of Foundation Models
127
To make the notions of model representations and mechanisms concrete, consider a simple
behavior exhibited by GPT-3: It was quickly observed what GPT-3 did when provided with examples
of the addition of small numbers and then queried to perform addition of two new numbers: with
high probability, it predicted the correct result of the addition [Branwen 2020; Brockman 2020].
When asking why GPT-3 performed as it did, one could find evidence in the input, like aspects of
its prompt that highly affected its response (these might be the two numbers to be added, though
not necessarily), or aspects of GPT-3’s training data that affected its response (these might be
examples of addition, though not necessarily). Delving into the model, we may envision a deeper
understanding of the mechanisms that GPT-3 uses to add a specific pair of numbers and the
mechanism that it uses to add other arbitrary pairs of numbers. We may also envision a deeper
understanding of whether these mechanisms are similar to the mathematical notion of ‘addition’
or merely correlated with this notion.
By understanding individual model mechanisms, we can build up a compositional understanding
of complex behaviors of a foundation model. A task slightly more complex than the addition of
numbers is solving mathematical word problems, in which numbers come with units and the
problem is presented in natural language. Once we understand the mechanism (or mechanisms)
by which a model performs addition, we can investigate whether this mechanism is used as an
intermediate step in solving word problems. If the addition mechanism is used, we have built up
our understanding of how the model solves word problems, we have increased confidence that the
foundation model generalizes the notions of quantities and addition (not another correlation or
heuristic), and, furthermore, we have increased confidence in our ability to predict the model’s why
(which parts of the inputs it is attending to) and the output’s what (addition of two numbers). If the
addition mechanism is not used, we may retain a healthy skepticism that this is truly addition, and
we can investigate which representations and mechanisms are used instead.
It is important to be aware that there are many potential cases of more complex and concerning
model mechanisms, for instance, the estimation of race from the characters in a name, or the
pixels in an image. Establishing evidence of such a mechanism in a foundation model and its use
can support a moral or legal responsibility to ban the model from tasks like predictive policing,
marketing, loan applications, and surveillance at large.
A plethora of methods have emerged to investigate these internal aspects of neural network
models. Typically, these approaches separate the model into nodes (e.g., neurons, layers, or parts
of layers), then interrogate either the representations captured in nodes or the mechanisms by
which nodes are assembled. Some approaches are hypothesis driven: by hypothesizing that nodes
may capture certain information (e.g., a grammatical feature of a word, or the race of a person),
one can probe all nodes to quantify how much of that information they make available [Alain
and Bengio 2016; Veldhoen et al. 2016; Belinkov et al. 2017; Adi et al. 2017; Conneau et al. 2018;
Hewitt and Liang 2019; Hewitt and Manning 2019; Voita and Titov 2020; Pimentel et al. 2020].
Other approaches build on explanatory met