Title: How to Forage for Mushrooms Without Dying: Integrating Language and Reinforcement Learning
Abstract: I'll cover recent work grounding communication into complex decision settings to explain linguistic phenomena such as relevance and generics. I'll first present theoretical work generalizing Lewis signaling games to multi-armed bandit settings, creating a new game: Signaling Bandits. This allows us to extend computational models of communication (the Rational Speech Act) to incorporate decision-theoretic utility based on the listener's subsequent actions. I'll then preview behavioral data supporting these models, demonstrating participants' sensitivity to nuanced tradeoffs between epistemic and decision-theoretic objectives. In the long term, I hope to extend artificial agents' linguistic abilities, moving beyond concrete instructions to language encoding abstract information about the environment.
Title: Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color
Abstract: Pretrained language models have been shown to encode relational information, such as the relations between entities or concepts in knowledge-bases -- (Paris, Capital, France). However, simple relations of this type can often be recovered heuristically and the extent to which models implicitly reflect topological structure that is grounded in world, such as perceptual structure, is unknown. To explore this question, we conduct a thorough case study on color. Namely, we employ a dataset of monolexemic color terms and color chips represented in CIELAB, a color space with a perceptually meaningful distance metric. Using two methods of evaluating the structural alignment of colors in this space with text-derived color term representations, we find significant correspondence. Analyzing the differences in alignment across the color spectrum, we find that warmer colors are, on average, better aligned to the perceptual color space than cooler ones, suggesting an intriguing connection to findings from recent work on efficient communication in color naming. Further analysis suggests that differences in alignment are, in part, mediated by collocationality and differences in syntactic usage, posing questions as to the relationship between color perception and usage and context.
3/9 Najoung Kim (BU/NYU)
Title: Compositional Generalization in Artificial Neural Networks
Abstract: Compositionality is considered a central property of human language. One key benefit of compositionality is the generalization it enables the production and comprehension of novel expressions analyzed as new compositions of familiar parts. I construct a test for compositional generalization for artificial neural networks based on human generalization patterns discussed in existing linguistic and developmental studies, and test several instantiations of Transformer (Vaswani et al. 2017) and Long Short-Term Memory (Hochreiter & Schmidhuber 1997) models. The models evaluated exhibit only limited degrees of compositional generalization, implying that their learning biases for induction to fill gaps in the training data differ from those of human learners. An error analysis reveals that all models tested lack bias towards faithfulness (à la Prince & Smolensky 1993/2002). Adding a glossing task (word-by-word translation), a task that requires maximally faithful input-output mappings, as an auxiliary training objective to the Transformer model substantially improves generalization, showing that the auxiliary training successfully modified the model's inductive bias. However, the improvement is limited to generalization to novel compositions of known lexical items and known structures; all models still struggled with generalization to novel structures, regardless of auxiliary training. The challenge of structural generalization leaves open exciting avenues for future research for both human and machine learners.
Title: Language Models Have Implicit Semantics
Abstract: Is "meaning" encoded in a model that has been trained to assign probabilities to sentences? While intuitively it may seem like it is not, this paper establishes a mathematical sense in which it is. For several different models of how speakers produce training data, we show that semantic entailment between sentences is isomorphic to a simple equation defined purely in terms of sentence probabilities. We also prove that such a relation must exist for data generated by any "reasonable" speaker, although it need not be easy to compute. Together, these results suggest that naturally occuring distributions of text can be understood to implicitly assign semantics to the text they contain. This implies that deciding entailment between sentences is reducible to language modeling, so a language model like GPT-3 can in theory learn entailment semantics. Assuming our idealized speakers are valid models of human speakers, it also suggests simple equations that can be used to "decode" entailment between two sentences from a language model in an unsupervised way.
4/6 Soo Hyun Ryu (University of Michigan)
Title: Transformer Language Models can Integrate Surprisal, Entropy, and Working Memory Retrieval Accounts of Sentence Processing
Abstract: Memory-based and expectation-based theories have been successful in accounting for a diverse set of sentence processing phenomena, and integrating the two approaches is an interesting challenge for the field. In this work, we show that Transformers can serve as an integrative model: Transformers are predictive cue-based retrieval parsers. We describe how the learned attention mechanism operates as a kind of cue-based retrieval, with the diffuseness of patterns of attention reflecting similarity-based interference. We introduce a simple metric, attention entropy, that quantitatively captures this interference, and show that it is a predictor of word-by-word reading times independent of surprisal, in both self-paced reading and eye-tracking paradigms. We then show how the Transformer account can explain a set of psycholinguistic phenomena that have challenged previous surprisal and cue-based models.
4/13 Aniketh Reddy (UC Berkeley)
Title: Can fMRI reveal the representation of syntactic structure in the brain?
Abstract: While studying semantics in the brain, neuroscientists use two approaches. One is to identify areas that are correlated with semantic processing load. Another is to find areas that are predicted by the semantic representation of the stimulus words. However, most studies of syntax have focused only on identifying areas correlated with syntactic processing load. One possible reason for this discrepancy is that representing syntactic structure in an embedding space such that it can be used to model brain activity is a non-trivial computational problem. Another possible reason is that it is unclear if the low signal-to-noise ratio of neuroimaging tools such as functional Magnetic Resonance Imaging (fMRI) can allow us to reveal the correlates of complex (and perhaps subtle) syntactic representations. To make it easier to study syntax representations, we develop novel multi-dimensional features that encode information about the syntactic structure of sentences. Using these features and fMRI recordings of participants reading a natural text, we model the brain representation of syntax. We find that our syntactic structure-based features can explain additional variance in the brain activity of various parts of the language system, even after controlling for complexity metrics that capture processing load, showing that fMRI data can indeed reveal brain representations of syntax. At the same time, we see that the regions well-predicted by syntactic features are distributed in the language system and are not distinguishable from those processing semantics.
4/20 David Bau (Harvard/Northeastern)
Title: Locating and Editing Factual Knowledge in GPT (ROME)
Abstract: Where is the knowledge inside a large language model? In this paper, we show evidence that factual knowledge within GPT also corresponds to a localized computation that can be directly edited. For example, we can make a small change to a small set of the weights of GPT-J to teach it the counterfactual "Eiffel Tower is located in the city of Rome." Rather than merely regurgitating the new sentence, it will generalize that specific counterfactual knowledge and apply it in very different linguistic contexts. We discuss methods we use to identify specific modules that mediate such factual knowledge, and the methods we use to alter parameters to edit the model's belief in individual facts.
5/4 Matteo Alleman (Columbia)
Title: Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Models
Abstract: While vector-based language representations from pretrained language models have set a new standard for many NLP tasks, there is not yet a complete accounting of their inner workings. In particular, it is not entirely clear what aspects of sentence-level syntax are captured by these representations, nor how (if at all) they are built along the stacked layers of the network. In this paper, we aim to address such questions with a general class of interventional, input perturbation-based analyses of representations from pretrained language models. Importing from computational and cognitive neuroscience the notion of representational invariance, we perform a series of probes designed to test the sensitivity of these representations to several kinds of structure in sentences. Each probe involves swapping words in a sentence and comparing the representations from perturbed sentences against the original. We experiment with three different perturbations: (1) random permutations of n-grams of varying width, to test the scale at which a representation is sensitive to word position; (2) swapping of two spans which do or do not form a syntactic phrase, to test sensitivity to global phrase structure; and (3) swapping of two adjacent words which do or do not break apart a syntactic phrase, to test sensitivity to local phrase structure. Results from these probes collectively suggest that Transformers build sensitivity to larger parts of the sentence along their layers, and that hierarchical phrase structure plays a role in this process. More broadly, our results also indicate that structured input perturbations widens the scope of analyses that can be performed on often-opaque deep learning systems, and can serve as a complement to existing tools (such as supervised linear probes) for interpreting complex black-box models.