Contact Us

Give us a call or drop by anytime, we endeavor to answer all inquiries within 24 hours.

map

Find us

PO Box 16122 Collins Street West Victoria, Australia

Email us

info@domain.com / example@domain.com

Phone support

Phone: + (066) 0760 0260 / + (057) 0760 0560

Linguistics

  • March 23, 2023
  • Kaleigh O'Merry

Structured Transformer Models for NLP

The field of natural language processing has recently unlocked a wide range of new capabilities through the use of large language models, such as GPT-4. The growing application of these models motivates developing a more thorough understanding of how and why they work, as well as further improvements in both quality and efficiency.

In this talk, I will present my work on analyzing and improving the Transformer architecture underlying today’s language models through the study of how information is routed between multiple words in an input. I will show that such models can predict the syntactic structure of text in a variety of languages, and discuss how syntax can inform our understanding of how the networks operate. I will also present my work on structuring information flow to build radically more efficient models, including models that can process text of up to one million words, which enables new possibilities for NLP with book-length text.

Read More
  • March 23, 2023
  • Kaleigh O'Merry

Constrained, Casual, and Logical Reasoning for Neural Language Generation

Today’s language models (LMs) can produce human-like fluent text. However, they generate words with no grounding in the world and cannot flexibly reason about everyday situations and events, such as counterfactual (“what if?”) and abductive (“what might explain these observations?”) reasoning that are important forms of human cognition activities. In this talk, I will present my research on connecting reasoning with language generation. Reasoning for language generation poses several key challenges, including incorporating diverse contextual constraints on the fly, understanding cause and effect when events unfold, and grounding on logic structures for consistent reasoning. I will first discuss COLD decoding, a unified energy-based framework for any off-the-shelf LMs to reason with arbitrary constraints. It also introduces differentiable reasoning over discrete symbolic text for improved efficiency. Secondly, I will focus on a particularly important form of reasoning, counterfactual reasoning, including its first formulation in language generation and our algorithm, DeLorean, that enables off-the-shelf LMs to capture causal invariance. Thirdly, I will present Maieutic prompting, which improves the logical consistency of neural reasoning by integrating with logic structures. I will conclude with future research toward more general, grounded, and trustworthy reasoning with language.

Read More
  • March 16, 2023
  • Kaleigh O'Merry

Uncovering the algorithmic foundations of language learning and processing

Human language is extraordinarily complex. Nevertheless, we readily acquire language as children, when we are most cognitively limited, and we comprehend language as adults with striking efficiency. My research seeks to understand the mental algorithms that allow us to accomplish this feat, with particular focus on how memory and prediction mechanisms are recruited to overcome the bottlenecks of real-time language processing. In this talk, I will review results from three of my lines of inquiry into this question. First, using diverse naturalistic reading datasets, I will show evidence that prediction is a central concern of the human language processing system. Second, using fMRI measures of naturalistic story listening, I will show evidence that memory and prediction processes are dissociable in the brain’s response to language, that syntactic structure building plays a major role in ordinary language comprehension, and that the neural resources that are responsible for structure building are largely specialized for language. Third, I will show evidence from computational modeling that memory and prediction pressures independently encourage discovery of phonological regularities from natural speech. Together, these results support an intricate coordination of memory and prediction abilities for language learning and comprehension. I will conclude by outlining planned directions for my future lab, integrating neuroimaging, behavioral methods, natural language processing, and computational modeling to study language learning and processing.

Read More

Post List