Contact Us

Give us a call or drop by anytime, we endeavor to answer all inquiries within 24 hours.

map

Find us

PO Box 16122 Collins Street West Victoria, Australia

Email us

info@domain.com / example@domain.com

Phone support

Phone: + (066) 0760 0260 / + (057) 0760 0560

Loading Events

« All Events

  • This event has passed.
Event Series Event Series: EnCORE Series

EnCORE : Theoretical Exploration of Foundation Model Adaptation, Kangwook Lee, UW Madison, Feb 9th, 1-2pm

February 9 @ 1:00 pm - 2:00 pm

Abstract: Due to the enormous size of foundation models, various new methods for efficient model adaptation have been developed. Parameter-efficient fine-tuning (PEFT) is an adaptation method that updates only a tiny fraction of the model parameters, leaving the remainder unchanged. In-context Learning (ICL) is a test-time adaptation method, which repurposes foundation models by providing them with labeled samples as part of the input context. Given the growing importance of this emerging paradigm, developing theoretical foundations for the new paradigm is of utmost importance.

In this talk, I will introduce two preliminary results toward this goal. In the first part, I will present a theoretical analysis of Low-Rank Adaptation (also known as LoRA), one of the most popular PEFT methods today. Our analysis of the expressive power of LoRA not only helps us better understand the high adaptivity of LoRA observed in practice but also provides insights to practitioners. In the second part, I will introduce our probabilistic framework for a better understanding of ICL. With our framework, one can analyze the transition between two distinct modes of ICL: task retrieval and learning. We also discuss how our framework can help explain and predict various phenomena, which can be observed with large language models in practice yet not fully explained.

Bio: Kangwook Lee is an Assistant Professor in the Electrical and Computer Engineering Department and the Computer Sciences Department (by courtesy) at the University of Wisconsin-Madison. Previously, he was a Research Assistant Professor at the Information and Electronics Research Institute of KAIST and was a postdoctoral scholar at the same institute. He received his PhD in 2016 from the Electrical Engineering and Computer Science department at UC Berkeley. He is the recipient of the IEEE Joint Communications Society/Information Theory Society Paper Award (2020) and the KSEA Young Investigator Grant Award (2022).

Details

Date:
February 9
Time:
1:00 pm - 2:00 pm
Series:
Event Categories:
,

Organizer

k1omerry@ucsd.edu
Email
k1omerry@ucsd.edu

Other

Format
Hybrid
Speaker
Kangwook Lee, UW Madison
Event Recording Link
https://ucsd.zoom.us/j/96580743215