Contact Us
Give us a call or drop by anytime, we endeavor to answer all inquiries within 24 hours.
Find us
PO Box 16122 Collins Street West Victoria, Australia
Email us
info@domain.com / example@domain.com
Phone support
Phone: + (066) 0760 0260 / + (057) 0760 0560
- This event has passed.
Special TILOS Seminar: Claire Boyer | Single location regression and attention-based models
March 27 @ 2:00 pm - 3:00 pm
Talk Information
Speaker: Claire Boyer (Université Paris-Saclay)
Date & Time: Thursday, March 27 @ 2pm PDT
Venue: HDSI 123
Title: Single location regression and attention-based models
Abstract: Attention-based models, such as Transformer, excel across various tasks but lack a comprehensive theoretical understanding, especially regarding token-wise sparsity and internal linear representations. To address this gap, we introduce the single-location regression task, where only one token in a sequence determines the output, and its position is a latent random variable, retrievable via a linear projection of the input. To solve this task, we propose a dedicated predictor, which turns out to be a simplified version of a non-linear self-attention layer. We study its theoretical properties, by showing its asymptotic Bayes optimality and analyzing its training dynamics. In particular, despite the non-convex nature of the problem, the predictor effectively learns the underlying structure. This work highlights the capacity of attention mechanisms to handle sparse token information and internal linear structures.