Events

Extrapolating scaling trends suggest that training dataset size for LLMs may soon be limited by the amount of text data available on the internet. In this talk we investigate scaling language models in data-constrained regimes. Specifically, we run a set of empirical experiments varying the extent of data repetition and compute budget. From these experiments we propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we discuss and experiment with approaches for mitigating data scarcity.

Abstract: Despite impressive recent advances, machine learning models exhibit a number of critical deficiencies. They are prone to leaking sensitive information about their training data. They remain alarmingly brittle to attacks by malicious parties. Troublingly, these issues stem from more fundamental statistical vulnerabilities, which remain unresolved even decades later, highlighting significant gaps in our understanding of how to deal with these important considerations. As long as these problems remain, our models will not be appropriate for use beyond deployment in toy settings. In this talk, I will discuss recent advances on a number of these problems, which give key new algorithmic insights into how to address these considerations, and enable real-world deployments that were previously thought infeasible. In a first vignette, we will explore how to guarantee individual privacy in machine learning models, with a particular focus on large language models and the important role played by public data in the training pipeline. In a second vignette, we focus on how to robustly perform mean estimation, giving the first efficient and accurate algorithms for multivariate settings. We will go on to discuss connections to robustness against data poisoning attacks, robust exploratory data analysis, and surprising conceptual and technical connections with privacy.

Abstract: "The previously discrete technologies of IoT and AI have now entered a tight virtuous embrace. IoT allows sensing and actuation in our physical, social, and urban spaces with unimaginable ubiquity. AI allows sophisticated inferences and decisions to be made algorithmically using deep neural networks, even from unstructured and high- dimensional data, with uncanny performance. Together they seek to perform sophisticated perception-cognition-communication-action loops in diverse applications. However, designers of learning-enabled IoT systems face the challenge of extremely resource-constrained edge platforms operating in uncertain environments while assuring performance and trustworthiness. Moreover, in many applications, the systems go beyond taking actions based on rich inferences about the world state to perform long-term reasoning about complex events and obey the underlying physics, rules, and constraints. Based on our experience in designing such systems in applications including mHealth, ocean animal health, agriculture robotics, and military, This talk explores meeting these challenges through a combination of (i) neurosymbolic architectures that allow the incorporation of physics awareness and human knowledge while enhancing user trust, (ii) automatic platform-aware architecture search and code generation, and (iii) techniques to efficiently adapt to the deployment environment."