ELLIS Institute Scientific Symposium 2024

Abstract

Sparsity, a fundamental characteristic of machine learning, plays a pivotal role in enhancing the efficiency and performance of neural networks. In neural networks, sparsity entails the deliberate reduction of parameters or activations, resulting in a leaner structure with fewer non-zero weights or activations. While existing research predominantly focuses on exploiting sparsity for model compression—such as deriving sparse neural networks from pre-trained dense ones—many other promising benefits such as scalability, robustness, and fairness remain under-explored. This talk aims to delve into these overlooked advantages. Specifically, we will showcase how sparsity can boost the scalability of neural networks by efficiently training sparse models from scratch. This approach enables a significant increase in model capacity without proportionally escalating computational or memory requirements. Additionally, we will explore the future implications of sparsity in the realm of large language models, discussing its potential benefits to efficient LLM scaling, lossless LLM compression, and fostering trustworthy AI.

Biography

Shiwei Liu is a Royal Society Newton International Fellow at the University of Oxford. He obtained his Ph.D. with the Cum Laude (distinguished Ph.D. thesis) from the Eindhoven University of Technology in 2022, under the supervision of Prof. Mykola Pechenizkiy and Dr. Decebal Constantin Mocanu. He was a postdoctoral fellow at UT Austin in 2023 working with Dr. Atlas Wang. He has received two Rising Star Awards from KAUST and the Conference on Parsimony and Learning (CPAL). His Ph.D. thesis received the 2023 Best Dissertation Award from Informatics Europe. At present, his core research interest is to leverage, understand, and expand the role of sparsity in modern deep neural networks, whose impacts span many important topics, such as efficient training/inference/transfer of large-foundation models, robustness, and trustworthiness. He has over 30 publications in top-tier machine learning conferences, such as ICLR, ICML, NeurIPS, ICCV, AAAI, IJCAI, UAI, and LoG. He has served as an area chair in ICIP'22 and ICIP'23; and a PC member of almost all top-tier ML/CV conferences. Shiwei has co-organized several tutorials in ICASSP 2024, IJCAI'23 and ECML-PKDD'22.

Abstract

Combining physics with machine learning is a rapidly growing field of research. Thereby, most work focuses on leveraging machine learning methods to solve problems in physics. Here, however, we focus on the converse, i.e., physics-inspired machine learning, which can be described as incorporating structure from physical systems into machine learning methods to obtain models with better inductive biases. More concretely, we propose several physics-inspired deep learning architectures for sequence modelling based on nonlinear coupled oscillators, Hamiltonian systems and multi-scale dynamical systems. The proposed architectures tackle central problems in the field of recurrent sequence modeling, namely the vanishing and exploding gradients problem as well as the issue of insufficient expressive power. Moreover, we discuss physics-inspired learning on graphs, wherein the dynamics of the message-passing propagation are derived from physical systems. We further prove that these methods mitigate the over-smoothing issue, thereby enabling the construction of deep graph neural networks (GNNs). We extensively test all proposed methods on a variety of versatile synthetic and real-world datasets, ranging from image recognition, speech recognition, natural language processing (NLP), medical applications, and scientific computing for sequence models, to citation networks, computational chemistry applications, and networks of articles and websites for graph learning models. Finally, we show how to leverage physics-based inductive biases of physics-inspired machine learning methods to solve problems in the physical sciences.

Biography

T. Konstantin Rusch is an SNSF postdoctoral research fellow at CSAIL, MIT. Before that, he obtained a PhD in Applied Mathematics and Machine Learning at ETH Zurich, supervised by Prof. Dr. Siddhartha Mishra. During his doctoral studies, he had a second affiliation at UC Berkeley, advised by Prof. Dr. Michael Mahoney. Moreover, he held visiting research appointments at UC Berkeley and the University of Oxford. His main research interest is in combining physics with machine learning. Thereby, he focuses on physics-inspired machine learning, which can be described as leveraging structure from physical systems to construct novel machine learning methods with better inductive biases. Additionally, he works on combining methods from numerical analysis with machine learning to solve problems in computational science and engineering.

Abstract

Traditional computing sciences have made significant advances with tools like Complexity and Worst-Case Analysis. However, Machine Learning has unveiled optimization challenges, from image generation to autonomous vehicles, that go beyond the analytical capabilities of past decades. Despite their theoretical complexity, such tasks are often more manageable in practice, thanks to deceptively simple yet efficient techniques such as Local Search and Gradient Descent.
In this talk, we will delve into the effectiveness of these algorithms in complex environments and the development of a theory that transcends traditional analysis, bridging theoretical principles with practical applications. We will further explore the behavior of these heuristics in multi-agent strategic environments, evaluating their capacity to achieve equilibria through advanced machinery from Optimization, Statistics, Dynamical Systems, and Game Theory. The discussion will conclude with an outline of future research directions and my vision for a computational understanding of multi-agent Machine Learning.

Biography

Emmanouil-Vasileios (Manolis) Vlatakis Gkaragkounis is currently a Foundations of Data Science Institute (FODSI) Postdoctoral Fellow at the Simons Institute for the Theory of Computing, UC Berkeley, mentored by Prof. Michael Jordan. He completed his Ph.D. in Computer Science at Columbia University, under Professors Mihalis Yannakakis and Rocco Servedio, and holds B.Sc. and M.Sc. degrees in Electrical and Computer Engineering. Manolis specializes in the theoretical aspects of Data Science, Machine Learning, and Game Theory. His expertise includes beyond worst-case analysis, optimization, and data-driven decision-making in complex environments. Applications of his work span multiple areas from privacy, neural networks, to economics and contract theory, statistical inference, and quantum machine learning.

Abstract

Recent developments in deep learning have led to significant advances in many areas of computer vision. However, the success of these methods often depends on having a well-defined task, corresponding training data, and measuring success by improved task-specific accuracy. However, in order to apply new methods in the real-world, other aspects become relevant as well, such as required labelled data, computational requirements, as well as, especially in safety critical scenarios, how trust-worthy a model is. In my talk, I will first discuss how motion in videos can be used to learn representations in an unsupervised way as well as methods to efficiently handle higher-resolution data. In the second part, I will show how attribution maps, which help to gain a better understanding of the predictions, can be obtained efficiently.

Biography

Simone Schaub-Meyer is an independent research group leader at the Technical University of Darmstadt, as well as affiliated with the Hessian Center for Artificial Intelligence. She recently got the renowned Emmy Noether Programme (ENP) grant of the German Research Foundation (DFG) supporting her research group for the next 6 years. The focus of her research is on developing efficient, robust, and understandable methods and algorithms for image and video analysis. Before starting her own group, she was a postdoctoral researcher in the Visual Inference Lab of Prof. Stefan Roth. Prior to joining TU Darmstadt, she was a postdoctoral researcher at the Media Technology Lab at ETH Zurich working on augmented reality. She obtained her doctoral degree from ETH Zurich, advised by Prof. Dr. Markus Gross and in collaboration with Disney Research Zurich. In her thesis, awarded with the ETH Medal, she developed novel methods for motion representation and video frame interpolation.

Abstract

The prior knowledge we have about the physical laws and the common geometric patterns of our 3D world is often neglected in the design and the learning of data-driven models for semantic and geometric visual perception. In this talk, I will by contrast advocate for a physics- and geometry-informed approach to visual perception. On the physics side, the talk will focus on the case of lidar optics and will demonstrate how the careful identification of the linear optical system which governs pulsed lidars enables a physically sound and controllable simulation of adverse weather conditions, such as snowfall, on real lidar point clouds, which in turn consistently enhances the performance of 3D object detectors on challenging real scenes from such conditions. On the geometry side, we will first review how the concrete prior of piecewise planarity can be "baked" into the design of networks for monocular depth estimation via an informed choice of the output space of the network. Taking one step further, a depth estimation architecture which implicitly learns distinct, generic geometric patterns via a soft discretization of the internal feature space will be outlined.

Biography

Christos Sakaridis is a lecturer at ETH Zürich and a senior postdoctoral researcher at the Computer Vision Lab of ETH Zürich. His research fields are computer vision and machine learning. The focus of his research is on semantic and geometric visual perception, involving multiple domains, visual conditions, and modalities. Since 2021, he is the Principal Engineer of TRACE-Zürich, a large-scale project on computer vision for autonomous cars and robots. He received the ETH Zürich Career Seed Award in 2022. He obtained his PhD from ETH Zürich in 2021, having worked at Computer Vision Lab. Prior to that, he received his MSc in Computer Science from ETH Zürich in 2016 and his Diploma in Electrical and Computer Engineering from National Technical University of Athens in 2014.

Abstract

Algorithmic predictions are emerging as a promising solution concept for efficiently allocating scarce societal resources. Fueling their use is an underlying assumption that predictive systems are necessary for identification---that we can target resources more efficiently through individual risk scores output by such systems. In this talk, we examine this central assumption empirically and theoretically.

Empirically, we present findings from a large-scale evaluation of Wisconsin's Dropout Early Warning System (DEWS)---an early warning system used to predict each public school student's likelihood of dropping out of high school. Using nearly a decade's worth of data, we show that DEWS accurately sorts students by their dropout risk, and it may have resulted in a single-digit increase in graduation rates. However, a simple allocation mechanism that only uses environmental information about schools, neighborhoods, and districts may have sufficed for targeting interventions just as efficiently.

We then examine this gap that emerges between predictions and allocations theoretically. Using a simple mathematical model, we evaluate the efficacy of individual prediction-based allocations with environmentally-based allocations, which only use aggregate school-level statistics. We find that prediction-based allocations outperform environmentally-based ones only when between-school inequality is low or the allocation budget is high. Our theoretical findings hold for a wide range of settings capturing heterogeneity of treatment effects, learnability of school-level statistics, and price of prediction.

Combined, our evaluation framework and analyses surface inequality as a fundamental mechanism linking prediction and allocation in settings where outcomes are structurally determined. Predictions may only improve allocations only if inequality is low.

This talk is based on joint work with Tolani Britton, Moritz Hardt, Juan Carlos Perdomo, and Ali Shirali. It is informed by discussions with Wisconsin's Department of Public Instruction, particularly Erin Fath, Carl Frederick, and Justin Meyer

Biography

Rediet Abebe is a Junior Fellow at the Harvard Society of Fellows and an Andrew Carnegie Fellow. Abebe’s research examines the interaction of algorithms and inequality, with a focus on contributing to the scientific foundations of this emerging research area. Abebe co-launched the ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (ACM EAAMO), for which Abebe serves on the executive committee and was a program co-chair for the inaugural conference. Abebe’s work has received recognitions including the MIT Technology Reviews’ 35 Innovators Under 35, the Bloomberg 50 as a one to watch, the ACM SIGKDD Dissertation Award, and an honorable mention for the ACM SIGecom Dissertation Award..

Abstract

Recent advances in visual generative models have led to the generation of high quality, diverse images of nearly any imaginable concept, thanks to increasingly large models and huge training datasets. Looking at the quality of images that these models are able to produce, one may wonder: "are we done?". In this talk, I argue that we’ve only just scratched the surface.

These models have seen billions of training images—what other knowledge have they amassed along the way? And what else can they be used for, beyond the image generation task they were trained for? My talk will cover my recent explorations in answering these questions: I'll demonstrate how the outputs of large text-to-image models can replace curated datasets for downstream supervised tasks, like image editing and 3D reconstruction. Delving deeper, I’ll show that by probing the internal representations of these models, one can extract underlying properties of the generated content, such as scene geometry, point correspondence, and more.

Biography

Aleksander Holynski is a postdoctoral scholar at UC Berkeley, working with Alyosha Efros and Angjoo Kanazawa, and concurrently a Research Scientist at Google Research.  He completed his PhD at the University of Washington, advised by Steve Seitz, Brian Curless, and Rick Szeliski, and he received his B.S. at the University of Illinois at Urbana-Champaign. His co-authored work has received a best student paper award at ICCV 2023.