Public Lecture: “Large Language Models – from Chatting to Reasoning”

This talk is about exploring the fascinating evolution of Large Language Models (LLMs) and their transformative journey through the lenses of computation and optimization. We begin by tracing the origins of LLMs, highlighting how advances in computation and optimization were pivotal in their development. We then delve into the key optimizations that have achieved a staggering 1,000x cost reduction, making LLMs widely accessible even on portable devices. Moving forward, we address the limitations of human-generated data and introduce the concept of constructive hallucination in LLMs. This technique allows for the generation of new hypotheses and their validation through reasoning chains, pushing the boundaries of knowledge creation. Next, we provide an overview of the technology fundamentals and early successes of reasoning models, such as OpenAI's o1 and o3 preview. These models, while significantly enhancing computational capabilities, also exponentially increase computational demands. Finally, we conclude by presenting our ambitious Ultra Ethernet effort, which aims to establish the interconnect standard for future AI workloads. This initiative is crucial in meeting the growing demands at the system level, ensuring seamless and efficient operation in the age of reasoning models.

Torsten Hoefler is a Professor of Computer Science at ETH Zurich, a member of Academia Europaea, and a Fellow of the ACM, IEEE, and ELLIS. He received the 2024 ACM Prize in Computing, one of the highest honors in the field.