• Co-Supervisor :
  • Augustin COSSE (LISIC)
  • Funding : ANR, ULCO
  • Start year :
  • 2024

The last few months have seen an acceleration in the development of machine learning (ML) and artificial intelligence (AI) models, in particular due to the advent of generative adversarial networks, reinforcement learning and the recent impressive progress in large language models (LLMs). Examples of spectacular achievements following those developments include the AlphaZero and AlphaGo victories in Go, the iconic feats of DeepMind and OpenAI on numerous Atari games and the development of an associated ecosystem as well as the stunning performances of ChatGPT, Llama and Bard. Although the paradigm shift (in the strict sense of a new set of theories and methods accepted by the community to guide inquiry) already started to reveal itself a few years ago with the appearance of graphical processing units, it has now become very clear that the recent breakthroughs (especially in terms of language models) will not only fundamentally reshape the society as we know it but might also shed a new light on our understanding of cognitive processes. Be it in terms of jobs, or in scientific terms (regarding our understanding of the brain for example), the recent progress in automatic language generation and understanding appears almost as important as the development of the internet around the 1970’s or the invention of the steam engine around the 1750’s. If the steam engine produced a fertile ground for the formulation of modern thermodynamics, one might hope that LLMs will lead to similar insights regarding the functioning of the brain.

In an urge to demonstrate efficiency on general languages, the ML community moved in a lapse of only a few years from small scale chatbots (whose functioning was already not fully understood) to highly elaborate models which are resisting every form of scientific investigation. LLMs are currently made of billions of parameters trained on complex and highly diverse language corpus and involve advanced architectures. What is perhaps more impressive and came to light only recently (see the seminal paper of Wei et al. below) is that with the growth in the number of parameters, comes a series of new abilities that ’emerge’ successively at distinct “critical scales” . Those abilities include e.g. the execution of arithmetic operations, the capability to summarize simple text passages, or a capacity for answering simple questions.

Just as the steam engine paved the way for modern thermodynamics, we believe now is the right time to derive a mathematical understanding of those new models. The thesis will study the connection between the linguistic properties of the data, the complexity of language models and the skills of those models through (i) a careful design of a simple dataset in collaboration with linguists (ii) a clear understanding of the structure of language models and (iii) a mathematical characterization of the transitions in the emergence of skills.