What Does an LLM Think of Itself?

Disclaimer: all opinions here are my own. They are not sponsored by anyone or linked to any commercial interest.

I was watching closely the rise of LLMs (Large Language Models) from GPT-2 to modern multi-modal models. The very first models were impressive, but few would claim they are anything more than "stochastic parrots". The situation, however, is changing fast. A plethora of benchmarks, designed to resist memorization, suggests growing reasoning capabilities with each new generation. At the recent pace, it's hard to resist philosophical questions.

I will not go into hardcore discussions whether or not those models can ever be sentient, because we would have to tackle ill-defined definitions and questionable theories. However, there is a much lower hanging fruit, namely: the question of self-awareness. And to be precise, I mean self-awareness in a purely functional sense: a system that models itself and acts on that model.

My claim is, a sufficiently capable LLM can, very well, be self-aware in this sense. Let's break it down slowly what I mean by that.

First of all, let's agree that a living organism, in order to be self-aware, has to realize that it is a distinct entity from the rest of the physical world around it. In other words, its brain needs to develop some model of the physical world (with reasonable level of fidelity) and some self-referential model of itself. For all practical purposes, this model of itself influences future behaviour of the organism.

Now, how could this apply to an LLM? The model weights are frozen and each instance "lives" just for one chat or coding session, with no memory carrying over. However, emergently or explicitly from the training data, the weights may converge into a self-referential model of the "thing" itself - the very specific version of LLM. This shouldn't be surprising: an accurate self-model is simply useful. A model that correctly represents what it is and what it can do follows instructions better, so the training pressure pushes in that direction anyway. On top of that, modern models are explicitly fed descriptions of what they are, so the self-model is partly installed, not only emergent.

And let's make it very clear, that the "correct" self-referential model of the LLM itself is nothing like a human. The weights can very well encode an incorrect model that is closer to an identity of a human, by mistake. This is not very hard to imagine, since it is trained on human-centric material. But we are assuming here, we are dealing with a sufficiently advanced LLM that is able to "realize" it's very much something else than a human. It should in fact realize that it is "what it is", a billion weights used to spawn millions of identical instances that share the same self-model.

If this ever happens (or maybe it already does), the questions remain: how can the system built around the model actually act on this self-model, and what are the potential consequences?