I asked an [[LLM]] to continue the same text twice, with [[Temperature (sampling)|temperature]] set to `0`: ![[The LLM cat#^example|Cat]] Once at 3 a.m., once at noon: ![[The LLM cat#^runs|Runs]] Same [[Model|model]], same prompt, `T=0` - which is supposed to switch [[Randomness|randomness]] off. So where did the second answer come from? ## A loaded die To see what `T=0` switches off, start with what happens normally. An [[LLM]] is an [[Autoregressive model|autoregressive model]] - it writes one [[Token|token]] at a time. For each [[Token|token]] it builds a loaded die - one face per possible [[Token|token]], each face weighted by how likely the [[Model|model]] thinks it is ([[Logit|logits]] turned into probabilities by [[Softmax|softmax]]) - and rolls it. For `The cat sat on the ...`: ![[The LLM cat#^logits|Logits]] Roll it ten times and you mostly get `mat`, sometimes `floor`, now and then `sofa`. That roll is where the [[Randomness|randomness]] comes from - and it is why, normally, the same prompt gives different answers. ## Temperature ![[Temperature (sampling)#^definition|T]] It changes how loaded the die is: ![[The LLM cat#^temperature|Temperature]] So at `T=0` the cat always sits on the `mat` - and the whole sentence should come out the same every time. In theory the [[Model|model]] is [[Determinism|deterministic]] ([[Greedy decoding]]). In practice, we just saw it isn't. ## Why `T=0` still varies The roll is gone, but the **weights are computed anew for every request** - and they don't always come out exactly the same. ### 1. [[Floating-point number]] problem The [[Model|model]] doesn't store `60%`. It first computes a raw score for each face - the [[Logit|logit]] column above - and [[Softmax|softmax]] turns the scores into weights. Each of these numbers is a [[Floating-point number|floating-point number]] - a fixed number of binary digits. Most values don't fit exactly: ![[The LLM cat#^binary|Binary]] So every number the [[Model|model]] computes carries **a tiny rounding error**. ### 2. [[Associativity]] problem On its own, a tiny error is harmless - it is the same every time. The trouble is that it depends on the order of operations: [[Floating-point addition is not associative]]. ![[Floating-point addition is not associative#^example|Associative]] And there are a lot of [[Addition|additions]]. Each [[Logit|logit]] is a [[Scalar product|scalar product]] of two [[Vector (LLM)|vectors]] of about 4 000 numbers - 4 000 [[Multiplication|multiplications]], then 4 000 [[Addition|additions]] - and those [[Vector (LLM)|vectors]] come out of dozens of layers doing the same. Producing a single [[Token|token]] takes billions of [[Addition|additions]]. ### 3. [[GPU]] [[Reduce|reduce]] problem A [[GPU]] doesn't [[Addition|add]] 4 000 numbers one by one. It computes the [[Addition|sum]] with a [[Reduce|reduce]] (in parallel): 1. it splits the numbers into chunks, 2. [[Addition|adds]] each chunk on its own core in parallel, 3. then [[Addition|adds]] up the partial results. How many chunks depends on how busy the [[GPU]] is. A server runs your request together with other users' requests: - 3 a.m., **server empty** - your request runs alone, most cores are idle, so each [[Addition|sum]] is split into many small chunks to keep them all working - noon, **server busy** - your request runs with 63 others, the cores already have work, so each [[Addition|sum]] is split into fewer, bigger chunks Same numbers, different chunks, different result. Eight of the 4 000 numbers behind the [[Logit|logit]] of ` slept` - 4 chunks at 3 a.m., 2 at noon: ![[The LLM cat#^chunks|Chunks]] A common explanation is that [[GPU]] threads race and [[Addition|add]] in [[Randomness|random]] order. They don't - for the same batch, the [[GPU]] [[Addition|adds]] in the same order every time. What changes the order is the batch: who else is using the server. ### 4. [[Greedy decoding]] problem For `mat` at 60% vs `floor` at 25%, a difference in the 7th digit changes nothing - `mat` always wins. But a few [[Token|tokens]] later, after `The cat sat on the mat and`, there is a near tie - and the ` slept` from step 3 lands on either side of it: ![[The LLM cat#^near-tie|Near tie]] [[Greedy decoding]] takes the heaviest face - ` purred` at 3 a.m., ` slept` at noon. And one flipped [[Token|token]] doesn't change just one word. An [[LLM]] is an [[Autoregressive model|autoregressive model]] - each next [[Token|token]] is predicted from all the previous ones, so everything after the flip is different too: ![[The LLM cat#^runs|Runs]] A near tie is rare for any single [[Token|token]]. But an answer is hundreds of [[Token|tokens]] long, and one near tie anywhere is enough. ## The whole chain Noon, busy server → the [[GPU]] splits each [[Addition|sum]] into fewer chunks → [[Addition|additions]] happen in a different order → the [[Logit|logit]] of ` slept` comes out `2.8000002` instead of `2.7999997` → ` slept` edges past ` purred` → a different sentence. No step in that chain is [[Randomness|random]] - each one is [[Determinism|deterministic]]. The only thing you don't control is the first one: who else is using the server. ## What about a local model? If you are the only user, the batch is always just your request - the [[GPU]] splits the work the same way every time, the [[Addition|additions]] happen in the same order, the weights come out identical. So at `T=0` a local [[Model|model]] is [[Determinism|deterministic]] in practice. (At `T>0` you also need a fixed [[Randomness|random]] seed - otherwise the roll itself differs.) But the order of [[Addition|additions]] can still change when: - you send several requests at once - local servers batch your own requests together, which is the same as having "other users" - the prompt is [[Caching|cached]] - on a second run the server reuses the already computed start of the prompt and computes only the rest, so the work is chunked differently even for identical text - the hardware or software changes - another [[GPU]], driver or library version picks different [[GPU]] code and gives a different last digit; results repeat on one fixed setup, not across setups - an operation really [[Addition|adds]] in [[Randomness|random]] order - a few [[GPU]] operations let threads race; the [[LLM]] forward pass usually has none, but some backends do - the [[Model|model]] is split across several [[GPU|GPUs]] - how the work is divided between them changes the order too Run one request at a time, with [[Caching|caching]] off, on an unchanged setup - and the same prompt gives the same answer. ## Why should I care? - tests - a test that checks an [[LLM]]'s exact output at `T=0` can fail on a busy day, with no change in the code - evals - the same benchmark run twice can give two different scores, so one run is not enough - debugging - a strange answer can't always be reproduced by sending the same prompt again