> The cat sat on the ... ^example ## Two runs at `T=0` The same prompt, sent at 3 a.m. and at noon, both at [[Temperature (sampling)|temperature]] `0`: | run | answer | | ------ | ---------------------------------------------------------- | | 3 a.m. | The cat sat on the mat and **purred** quietly in the sun. | | noon | The cat sat on the mat and **slept** until dinner. | ^runs ## [[Token|Tokens]] `The cat sat on the mat.` split by the GPT-4o tokenizer: | [[Token\|token]] | `The` | ` cat` | ` sat` | ` on` | ` the` | ` mat` | `.` | | ----- | ----- | ------ | ------ | ----- | ------ | ------ | --- | | ID | 976 | 9059 | 10139 | 402 | 290 | 2450 | 13 | - a [[Token|token]] usually carries its leading space - ` cat`, not `cat` - a longer word can split - `Tokenization` becomes `Token` + `ization` - the [[Model|model]] never sees letters, only the IDs - `[976, 9059, 10139, 402, 290, 2450, 13]` The tokenizer knows about 200 000 [[Token|tokens]] - that list is the [[Model|model]]'s vocabulary. ## [[Vector (LLM)|Vector]] The [[Token|token]] ` cat` stored as a [[Vector (LLM)|vector]] (illustrative values; a mid-size [[Model|model]] uses about 4 000 numbers): ``` cat → [0.12, -0.83, 0.05, 0.67, ..., 0.41] ``` ^vector ## [[Logit|Logits]] and weights `The cat sat on the ...` (illustrative values) - and about 200 000 more rows, almost all of them far lower: | face | ID | [[Logit\|logit]] | weight | | ---------- | ----- | ------ | ------ | | ` mat` | 2450 | 4.489 | 60% | | ` floor` | 8350 | 3.614 | 25% | | ` sofa` | 34790 | 3.034 | 14% | | ... | ... | ... | ... | | ` balcony` | | -1.908 | 0.1% | ^logits ## Weights in binary Each weight is a [[Floating-point number|floating-point number]] - a sign, an exponent and a fixed number of binary digits, the [[Scientific notation#Significand|significand]] (often called [[Scientific notation#Significand|mantissa]]). In 32-bit float: | weight | sign | exponent | [[Scientific notation#Significand\|significand]] (23 bits) | value really stored | | ------ | ---- | -------- | ------------------------- | ---------------------------- | | 0.25 | 0 | 01111101 | 00000000000000000000000 | 0.25 exactly | | 0.6 | 0 | 01111110 | 00110011001100110011010 | 0.60000002384185791015625 | | 0.14 | 0 | 01111100 | 00011110101110000101001 | 0.14000000059604644775390625 | ^binary 0.25 is $2^{-2}$, so it fits exactly. 0.6 in binary is `0.1001 1001 1001...` forever - like $1/3 = 0.333...$ in decimal - so it is cut off and rounded. 16-bit bfloat16 keeps only 7 [[Scientific notation#Significand|significand]] bits - there 0.6 is stored as 0.6015625. ## [[Temperature (sampling)|Temperature]] Different temperature setting will return different results: | T | meaning | results | | --- | --- | --- | | high | weights evened out | `sofa` comes up often, sometimes even `balcony` | | low | | `mat` dominates, sometime `floor`, but almost none `balcony` | | 0 | no roll at all | just `mat` | ^temperature ## The [[Logit|logit]] of ` slept` A few [[Token|tokens]] later, after `The cat sat on the mat and`, the [[Model|model]] computes the [[Logit|logit]] of ` slept` - a [[Scalar product|scalar product]] of about 4 000 [[Multiplication|products]]. Eight of them, [[Addition|summed]] in 32-bit float with a [[Reduce|reduce]]: | run | chunks | partial [[Addition\|sums]] | total | | ------ | ----------------------------------------------- | --------------- | --------- | | 3 a.m. | `0.3 0.2` · `0.7 0.1` · `0.5 0.4` · `0.1 0.5` | 0.5 + 0.8 + 0.9 + 0.6 | 2.7999997 | | noon | `0.3 0.2 0.7 0.1` · `0.5 0.4 0.1 0.5` | 1.3000001 + 1.5 | 2.8000002 | ^chunks ## Near tie After `The cat sat on the mat and`, two faces are almost equal. The [[Logit|logit]] of ` purred` comes out `2.8` in both runs, the one of ` slept` differs: | face | 3 a.m. | noon | | --------- | ------------- | ------------- | | ` purred` | **2.8** | 2.8 | | ` slept` | 2.7999997 | **2.8000002** | ^near-tie [[Greedy decoding]] takes the heaviest face - ` purred` at 3 a.m., ` slept` at noon. ## [[Top-K]] `K = 3`: ```md The cat sit on the mat, chair or windowsill. ``` Ignore piano or rainbow. ## [[Top-P]] `P = 0.8` ```md The cat sit on the mat, chair, porch or windowsill. ``` They add to `>= 0.8`