> The raw score an [[LLM]] gives every [[Token|token]] in its vocabulary when predicting the next one - higher means more likely, but it is not yet a probability. ^definition ## What it is exactly After reading the text so far, the [[Model|model]] ends up with one long list of numbers - a [[Vector (LLM)|vector]] - describing "what should come next". It also keeps one stored [[Vector (LLM)|vector]] per [[Token|token]] in its vocabulary. The logit of a [[Token|token]] is how well the two match - their [[Scalar product|scalar product]]. One number per [[Token|token]] - about 200 000 numbers at every step. It is not a probability because: - it can be any real number - negative, or far above 1 - the logits don't [[Addition|sum]] to 1 - only differences between them matter - [[Addition|adding]] the same number to every logit changes nothing A logit is a log-probability up to a constant: $z_i = \log p_i + C$. To get real probabilities, the constant has to be found and removed.