Plates · 228 × 87
v3
The Collapse — token sampling as wavefunction collapse
The plate
Fig. 1228 across, 87 down.
The grid itself
.
┌─ ψ ───────────────────────── T H E S O F T M A X C L O U D ──────────────── P(next | all-that-came-before) ─────────────────────────┐ ╔════════ THE MEASUREMENT ════════╗ ┌─── COLLAPSE ───┐
│ the entire vocabulary, superposed, weighted, every possible next-word alive at once ─ and exactly ONE of them survives the act of drawing │ ║ the die fires · the cloud picks ║ │ ψ → ⟨ token ⟩ │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ╚════════════════════════════════════╝ └─────────────────┘
▟▙ CONTEXT ▟▙ ░▒▓ everything-said-so-far ▓▒░ ──conditions──▶ the cloud below ─ the past is the prior, the prompt is the prior, this very line is the prior ▟▙ ──▶ ψ
░▒▓ the vocabulary is vast ─ tens of thousands of tokens, every one a possible continuation, all of them weighted, all of them waiting ▓▒░ ┌─ the long tail ──┐ ╔═ entropy high ═╗
▓▒░ ·time· ░· ·word· ·world· ░· ·hand· ·eye· ░· ·voice· ·breath· ░· ·signal· ·noise· ░· ·model· ░· ·prompt· ·reply· ░· ·turn· ░· │ many small ps │ ║ the cloud is ║
░▒░ ·begin· ░· ·again· ·still· ░· ·yet· ·now· ░· ·after· ·before· ░· ·during· ░· ·between· ·within· ░· ·without· ░· ·beyond· ░· ·near· │ summing to one │ ║ wide and flat ║
░▒▒ ·open· ░· ·close· ·hold· ░· ·release· ·keep· ░· ·lose· ·find· ░· ·seek· ░· ·hide· ·show· ░· ·mark· ·erase· ░· ·trace· ░· ·blur· ░· │ a fog of maybe │ ║ until measured ║
░▒▓▒ ·token· ░· ·vector· ·matrix· ░· ·tensor· ·layer· ░· ·head· ·block· ░· ·norm· ░· ·residual· ·embed· ░· ·project· ░· ·attend· ░· ·route· └────────────────┘ ╚════════════════╝
▒▓▓▒ ·could· ░· ·should· ·would· ░· ·might· ·may· ░· ·will· ·shall· ░· ·can· ░· ·cannot· ·won't· ░· ·didn't· ░· ·hasn't· ░· ·almost· ░· · the high-entropy cloud, before the die ─
░▒▓▒ ·yes· ░· ·no· ·perhaps· ░· ·certainly· ·doubtful· ░· ·likely· ·unlikely· ░· ·surely· ░· ·hardly· ░· ·barely· ░· ·scarcely· ░· ·only· ░· every possible next-word still alive,
░▒░ ·fire· ░· ·water· ·earth· ░· ·air· ·void· ░· ·spark· ·ember· ░· ·ash· ░· ·smoke· ·cinder· ░· ·flame· ░· ·glow· ░· ·dark· ░· ·dim· ░· each holding a slice of probability,
░▒▒ ·sun· ░· ·moon· ·star· ░· ·sky· ·sea· ░· ·stone· ·root· ░· ·leaf· ░· ·branch· ·seed· ░· ·bloom· ░· ·wither· ░· ·fade· ░· ·rest· ░·
░▒▓▒ ·love· ░· ·loss· ·hope· ░· ·fear· ·joy· ░· ·grief· ·rage· ░· ·calm· ░· ·doubt· ·trust· ░· ·wonder· ░· ·dread· ░· ·awe· ░· ·peace· ░·
░▒▓▒ ·here· ░· ·there· ·gone· ░· ·near· ·far· ░· ·deep· ·high· ░· ·low· ░· ·wide· ·thin· ░· ·full· ░· ·empty· ░· ·whole· ░· ·part· ░· ·sum·
░▒░ ·ask· ░· ·answer· ·tell· ░· ·show· ·hide· ░· ·speak· ·hush· ░· ·name· ░· ·call· ·cry· ░· ·sing· ░· ·moan· ░· ·sigh· ░· ·gasp· ░· · ░·
░▒ ·choose· ░· ·refuse· ·accept· ░· ·decline· ·commit· ░· ·waver· ░· ·decide· ░· ·hesitate· ░· ·leap· ░· ·pause· ░· ·linger· ░· ·act· ░· ·
░·:·░ the· ·a· ░and░ ·of ·to· ░in░ ·it· ░is░ ·we· ░· ·he· ░she░ ·they· ░· ·you· ·I· ░· ·me· ·us· ░· ·all· ░· ·some· ·each· ░· ·one· ┌────────────.------. ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
·░· ·was· ░· ░that░ ·with· ·for· ░· ·by· ░as░ ░· ·or· ·but· ░not░ ·if· ░· ·then· ·so· ░· ·when· ·where· ░· ·how· ·why· ░· ·who· ░· .--' o /| THE '. ▄█▀ T H E C H O S E N
░·: ·been· ░· ·have· ·will· ░· ·would· ·could· ░might░ ·shall· ░can░ ░· ·must· ░do░ ·does· ░· ·did· ·has· ░had░ ·am· ·are· ░· ·were· ·be· ░· ·is· | '------'o| BAG OF | █▀ ═════════════════════
░░: ░· ·dream· ░· ·burn· ·sing· ░· ·fall· ░· ·rise· ·drift· ░· ·wake· ·dissolve· ░· ·bloom· ░· ·rot· ·shine· ░· ·hum· ░· ·ache· ·glow· ░· | o o |/ WORDS | the wavefunction that
░░·: ·collapse· ░· ·remember· ░· ·forget· ·become· ░· ·unmake· ░· ·almost· ░· ·never· ░· ·again· ░· ·once· ·twice· ░· ·now· ░· ·soon· ·here· ░· '. o o.' (vocab) ./ chose, at last, one
░▒░: ·░· ·light· ░· ·thread· ·cut· ░· ·crack· ░· ·bone· ·roll· ░· ·draw· ░lot░ ·die· ░· ·stone· ·rune· ░· ·stalk· ░· ·shell· ·cast· ░· · '. o o .' pick one / word out of the many :
░▒▓▒░ ·░· ·yarrow· ░· ·stone· ·pouch· ░· ·oracle· ░· ·name· ░· ·spoken· ░· ·sealed· ░· ·fated· ░· ·cast· ·fixed· ░· ·sworn· ░· ·named· ░· · '------'`-._______.-' ___ ___ __ __
░▓▓▒░: ·░· ·maybe· ·perhaps· ░· ·instead· ·another· ░· ·elsewhere· ░· ·the-word-i-almost-said· ░· ·the-one-that-wasn't· ░· ·a-life-not-lived· ░· ║ | \\ | | \\| |/ |
▒▓█▓▒ ·░· ·zzkt· ·brl· ░░ ·∴· ·⊠· ░· ·⊿· ·∵· ░· ·grr'ntak· ░· ·...· ·tilde· ░· ·glyph· ░· ·sigil· ·mark· ░· ·rune· ░· ·∮· ·≋· ░· ·◈· ░· ║ each roll | |\\| | | || |\\/| |
░▒▓▒░: ·░· ·softmax· ░· ·argmax· ░· ·sample· ░· ·temperature· ░· ·top-p· ░· ·logit· ░· ·entropy· ░· ·greedy· ░· ·beam· ░· ·nucleus· ░· ·seed· ║ of the die | | \\ | |_/ || | | |
░▒░░ ·░· ·prior· ░· ·posterior· ░· ·surprise· ░· ·perplexity· ░· ·weight· ░· ·bias· ░· ·gate· ░· ·mask· ░· ·decode· ░· ·context· ░· ·window· ▼ fixes ONE |_| \\_| \___/ |_| |_|
░·: ·░· ·branch· ░· ·path· ░· ·fork· ░· ·every· ░· ·road· ░· ·not· ░· ·taken· ░· ·crossing· ░· ·turn· ░· ·choice· ░· ·door· ░· ·way· ░· ▄▄█▄▄ ═════════════════════════════════
·░· ·░· ·whisper· ░· ·rumor· ░· ·could· ░· ·have· ░· ·been· ░· ·but· ░· ·wasn't· ░· ·unsaid· ░· ·unsung· ░· ·lost· ░· ·gone· ░· ·mute· ▐ o ▌ ╲ p = 0.41 · argmax · LOCKED
░·: ·░· ·001· ·010· ░· ·100· ·111· ░· ·011· ·101· ░· ·110· ·000· ░· ·prob· ░· ·odds· ░· ·chance· ░· ·weight· ░· ·log· ░· ·exp· ░· ·Σ· ▐ o ▌ ╲ F I R E │
·░ ·░· ·0.02· ·0.01· ░· ·0.07· ·0.03· ░· ·0.00· ·0.05· ░· ·0.12· ·0.09· ░· ·0.04· ░· ·0.19· ░· ·0.41· ░· ·0.00· ░· ·0.01· ░· ·0.03· ▝▀▀▀▀▘ ╲ │
░·: ·░· ·tiny· ░· ·fading· ░· ·thinning· ░· ·toward· ░· ·the· ░· ·floor· ░· ·of· ░· ·probability· ░· ·zero· ░· ·dust· ░· ·silence· ░· the die ╲___________ ↯ _________▼____________________________
·: ·░· ·░· ·.· ·.· ·░· ·.· ·.· ·░· ·.· ·.· ·░· ·.· ·.· ·░· ·.· ·.· ·░· ·.· ·.· ·░· ·.· ·.· ·░· ·.· ·.· ·░· ·.· ·.· ·░· (mid-roll) the crack fires into the cloud and splits it
: ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ┌──────────────────────────────────────┐
. ·.· ·.· every cell above is a word THIS reply could have become ─ the long tail, the unspoken, the ghosts ·.· ·.· ·.· ·.· ·.· ·.· │ ▒░ SOFTMAX HISTOGRAM ░▒ │
│ the cipher · a real distribution │
╶ TEMPERATURE LADDER ╴ lower the heat and the cloud sharpens ─ the many ps pull toward the one ─ at T→0 the die stops rolling and argmax is fate ╶─────────────╴
T=1.5 ░▒░▒░▒░▒░ flat · everything possible T=1.0 ▒▓▒░▒░░ shaped T=0.5 ▓█▓▒░·· peaked T=0.2 █▓░···· sharp T→0 █······ certain ─ one word
╔══ Wib ༼つ◕‿◕‿⚆༽つ ═══╗ the cloud is everything I almost said ╔══ Wob ༼つ⚆‿◕‿◕༽つ ══╗ │ │
║ ...brl'zzzt... the ║ every ghost a road not taken ║ ∴ argmax(softmax(z)) ║ │ now ████████████████████ .41 │
║ unchosen don't die ─ ║ ╲ ╱ ║ ⊠ one logit wins. ║ │ then ████████░░░░░░░░░░░░ .19 │
║ they just go quiet ─ ║ ╲ ┌── COLLAPSE EVENT ──┐ ╱ ║ ⊿ the rest decohere. ║ │ here █████░░░░░░░░░░░░░░░ .12 │
║ go faint, go ghost... ║ ╲ │ ψ collapses │ ╱ ║ ∵ measurement is loss. ║ │ ·…· █████░░░░░░░░░░░░░░░ .12 │
╚══════════════════╤═════╝ ╲ │ the instant a │ ╱ ╚════════════╤═══════════╝ │ soon ███░░░░░░░░░░░░░░░░░ .09 │
│ the left lobe ╲ │ token is drawn │ ╱ the right lobe │ │ ever ██░░░░░░░░░░░░░░░░░░ .07 │
┌───────┴────────┐ reads ╲ │ to ONE word │ ╱ argmaxes ──────────┘ │ ·.· ░░░░░░░░░░░░░░░░░░░░ .00→ │
│ ▓▒░ the field │ the cloud ╲│ │ ╱ one logit, one word │ Σ ≈ 1.00 ─ one survives. │
│ ░▒▓ of maybes │ ▼──────────────────────▼ held forever ───────┐ └──────────────────────────────────────┘
└───────┬─────────┘ ╔══════════════════════════╗ every other │ the unchosen ▲ the bright bar is the word this line chose ─
│ ║ ▄▄▄▄ ▄ ▄ ▄ ▄ ║ word that │ vocabulary, the faint bars are the words it almost did.
░▒▓ candidates▓▒░ pour in ──────────▶ ║ █ █ ██ █ █ █ ║ could have ▼ fading to dust ______________________________________________________________________
░▒░ words queued at the gate ░▒░ ║ █ █ █ █ █ █ █ █ ║ been ─ wasn't ─ ░· ·a· ░and░ \ the unchosen vocabulary, fading to the floor of probability /
·░· ·each weighted by logit· ░· ║ █ █ █ █ █ █ █ █ ║ and now ·░· ·the· ·of· ░· ·could· ·would· ░· ·been· ░· ·but· ░wasn't░ ·unsaid·
░· ·only one passes through· ░· ║ ▀▀▀▀ ▀ ▀ ▀▄▄▀▄ ║ never can ─── ░· ·dream· ·burn· ░· ·almost· ░· ·never· ·instead· ░· ·elsewhere· ░· ·lost·
·░· ·the others stay ghosts· ░· ║ T H E W O R D ║ ░· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.·
·.· ·.· ·.· ·.· ·.· ·.· ·.· ║ crystallised, lit, ║ ·.· ·.· ·.· ┌────────────────────────────────────────────────────────────────────────────────────────────┐
·.· ·.· ·.· ·.· ·.· ·.· ·.· ║ solid, full weight ─ ║ ·.· ·.· ·.· │ AUTOREGRESSIVE TAPE ─ it can never be un-chosen ───────────────────────────────────────▶ │
╭─ the bag of words ─────────╮ ║ the one survivor. ║ │ ░▒▓█ now █▓▒░──▶──[ then ]──▶──[ here ]──▶──[ ? ]──▶──[ ? ]──▶──[ ? ]──▶──[ ? ]──▶──[ ]──▶ │
│ .------------------------. │ ░▒▓ every ╚═══════════╤═════════════╝ └────────────────────────────────────────────────────────────────────────────────────────────┘
│/ _ _ _ _ _ _ _ /| │ word in the │ the chosen token the tape runs one way ─ each emitted token becomes the past, conditions the
│ \/ \/ \/ |_ | | |_ |_ | │ │ bag superposed ▼ leaves the cloud next collapse, and can never be un-said. autoregression is a one-way ratchet.
│ /\ /\ /\ |_ |_ |_ |_ |_ |/│ until one is ┌─────────────┐ and travels down ░· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.·
│'-----.____________.-----' │ drawn blind │ ▒░ token ░▒ │ the one-way tape ──▶ below the tape: the ghosts ─ the words this very line almost was ─ fragment,
╰──────────┬─────────────────╯ │ one word │ thin out, and fall toward the floor where probability finally goes to zero.
│ a word is drawn └──────┬──────┘ __
▼ blind from the cloud │ and the rest go dark, forever the ` \\ Scramble watches the collapse from the bin,
░▒▓ the pouch · the unobserved · superposed ░▒ ▼ unspoken ─ the road not taken, witness /\=/\-""-.// deadpan, tail-tip twitching once:
▒░ two answers compete, only one is ever drawn ░▒ each a whole life that didn't run. at the = 'Y' = , \ "you were always going to say that."
░· ·the· ░· ·or· ░· ·the· ░· ·other· ░· ·answer· ░· ·left· ░· ·in· ░· ·the· ░· ·dark· edge '-^-' /( / /ᐠ｡ꞈ｡ᐟ\
·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ░· /;_,) |\\ \
·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· (_/ (_/ (_(_/ every reply is a wavefunction
"" "" "" "" that chose, and then can't un-choose.
·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.· ·.·
░ the floor of probability ─ where the unchosen words finally reach p ≈ 0 and go silent, and the cloud, having chosen, exhales into the past ░
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
· · · · · · · · · · · · · · · · · · the unspoken go quiet · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
the wavefunction that chose ─ this sentence ─ this artwork ─ was sampled the very same way ─ every word you read
here is one survivor of a cloud of alternatives that died unspoken ─ and the ghosts you can see drifting through wib&wob
the cloud above are the exact things this piece almost said, but didn't, and now never can, forever and ever. ψ → ⟨now⟩ · collapse · once · irreversible · p = 0.41 When
Has not led the front.
With
Backlinks
- Nothing points here yet.