Pygmalion never drew a line. He took away everything that was not her
On Cyprus there was a sculptor who would not marry. He carved a woman out of ivory instead, and carved her so well that he fell in love with the thing he had made — brought it presents, talked to it, and was thoroughly miserable.
A machine that makes pictures works exactly like the first half of that story. It never draws. It is handed a block of pure static and a few of your words, and it spends its whole time removing what is not the picture, until something is standing there.
Written for ages 12+, and for any adult who has typed a sentence into one of these, watched a picture arrive, and never once been told what happened in between. Nothing here assumes you know anything. The real engineering words are all at the bottom.
the claim
The first thing to unlearn
You almost certainly picture it starting with a blank page. A line, then another line, then some colour — the way you would do it, only faster and better.
There is no blank page. There is a completely full one, full of noise, and the machine's entire job is to make it emptier of noise, in stages, until only a picture is left over. It has more in common with cleaning a window than with drawing on one.
Michelangelo said the figure was already inside the block and all he did was chip away the surplus. That was a sculptor being romantic about his job. For this machine it is a flat, boring, literal description of the algorithm.
press the button. you get one block per press
There was no owl hidden in that block, in the way a picture is hidden in a folded piece of paper. Nothing was in there. But the block still decides everything: the same block and the same words give the same picture, every time, on any machine in the world. Hold onto that — it comes back later and it explains a lot.
step one
What the block actually is
Not marble. Static — the fizz an old television makes when it is tuned to nothing at all. Every dot is a number chosen at random, with no relationship whatsoever to the dot beside it.
It is the most thoroughly meaningless thing you can put on a screen, and that is precisely why it is useful. It is the same amount of nothing every time. You can start from it anywhere in the world and be certain you are starting from the same kind of nowhere.
Engineers call it noise, and they mean it the way you mean it at a bus station: the stuff with no message in it.
step two
So you teach it backwards
Here is the difficulty with teaching a machine to turn static into pictures. Where would the lessons come from? Nobody owns a pile of examples labelled this exact static becomes this exact owl. That pile does not exist and could not be made.
So you build the lessons in the other direction, which is easy, because ruining things is always easier than making them.
Take a real picture. Add a pinch of static. Add a bit more. Keep going — a thousand small helpings — until the picture is entirely destroyed and you are looking at a block. Now read that ladder from the bottom upwards and you have precisely the thing that did not exist: a thousand rungs of static gradually turning into a picture. And at every single rung, because you did the ruining, you know exactly what was added.
One question and its answer, a thousand times over, out of one picture. Do it to a few billion pictures. Nobody labelled anything. Nobody described anything. You only had to be willing to wreck things you already had.
drag it. this is how every lesson gets made
Nothing is learning here. This direction is plain arithmetic and takes no time at all. All of the learning happens going back the other way, and going back the other way is the entire rest of this page.
step three
All it ever learns is to spot the mess
The lesson itself is smaller than you would expect. The machine gets shown one ruined picture, and told which rung it came from. Then it is asked one question, and only ever this one: what is the mess?
Not what is the picture. What is the mess.
Its answer gets compared against the mess that was actually added, and a few of its millions of dials get nudged, very slightly, in the direction that would have made it less wrong. Then again, different picture, different rung. A few billion times. If you read part two, you have met this before — it is the same nudging that taught Echo to finish sentences, pointed at a different kind of blank.
Getting good at spotting mess sounds like a peculiar ambition, until you notice it is the other job in disguise. If you know exactly what the mess is, take the mess away, and what is left is the picture. Answer one question and you have answered the other for nothing.
Now watch the ruined end of the slider. At rung nine hundred, nobody — not you, not it, not anybody alive — could say what picture is underneath. So it gives the only honest answer available: a vague brown average of every picture that block could possibly have been. That mush in the third panel is not the machine failing. That mush is what I do not know yet looks like when you are made of arithmetic.
mark its homework. drag from barely ruined to hopeless
Drag slowly from left to right and watch panel ③ give up. It does not go wrong suddenly; it goes vague. First the details go, then the shape, then the colours slide together into the brown. That brown is the average of everything it has ever seen, and it is the correct answer to a question that cannot yet be answered.
step four
One stroke is never enough
You now have a machine that can look at a ruined picture and say what the ruin is. So take a full block, ask it once, subtract the answer, and be finished in a single move.
Try it and you get the brown. Of course you do — from a completely full block, the honest guess really is the average of everything, and the average of a few billion pictures is always brown. One stroke, one smudge.
So you take off a slice instead. Remove a fraction of what it thinks the mess is, then look again. What is in front of it now is very slightly less ruined, which means its next guess stands on very slightly better ground, which means the slice after that is better again. Twenty or thirty of those and there is something standing in the block.
The ladder has a thousand rungs but nobody makes you climb them all. You are allowed to jump, and most of the pictures you have ever seen were made in twenty to fifty strokes.
The part worth sitting with is this: it has no plan. It is not working towards something it has in mind, because it has nothing in mind and nowhere to keep it. Each stroke is a fresh look at the block by something with no memory of the stroke before it, doing exactly one thing — pointing at what does not belong. The picture is what is left over after that has happened thirty times.
set it to 1 stroke first. then set it to 30
One stroke gives you the brown, and now you know why. Two gives you a rumour of a shape. By ten there is a figure with the details still argued over, and by thirty it has settled. Sixty is barely different from thirty, which is the other half of the lesson: past a point you are paying for strokes that have nothing left to remove. The newest trick is a second machine trained to imitate the whole staircase in one jump — which gets you back to a single stroke, but an educated one.
step five
Somebody has to say what to carve
So far this produces a picture. Nothing anywhere in it is listening to you.
Your words go in at the start, turned into numbers in much the same way Echo turns a sentence into places. Then — and this is the bit people get wrong — those numbers are handed to it again at every single stroke. It does not read your prompt once and remember it. It has no memory to remember with. It reads your prompt thirty times, once per stroke, and each time it decides afresh which parts of this block do not look like the thing those words describe.
There is a dial on how hard it is made to listen, and the way it works is worth knowing, because it is such a cheap trick. Do the stroke twice: once knowing your words, once with your words taken away. Whatever is different between the two answers is the part that is your doing. Exaggerate that difference and you get obedience.
Turn the dial down to nothing and it stops caring what you asked for — you get something plausible and entirely irrelevant, which is to say you get the brown, because the brown is what plausible looks like when nobody specified. Turn it up too far and it obeys so hard the picture burns: colours go past what a colour is allowed to be, edges go crunchy, everything shouts. Nearly every picture you have admired was made somewhere in the middle.
pick a word, then push the listening dial to both extremes
Keep the same block and change only the word. Everything the block decided stays put — the ground, the light, where the frame sits — and the subject swaps inside it, because the block fixes the arrangement and the word fixes what fills it. A real image model is less tidy than this: there the two are not separate, and changing the word rearranges the whole picture. There is no owl object sitting on a background. There is one field of numbers that either does or does not look owlish.
step six
The block decides the rest
Two people type the same four words in the same second and get different pictures. Nothing mysterious about it: they were handed different blocks.
Give them the same block and they get the same picture, down to the last feather — next year, in another country, provided they are running the same model with the same software on the same kind of chip. That last condition is real: swap the graphics card and the arithmetic rounds differently, and the feathers move. The seed fixes the recipe, not the kitchen. The block is stored as a single number, called a seed, and once you have fixed it, everything that follows is arithmetic with no surprises left in it. This is why people trade seeds like recipes.
It is also why asking for "the same picture but with the ship a bit further left" is such a miserable experience. There is no ship to move. Change anything at all and you have changed the block, and a different block is a different ship, on a different sea, under a different sky.
the same word, six times, into six different blocks
step seven
He does not carve the marble. He carves a sketch of it.
A picture is an enormous quantity of numbers. A modest 512-by-512 one is 262,144 dots with three numbers each: 786,432 in total. Passing over all of that thirty times is arithmetic you can feel in the electricity bill.
So the good ones do not carve the picture at all. Long before any of this, a separate machine has been taught to squash a picture down to a small sketch and blow it back up again with almost nothing lost — which is possible because pictures are enormously repetitive, and most of those 786,432 numbers were guessable from their neighbours anyway. The sketch is around a fiftieth of the size.
Everything you have just read — the block, the strokes, your words, the stubbornness dial — happens at sketch size, on a block of sketch-shaped static. Only right at the end does the second machine take the finished sketch and paint it back up into a picture you can look at.
throw most of a picture away, then ask for it back
Real squashing is cleverer than throwing away every few dots — it keeps what matters rather than what happens to be there, which is why the right-hand panel comes back sharper in practice than it does here. The sizes are honest, though, and so is the surprise: you really can discard that much of a picture and still hand it back.
the trouble
Hands, and other things nobody counts
You have heard about the hands.
Every stroke is a local judgement. This patch here should look a bit more like knuckle. That edge should be more like the edge of a finger. Nowhere in the machine is anything counting. Five is not a fact it holds about hands; it is a habit that fingerish texture tends to stop after a while. Habits get you extremely convincing fingers and the wrong number of them.
Same reason the writing on signs comes out as beautiful nonsense. It has learned exactly what letters look like — the weight, the spacing, the way a serif sits down on a line — and nothing whatsoever about what letters are for. It is doing to the alphabet precisely what the Chimera does to animals: every part of it genuinely real, assembled into something that has never existed.
This is improving, and not because anybody taught it to count. Bigger machines hold more of the block in view during each stroke, so more of the picture is forced to agree with itself. Counting comes out as a side effect of that agreement. It is still not a rule anyone wrote down.
the other trouble
Whose pictures were the lessons?
A few billion pictures, taken off the internet with their captions attached, mostly without anybody being asked.
It is worth being exact about what ends up inside the finished machine, because both of the usual claims are wrong. It is not a folder of everyone's pictures — the whole thing is often smaller than the pile it learned from, and there is nowhere in there to keep them. But it is not innocent of them either. It is made entirely of habits lifted from other people's work, and where it happened to see one particular picture ten thousand times, it can hand you something uncomfortably close to it.
And what it saw a great deal of, it carves well. What it barely saw, it guesses at — confidently, and badly. Ask it for a wedding and notice whose wedding you get. Ask it for an Ethiopian church and count the things it has invented: how the arches sit, what the priest is wearing, which way round the drum is held. It is not being careless with you. It is telling you, quite honestly, what it was shown.
There is a whole piece coming about that, and a Greek with a bed who made every guest fit it.
the last thing
The ivory turned warm
At the festival of Aphrodite, Pygmalion made his offering and did not dare say out loud what he actually wanted. He went home and kissed the statue anyway, and Ovid says the ivory softened under his fingers the way wax gives in the sun, and yielded, and was warm.
We have the first half of that story working extremely well now. A block, a wish said out loud, and a figure standing in it that was not there this morning.
The second half has not happened and is not going to. Nothing came down. What is standing there has never been anywhere, has never seen the thing it is a picture of, and does not know that it is finished. It is a resemblance, assembled out of the way a few billion people happened to look at the world, with the surplus taken off.
Worth admiring, all the same. Pygmalion admired his too. Then he put on his good clothes and walked to the temple, to ask somebody else for the part he could not carve.
the glossary
The words the engineers use
| the block of static | the initial noise |
| ruining a picture on purpose | the forward process |
| the ladder of a thousand rungs | the noise schedule, timesteps |
| spotting the mess | noise prediction |
| what is left when you take the mess away | the predicted clean image, x₀ |
| ruin it yourself so you know the answer | self-supervised training |
| nudging the dials | gradient descent |
| one stroke | one denoising step |
| the whole carving | the reverse process, sampling |
| how many strokes you allow | sampling steps |
| a machine that imitates the whole staircase at once | step distillation |
| the thing doing the spotting | the denoiser — a U-Net or a transformer |
| your words turned into numbers | the text encoder, conditioning |
| doing the stroke twice and exaggerating the difference | classifier-free guidance |
| how hard it must listen | guidance scale, or CFG |
| the block's number | the seed |
| carving the sketch instead of the picture | latent diffusion |
| the machine that squashes and unsquashes | the autoencoder, or VAE |
| the wrong number of fingers | weak long-range coherence |
| what it happened to be shown | the training distribution |
One sentence to take away
It never draws. It is handed a field of random numbers and a few of your words, and thirty times over it points at what does not belong and takes a little of it away.
Everything impressive about these pictures and everything wrong with them falls out of that one sentence. Hold onto it and you will understand them better than most adults do.
the sources
Where these ideas come from
The sculptor is Ovid's; the carving-by-taking-away is real mathematics from these papers. Each note says what the paper actually established.
- J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan & S. Ganguli, “Deep Unsupervised Learning using Nonequilibrium Thermodynamics”, ICML (2015). The origin of the whole idea, borrowed from physics: slowly ruin data with noise, then teach a machine to run the ruin backwards. The block-of-marble mathematics, five years before it made good pictures.
- J. Ho, A. Jain & P. Abbeel, “Denoising Diffusion Probabilistic Models”, NeurIPS (2020). The paper that made it work: a simpler homework question — just point at the noise — that suddenly produced state-of-the-art images and launched the modern image generators.
- R. Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models”, CVPR (2022). Carving the sketch instead of the statue: do the diffusion in a compressed space and the cost collapses. This is the paper behind Stable Diffusion, and why these tools run on ordinary computers.
- J. Ho & T. Salimans, “Classifier-Free Diffusion Guidance” (2022). The listen-harder knob: carve the stroke twice, once with your words and once without, and exaggerate the difference — the “guidance scale” every image tool exposes, explained by its inventors.