Picking the puppy

Can you tell, early in a training run, whether it’s going to come good? In the last post I evolved a sheepdog three times with different random seeds. One run had a dog penning whole flocks by generation 2, one by 17, and one not until generation 114. Had I only run the third, at generation 50 I’d have been staring at a flat line, deciding whether to give up on it and rehome it. Another run costs me a minute of laptop, but for a lab that has burned millions it’s a serious question. So: lots of runs, watched early, to see whether generation 15 can predict generation 120.

The setup #

Same dog, same sheep, same open field and pen as last time.

One thing about the score. It’s sheep penned, plus a bonus for finishing quickly, plus up to 0.3 for how close the loose sheep got to the pen. That last bit was added last time so the early generations aren’t all zero, and it means the score curve is already a hand-built early signal. Worth remembering when we ask whether anything beats it.

The lottery #

Every run sits on a floor until one generation a dog pens a whole flock and the score jumps from about 0.3 to 1.5. Here is when that happened for each of the 100 evolution runs, coloured by how the run ended.

ended in the top quarterthe middleended in the bottom quarter

Half the runs had a penning dog by generation 12, three quarters by generation 60, and 11 never did. The colours are the point. Of the 48 runs that jumped by generation 15, 10 ended in the bottom quarter, and 7 of the top-quarter runs didn’t jump until after generation 30. Exam scores ran from 0.19 to 2.04, with 39 dogs that penned all thirty flocks and 20 that couldn’t reliably pen one.

Can you pick the puppy? #

So here’s the game. Four real training runs from the experiment, paused at generation 15, which is an eighth of their budget. For each one you get what a score-watcher would see, which is its score curve so far, and what a shepherd would see, which is the best dog from its latest batch working the same flock as the other three. One of the four ends up the best dog of the litter. Back one.

Once you’ve picked, the chart plays the four futures out, and each field switches to the dog that puppy grew up to be, on the same flock, so you can see what another 105 generations bought. The tally keeps your score against the score-watcher (who always backs the highest curve) and against luck (which gets one in four).

Play a few rounds and two things come through. The first is that at generation 15 most of the puppies look equally hopeless. A dog that has penned a flock in training usually loses one on the demo flock, and the ones that haven’t all do the same thing, which is get roughly behind the sheep and shove. The second is that the score-watcher isn’t much of a rival. Over the draws this game makes, backing the highest score so far picks the winner 47% of the time, against 25% for guessing. Whether a run had produced a penning dog by generation 15 tells you almost nothing about how it ends: 40 of the 48 that had went on to finish above 1, and so did 40 of the 52 that hadn’t.

There is a trick, and it’s a fair one. Look at where each line is now, not where it peaked. Every run trains on the same flocks in the same order, so at generation 15 all four puppies have just sat the same two-flock test, and comparing those scores is comparing like with like. A peak back at generation 8 might just mean generation 8 had an easy flock. Backing the best current score picks the winner 69% of the time in this game. The puppy that peaked at 1.09 and is sitting on 0.19 is the one to avoid.

What the signals know #

The game is four runs at a time. This is the same question over all hundred: at nineteen checkpoints in every run I logged everything an early observer could reasonably know, then measured how well each of it predicted the exam.

best score so farbest score this generationscore on probe flockstime behind the flockflock progress to pengenome diversity

best score this generation ρ 0.36 at 15, 0.71 at 120 The best thing you can see at 15, and it isn't much.

best score so far ρ 0.18 at 15, 0.72 at 120 The thing you'd naturally watch. Even at the end of training it only agrees 0.72 with the exam.

spread of scores in the batch ρ 0.40 at 15, 0.66 at 80 Looks good, but it's the maximum in disguise: when one dog pens a flock the two jump together (they correlate at 0.94), so it isn't on the chart.

score on the probe flocks ρ 0.04 at 15, 0.78 at 120 Near zero at 15, and only catches the score curve at about generation 60.

flock progress to pen ρ 0.04 at 15, 0.69 at 120 Near zero at 15, catches up at about generation 60.

time behind the flock ρ 0.10 at 15, 0.41 at 120 Hovers around zero until generation 80.

time close to the sheep ρ -0.19 at 15, -0.58 at 80 Points the other way: the dogs that end up best press the flock rather than keeping their distance. The strongest behavioural signal late on, and nothing at 15.

genome diversity ρ 0.11 at 15, -0.46 at 50 Slightly positive early, then negative from 25, because a population that has converged has found something worth converging on. Not a leading signal, but a sign that a jump is real.

The probes were the ones I wanted to work. My best guess at why they don’t is that the score is a statistic of a whole batch of 32 dogs, while a probe watches one dog on three flocks, and early on one dog is flaky: it pens one flock and loses the next.

The same thing done the way the game does it: draw four runs at random, back the one the signal ranks top, and count how often that turns out to be the best of the four.

best score so farbest score this generationscore on probe flockstime behind the flockflock progress to penguessing

Picking the best of fourHow often the run a signal ranks top at that generation ends best of a random four. Guessing gets 25%.open

signal read at generation…	5	15	30	50	80	120
best score so far	32%	35%	42%	47%	48%	50%
best score this generation	36%	40%	47%	49%	48%	48%
score on three probe flocks	35%	35%	43%	46%	54%	58%
flock progress to pen	32%	32%	42%	39%	52%	45%
time behind the flock	28%	35%	37%	36%	36%	38%
guessing	25%	25%	25%	25%	25%	25%

Keep or rehome #

Prediction is only worth anything if it changes what you do with your compute. The practical version, which Hyperband and successive halving built an industry on, is triage: start more runs than you can afford to finish, rehome the unpromising ones early, and give the ones you keep the budget. At the same total compute, is that better than running fewer to the finish?

The sums: the budget is four full runs. Triage starts eight, rehomes at some generation down to however many the leftover budget can carry to the end (three up to generation 20, two up to 40, one at 50, never over budget), and keeps the ones with the best score so far. Both policies hand over their best dog. This is how often triage wins, over a few thousand draws from the real runs:

evolutionhill climbingevolution strategyrehome at random

On the evolution runs it doesn’t pay: about 50% at every generation, a draw. Rehoming at random gets 40%, which is the price of finishing three runs instead of four. The reason is the base rate. 39 of the 100 runs end with a dog that pens everything, so the best of four random runs already averages 1.99 out of a possible 2.04, and there’s nothing for triage to buy except the 10 to 20% of compute it doesn’t spend. The other two lotteries have more blanks in them, and there triage pays: 59% for hill climbing at generation 5, and two times in three for the evolution strategy at generation 15, because half of its runs never get off the floor and by then the score knows which half. Rehoming by what the dog does rather than what it scores never beats rehoming by score where triage pays, and for hill climbing it’s worse than random.

Rehoming by other signalsHow often triage at generation 15 beats four full runs, by the signal you rehome on.open

rehome at generation 15 by…	score so far	score this generation	probe score	time behind flock	flock progress	random
evolution	48%	54%	52%	52%	49%	40%
hill climbing	51%	53%	41%	37%	38%	41%
evolution strategy	66%	60%	48%	47%	49%	39%

Three ways to train, three kinds of luck #

The same study for the other two optimizers. Every run’s exam score, sorted:

evolution (the GA)hill climbingevolution strategy

Evolution is the jagged one: 39 of 100 runs end with a dog that pens everything, 20 with a dog that can’t pen one, and which is which is nearly invisible at generation 15. Hill climbing is the cautious one. One family line, keep the child only if it beats the parent, and it mostly ends up good enough: 36 of 60 runs above 1, but only 3 that pen everything. Once a lineage has a habit that works, a mutation rarely finds a better one that also beats it on the next two flocks, so it settles down. The evolution strategy was the surprise. When I was setting this up it looked stuck: it climbs the slope the shaping bonus gives it, learns to get behind the flock and push, and then sits on 0.2 for fifty generations. But 17 of 60 runs did get there, 9 of them with dogs that pen everything, and the way they get there is different. It isn’t a jump, it’s a ramp, twenty or thirty generations of the score creeping from 0.5 up to 2, mostly starting after generation 50. The other 32 never left the floor.

It is also the most predictable of the three. The best score so far at generation 15 correlates 0.59 with the ending for the evolution strategy, 0.48 for hill climbing and 0.18 for evolution. So the trade is this: the optimizer you can forecast is the one with the most duds, and the one that wins most often is the one you can’t read.

What this looks like with real money #

Scale the field up and the question doesn’t change, but the stakes do. There are two kinds of training run out there, and they’re the two shapes in this post.

I expected the transferable idea to be that what a model does carries earlier signal than what it scores. This study is no evidence for that: the score curve won, and what helped was reading it properly and starting more runs than you finish. I’d still bet the idea is right at scale, because the labs’ evaluation suites are exactly that, a probe of what a model can do rather than its loss. My dogs didn’t show it.

What I took from it #

Hover a bar for the numbers behind it.

40% How often the best signal at generation 15, an eighth of the budget, picks the best of four runs. Guessing gets 25%, so you can't really pick the puppy that early.

69% The score-watcher in the game once it compared runs on the same test at the same time rather than on their best-ever scores, which are often just an easy flock.

35% I wanted watching the dog to beat the score at generation 15 and it didn't, at least not here. One dog on three flocks against a score that sums a batch of 32, and early dogs are flaky, so it isn't the last word.

66% How often rehoming half the litter at generation 15 beats four full runs. It depends on how many runs were going to fail anyway: a draw for evolution, where four in ten come good, and two times in three for the evolution strategy, where half never get off the floor.

0.59 How well the score at generation 15 predicts the ending, per optimizer. The evolution strategy is the most predictable and the one that fails most often, and evolution is the least predictable and has the most winners.

0.72 How well a finished evolution run's training score agrees with its exam on unseen flocks. Even a finished run is a bit of a lottery, so a fair amount of the unpredictability is the exam rather than the training.

---

The sheep, the field and the dog are from the last two posts. Every number and every run in this post comes from one script on the same code as the demos, seeded: experiment script · demo code. The grown-up versions of this question: scaling laws (Kaplan et al. 2020, Hoffmann et al. 2022, and a practical guide); learning-curve extrapolation (Domhan et al. 2015, LC-PFN, NeurIPS 2023, a review of curve shapes); early stopping at scale (Hyperband); judging networks without training them (zero-cost proxies); the jump problem (emergent abilities, and the case they’re a mirage of the metric); and the evolution strategy is Salimans et al. 2017.