What the Festival taught us

Why we try to predict a cross country race at all

Ask most people how fast a high school runner is and they will quote a best time. The trouble is that a 5K in Maine is not a 5K in Maine. Some courses are flat and fast, some climb and twist, some are a little long, and a time run in a September heat wave is not the same as one run on a cool October morning. A best time mixes all of that together. So before the Festival of Champions we set ourselves a simple test: if we account for the course, the weather and the time of year, can we say ahead of time how the biggest meet of the regular season will go, and then show everyone how close we came?

How we built the projection

We started from every result we could gather from this season and the three before it. Each race tells us two things at once: how good the runners are, and how hard that day was. A runner who is always about thirty seconds behind the same rival, on every course, gives the model a fixed point; a course where everyone runs slow tells us the course is hard rather than the runners. Put thousands of those races together and every runner gets a level and every course gets a rating.

On top of that we added the things every coach already knows. Runners get fitter as the season goes on, and younger runners tend to improve faster than seniors. Heat and humidity cost time, so we used the forecast for Belfast. And we checked our own habits: projections built from only a few races tend to place runners too far apart, so we pull every projected time a little toward the middle of the field, by an amount measured on past Festivals.

Then we did the part that matters most. On Thursday evening, two days before the race, we froze the projection. The file was saved with a fingerprint and has not been touched since. Everything on the results page is compared against that frozen copy, not against anything we could have adjusted after the fact. We compared Maine teams and Maine runners only; out-of-state schools raced, but we had no way to rate them fairly.

What we got right

The front of the race went the way the numbers said it would. The leading boys and girls ran very close to their projected times, typically within ten to twenty seconds, and eight of our ten projected top boys finished in the top ten. In the girls race all ten of our projected top ten finished in the top ten, and we had the winner. That is the part of the projection a coach cares about most when setting goals for front runners, and it held.

The best teams were where we expected them. Yarmouth and Portland finished first and second in the boys team race, exactly as projected. In the girls race the same three teams we put on top, Falmouth, Portland and Hampden Academy, were the top three. Across both races, three of our five projected leading teams finished in the top five.

The middle of our time estimates was honest. We said beforehand to expect a typical miss of about half a minute per runner. The boys came in better than that and the girls right on it, and neither race leaned much faster or slower than we said. The likely-place range we showed for each runner, the band where most good and bad days should land, held for more than eight runners in ten once you count only the runners who actually started, which is what it was built to do.

What we got wrong

We missed the girls winner. We had Falmouth first by a comfortable margin; Hampden Academy won it, and won it clearly. Our numbers had Hampden close but third. Mt. Desert Island and Cape Elizabeth also ran well above where we had them and broke into the top five, while Presque Isle, our fifth, finished tenth. On the boys side Thornton Academy, Scarborough and Gorham all moved up, and Houlton/GHCA and Winthrop slipped.

The back half of the field was much harder to call than the front. Runners we projected far back ran faster than we said, while runners in the middle of the pack ran a little slower. That pattern, front slightly slow and back clearly fast, is the signature of a projection that still places people too far apart, and it is exactly what our calibration is meant to correct.

The biggest single source of error was runners we barely knew. A runner with four or more races in our data was usually projected well. A runner with only one race, or with nothing but a seed time from the entry list, was often off by a minute or more. That is no surprise, but it was bigger than we had allowed for.

Freshmen boys beat their projections. As a group they ran noticeably faster than we expected, which says first-year runners are still improving faster at this point in the season than our fitness curve assumes.

Whole teams have days. When we look team by team, some squads ran well ahead of their projections almost to a runner, and others ran behind almost to a runner. Lewiston, Nokomis, Mt. Ararat and Oxford Hills had strong days; a few others had tough ones. Our team ranges treat each runner's day as independent, and the Festival says they are not: a team tends to rise or fall together, on a bus, on a cold start line, in a shared warm-up.

And a lot of runners we projected never toed the market price. Roughly one in five of the boys we projected and one in four of the girls did not start. Some were on the entry list from earlier in the week and scratched; others were roster runners who were never entered. A handful of schools did not bring a varsity team at all. Every one of those no-shows shuffles the team order a little, which is why the results page has an "Only runners who started" switch: it puts both sides on the same runners and shows how much of each miss came from the race and how much came from who was there.

A correction of our own

On race night we first reported that the likely-place ranges were far too narrow. That was our mistake, not the projection's. The grading tool checked the ranges with an older, much tighter setting than the one the Festival page actually used. Graded against what members really saw, the ranges did their job. We have fixed the grading and corrected the numbers.

What the Festival gives us for the regionals

More data, and the right kind. Every runner who raced in Belfast now has one more race in the model, and for runners who had only one or two races before, that is a big improvement. The Festival also adds a large, single-day result where nearly every good team in Maine raced the same course in the same conditions, which ties the courses of the whole state more tightly together than any regular-season meet can.

It also tells us about Belfast itself. In recent seasons the Northern regionals have been run on the same Troy Howard course, so if that holds this year, Northern runners now have a race on the regional course three weeks before the regional, and we have a fresh check of how that course plays.

We have already re-measured our calibration with this race included. The adjustment for Festival-style meets moved slightly, and the next projections use it.