FiveThirtyEight's big forecast errors
Accountability or something
The past two election cycles have been bad for FiveThirtyEight. I don't mean “they made bad calls” or “missed what should have been easier forecasts” (though they did, and I call them on specifically where and why here)
But there's a more glaring problem in their forecasts.
That is, they miss a lot of races, by a lot, in the same direction.
While their definition of poll error is statistically invalid (you can't compare election margin to poll margin and call the difference an error apples to apples, because polls and elections do not measure the same thing)
It's perfectly reasonable to compare their projected election margin to the actual election margin.
So I'll call that difference a forecast error.
My analysis includes the closest statewide races of 2022 (Gov and Sen, 10 and 12 of them, respectively) and 2020 (Pres and Sen, 17 and 16 total, respectively).
Why only include the closest races? Easy: those are the races with the most polling, and most at stake electorally. Close (and close-ish) races are why forecasts exist in the first place.
(You can read a little more on high-data vs low-data modeling in my Twitter post here if you'd like)
Forecast Error 2020 & 2022
On average, FiveThirtyEight missed the margin in these Battleground States by 4 points. That's not good. 4 points on the margin is the difference between a race easily called on Election Night and one too close to call for days.
That's also the difference between clearly having the wrong favorite in important swing states, like they did in PA in 2022.
On the note of wrong favorites, they shared this chart that supports their win probability forecasts. It does not do that.
What this shows is that in the closest races (races in which they give one candidate a 55-60% chance to win) that candidate loses more than 50% of the time. Given the importance of these races and the large sample size…that's not good
But it's not just that they miss that's of concern (or interest, depending on whether you're doing research or reading).
It's that they miss in the same direction, often by a lot.
In 2022, they had a heavy bias towards Republicans, and a 3 point (2.99 point) forecast error, on average. That average isn't terrible! But again, the average isn't the whole story.
Of the 22 “Battleground” State elections in 2022, 20 of their misses were to the right, 90.9%. The only ones they missed to the left were NYGov (D+9.6 forecast vs D+5.8 result) and OHSen (R+6.4 forecast vs R+6.6 result).
They were within 1 point (what I'd consider elite) in 6 forecasts, including the aforementioned OH Senate.
They were off by more than 1 point but less than 2 points (very good) in 3 more.
That puts them at 9/22 (40.9%) “great” forecasts, in terms of forecast error.
The next tier, off by 2-3, contains 3.
The next tier, off by 3-4, contains 3 as well.
6/22 (27.3%) of their forecasts were “okay” (off by 2-4) in terms of forecast error.
The bottom tier, off by 4 or more, is where the value, predictiveness, and methodology of their forecasts is drawn into question.
These 4+ point forecast errors comprise 7/22 (31.8%) of their forecasts for 2022.
They had 5.8 and 5.7 misses in each of PA and NH both of which I called them on not so subtly, and well in advance, and in many different places.
It'd be one thing if Nate Silver simply missed something that he should have seen. What's problematic to me as an analyst is that he saw the exact thing I pointed out would skew their forecasts in these states and didn't care.
Now, this is not the only flaw in 538’s forecast (a separate topic for a separate post) but seeing and ignoring a partisan poll flood because “free market?” could easily be the difference between, say, a bad 6 point miss and a respectable 2 or 3 point miss.
Nate, polls are not markets. Polls are data. Signal vs noise was supposed to be your thing. Anyways
COSen and WAsen by 6.6 and 6.0 were their biggest misses of the cycle - along with MIGov (5.8), PAGov (4.8), MEGov (4.9).
And in 2020
You'll notice I never said 538 had a partisan bias, and for good reason. It's telling that Nate seems to be perceived as biased by both the left and right depending on the year.
He's not, but his model sure is.
I noted that 20/22 of their forecast errors were in the same direction in 2022, overestimating Rs.
Well in 2020, 31/33 (93.9%) of their forecast errors missed in the same direction, this time overestimating Ds.
That is to say, over the past two election cycles, 51/55 (92.7%) of their forecasts have been biased in the same direction.
The direction of bias matters, but the amount of bias matters too. Their average forecast error in 2020 was a whopping 4.5 points; but again, the average doesn't tell the whole story.
Using the same (admittedly subjective, but fair) elite-good-okay-bad scale as above:
0-1 point forecast error: 2/33
1-2 point forecast error: 3/33
2-3 point forecast error: 6/33
3-4 point forecast error: 4/33
4-5 point forecast error: 1/33
5+ point forecast error: 17/33
This was so wild I had to double check my numbers.
They were off by FIVE OR MORE POINTS in MORE THAN HALF of the statewide battleground races in 2020.
When I say my forecast was better than theirs a lot of people take that to mean “oh you think you're a know it all genius, huh?”
No. My forecast was “off” in 2020 also. Just…not by as much. Which, as you've seen, isn't saying a lot.
2022 on the other hand, mine was much better because I was much closer on the states they missed by the most.
What does this mean for 2024 and beyond?
Well, I'll be using this article to establish what I'll be referring to as a normal forecast error regarding 538’s forecasts.
If anyone wants to go back farther in time (2018, 2016, 2014, 2012) to do the same research I'll be grateful. I'll likely get to it some point but I have something else I'm working on that you all will like a lot more.
That poll data series is a really fun project that I'll be posting on twice/week starting later this week, that you'll want to be a part of - please subscribe!
Carl…seems OK to score election models, which I imagine in this instance Nate would welcome assuming all assumptions correct - but why you think your model is more accurate would be just as worthwhile. Also noting weaknesses in analysis - e.g. small sample size of election cycles for yours. My gut take, which may be totally off given difficulty in understanding context, is this seems personal to you. If rightly or wrongly I am feeling this, does that feeling take away from what you are trying to point out?
Interesting post. Which way do you think Nate's forecast is biased this year? Is his more pro R than your forecast?