Thanks for this important work distinguishing polls and predictions.
It makes me think of Daniel Kahnemann's work on human psychology.
We are biologically programmed to make mental shortcuts.
We substitute a harder question like who will win the election in 6 weeks, with an easier one like "who is ahead in the polls now". But we don't realize we have done the mental substitution.
So, great work pointing out this example of unconscious mental substitution in a very important and timely topic: this US election and polls.
Exit polls are still polls - which are also estimates of a present state. But because the population of interest is "actual voters" (with no undecideds) an exit poll should be pretty close to the result.
In the book, I differentiate between present polls and plan polls for this reason, and others.
In a present poll, you can take the data "as is" as the best estimate of the population
In a plan poll, you have to incorporate some assumptions.
Is there an argument for saying the simplest forecast is a poll with no adjustments for undecideds, sampling bias, time till election etc. And then this simple forecast is the benchmark against which complex forecasts compete against. Similar to how a linear model with a single feature can be used as minimum performance to beat when looking at more complex models like Multiple Linear Regression, tree based methods, Neural Networks etc?
There's no such thing as a forecast with no adjustments for undecideds though, because if the poll has them (almost all do) you have to assume something about them
I see where you're coming from, and I'm not even opposed to using that as a benchmark of sorts, but right now the field calls anything other than 50-50 undecided a "poll error" when in many cases that's a very poor assumption
Then you get into the third-party problem.
When a poll says 42-41 with 10% third-party, and the result is 50-49 with 1% third-party, was that poll right or not?
Currently, they say "yep, margin margin margin we subtracted the numbers"
Yea ok I see your point about requiring some adjustment for undecideds. I guess the naive 50-50 split would work for this simple “forecast”.
And yea, regarding poll error, I’ve been convinced by your explanations years ago, the way the media, Nate Silver, etc refer to it doesn’t make sense. And yea I can see how this is related to distinguishing between a poll and a forecast. I guess being more specific about that would lead to an easier discussion of poll error (as in sampling error from population) vs forecast error (how undecideds voted vs the prediction) as people are able to differentiate between a poll and a forecast
It's refreshing to me that people understand it and I'm not crazy.
If my forecast assumption is that undecideds split 50-50, but they actually split 60-40, that's not always negligible.
At just 10% undecided, splitting 6 and 4, to most people that's "only" 2 points, who cares?
Well, if some pollster actually does have, say, a two point error, it would now be said they have a four point error.
The poor assumption has contributed as much error as the poll itself had.
And on the other end, a poll that had an error in the other direction, such that the 6-4 split of undecideds "corrected" their error, they could be said to have 0 error
There are so many problems, and so much to be done.
But you are crazy, what was your forecast of a biden win % when he dropped out? What was that based on?what does it mean to be a undecided likely voter?
Thanks for this important work distinguishing polls and predictions.
It makes me think of Daniel Kahnemann's work on human psychology.
We are biologically programmed to make mental shortcuts.
We substitute a harder question like who will win the election in 6 weeks, with an easier one like "who is ahead in the polls now". But we don't realize we have done the mental substitution.
So, great work pointing out this example of unconscious mental substitution in a very important and timely topic: this US election and polls.
The election result is a deceptive placeholder for the simultaneous census.
I have an example at the end of the book that illustrates the problem with the "directionally correct" obsession held by the field's current analysts
EXTEMELY helpful sir. Thank you.
This is perhaps the most self inflated, plagiarized and stolen content, mixed with bad writing and inaccurate conclusions I've ever seen.
Your level of being confidently and arrogantly wrong is staggering.
I hope you stop sharing whatever small minded thoughts you have publicly as it might help make the world a little smarter.
You wrote three sentences and there's nothing of substance in any of them.
Could you cite anything specific or are you just here to cry?
What about “exit polls” then? Are they not forecasts as well?
Not exactly.
Exit polls are still polls - which are also estimates of a present state. But because the population of interest is "actual voters" (with no undecideds) an exit poll should be pretty close to the result.
In the book, I differentiate between present polls and plan polls for this reason, and others.
In a present poll, you can take the data "as is" as the best estimate of the population
In a plan poll, you have to incorporate some assumptions.
Really useful info!
Is there an argument for saying the simplest forecast is a poll with no adjustments for undecideds, sampling bias, time till election etc. And then this simple forecast is the benchmark against which complex forecasts compete against. Similar to how a linear model with a single feature can be used as minimum performance to beat when looking at more complex models like Multiple Linear Regression, tree based methods, Neural Networks etc?
There's no such thing as a forecast with no adjustments for undecideds though, because if the poll has them (almost all do) you have to assume something about them
I see where you're coming from, and I'm not even opposed to using that as a benchmark of sorts, but right now the field calls anything other than 50-50 undecided a "poll error" when in many cases that's a very poor assumption
Then you get into the third-party problem.
When a poll says 42-41 with 10% third-party, and the result is 50-49 with 1% third-party, was that poll right or not?
Currently, they say "yep, margin margin margin we subtracted the numbers"
I disagree.
Yea ok I see your point about requiring some adjustment for undecideds. I guess the naive 50-50 split would work for this simple “forecast”.
And yea, regarding poll error, I’ve been convinced by your explanations years ago, the way the media, Nate Silver, etc refer to it doesn’t make sense. And yea I can see how this is related to distinguishing between a poll and a forecast. I guess being more specific about that would lead to an easier discussion of poll error (as in sampling error from population) vs forecast error (how undecideds voted vs the prediction) as people are able to differentiate between a poll and a forecast
Bingo.
It's refreshing to me that people understand it and I'm not crazy.
If my forecast assumption is that undecideds split 50-50, but they actually split 60-40, that's not always negligible.
At just 10% undecided, splitting 6 and 4, to most people that's "only" 2 points, who cares?
Well, if some pollster actually does have, say, a two point error, it would now be said they have a four point error.
The poor assumption has contributed as much error as the poll itself had.
And on the other end, a poll that had an error in the other direction, such that the 6-4 split of undecideds "corrected" their error, they could be said to have 0 error
There are so many problems, and so much to be done.
But you are crazy, what was your forecast of a biden win % when he dropped out? What was that based on?what does it mean to be a undecided likely voter?