Kevin Lewis pointed me to this comment from Nate Silver on a recent post:
Having a few elections under your belt helps a *lot*. No matter how much you test things in the lab, there are some things you’re going to learn only by seeing how your forecast reacts to real data in real time. (I’m sure this applies to lots of other stuff too.)
It’s an interesting thought. Nate and I both have experience with election forecasting: he’s been doing it since 2008 and I’ve been doing it since 1992, on and off. And, as Nate wrote, our forecasts are pretty similar, so I guess I can take his comment as being very positive!, in that he’s putting us (the Economist) and them (Fivethirtyeight) in the same category, as the product of experienced forecasters who have learned by seeing how our forecasts react to real data in real time and have ended up with similar results.
I do agree that our forecasts are similar, especially at the national level. We have some differences in how we handle polls, which is why we’re forecasting Biden at 54.2% of the total vote and their forecast is 54.0%—no way to tell these apart! A few months ago, our forecasts differed by more, but that’s because our fundamentals-based predictions were different, and the fundamentals-based prediction becomes less important as election day approaches. We’ve discussed some interesting differences between the two forecasts, but these don’t have much of an impact on the headline numbers. For example, Fivethirtyeight gives Biden a 6% chance of winning South Dakota and we don’t, but . . . that’s only 6%, and these are elections that Biden would win anyway. Their predictive interval for Vermont is much wider than ours, but, again, we all know who’s gonna win Vermont anyway. These internals can be helpful in understanding how a model works, so they’re worth studying, but no matter how you slice it, the data are gonna say, “Biden’s the clear favorite but there’s an outside chance he won’t win enough swing states to pull it off.”
In his tweet, Nate also brings up the stability of his forecast: “it’s been pretty smooth, in contrast to our bouncier 2016 model (though of course the polls were way bouncer in ’16).” This indicates an example of the value of experience. Nate and I both have “a few elections under our belts,” so we know not to take polling bounces seriously, as they can come from differential nonresponse. This is a point that Mark Palko made back in 2012!, which my colleagues and I rediscovered in our Xbox study, which was also apparent in 2016 (see this graph from Alan Abramowitz), and which we formally incorporated into our model in 2020 by allowing a time-varying national polling bias term for polls that don’t adjust for partisanship. So, yeah, our experiences and insight from 2012 and 2016 have helped us get more sensible forecasts and not overreact to polling bounces in 2020.
There’s one thing, though. Much as I enjoy the respect that Nate shows to me from my years of experience, and much as I respect his years of experience in politics and sports analytics, couldn’t someone without all those elections under their belt do as well as we can? After all, Nate and I have written a ton about what we do: I’ve published a few books and many articles about political data analysis, including a few directly on the analysis of pre-election polls, and Nate’s been blogging and sharing his analyses for over a decade.
So couldn’t some newcomer with an open mind and an empty belt just read all our stuff, play with our code, and do better than the Economist or Fivethirtyeight?
I think they could. I think the right newcomers could do better, because they’d have the benefit of all our experience, plus their own unique perspectives.
Indeed, this has happened before. Back in 2010, old-school pollster wrote an insulting open letter to Nate Silver, basically calling Nate a young whippersnapper and telling him to grow up. I thought Zogby was wrong because he didn’t recognize the value of the division of labor. Zogby’s a pollster, Nate’s an analyst: These are two different, complementary, roles.
So, again, I have mixed feelings. I do think my work is enriched by my experience of studying all those elections in real time, and that I can avoid mistakes that I might have made without that experience, and I think this holds for Nate as well. But I’d hesitate to say that this experience is necessary or even desirable for all people. I was a young person once, and I recall figuring things out that oldsters were stuck on: sometimes lack of experience can help too.
I think young researchers should be able to do better than Nate or me, by making use of our experiences through our writings but without being stuck with our preconceptions, whatever they are.
P.S. Nate describes my post as “from the Economist team” but it’s actually by me! I can’t complain, though. I like being part of a team.
from Statistical Modeling, Causal Inference, and Social Science https://ift.tt/2TPNu8J
via IFTTT
Comments
Post a Comment