Prediction is a messy business. Those of us who cover horse racing have known this for a long time. I know I’ve told the story about the old guy at the racetrack who gave me some advice I’ve never forgotten, “I don’t need to know what happened yesterday. Tell me what will happen tomorrow.” That’s a problem for American news reporting in a nutshell. Predictions are highly unreliable. But the market for predictive news is huge. Look at NFL Draft reporting. Every year, the mock drafts are wrong. But we keep going back to the same experts. They should put a disclaimer on the screen every time one of those guys talks.
It’s one reason why I found all the “simulated” NCAA Tournaments so laughable. These predictors can’t tell us what will happen when the games are being played, but we’re going to give them any notice when the games aren’t? No thanks.
Sometimes, however, you can’t ignore predictions. A bigger problem arises when the subject turns away from fun and games and toward the subject of American lives. It gets much more serious when government actions are based on predictions, even educated ones, and they predictably turn out to be off.
Early on, a projection from London’s Imperial College got everyone’s attention by producing the notion that as many as 2.2 million in North America could die of COVID-19. The tone was especially somber when President Donald Trump held a press conference to share the projection that as many as 100,000 Americans could succumb.
Now, as many American cities approach their peak, the nation just surpassed 40,000 deaths — generally believed to be an undercount, but still not close to original projections — and one model the president’s team had used to project 100,000 deaths now is projecting around 60,000 — still an appalling loss, but a long way from what was originally speculated.
What happened? Several things. But before getting on with this discussion, I need to say two things. One, some of these projections could turn out to be right. We’re not though this yet, and the pandemic continues. And second, every statistic here, every death mentioned, is the loss of a life, and this outbreak, no matter what the final numbers turn out to be, has been a large-scale tragedy for this nation and world.
Now, the models. First, the projections that got the most attention, out of the University of Washington’s Institute for Health Metrics and Evaluation, were not true epidemiological models. Those in the industry would tell you that they are more statistical in nature, that they relied on the behavior of the virus in other countries, then plotted the U.S. over top of those numbers to come up with a range of expectations. The problem with the model, many would say, is that it varies so widely from week to week.
So why is it the one Americans have focused on? My guess – because it is the most easily accessible. It has the most user-friendly website. It has nothing to do with accuracy or underlying data, it’s just a good presentation.
“That the IHME model keeps changing is evidence of its lack of reliability as a predictive tool,” epidemiologist Ruth Etzioni of the Fred Hutchinson Cancer Center, who has served on a search committee for IHME, told the online publication STAT. “That it is being used for policy decisions and its results interpreted wrongly is a travesty unfolding before our eyes.”
In any prediction — educated or not — there are plenty of x-factors, things you just can’t predict. In sports, these are injuries, or the simple human factor of a key player having a bad game or a little-known player coming out of nowhere to have a great one. And there are many others.
In coronavirus projections, there were (at least) two major unknowns — the virus, and the behavior of people in response to it. For all we know about COVID-19, we still don’t know plenty. Even those involved in the Imperial College models — which use a more traditional method of measuring susceptibility, exposure, infection and recovery to put together a range of expectations — have said from the outset that their understanding of the virus led to an incomplete picture. For instance, the role of children in spreading the virus is relatively unknown.
And there’s one other factor involved that has nothing to do with the projections themselves but everything to do with us — people in the media and just people at home. We immediately go to the worst-case scenario. The media often does that because it sells. The public often does it because it’s human nature.
Even in the IHME model, while giving a single number it highlights as its midlevel projection, gives in much smaller print a range of outcomes, and it’s that range for which the model is primarily responsible. But in the U.S. today, we don’t want ranges. We don’t want a lot of qualifiers. We want to look at one number.
Regardless, a field hospital of empty beds that was dismantled in Seattle — after never seeing a patient — is testament to something: Models that were way off, or mitigation that was wildly successful. As usually is the case, the truth likely is somewhere in the middle, and both statements can be true.
The fact of the matter is, we still don’t entirely know how this virus will play out. We don’t know what kind of immunity those who have recovered from it will really have. We don’t know what kind of long-term health problems it causes. We don’t know how it will react this summer, or how strongly it will return in the fall.
Here’s one thing I do know — people are not going to stay cooped up a whole lot longer, especially those who don’t fall into the at-risk category. There are 22 million unemployed who have waited about as long as they feel like they can wait. And the instability in the models they have seen don’t inspire trust in what government officials or doctors will tell them moving forward.
Human beings are impossible to predict. Human nature, not so much. It doesn’t really change. It has been the same for centuries.
On Sunday, New York Gov. Andrew Cuomo harkened back to the early projections, before turning to recent outcomes, which have been terrible in New York, but far less than projected.
“Remember the context here,” he said. “We have accomplished what no one thought could be accomplished. The top experts, the CDC, the President’s Coronavirus Task Force, they all had multiples of more infections projected and multiples of more deaths projected. They were talking about 2 million people hospitalized in this country just a month ago. 2 million people. You know how many hospital beds we have in this country? 1 million. Their low estimate was for double the hospital capacity of the nation on like March 13. That’s what the top experts were predicting. Here in New York State, McKinsey, Cornell, Columbia, the Gates funded operation, they all had multiples of what we did.
“This is a great success story on everyone’s behalf. You know, if people weren’t so angry and frustrated right now, this has really been a great triumph. What the federal government did, what the states did, what the healthcare workers did, the way Americans responded and acted responsibly, it is better than any of the predictors. The President is right when he gets up there and says the models had many more people dying. President’s right. Or the experts said that. So this was heroic efforts that changed that curve. God bless America, but don’t under appreciate what you just did and don’t go backwards. That would be a real mistake.”
The problem is that fear often fades into frustration, especially when people are missing a paycheck. And human nature dictates that many will turn and blame the models for their predicament rather than seeing the actions taken as having mitigated what could have been a worse disaster.
In any event, the job of using data to justify restrictions for a significant time longer is a lot more difficult than it was a month ago. That turn of events was easy to predict, and Dr. Anthony Fauci, director of the National Institute for Allergy and Infectious Disease, predicted it last month when he said that if the restrictive measures work, people will blame him for overstating the threat. It was a prediction he said he hoped would come to pass. It appears that it has.
But for those people sure that the threat to them is passing and that it’s time to resume some degree of normalcy, a bit of caution — your predictions are as likely to go south as everyone else’s.
Even as the wheels of re-opening business and society finally begin to turn, let’s continue to be careful, and realize that none of us knows exactly what the future holds.