Lies, damn lies and modelling? Why COVID forecasts haven’t all come to pass

For our free coronavirus pandemic coverage, learn more here.

Hundreds of thousands of deaths. ICUs in overwhelmed hospitals struggling to cope under extreme pressure on beds and ventilators. The numbers were splashed in headlines across Australia.

In March last year, federal government modelling said that in a worst-case scenario up to 150,000 Australians could die from COVID-19. Modelling also forecast nearly 3000 cases a day in Victoria during an October peak of the latest outbreak and hundreds of cases in ICU and hospital.

But thankfully neither of these scenarios – nor other grim, headline-grabbing forecasts – have come to pass.

Critics have said this is proof that the scientists and public health experts behind the modelling are wrong and just want to see the country locked down for good.

But modelling is not a crystal ball prediction of the future. The experts behind some of the country’s best-known modelling, from the Doherty Institute and Burnet Institute, have been saying that for anyone who cares to listen.

It’s a tough science communication gig, according to Dr Nick Scott, head of modelling and biostatistics at the Burnet.

“You can’t expect them to be correct for more than a week or even two weeks at most, in the same way that you could never expect a weather forecast to be accurate that far out,” he says.

Why does it change? Why does some modelling paint a disastrous picture? And why do different experts or institutions end up with wildly different results?

Firstly, modelling is based on data. And at the beginning of the pandemic, there wasn’t much of it.

“Your model will only be as good as the information you have,” says Professor Jodie McVernon, director of Doherty Epidemiology at the Doherty Institute.

New variants. Different public health measures. More information about how effective vaccines are. As all this changes, so too does the modelling. “The model projections are basically the best we can do with the available evidence at the time,” Scott says. “Over time, our picture of that data gets clearer and clearer.”

Even in the months since completing the major modelling used to underpin the country’s reopening plan, the data has vastly improved, McVernon says.

“At that point, Delta was a brand new thing, we didn’t know as much about it, we hadn’t had as much opportunity to see how well the vaccines worked. And so we made a whole set of assumptions based on what we knew back in July,” she says.

But after going back and adjusting some of their parameters for the latest November modelling, McVernon says when they were put in the model, they got essentially the same results as July.

So why didn’t Sydney, for instance, end up with thousands of hospitalisations and ICU admissions, and a peak of 2000 cases a day?

Jodie McVernon, Director of epidemiology at the Doherty Institute in Melbourne.Credit:Simon Schluter

McVernon says it’s important to note that if the modelling said there was going to be hundreds of new cases a day, that’s a scenario, not a prediction. “That was allowing you to compare one strategy with another strategy; a different coverage level, a different set of measures, to see whether you would expect one [strategy] to be better or worse,” she adds.

New data has also played a role, specifically around the effectiveness of vaccines. In a huge positive, they have been far more effective at preventing hospitalisation and severe illness than originally thought, with all three vaccines currently being used in Australia performing better in real-world settings than they did in the original trials.

“It’s actually quite a clear example of why projections will change over time when new data becomes available,” Scott says. “In updated models, we have the benefit of more data on vaccines. So that all goes in and it helps to inform the next set of decisions that need to be made.”

As the data changes, so too do the questions that governments want answered.

Prime Minister Scott Morrison in national cabinet in March 2020, at the start of the pandemic.Credit:Alex Ellinghausen

McVernon has regularly briefed national cabinet on the latest modelling. Early on, she said the country’s leaders needed broad answers.

“In very early scenarios, the biggest question we had was ‘how severe was this disease going to be?’,” she said. “And we had vast ranges of possibilities right back at the beginning.”

Models themselves can also vary – from back-of-the-envelope calculations to detailed and complex mathematical equations. Assumptions about things including how effective the vaccine will be can have a huge impact on the results.

“While most modellers will try to use the best available evidence, the way they incorporate that into the model might be slightly different,” McVernon says. “So we have seen some models that have had more pessimistic predictions about clinical cases and others that have been less so.”

Projected case numbers and hospitalisations are one thing. But what governments and policymakers really want to know from the modelling is what is the best decision they can make.

The original modelling looked at scenarios around how bad the pandemic could be for the country. The latest tranche was much more detailed. Rather than looking at a national scenario, there are granular investigations of outbreaks in school settings, Aboriginal and Torres Strait Islander communities, local government area-specific modelling and a look at the risk of COVID-19 from international arrivals.

“The more information we have, the more we can bring that into the model. And that will actually make it more likely to produce meaningful outcomes,” McVernon says.

Modelling is also not in itself a solution. It’s up to governments to decide what actions they should take and what risks they’re willing to bear.

“The reason that the actual model is useful is that a lot of these things are not linear,” McVernon says.

This has become increasingly obvious as states and territories reveal opening plans: NSW and Victoria are more willing to continue living with levels of COVID-19 in the community, while Western Australia wants at least 90 per cent of its population vaccinated before it considers fully reopening.

The strength of modelling is it pulls together all the pieces of information at hand – the effectiveness of vaccines, the effect of social distancing and working-from-home measures – puts them in one spot and shows what happens when governments do or don’t combine them.

“They just need continual updating. But they are helpful when you need to make a decision about what you’re going to do tomorrow or next week,” Scott concludes.

Fascinating answers to perplexing questions delivered to your inbox every week. Sign up to get our new Explainer newsletter here.

Most Viewed in Politics

From our partners

Source: Read Full Article