It cannot have escaped the attention of IMA members that the use of mathematical models, and associated algorithms, has had a significant amount of coverage in the press recently. Much of this is for good reasons, for example the use of models has played an important role in fighting COVID-19. However, there have been some serious errors in their use, for example the notorious use of an algorithm to determine the A-level results in 2020.
Models and algorithms have a long, and successful, track record when applied to problems in physics, engineering and biology. They are now being used to help to guide public policy. This has to be done very carefully, and naturally leads to controversy. It is essential that any such model is carefully constructed, used appropriately, with the assumptions and limitations discussed, and that the conclusions from using it are communicated as clearly as possible.
In recognition of the importance of this topic the IMA has set up a ‘Modelling and Algorithms Working Party’. The IMA partnered with the Parliamentary & Scientific Committee (P&SC) to organise their September discussion on the subject of mathematical models and algorithms, as part of the P&SC regular series of meetings. The P&SC is made up of a number of Westminster MPs concerned about science, together with universities, scientific and technical organisations, industrial companies and organisations representing those affected by science – including the IMA. It is chaired by Stephen Metcalfe MP, with Stephen Benn, Viscount Stansgate, as a vice-president. Recent topics for debate have been climate change and the scientific brain drain.
Led by David Youdan FIMA and Martine Barons CMath FIMA, the IMA assembled a crack team of speakers comprising our president Nira Chamberlain; Kit Yates from the University of Bath and the Independent Scientific Advisory Group for Emergencies; Ellen Brooks-Pollock, from Bristol, Scientific Pandemic Influenza Group on Modelling (SPI-M), and also a Mathematics Today Editorial Board Member; and Hannah Fry from University College London who has a terrific reputation for her books, and TV programmes, about the use of algorithms.
The webinar meeting was introduced, and chaired by Stephen Metcalfe, apart from a short period in the middle when he had to leave to vote. The peril of being an MP! Each speaker gave a ten minute presentation which was followed by a series of questions from the online audience, which was well over 100 in number.
Algorithms in industry
Nira Chamberlain CMath CSci FIMA started the discussion. He described how he has worked for over thirty years as a mathematical modeller in industry. He has seen the power of mathematical models to transform the understanding of a problem and then to lead to its solution.
As an example he described his own work on the systems of the new aircraft carrier HMS Queen Elizabeth where he had to convince sceptical military, government and industrial stakeholders of the best way to design the ship. He also talked about the Kursk submarine disaster where many sailors died following an underwater explosion leading to them getting trapped. If the results of a mathematical model on escape from a submarine had been applied it is possible that many of these sailors might have been saved.
Nira then spoke generally, saying that when a mathematical model was good it could be used to understand, predict and influence the design of a system. However, it could also lead to bad results if used for a problem it was not designed for. He emphasised the importance of explaining the basis of a mathematical model to people and that explanations have to include the limitations and assumptions made in the model.
Quoting Richard Feynman he said that we could understand the world via mathematics, but that it was important that the public understands the difference between a good and a bad model.
Communicating infection statistics
Kit Yates FIMA described the use of mathematical modelling and evidence based policy decisions, in the context of COVID-19 and data for the number of hospital admissions, deaths, and vaccinations. He said that whilst many people claim that they don’t believe in a mathematical model, without the use of mathematical models it is simply not possible to interpret data.
To illustrate this he considered different ways to measure the seriousness of COVID-19. A common measure is the case to fatality ratio (CFR), which is calculated as the number of deaths divided by the number of cases. As an example on one day in Germany there were 93 deaths and 23 000 cases leading to an incorrectly calculated CFR of 0.5%. In fact, as the number of people infected is often underestimated the CFR was divided by two to find an infection fatality ratio (IFR) of 0.25%, but even this revised figure is a mistake. This appears to be a relatively small number implying that COVID-19 might not be that dangerous.
However, there are a number of incorrect assumptions in the use of this measure. Firstly, it takes about three weeks for a COVID-19 infection to lead to a possible fatality. Thus the IFR should really look at the number of deaths three weeks after the number of cases. In the Germany example this figure was 3022, giving a corrected CFR of 13%. This looks much more serious, however, the estimate for 1/2 the infections being spotted is actually way too high. In fact only about 1/20 of the infections were detected at early stages of the pandemic. Under this assumption the IFR becomes 0.65%. This all shows that how we interpret metrics which characterise the severity of a disease depends crucially on the assumptions made in the model.
Another problem with the use of models, which can reduce trust in them, is that the predictions of a model can themselves change the situation it is modelling. For example, the models used at the start of the pandemic projected that, if left to progress without any intervention, there would be a very large number of deaths (around 500 000) in the UK. This projection led, eventually, to the UK’s first lockdown, and so that number of deaths has not materialised. Thus the initial prediction of the model never occurred. This led to a lot of criticism in the press. Of course, this does not necessarily mean that the model was wrong. In fact, given that we have had at least 150 000 deaths even with lockdown, the model prediction was very much in the right ball park, and certainly of the right order of magnitude.
There are always problems with using models, and (as Nira had also said) it is essential to communicate the assumptions made as clearly as possible so that the nuances in their use can be fully appreciated. This is particularly important given the generally low level of numeracy amongst many politicians and journalists. Having said that, modelling is still the best way to describe the world around us and to inform evidence based policy. It is not perfect, but it is honest.
Calculating R for policy makers
Ellen Brooks-Pollock OBE FIMA has been at the heart of using modelling to advise government policy and has served throughout the pandemic on SPI-M. This is a group of about 50 modellers taken from many different groups, which provides evidence based advice to the Scientific Advisory Group for Emergencies (SAGE). Her presentation described the use of mathematical modelling for policy advice during the COVID-19 pandemic. One of the main tasks of SPI-M is to provide both short and medium term estimates for the infection rate R. These estimates are obtained by making a comparison of the predictions from a number of independent models.
An important aspect of calculating R are the types of social contact that a person has, and this can be estimated by using a close social contact model. The number and type of contacts that a person has depends significantly on the sort of occupation/lifestyle that they have and also what actions are prohibited due to lock down. Ellen compared the nature and type of social contacts of a school pupil and of a flight attendant. One of the predictions of the model that she constructed was the level of R as a function of active work and leisure contacts. This allowed R to be compared under the cases of keeping the schools open and keeping them closed. For the models she worked on the predicted value of R changed from 1.5 if the schools are open to 0.7 if the schools are closed. This is a very large difference with clear policy implications.
In fact the model was refined enough to look at the differences between opening primary and secondary schools, and also the implications of using contact tracing. This was an important example of the use of the modelling approach which was able to determine the right time for a relaxation of the strict measures on schools and the use of support bubbles. This work is reported in the 2021 Philosophical Transactions of the Royal Society B themed issue on Modelling that shaped the early COVID-19 pandemic response in the UK, edited by Ellen Brooks-Pollock, Leon Danon, Thibaut Jombart and Lorenzo Pellis. This issue introduces ‘the UK scientific advisory system and how it operates in practice, and discuss how infectious disease modelling can be useful in policy making’.
Modelling cancer and A-levels
Hannah Fry looked at the use of modelling and algorithms in a broader context. She started by considering medical screening for breast cancer. Machine learning based algorithms have been used with great success in doing this. However they can display a bias due to the way that they may be trained on people of only European descent. Another problem with using screening is that whether someone has cancer is not a binary issue, and there is a spectrum of levels of the amount of cancerous tissue in a body.
The disadvantage of using a sensitive test for breast cancer is that it has a higher risk of false positives, and that such a false result can be life-changing for the patient. This leads to what is called a ‘Kahnemanian swap’ between the easy decision of which patient has abnormal cancer cells to the much harder decision of whose life should be saved. Going from one conclusion to another is difficult and can cause problems if the model behind the data is not well understood.
A similar issue arose in the use of algorithms in 2020 to determine the A-level results in which the algorithm was constructed to make the ‘easy’ decision of keeping the results in 2020 consistent with those in previous years, rather than the much ‘harder’ decision of producing awards of results which were fair to the candidates.
Hannah concluded by warning against over trusting an algorithm and gave examples of where this could be a problem, from people following a robot, to the use of a grossly oversimplified algorithm to determine the budget in Idaho. As with all of the other speakers she agreed that mathematical tools are ‘the most right’ for making decisions but that we should all be aware of their limitations.
Q&A
At the conclusion of the presentations there was then a period of excellent questioning of the panellists. This was lively and good spirited. Some of the questions are paraphrased below.
What do the panellists disagree about?
Stephen Metcalfe MP asked this first question and Kit Yates said that there were often differences of opinion about the ethical issues in mathematics, and how you communicate the results of a model. Hannah Fry said that it was our duty to communicate clearly especially as ‘the world fears mathematicians’ and that mathematicians should also consider that they were communicating to non experts. Ellen Brooks-Pollock said that there were often issues about the balance between the use of complex and simple models, and Nira Chamberlain said that there were usually disagreements about the assumptions made in constructing a model.
Can a model be precise but totally wrong?
In fact this is a big problem with modern machine learning where models with many parameters can be overfitted to a particular set of data but tell very little about a general system. A historical example was the use of epicycles in the Ptolemaic model of the solar system. These gave good predictions but were a totally wrong description of the physical system.
How can we work with politicians to increase public confidence in using models?
Stephen Metcalfe made the general point that school students are often unaware of the connection between the maths that they learn at school and the game changing role that it plays later on.
Nira Chamberlain said that this was exactly the remit and ambition of the IMA. Hannah Fry argued that if more school subjects such as history and geography made more use of data then school students would see how important mathematics was in the real world. Kit Yates and Ellen Brooks-Pollock both agreed that MPs had a role to play by leading by example in the use of data to make decisions.
Should we have stuck to the original COVID-19 plan?
Kit Yates said that the original models were based on influenza modelling and that they had to be rapidly changed as data became available about the special nature of the COVID-19 virus. Many lives would had been saved if we had gone into lock down earlier both at the start of the pandemic and also at the start of the second wave.
Model assumptions and close
The last point was raised by Martine Barons (chair of the IMA Research Committee) who said that it would be good if ten pieces of information were given about any model used. Ellen Brooks-Pollock said that it was often challenging to list all of the assumptions in a model, and that the use of open code was needed to make models transparent.
Nira Chamberlain concluded the evening by stating simply that mathematics is the greatest subject in the world.
A statement with which I heartily agree.
Thanks and webinar recording
I would like to thank all of the panellists and organisers of the event and Stephen Metcalfe MP for his excellent chairing. A recording of the webinar is available. I hope that we will have similar meetings on other mathematically related topics in the future. IMA members are most welcome to make suggestions.
Chris Budd OBE CMath FIMA
University of Bath
Reproduced from Mathematics Today, December 2021
Download the article, P&SC Mathematical Modelling and Algorithms Webinar (pdf)
