It is necessary to report the 95% confidence intervals around both the overall vaccine trial result and for the values being reported for any subgroup of interest
By K Srinath Reddy
Mirror, mirror on the wall! Who is the fairest one of all?”, asked Snow White’s narcissistic stepmother. Her confidence that she would be the chosen one was shattered when the mirror named her stepdaughter. Covid-19 vaccine manufacturers are also running a race to be regarded as the most effective vaccine produced. Their confidence is being tested by the mutants that have entered the extended family of SARS-CoV-2 viruses.
Even without the questions thrown up by the mutants, the claims made of highly specific levels of protection need scrutiny. When the media reports a number, say 90% or 95% efficacy in a trial, it appears that is exactly the level of protection we should expect in every population where the vaccine is administered. That is not true even of a population that is identical to the trial population, leave aside the differences across global populations in their susceptibility and immune response.
To explain why we need to understand the concept of confidence intervals.
Every statistical test of efficacy considers the possibility that the observed result could have arisen through a play of chance, rather than represent the absolute ‘truth’. Even if the repeat trial is conducted in the same population, a different value may arise due to chance alone. To minimise the play of chance, tests are designed to test a hypothesis through adjustments for the ‘alpha error’ (the probability that the observed result arose by chance alone). The test of hypothesis tells us whether the observed result is statistically significant or not (usually after allowing room of up to 5% for a false-positive result).
However, it is also important to gauge how well the observed result may approximate the true value which may be different from the one observed. The 95% confidence interval provides a band of values within which the values will fall 95 times if the study were repeated 100 times in identical conditions. The larger the sample size of the study, narrower the band between the lower and upper confidence limits around the observed value. So, we derive greater confidence that the observed value is not far from the truth even if it is not ‘the actual truth’.
The confidence interval also gives an idea of the clinical significance of the result, which goes beyond mere statistical significance. There is a natural tendency to conclude that a vaccine with a 90% efficacy is better than a vaccine with an 82% efficacy. It well maybe so, but it is also possible that the observed difference is not real and is seen only due to a play of chance. If the 95% confidence intervals around the observed point estimates overlap, we cannot conclude that one vaccine is superior to the other. If they do not overlap, we can conclude that there is indeed a very high probability of a true difference in efficacy.
So, it is necessary to report the 95% confidence intervals around both the overall trial result and for the values being reported for any subgroup of interest. Since subgroups will have a smaller sample size than the overall trial population, their confidence bands will be wider than the main trial result. To jump to conclusions on subgroup results is unwise if their sample sizes are small, and 95% the confidence intervals are wide.
Even as several vaccines were publishing their trial results, virus variants emerged due to mutations, making them more infectious. It is possible that the evolutionary pressure being exerted on the virus by the vaccine trials, as well as various public health measures ranging from masks to physical distancing, led to the emergence of more infectious mutants. Questions naturally arose as to whether the vaccines have the same level of efficacy against the mutants as the earlier strain that was targeted by the vaccine.
This information comes from laboratory studies which examine how well antibodies developed by vaccinated persons can neutralise the mutant strain or a pseudo-virus created to simulate it. Vaccine trials in countries where the mutants emerged, and are circulating along with the less infectious strain, also give insights into the efficacy of vaccines against such a mixed group.
It is worth examining the efficacy of vaccines, both through a comparison of the reported point estimates and the 95% estimates around them (see graphic). These values are from published trial reports. It is worth remembering that all of these estimates are related to the efficacy of the vaccine in reducing the risk of manifesting symptomatic Covid-19 and not the risk of viral infection per se or the risk of transmission by an immunised person to others.
For each of these vaccines, there is a 95% probability that the true efficacy value may lie anywhere within the reported confidence interval.
The 95% confidence intervals for all the three Astra-Zeneca groups overlap. So, it is not possible to affirm that there is indeed higher efficacy for the half dose-full dose combo. It is also obvious that this group suffers from a small sample size, leading to a wide confidence interval. Further, that group also had a problem of delay in the second dose up to three months in several cases. Efficacy estimates are clouded by these mixed groups and protocol deviations. The confidence interval of Novavax overlaps at one end with Astra-Zeneca and at the other with Pfizer-BioNTech and Moderna. Since we do not keep repeating trials, we tend to accept the reported point estimates as our best estimates, but it would be incorrect to interpret them as conclusive proof of the difference in efficacy if the confidence intervals overlap.
The most recently reported trial of the single-dose J&J vaccine has only stated the point estimate of efficacy, without the 95% confidence intervals, in its press release. Interestingly, it reports different rates of efficacy in different geographic locations—72% in the US, 66% in Latin America and 57% in South Africa. The efficacy is reported for the prevention of moderate or severe Covid-19 infection. It is possible that the 95% confidence intervals of these three estimates may overlap. However, the difference in the point estimates raises the possibility that the geographical variations could have arisen due to the earlier spread of the more infectious and vaccine-resistant mutants in South Africa and Brazil than in the US.
What then is the efficacy level that we should accept? When the vaccine trials were initiated in 2020, the regulators set a point estimate of 50%, with a large enough sample size to keep the lower boundary of the 95% confidence interval at or above 30%. That was a modest goal, but it was felt that even that level of efficacy would help to limit the danger from the pandemic and possibly stall it. The high level of efficacy reported from the two mRNA vaccines is really a big bonus, but that should not lead us to dismiss vaccines that meet the original standard as ineffective vaccines. Higher the point estimate and narrower the 95% confidence interval the better it is, but all those who meet the pre-set standard are acceptable for use in mass vaccination, especially if their 95% confidence intervals overlap with those of the leaders.
Author is a cardiologist and epidemiologist, is President, PHFI. He is the author of Make Health in India: Reaching a Billion Plus. Views are personal