‘Science by press release’ may be good for value of developers’ stocks, but regulatory approvals & public-health use require rigorously gathered, honestly reported data
The recent rush of claims on Covid-19 vaccine efficacy, in four interim trial results announced by their makers, resembled competitive bidding at an auction. In three of them, frequently revised numbers moved up from 90% to 95% in a spree of media announcements a few days. The Pfizer-BioNTech, Moderna and Sputnik-V vaccines have not yet published their data in peer-reviewed scientific journals nor have the regulators scrutinised the data so far.
With greater modesty, the AstraZeneca vaccine came forth a bit later to claim a 70% efficacy rate. However, it chose to add mystique to modesty by reporting that two different levels of efficacy had emerged from the study. In one group, where the vaccine was given in two equally high doses, the observed efficacy was 62%. In another group, where the first dose administered was at half the intended level, and the second one was a full dose, the observed efficacy was 90%. Combining both groups, the overall efficacy level was reported to be 70%. This was because the group which received both doses in full (8,895 participants) was numerically much larger than the modified dose group, which exhibited greater efficacy (2,741 participants).
When Covid-19 vaccine trials began, there was uncertainty about whether one or more of the candidates would succeed and, if so, at what level of efficacy. At that time, most regulators and WHO set a protection rate of 50% as the criterion for success. Recognising that any observed estimate is compatible with a variable range of the true values of efficacy, a lower bound of 30% probability was set for any single estimate of protection that was reported as 50% or above.
What this means is that any reported estimate from a single trial is just one observation which falls within a potential band of values, in which, the truth lies. That band extends around the observed value, which is called the point estimate. Larger the sample size, narrower the band around that estimate. We will never be able to say that a single value definitely represents the absolute truth, however, large a finite sample may be. A 95% confidence interval provides the upper and lower bounds of the spread of values where the point estimates would fall 95 times if the trial were repeated identically 100 times. The lower boundary represents (almost) the worst-case scenario, based on the observed estimate while the upper boundary represents the (almost) best-case scenario.
When the lower boundary remains at or above 30% in the 95% confidence interval, we are fairly assured that the true efficacy will not slip below 30%, whether the observed point estimate was 50% or 60%. For a new virus that was spreading fast across the globe, a modest success rate would be acceptable for adopting a vaccine. Anything more would be a very welcome bonus. To be declared as an effective vaccine, both criteria (point estimate of 50% or higher; lower limit of the confidence interval at 30% or higher) needed to be met.
So, a 90% or 95% value of efficacy observed in a large trial is happy news indeed. In a small trial, the confidence interval would be very wide and will not carry conviction, as the lower boundary of that band would slip below 30%, even if the reported point estimate is above 50%. We still await the publication of the large Phase 3 trials to calculate the 95% confidence limits of the observed point estimates of protection in each, but the press release promos so far suggest that the lower bound criterion would be well met. So far, there is no head to head comparison of the different vaccines in a single trial. The trials are also being conducted in different populations. Even if they were performed in very similar populations, the protocols would need to be similar. Only then can the 95% confidence intervals around the observed point estimates of the different trials be compared. If those confidence intervals overlap, we cannot conclude that they differ in efficacy even though the point estimates suggest that they do. Therefore, claims from trials do not convey much unless accompanied by a report of the 95% confidence interval around each reported point estimate.
The AstraZeneca trial of the Oxford vaccine has, however, thrown a surprise that goes beyond the size of the trial and width of the confidence interval. The interim trial report stated that the trial unintentionally split into two designs. The larger strand of the trial kept to the protocol as designed, administering two equally high doses of the vaccine to the active intervention group and inactive placebo injections to the control group. An inadvertent error, reportedly by ‘a contractor’, led to another strand of the same trial following a different dosage schedule. The initial dose in this strand was low (half the intended dose), while the second was the usual ‘high’ dose. When the results were examined, it appeared that the accidentally created ‘low-high’ vaccine strand had a more efficacious effect than the originally planned ‘high-high’ vaccine strand of the trial.
Serendipity spawning success? Not unknown to science. Alexander Fleming, known to be ‘a careless lab technician’ returned from a two-week vacation to find a mould growing on bacterial culture plates and noted, to his initial surprise and later delight, that a dreaded bacterium (staphylococcus) was destroyed by the penicillium mould. Later knighted, he went on to receive the Nobel Prize for the discovery of penicillin, the first antibiotic that transformed the treatment of deadly infections. “One sometimes finds what one is not looking for” was Fleming’s candid confession, one which the Oxford researchers might well cite as evidence of happy happenstance.
The different levels of protection observed in the two limbs of the trial involved an unplanned deviation from the original protocol and cannot be combined into a single point estimate of 70% efficacy. It appears also that the 95% confidence intervals of the two-point estimates (from the two strands) overlap. It is, therefore, difficult to conclude that one dosing pattern is more effective than the other. Faced with the criticism on several counts, AstraZeneca has now announced a new trial, which would be conducted only with the modified ‘low-high’ dosing schedule, even as the earlier trial continues. Whether the new trial confirms the earlier observation that the modified dosing schedule works better is to be seen. The new trial will also try to address another criticism about non-comparable age groups in the two strands of the original trial. The accidentally created ‘low-high’ dose group had only young persons, all below 55 years of age. The protective value may have appeared high for that reason. The new trial will include older persons.
Despite the valid criticism directed at the Oxford trial, there are several strengths of the study. The participants were lab tested for proof of infection and immune response, unlike other trials which relied on self-reported illness. The vaccine is less expensive, easier to store and transport, and can be produced at large scale on previously well-established platforms. The fact that the trial involves several countries is actually a plus point because it accommodates diversity and enhances generalisability of protective results to many populations.
The science behind the greater protective effect of the unplanned ‘low-high’ dosing schedule is speculative but intriguing. It is being proposed that the lower initial dose may efficiently prime the body’s immune system rather than overwhelming it. That may help the second (‘high’) dose to elicit a better immune response. Like a car picks up speed efficiently when we move steadily from low to high gear, rather than jerkily stall when we abruptly shift it into high gear at the very start. All of this is speculative. As Mark Twain wryly remarked, “There is something fascinating about science. One gets such wholesale returns of conjecture out of such a trifling investment of fact”.
Despite the competitive bombast and controversies, there are hopeful signs of several efficacious vaccines emerging soon. However, we need to see even interim results of the trials published in peer-reviewed scientific journals, trials completed and all data submitted to regulators. In all the hullabaloo of interim announcements of efficacy, the need to examine the safety of a vaccine and the likely duration of its protection must not be forgotten. The feasibility of large scale production and equitable distribution follow quickly as the other elements to evaluate. Affordability for public procurement becomes an important consideration after the trials are deemed successful.
‘Science by press release’ may be good for the stock value of companies, but regulatory approvals and adoption by public health practice require rigorously gathered, appropriately analysed and honestly reported data. That too is a ‘confidence interval’ between promise and proven performance, which needs to be narrowed by the vaccine candidates before they step in to save the world from further ravages of Covid-19.
The author is Cardiologist & epidemiologist, and president, PHFI. Author of Make Health in India: Reaching a Billion Plus. (Views are personal)