Over a month ago, there was some news about the first possible HIV vaccine. I remember reading that this was the first trial of an HIV vaccine that showed any success pretty much ever. But there weren’t any details at the time. Now, the full report has been published in the New England Journal of Medicine, and I find it unconvincing. The vaccine is a combination of two vaccines previously known not to work. 8198 people in Thailand were given saline injections, and over the time of the study, 74 got HIV. That is the control group. To compare, 8197 people were given the vaccine, and at the end of the same period, the number with HIV was… 51.
That’s it. Those are the raw data. Everything else after that is statistical analysis and interpretation. Are you impressed? I’m not. 51 is less than 74, but does it mean anything? The researchers did a statistical test called a p-test to see if the results were significant. They did the test three times, using different data in each case, and in only one of them did they get a p value that was less than 0.05 (it was 0.04.) Usually, a p-test has to be less than 0.05 to be considered significant. And what that means is that there was a 4% chance, even in their best case scenario, that the observed effect could be observed due to random chance (the other p-tests said this chance was even higher). But that is regardless of whatever that “observed effect” might be. In this case, the effect is merely a 40% maximum reduction in infection rates. And when observed over time, the benefit is only immediate, and then after the first year the rates track the same as the control group.
So, to review:
1. It provides an initial benefit, but after that the rates of infection are the same.
2. That benefit only applies to a minority of people receiving the vaccine.
3. It *barely* passes statistical scrutiny that this modest benefit is even real.
4. Even if it is real, we have no idea how it works.
The AIDS/HIV vaccine community has been so starved for good news for so long, that it’s easy to understand them for grasping at good news where very little exists. By all means, it could be a good thing, and it’s good to follow up on it, as they are planning to. But the point here is that, as is frequently the case in science, results can be murky.
Or, as a labmate put it to me the other day: (impact)*(certainty) = a constant
What always annoys me about news reports about studies is that they’re often written by reporters who either don’t seem to read the actual study or don’t seem to have a very serious grasp of how to read scientific reports. It stuns me to see the difference between what the news says the study says and what the results actually prove. Correlation is NOT the same thing as cause-and-effect, news writers!!!!