While hanging out at Columbia the past couple of years, i was comforted to realize that i wasn’t alone in my lack of enthusiasm for p-values. In fact, while some folks in some disciplines are completely infatuated with them, did you know there are others out there that dismiss them? i mean, entirely!? [removes tongue from cheek]
While it might be an overstatement to suggest that the variety of headlines that accompanied the news of the HIV vaccine trial in Thailand claimed that “AIDS is now a disease of the past,” it really wouldn’t be stretching the tone of some of those headlines too far. Some people really did seem to think this was the breakthrough we’d been waiting for. That is, those people who focused solely on the p-values. E.g., from one of the NYT pieces on it “Although the difference was a mere 23 people…it was statistically significant.”
But what most people seemed to fail to pick up on was the very next part of that sentence –“…and meant that the vaccine was 31.2 percent effective.” And the nuisance of a realization with that statement that no vaccine would ever be seriously considered for scale up before that number reached at least 80%, if not higher.
So, yes, this was the first such trial that showed any difference whatsoever. But was it worth all the hoopla that followed? Well, given Fauci’s quick re-interpretation for the public shortly thereafter, followed by some minor tweaks in analytic strategy of the data (mind you to follow more standard reporting practices) revealed that the “statistically significant” portion of the interpretation may not even be warranted, i think maybe no.
Now, i’m not posting about this to quibble about the significance (or lack thereof) of their findings. What i want to bring up is the important difference this highlights between statistical and substantive significance. In practical terms, this particular study’s findings isn’t likely to have much of an impact at all on the way we approach HIV-prevention (not even if the statistical significance were a slam dunk, i.e., p<0.001). Because while (potentially) statistically significant, it just wasn’t large enough to really have a meaningful impact. i say this as a disappointed HIV researcher who would be more than happy to welcome a vaccine to the available arsenal. However, i am not going to start doing cartwheels over small statistical differences, that could have happened by chance,* and ultimately are substantively small enough to not be implemented in any helpful way (i.e., producing a vaccine that reaches the public and actually prevents new infections).
All that to say something that is likely obvious to many reading this, but is often worth repeating – “Statistical significance is not always all it’s cracked up to be.”
*Remember p<0.05 does still mean that 1/20 times it could be random. And i don’t have a hard count on how many of these trials we’ve seen at this point, but the numbers have got to be adding up.