I’m preparing to release a yearly update on new research about the various supplements, and I’ve come across a couple studies (all in the glucosamine/chondroitin sphere) that conclude thusly:
Both the test group and control group were found to be significantly improved from baseline (p < 0.05). There was a trend for greater change in the test group.
That sounds meaningful, but it’s just using p-values to paint an inaccurate picture of what the study really tells us, which is that the test treatment didn’t cause any further significant improvement over the control group—or in other words, the improvements witnessed can be explained by the placebo effect.
What about that trend for greater change? Unfortunately, it doesn’t actually mean anything. Assuming they tested for between-group changes, statistical analysis revealed there to not be any significant difference (even if one group appeared to improve more). If they didn’t test between the groups, we can’t make any assumptions at all.
All of this is aside from any problems with the studies themselves, which there frequently are—things like open-label design (candidates know whether they are receiving treatment or not), subjective pain measurements being used as criteria for improvement, and small and/or non-representative sample groups. These are also quite common in supplement studies!
It’s easy to use p-values to make a completely mundane result seem meaningful, but correct interpretation shows the flaws. Any apparent trend is meaningless without appropriate statistical analysis; in the cases described above, it’s likely that the trend emerged solely as a random event, not as a result of treatment.
It’s also possible that the trend in the treated group wasn’t due to random chance, but the effect size was not sufficiently large enough for the study’s power to detect. But, assuming the study had reasonable power, then the effect of the supplement isn’t big enough to care about anyway. Happy to see more science-based health writing!
Thanks for this explanation. I only recently started to dig into the studies themselves and always find it quite hard to interpret the phrasing. It surely is common domain language in the field of scientific studies but looks totally confusing to me.
Every bit of clarification in that direction helps a lot.
Unfortunately, I think a lot of scientists (as well philosophers, artists, and experts in many fields in general) equate complicated speech with impressive thoughts. The reality is that much of the time they could have said the exact same thing in a much simpler way. As such, I do what I can to phrase things clearly!