New England Journal of Medicine for the last time
Sorry for going on yet again about the New England Journal of Medicine, but I recently had the experience of publishing a trial there (this one) and interacting with their statistical guidelines, which I have discussed before (here and here).
The issue about p-values for baseline characteristics came up, of course. Here’s what came back from the Editor:
It is a general policy of the Journal to request that the table of baseline characteristics (here Table 1) should include a footnote indicating which (if any) of the listed characteristics differ at P<0.05. The editors appreciate that, in a randomized trial, all differences between groups occur at random, and therefore the P value does not have its conventional meaning here. However, it is the view of our statistical advisers that such P values can be useful in calling attention to notable imbalances (if any) between the groups.
The table duly appeared with the footnote as required:
This strikes me as possibly the worst of the available options. If we gave the exact p-value for each of the variables, that would give us a measure of how unusual the data were compared to the expectation of zero difference, which makes some sense. Just saying there were no significant differences is misleading, because the whole concept of significance tests does not apply here. We’re not performing tests with the intention of rejecting the null hypothesis if we find p < 0.05, because we already know that it is true. It’s randomised, so we already know that differences are only due to chance.
It’s good that the editor recognised that the use of the p-value for baseline characteristics is different from the usual use, but not so good that they ask for those with p < 0.05 to be indicated. That will inevitably get interpreted by almost everyone as indicating “significance,” “rejection of the null hypothesis” and all of that stuff. I think it would be much better to call attention to large imbalances using something other than a p-value, and definitely something other than just indicating where p < 0.05.