• gender issues
• genetics
• sex testing
• validation
• women In sport

How can performance data resolve the arbitration of sensitive matters in the world of sports? In the absence of experimental data (ie, clinical trials), researchers must build an argument based on correlations in observational data. Such data are often not widely available. The Dutee Chand v. AFI & IAAF case is a case in point (box ).1

Bermon and Garnier2 use correlations between free testosterone (fT) and athletic performance across 21 women’s events to claim that women with high fT have a performance advantage in a very specific subset of athletic events. On 29 September 2017, their data were filed as evidence by the International Association of Athletics Federations (IAAF) to the Court of Arbitration for Sport (CAS) in the Dutee Chand v. AFI & IAAF case. These data added to draft revised regulations (not available in the public domain) that would apply only to female track events over distances of between 400 m and 1 mile.

The CAS Panel has made no ruling (as of 28 January 2018) about the sufficiency of the evidence put forward by IAAF. As of a press release dated 19 January 2018, the regulations remain suspended for an additional 6 months at which time the IAAF is  to advise CAS on how it intends to implement the regulations moving forward.3

We argue that the evidence put forward by Bermon and Garnier2 is not sufficient to sustain even draft revised regulations applying only to specific events. Our reanalysis of the available data presented by Bermon and Garnier2 suggests, at the very least, that further analysis is required to establish the claims made in the paper.

The application of statistical techniques, and interpretation of results, in such studies is not neutral, nor standardised; correlations in observational data require careful interpretation by independent researchers with access to the original data. Can the data used in Bermon and Garnier2 tell us something about whether testosterone confers an advantage in particular events, individually, or just whether there is an overall correlation across all events?

We argue that, given the sample sizes used for each event, and the number of statistical tests conducted by the authors, any particular significant result in an event is more likely to have arisen by chance. Unfortunately, without publicly available raw data, it is not possible to perform all the desired robustness checks on the data. In lieu of access to such data, we performed a Fisher’s combination test using the P values calculated from the published data. After performing such a test, we were unable to reject the global null hypothesis that all null hypotheses are true, that is, the pattern of P values is not inconsistent with there being no advantage to high fT women, in any one of the events. In simpler terms: it is reasonably likely that the correlations presented in the paper (even the largest ones) occurred by chance.

Given the number of tests performed, the few significant findings detected could have arisen without there being a true correlation between testosterone and performance for female athletes. To avoid these chance findings (also known as false positives) , appropriate multiple hypothesis testing corrections ought to be applied. 

Our reanalysis of the data (see online supplementary web content for additional details) suggests that this correction would not yield a robust and significant correlation in any event. Given these findings, we believe that it is scientifically incorrect to draw the conclusions in the Bermon and Garnier2 paper from the statistical results presented. Their paper claims that certain athletes have an advantage in precisely the five events where a significant effect was found: we calculate that a high share of those five significant effects are likely to be false positives. The overall range of coefficients across all events is large: we estimate the average advantage for high testosterone women to be 0.7%, with a minimum of −2.6% and a maximum of 4.5% across the 21 events.Only in 12 (57%) of the events do higher fT athlete perform better on average. With access to the data, a more sensible test might be one single test of correlation conducted across all events.

In light of our reanalysis, we conclude that

1. Raw data used in such studies, which will have direct implications for real-world outcomes, should be made publicly available for other researchers to analyse.

2. Interpretation of estimated correlations should also be conducted with great caution and be referred to independent statisticians. While we do not claim to play the role of such an independent statistical arbitrator in this case (especially since we have not had access to the raw data), our statistical analysis already allows one to conclude that the article by Bermon and Garnier2 does not meet the standard of proof set by the CAS, without further analysis. Independent analysis is necessary in this situation, and others like it.