Acceptability and perceptions of end-users towards an online sports-health surveillance system

Design and participants

This was a mixed-methods study conducted between April 2014 and January 2015. Participants were prospectively followed online during a sporting season, and were interviewed face-to-face at one follow-up point of the study (figure 1). The participants of this study were from a Brazilian private multisport club. Each sport-discipline of this club has its own responsible trainer and medical staff.

The head of the physiotherapy department of the club selected 3 sport-disciplines by convenience to participate in this study. National elite athletes from judo, swimming, and volleyball, and their respective staff, were verbally informed on the purpose and procedures of this study. Detailed information on the study and procedures was also provided digitally. All participants agreed to an electronic participation consent form approved by the ethics committee of the VU University Medical Center Amsterdam, the Netherlands.

The online system

At baseline, athletes were requested to complete an online questionnaire on age, body height and weight, years of sport experience, injury history and current health status. Subsequently, athletes were followed-up by email on a regular basis. The emails included a secure link to an online questionnaire that asked how many hours athletes had spent on sport specific training, general strength and conditioning, training and competition. The online questionnaires also captured athletes’ health complaints using the Oslo Sport Trauma Research Center (OSTRC) questionnaire.8

The 4 key questions of the OSTRC questionnaire measure to what extent a health complaint has affected (1) sport participation, (2) training volume, (3) sport performance and (4) the extent of health symptoms experienced by athletes. The answers to the 4 questions generate a severity score for the reported health complaint that ranges from 0 (ie, no health complaint) to 100 (ie, severe health complaint). For the purposes of this pilot study, the 4 key questions of the OSTRC questionnaire were translated to Brazilian Portuguese by the primary researcher (SDB).

In case of a health complaint, athletes reported their training and/or competition time loss in days, whether the health complaint was registered for the first time and if it concerned an injury or illness. In case of injury, athletes described the injury onset, the affected body location, the injury type and the medical attention received. In case of illness, athletes were asked to specify the symptoms and the medical attention received. In case of more than one health complaint, athletes were instructed to report the most severe complaint first. Further complaints could be registered subsequently using the same procedure.

In order to best fit the system in the teams’ usual practice, the starting point of the surveillance and the frequency of follow-up was different among sport-disciplines (figure 1), and athletes were formally included in the study after answering the baseline questionnaire. Judo athletes were followed-up every 2 weeks, and reported their sport exposure and health complaints regarding to the past 2 weeks. Swimming and volleyball athletes were followed-up weekly, reporting at every follow-up, their sport exposure and health complaints regarding to the past week. Athletes’ responses were gathered and reported to their respective medical and trainer staff on a regular basis (ie, every 2 weeks for judo, and weekly for swimming and volleyball), as presented in figure 2A.

End-users’ perceptions

A semistructured interview guide (see online supplementary file 1) was prepared to (1) investigate end-users’ (ie, athletes and team staff members) satisfaction and expectations towards the sports-health surveillance system, and (2) the relationship between the system and within-team communication, perceived health status of the athlete and quality of healthcare (figure 2B). Subsequently, the primary researcher (SDB) visited the club location for 4 consecutive days (figure 1) and approached a convenience sample of end-users that was available and willing to share perceptions regarding the online system. After a short ‘ice-breaker’ conversation, end-users agreed upon having the interview recorded in digital audio format. Confidentiality and anonymity were assured before the recording device was turned on for the interview.

Data analysis and presentation

Descriptive analysis was conducted to present athletes’ characteristics at baseline and follow-up measures. The adherence of athletes to the online system (ie, response rate to the follow-up questionnaires) was calculated for each follow-up point. The overall response rate per sport-discipline is presented as mean and its SD. Data analysis was performed using R, V.3.3.2.9

Qualitative analysis of interview data was based on the constant comparative method using principles of the grounded theory.10 The interviews audio was transcribed by the primary researcher (SDB) and coded independently by two researchers (SDB and CSB) using the R-based Qualitative Data Analysis package.11 The main coding categories were predetermined by the research question, which included the end-users’ perceptions towards the online system (ie, satisfaction, expectations, perceived relationship between the system and within-team communication, perception of the athletes’ health status and quality of healthcare).

The coded transcripts from SDB and CSB were compared and discussed with a third researcher (EV). Accordingly, a final consensus on the coding framework was reached (figure 2B). Qualitative data are presented according to the final coding framework, and grouped by athletes and team staff’s perspectives, respectively. Interview quotes are presented in italic format.

Baseline data

Ninety-nine athletes were invited to participate in this study, and 74 (75%) were included after they had answered the baseline questionnaire. Athletes’ characteristics at baseline are presented in table 1.

Athletes’ adherence to the online system

Of the 74 athletes who had answered the baseline questionnaire, 5 (7%) did not respond to any follow-up questionnaire. The mean of the follow-up period per athlete was 42 (SD 2) weeks for judo, 18 (SD 2) weeks for swimming and 15 (SD 3) weeks for volleyball athletes. The response rate per sport over the study period is presented in figure 3. The mean of the biweekly response rate was 50% (SD 23) for judo. The mean of the weekly response rate was 61% (SD 27) for swimming, and 56% (SD 25) for volleyball (table 2).

End-users’ perceptions

All users approached by the researcher in the club facility accepted to be interviewed. Fifteen athletes were interviewed until saturation was reached, that is, the point after which no new concepts were introduced by athletes in interviews. Five staff members were interviewed, without reaching saturation. The interviews lasted 8 min on average (ranging from 4 to 16 min). Most of the staff members mentioned that the physician responsible for the three sport-disciplines was using the reports from the system to follow-up athletes with health complaints. The physician was not available for a face-to-face interview. Therefore, the physician was contacted by email and answered the interview questions in writing. An overview of the 21 end-users who were interviewed is presented in table 3.

End-users’ satisfaction

Out of 10, athletes graded the system as 8.2 on average (ranging from 5 to 10). Most were satisfied with the user interface because they could answer the questionnaire with a few clicks and did not need to type much.

Athlete: I like it because it is practical, we can answer [the questionnaire] on our phones.

However, a drawback was that athletes needed to detail the health complaint at every follow-up when the same complaint was lingering on for multiple weeks. An athlete said he/she usually reported no health complaints, because it took time to detail the health complaint.

Staff members graded the system as 9.2 on average (ranging from 8 to 10).

Trainer: [The system] establishes a procedure, avoiding that [athletes’ health] information go unnoticed.

According to staff members, the system was not graded 10 because reports were not mobile friendly.

End-users’ expectations

Athletes expected to receive some feedback on their inputs to the system. This feedback should motivate them to keep responding to the questionnaires.

Athlete: Some people were answering [the questionnaires] and realized they did not receive any feedback; then they stopped [responding].

Athlete: I think it would be important to have this information (feedback) so I could investigate together with the team staff if there is a problem that is happening frequently, something that is bothering me often, that I am not realizing.

The feedback to athletes could be in the form of a report showing their responses over time.

Athlete: I like graphs, you can visualise what is happening with your body. It is interesting to see how your health [status], your performance is.

Staff members’ first expectation was to see a complete adherence of athletes to the online questionnaires. They also mentioned that it would be important to incorporate a feedback loop to the athletes in the system, to increase athletes’ response rate.

The online system and within-team communication

Although athletes preferred to communicate face-to-face, some think that it is easy to use the online system to communicate their sport exposure and health complaints to all staff at once.

Athlete: We are in contact [with team staff] frequently. It is easier to communicate directly.

Athlete: [The system] helps because we provide a complete feedback to all team staff, not only to the physician or the physiotherapist.

Staff members did not perceive the system as a communication tool with athletes once their main communication was conducted face-to-face. However, the system facilitated the communication between the medical and training department.

Trainer: The system improved the communication between medical and trainer staff.

Besides the system, staff members used different tools to communicate with and collect data from athletes. However, integrating the information from these different tools into one platform remains a challenge for them.

Trainer: The amount of information we have up to now is enough. What is lacking is a way to combine this information in a way that is easy for everybody [to access].

The online system and the perception of health

Athletes had to reflect on their training/competition load and health when answering the follow-up questionnaires. This reflection made them more aware of their health status.

Athlete: I started to pay attention to things that I usually did not. Pain or some conditions, I realized, were getting repetitive…

However, an athlete said that nothing had changed after using the system, given that no feedback was provided on athletes’ answers.

According to team staff, the system reports displayed health complaints that were not necessarily communicated personally by athletes.

Trainer: With the system reports I realized that there were things happening [with athletes] and I was not aware about them.

As athletes were followed-up online every 2 weeks (judo) and weekly (swimming and volleyball), the system was not able to capture the fluctuation in health complaints on a daily basis.

Physiotherapist: Oftentimes what I see in the reports is what the athlete complains about [to me], and sometimes not. Maybe the athlete had answered the questionnaire when he/she had a complaint, but he/she came to me 3 or 4 days after and the complaint had changed, improved, or he/she got a new complaint.

However, a trainer said that it would not be feasible to receive reports more often than weekly.

Trainer: It is not real to imagine a system that provides feedback more often than this (weekly) […] I do not think that [more frequent reporting] is necessary, because we do not have human resources to process such information faster [than weekly].

The online system and the provision of healthcare

Athletes did not perceive an effect of the system on the provision of healthcare.

Athlete: We had always all [health] assistance that we need [regardless of the system]. But I believe the more information, the better.

However, an athlete said that questionnaire answers would be useful to investigate patterns in health problems and to develop preventive measures accordingly. Another athlete mentioned that the questionnaire answers might be used by the medical staff as a preassessment, before an actual appointment.

Staff members also did not perceive a direct effect of the system on the provision of healthcare. However, the physician said it was easier to track athletes’ health with the system reports as he/she usually has less personal contact with athletes than physiotherapists and trainers.

Physician: With the system usage, I had more control over what happens with athletes, and I could intervene early when necessary.

Main findings

In the present pilot study, athletes registered their health complaints online through the 4 key questions of the OSTRC questionnaire. The OSTRC questionnaire has been used to register health problems in sports in both weekly8 12 and biweekly basis.13 14 In our current elite athletic sample, the questionnaire picked up an average prevalence of health complaints, ranging from 28% to 36% across the 3 sport-, showing the presence of health complaints within the studied population. However, the adherence of athletes to the online system in the current study was relatively low when compared with previous studies using the OSTRC questionnaire. The average response rates reported in previous studies ranged from 80% to 95% using a similar registration system.8 13 The lower adherence of athletes as found in the present study may be explained by the lack of feedback on their inputs. Athletes stated during interviews that they expected to receive feedback on their responses to the questionnaires, and this should motivate them to keep responding.

The present study evaluated the online system as a practical tool (ie, not as a research instrument), without control or influence from researchers on the use of the system by end-users. This may also explain the lower response rate of athletes to the online follow-ups compared with previous research. Despite low engagement of athletes to the online system, most stated that the system was simple to use. The questionnaires made athletes reflect on their health status while facilitating the reporting of sport exposure and health complaints to team staff. This procedure also enabled the physician, who did not see athletes as frequently as physiotherapists and trainers, to have additional oversight of athletes’ health status, despite having less contact time with athletes as compared with trainers and physiotherapists. Despite these benefits, athletes and staff members did not perceive that the system provided a large enough effect on the provision of healthcare to be worthwhile.

Athletes preferred to communicate health complaints face-to-face, and staff members could identify in the system reports athletes’ complaints that had not been communicated personally. However, an athlete usually reported no complaints because, according to the athlete, the time burden associated with entering the complaints in the system was too great. This information was provided by one athlete during a face-to-face interview and it is unknown whether more athletes used the same ‘strategy’. Athletes were informed verbally and digitally about the study procedures, yet self-reported methods are still dependent on honest information. Identifying whether athletes report the truth remains a challenge, and threats the validity of the self-reported data.15

Practical implications

This pilot study provides knowledge on the acceptability and the perceptions of end-users towards an online sports-health surveillance system. This information is important for those interested in using online technology to monitor athletes’ health. The findings of this study will help our research group to update the system according to the perception of end-users, before large scale implementation. The main feature to be incorporated is a feedback loop to athletes that provides a visualisation of their responses to the questionnaires over time. This feedback will potentially increase the adherence of athletes to the system, and provide them personalised information that they can use for their own purposes.