Composition of the RU

The RU 5187 PREACT (‘Towards precision psychotherapy for non-respondent patients: from signatures to predictions to clinical utility’) is a collaborative initiative grounded in the rich academic landscape of Berlin (Germany) and is carried out at Humboldt-Universität zu Berlin (HU Berlin), Freie Universität Berlin (FU Berlin), Charité—Universitätsmedizin Berlin (Charité) and Psychologische Hochschule Berlin (PHB). The RU brings together a diverse team of 13 principal investigators (PIs) alongside 36 additional members, including postdoctoral researchers, doctoral candidates and student assistants. Recruitment is based on four academic outpatient clinics (in German: ‘Hochschulambulanz’—HSA). Those HSAs are based in three of the participating sites (HU Berlin: HSA-HU and ZPHU—‘Zentrum für Psychotherapie der Humboldt-Universität zu Berlin/Center for psychotherapy of the Humboldt-Universität zu Berlin’; FU Berlin: HSA-FU, PHB: HSA-PHB). The RU 5187 is funded by the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG, project number: 442075332) with a grant spanning 4 years, from 1 July 2022 to 30 June 2026. The RU is funded from 1 July 2022 to 30 June 2026. Recruitment started in May 2023 and is expected to continue until May 2025. The RU’s structure is intricate, consisting of nine specialised subprojects (SPs), all orchestrated under a comprehensive coordination project (table 1 and figure 2).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Structure and content of the RU and major collaborations among the subproject (SP). RU, research unit.

The coordination project’s mandate extends beyond administrative oversight, embedding within its core objectives the promotion of young researchers, the advancement of gender equality, the management of network funds and the enhancement of scientific communication. This multidimensional approach not only fosters a nurturing environment for research but also cultivates a framework for inclusive and interdisciplinary collaboration (table 2).

The RU encompasses three core science and service projects (SP1, SP2 and SP3). Additionally, six specific science projects target narrower research questions (as detailed in table 1). These include EEG/ECG assessment (SP4), psychological measures (SP5), ambulatory assessment (AA, SP6) and neuroimaging (SP7, SP8, SP9). These projects are tightly integrated to increase the added value and answer research questions reaching beyond particular SPs. The rationale, hypotheses and analytical plans for all projects are preregistered with the open science framework, ensuring transparency and rigour. For additional information about the projects, refer to the preregistrations listed in table 1.

Naturalistic prospective-longitudinal observational clinical trial: study design, participants, eligibility criteria and current status

We include patients with a primary diagnosis of an internalising disorder receiving CBT at one of the four participating academic outpatient clinics. These university-based clinics are integral to Germany’s psychotherapy landscape, serving dual roles in research and education within the German healthcare system, hence ensuring the high ecological validity and representativeness of our sample. The inclusion criteria of the study are as follows: (1) Age of 18 years or older, (2) a primary diagnosis based on the Diagnostic and Statistical Manual of Mental Disorders(DSM)-5 criteria for either SPH, SAD, PD, AG, GAD, OCD, PTSD or unipolar depressive disorder (major depression or dysthymia, DD), (3) an indication for outpatient CBT treatment, (4) a minimum treatment plan of 12 sessions (excluding mandatory probatory sessions required in Germany before treatment), (5) a symptom severity indicated by a General Severity Index of the Brief Symptom Inventory (BSI-GSI30) greater than 0.56 (except for SPH31), (6) a Clinical Global Impression–Severity Scale (CGI-S32) score of 3 or higher (indicating at least mild illness) and (7) a willingness to participate in the study. We abandoned the BSI-GSI inclusion criteria for SPH because too many patients had to be excluded at the beginning of recruitment due to this criterion, making it impossible to reach our recruitment goal. Exclusion criteria were held to a minimum in order to foster ecological validity of the sample. They mainly encompass contraindications for outpatient treatment, including (1) a current secondary diagnosis of moderate to severe substance use disorder (including regular use of benzodiazepines), (2) current psychosis, (3) current bipolar disorder, (4) more than moderate suicidality and (5) medical conditions that contraindicate CBT. Patients who require inpatient treatment during the course of the outpatient treatment will be excluded. However, the presence of other comorbid disorders will not lead to exclusion. Concurrent pharmacological treatment is permitted and will be thoroughly documented.

Patients opting for the experimental assessments (EMA, fMRI and EEG/ECG) will receive detailed information about possible risks and additional eligibility criteria including specific safety-related exclusion criteria will be applied.

In our sample (28 November 2024), the distribution of diagnoses was as follows: ADs 35.5%, OCD 18.6%, DD 41.0% and PTSD 4.9%. Compared with a nationwide sample from the KODAP consortium with 6098 patients (ADs 29.29%, OCD 5.76%, DD 58.53% and PTSD 6.42%), the distributions are largely similar.33 34 However, the proportion of patients with OCD is notably higher in our sample. This can be attributed to the specialisation of one of our outpatient clinics in OCD treatment, which led to an intentional enrichment of the sample.

For current study progress, see figure 3.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Study CONSORT flow chart, 28 November 2024. ¹Termination during probatory sessions. ²Mandatory preliminary sessions (up to four) required in the German healthcare system before initiating treatment. CONSORT, Consolidated Standards of Reporting Trials.

Sample size calculation

We estimate effect size from two perspectives. First, we aim to achieve or surpass the best ML model prediction accuracy reported in existing literature, which is around 75%. We expect that improving prediction accuracy by 25% from the chance level (50%) will have clinical and health-economic significance.

To accurately estimate the necessary sample size, two key factors are considered: (1) prediction accuracy is expected to increase with a larger training dataset, up to the maximal achievable accuracy based on the underlying effect and (2) fluctuations in detected prediction accuracy around the true value will decrease as the test set size increases. The exact training set size needed to achieve a 75% prediction accuracy is uncertain and depends on both predictor characteristics and the chosen algorithm. Our simulations, consistent with previous research,35 suggest that prediction accuracies plateau at around 300 samples in the training set across various models (figure 4). Further simulations indicate that a test set of about 100 patients will likely yield a measured prediction accuracy of 75%–85%.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

Simulation of association between sample size and prediction accuracy. Association of train and test set sizes with prediction accuracy across simulations. (A) Across increasing size of n train, prediction accuracy is increasing until reaching a plateau (ie, the maximum prediction accuracy possible based on the underlying effect). Starting point and gain in prediction accuracy depend on data and model parameters. (B) Across increasing size of n test, the measured prediction accuracy varies less and less around the true prediction accuracy for the underlying effect.

Based on these insights, a fivefold cross-validation method appears suitable. This involves assigning 400 patients to the training set and 100 to the test set, with the analysis repeated five times to estimate variability.

Additionally, for comparative purposes with traditional studies, we also present an effect size calculation for group-based linear multiple regression analyses, despite not using this method in our study. Considering an alpha level of 5%, a power (β) of 80% and 19 predictors, small effects (R²=0.07) can be detected with a sample size of 500 patients. This calculation is based on the power analysis conducted using G*Power V.3.1.9.2 for linear multiple regression. The recruitment plan for the study is designed to ensure a steady and sufficient number of participants throughout the trial. Initially, at baseline, 585 patients will be recruited. By the fifth session (S5), it is anticipated that the patient number will decrease to 555 due to an expected attrition rate of 5%. As the study progresses to the 20th session (S20), the number of participants is projected to be 527, accounting for a cumulative attrition rate of 10%. Finally, at the postassessment, the study expects to have 500 patients, reflecting a cumulative attrition rate of 15%. This recruitment strategy considers potential dropouts and aims to maintain a robust sample size for the study’s duration.

Treatment setting, contents and procedure

The treatment follows the same basic CBT approach at all sites and is based on established manuals,36–39 but not strictly manualised to mirror the ecological approach. We aim to predict TNR in a naturalistic setting, where CBT is usually tailored (based on clinician expertise, ie, ‘eminence-based personalisation’) to the specific requirements of each patient. However, despite this form of personalisation, TNR remains a frequent outcome. Therefore, this naturalistic CBT-based treatment addresses individual variation in clinical phenomenology, including comorbid disorders. These elements are not only addressed through individualised therapy plans but are also analysed as potential predictors of TNR.

Standard manuals36–39 typically encompass psychoeducational components, various intervention forms like exposure techniques, behavioural activation or cognitive restructuring and relapse prevention strategies. Research40 41 indicates that despite their differences, these techniques collectively enhance basic ER skills. The regimen and duration of treatment adhere to the specific regulations of the German public healthcare system, markedly distinct from those in other countries such as the UK or USA. The legal framework permits up to 80 sessions of 50 min each. Insurance companies approve varying numbers of sessions, categorised into ‘short-term therapy part 1’ for sessions S1–12, short-term therapy part 2 for sessions S13–24, and extensions for long-term treatment usually up to 60 or in some cases up to 80 sessions. Typically, treatments involve one session per week, with longer sessions for exposure therapy.

The number and length of sessions at our outpatient clinics vary between the type and severity of the disorder, averaging 40–50 sessions over 1–1.5 years. Nonetheless, treatment can deviate to as few as 10–20 sessions (short-term treatment) over 4 months or extend to 80 sessions (long-term treatment) over 2 years. Considering these variations and trial time constraints, we decided to evaluate treatment outcomes after a predetermined number of sessions (‘treatment dose’ at S20) and at the end of treatment with a maximum duration of 12 months (‘treatment time’; although the naturalistic treatment may go beyond, but not within the study regimen). Additionally, we will monitor early psychological changes after the fifth session (S5).

Patients seeking treatment will first undergo routine screening, receive a detailed explanation of the study protocol and provide informed consent before participating. The initial five preliminary/probatory sessions, designated as the diagnostic interval, serve as the baseline period for eligibility assessment based on inclusion and exclusion criteria, and for collecting multimodal predictors such as clinical data, AA including EMA and passive sensing, fMRI and EEG/ECG data. All clinical data are electronically compiled in a central database via ‘Research Electronic Data Capture’ (REDCap), which will be integrated into a shared and secured server alongside our multimodal datasets.42 43

For an in-depth overview of the study flow, see figure 5.

• Download figure
• Open in new tab
• Download powerpoint

Figure 5

Study workflow. AA, ambulatory assessments; EEG, electroencephalogram; HRV, heart rate variability; S, treatment session.

The study was approved by the Institutional Ethics Committee of the Department of Psychology at Humboldt Universität zu Berlin (approval no. 2021-01) and the Ethics Committee of Charité-Universitätsmedizin Berlin (approval no. EA1/186/22). Participants in the experimental sessions, including AA, EEG/ECG and neuroimaging studies, will receive financial compensation. The trial has been registered with the German Clinical Trials Register (DRKS00030915) and on the open science framework platform of the Center for Open Science, available at https://osf.io/bcgax.

Data analysis: ML preprocessing and validation pathway

Here, we focus on ML models for the predictive analytics. For detailed analyses of single SP, see preregistrations in table 1.

For predicting TNR, we will initially develop separate models for each data modality—including clinical, behavioural, psychophysiological and neuroimaging data. Subsequently, we will integrate these modalities to enhance the predictive accuracy.

For tabular data, which includes demographic, clinical and psychological data as well as features derived from EMA data, we will primarily use classical ML algorithms like gradient boosting and random forests, known for their efficacy with structured data.53 We will also explore the use of elastic nets, support vector machines and experimental deep learning techniques,54 although cautiously due to their unproven benefits for non-hierarchical data and small sample sizes. For the interpretation and assessing the importance of the individual features, we will use model-agnostic shapley values (SHAP,55). This will also allow for comparing the relative importance of the different data sources (eg, demographic vs EMA data).

For neuroimaging data, we will systematically compare classical ML methods relying on extracted features (ie, image-derived phenotypes) with deep learning methods directly applied to minimally processed (ie, harmonised and linearly registered to MNI space) MRI data. Starting point will be a comparably simple CNN model that we originally trained for differentiating between patients with Alzheimer’s disease and healthy controls based on T1-weighted (MPRAGE) data,56 but used as a baseline model for other use cases.57 Based on this baseline model, we will experiment with different settings of the network based on the training data to optimally adapt to the acquired data while still allowing to generalise (eg, numbers of layers or filters, different loss functions, different regularisation schemes, including short-cuts or dense connections, etc).

To manage small sample sizes in deep learning models based on MRI data, we will implement two key strategies. First, data augmentation will be employed, where modified versions of the input data are incorporated into the dataset. This approach not only serves as regularisation but has also proven to enhance performance in MRI studies.58 Second, we will use transfer learning.59 Here, models are initially trained on a source dataset (eg, UK Biobank or other retrospective datasets) for a specific task (like disease classification) and subsequently fine-tuned on a target dataset for a different task (such as outcome prediction). This method leverages pretrained weights to refine the model for the target task, a technique demonstrated to significantly boost performance in our previous studies, including across different tasks and MRI sequences.57

Additionally, we will explore various scenarios for effective transfer learning, such as the adaptation of models from general disease classification to specific outcome prediction within patient groups. This process not only helps in identifying key areas for differentiation between patient groups but also in assessing the transferability and efficacy of models across different tasks. Our approach will include a comparative analysis of models with and without pretraining to determine the most effective settings for transfer learning. Successful model transfer to clinical practice will be a key consideration. To elucidate the decision-making process of these neural networks, we will use advanced visualisation techniques such as SHAP and layer-wise relevance propagation.60 These post hoc methods will help in understanding the neurobiological correlates of TNR as identified by neuroimaging techniques, and in assessing the impact of different transfer learning models.

After analysing each data modality separately to evaluate its stand-alone predictive performance, we will focus on integrating diverse data types, including neuroimaging data, psychophysiological, clinical data, psychological test scores and EMA data. Combining these varied data sources presents unique challenges, especially given the unknown interactions between them and the uncertainty regarding the informativeness of specific features.61 We will explore both intermediate and late data fusion techniques. In intermediate fusion, features from different modalities are combined into a single model after separate processing and extraction. In contrast, late fusion uses these features as inputs for different classifiers in an ensemble approach. For MRI data, we will use either image-derived phenotypes or CNN-based representations. Our aim is to determine the relevance of different data domains, including emerging putative biomarker candidates from various sources. We will also pursue a cost-sensitive strategy, focusing on minimal yet effective feature sets, applying dimensionality reduction and feature selection techniques like recursive feature elimination. Data fusion methods will range from simple concatenation to more complex techniques like ensemble learning, boosting, attention-based methods and multiple kernel learning, the latter of which allows for consideration of interdependencies between data sources.61 62 For deep learning-based multimodal data integration, we will rely on the recently published toolbox fusilli (https://iopscience.iop.org/article/10.1088/2516-1091/acc2fe),

To maintain comparability between methods and data types (tabular or neuroimaging, unimodal vs multimodal analysis), we will apply a consistent validation strategy. Data will be randomly divided into training, validation and test sets. To account for the relatively small sample size, we will perform these splits multiple times (based on computational feasibility and prediction variance) and calculate average performance metrics as well as variability indices to estimate the reliability of prediction. In cases of data imbalance to the respective label (age, sex or TNR), stratified splitting will be used. The effectiveness of the models will be assessed based on their ability to predict treatment outcomes for individual patients and across various disorders. This evaluation will also consider the interpretability of the models by clinical or domain experts. Further, we will examine models for age-specific or sex-specific effects by separately analysing performance across different subgroups, such as women vs men or younger versus older individuals, and our developed deep representation visualiser allowing to estimate the effects of different confounding variables in a trained ML model.63 The significance of each model’s performance will be verified using permutation tests, where labels are randomly shuffled. This comprehensive approach ensures rigorous evaluation and comparability of the models across different data types and research scenarios.

We are committed to using Python for model development, leveraging libraries like sklearn64 and PyTorch, and making our code publicly available for broader use and collaboration. ML experiments will be conducted at the computing facility of Charité–Universitätsmedizin Berlin.

Data handling and dissemination

In compliance with the stringent requirements of the General Data Protection Regulation in Germany (‘Datenschutz-Grundverordnung’—DSGVO), we have set up a data protection concept for prospective clinical studies and signed a Joint Controller Agreement by all participating institutions for data handling. These documents have been approved by all respective data protection officers. Given the sensitivity of these health data, they are subject to strict protection protocols and are not shared with third parties as long as they remain identifiable. To ensure the privacy of the patients is preserved, all data are safeguarded by appropriate technical (audit-trails and user rights management in access group) and organisational measures (standard operating procedures and standardised personal training) throughout the study phase. One year after the completion of the trial, all data will be irreversibly anonymised. MRI will be defaced to remove identifiable features. Only after this anonymisation process will the data be shared with other consortia such as ENIGMA (Enhancing neuro imaging genetics through meta-analysis; https://enigma.ini.usc.edu/).

Patient and public involvement

Implementing a participatory research approach right from the outset together with the DZPG (Deutsches Zentrum für psychische Gesundheit–German Research Center for Mental Health) PPI (patient participation involvement), we developed the research objectives and study design of the renewal proposal conjointly with a patient advisory board encompassing five experts with lived experience. Those were invited to join our consortium and consulted us in several meetings. Importantly, principal investigators wrote layman’s sketches of each study plan which were uploaded in a joint document that was reviewed and commented by those five experts.