ITCs are increasingly becoming part of the core evidentiary base in HTA submissions, particularly with the growing need for indirect evidence to compare a novel treatment's relative efficacy and safety with all potential standards of care. In preparation for the newly instituted EU JCA, EUnetHTA completed 23 REAs for pharmaceuticals from 2010 to 2021, all as part of EU-funded projects.

The current study systematically reviewed and evaluated the ITC-related components from the EUnetHTA REAs to better understand their role in future EU JCAs. Across the 23 REAs, 12 included an ITC. In total, 64 treatment comparisons were required, with a median of four comparators identified per REA. The suitability of ITC evidence by EUnetHTA was largely categorised as ‘unclear’—only one ITC was designated as ‘appropriate’. No ITCs were considered as ‘unsuitable’. Fifty-two critiques of ITCs were identified across the four main categories of data, methods, uncertainty and ‘other’, more than one-third of which were related to data/evidence shortcomings. There are underlying limitations of ITCs rooted in data availability. In this study, heterogeneity between trials and populations was the primary driver of issues with data.

These findings highlight the significant necessity for ITCs and underscore the ongoing ambiguity regarding their use in HTA positions. The main critiques of the use of ITCs in the EUnetHTA REAs regarding heterogeneity and suboptimal data in the current study were consistent with findings from the literature. In a 2023 systematic review of PAIC methods in HTA, the authors concluded that in current HTA practice, the conduct and reporting of PAIC methods was considerably heterogeneous and suboptimal [35]. In a review of HTAs submitted to NICE, the appraisal committee expressed concern about the quality of the data underpinning the analyses [36]. Although data derived from ITCs are generally considered insufficient in HTAs, their acceptance varies from country to country; for example, Macabeo, et al., 2024, evaluated the acceptance rate of ITCs in HTAs in England, France, Germany, Italy and Spain, which ranged from 0% in France to 47% in England [10]. In Ireland, the National Centre for Pharmacoeconomics issued positive recommendations for 34% of submissions from 2018 to 2023 that relied on indirect evidence versus 28% of submissions based on head-to-head trial data [37].

The current review confirms the inevitable need for comparative evidence based on ITCs to demonstrate the relative efficacy and safety of a novel health technology compared with a spectrum of standard-of-care options. Direct head-to-head treatment comparisons versus all treatment options are often unavailable, leading to gaps in critical evidence. ITCs bridge evidence gaps and inform decisions by different stakeholders. This valuable role of ITCs has prompted guidance on the use of direct and indirect comparisons from the Member State Coordination Group on HTA [38, 39]. This guidance provides details on the available methods (e.g. their strengths, weaknesses and underlying assumptions) as well as the appropriateness of the method as it relates to the data situation; these guidelines are descriptive in nature without stating a preference for a particular method [38, 39]. The array of standard and novel methods discussed underscores the indispensable need for ITCs and highlights that certain methods, while justified by specific research questions and data availability, are associated with inherent limitations.

Although data from randomised controlled trials are still the preferred source of evidence, HTDs will need to demonstrate the clinical effectiveness of a product using the best available evidence to enable timely downstream decision-making and avoid delays in patient access to novel health technologies. However, the introduction of the EU JCAs, which aims to address national policy questions across all EU member states in a single assessment, triggers uncertainty in the ability of HTDs to confidently meet the expected PICO scope with direct and indirect comparative evidence.

Despite the issued guidance, additional advice from the Member State Coordination Group is recommended to provide more transparency into what factors influence how indirect evidence is evaluated within the context of data limitations. The European Access Academy has stated that some current guidance from the Member State Coordination Group will ‘be impossible to fulfil’ and that a more realistic approach to ‘best available evidence’ should be considered in light of the fact that ITCs will be a critical component of upcoming HTAs [40].

The overarching aim of the comprehensive scope of the PICO is to consider the requirements of all 27 EU member states. The reality of addressing and assessing multiple PICOs, however, will have a consequential effect on HTDs as well as on assessors/co-assessors. The EUnetHTA Practical Guideline provided an example in which five PICOs were consolidated [9]. However, in a hypothetical case study in first-line non-small cell lung cancer, an estimated 10 different PICO combinations across six member states would be needed, six of which were for a single country; 17 unique outcomes per PICO—for a total of 170 outcomes—would be expected [41]. In the current study, the number of comparators in the included EUnetHTA REAs with ITCs ranged from one to 18. The potential increase in evidentiary requirements (based on PTJA06 [12]) is illustrated in Supplementary Fig. 1. There is a risk that many comparisons will be unfeasible owing to missing evidence (as learned from EUnetHTA) or the analytical workload (i.e. multiple PICOs) in the limited timeframe available for JCA dossier preparation and finalisation. PICOs are policy driven and established ahead of the manufacturer submission. They do not account for availability of evidence or findings of an indirect comparison feasibility assessment.

The current JCA scope, as defined by PICOs, is based on the anticipated label indication wording and specified through a scoping process that involves all EU member states. During this scoping process, the policy needs of EU member states are translated into research (PICO) questions. However, addressing these policy-relevant PICOs with high-quality evidence can present several challenges for HTDs. First, some PICOs may vary from the registrational trial population. The patient (sub)population requested by EU members states to be addressed may either be excluded by design (hence, no clinical trial evidence exists) or may represent a subpopulation of the registrational trial, thus, requiring HTDs to conduct a subgroup analysis. In this latter case, depending on the specific trial design, randomisation and balance of confounding factors may no longer be guaranteed. Second, many PICOs will involve different comparators, likely requiring evidence external to the registrational trial and necessitating the use of the most appropriate ITC methods. Policy-driven PICOs should explicitly consider these factors and focus on the most relevant comparators rather than applying an overly inclusive selection. A ranking or weighting of these PICOs could help decision-makers appraise the evidence based on the most relevant PICOs. When PICOs are policy driven instead of evidence driven, it can lead to challenges in finding sound evidence to support them. This means the available evidence may not fully back some PICOs, so it may be considered insufficient. This increases uncertainties and underscores the importance of aligning PICOs with the evidence base to ensure they are realistic and well supported.

Although HTDs have no direct involvement in the PICO scoping process, steps can be taken by HTDs before and after this process to maintain an open dialogue with local jurisdictions to allow for the best preparation of evidence packages. More definitive direction from national HTA agencies on how they regard ITCs in their appraisal/decision-making is integral for the practical application of enabling HTDs to appropriately prepare for ITCs, incorporate them into JCA or into local HTA submissions, and design studies that will meet these expectations. Any such guidance would be a starting point that would need to be regularly updated to keep pace with new ITC methods and changes in how indirect evidence is assessed by HTA decision-makers. The recently published guidance on direct and indirect evidence by the HTA Coordination Group includes novel and complex ITC methods [38, 39]. It is unclear how national HTA authorities and decision-makers will handle these methods in practice, owing to either limited familiarity or differing opinions on their role/suitability to inform decision-making. Although EU member states must give the EU JCA ‘due consideration’, national HTA agencies may also conduct complementary clinical assessments (including ITCs) that consider a different PICO and/or methodology [2, 42], ultimately leading to a conclusion that may differ from the EU JCA. The EU JCA report should include pragmatic and clear statements regarding the situational suitability of the conducted ITC method to enable more informed decision-making at the national level.

Member states should remain responsible for drawing conclusions at the national level on the clinical added value of a health technology, as such conclusions depend on the specific local healthcare context in any given member state, and on the relevance of individual analyses included in the JCA report (e.g. several comparators could be included in the JCA report, of which only a selection is relevant to a given member state). In line with the HTA regulation [3], the JCA report should include a description of the relative effects observed for the health outcomes analysed, including numerical results and confidence intervals, and an analysis of scientific uncertainty and strengths and limitations of the evidence (e.g. internal and external validity). The JCA should be factual and should not contain any value judgement, ranking of health outcomes, conclusions on the overall benefit or clinical added value of the assessed health technology, any position on the target population in which the health technology should be used, or any position on the place the health technology should have in the therapeutic, diagnostic or preventive strategy.

The intent behind JCA is to contribute to the promotion of innovation; it focuses on a high-quality system that is equitable and efficient for all member states. In practice, however, it will be challenging to balance the European perspective with the individual needs of the member states. The European Federation of Pharmaceutical Industries and Associations issued a statement encouraging local HTA agencies to leverage the scientific expertise of EU JCA assessors/co-assessors, including on the validity of evidence [43]. Consistent reporting in JCAs and greater direction at the EU level on the appropriateness of ITC analyses based on rigorous methodology will potentially increase trust in these results at the national level [43, 44].

JCAs can offer the benefit of having one comprehensive joint scientific assessment of available clinical evidence to address relevant EU PICOs (relative efficacy and relative safety), available in a timely manner shortly after EU approval. Although head-to-head comparisons are valuable and ideal, they are often not feasible when there are many comparators. Discussions with regulatory agencies and expert opinions usually guide the choice of an active trial comparator. Therefore, running trials with just one comparator would not remove the need for ITCs. Scoping exercises have shown that the variety of potential comparisons is far greater than a single head-to-head study could address, highlighting the critical role of ITCs in filling these gaps. The EU JCA will encourage more alignment on the use of ITCs to address multiple PICOs despite data limitations and can highlight the value of indirect methods to inform national decision-making. The inclusion of clear statements on appropriateness of ITCs in JCA reports will contribute to increased transparency, thereby increasing confidence in the results based on ITC analyses among member states.

This review did not reveal any association between the acceptability of ITC methods in the EU JCA report and the outcome of the appraisal (i.e. ‘acceptable’ did not equal a positive appraisal). Although the acceptability of the ITC analyses was overwhelmingly (96%) categorised by EUnetHTA authors as ‘unclear’, evidence from these comparisons were still included in the final reports to be used as part of the evidence for national decision-making. This underscores the value of data derived from indirect evidence.

Future research is needed to support the transferability of scientific expertise at the EU JCA level to implementation at the member state level. Data gaps in submission packages are unavoidable, as demonstrated by the current review (the largest category of all criticisms, at 37%), leaving national HTA agencies to strike a balance between rigorous methodological review and practical decision-making based on the best available evidence. Additional guidance is needed on the situational suitability and subsequent acceptability of ITC methods given data availability and the uptake of more complex analytical methods by HTA agencies with different needs.

The current study followed rigorous methodology in the systematic review and extraction of data. However, the study should be viewed considering the following limitations. Although EUnetHTA completed 23 REAs in preparation for the start of the EU JCA, this does not necessarily exactly translate into how ITCs will be assessed in actual EU JCAs. In addition, none of the ITCs were categorised as unsuitable. Information was limited on how the ITC influenced the assessment of benefit at the national level; therefore, categorisations were subject to author interpretation of the EUnetHTA critiques. Some ITCs currently in the unclear group could be deemed to be categorised as unsuitable; however, in the current study the ‘unclear’ category was only selected in the absence of language in the reports that indicated a clear rejection of the ITC (e.g. ‘not appropriate’, ‘not adequate’ or ‘incorrect’). In Macabeo et al., 2024 [10], the authors noted that ‘the categories of criticisms were limited, due to unclear reporting across HTA agencies resulting in overlap between the categories provided’. Likewise, in the current paper, categorisation was reliant on clear language in the EUnetHTA REAs. Greater transparency is needed in HTA reports to more accurately assess these decisions.

In the last 10 years, EUnetHTA has issued 18 guidance documents referencing ITCs, only four of which are considered comprehensive [45]. It should be noted that the ITC guidelines have evolved over time. However, this analysis was unable to assess how changes in methodological rigor or the incorporation of updated best practices impacted the EUnetHTA authors’ description of ITC limitations.