3.1 Feasibility

An approved, already in market locally advanced and mBC treatment was selected as the study treatment test case for this collaboration, because there were sufficient follow-up data, an established electronic health registry, meaningful outcomes in the timescale available and a large enough patient population in Wales to avoid deductive disclosure of patient identities.

3.2 Patient Population

The inclusion criteria were defined through an iterative process that considered clinical input and data feasibility: human epidermal growth factor receptor 2 negative (HER2−), oestrogen receptor (ER+), aged 18 or older, locally advanced or metastatic breast cancer (including T3+ or N1+ or M0/M1 from the Tumour, Node, Metastasis staging system [37]) and non-operable breast cancer (when surgery was not completed prior to treatment). There were no exclusions based on sex.

This inclusion criteria resulted in a patient sample size of n = 99 for the total population in the treatment cohort (2017–2020) and n = 286 in the control cohort (2014–2016) (Fig. 1). A sizable sample of patients met the inclusion/exclusion criteria from 2017 to 2020 but were excluded from the analysis, because they were not prescribed the study treatment (n = 311). Additional demographic details of the treatment and control cohorts are available in Table 1.

Fig. 1

Patient sample by inclusion/exclusion, identifying the final cohorts (control, treatment, and eligible population not prescribed the medication). M1 = metastatic disease at inclusion

Table 1 Demographic characteristics of treatment and control cohorts at baseline, 2014–2020

The number of patients with metastases (M1) versus locally advanced disease (M0) differed considerably between the treatment cohort (92% metastatic disease) and control cohort (31% metastatic disease). This suggested a higher severity of the disease in the study treatment population. As 92% of the study population had M1 at inclusion, this subgroup was used for modelling of the primary scenario to improve comparability of results across the two cohorts (i.e. M0 patients were not included in the primary volatility analysis). Staging of patients was determined by the staging tables within ChemoCare™ or CaNISC at diagnosis and not split into histopathologic or clinical assessment.

3.3 Outcomes

A pragmatic literature review identified 47 sources of outcomes for mBC [e.g. International Consortium for Health Outcomes Measurement (ICHOM), the Welsh Cancer Network]. Within these sources, a long list of 219 health outcomes relevant to metastatic breast cancer were identified. Following two workshops and a voting exercise, clinicians selected a short-list of the top 57 outcomes of importance for patient long-term health and health-related quality of life. The feasibility of inclusion of these variables for use in an OBA was considered in two ways: (i) multiple outcomes that measured the same domain were reduced to a single variable via clinician input and (ii) variables were assessed to determine their likely availability in the Welsh data environment. The ten outcomes that were considered to be feasible for inclusion were (1) days disrupted by care, (2) intolerance to treatment (deferral and discontinuance), (3) 30 day mortality, (4) 1 year survival, (5) progression free survival (PFS), (6) spinal cord compression (SCC) incidence, (7) severe bowel symptoms, (8) symptom control in palliative care, (9) pain management, and (10) treatment response (Table 2, definitions provided). The variable “intolerance to treatment” was considered in two ways: discontinuance and deferral.

Table 2 Shortlisted outcomes, definitions, and reason for exclusion

Once the linked datasets became available in UK SeRP, outcomes were reviewed for data availability during the study period, missingness, type of data (e.g. free text) and low incidence (risk of deductive disclosure of patients). In total, seven of the remaining variables were excluded (as well as intolerance to treatment: discontinuance) (Table 2).

The final set of three outcome variables for use in the contracting exercise was identified: (i) 1-year survival, (ii) days disrupted by care and (iii) intolerance to treatment-deferral. “One-year survival” measured the number of patients who survived one year after diagnosis. “Days disrupted by care” used inpatient, outpatient and accident and emergency (A and E) data to identify planned and unplanned admissions to secondary care. “Intolerance to treatment-deferral” calculated the percent of treatment cycles delayed or stopped due to intolerance of treatment. The outcome selection process was carried out iteratively and refined over a few months.

Outcome data are reported for both treatment and control cohorts, including exploratory sub-group analysis by metastases (Table 3). The survival rate at 1 year was higher in the treatment cohort (total: 91.9%, M0: 100%, M1: 91.3%) than in the control cohort (total: 83.8%, M0: 92%, M1: 64.7%). The intolerance to treatment was lower in the treatment cohort (total: 5%, M0: 2%, M1: 5%) than the control cohort (total: 8%, M0: 11%, M1: 7%), and the total days disrupted by care per year was lower in the treatment cohort (total: 19.97, M0: 15.2, M1: 20.5) than the control cohort (total: 21.9, M0: 17.4, M1: 30.2).

Table 3 Outcome data by control and treatment cohort (2018–2020), including M0 and M1 sub-groups3.4 Contract Design, Modelling and Implementation3.4.1 Contract Scenarios

During the interdisciplinary workshops, three contracting scenarios were agreed: (1) one scenario including all three outcomes (1-year survival, days disrupted by care and intolerance to treatment), (2) a second scenario including mortality and morbidity (1-year survival and days disrupted by care), and (3) a third scenario with mortality alone (1-year survival). Scenario 1 was deemed the primary OBA scenario (Fig. 2e) and is presented in the results; the other two scenario outputs are reported in Supplementary Fig. 3.

Fig. 2

OBA payment by outcome using a population-based, linear payment model with M1 treatment cohort, including volatility analysis. (a) All outcomes (2017–2020), (b) 1-year survival (2018–2020), (c) intolerance to treatment (2018–2020), (d) days disrupted by care (2018–2020), (e) all outcomes = primary scenario (2018–2020). OBA outcome-based agreement, M1 metastatic disease at inclusion, P Percentile

3.4.2 Contract Parameters

Contract design parameters for the primary OBA scenario are reported in Table 4. Per patient limits were included despite the overall contract being measured on a population basis to prevent distortion in averages due to outliers in the cohorts. A payment cap was incorporated for the 1-year survival outcome to ensure performance could not result in greater than 100% payment contribution. The other outcomes in the primary OBA scenario (days disrupted by care and intolerance to treatment) were not capped. This meant that if the performance measure was higher than the target—for example, in such cases where outcomes are better than expected—the payment is calculated using linear interpolation allowing bonus payments beyond the base 100% agreed payment.

Table 4 Contract design parameters for primary OBA scenario (2018–2020)

We defined the minimum outcome level for the 1-year survival outcome as the 1-year survival rate of the placebo population used in the HTA cost effectiveness model [38] with a 5% marginal reduction. We defined the maximum outcome level for the 1-year survival outcome as the 1-year survival rate observed for the mBC study population plus a 5% margin. The 5% margin is required to allow variation between the minimum and maximum outcome levels due to variation in the confidence intervals, as a narrower margin would increase payment volatility. We defined the benchmark for days disrupted by care as the adjusted days disrupted of the comparator cohort in the first 4 years since diagnosis and weighted by the time since diagnosis of the study population. The target was defined as 10 days less than the benchmark (i.e. the difference between the study and comparator population in 2014–2020). The intolerance to treatment benchmark was 15% while the target was set to the minimum possible (0%) based on discussions with clinicians. Exploratory sensitivity analyses of the benchmarks and targets for each outcome showed that narrowing the benchmark to target gap made the target more difficult to achieve, adversely affecting total contract value.

3.4.3 Contract Payment3.4.3.1 Median Percent Contribution

Since medicine pricing was not included in the OBA parameters, total contract value was modelled to understand the proportion of total payment that would be made for the primary scenario based on performance of the outcomes (Table 5). Each outcome’s performance translates to a percent contribution relative to the total contract value (i.e. each of the three outcomes could contribute a maximum of 33% to the total contract over three years; contract parameters in Table 4). Therefore, the sum of the “1-year survival”, “intolerance to treatment” and “days disrupted by care” contributions equal the percent of the maximum contract value that could be paid (when considered in relation to the total possible OBA payment if all outcomes performed at 100%).

Table 5 Outcome measurement translated to financial performance (median contribution to total OBA payment by outcome per year), 2017–2020

While target population data from 2017 to 2020 and comparator data from 2014 to 2020 were used to inform benchmarks and targets, significant variability was observed in the contribution to payment when including 2017 within the contract period. This was due to low contract volumes (nine actual patients within the contract but not all were eligible for outcomes) (Fig. 2a). Additionally, no patients were eligible for the 1-year survival outcome until 2018, so it was determined most appropriate to only consider 2018–2020 data for payment and volatility analysis.

“One year survival” demonstrated the volatility of a binary outcome with narrow benchmark and target ranges. In 2019 the observed performance was 24%, despite 100% and 87% contributions in 2018 and 2020 respectively due to high expected survival for patients with the disease, irrespective of receiving the study treatment, and small patient numbers. “Intolerance to treatment” showed the most consistent performance across 2018–2020 ranging from 69–85% median contribution to payment per year. The contract design did not include a patient cap for “days disrupted by care” which is relevant as the treatment cohort did outperform in all years except 2018 (Table 5; over 100% 2017 to 2020, except 87% in 2018).

When measuring the performance of all outcomes together, the overall payment due remains less than 100% for all years (92% in 2018, 61% in 2019 and 87% in 2020) (Fig. 2e, M1 population). When comparing the OBA model with the total population (M0 and M1), not only does the overall contract perform less well, but there is also greater volatility (Fig. 3c).

Fig. 3

Exploratory analyses of OBA payment (2018–2020), including volatility analysis. a Using a per-patient, threshold payment model for 1-year survival only with M1 treatment cohort, b using a population-based, threshold payment model with M1 treatment cohort, c using a population-based, linear payment model with the total (M0 + M1) treatment cohort. OBA outcome-based agreement, M0 no metastatic disease at inclusion, M1 metastatic disease at inclusion, P Percentile

3.4.3.2 Type of Outcome and Volatility Analysis

To understand the impact of each individual outcome on the total OBA contract output (payment) between 2018 and 2020 and how different types of outcomes behave (binary or continuous), each of the contract outcomes was assessed separately before being considered in the same model (Fig. 2).

The binary outcome, 1-year survival, showed payment contributions with a dramatic range, which resulted in very high volatility and higher financial risk (Fig. 2b). In 2018, the performance of the outcome is high but there was small contract volume. This compared well with 2019 when there was a decrease in outcome performance (due to the lower survival rate observed) and a peak in contract volume. In 2020, performance improves and although contract volume decreases it remains sizeable.

In comparison, the continuous variables, i.e. days disrupted by care and intolerance to treatment, showed a tighter range around the median payment contribution and, as such, lower volatility (Fig. 2c, d) and lower financial risk. Of the three outcomes included in this primary scenario, intolerance to treatment (Fig. 2c) has the narrowest overall payment band around the median proportion of payment, and days disrupted by care (Fig. 2d) shows a moderate payment band widened by overperformance in most years. These outcomes thus demonstrated less volatility than one year survival, with intolerance to treatment showing the most stable performance of the three in the scenario. Combined in the same model (Fig. 2e) the outcomes demonstrate the impact of the binary variable’s volatility offset by the less volatile continuous variables.

3.4.3.3 Outcome Measurement Level and Volatility Analysis

Both population and per patient calculations for the intolerance to treatment and days disrupted by care outcome should result in similar payment contributions. As 1-year survival on a per patient basis becomes a threshold payment model rather than linear, payment contributions would be made at 100% if a patient survives and 0% if they die. This results in a much narrower band across 2019 and 2020 (2018 will still remain at 100%) (Fig. 3a).

3.4.3.4 Payment Model and Volatility Analysis

While a linear payment model was used to calculate the output of the primary OBA scenario and investigate the impact of each outcome on the overall proportion of payment (Fig. 2e), a threshold payment model was developed for comparison (see Fig. 3b). The threshold model resulted in sharp contraction of the total contract payment based on lower outcome performance and contribution by year; this is paired with considerably higher volatility than the linear model for the primary scenario.