This was a retrospective observer performance study using digital mammogram reader data from BreastScreen Reader Assessment Strategy (BREAST) platform between 2020 and 2023 and received Human Research Ethics Committee (HREC) approval from the University of Sydney (2019/013). The two comparison groups that participated in this observational study were Australian breast surgical trainees and general radiology trainees, who each completed the same mammographic test set on the BREAST platform, which contained 20 mammogram cases consisting of 6 breast cancer cases. The test set consisted of 3 low BD cases (levels A and B) and 3 high BD (levels C and D) cancer cases plus 14 cancer-free cases (5 low BD and 9 high BD).

The cancer-free mammogram cases were confirmed by at least two senior radiologists with consensus reads in two rounds of normal screening, while the cancer cases were confirmed by breast tissue biopsy. Each case in the test set contained two mammographic views, a cranio-caudal (CC) and mediolateral oblique (MLO) for each side. The test set did not include mammogram cases with post-biopsy markers, surgical clips, or scars. The Cancer Institute New South Wales (CINSW) waived the need for obtaining informed consent from the patients whose anonymised mammograms were used in the test set. All reading examinations were performed after obtaining informed consent from participants.

Recruitment of surgical trainees was conducted by the unit coordinator of a Breast Surgery unit of study at the University of Sydney (as part of the Graduate Certificate of Breast Surgery (GCBS)), and test set readings of surgical trainees were conducted between 2020 and 2023. The data sample of radiology trainees was collected through the BREAST programme with data collection between 2022 and 2023. Recruitment of radiology trainees and surgical trainees was a voluntary process and yielded de-identified data. Radiology trainees include readers registered for reading the BREAST test set at their clinic or at the BREAST workshop which has similar reading environment.

The sample size included 18 surgical trainees (10 with <3 months experience and 8 with ≥3 months experience) and 32 radiology trainees (28 with <3 months experience and 4 with ≥3 months experience) who completed reading the same cases between 2022 and 2023. Each reader’s information was collected through an online demographic questionnaire embedded in the BREAST platform, including their current role, medical specialty (breast or other specialisation), time in current role (average 5 years for surgical trainees and average 3 years for radiology trainees), years reading mammograms (average 11 months for surgical trainees and average 2 months for radiology trainees), number of mammographic cases read per week, number of hours reading per week and completion of breast fellowship. All data were collected with de-identified ID numbers.

The participants were asked to read the mammograms in full resolution on diagnostic monitors and were required to localise all suspected breast lesions for each mammogram case and rate each lesion in accordance with the RANZCR Imaging Classification: 1—no significant abnormality (no marking on the mammogram), 2—benign, 3—indeterminate/equivocal, 4—suspicious, or 5—malignant. The participants’ responses were recorded by the online BREAST platform.

The performances of participants were calculated in term of case sensitivity, specificity, location sensitivity, receiver operating characteristics (ROC) area under curve (AUC) and jackknife alternative free response receiver operating characteristics (JAFROC). Case sensitivity is a true positive rate measuring the proportion of cancer cases that were correctly marked with a positive rating (3, 4, or 5). Specificity measures the free-cancer cases that were correctly identified cancer-free (rating 1 or 2). Location sensitivity refers to the proportion of each cancer lesion that was correctly localised and identified. It assesses the ability of readers to accurately localise each breast lesion. ROC AUC evaluates the readers ability to detect cancer cases and identify normal cases, while JAFROC considers the performances of the readers in cancer localization in association with normal reporting and their ratings [12]. The participants were given access to their scores and the answers of each case after completing the test set in the BREAST platform.

The performance metrics of participants were analysed, first through an overall comparison between the surgical trainees and radiology trainees on all cases. The performance of surgical trainees was then compared to the performance of radiology trainees on low-density mammograms and high-density mammograms separately. Following this, the performance of surgical trainees was compared to the performance of radiology trainees on for trainee groups with <3 months experience and ≥3 months experience, separately. Finally, the performances of surgical and radiology trainees were compared by simultaneously stratifying the breast density levels, trainee experience and mammogram density.

The groups were compared in terms of specificity, sensitivity, lesion sensitivity, ROC and JAFROC values, with lesion sensitivity determined by a localisation within the radius of the true cancer lesion which was recorded based on the radiology and pathological reports. All cancer cases had only one actionable cancer. XXX statistical tests were used to for the comparisons. All statistical analyses were conducted with SPSS, and P value < 0.05 was considered significant.