Study characteristics

A total of 6950 results were initially obtained from the search. An additional 153 articles were identified by reviewing BRFSS reference lists [17]. A flow diagram documenting the search process and reasons for excluding studies is shown in Fig. 1. Of the 7103 articles, 2339 were duplicates and were excluded before the abstract/title review. After reviewing 4764 abstracts/titles, 3744 articles were excluded. Of the 1020 articles, a full-text review of the first 10% of articles demonstrated an almost perfect inter-rater reliability agreement between reviewers on which articles met the inclusion criteria (Gwet’s AC: 0.8517 (0.8000–1.0000)). Following review of the full article text, 899 articles were removed. The key reasons for excluding the articles were because they [1] did not conduct validity testing (n = 874), [2] were conducted outside the United States (n = 1105), or [3] were focused on topics other than substance use (n = 878). For this review, a total of 46 articles met the inclusion criteria (Fig. 1). The characteristics of those 46 selected studies are presented in Table 1.

Fig. 1

Flow chart for the selection of studies*. *Studies could have been excluded for multiple reasons

Table 1 Characteristics of included studies of validation testing

The included studies were published between 1979 and 2021, with a wide variation in demographic characteristics. Of the 46 studies, seven had over 80% male participants (Han et al., 2017; Peters et al., 2000; Tiet et al., 2016; Tiet et al., 2019; Tiet et al., 2015; Tiet et al., 2017; Zanis et al., 1994). Among these studies, two recruited only male participants (Peters et al., 2000; Zanis et al., 1994). Additionally, there was one study that only recruited female participants [22]. Racial and ethnic differences also existed among these study samples. Six studies had study sample of primarily (70% or more) White participants [41, 58], and three studies had a study population sample of 70% or more Black/African American (AA) participants [32, 42, 58]. Furthermore, 12 studies only recruited White and Black/AA participants [6, 18, 21, 24, 25, 31, 32, 36, 39, 41, 58, 59]. Seven studies (15.2%) did not report information on race/ethnicity characteristics [19, 22, 27, 34, 55, 57].

All 44 studies included in this review reported the final sample size, with a mean of 1427 (median = 449) participants with an overall range of 23–10,167 participants. Only 13 studies reported response rate, and the response rates ranged between 13.4 [18] and 100% [32]. Twenty-six studies reported the survey duration, and it ranged from 1 month [32] to 120 months [20], with mean 28.48 months (median 13 months). Moreover, studies reported the mean age of the participants as < 30 years (n = 4), between 30 and 39 years (n = 16), and ≥ 40 years (n = 18). Another eight studies reported age groups or median age of the study population. Additionally, a majority (n = 37) of the studies were conducted in non-population-based clinical settings (e.g., inpatient, outpatient).

Participant recruitment strategy

The participant recruitment strategies from included studies in this review were shown in Table 2. Of the 46 studies, only 4% (n = 2) examined SUD in the general population [20, 28]; the rest (n = 44) of the studies were conducted in clinical or other population subgroups. In the first population-based study, 6664 adult Medicaid enrollees were recruited from 1 of 7 Florida regions who took part in the Florida Health Services Survey at least once between 1998 and 2008 [20]. Researchers assessed the internal psychometric properties of the Simple Screening Instrument for Substance Abuse (SSI-SA) but did not compare survey responses with SUD diagnoses in Medicaid clinical records. In the second population-based study, participants were selected from the National Epidemiologic Survey on Alcohol and Related Conditions-III (NESARC-III) sample, which included noninstitutionalized US adult residents (aged 18 years or older) [28]. The authors then selected 777 respondents for the procedural validity study and used a test–retest design to compare concordance of respondents’ answers to the NESARC-III survey questions with a semi-structured interview, the Psychiatric Research Interview for Substance and Mental Disorders, DSM-5 version (PRISM-5), administered by a clinician.

Table 2 Participant recruitment strategies

Of the remaining 44 studies not in the general population, over three quarters (n = 35) were conducted in the clinical setting, with the majority (n = 19) in the inpatient setting [6, 19, 21, 22, 24, 26, 32, 35,36,37,38,39, 42, 46, 48,49,50, 59]. Eleven studies were conducted in the outpatient clinical setting [5, 23, 31, 40, 43, 47, 51,52,53,54,55], 5 studies were conducted in both the inpatient and outpatient settings (Harris et al., 2015; Hser et al., 1999; Kellogg et al., 2002; Kupetz et al., 1979; Salyers et al., 2000), and 1 study was conducted in a Veterans’ Administration shelter [58]. The remaining studies (n = 8) were conducted outside the clinical setting. For example, participants were recruited from an alcohol and drug program [18], prison substance abuse treatment programs [25], Holiday Transfer Facility [41], and a novel jail-release program [56]. Lastly, four studies consisted of sub-population samples within the National Epidemiologic Survey on Alcohol and Related Conditions-III (NESARC-III) [29] and universities using student participants [45, 57].

Quality of studies

Risk of bias was assessed based upon the methodology used for instrument comparison and the statistical analysis conducted. Although several studies adopted recruitment strategies that limited their study population to specific groups (for example, only recruiting male or white populations), the risk-of-bias assessment employed by the current study did not account for recruitment. As a result, most of the included studies (n = 41) had a risk-of-bias score of 4 or higher (Table 1). Two studies had a score of 3 [30, 34], one studies had a score of 2 [29, 48], and two studies had a score of only 1 [27, 44]. Among those studies with low-quality assessment scores, four studies lacked statistical comparisons and reported prevalence estimates only [27, 30, 34, 44]. There were three studies that did not report on validation methodology [27, 44, 48].

Survey measure

Among the articles included in this review, 89% (n = 41) used measures specifically designed for screening SUDs. For example, seven studies tested the validity of the measure’s ability to screen for a specific SUD, including marijuana use [18, 21, 29, 40, 45], cocaine use [30], and opioid use [56]. Five studies validated measures for both substance use and mental health [23, 25, 40, 44, 54], of which one study used a measure for post-traumatic stress disorder (PTSD) screening [54]. The rest of the included studies did not specify a specific SUD for screening purposes but used a generic term for defining SUD. All measures and their frequency of use in the included studies are depicted in Fig. 2.

Fig. 2

Frequency of survey measures used in included studies. Abbreviations in order: Texas Christian University Drug Screen (TCUDS), Substance Use and Abuse Survey (SUAS), the Simple Screening Instrument for Substance Abuse (SSI-SA), the Simple Screening Instrument (SSI), screen of drug use (SoDU), single-item screening questions (SISQs), Substance Dependence Severity Scale (SDSS), Substance Abuse Subtle Screening Inventory-2 (SASSI-2), Rapid Opioid Dependence Screen (RODS), Personality Assessment Inventory Drug Problem Scale (PAI DRG), National Epidemiologic Survey on Alcohol and Related Conditions (NESARC), the Marijuana Screening Inventory (MSI-X), the Longitudinal Substance Use Recall Instrument Recall for 12 Weeks instrument (LSUR-12), the Longitudinal Substance Use Recall Instrument (LSUR), Lifetime Severity Index for Cocaine Use Disorder (LSI-Cocaine), Healthcare Effectiveness Data and Information Set (HEDIS), the Drug Use Screening Inventory (DUSI), Dartmouth Assessment of Lifestyle Instrument (DALI), Cut down, Annoyed, Guilty, and Eye-Opener Substance Abuse Screening Tool (CAGE), the Alcohol Use Disorder and Associated Disabilities Interview Schedule (AUDADIS), Alcohol, Smoking, and Substance Involvement Screening Test-Drug (ASSIST-Drug), Parents, Partners, Past, and Pregnancy Plus (4P’s Plus), tobacco, alcohol, prescription medication, and other substance use (TAPS tool), Substance Use Brief Screen (SUBS), single question used from short inventory of problems-drug use (SIP-DU), Drug Abuse Screening Test (DAST), the Chemical Use, Abuse, and Dependence (CUAD), Addiction Severity Index (ASI)

The majority of studies validated one single measure, of which five studies validated the Addiction Severity Index (ASI), [32, 35, 41, 58, 59] and one study validated drug use subscales of ASI [26, 37]. Five studies validated multiple survey measures:

(1)

Duncan et al. validated two survey measures: (a) CJDAT Co-Occurring Disorders Screening Instruments for any Mental Disorder (CODSI-MD) and (b) CJDAT Co-Occurring Disorders Screening Instruments for Severe Mental Disorder (CODSI-SMD) [25].

(2)

Ramsay et al. also validated two different survey measures: (a) The Lifetime Substance Use Recall Instrument (LSUR) and (b) the Longitudinal Substance Use Recall for 12 Weeks instrument (LSUR-12) [