Advertisement

Interobserver variability among expert readers quantifying plaque volume and plaque characteristics on coronary CT angiography: a CLARIFY trial sub-study

Open AccessPublished:August 15, 2022DOI:https://doi.org/10.1016/j.clinimag.2022.08.005

      Highlights

      • There is significant interobserver variability among expert readers when quantifying plaque composition
      • Interobserver variability increases among expert readers when attempting to identify high risk plaque features
      • There is high discordance between expert readers and AI guided QCT when characterizing plaque components, especially high risk plaque features

      Abstract

      Background

      The difference between expert level (L3) reader and artificial intelligence (AI) performance for quantifying coronary plaque and plaque components is unknown.

      Objective

      This study evaluates the interobserver variability among expert readers for quantifying the volume of coronary plaque and plaque components on coronary computed tomographic angiography (CCTA) using an artificial intelligence enabled quantitative CCTA analysis software as a reference (AI-QCT).

      Methods

      This study uses CCTA imaging obtained from 232 patients enrolled in the CLARIFY (CT EvaLuation by ARtificial Intelligence For Atherosclerosis, Stenosis and Vascular MorphologY) study. Readers quantified overall plaque volume and the % breakdown of noncalcified plaque (NCP) and calcified plaque (CP) on a per vessel basis. Readers categorized high risk plaque (HRP) based on the presence of low-attenuation-noncalcified plaque (LA-NCP) and positive remodeling (PR; ≥1.10). All CCTAs were analyzed by an FDA-cleared software service that performs AI-driven plaque characterization and quantification (AI-QCT) for comparison to L3 readers. Reader generated analyses were compared among readers and to AI-QCT generated analyses.

      Results

      When evaluating plaque volume on a per vessel basis, expert readers achieved moderate to high interobserver consistency with an intra-class correlation coefficient of 0.78 for a single reader score and 0.91 for mean scores. There was a moderate trend between readers 1, 2, and 3 and AI with spearman coefficients of 0.70, 0.68 and 0.74, respectively. There was high discordance between readers and AI plaque component analyses. When quantifying %NCP v. %CP, readers 1, 2, and 3 achieved a weighted kappa coefficient of 0.23, 0.34 and 0.24, respectively, compared to AI with a spearman coefficient of 0.38, 0.51, and 0.60, respectively. The intra-class correlation coefficient among readers for plaque composition assessment was 0.68. With respect to HRP, readers 1, 2, and 3 achieved a weighted kappa coefficient of 0.22, 0.26, and 0.17, respectively, and a spearman coefficient of 0.36, 0.35, and 0.44, respectively.

      Conclusion

      Expert readers performed moderately well quantifying total plaque volumes with high consistency. However, there was both significant interobserver variability and high discordance with AI-QCT when quantifying plaque composition.

      Abbreviations:

      QCT (Quantitative Coronary Computed Tomographic Angiography), CCTA (Coronary computed tomography angiography), LA-NCP (Low Attenuation Non-Calcified Plaque), HRP (High Risk Plaque), FDA (Food and Drug Administration), AI (Artificial Intelligence), CAD (Coronary Artery Disease), MDCT (Multidetector Computed Tomography), LM (Left Main), LAD (Left Anterior Descending), LCx (Left Circumflex), RCA (Right Coronary Artery), CP (Calcified Plaque), NCP (Noncalcified Plaque), QCA (quantitative coronary angiography), L3 (Level 3), PV (plaque volume), PAV (percent atheroma volume), APCs (atherosclerotic plaque characteristics), HU (Hounsfield units)

      Keywords

      1. Background

      Coronary computed tomography angiography (CCTA) is currently indicated for imaging symptomatic patients at low to intermediate risk for coronary artery disease (CAD) due to its ability to accurately rule out severe stenosis.
      • Narula J.
      • Chandrashekhar Y.
      • Ahmadi A.
      • et al.
      SCCT 2021 expert consensus document on coronary computed tomographic angiography: a report of the Society of Cardiovascular Computed Tomography.
      However, CCTA has a wider scope of capabilities including quantifying plaque volume and plaque characteristics, which can subsequently help clinicians identify CAD plaque burden, prognosticate disease progression, and risk stratify patients for future cardiovascular events.
      • Ferencik M.
      • Mayrhofer T.
      • Bittner D.O.
      • et al.
      Use of high-risk coronary atherosclerotic plaque detection for risk stratification of patients with stable chest pain: a secondary analysis of the PROMISE randomized clinical trial.
      • Stuijfzand W.J.
      • van Rosendael A.R.
      • Lin F.Y.
      • et al.
      Stress myocardial perfusion imaging vs coronary computed tomographic angiography for diagnosis of invasive vessel-specific coronary physiology: predictive modeling results from the computed tomographic evaluation of atherosclerotic determinants of myocardial ischemia (CREDENCE) trial.
      • Bittencourt M.S.
      • Hulten E.A.
      • Murthy V.L.
      • et al.
      Clinical outcomes after evaluation of stable chest pain by coronary computed tomographic angiography versus usual care: a meta-analysis.
      Specifically, as the prognostic nature of plaque burden and high-risk plaque characteristics (HRP), is substantiated in the literature, it underscores the need to assess our capability for accurately and consistently assessing these markers.
      • Motoyama S.
      • Ito H.
      • Sarai M.
      • et al.
      Plaque characterization by coronary computed tomography angiography and the likelihood of acute coronary events in mid-term follow-up.
      • Williams M.C.
      • Moss A.J.
      • Dweck M.
      • et al.
      Coronary artery plaque characteristics associated with adverse outcomes in the SCOT-HEART study.
      • Motoyama S.
      • Sarai M.
      • Harigaya H.
      • et al.
      Computed tomographic angiography characteristics of atherosclerotic plaques subsequently resulting in acute coronary syndrome.
      Most of the work published on interobserver variability within CCTA analysis focuses on identifying and quantifying degrees of plaque stenosis.
      • Pagali S.R.
      • Madaj P.
      • Gupta M.
      • et al.
      Interobserver variations of plaque severity score and segment stenosis score in coronary arteries using 64 slice multidetector computed tomography: a substudy of the ACCURACY trial.
      • Choudhary G.
      • Atalay M.K.
      • Ritter N.
      • et al.
      Interobserver reliability in the assessment of coronary stenoses by multidetector computed tomography.
      • Kerl J.M.
      • Schoepf U.J.
      • Bauer R.W.
      • et al.
      64-Slice multidetector-row computed tomography in the diagnosis of coronary artery disease: interobserver agreement among radiologists with varied levels of experience on a per-patient and per-segment basis.
      • Maroules C.D.
      • Hamilton-Craig C.
      • Branch K.
      • et al.
      Coronary artery disease reporting and data system (CAD-RADS(TM)): inter-observer agreement for assessment categories and modifiers.
      However, the application of artificial intelligence enabled quantitative CCTA evaluation (AI-QCT) has advanced our ability to make plaque quantification and characterization assessments both accurately and quickly compared to human readers.
      • Choi A.D.
      • Marques H.
      • Kumar V.
      • et al.
      CT EvaLuation by ARtificial Intelligence For Atherosclerosis, Stenosis and Vascular MorphologY (CLARIFY): A Multi-center, international study.
      The diagnostic benefit of quantifying high-risk plaque components and the quickly expanding capabilities of applied AI makes the baseline interobserver variability for quantifying plaque components, and how that performance compares to AI, relevant questions that are unanswered by the current literature. This is the first study to assess expert interobserver performance for categorizing both plaque burden and plaque composition against a validated AI reference.

      2. Methods

      2.1 Study enrollment and design

      This is a retrospective study evaluating the imaging data of 232 patients enrolled in the CLARIFY (CT EvaLuation by ARtificial Intelligence For Atherosclerosis, Stenosis and Vascular MorphologY) trials,
      • Choi A.D.
      • Marques H.
      • Kumar V.
      • et al.
      CT EvaLuation by ARtificial Intelligence For Atherosclerosis, Stenosis and Vascular MorphologY (CLARIFY): A Multi-center, international study.
      • Griffin W.F.
      • Choi A.D.
      • Riess J.
      • et al.
      AI evaluation of coronary stenosis on CT coronary angiography, comparison with quantitative coronary angiography and fractional flow reserve; a CREDENCE trial sub-study.
      which identified consecutive patients undergoing CCTA for acute and stable chest pain at high volume centers for cardiac CT. Parameters including age, gender, body mass index (BMI), hypertension, hyperlipidemia, diabetes, smoking history, family history of coronary artery disease (CAD), statin use, antiplatelet therapy, and use of beta-blockers were all collected on enrollment and subsequently de-identified. The CLARIFY 1 and CLARIFY 2 trials used the data from this cohort to test the performance of AI-QCT against various gold standards for plaque assessment including L3 readers, quantitative coronary angiography (QCA) and fractional flow reserve (FFR), focusing on stenosis grading and plaque quantification.
      • Choi A.D.
      • Marques H.
      • Kumar V.
      • et al.
      CT EvaLuation by ARtificial Intelligence For Atherosclerosis, Stenosis and Vascular MorphologY (CLARIFY): A Multi-center, international study.
      • Griffin W.F.
      • Choi A.D.
      • Riess J.
      • et al.
      AI evaluation of coronary stenosis on CT coronary angiography, comparison with quantitative coronary angiography and fractional flow reserve; a CREDENCE trial sub-study.
      This study assesses the interobserver variability of L3 readers for characterizing plaque and identifying HRP using AI-QCT as a reference.

      2.2 CCTA scan acquisition

      CCTA scans were performed on a 64-MDCT General Electric VCT (General Electric Healthcare, Milwaukee, Wisconsin; CVMG), and a 128-DSCT Siemens FLASH (Siemens Healthcare, Erlangen, Germany; GWU and Hospital de Santa Marta). Acquisition techniques included prospective and retrospective gating based upon institutional protocols. Iterative reconstruction was used on the DSCT scanners but not on the CVMG VCT. Patients received beta blockade, nitroglycerin and iodinated contrast in accordance with institutional and Society of Cardiovascular Computed Tomography guidelines.
      • Abbara S.
      • Blanke P.
      • Maroules C.D.
      • et al.
      SCCT guidelines for the performance and acquisition of coronary computed tomographic angiography: a report of the Society of Cardiovascular Computed Tomography Guidelines Committee: endorsed by the North American Society for Cardiovascular Imaging (NASCI).
      Exams were reconstructed in 5–10% increments. A diagram outlining CCTA exclusions is provided in Fig. A1 of Appendix A.

      2.3 Artificial intelligence segmentation, plaque characterization and quantification

      CCTA studies were uploaded to and analyzed by an FDA-cleared software in a blinded manner providing statistical services as determined and requested by the study investigators.
      • Hwang T.J.
      • Kesselheim A.S.
      • Vokinger K.N.
      Lifecycle regulation of artificial intelligence- and machine learning-based software devices in medicine.
      United States Food and Drug Administration
      Cleerly Labs 510 (k) premarket notification.
      This is an AI-aided approach that performs automated analysis of CCTA using a series of validated convolutional neural network models (including VGG 19 network, 3D U-Net and VGG Network Variant) for image quality assessment, coronary segmentation and labeling, lumen wall evaluation and vessel contour determination and plaque characterization.
      • Singh G.
      • Al'Aref S.J.
      • Van Assen M.
      • et al.
      Machine learning in cardiac CT: basic concepts and contemporary data.
      • Carin L.
      • Pencina M.J.
      On deep learning for medical image analysis.
      A full graphical representation of the algorithm is presented in Appendix B. No manual interaction is required from the reader.
      First, the AI-aided approach leverages 2 deep convolutional neural networks, one to produce a centerline along the length of the vessel, and another for lumen and outer vessel wall contouring. This approach is applied to multiple phases of the CCTA examination, if present, and enables phase-specific evaluation at the coronary segment vessel. The algorithm reviews all series and determines the top 2 optimal series for further analysis including vessel and lumen segmentation, plaque and stenosis quantification. The algorithm rank-orders all available phases for the segmentation of arteries. It then uses the top two phases interactively on a per vessel basis, e.g., the right coronary artery (RCA) will be reconstructed from the phase which yields the highest RCA image quality, while the posterior descending artery (PDA) may come from the second phase if the PDA has a higher image quality on that phase. Once coronary artery segmentation is performed, an automated labeling is done to classify arteries by their location in the coronary tree as well as within the proximal, mid or distal portions in a single vessel. The AI further allows for defining of coronary artery lesions (i.e., those areas where plaque is present). Utilizing a normal proximal reference vessel cross-sectional slide and the cross-sectional slice that demonstrates the greatest absolute narrowing, the software determines the start and end of lesions and drops markers between which it calculates % diameter stenosis and quantifies plaque burden. Within the lesions, plaque is further characterized as low-attenuation non-calcified plaque, non-calcified plaque and calcified plaque based upon Hounsfield unit (HU) densities of <30, 31 to 350, >350, respectively. Positive arterial remodeling was identified as a remodeling index ≥1.10 by diameter when compared to a normal proximal vessel reference.
      We used a coronary artery territory-based analysis which included the left main (LM), left anterior descending (LAD) including diagonals and ramus intermedius, left circumflex (LCx) including obtuse marginals and left-posterior descending and posterolateral branches, and RCA including right posterolateral and PDA. For each territory we recorded vessel length, vessel volume, lumen volume, total plaque volume, calcified plaque volume, noncalcified plaque volume, low density noncalcified plaque volume, maximum diameter and area stenosis, and maximum remodeling index. After the AI algorithm finishes all operations, as mandated by the FDA, a quality control cardiac CT trained technician reviews the results of the AI analysis in all cases with manual adjustment if necessary. The QA process included visual inspection of the lumen and vessel boundaries on the straightened multiplanar reformat views of all vessels 1.5 mm and larger, as well as every cross-section of each of these vessels, placed at contiguous 0.25 mm increments. The time from data upload until AI completed processing was recorded as well as any additional time required for technician or physician quality assurance review.

      2.4 Level 3 expert reads

      Three advanced imaging attending physicians who were L3 readers, ranging from 7 to 17 years of experience, performed blinded assessment of CCTA. Each reader read each case independently and in distinct reading sessions. The readers interpreted the original dataset and chose phases independent of the AI image segmentation to calculate plaque volume using a semi-quantitative software. They subsequently categorized plaque volume and its characteristics including %CP, %NCP, and HRP on a territory basis.

      2.5 Atherosclerotic plaque volume and characterization

      Readers categorized plaque volume into one of five quintiles on a per vessel basis. Similarly, plaque components including %NCP vs. %CP were evaluated on a vessel territory basis and assigned to quintiles 1–5 with 1 signifying a vessel with >90% NCP, 2: %NCP 60–90%, 3: %NCP 40–60%, 4: %NCP 10–40% and 5: %NCP <10%. Readers also recorded presence or absence of high-risk plaque features including positive remodeling (PR) and low attenuation non-calcified plaque (LA-NCP). Readers were provided multiple correlative examples of plaque categories that corresponded to the quantified plaque quintiles described. One example is provided in Fig. 3. A full evaluation of atherosclerotic plaque volume and composition is provided in Table A1 of Appendix A.
      Fig. 1
      Fig. 1Box plots comparing level 3 reader versus AI derived plaque volume measurements
      On a per vessel basis, there were moderately strong trends demonstrated between AI and readers. Intra-quintile distribution increased as plaque volumes increased. Spearman coefficients were 0.70262, 0.68224, and 0.73835 for reader 1, 2, and 3, respectively compared to AI.
      Fig. 2
      Fig. 25 × 5 Contingency table of AI versus reader categorization of NCP and CP percentages by vessel
      On a per vessel basis, there were poor trends demonstrated between AI and readers. Spearman coefficients were 0.37660, 0.50706 and 0.60281 for reader 1, 2, and 3, respectively, with kappa coefficient showing poor agreement at 0.232, 0.337, and 0.239, respectively. Overall, readers tended to overestimate % CP.
      Fig. 3
      Fig. 3An example of coronary vessels with plaque volumes appropriate for each quintile vessels were subjectively categorized into groups 0–4 based on whether there was no plaque present (0), minimal plaque present (1), mild amounts of plaque present (2), moderate amounts of plaque present (3) or severe volumes of plaque present (4). High risk features are present in the final panel. LA-NCP is circled in red, evidence of positive remodeling is circled in blue. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

      2.6 Statistical analysis

      The variability among L3 readers was evaluated using the intra-class correlation coefficient. Correlation between L3 readers and AI was evaluated using Spearman's correlation coefficient while agreement between L3 readers and AI was evaluated with weighted kappa statistic.
      Readers determined presence of two high risk plaque features—low attenuation plaque <30 HU and positive arterial remodeling with a remodeling index ≥1.10 by diameter on a per vessel territory basis. This analysis was then compared with AI. This binary outcome was compared by calculating the percent agreement and kappa statistic.

      3. Results

      3.1 Demographics and analysis time

      The study population consisted of n = 232 patients who were mean age 60 ± 12 years and 37% female. Among the study cohort, 61% had hypertension, 69% had hyperlipidemia, 29% had diabetes and 38% were smokers. The AI analysis time was 9.7 ± 3.2 min. AI analysis plus quality assurance analysis and report generation was 23.7 ± 6.4 min (Table 1).
      Table 1Study Cohort demographics with AI analytic times.
      Demographics and AI analysis data (N = 232)
      VariableN (%)
      Age ± SD60 ± 12 years
      Female sex, mean (%)86 (37)
      Body mass index (BMI) ± SD27.5 ± 6 kg/m2
      Hypertension, n (%)142 (61)
      Hyperlipidemia, n (%)161 (69)
      Diabetes mellitus, n (%)67 (29)
      Smoking, n (%)88 (38)
      Family history of coronary artery disease, n (%)116 (35)
      Statin therapy, n (%)159 (68)
      Antiplatelet therapy, n (%)84 (36)
      Beta-Blocker therapy, n (%)58 (25)
      Coronary artery calcium score, mean ± SD
      Coronary artery calcium score by the Agatston method was available for 147 of the 232 (63%) of patients as one of the sites does not routinely perform non-contrast calcium scoring prior to CCTA.
      150 ± 495 (Range 0–3607)
      AI analysis data
      AI analysis series available, mean ± SD minutes3.6 ± 1.6 (Range 1–10)
      AI analysis time, mean ± SD minutes9.7 ± 3.2
      AI analysis + QA analysis and report generation, mean ± SD minutes23.7 ± 6.4
      a Coronary artery calcium score by the Agatston method was available for 147 of the 232 (63%) of patients as one of the sites does not routinely perform non-contrast calcium scoring prior to CCTA.
      Non-negligible plaque (>3 mm3) was detected by AI in the following distributions: 170/232 (73.2%) in the RCA, 154/228 (67.5%) in the LM, 196/232 (84.5%) in the LAD, and 150/232 (64.7%) in the LCx. Fig. 1 is a box plot illustration reflecting reader performance for quantifying plaque on a per-vessel territory basis compared to those AI reference measurements. The Spearman coefficients for readers 1, 2, and 3 compared to AI were 0.70262, 0.68224, and 0.73835, respectively (Table 2). The intra-class correlation of a single score across readers was 0.77921 with a high intra-class correlation coefficient of 0.91370 for mean scores.
      Table 2Correlation and agreement trends between L3 readers and AI.
      Plaque volume% NCP vs. % CPHRP
      ReaderSpearman correlation coefficientSpearman correlation coefficientWeighted kappa coefficientSpearman correlation coefficientWeighted kappa coefficient
      10.702620.376600.2320.36200.224
      20.682240.507060.3370.35290.261
      30.738350.602810.2390.44210.166

      3.2 Interobserver variability and correlation with AI

      Fig. 2 shows the 5 by 5 tables comparing reader assigned quintiles for % NCP and % CP on a per-vessel territory basis with AI reference ranges. Spearman coefficients for readers 1, 2, and 3 compared to AI were 0.37660, 0.50706, and 0.60281, respectively. The intra-class correlation coefficient for a single score across readers was 0.68, higher than the Spearman coefficient achieved by any individual reader to AI. The weighted kappa coefficient measuring agreement between AI and readers 1, 2, and 3 for quantifying % NCP vs. % CP was 0.232, 0.337, and 0.239, respectively. The Spearman coefficient demonstrating correlation among readers 1, 2, and 3 for quantifying HRP was 0.362, 0.3529, and 0.4421, respectively. Agreement between AI and Readers 1, 2, and 3 for quantifying HRP was poor with weighted kappa coefficients of 0.224, 0.261, and 0.166, respectively (Fig. 4, Fig. 5). A matrix comparing HRP detected by AI versus L3 readers is included in Fig. A2 of Appendix A.
      Fig. 4
      Fig. 4Example of L3 interobserver variability determining % NCP and % CP
      63 year old man with stable chest pain. CCTA with discordant plaque composition was reported by readers. Curved multiplanar reformatted images (CPR) (2 left images) depict a moderate volume of mixed density atherosclerotic plaque. Level 3 readers' estimates of the % non-calcified plaque included 10–40%, 40–60%, and 60–90%. AI-QCT images and analysis (3rd and 4th images from left, the yellow line is outer wall boundary, purple line lumen wall boundary; the color plaque overlay: blue is calcified plaque, yellow noncalcified plaque, red low density noncalcified plaque) revealed that the percent noncalcified plaque was 58%. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
      Fig. 5
      Fig. 5Example of discordance between AI and L3 readers for identifying HRP
      53 year old man with chest pain. This example illustrates a circumstance in which L3 readers and AI disagreed regarding the presence of HRP features (defined as LA-NCP and PR). Panel A shows a frontal view of a vessel that L3 readers identified as having HRP features. Reader perception of LA-NCP is circled in red. PR is circled in blue. Panels B and D show sagittal views of that same vessel with color overlay representing signs of PR identified by AI. Panels C and E show transverse views of the same vessel with AI analysis showing HU >30 (yellow), and consequently no LA-NCP. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

      4. Discussion

      This study evaluates the interobserver variation for total plaque and plaque component quantification among expert readers using reference measurements from an FDA approved AI validated CCTA analysis software and showed that expert readers perform moderately well quantifying plaque volume at the vessel territory level with moderate to high consistency among readers, but demonstrate high levels of interobserver variability as well as discordance with AI when quantifying plaque composition including %NCP, %CP and presence of HRP.
      Our data reflect that there was high consistently among expert readers but only moderate range performance quantifying total plaque by vessel territory compared to AI-QCT. Our findings are consistent with studies by Hoffmann et al. and Pearsons et al., both of which assessed interobserver variation among expert readers quantifying total plaque severity and found high interobserver consistency.
      • Pagali S.R.
      • Madaj P.
      • Gupta M.
      • et al.
      Interobserver variations of plaque severity score and segment stenosis score in coronary arteries using 64 slice multidetector computed tomography: a substudy of the ACCURACY trial.
      • Hoffmann H.
      • Frieler K.
      • Hamm B.
      • Dewey M.
      Intra- and interobserver variability in detection and assessment of calcified and noncalcified coronary artery plaques using 64-slice computed tomography: variability in coronary plaque measurement using MSCT.
      However, our findings diverge when assessing reader performance. While Hoffmann et al. and Pearsons et al. had used consensus reads as a reference standard and concluded that reader performance was strong, this study notably uses AI-QCT as a reference, and in doing so, brings the shortcomings of human reads into relief. Namely, while expert readers share a high level of skill that may result in similar outcomes among other expert readers, they nonetheless remain equally limited by the capabilities of human perception. By contrast, AI algorithms can evaluate images at the level of each individual voxel providing an accuracy that surpasses human capabilities.
      • van Rosendael A.R.
      • Maliakal G.
      • Kolli K.K.
      • et al.
      Maximization of the usage of coronary CTA derived plaque information using a machine learning based algorithm to improve risk stratification; insights from the CONFIRM registry.
      The extent of human limitation is magnified when evaluating expert performance quantifying plaque components. In the literature, as in our findings, interobserver variability increases and reader performance declines. In Hoffmann et al., not only does the kappa coefficient for quantifying %NCP v. %CP drop from high to moderate, but intra-observer variability also increases with a correlation of 0.68 between reads (down from 0.90 for quantifying plaque).
      • Hoffmann H.
      • Frieler K.
      • Hamm B.
      • Dewey M.
      Intra- and interobserver variability in detection and assessment of calcified and noncalcified coronary artery plaques using 64-slice computed tomography: variability in coronary plaque measurement using MSCT.
      While our study compared expert readers among one another, their calculations were also gauged against an FDA validated AI guided software that has shown non-inferiority to invasive coronary angiography measurements for stenosis, plaque quantification and characterization
      • Choi A.D.
      • Marques H.
      • Kumar V.
      • et al.
      CT EvaLuation by ARtificial Intelligence For Atherosclerosis, Stenosis and Vascular MorphologY (CLARIFY): A Multi-center, international study.
      United States Food and Drug Administration
      Cleerly Labs 510 (k) premarket notification.
      providing a non-biased comparison for reader performance. Compared to AI-QCT performance, expert readers showed a weak to moderately positive trend for quantifying %NCP vs. %CP with only fair correlation with AI. Consistently, there was only slight to fair correlation with AI-QCT when identifying HRP. The significance of these findings lies in the established prognostic benefit of plaque characteristics and a growing need for both quantifying high risk plaque features and tracking their progression. For instance, the CREDENCE trial showed that stenosis and plaque features together provide a non-invasive approach for predicting downstream ischemia that outperformed either stenosis alone or myocardial perfusion imaging.
      • Stuijfzand W.J.
      • van Rosendael A.R.
      • Lin F.Y.
      • et al.
      Stress myocardial perfusion imaging vs coronary computed tomographic angiography for diagnosis of invasive vessel-specific coronary physiology: predictive modeling results from the computed tomographic evaluation of atherosclerotic determinants of myocardial ischemia (CREDENCE) trial.
      Similarly, sub-studies of the SCOT-HEART trial show that specific atherosclerotic plaque characteristics (APCs) confer increased risk of myocardial events at five year follow up
      • Williams M.C.
      • Moss A.J.
      • Dweck M.
      • et al.
      Coronary artery plaque characteristics associated with adverse outcomes in the SCOT-HEART study.
      while the PARADIGM trial showed that baseline plaque volumes and the presence of HRP influence the rate of plaque progression over a two year period.
      • Lee S.E.
      • Sung J.M.
      • Rizvi A.
      • et al.
      Quantification of coronary atherosclerosis in the assessment of coronary artery disease.
      • Lee S.E.
      • Sung J.M.
      • Andreini D.
      • et al.
      Differences in progression to obstructive lesions per high-risk plaque features and plaque volumes with CCTA.
      • Han D.
      • Kolli K.K.
      • Al'Aref S.J.
      • et al.
      Machine learning framework to identify individuals at risk of rapid progression of coronary atherosclerosis: from the PARADIGM registry.
      These findings collectively emphasize the importance of accurately identifying, quantifying and tracking plaque features. Our study adds to this conversation by showing that not only are there limitations on expert reader performance quantifying plaque and identifying plaque features, but importantly that these limitations are mitigated by using AI-QCT.
      This is the first study to reflect expert interobserver performance for categorizing plaque characteristics against a validated quantitative reference. Given that new prognostic data support the quantification and tracking of APCs including HRP, our data suggest that AI may have a practical place in screening and monitoring plaque progression in appropriate patients, as high variability and difficulty accurately categorizing plaque components exists among expert readers. While the AI program has been validated against expert readers,
      • Choi A.D.
      • Marques H.
      • Kumar V.
      • et al.
      CT EvaLuation by ARtificial Intelligence For Atherosclerosis, Stenosis and Vascular MorphologY (CLARIFY): A Multi-center, international study.
      and quantitative coronary angiography,
      • Griffin W.F.
      • Choi A.D.
      • Riess J.
      • et al.
      AI evaluation of coronary stenosis on CT coronary angiography, comparison with quantitative coronary angiography and fractional flow reserve; a CREDENCE trial sub-study.
      studies are currently underway evaluating AI performance against IVUS,
      • Hakim D.
      Comparison of endothelial shear stress (ESS) computation utilizing non-invasive coronary computed tomography angiography (CCTA) vs Invasive Intravascular Ultrasound (IVUS) Imaging. [Poster presentation] American Heart Association Scientific Sessions, Boston, Massachusetts, USA (and Virtual Experience).
      optical coherence tomography, and near field infrared spectroscopy to establish all-around performance against the current gold standards.
      This study has limitations. The present study was a post-hoc analysis of the CLARIFY trial and, while it is unexpected that significant bias would be introduced in a retrospective evaluation leveraging blinded core laboratory readers, it nevertheless emphasizes the absence of a prospective clinical trial that could be performed in the future. Additionally, in this study, established HU thresholds for plaque characterization were utilized without adjustment in the absence of a standardized methodology for high luminal contrast enhancement. Ultimately, AI was not performed on CCTAs of poor image quality deemed uninterpretable by expert readers, further emphasizing limitations by human readers versus AI. Finally, while the prognostic significance of atherosclerotic plaque quantified by AI is still unknown, AI's high performance and the expanding knowledge surrounding the prognostic value of adverse plaque substantiate that further investigation is warranted.

      5. Conclusion

      Expert readers can quantify coronary plaque volume moderately well with high interobserver consistency. However, quantifying specific high risk coronary plaque components remains a challenge as high variability remains among readers with high discordance compared to AI-QCT.

      Declaration of competing interest

      The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:
      Hugo Marques reports a relationship with Cleerly Inc. that includes: consulting or advisory and equity or stocks. Andrew D. Choi reports a relationship with Cleerly Inc. that includes: equity or stocks. James P. Earls reports a relationship with Cleerly Inc. that includes: board membership, employment, and equity or stocks. Robert S. Jennings reports a relationship with Cleerly Inc. that includes: employment. Tami R. Crabtree reports a relationship with Cleerly Inc. that includes: employment.

      References

        • Narula J.
        • Chandrashekhar Y.
        • Ahmadi A.
        • et al.
        SCCT 2021 expert consensus document on coronary computed tomographic angiography: a report of the Society of Cardiovascular Computed Tomography.
        J Cardiovasc Comput Tomogr. 2021; 15: 192-217
        • Ferencik M.
        • Mayrhofer T.
        • Bittner D.O.
        • et al.
        Use of high-risk coronary atherosclerotic plaque detection for risk stratification of patients with stable chest pain: a secondary analysis of the PROMISE randomized clinical trial.
        JAMA Cardiol. 2018; 3: 144-152
        • Stuijfzand W.J.
        • van Rosendael A.R.
        • Lin F.Y.
        • et al.
        Stress myocardial perfusion imaging vs coronary computed tomographic angiography for diagnosis of invasive vessel-specific coronary physiology: predictive modeling results from the computed tomographic evaluation of atherosclerotic determinants of myocardial ischemia (CREDENCE) trial.
        JAMA Cardiol. 2020; 5: 1338-1348
        • Bittencourt M.S.
        • Hulten E.A.
        • Murthy V.L.
        • et al.
        Clinical outcomes after evaluation of stable chest pain by coronary computed tomographic angiography versus usual care: a meta-analysis.
        Circ Cardiovasc Imaging. 2016; 9e004419
        • Motoyama S.
        • Ito H.
        • Sarai M.
        • et al.
        Plaque characterization by coronary computed tomography angiography and the likelihood of acute coronary events in mid-term follow-up.
        J Am Coll Cardiol. 2015; 66: 337-346
        • Williams M.C.
        • Moss A.J.
        • Dweck M.
        • et al.
        Coronary artery plaque characteristics associated with adverse outcomes in the SCOT-HEART study.
        J Am Coll Cardiol. 2019; 73: 291-301
        • Motoyama S.
        • Sarai M.
        • Harigaya H.
        • et al.
        Computed tomographic angiography characteristics of atherosclerotic plaques subsequently resulting in acute coronary syndrome.
        J Am Coll Cardiol. 2009; 54: 49-57
        • Pagali S.R.
        • Madaj P.
        • Gupta M.
        • et al.
        Interobserver variations of plaque severity score and segment stenosis score in coronary arteries using 64 slice multidetector computed tomography: a substudy of the ACCURACY trial.
        J Cardiovasc Comput Tomogr. 2010; 4: 312-318
        • Choudhary G.
        • Atalay M.K.
        • Ritter N.
        • et al.
        Interobserver reliability in the assessment of coronary stenoses by multidetector computed tomography.
        J Comput Assist Tomogr. 2011; 35: 126-134
        • Kerl J.M.
        • Schoepf U.J.
        • Bauer R.W.
        • et al.
        64-Slice multidetector-row computed tomography in the diagnosis of coronary artery disease: interobserver agreement among radiologists with varied levels of experience on a per-patient and per-segment basis.
        J Thorac Imaging. 2012; 27: 29-35
        • Maroules C.D.
        • Hamilton-Craig C.
        • Branch K.
        • et al.
        Coronary artery disease reporting and data system (CAD-RADS(TM)): inter-observer agreement for assessment categories and modifiers.
        J Cardiovasc Comput Tomogr. 2018; 12: 125-130
        • Choi A.D.
        • Marques H.
        • Kumar V.
        • et al.
        CT EvaLuation by ARtificial Intelligence For Atherosclerosis, Stenosis and Vascular MorphologY (CLARIFY): A Multi-center, international study.
        J Cardiovasc Comput Tomogr. 2021; 15: 470-476
        • Griffin W.F.
        • Choi A.D.
        • Riess J.
        • et al.
        AI evaluation of coronary stenosis on CT coronary angiography, comparison with quantitative coronary angiography and fractional flow reserve; a CREDENCE trial sub-study.
        JACC Cardiovasc Imaging. 2022; (In press)
        • Abbara S.
        • Blanke P.
        • Maroules C.D.
        • et al.
        SCCT guidelines for the performance and acquisition of coronary computed tomographic angiography: a report of the Society of Cardiovascular Computed Tomography Guidelines Committee: endorsed by the North American Society for Cardiovascular Imaging (NASCI).
        J Cardiovasc Comput Tomogr. 2016; 10: 435-449
        • Hwang T.J.
        • Kesselheim A.S.
        • Vokinger K.N.
        Lifecycle regulation of artificial intelligence- and machine learning-based software devices in medicine.
        JAMA. 2019; 322: 2285-2286
        • United States Food and Drug Administration
        Cleerly Labs 510 (k) premarket notification.
        (URL)
        • Singh G.
        • Al'Aref S.J.
        • Van Assen M.
        • et al.
        Machine learning in cardiac CT: basic concepts and contemporary data.
        J Cardiovasc Comput Tomogr. 2018; 12: 192-201
        • Carin L.
        • Pencina M.J.
        On deep learning for medical image analysis.
        JAMA. 2018; 320: 1192-1193
        • Hoffmann H.
        • Frieler K.
        • Hamm B.
        • Dewey M.
        Intra- and interobserver variability in detection and assessment of calcified and noncalcified coronary artery plaques using 64-slice computed tomography: variability in coronary plaque measurement using MSCT.
        Int J Cardiovasc Imaging. 2008; 24: 735-742
        • van Rosendael A.R.
        • Maliakal G.
        • Kolli K.K.
        • et al.
        Maximization of the usage of coronary CTA derived plaque information using a machine learning based algorithm to improve risk stratification; insights from the CONFIRM registry.
        J Cardiovasc Comput Tomogr. 2018; 12: 204-209
        • Lee S.E.
        • Sung J.M.
        • Rizvi A.
        • et al.
        Quantification of coronary atherosclerosis in the assessment of coronary artery disease.
        Circ Cardiovasc Imaging. 2018; 11e007562
        • Lee S.E.
        • Sung J.M.
        • Andreini D.
        • et al.
        Differences in progression to obstructive lesions per high-risk plaque features and plaque volumes with CCTA.
        JACC Cardiovasc Imaging. 2020; 13: 1409-1417
        • Han D.
        • Kolli K.K.
        • Al'Aref S.J.
        • et al.
        Machine learning framework to identify individuals at risk of rapid progression of coronary atherosclerosis: from the PARADIGM registry.
        J Am Heart Assoc. 2020; 9e013958
        • Hakim D.
        Comparison of endothelial shear stress (ESS) computation utilizing non-invasive coronary computed tomography angiography (CCTA) vs Invasive Intravascular Ultrasound (IVUS) Imaging. [Poster presentation] American Heart Association Scientific Sessions, Boston, Massachusetts, USA (and Virtual Experience).
        2021 (November 13-15)