Advertisement

Radiology trainee and attending satisfaction with virtual readouts during the COVID-19 pandemic

      Highlights

      • Virtual platforms resembling in-person readouts are overall preferred by both trainees and attendings.
      • Between similar platforms, technical stability determines satisfaction and perceived effectiveness of a readout system.
      • Trainees and attendings support use of virtual readouts in combination with in-person readouts in the post-COVID-19 era.
      • Virtual platforms may be incorporated into radiology residency education to create a hybrid model for readouts.

      Abstract

      Rationale and objectives

      In response to COVID-19, our institution implemented three virtual readout systems: a commercial HIPAA compliant web-based video conferencing platform used for screen-sharing (Starleaf), an interactive control sharing system integrated into PACS allowing simultaneous multi-user mouse control over images (Collaborate), and the telephone. Our aim was to assess overall satisfaction with and perceived effectiveness of these virtual readout methods to optimize best practices for the future.

      Materials and methods

      An IRB-exempt survey was electronically distributed to 64 trainees and 76 attendings at one tertiary-care institution via Survey Monkey. Questions focused on overall satisfaction, perceived effectiveness, technical difficulties, and continued future use of the three virtual readout strategies. Answers were collected with Likert scales, tick boxes, and open-ended questions.

      Results

      32/64 trainees (50%) and 32/76 attendings (42%) completed the survey. Trainees and attendings were more satisfied with screen sharing (Starleaf) and perceived it more effective than control sharing (Collaborate) or the telephone (p < 0.0001). Respondents experienced more technical difficulties with control sharing versus screen sharing (p = 0.0004) with a negative correlation between level of technical difficulties and satisfaction with screen sharing (r = −0.50, p < 0.0001) and control sharing (r = −0.38, p = 0.0006). Trainees and faculty supported a combination of in-person and virtual readouts in the future (p < 0.0001).

      Conclusion

      Platforms mirroring in-person readouts, such as Starleaf, are preferred by both trainees and attendings over non-screen sharing platforms such as the telephone. However, technical stability determines satisfaction between similar platforms. Both trainees and attendings support incorporation of virtual readout methods in combination with traditional in-person readouts in the post-COVID-19 era.

      Abbreviations:

      COVID-19 (Coronavirus Disease 2019)

      Keywords

      1. Introduction

      The COVID-19 pandemic forced radiology residency programs to restructure their learning environments in order to comply with the mandated social distancing measures.
      • Slanetz P.J.
      • Parikh U.
      • Chapman T.
      • Motuzas C.L.
      Coronavirus disease 2019 (COVID-19) and radiology education—strategies for survival.
      • England E.
      • Kanfi A.
      • Flink C.
      • Vagal A.
      • Sarkany D.
      • Patel M.D.
      • et al.
      Radiology residency program management in the COVID era – strategy and reality.
      Specifically, traditional in-person readouts were disrupted, and new virtual readout methods were subsequently developed. Several papers published during the earlier stages of the pandemic detailed the different virtual readout methods employed, which included videoconferencing and screening sharing software,
      • Matalon S.A.
      • DAT Souza
      • Gaviola G.C.
      • Silverman S.G.
      • Mayo-Smith W.W.
      • Lee L.K.
      Trainee and attending perspectives on remote radiology readouts in the era of the COVID-19 pandemic.
      • Li C.H.
      • Rajamohan A.G.
      • Acharya P.T.
      • Patel V.
      • Go J.L.
      • Lui C.-S.J.
      • et al.
      Virtual read-out: radiology education for the 21st century during the COVID-19 pandemic.
      • Alvin M.D.
      • George E.
      • Deng F.
      • Warhadpande S.
      • Lee S.I.
      The impact of COVID-19 on radiology trainees.
      • Chong A.
      • Kagetsu N.J.
      • Yen A.
      • Cooke E.A.
      Radiology residency preparedness and response to the COVID-19 pandemic.
      • Virarkar M.
      • Jensen C.
      • Javadi S.
      • Saleh M.
      • Bhosale P.R.
      Radiology education amid COVID-19 pandemic and possible solutions.
      • Nadgir R.
      Teaching remotely: educating radiology trainees at the workstation in the COVID-19 era.
      however few focused on trainee and faculty satisfaction with these practices. One paper investigated resident satisfaction with virtual learning platforms, including virtual readouts, and found that residents were less satisfied with virtual readouts compared to other components of their virtual curriculum, but did not go into further detail.
      • Larocque N.
      • Shenoy-Bhangle A.
      • Brook A.
      • Eisenberg R.
      • Chang Y.M.
      • Mehta P.
      Resident Experiences With Virtual Radiology Learning During the COVID-19 Pandemic.
      At the start of the pandemic, our department established three different virtual readout systems, including a commercially available HIPAA compliant web-based video conferencing screen sharing platform (Starleaf; Watford, UK), an interactive control sharing system integrated into the institutional PACS allowing simultaneous multi-user mouse control over images (Collaborate; Change Healthcare, Nashville TN), and telephone readouts. Both the screen sharing and control sharing methods permitted the other user(s) to see the presenter's PACS images. However, screen sharing allowed only the presenter to control the mouse and manipulate PACS images, while control sharing allowed both the presenter and the trainee to control the mouse to scroll through images and point to areas of interest, theoretically allowing for a more interactive learning experience. The telephone method only allowed for a verbal description of imaging findings, presumably preferred for a quick discussion of simple points decreasing the added time needed to load screen and control share applications.
      The purpose of this study was to assess both radiology trainee and attending overall satisfaction with and subjective perceived effectiveness of three different methods of virtual readout implemented since April 2020 at a single large residency program and to determine the best virtual readout platform for optimal trainee education during the COVID-19 pandemic, as well as the opportunity to continue using these methods in the post pandemic era. Additionally, while there is already much agreement that in-person readouts are generally preferred, there seems to be disagreement institutionally regarding the utility of virtual readout platforms, therefore, this study was also meant to elucidate a general consensus in order to develop best practices for readouts going forward.

      2. Materials and methods

      This study was submitted to the Medical Board of Ethics at our institution and was approved for institutional review board exemption.

      2.1 Virtual readout platforms

      Our diagnostic radiology residency program currently trains 46 radiology residents (34 diagnostic radiology residents, 9 integrated interventional radiology residents, 3 interventional radiology residents) and 18 fellows, and is affiliated with a large tertiary-care academic institution. In response to the COVID-19 social distancing mandates, our department implemented three different virtual readout methods starting in April 2020. These included: a HIPAA compliant web-based video conferencing screen sharing platform (Starleaf, Watford, UK), a screen and control sharing tool integrated into the PACS (Collaborate, Change Healthcare, Nashville, TN); and telephone readout sessions. The questionnaire surveys covered the time period between April 2020 and April 2021.
      Starting in July 2020, some sections in our department returned to a hybrid mixture of in-person and virtual readouts, including the Body and Musculoskeletal sections, with individual sections slowly increasing their percentages of in-person readout to April 2021. This was done on a section-by-section basis as well by individual attending preference. Other sections remained completely virtual, including our Neuroradiology and Emergency department sections. Therefore, the new first year residents starting in July 2020 experienced both virtual and in-person readouts. Thus, all residents experienced both in-person and virtual readout sessions at some point during the survey period and could make direct comparisons between them.

      2.1.1 Screen sharing (Starleaf)

      Starleaf is a commercially available cloud-based HIPAA compliant videoconferencing system which allows for a single participant to share his/her screen (Fig. 1). The platform also allows for multiple users to participate in a single session. Multiple participants can be seen using separate webcams if desired and can converse through microphones and speakers. The participant sharing his/her screen is the only one able to control the mouse to scroll through images and point out pertinent findings. Since the creation and sharing of ad-hoc meeting invitations can be cumbersome, an institutional “virtual reading room” web application was created (Fig. 1A). This application, accessible via a web browser through a link built into the PACS workstation, provides a list of links that can launch “virtual reading rooms” named after the physical reading rooms, with larger reading rooms connected via multiple virtual links. The links associated with the meetings are dynamically regenerated every 24 h using the Starleaf application programming interface (API). This approach provided an additional layer of protection against the meeting links being inadvertently discovered by an unauthorized person.
      Fig. 1
      Fig. 1Screen sharing (Starleaf). A. Multiple Starleaf sessions accessible through the institutional intranet. The virtual reading rooms were divided by service, with a up to 4 different rooms per service. B. After a participant selects a virtual reading room, a screen loads for the participants to enter his/her name as well as select the option to connect to webcam and microphone. C. Main screen of the virtual reading room. The menu bar at the top has several features including: list of the number of participants in the session, option to launch a text chat box, ability to share screen, webcam and microphone settings, and ability to record a session.
      Fig. 1
      Fig. 1Screen sharing (Starleaf). A. Multiple Starleaf sessions accessible through the institutional intranet. The virtual reading rooms were divided by service, with a up to 4 different rooms per service. B. After a participant selects a virtual reading room, a screen loads for the participants to enter his/her name as well as select the option to connect to webcam and microphone. C. Main screen of the virtual reading room. The menu bar at the top has several features including: list of the number of participants in the session, option to launch a text chat box, ability to share screen, webcam and microphone settings, and ability to record a session.

      2.1.2 Control sharing (Collaborate)

      Collaborate is an integrated software tool accessible through PACS which allows for both participants to work on a shared window simultaneously (Fig. 2). Each participant has the ability to take control and use the mouse and PACS functions within the same shared PACS window, allowing for an interactive session. Each participant can scroll, zoom, pan, window/level and annotate the images using the same tools in the PACS viewer, and these manipulations are viewable in real time by the other participant. While the shared mouse is being used by one participant, the other person can see the mouse but cannot control it for his/her monitor or other linked computer monitors. This platform does not have a built-in microphone or webcam, therefore discussion needs to be done using either the telephone or another system with a microphone. To initiate the control sharing feature on PACS, the participant first launches the study they wish to collaborate on and clicks the “Collaborate” link on the PACS menu bar (Fig. 2A). From there, a menu pops up listing everyone currently logged into a PACS workstation (Fig. 2B). The participant must select the additional collaborator from the menu to send an invitation. The invited collaborator will then see the invitation pop up in the corner of his/her PACS screen (Fig. 2C). After clicking accept, a new integrated PACS window is launched that includes both participants (Fig. 2D). After the current study is finished, the Collaborate platform is exited. The initial version of this platform required a new invitation to be sent for every new study to be reviewed. After providing feedback to the vendor, a software patch was released to enable multiple studies to be reviewed in a single Collaborate session.
      Fig. 2
      Fig. 2Control sharing (Collaborate). A. Menu bar on every PACS workstation has a link to the Collaborate platform. B. Once the participant clicks the link, a menu launches listing every radiology trainee and attending in the system. The participant chooses a single collaborator to send an invitation. C. The invitation populates on the collaborator's PACS station to be accepted. D. Once the invitation is accepted, a shared screen launches with the case on the original participant's screen. The shared screen contains all the PACS features and allows both participants to manipulate the images.
      Fig. 2
      Fig. 2Control sharing (Collaborate). A. Menu bar on every PACS workstation has a link to the Collaborate platform. B. Once the participant clicks the link, a menu launches listing every radiology trainee and attending in the system. The participant chooses a single collaborator to send an invitation. C. The invitation populates on the collaborator's PACS station to be accepted. D. Once the invitation is accepted, a shared screen launches with the case on the original participant's screen. The shared screen contains all the PACS features and allows both participants to manipulate the images.

      2.1.3 Telephone

      Trainees and faculty discussed cases over the telephone while viewing images separately on their own workstations.

      2.2 Data collection and analysis

      A 10-question survey was distributed to all current radiology residents, fellows, and attendings between April to June 2021, after the virtual readout platforms were employed for one year. The survey was first distributed electronically via email using SurveyMonkey (San Mateo, CA) in April 2021, followed by two additional follow-up emails in May and June 2021.
      Survey questions (Appendix A) focused on satisfaction with and perceived effectiveness of the three different virtual readout strategies in comparison to in-person readouts, as well as the desire to continue using these methods in the future, including in the post-pandemic era. Only subjective perceived effectiveness of the platforms relative to in-person readouts was evaluated given the difficulty in controlling for confounding variables influencing quantitative measures of effectiveness, such as standardized or in-house exams. For instance, the senior residents had years of prior training based on traditional in-person readouts while more junior residents did not. Additionally, there is a lack of statistical power in the number of junior resident scores to compare with previous years to assess scores before and after the use of virtual readouts. Thus, no metric on actual teaching effectiveness can be assessed at this time. Technical difficulties with each of the virtual readout methods and suggestions for improvement were also assessed. Specifically relating to the control sharing platform, we assessed whether the ability for both users to manipulate the PACS images was an important feature for teaching and learning. Question types included answers with 3 to 5-point Likert scales, tick boxes, and open-ended questions.
      Data were exported from Survey Monkey and analyzed via Microsoft Excel and Matlab (Mathworks, Natick, MA). Answers made on a 5-point scale were converted to numbers from 1 to 5 for most analyses, and for computing mean and/or median grades. Some questions analyzed on a 5-point scale included an additional question about whether or not the respondent had used the specific platform, which was not ultimately used for generation of mean and/or median grades. Answers made on a 3-point scale were converted to numbers from 1 to 3 for most analysis, and for computing mean and/or median grades. The quantitative data were analyzed using proportions, Kendall rank correlation, Kruskal-Wallis and Friedman tests. Pairwise comparisons were performed using Tukey post-hoc procedure after the overall significant difference was established. p-Values ≤ 0.05 were considered statistically significant. Data from the open-ended questions were reviewed and analyzed by three study authors. The main themes discussed in the comments are presented in this study.
      To assess possible effects of virtual readouts on resident productivity, additional data was gathered via CPT codes regarding the percentage and number of reports generated by residents between April 1, 2020-July 1, 2020 (a period during the COVID-19 pandemic when all nonessential medical procedures were placed on hold) and between April 1, 2020-March 31, 2021, to assess overall studies performed during the time period of our study. This was compared to the percentage and number of reports generated by residents before the COVID-19 pandemic time periods of April 1, 2019-July 1, 2019, and April 1, 2019-March 31, 2020. Absolute numbers were not compared given the ~30% decline in radiology volume during the months of April 1, 2020-July 1, 2020, due to the hold on all nonessential radiology studies.

      3. Results

      A total of 32 out of 64 trainees (50%) completed the survey, which included 26 out of 46 residents (57%) and 6 out of 18 fellows (33%). The distribution of responses by level of training was: 8 (13%) first year residents, 7 (11%) second year residents, 6 (9%) third year residents, and 5 (8%) fourth year residents. A total of 32 out of 76 radiology attendings (42%) completed the survey. Two respondents did not specify level of training. Thirty-three (50%) women and twenty-eight (42%) men responded, while five (8%) preferred not to disclose their gender. A summary of the survey results is listed in Table 1.
      Table 1Virtual readout survey results.
      Questionn%
      Level of training
       R1813%
       R2711%
       R369%
       R458%
       Fellow69%
       Faculty3250%
       Total64
      Gender
       Female3350%
       Male2842%
       Prefer not to disclose58%
       Total66
      Level of satisfaction with virtual readout strategies
      Screen sharing (Starleaf)
       Did not use58%
       Extremely satisfied1523%
       Very satisfied1218%
       Moderately satisfied1929%
       Slightly satisfied1320%
       Not at all satisfied23%
       Total66
      Control sharing (Collaborate)
       Did not use711%
       Extremely satisfied711%
       Very satisfied914%
       Moderately satisfied1422%
       Slightly satisfied2234%
       Not at all satisfied69%
       Total66
      Telephone
       Did not use69%
       Extremely satisfied58%
       Very satisfied1421%
       Moderately satisfied1827%
       Slightly satisfied1929%
       Not at all satisfied46%
       Total66
      Learning as effective with virtual readout as in-person readouts
      Screen sharing (Starleaf)
       Did not use46%
       Strongly agree1117%
       Agree2030%
       Neutral69%
       Disagree1726%
       Strongly disagree812%
       Total66
      Control sharing (Collaborate)
       Did not use69%
       Strongly agree914%
       Agree1625%
       Neutral812%
       Disagree1523%
       Strongly disagree1117%
       Total66
      Telephone
       Did not use46%
       Strongly agree46%
       Agree1117%
       Neutral1219%
       Disagree2437%
       Strongly disagree1015%
       Total66
      Technical difficulties (Ex. System crashes, trouble loading platform, slowing/freezing of software, etc.)
      Screen sharing (Starleaf)
       Did not use58%
       Strongly agree46%
       Agree711%
       Neutral1320%
       Disagree2233%
       Strongly disagree1523%
       Total66
      Control sharing (Collaborate)
       Did not use914%
       Strongly agree711%
       Agree1827%
       Neutral914%
       Disagree1624%
       Strongly disagree711%
       Total66
      Telephone
       Did not use58%
       Strongly agree12%
       Agree58%
       Neutral914%
       Disagree2742%
       Strongly disagree1828%
       Total66
      Do you agree that the ability for both users to manipulate the PACS images on Collaborate was important for learning/teaching?
       Strongly agree2445%
       Agree1426%
       Neutral1121%
       Disagree24%
       Strongly disagree24%
       Total53
      In the future, would you agree to support the use of some form of virtual read-out combined with in-person readouts?
       Yes4177%
       No48%
       Neutral815%
       Total53
      Support of these virtual programs in the future combined with in person readouts
      Screen sharing (Starleaf)
       Did not use36%
       Strongly support2548%
       Support1733%
       Neutral36%
       Do not support36%
       Strongly do not support12%
       Total52
      Control sharing (Collaborate)
       Did not use48%
       Strongly support1325%
       Support1631%
       Neutral917%
       Do not support815%
       Strongly do not support24%
       Total52
      Telephone
       Did not use12%
       Strongly support1121%
       Support2038%
       Neutral815%
       Do not support917%
       Strongly do not support36%
       Total52
      Trainees and attendings combined were significantly more satisfied with the screen sharing platform compared to the control sharing platform (3.41 vs. 2.81; p = 0.001) and compared to the telephone (3.41 vs. 2.95; p = 0.0003). There was no significant difference in satisfaction between the control sharing and the telephone methods (2.81 vs. 2.95; p = 0.93).
      Overall, trainees and attendings experienced significantly more technical difficulties with control sharing than with screen sharing (3.04 vs. 2.39; p = 0.0005) and the telephone (3.04 vs. 2.07; p < 0.0001). There was no significant difference in technical issues experienced between screen sharing and the telephone (2.39 vs. 2.07; p = 0.2).
      With regards to these virtual methods in comparison to in-person readouts, screen sharing and control sharing were both perceived to be significantly more effective than the telephone (3.14 vs. 2.59; p = 0.0001 and 2.95 vs. 2.59; p = 0.007 respectively). There was no significant difference in perceived effectiveness between screen sharing and control sharing in comparison to in-person readouts (3.14 vs. 2.95; p = 0.5). Means for satisfaction, perceived effectiveness, and experiences of technical difficulties are summarized in Table 2.
      Table 2Mean responses by all participants.
      VariableMeanVariable comparisonsp-Value
      Satisfaction
      Means calculated from the following 5-point scale: 5 = Extremely Satisfied, 4 = Very satisfied, 3 = Moderately satisfied, 2 = Slightly satisfied, 1 = Not satisfied at all.
       Screen sharing (Starleaf)3.41Starleaf vs. Collaborate0.001
       Control sharing (Collaborate)2.81Starleaf vs. Telephone0.0003
       Telephone2.95Collaborate vs. Telephone0.9
      Perceived effectiveness
      Means calculated from the following 5-point scale: 5 = Strongly agree, 4 = Agree, 3 = Neutral, 2 = Disagree, 1 = Strongly disagree.
       Screen sharing (Starleaf)3.14Starleaf vs. Collaborate0.5
       Control sharing (Collaborate)2.95Starleaf vs. Telephone0.0001
       Telephone2.59Collaborate vs. Telephone0.007
      Experienced technical difficulties
      Means calculated from the following 5-point scale: 5 = Strongly agree, 4 = Agree, 3 = Neutral, 2 = Disagree, 1 = Strongly disagree.
       Screen sharing (Starleaf)2.39Starleaf vs. Collaborate0.0005
       Control sharing (Collaborate)3.04Starleaf vs. Telephone0.2
       Telephone2.07Collaborate vs. Telephone<0.0001
      Support of virtual platform in future
      Means calculated from the following 5-point scale: 5 = Strong support, 4 = Support, 3 = Neutral, 2 = Do not support, 1 = Strongly do not support.
       Screen sharing (Starleaf)4.26Starleaf vs. Collaborate0.07
       Control sharing (Collaborate)3.60Starleaf vs. Telephone0.0009
       Telephone2.54Collaborate vs. Telephone0.3
      Bolded text indicates statistical significance
      a Means calculated from the following 5-point scale: 5 = Extremely Satisfied, 4 = Very satisfied, 3 = Moderately satisfied, 2 = Slightly satisfied, 1 = Not satisfied at all.
      b Means calculated from the following 5-point scale: 5 = Strongly agree, 4 = Agree, 3 = Neutral, 2 = Disagree, 1 = Strongly disagree.
      c Means calculated from the following 5-point scale: 5 = Strong support, 4 = Support, 3 = Neutral, 2 = Do not support, 1 = Strongly do not support.
      There was somewhat greater satisfaction and significantly more positive perception of effectiveness with the telephone by attendings compared to trainees (3.18 vs. 2.68; p = 0.07 and 2.86 vs. 2.26; p = 0.05 respectively). Otherwise, there were no other significant differences between trainee and attending experiences with the three virtual readout platforms (Table 3).
      Table 3Comparison of responses by trainees and attendings.
      VariableTrainee (mean)Attending (mean)p-Value
      Satisfaction
      Means calculated from the following 5-point scale: 5 = Extremely Satisfied, 4 = Very satisfied, 3 = Moderately satisfied, 2 = Slightly satisfied, 1 = Not satisfied at all.
       Screen sharing (Starleaf)3.343.430.83
       Control sharing (Collaborate)2.912.600.32
       Telephone2.683.180.07
      Perceived effectiveness
      Means calculated from the following 5-point scale: 5 = Strongly agree, 4 = Agree, 3 = Neutral, 2 = Disagree, 1 = Strongly disagree.
       Screen sharing (Starleaf3.093.140.83
       Control sharing (Collaborate)2.813.040.54
       Telephone2.262.860.05
      Experienced technical difficulties
      Means calculated from the following 5-point scale: 5 = Strongly agree, 4 = Agree, 3 = Neutral, 2 = Disagree, 1 = Strongly disagree.
       Screen sharing (Starleaf2.382.460.72
       Control sharing (Collaborate)3.063.080.99
       Telephone2.132.040.71
      Ability for both users manipulate mouse on Collaborate aids in learning/teaching
      Means calculated from the following 3-point scale: 3 = Agree, 2 = Neutral, 1 = Disagree.
      4.073.950.95
      Support the use of some form of virtual read-out combined with in-person readouts
      Means calculated from the following 3-point scale: 3 = Agree, 2 = Neutral, 1 = Disagree.
      2.552.860.13
      Support of virtual platform in future combined with in-person readout
      Means calculated from the following 3-point scale: 3 = Agree, 2 = Neutral, 1 = Disagree.
       Screen sharing (Starleaf4.174.410.46
       Control sharing (Collaborate)3.553.690.54
       Telephone3.034.320.0004
      Bolded text indicates statistical significance
      a Means calculated from the following 5-point scale: 5 = Extremely Satisfied, 4 = Very satisfied, 3 = Moderately satisfied, 2 = Slightly satisfied, 1 = Not satisfied at all.
      b Means calculated from the following 5-point scale: 5 = Strongly agree, 4 = Agree, 3 = Neutral, 2 = Disagree, 1 = Strongly disagree.
      c Means calculated from the following 3-point scale: 3 = Agree, 2 = Neutral, 1 = Disagree.
      When the trainee and attending groups were analyzed separately, for trainees there was only a significant difference in satisfaction between screen sharing and the telephone (3.35 vs. 2.68; p = 0.0006) but not between control sharing and screen sharing (p = 0.1) or the telephone (p = 0.2). For attendings, there was a significant difference in satisfaction between screen sharing and control sharing (3.57 vs. 2.65; p = 0.005) and no significant difference between screen sharing or control sharing compared to the telephone (p = 0.3 both), which differs from trainees. In terms of perceived effectiveness compared to in-person readouts, the trainee group showed that both screen sharing and control sharing were perceived as more effective than the telephone (3.06 vs. 2.26; p = 0.0004 and 2.81 vs. 2.26; p = 0.03 respectively). In the attending group, there was no significant difference in perceived effectiveness between the virtual platforms (p = 0.12). Both trainees and attendings had significantly more technical difficulties with control sharing compared to the other virtual platforms (p = 0.0002 and p = 0.0007 respectively), with no significant difference between screen sharing and the telephone (p = 0.64 and p = 0.26 respectively).
      There was a significant negative correlation between the level of technical difficulties and satisfaction for screen sharing (r = −0.50, p < 0.0001) and control sharing (r = −0.38, p = 0.0006), but not for the telephone (p = 0.13). Additionally, there was a significant negative correlation between the level of technical difficulties and perceived effectiveness of screen sharing (r = −0.28, p = 0.009) and control sharing (r = −0.25, p = 0.02) relative to in-person readouts, but not for the telephone (p = 0.8).
      Regarding the control sharing platform, most agreed to strongly agreed that the ability for both users to manipulate the PACS images was important for learning and teaching (p < 0.0001). There was no significant difference in responses between trainees and attendings (p = 0.95) (Table 3).
      Of the 33 comments regarding screen sharing, 11 (33%) mentioned that the resolution quality was not ideal. Six out of 31 (19%) comments about control sharing expressed dissatisfaction that only one participant could use the mouse at a time and 11 comments (35%) stated that control sharing was often slow, crashed, or did not work. The lack of visual learning with the telephone was specified in 21 of the 34 comments (62%). Representative positive and negative comments are detailed in Table 4.
      Table 4Representative comments by trainees and attendings.
      PositiveNegative
      Screen sharing (Starleaf)Able to review multiple studies at once without needing to start new session for each study.

      When used continuously with a webcam, there was continuous oversight of workflow and real time discussion of cases, which re-created the environment of in-person readouts.

      Works well and allows nearly immediate access to trainee or attending.
      Poorer resolution, harder to see smaller findings/suboptimal image quality.

      Spectator readout.

      Only one participant able to move the mouse.
      Control sharing (Collaborate)Easier to see images and navigate since it is on PACS.

      Both participants can use the mouse to navigate.
      Cannot concurrently use the shared mouse.

      Had trouble getting Collaborate to work on many occasions.

      Slow, often did not work.

      Can only open one study at a time.
      TelephoneHelpful for quick conversation.

      Great for efficiency for simple cases.

      Used only for select cases that don't need much interactive review.
      Cannot share screen, which is okay for easier things, but makes showing findings more difficult.

      Telephone readouts only really work with more senior residents, who have basics down. Many more junior residents need specific findings pointed out directly which is tricky on the phone.

      Terrible for cases that are complex and of high learning value.
      Of the 35 comments provided by respondents regarding suggestions for improvement, a large portion of respondents (40%) expressed a strong preference for in-person readouts, however there was not an overwhelmingly negative attitude towards the virtual readout platforms overall. Five respondents (14%) voiced positive feedback regarding the screen sharing system. Eight respondents (23%) provided feedback to improve the platforms and did not convey a strong opinion against virtual readouts. Two respondents (6%) expressed strongly negative feedback regarding the control sharing function and only one respondent expressed strongly negative feedback regarding the telephone readouts. There was no strongly negative feedback written about the screen sharing virtual platform. Positive responses regarding screen sharing were that multiple participants could log into the same virtual room and participate in the readout. Several respondents liked the image resolution on the control sharing platform and the convenience of the telephone for quick questions or straightforward cases. Several comments mentioned the importance of having a webcam in order to see each other and receive non-verbal cues similar to in-person readouts, which was an optional feature built into the screen sharing application.
      The majority of respondents supported the use of some form of virtual readout method in conjunction with in-person readouts in the future. The number of responses in support of incorporation of virtual platforms in the future was significantly higher than the number of responses against it (p < 0.0001), with no significant difference between trainees and attendings (p = 0.13). Out of the three virtual platforms, respondents significantly supported the use of screen sharing over the telephone (4.26 vs. 3.54; p = 0.0009) and showed some support for screen sharing over control sharing in conjunction with in-person readouts in the future (4.26 vs. 3.60; p = 0.07). There was no significant difference in support between control sharing and the telephone (p = 0.34). There was no significant difference in support between trainees and attendings regarding screen sharing (p = 0.46) and control sharing (p = 0.54), however attendings more strongly supported the use of the telephone in future readouts compared to trainees (4.32 vs. 3.03; p = 0.0004) (Table 3).
      Additional data regarding the percentage and number of reports generated by residents during and before the COVID-19 pandemic was obtained to assess possible effects of virtual readouts on resident productivity. From April 1, 2020-July 1, 2020, during the pandemic when all non-essential medical procedures were on hold, 53,587 exams were read in total with 27,039 (50.5%) exams having a resident read. During the entire period from April 1, 2020- March 31, 2021 (the time period assessed in this study), 309,603 exams were read in total with 151,285 (48.9%) exams having a resident read. From April 1, 2019- July 1,2019 before the pandemic, 94,360 exams were read in total with 42,593 (45.1%) exams having a resident read. Between April 1, 2019-March 31,2020, 357,140 exams were read in total with 168,836 (47.3%) having a resident read.

      4. Discussion

      The COVID-19 pandemic has changed the way in which radiology residency programs structure their learning environments, particularly in terms of the traditional in-person readouts, which now include numerous virtual platforms. Our study provides both trainee and attending feedback regarding three different virtual readout methods implemented at our institution, including a commercially available HIPAA compliant web-based videoconferencing screen sharing platform (Starleaf), a control sharing integrated PACS platform (Collaborate), both of which most closely resemble in-person readouts by allowing for combined visual and verbal direction with control sharing further allowing for shared interaction with the images, and telephone readout sessions.
      Trainees and attendings were significantly more satisfied with the screen sharing platform. This was likely in part related to the fact that screen sharing had significantly fewer technical issues than control sharing and also allowed users to concurrently review images unlike with the telephone. The key features to a traditional readout described by Li et al. include direct two-way communication between the trainee and attending, ability to view images simultaneously, potential for all parties to scroll through images, and an ability to annotate the images, and the ideal virtual system would incorporate these key features.
      • Li C.H.
      • Rajamohan A.G.
      • Acharya P.T.
      • Patel V.
      • Go J.L.
      • Lui C.-S.J.
      • et al.
      Virtual read-out: radiology education for the 21st century during the COVID-19 pandemic.
      Screen sharing includes most of these features in some form, while the telephone does not since it does not allow for simultaneous review of images and makes it much more difficult to point out findings. Control sharing also allows for screen sharing, however lacked a build in microphone and webcam. While control sharing permits both participants to view and scroll on the same PACS screen, its technological issues limit complete assessment of the benefit of this increased interactivity.
      Although there was no statistical difference in perceived effectiveness of screen sharing versus control sharing, both visually based systems were considered significantly more effective compared to the telephone. This likely reflects a preference for traditional in-person readouts, most closely resembled by the screen sharing and control sharing platforms. It is important to note that only subjective perceived effectiveness of the platforms relative to in-person readouts was able to be evaluated and no metric on actual teaching effectiveness could be assessed at this time.
      As alluded to above, technical difficulty with a virtual platform negatively impacts its ability to be successfully implemented for readouts. In our quantitative data, there was a significant negative correlation between technical difficulties and satisfaction as well as with perceived effectiveness of both screen sharing and control sharing. In our qualitative data, this was reflected in comments regarding the low image resolution in the screen sharing program and the inability for both participants to simultaneously use the mouse in the control sharing tool. Additionally, control sharing was often slow or did not work appropriately. Respondents also did not like that only one study could be opened at a time on control sharing. Although multi-study control sharing sessions were added via a software patch, most remote readouts during the first COVID surge occurred before this fix was available. Respondents liked the ability for multiple participants to log into a virtual reading room at the same time on the screen sharing platform to review cases together. Several respondents mentioned the need to always have a webcam in order receive non-verbal cues during virtual readouts and better gauge understanding, similar to the finding detailed by Matalon et al..
      • Matalon S.A.
      • DAT Souza
      • Gaviola G.C.
      • Silverman S.G.
      • Mayo-Smith W.W.
      • Lee L.K.
      Trainee and attending perspectives on remote radiology readouts in the era of the COVID-19 pandemic.
      Although most participants felt that the ability for both users to manipulate the PACS images in the control sharing tool was important for teaching and learning, there was still an overall dissatisfaction with this platform, presumably due to the higher degree of technical issues.
      Trainees and attendings generally agreed regarding satisfaction with and perceived effectiveness of the different virtual readout platforms, though there was a trend towards greater satisfaction with the telephone by attendings compared to trainees. This result is likely related to the faster speed and greater ease of use with the telephone compared to the other methods. Feedback regarding the telephone however was predominately negative in terms of education since findings could not be visually pointed out like with in-person readouts. In a time where case volume continues to rise, speed and efficiency are important factors to consider, while also not compromising education. The greater degree of technical issues encountered on screen and control sharing tools relative to the phone likely disrupts workflow. Therefore, the telephone could provide benefit and increase efficiency in certain circumstances, such as when confronted with very high exam volumes, for addressing straightforward questions, or when reading out with more senior trainees who may not require as much direct supervision and discussion.
      • Matalon S.A.
      • DAT Souza
      • Gaviola G.C.
      • Silverman S.G.
      • Mayo-Smith W.W.
      • Lee L.K.
      Trainee and attending perspectives on remote radiology readouts in the era of the COVID-19 pandemic.
      • Harari A.A.
      • Conti M.B.
      • SAJ Bokhari
      • Staib L.H.
      • Taylor C.R.
      The role of report comparison, analysis, and discrepancy categorization in resident education.
      Though based on our results, the telephone should not be the primary method for virtual readout.
      In general, respondents still prefer in-person readouts over virtual readouts. While virtual methods should not replace in-person interactions, there is a strong support for incorporation of virtual platforms in the future. Respondents most strongly supported the continued use of the screening sharing application in the future, in keeping with the overall satisfaction and perceived effectiveness with the platform. Attendings significantly supported the use of the telephone in future readouts, likely due to improved efficiency compared to trainees, who valued visual learning more.
      Virtual readouts did not impact resident productivity. A quantitative comparison of efficiency between virtual and in-person readouts is difficult to assess on a numerical level because in the first 4–5 months of implementing virtual readout (and when nearly all readouts were virtual), there was a considerable drop in radiology volume due to suspension of nonessential hospital visits. However, the percentages of cases relative to overall volume was essentially unchanged to slightly higher during those months compared to the prior year before the pandemic, suggesting that there was no decreased efficiency in readout due to virtual methods. Once the suspension of nonessential hospital visits was lifted, some sections resumed readouts in a hybrid manner while others continued with virtual readouts only. During the time period which included these months, there was also no difference in percentages of resident reports relative to total number of studies, further suggesting that there is no significant effect of virtual readouts on resident efficiency.
      Limitations of this study include a small sample size at a single institution, which may limit generalizability of the findings across all institutions, however, given the presumed heterogeneity in implementation and duration of virtual readouts across institutions, we feel that our single institution paper allowed for specific and granular comparisons across distinct virtual platforms that would be impossible over multiple training programs. In addition, the smaller sample size did not allow for more detailed analysis of satisfaction of the virtual readout methods among differing resident training levels or years of experience among attendings. Future multi-institutional research would be needed to assess for differences between training levels. Furthermore, radiology subspeciality differences in virtual readout method preferences were not assessed in this study. It is possible that some sections would prefer or view different platforms more favorably based on specific needs. Future studies are needed to elucidate these differences. Selection bias may impact our results as potentially trainees and attendings who were more or less enthusiastic about virtual readouts may have been more likely to participate. Additionally, certain survey questions received fewer responses than others, which may also contribute to selection bias. Our study chose to evaluate subjective trainee and attending feedback metrics, though incorporating more objective metrics could help strengthen our results. For example, only perceived effectiveness of the virtual platforms relative to in-person readouts was able to be assessed and no metric on actual teaching effectiveness could be assessed. Given the relatively short time frame of virtual only readouts and lack of power in the number of junior residents where the potentially negative impacts of virtual readouts might be felt most strongly, no statistical analysis could be made on quantitative measures of learning. Nevertheless, attendings, whose evaluation of residents make up an important component of trainee core competency progress assessments, felt that virtual readouts were comparable to in-person readouts. Future multi-institutional studies will be required to assess if teaching was comparably effective between virtual versus in-person readouts in a quantitative manner. We were also unable to track the number of cases read using each virtual platform. Furthermore, given the technical issues faced with the control sharing platform, analysis of its true utility, specifically relating to increased interactivity between the attending and trainee during virtual readouts, could not be adequately assessed. A follow-up study with technical improvements to the control sharing tool may be needed to fully describe the impact of the added functionality on satisfaction and perceived effectiveness on teaching versus other virtual readout platforms. The return to a hybrid readout model starting in July 2020 introduces some confounding variables due to section inhomogeneity in virtual and in-person readouts. However, since the purpose of this study was to assess overall satisfaction and a relative sense of effectiveness of the virtual platforms, the hybrid system is not believed to impede the overall goals of the study because this allowed for a direct comparison between virtual and in-person readouts in the sample population at the same time. Thus, the results showing similar perceived teaching effectiveness of screen and control sharing platforms versus in-person readouts as well as overall satisfaction with virtual readouts is not due to recency effect but is mostly a contemporaneous comparison. Finally, the virtual readout platforms studied in this paper are representative of our institution and may not necessarily match exactly with the platforms at other institutions somewhat limiting generalizability of the study results.
      The results of our study can provide guidance for radiology residency readouts in the post-pandemic era. It is evident that the most important features of any virtual readout system are stability and functionality. We found that our screen sharing application served as a stable platform and encompassed most of the key elements of a traditional in-person readout. While solely virtual readouts are not ideal, they can be useful in certain circumstances, such as when the resident and attending are in separate facilities. These new technological advances that came with the COVID-19 pandemic can be integrated into radiology residencies and help improve overall education and work efficiency. A hybrid virtual/in-person model for readouts may be the future of radiology education.
      • Awan O.A.
      Virtual radiology readouts after the COVID-19 pandemic.

      5. Conclusion

      Virtual platforms that most closely resemble in-person readouts are overall preferred by both trainees and attendings. These platforms include a combination of audio and visual cues.
      Between similar platforms, technical stability determines overall satisfaction and perceived effectiveness of a virtual readout system. Both trainees and attendings support incorporation of virtual readout methods in combination with traditional in-person readouts in the post-COVID-19 era. Our study alludes to how these new virtual platforms may be effectively incorporated into radiology residency education in the future.

      Appendix A

      A. Virtual Readout Survey
      1. Current level of training:
      • A.
        R1
      • B.
        R2
      • C.
        R3
      • D.
        R4
      • E.
        Fellow
      • F.
        Faculty
      2. What is your gender?
      • A.
        Female
      • B.
        Male
      • C.
        Prefer not to disclose
      3. Please rate your level of satisfaction with the following virtual read-out strategies:
      Virtual Reading Rooms (Starleaf Platform)
      • A.
        Did not use
      • B.
        Extremely satisfied
      • C.
        Very satisfied
      • D.
        Moderately satisfied
      • E.
        Slightly satisfied
      • F.
        Not at all satisfied
      Collaborate (Integrated PACS Platform)
      A. Did not use
      B. Extremely satisfied
      C. Very satisfied
      D. Moderately satisfied
      E. Slightly satisfied
      F. Not at all satisfied
      Telephone
      A. Did not use
      B. Extremely satisfied
      C. Very satisfied
      D. Moderately satisfied
      E. Slightly satisfied
      F. Not at all satisfied
      4. Learning is as effective with virtual read-out as live read-outs:
      Virtual Reading Rooms (Starleaf Platform)
      • A.
        Did not use
      • B.
        Strongly agree
      • C.
        Agree
      • D.
        Neutral
      • E.
        Disagree
      • F.
        Strongly Disagree
      (free text)
      Collaborate (Integrated PACS Platform)
      A. Did not use
      B. Strongly agree
      C. Agree
      D. Neutral
      E. Disagree
      F. Strongly Disagree
      (free text)
      Telephone
      A. Did not use
      B. Strongly agree
      C. Agree
      D. Neutral
      E. Disagree
      F. Strongly Disagree
      (free text)
      5. Do you agree that the ability for both users to manipulate the PACS images in Collaborate was important for learning/Teaching?
      • A.
        Did not use
      • B.
        Strongly agree
      • C.
        Agree
      • D.
        Neutral
      • E.
        Disagree
      • F.
        Strongly Disagree
      6. Did you have substantial technical difficulties using the following virtual read-out platforms?
      Starleaf Platform
      A. Did not use
      B. Strongly agree
      C. Agree
      D. Neutral
      E. Disagree
      F. Strongly Disagree
      Collaborate (Integrated PACS Platform)
      A. Did not use
      B. Strongly agree
      C. Agree
      D. Neutral
      E. Disagree
      F. Strongly Disagree
      Telephone
      A. Did not use
      B. Strongly agree
      C. Agree
      D. Neutral
      E. Disagree
      F. Strongly Disagree
      7. Do you have any suggestions for improving virtual read-outs?
      (free text)
      8. Do you agree that the ability for both users to manipulate the PACS images on Collaborate was important for learning/teaching?
      A. Strongly agree
      B. Agree
      C. Neutral
      D. Disagree
      E. Strongly disagree
      9. In the future, would you agree to support the use of some form of virtual read-out combined with in-person readouts?
      A. Yes
      B. No
      C. Neutral
      10. Support of these virtual programs in the future combined with in person readouts:
      Starleaf
      A. Did not use
      B. Strongly support
      C. Support
      D. Neutral
      E. Do not support
      F. Strongly do not support
      Collaborate
      A. Did not use
      B. Strongly support
      C. Support
      D. Neutral
      E. Do not support
      F. Strongly do not support
      Telephone
      A. Did not use
      B. Strongly support
      C. Support
      D. Neutral
      E. Do not support
      F. Strongly do not support

      References

        • Slanetz P.J.
        • Parikh U.
        • Chapman T.
        • Motuzas C.L.
        Coronavirus disease 2019 (COVID-19) and radiology education—strategies for survival.
        Journal of the American College of Radiology. 2020 Jun; 17 (Available from): 743-745
        • England E.
        • Kanfi A.
        • Flink C.
        • Vagal A.
        • Sarkany D.
        • Patel M.D.
        • et al.
        Radiology residency program management in the COVID era – strategy and reality.
        Acad Radiol. 2020; 27 (Available from:) (Aug [cited 2021 Jun 13): 1140-1146
        • Matalon S.A.
        • DAT Souza
        • Gaviola G.C.
        • Silverman S.G.
        • Mayo-Smith W.W.
        • Lee L.K.
        Trainee and attending perspectives on remote radiology readouts in the era of the COVID-19 pandemic.
        Academic Radiology [Internet]. 2020 Aug; 27 (Available from): 1147-1153
        • Li C.H.
        • Rajamohan A.G.
        • Acharya P.T.
        • Patel V.
        • Go J.L.
        • Lui C.-S.J.
        • et al.
        Virtual read-out: radiology education for the 21st century during the COVID-19 pandemic.
        Academic Radiology [Internet]. 2020 Jun; 27 (Available from): 872-881
        • Alvin M.D.
        • George E.
        • Deng F.
        • Warhadpande S.
        • Lee S.I.
        The impact of COVID-19 on radiology trainees.
        Radiology [Internet]. 2020 Aug; 296 (cited 2021 Jun 13) (Available from): 246-248https://doi.org/10.1148/radiol.2020201222
        • Chong A.
        • Kagetsu N.J.
        • Yen A.
        • Cooke E.A.
        Radiology residency preparedness and response to the COVID-19 pandemic.
        Acad Radiol. 2020; 27 (Available from:) (Jun [cited 2021 Jun 13): 856-861
        • Virarkar M.
        • Jensen C.
        • Javadi S.
        • Saleh M.
        • Bhosale P.R.
        Radiology education amid COVID-19 pandemic and possible solutions.
        J Comput Assist Tomogr [Internet]. 2020 Jul; 44 (Available from): 472-478
        • Nadgir R.
        Teaching remotely: educating radiology trainees at the workstation in the COVID-19 era.
        Academic Radiology [Internet]. 2020 Sep; 27 (Available from): 1291-1293
        • Larocque N.
        • Shenoy-Bhangle A.
        • Brook A.
        • Eisenberg R.
        • Chang Y.M.
        • Mehta P.
        Resident Experiences With Virtual Radiology Learning During the COVID-19 Pandemic.
        Academic Radiology. 2021 May; 28: 704-710https://doi.org/10.1016/j.acra.2021.02.006
        • Harari A.A.
        • Conti M.B.
        • SAJ Bokhari
        • Staib L.H.
        • Taylor C.R.
        The role of report comparison, analysis, and discrepancy categorization in resident education.
        American Journal of Roentgenology [Internet]. 2016 Dec; 207 (Available from): 1223-1231https://doi.org/10.2214/AJR.16.16245
        • Awan O.A.
        Virtual radiology readouts after the COVID-19 pandemic.
        American Journal of Roentgenology. 2021 Feb 17; (cited 2021 Apr 19) (Available from)AJR.21.25607https://doi.org/10.2214/AJR.21.25607