Presentation Authors: Daniel Nethala*, David Mikhail, Shannon Smith, Zachary Kozel, Lee Richstone, Jessica Kreshover, Lake Success, NY
Introduction: The need for critical evaluation of resident surgical performance has become a necessity to evaluate competence. Despite the impetus for timely intra-operative feedback, there is a paucity of data in the literature assessing the surgical performance and autonomy of urologic residents and the accuracy of self-evaluations. To study this, we compared the perceived autonomy, performance, and complexity scores of common urologic procedures between residents and attendings using a real-time smart phone application.
Methods: We utilized the System for Improving Procedural Learning (SIMPL) mobile application from 2/18-6/18. Evaluations were logged post-operatively assessing the autonomy, performance and complexity perceived by both the resident and attending. The validated Zwisch scale was used to assess autonomy/supervision; performance was a 5-point Likert scale (1=critical deficiency; 5=exceptional), and complexity by tertile (easy, medium or hard). Concordance was defined as identical evaluations by the resident and attending. Discordance was calculated by subtracting the attending score from the resident's.
Results: A total of 294 evaluations were completed; 70 of which were excluded for lack of pairing. We analyzed 224 evaluations (112 paired evaluations). Performance data was missing from 11% of the evaluations. Concordance of autonomy, performance, & complexity was seen in 43%, 55%, and 65% of cases respectively. Of the 64 cases with discordant autonomy scores, there was relatively equal discordance rates between the resident and attending perception of the assistance provided. Of the discordant evaluations of performance and complexity (n=45 and 39 respectively) residents mostly believed themselves to perform worse and thought cases were less complex than the attending perceived.
Conclusions: Evaluations by urologic residents & attendings showed significant concordance rates in assessment of the case complexity, but lower than expected in autonomy and performance. Much of the discordance in these variables came from the resident undervaluing their performance and autonomy during a case as well as underestimating the complexity of the case. More data is needed to identify the effect of procedure type and level of training on the concordance rates to better develop and evaluate training opportunities.