SRFP117: Using Chart Reviews to Evaluate a Continuing Medical Education (CME) Program (Pearls)
Jane Zhao, MSc; Pardis Balari, BSc; Bayley Inniss; Bryan MacLeod, MD, BSc, CFPC, FCPC; Paul Taenzer, PhD; Leslie Carlin, PhD; Andrea Furlan
Abstract
Context
Moore’s Expanded Outcomes Framework is a 7 level framework commonly used to assess the outcomes of continuing medical education (CME) programs. Levels 1 to 5 are provider-level outcomes (participation, satisfaction, knowledge, competence, and performance) while levels 6 and 7 are patient- and community-level outcomes. Chart reviews are one method to assess level 5 (performance). ECHO Ontario Chronic Pain and Opioid Stewardship (“ECHO”) is a CME telementoring program that aims to increase capacity and access for primary care providers (PCPs) who manage patients with chronic pain.
Objective
This study uses chart reviews to evaluate ECHO’s impact on PCP performance and to discuss the feasibility of performing chart reviews for evaluation purposes as per Moore’s framework.
Study Design
Retrospective chart review
Setting
The practices of 12 primary care providers across Ontario who attended ECHO between June 2014 to August 2018. The inclusion criteria for PCPs was 1) attended a minimum of four ECHO sessions, and 2) clinic site must be approving of a site visit for chart reviews.
Population Studied
47 patient charts were included. For each patient chart reviewed, PCPs were asked to choose patients in their practice on whom they had used ECHO-taught knowledge. Inclusion criteria for patients was 1) have chronic pain and be managed by the ECHO-participating PCP, 2) was prescribed opioids during the time frame of the study, and 3) not presented during ECHO sessions. Informed consent was obtained prior to each site visit.
Outcome Measures
1) Increased use of ECHO-taught pain and opioid management strategies
2) Feasibility of using chart reviews to evaluate PCP performance following a CME activity
Results
25 (53%) patients were male and the average age was 59 (± 14) years. 24 (51%) patients had two or more pain diagnoses at baseline, with musculoskeletal pain being the most prevalent at 81%. 26 (55%) patients had comorbid mental health conditions and 13 (28%) had sleep disorders. Trends in results showed marginal, but non-significant, improvements in PCP performance after ECHO as indicated by increased use of pain and opioid management strategies.
Conclusions
Conducting chart reviews was a challenging method to assess provider performance. Future work to assess provider performance should include a qualitative component (in-depth interviews or focus groups) in order to complement the quantitative data and provide context for care and management decisions.
Moore’s Expanded Outcomes Framework is a 7 level framework commonly used to assess the outcomes of continuing medical education (CME) programs. Levels 1 to 5 are provider-level outcomes (participation, satisfaction, knowledge, competence, and performance) while levels 6 and 7 are patient- and community-level outcomes. Chart reviews are one method to assess level 5 (performance). ECHO Ontario Chronic Pain and Opioid Stewardship (“ECHO”) is a CME telementoring program that aims to increase capacity and access for primary care providers (PCPs) who manage patients with chronic pain.
Objective
This study uses chart reviews to evaluate ECHO’s impact on PCP performance and to discuss the feasibility of performing chart reviews for evaluation purposes as per Moore’s framework.
Study Design
Retrospective chart review
Setting
The practices of 12 primary care providers across Ontario who attended ECHO between June 2014 to August 2018. The inclusion criteria for PCPs was 1) attended a minimum of four ECHO sessions, and 2) clinic site must be approving of a site visit for chart reviews.
Population Studied
47 patient charts were included. For each patient chart reviewed, PCPs were asked to choose patients in their practice on whom they had used ECHO-taught knowledge. Inclusion criteria for patients was 1) have chronic pain and be managed by the ECHO-participating PCP, 2) was prescribed opioids during the time frame of the study, and 3) not presented during ECHO sessions. Informed consent was obtained prior to each site visit.
Outcome Measures
1) Increased use of ECHO-taught pain and opioid management strategies
2) Feasibility of using chart reviews to evaluate PCP performance following a CME activity
Results
25 (53%) patients were male and the average age was 59 (± 14) years. 24 (51%) patients had two or more pain diagnoses at baseline, with musculoskeletal pain being the most prevalent at 81%. 26 (55%) patients had comorbid mental health conditions and 13 (28%) had sleep disorders. Trends in results showed marginal, but non-significant, improvements in PCP performance after ECHO as indicated by increased use of pain and opioid management strategies.
Conclusions
Conducting chart reviews was a challenging method to assess provider performance. Future work to assess provider performance should include a qualitative component (in-depth interviews or focus groups) in order to complement the quantitative data and provide context for care and management decisions.
Jack Westfall
jwestfall@aafp.org 11/21/2021ECHO is widely used. but has not been adequately evaluated. Thanks for your work in this area. It is crucial to understand the best ways to improve clinical care and whether that should be done in-person and with practice coaching, or simply via virtual offerings. Nice work. thanks for sharing at NAPCRG.