Outcomes Standardization Project (OSP): Let’s Speak the Same Language!

Download this article as PDF

Jennifer Donnelly, MLIS
Chief Accreditation Officer
©CloudCME, February 21, 2020

 

Are you familiar with the Outcomes Standardization Project (OSP)? Started in 2018 by seven steering team members, the Outcomes Standardization Project works to create standard outcomes terminology and address a lack of accepted methods in measuring the impact of educational activities by aggregating and analyzing outcome measurement data. ACCME’s Criteria 11 requires, “The provider analyzes changes in learners (competence, performance, or patient outcomes) achieved as a result of the overall program’s activities/educational interventions.” Because it is a challenge to aggregate data for individual activities, analyzing the outcomes of one’s overall CE/CPD program seems like an impossible task! Regardless of how enormous this task seems, we must strive to find common ground in outcome terminology and methods to advance as a profession and implement improvements to our overall CE/CPD programs. The work of the OSP and its volunteers is providing a foundation to this end result.

Published online on February 13, 2020 in the Journal of European CME, 9:1, the article titled, “Outcomes Standardisation Project (OSP) for Continuing Medical Education (CE/CME) Professionals: Background, Methods, and Initial Terms and Definitions,” outlines the OSP’s consensus building methods for selecting the first twenty-five outcome terms and definitions. This process took place over a year of 1:1 interviews, focus groups, call for comments and much discussion between the OSP Steering Team (OSPST) and volunteer educational research scientists and CE/CME professionals.

 

The End Result: Outcome Terms and Definitions

The following table includes the final version of outcome terms and definitions developed thus far by the OSP. The OSP plans to expand upon these foundational outcome terms and definitions.

Term

Definition

 

 

Participation Funnel

Term used to describe the series of events experienced by an HCP from exposure to an available educational experience through to the request and fulfilment of credit (if available).

 

 

 

Intended Reach

Term used to indicate the number of unique HCPs to whom the availability of educational activities is being promoted.

 

 

Participant

This term, perhaps more than any other considered by the OSP Steering Team, has such varied application and understanding by the community that it SHOULD NOT be used in outcomes efforts or reporting without additional context. Absent of this additional context, the term creates ambiguity and confusion.

 

 

Start

Term used to describe the action an HCP takes to begin the core educational content/intervention. If an Activity is preceded with CE/CME front matter or a pre-test, a Start occurs AFTER an HCP has navigated through these items.

 

 

Learner

Term used to describe an HCP who Starts the core educational content/intervention. The term is designated only for individuals that have progressed beyond the CE/ CME front matter and pre-test (if available) and have started to consume/participate in the educational experience.

 

 

Completion

Term used to describe when an HCP has finished the core educational content/intervention. Importantly, whether a Learner chooses to participate in the post-test or evaluation that may follow the education activity does NOT impact completion.

 

 

Completer

Term used to describe an HCP that has finished the core educational content/intervention. Importantly, whether a Learner chooses to participate in the post-test or evaluation that follows the education activity does NOT impact completion.

 

 

Completion Rate

Term used to define the percentage of Learners that completed the core educational content/intervention. The Completion Rate is a ratio of one stage of the Participation Funnel and provides specific insights into the quality of the educational content and experience.

 

 

Learning Actions

Term used to describe the behaviour of a Learner while consuming/participating in the core educational content/ intervention.

 

 

Engagement

Term used to describe the learning actions or behaviours of an HCP while consuming/participating in the core educational content/intervention.

 

 

 

Moore’s Level 1, Participation

The first level in one established outcomes framework, Moore’s Level 1 emphasises the need to count the number of HCPs progressing through each stage of the Participation Funnel.

 

 

Moore’s Level 2, Satisfaction

The second level in one established outcomes framework, Moore’s Level 2 emphasises the need to measure the degree to which the expectations of the Learners about the setting and delivery of the CME activity were met.

 

 

Moore’s Level 3a, Declarative Knowledge

 

The third level in one established outcomes framework, Moore’s Level 3a emphasises the need to measure the changes in declarative knowledge that are associated with an educational intervention.

 

 

Moore’s Level 3b, Procedural Knowledge

 

The third level in one established outcomes framework, Moore’s Level 3b emphasises the need to measure the changes in procedural knowledge that are associated with an educational intervention.

 

 

Moore’s Level 4, Competence

 

The fourth level in one established outcomes framework, Moore’s Level 4 emphasises the need to measure the changes in competence that are associated with an educational intervention.

 

 

Moore’s Level 5, Performance

 

The fifth level in one established outcomes framework, Moore’s Level 5 emphasises the need to mea- sure the changes in performance that are associated with an educational intervention.

 

 

Moore’s Level 6, Patient Health

 

The sixth level in one established outcomes framework, Moore’s Level 6 emphasises the need to measure the changes in patient health outcomes that are associated with an educational intervention.

 

 

Moore’s Level 7, Community Health

 

The seventh level in one established outcomes framework, Moore’s Level 7 emphasises the need to measure the changes in community health outcomes that are associated with an educational intervention.

 

 

Assessment

Term used to define the measurement of changes in knowledge, competence, performance or healthcare outcomes that are associated with the planned educational intervention.

 

 

Pre-test

 

Term used to define the measures (data from question(s) or data collection) BEFORE educational content is presented. Pre-tests can measure baseline knowledge, competence, present or anticipated behaviour, experienced or observed health outcome, or other topics.

 

 

Post-test

 

Term used to define the measures (data from question(s) or data collection) AFTER educational content is presented. Post- tests can measure resultant knowledge, competence, present or anticipated behaviour, experienced or observed health outcome, or other topics. This is typically intended as a measure of immediate change or impact.

 

 

First Post-test Score

 

Term used to define a measure of a Learner’s performance on his or her first attempt at a post-test, before any feedback or additional rational is provided.

 

 

Final Post-test Score

 

Term used to define a measure of a Learner’s performance on their final attempt at a post-test. With each repeated attempt at a post-test a Learner’s experience with the test evolves. The final post-test score is a measure of what a Learner was able to achieve through this evolving experience.

 

 

Evaluation

 

Term used to define the measurement of a Learner’s satisfaction with the content and learning experience and/or the perception of bias within the activity. Evaluations can provide a far richer understanding of learning and impact beyond correct/incorrect test questions.

 

 

Follow-up Assessments/Evaluation

 

Term used to define the data collected in the days, weeks or months following an educational experience. While Assessments or Evaluations are typically designed to make a measurement immediately after a learning experience, Follow-up Assessments or Evaluations are designed to make measurements over time.

Brian S. McGowan, Anthia Mandarakas, Sue McGuinness, Jason Olivieri, Karyn Ruiz-Cordell, Greg Salinas & Wendy Turell (2020) Outcomes Standardisation Project (OSP) for Continuing Medical Education (CE/CME) Professionals: Background, Methods, and Initial Terms and Definitions, Journal of European CME, 9:1, 1717187, DOI: 10.1080/21614083.2020.1717187

 

Summary

How to achieve outcomes has long perplexed CE/CPD professionals. With increasing activity development, CE/CPD teams are stretched with time to truly develop an outcomes-based educational program. Least we not forget, partnerships with QI/QA departments, risk management department, C-suite leadership involvement and support are key in developing outcome measurements that demonstrate the impact of one’s overall CE program. The terms and definitions that the OSP has developed is a start in creating a foundation that leads to common ground among CE/CPD professionals. To learn more about the OSP, click here.

  

This article represents the insight and opinions by CloudCME®, a technology company that provides a platform for managing all aspects of an accredited continuing education (CE) department. Accredited Continuing Education Providers are solely responsible for ensuring compliance with accrediting bodies, providerships and approving bodies criteria.