How to Measure CME Outcomes Beyond Level 3 (Moore's Pyramid in Practice)

Outcomes measurement is the single greatest challenge facing accredited CME providers today. In a 2024 survey by the Alliance for Continuing Education in the Health Professions, 78% of CME professionals identified measuring outcomes beyond learner satisfaction as their top operational difficulty. The reason is straightforward: most providers have mastered the lower levels of Donald Moore's outcomes framework but hit a wall when attempting to measure whether their activities actually changed clinical practice or improved patient health.

Moore's Expanded Outcomes Framework, commonly called Moore's Pyramid, provides a seven-level hierarchy for evaluating CME effectiveness. Most providers comfortably measure through Level 3 (declarative knowledge). Very few consistently measure at Level 5 (performance) or above. This article walks through each level practically, with specific measurement strategies and tools, and shows how interactive on-demand modules make higher-level measurement not just possible but built into the learning experience itself.

Moore's Seven Levels: A Practical Overview

Dr. Donald Moore Jr. developed his outcomes framework specifically for continuing education in the health professions. Unlike Kirkpatrick's four-level model, which was designed for corporate training, Moore's framework accounts for the unique regulatory environment, professional obligations, and patient safety implications of medical education. Understanding each level is essential before you can design measurement strategies that reach beyond the conventional ceiling.

Level 1: Participation

This is the simplest measurement: did the learner attend or complete the activity? Participation data includes attendance records, login timestamps, module completion rates, and credit claims. Every accredited provider already collects this data because it is required for credit reporting. The limitation is obvious: participation tells you nothing about whether learning occurred.

Level 2: Satisfaction

The classic post-activity evaluation survey. Did learners rate the activity favorably? Was the content relevant to their practice? Would they recommend it to colleagues? Satisfaction surveys are ubiquitous in CME and serve a useful quality control function. However, research consistently shows that learner satisfaction has minimal correlation with knowledge acquisition or behavior change. A highly entertaining lecture can score a 4.9 on satisfaction while producing zero measurable learning.

Level 3A: Declarative Knowledge

Can the learner recall facts and concepts presented in the activity? This is where the standard post-test lives. Multiple-choice questions that assess whether the learner can identify correct answers from a list of options. Most CME providers measure at this level and consider the job done. The problem is that knowing the right answer on a test does not predict whether the physician will apply that knowledge in clinical practice.

Level 3B: Procedural Knowledge

This is the first level where measurement becomes meaningfully harder. Procedural knowledge asks whether the learner knows how to apply the information in context. Instead of asking "Which medication is first-line therapy for X?" a procedural knowledge assessment presents a clinical scenario and asks the learner to determine the appropriate treatment plan, considering comorbidities, contraindications, and patient preferences. This distinction matters enormously for CME outcomes because procedural knowledge is a much stronger predictor of clinical behavior change than declarative recall.

Breaking Through the Level 3 Ceiling

The gap between Level 3 and Level 4 is where most CME outcomes measurement programs stall. Levels 1 through 3 can all be measured within the learning activity itself, using attendance logs, satisfaction surveys, and knowledge assessments. Starting at Level 4, you need to measure what happens after the learner completes the activity and returns to clinical practice. That requires different tools, longer timeframes, and fundamentally different data collection strategies.

Level 4: Competence

Competence means the learner can demonstrate the ability to apply knowledge in a simulated or controlled environment. This is distinct from performance (Level 5), which measures what the learner actually does in practice. Competence assessment answers the question: "Can this physician do it?" rather than "Does this physician do it?"

Interactive modules are uniquely suited to competence measurement because they can embed realistic clinical simulations directly into the learning experience. A branching case scenario where the learner must diagnose a patient, order appropriate tests, interpret results, and select a treatment plan is simultaneously a learning activity and a competence assessment. The learner's choices at each decision point generate data about their ability to apply the knowledge in a clinically realistic context.

Practical measurement strategies for Level 4 include:

• Case-based simulation assessments — Present multi-step clinical scenarios that require the learner to make sequential decisions. Score based on the clinical appropriateness of the decision pathway, not just the final answer.
• Unannounced standardized patient encounters — For procedural or communication skills, simulated patient interactions (whether in-person or virtual) provide direct competence evidence.
• Chart-stimulated recall — Present de-identified patient charts and ask learners to describe their clinical reasoning. This can be embedded in post-activity follow-up modules.

The shift from Level 3 to Level 4 is the shift from "Does the learner know the right answer?" to "Can the learner make the right decision when the situation is complex and ambiguous?" Interactive modules bridge this gap because they test decision-making, not just recall.

Level 5: Performance

Performance measurement asks whether the learner actually applies what they learned in their real clinical practice. This is the holy grail of CME outcomes measurement and the level that ACCME most wants to see evidence of. Measuring performance requires data from outside the learning activity itself, which is what makes it so difficult.

Effective Level 5 measurement strategies include:

• Commitment-to-change statements — Collected at the end of the activity, these ask learners to specify one or two concrete changes they will make in their practice. Follow-up surveys at 30 to 90 days assess whether those changes were implemented. Research shows that commitment-to-change instruments predict actual behavior change with moderate accuracy, particularly when the commitments are specific and actionable.
• Practice audit data — Partner with health systems or registries to compare pre-activity and post-activity clinical metrics for participating physicians. For example, if your activity targeted appropriate statin prescribing, measure statin prescription rates before and after the activity for learners versus non-learners.
• Self-reported practice change with verification — Follow-up surveys asking learners to describe specific cases where they applied the learning. While self-report has limitations, structured questions that ask for concrete examples produce more reliable data than generic "Did this activity change your practice?" questions.

Reaching Levels 6 and 7: Patient and Community Health

Level 6: Patient Health

Can you demonstrate that your CME activity improved patient outcomes? This level requires linking educational interventions to clinical results, an inherently complex attribution challenge. No single CME activity operates in isolation; patients are managed by multiple providers, and clinical outcomes are influenced by countless variables beyond physician knowledge.

Despite these challenges, practical approaches exist. The most rigorous involve partnering with health systems that can provide de-identified outcome data for patients treated by CME participants versus non-participants. For example, a CME activity on sepsis management could be evaluated by comparing sepsis mortality rates or time-to-antibiotic metrics at facilities where clinicians completed the training versus matched control facilities.

More accessible approaches include patient-reported outcome measures (PROMs) collected through follow-up surveys of participating clinicians. Ask physicians to report on specific patient outcomes related to the topic area, using structured instruments that minimize recall bias. While not as rigorous as registry data, PROMs provide directional evidence that can be aggregated across learners to identify trends.

Level 7: Community Health

The apex of Moore's Pyramid measures population-level health impact. Did your CME program reduce disease incidence, improve screening rates, or shift public health metrics in the communities served by your learners? This level is aspirational for most individual CME activities but achievable for large-scale, sustained educational programs.

Public health data sources, including CDC surveillance reports, state health department metrics, and population-based registries, can provide outcome data for Level 7 measurement. The key challenge is attribution: even if screening rates improved in a region where your CME activity had high penetration, can you credibly attribute the improvement to the education rather than other concurrent interventions?

The most defensible Level 7 measurement designs use quasi-experimental methods: comparing outcome trends in regions with high CME participation versus matched regions with low participation, controlling for confounding variables. This requires large learner populations and long observation windows, but it produces the strongest evidence of educational impact.

How Interactive Modules Enable Higher-Level Measurement

Traditional CME formats, whether live lectures, recorded webinars, or text-based enduring materials, are structurally limited in their ability to measure beyond Level 3. A lecture cannot assess competence. A recorded webinar cannot simulate clinical decision-making. A PDF monograph cannot track whether a learner paused to consider a case before answering.

Interactive on-demand modules, by contrast, are purpose-built for higher-level measurement. Here is how the format maps to Moore's framework.

• Level 4 (Competence) — Branching clinical scenarios embedded within the module test the learner's ability to apply knowledge in realistic contexts. Every decision point generates scored data. The module can adapt difficulty based on learner responses, providing a more precise competence assessment than static tests.
• Level 5 (Performance) — Commitment-to-change prompts, integrated directly after key learning moments rather than tacked on at the end, produce more specific and actionable commitments. AI-powered follow-up modules can re-engage learners at 30 and 60 days to assess whether commitments were implemented and collect practice change evidence.
• Levels 6-7 (Patient and Community Health) — Interactive platforms that integrate with clinical data systems can correlate individual learner engagement data with downstream patient outcomes, creating a continuous feedback loop between education and practice. While this integration is still emerging, the technical foundation exists today.

The critical advantage is that interactive modules generate measurement data as a byproduct of the learning experience. Learners do not need to complete a separate assessment instrument. Their interactions with the content, including their choices, hesitations, corrections, and time-on-task patterns, are themselves the outcome data. This is a fundamentally different paradigm from bolting on a post-test after a passive learning experience. For providers navigating ACCME's evolving expectations for enduring materials, this built-in measurement capability is a significant advantage.

Key Takeaways

Measuring CME outcomes beyond Level 3 is not optional in 2026. It is what separates providers who earn ACCME commendation from those who maintain baseline accreditation. Moore's Pyramid provides the framework; interactive on-demand modules provide the mechanism.

1. Most providers plateau at Level 3 (knowledge recall) because their formats cannot measure higher. Breaking through requires interactive formats that assess competence and generate performance data as part of the learning experience.
2. Level 4 (competence) is the most achievable next step. Case-based branching scenarios embedded in interactive modules measure clinical decision-making ability, not just factual recall. Start here if you are currently limited to post-test assessments.
3. Level 5 (performance) requires post-activity follow-up. Commitment-to-change instruments and structured follow-up surveys at 30 to 90 days provide evidence of practice change. Build these touchpoints into your activity design from the start.
4. Interactive modules make measurement a byproduct of learning. When learners make decisions within simulated clinical scenarios, their choices generate scored competence data automatically. No separate assessment instrument needed, no additional learner burden, and the data is directly applicable to MOC credit documentation.