Evaluations 101: Designing Effective Test Questions

🔎 Background:

A learning needs assessment for Nursing Professional Development Specialists at CHOP revealed a learning gap in creating robust evaluations that measure learning outcomes. My manager approached my colleague and me and asked if we’d be interested in partnering to design and develop curriculum to address this need. Dr. Kai Nissen and I collaborated over a six-month period, creating a class together and developing all of its components. Dr. Nissen left the organization prior to the first class date, so Dr. AnneMarie Monachino joined me as co-facilitator. We offered the course twice in September 2023.

🎯Aims:

• Define the criteria for designing robust evaluations.

• Recognize and implement Bloom's Taxonomy verbs in course and evaluation design.

• Align learning outcome design with evaluation design.

• Apply learning by developing an appropriately written question.

💡 Methods:

1. Prior to attending class, learners completed a pre-assessment. We used this data to personalize and differentiate the classroom experience to meet learners’ needs.

2. Learners attended a two-hour instructor-led session. This engaging, interactive class featured minimal didactic content. Instead, it focused on collaboration, discussion, kinesthetic activities, and small group work. We created a custom branding kit for the course as well as a dissemination plan to share it with staff.

3. Upon completion of class, learners completed a post-assessment. We used this data to measure educational outcomes of the course as well as plan for quality improvement.

📊 Results:

So far, 17 learners attended Evaluations 101 class. Comparing pre- and post-assessment data, learners’ scores increased an average of 14.42 points, and their average growth percentage increase was 35%. 94% of attendees agreed or strongly agreed with the following three statements:

• “My learning was enhanced by the facilitators.”

• “I understand how to apply what I learned on the job.”

• “This class was applicable to my role.”

When asked, “what are the most meaningful concepts you learned in this session?” learners provided positive feedback:

• “I really enjoyed this training - the length, how interactive it was, the participant guide, aligning learning outcomes, and applying.”

• “Length of and number of questions appropriate for hospital-based educational sessions was useful.”

• “The importance of identifying learning outcomes prior to designing a program or questions.”

📄 Conclusion:

Evaluations 101 was a pleasure to work on, especially as an end-to-end project where I served as an instructional designer and a facilitator. The data supports offering it again in the future. As such, we plan to facilitate the course again twice in January 2024 after making some improvements based on learner feedback. The course was also submitted to the CHOP Interprofessional Education (IPE) Committee for consideration as a workshop for the upcoming 7th Annual Education Day on January 17, 2024.