Leading an Effective Intern Prep Course (IPC)

Tenets of Evaluation

“There are no mistakes or failures, only lessons.” - Denis Waitley

Key Learning Points

  1. Evaluation practicess must be aligned with stated goals and objectives.

  2. The Kirkpatrick Evaluation Model serves as a guide to assessing students and programs on multiple levels, with level one representing learner reactions and level four the impact of student learning on target outcomes.

  3. Results of evaluations should be used to inform revisions to your teaching sessions.

Abstract of many dull unsharpened pencils contrasted with one perfectly sharpened pencil

1. Overview


Evaluation of student performance and program effectiveness is an important step in the teaching and learning cycle. Evaluation allows educators to assess program effectiveness, identify areas for improvement, and if using pre-curriculum assessments, gauge learners’ prior knowledge. Evaluative procedures must be informed by stated goals and objectives, emphasizing the key points learners should take from your teaching session.

Evaluation can take many forms, from simple satisfaction surveys to more intense direct observation and feedback. The methods of assessment should be tailored to your desired outcomes. Here we will present one highly regarded paradigm of evaluation, the Kirkpatrick Evaluation Model, as a way to conceptualize assessment practices.

 

2. Guiding Principles

Kirkpatrick’s Evaluation Model

The Kirkpatrick Evaluation Model is a widely used paradigm to discuss evaluation of teaching effectiveness. The impact of a program can be assessed on four levels, with level one, reaction, reflecting learner satisfaction or dissatisfaction, to level four, which assesses outcomes based on goals and objectives set forth by the program. Throughout this module, we will use an example of a teaching session on managing sepsis to illustrate these principles.

Level One: Reaction

Kirkpatrick's Evaluation Model

Level one evaluation assesses how learners felt about the teaching session. For example, did they enjoy the session? Were they satisfied with the content? Do they believe the content is relevant?

Often, this feedback is gathered with immediate post-session evaluation forms and is relatively easy to analyze for future improvements. Reactionary feedback does not assess participant learning or behavioral change.

Example. For a teaching session on managing sepsis, you might ask the following of participants:

  1. On a scale of one (did not enjoy at all) to five (enjoyed very much), indicate how much you enjoyed this session on managing sepsis.
  2. How relevant do you believe this session on managing sepsis is for your internship preparation?

Level Two: Learning

Level two evaluation practices assess how student knowledge has changed after a program. That is, did they learn what they were supposed to learn?

Learning evaluation is often achieved through pre- and post-program assessments of knowledge. This can take the form of self-assessment or written or verbal quizzes or tests, for example. This provides more meaningful data than reactionary feedback, however, it can be difficult to develop a measure for abstract skills and can be costly to implement. 

Example. For a teaching session on managing sepsis, you could assess individual learning with a brief written quiz. For group assessment, you could use an audience response system to test knowledge-based questions.

Level Three: Behavior

Level three evaluations assess the degree to which learners applied the knowledge gained from the session to change their behavior. For example, did students apply the concepts discussed in class to a real-world scenario?

Evaluation for behavior change often requires observation of a learner in their environment to determine whether they are employing skills and knowledge taught in the session. This is highly meaningful assessment data, but is challenging to implement in the context of a single teaching session.

Example. To determine whether students can apply knowledge gained in a teaching session on sepsis, one could observe a student’s approach to managing sepsis in a real-world scenario. For example, when presented with a septic patient, did they manage fluid resuscitation, antimicrobials, and source control appropriately? In the absence of real-world assessment, simulation can be used as a proxy. 

Level Four: Results

Evaluation of results assesses the impact of the enhanced learner skill on their environment. For example, how are patient outcomes affected by learner skill development? How are team dynamics or patient satisfaction impacted by communication skill development?

This is the highest level of evaluation and is the most difficult to implement, particularly on a large scale. With the lack of longitudinal contact with your learners in the Internship Preparation Course, this level of evaluation is unlikely to be achieved.

Example. If students are being followed longitudinally, one could assess the outcomes of the septic patients managed by students participating in the teaching session on sepsis. Does their knowledge of sepsis management translate to improved patient outcomes?

3. Practice

 

4. Apply

  1. Write two questions or statements you would like included on the student evaluation form for your teaching session.
  2. Write a brief description of how you plan to gather reactionary and learning assessments during your session.
  3. Develop one idea on how you could assess behavioral change from your learning session.

5. Wrap Up

Well planned evaluation procedures closely aligned with session goals and objectives are necessary to effectively assess student learning and to inform subsequent revisions of your teaching session. The Kirkpatrick Model can be used to plan assessment procedures to evaluate student learning on multiple levels.

Other Resources

Gross Davis, B. Tools for Teaching. 2nd edition. San Francisco, CA: Jossey-Bass; 2009.

Lang, J.M. On Course: A Week-by-Week Guide to your First Semester of College Teaching. Cambridge, MA: Harvard University Press; 2008.

Schuh, J.H., Biddix, J.P., Dean, L.A., and Kinzie, J. Assessment in Student Affairs. 2nd edition. San Francisco, CA: Jossey-Bass; 2016.

Skeff, K.M. and Stratos, G.A. Methods for Teaching Medicine. Philadelphia, PA: American College of Physicians Press; 2010.

References

Bree, K.K., Whicker, S.A., Fromme, H.B., Paik, S., and Greenberg, L. Resident-as-teachers publications: What can programs learn from the literature when starting a new or refining an established curriculum? Journal of Graduate Medical Education, 2014;237-48.

Kirkpatrick Partners, LLC. The Kirkpatrick Model. 2017. Retrieved from https://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model

Suski, L. Assessment Student Learning: A Common Sense Guide. 2nd edition. San Francisco, CA: Jossey-Bass; 2009.

 

Mentor Guide: Direct Observation
Mentor Guide: Post-course follow-up