As classroom observations become a common element of systematic teacher evaluations, it is essential to establish and maintain consistency among all the administrators, coaches, and other educators who are conducting the observations.
The Observation Engine is an affordable online tool that helps you prepare and certify school leaders to conduct classroom observations.
School systems use the Observation Engine to support training of both evaluators and teachers, to calibrate and achieve inter-rater reliability among their observers, and to certify educator evaluators to conduct classroom observations. It works with any observation protocol: homegrown, licensed from a protocol developer, or adapted from existing protocols.
Here are some solutions that Observation Engine offers.
Training and Online Learning
Using the Engine’s “Lessons” feature, school systems can accomplish the following.
- Create targeted, self-paced practice exercises for evaluators and teachers
- Select exemplar video clips that highlight your teaching standards
- Provide guided viewing experiences and immediate on-screen, formative feedback for evaluators
- Engage teachers in valuable conversations around teaching practice
The Observation Engine team can work with any training providers to seamlessly incorporate the Engine into training programs. We can also recommend technical assistance providers who can assist in customizing the Observation Engine to your protocol.
Here is an example scenario of how the Observation Engine can be used for training and practice.
Rossmor Public School District just completed a two-day, in person training on their observation protocol. To continue providing practice and targeted mini-learning modules, Rossmor uses the Observation Engine’s “Lessons” feature to author a number of exercises that their school principals can access on their own time and at their own pace. After the principals complete these exercises, they decide to assign the same modules to their teaching staff, to be used as learning exercises and to facilitate discussion around best practices during their PD days throughout the year.
The Observation Engine’s “Scoring Studies” can help school systems do the following.
- Facilitate group learning through discussion of areas of disagreement
- Support teamwork in generating target ratings (or master scores) for videos used in training and testing
- Deepen evaluators’ understanding of your observation rubric through consensus building activities
Here is an example scenario of how the Observation Engine can be used to calibrate evaluation staff.
Rossmor Public School District is conducting a pilot implementation of their new classroom observation rubric. After all lead coaches have been trained, Rossmor uses Observation Engine’s Scoring Study functionality to conduct a baseline “calibration study” to check the overall level of agreement among their coaches. At roughly 60% agreement among them, Rossmor’s evaluation team is able to pinpoint the specific rubric indicators that have the most disparity in terms of scoring. At the next PD meeting, they decide to focus on consensus building around those particular indicators.
After several more targeted PD sessions, a subgroup of coaches is selected to comprise Rossmor’s master scoring committee. Through the Engine, Rossmor conducts separate Scoring Studies to master score three sets of videos to use for practice and certification with the rest of their evaluation staff.
School leaders administer and take Practice and Certification Tests through the Observation Engine for these purposes.
- Practice scoring classroom lessons and receive individualized, comprehensive feedback
- Certify through secure online testing and take retests with unique videos
- Check for “rater drift” throughout the year
Here is an example scenario of how the Observation Engine can be used for certification.
After the initial pilot year of their classroom observation protocol, Rossmor Public School District decides to implement a high-stakes certification process for all of their administrators and peer evaluators. They administer an in-person proctored certification test through the Observation Engine. For those administrators who cannot attend in person, they offer a remote online option as well. Seventy six percent of their evaluators pass the certification test upon their first try. Those who did not pass are required to take a few online lesson modules before attempting a retest, for which Rossmor has master coded a unique set of videos.
Four months later, all administrators and peer evaluators take a short two-video quiz to check for “rater drift.” Those who do not meet the passing thresholds are required to review specific online learning modules based on the results of their quiz.
They are incredible from beginning to end—from the quality of their platform and statistical expertise to their customer service and project management.
—Jana Burk, Executive Director of Teacher and Leader Effectiveness of Tulsa Public Schools