Background: A new approach to testing operative technical skills, the Objective Structured Assessment of Technical Skill (OSATS), formally assesses discrete segments of surgical tasks using bench model simulations. This study examines the interstation reliability and construct validity of a large-scale administration of the OSATS.
Methods: A 2-hour, eight-station OSATS was administered to 48 general surgery residents. Residents were assessed at each station by one of 48 surgeons who evaluated the resident using two methods of scoring: task-specific checklists and global rating scales.
Results: Interstation reliability was 0.78 for the checklist score, and 0.85 for the global score. Analysis of variance revealed a significant effect of training for both the checklist score, F(3,44) = 20.08, P <0.001, and the global score, F(3,44) = 24.63, P <0.001.
Conclusions: The OSATS demonstrates high reliability and construct validity, suggesting that we can effectively measure residents' technical ability outside the operating room using bench model simulations.