Background:
Objective Structured Clinical Examinations (OSCEs) are critical for assessing clinical competencies in medical education. Ensuring their validity and reliability is essential for high-stakes decisions. This study retrospectively evaluates the psychometric quality of four OSCE stations completed by second-year medical students in the Practice of Medicine (POM) module at the University of Medicine and Pharmacy at Ho Chi Minh City (UMP-HCMC), focusing on both station-level and item-level quality indicators.

Methods:
A cross-sectional analysis was conducted using data from 406 second-year students who completed a four-station OSCE and 6 checklists. Each station employed structured checklists and global rating scales. The Borderline Regression Method (BRM) was used to determine passing score for each station. In accordance with AMEE Guide Number 49, the psychometric quality of each station was evaluated using several indicators, including the coefficient of determination (R²), inter-grade discrimination, Cronbach’s alpha, and between-group variation. Additionally, item-level analysis was performed using item difficulty and item discrimination indices.

Results:
Across all stations, R² values remained below the optimal 0.5 threshold. The cardiac examination, abdominal, anterior lung and stations demonstrated relatively stronger psychometric properties, with nearly acceptable R² values (0.45–0.49), internal consistency (α = 0.66, 0.67 and 0.68, respectively) and acceptable intergrade discrimination. However, the posterior lung station showed the weakest R² (0.26), lowest intergrade discrimination, and a high percentage of overly easy and poorly discriminating items. Item-level analysis confirmed widespread deficiencies in the lung examination checklists.

Conclusion:
This study marks the first structured psychometric evaluation of OSCE stations at UMP-HCMC. Key recommendations include revising underperforming stations, refining global rating criteria, and enhancing examiner calibration. By integrating station- and item-level analyses, this approach provides a robust framework for improving OSCE quality, enhancing validity and reliability, and serving as a model for broader assessment improvement.

Keywords: Psychometrics, OSCEs, AMEE Guide No, 49, medical education.