Reliability of a script agreement test for undergraduate speech-language therapy students.
Angélica Pilar Silva Ríos, Manuel Nibaldo Del Campo Rivas, Patricia Katherine Kuncar Uarac and 1 others
PMID 37970957WHAT IT FOUND
Experts rated clinical scripts with low overall consistency, meaning answers varied significantly between therapists.
While agreement was acceptable for some diagnostic tasks, it was poor for study scripts in Audiology and Vestibular areas. This test may not yet reliably measure clinical reasoning in students.
Key findings
01The overall reliability of the script corpus was low, with a Cronbach's alpha of 0.67.
02Inter-observer agreement among experts was also low, with a Fleiss's Kappa of 0.29 for the entire corpus.
03Reliability varied by area and task, with particularly poor results for study scripts in Audiology and Vestibular (alpha 0.34) and low agreement in Voice and Cognition areas.
STILL TO COME
How it was doneWhat they foundWhat it means for SLPs
Read the rest of this summary
You get three full summaries a month, free, and we do not ask for a card. Search, the TL;DRs and your library stay unlimited either way.
What it does not show
The sample size of experts was small (41 total, roughly 10 per area), which limits the generalizability of the reliability estimates. The study used convenience and snowball sampling, which may introduce selection bias. The scripts were not tested on students in this study, only on experts, so its actual utility for assessing student progress remains unproven. The low reliability in certain areas (like Audiology study scripts) suggests specific items were poorly constructed or ambiguous.
Declared interests
The study was funded by Proyecto de Innovación Educativa at Universidad Santo Tomás, Chile. No financial conflicts of interest with commercial entities were declared.
The easy way to misread this
Do not assume this script agreement test is a valid or reliable instrument for evaluating student clinical reasoning. The low inter-observer agreement among experts indicates that the test results are inconsistent and may not accurately reflect a student's ability.