Evaluating Fluency in Aphasia: Fluency Scales, Trichotomous Judgements, or Machine Learning.
Jeet Metu, Vishal Kotha, Argye E Hillis
PMID 38425350WHAT IT FOUND
Experienced speech-language pathologists judged fluent, non-fluent, or mixed aphasia speech very consistently when they used overall impressions.
The WAB-R fluency scale was less consistent, and machine learning did not reliably match non-fluent judgements.
Key findings
01Experienced SLPs agreed almost perfectly when classifying speech as fluent, non-fluent, or mixed (Kappa 0.94), while agreement using the WAB-R fluency score was moderate (Kappa 0.49).
02The algorithm agreed with SLPs on fluent speech in 81.8% of samples (Kappa 0.65) but on non-fluent speech in 63.6% (Kappa 0.24).
03For three samples, SLPs disagreed on whether speech was fluent or non-fluent when using the WAB-R fluency score.
STILL TO COME
How it was doneWhat they foundWhat it means for SLPs
Read the rest of this summary
You get three full summaries a month, free, and we do not ask for a card. Search, the TL;DRs and your library stay unlimited either way.
What it does not show
The validation set was only 11 samples, so the agreement estimates are based on a small number of cases. Training videos came from YouTube, and the paper says there was no independent evidence that the control speakers had no neurological or psychological disease. The language tasks in the YouTube training videos differed across samples and might have influenced fluency ratings. The paper tested only two machine-learning algorithms, trained on relatively small numbers of samples. The SLP raters were all from Johns Hopkins University School of Medicine and were very experienced with the WAB-R, so their agreement may not represent all SLPs. The paper notes that fluency as a construct is debatable.
Declared interests
The authors report no competing interests to declare.
The easy way to misread this
Do not conclude that machine learning can replace SLP fluency judgements. It matched SLPs on fluent speech in 81.8% of samples, but on non-fluent speech in only 63.6%.