Evaluating inter- and intra-rater reliability in assessing upper limb compensatory movements post-stroke: creating a ground truth through video analysis?
Lena Sauerzopf, Celina G Chavez Panduro, Andreas R Luft and 7 others
PMID 39702329WHAT IT FOUND
Experienced therapists agreed moderately well when rating specific arm movements in stroke videos, but disagreed heavily on overall smoothness and coordination.
These wide gaps mean therapist video ratings alone are not yet reliable enough to train automated assessment tools.
Key findings
01Therapists showed moderate to excellent agreement when rating individual phases of a drinking task, but poor reliability when judging global movement smoothness and interjoint coordination.
02The confidence intervals for reliability scores were very wide, indicating that the observed agreement might not be replicable in other groups of therapists.
03The authors concluded that current therapist video ratings are not viable for establishing a ground truth for machine learning algorithms without further training and calibration.
STILL TO COME
How it was doneWhat they foundWhat it means for PTsWhat it means for OTs
Read the rest of this summary
You get three full summaries a month, free, and we do not ask for a card. Search, the TL;DRs and your library stay unlimited either way.
What it does not show
The patients shown had mild to moderate impairments (Fugl-Meyer scores 46-66), so results do not apply to severe stroke cases. The study was likely underpowered, leading to very wide confidence intervals that make the reliability estimates uncertain. Face pixelation obscured the wrist and hand during the drinking phase, increasing 'not assessable' ratings. The rating scale lacked clear definitions for the upper limit of compensation, leading to varied interpretations among therapists. Markers were placed on patients' bodies, which may have influenced their movement performance.
Declared interests
Funded by the DFF Futures Fund (ZHAW Digital) and ZHAW Zurich University of Applied Sciences.
The easy way to misread this
Do not assume that experienced therapists can reliably assess movement smoothness or coordination from 2D videos. The study found poor reliability for these global characteristics, meaning a therapist's rating of 'smooth' vs 'clumsy' is highly subjective and inconsistent.