The development, reliability, and validity of the Facilitator Assessment Tool: An implementation fidelity measure used in Parenting for Lifelong Health for Young Children.
Mackenzie Martin, Jamie M Lachman, Hugh Murphy and 2 others
PMID 36316789WHAT IT FOUND
The tool for checking how well facilitators deliver parenting sessions shows mixed reliability.
Assessors were mostly consistent when rating the same video twice, but often disagreed when rating together. The section counting praise and feedback was unreliable and should be dropped or modified.
Key findings
01Content validity consultations with trainers, experts, and assessors led to revisions that expanded the tool to 62 items across three subscales.
02Intra-rater reliability was generally acceptable, with overall percentage agreements ranging from 57.6% to 91.5% and ICCs from 0.52 to 0.94.
03Inter-rater reliability varied widely by country, with overall percentage agreements ranging from 18.1% to 74.0% and ICCs from 0.49 to 0.91.
STILL TO COME
How it was doneWhat they found
Read the rest of this summary
You get three full summaries a month, free, and we do not ask for a card. Search, the TL;DRs and your library stay unlimited either way.
What it does not show
The study used a small sample of 11 assessors and a limited number of videos, which means the results should be interpreted with caution. Assessor training was not completed in North Macedonia and Romania due to scheduling conflicts, which may have affected how consistently the tool was used. Videos were selected purposively rather than randomly, so they may not represent typical facilitator performance. Some items were left blank because the behaviour did not occur, which complicates the scoring and reliability analysis.
Declared interests
JML and FG are co-developers of the Parenting for Lifelong Health for Young Children programme and co-founders of the Parenting for Lifelong Health initiative. JML receives occasional fees for training and supervision. The University of Oxford receives research funding for studies on the programme. MM, HM, and HMF declare no competing interests.
The easy way to misread this
Do not assume the tool is ready for high-stakes evaluation of facilitator performance. The inter-rater reliability was low in some countries, and the Frequency Subscale was found to be unreliable and recommended for removal, meaning different assessors may score the same facilitator very differently.