Examining the Validity of Mental Health Clinicians' Self-Reported Treatment Targets on a Measure of Usual Care
Presentation Type
Oral Presentation
Abstract/Artist Statement
Several decades of research have been spent identifying and testing evidence-based therapies (EBTs), but there is currently very little research that examines therapeutic practices within usual care. The lack of understanding in this area has been implicated as a barrier to the successful implementation of EBTs into usual mental health care settings. The Monthly Treatment Progress Summary (MTPS) is a measure developed to enable monthly tracking of intervention strategies and content within a statewide system of children’s mental health care. Although a growing body of research exists examining the reliability and validity of the treatment practice and treatment progress sections of the MTPS, less research has been conducted on the treatment target section. Specifically, no information exists on the validity of the treatment target section, which is a significant limitation of the measure and has implications for gaining a full picture of usual care treatment. The purpose of the current study is to assess the validity of the treatment target section of the MTPS. The study aims to evaluate whether clinicians’ self-reported treatment targets are consistent with their treatment objectives in session, as judged by trained coders reviewing audiotapes of therapy sessions. The current study also examines patterns of clinicians’ self-reported treatment targets in order to examine tendencies to over- or underreport targets. Data for this project are a set of 47 audio-recorded therapy sessions and the corresponding MTPS’s that were completed for each case, as part of a larger treatment effectiveness project (Weisz et al., 2012). The sessions are coded for therapist focus on treatment targets and the results will be compared to the therapist-endorsed targets on the MTPS. Although analyses are currently underway, preliminary analysis of interrater reliability among coders is high, with 79% of endorsed targets coded at a reliability of .60-1.0 (moderate to excellent). The findings of this study will add to our current knowledge of the validity of the MTPS as a measure of specific targets on which therapists focus in treatment. The need for valid, reliable, and pragmatic measures of mental health services is high (Glasgow & Riley, 2013), given the increased demand for accountability within mental healthcare settings. Although the MTPS has the potential to shed light on therapeutic practices and foci of usual care, it is important to understand therapists’ tendencies to over- or underreport specific treatment targets on such a measure. Thus, by providing insights into the accuracy of therapists’ self-reported treatment content, the results of this study could help to inform the structure and goals of future service delivery efforts.
Examining the Validity of Mental Health Clinicians' Self-Reported Treatment Targets on a Measure of Usual Care
UC Ballroom, Pod #1
Several decades of research have been spent identifying and testing evidence-based therapies (EBTs), but there is currently very little research that examines therapeutic practices within usual care. The lack of understanding in this area has been implicated as a barrier to the successful implementation of EBTs into usual mental health care settings. The Monthly Treatment Progress Summary (MTPS) is a measure developed to enable monthly tracking of intervention strategies and content within a statewide system of children’s mental health care. Although a growing body of research exists examining the reliability and validity of the treatment practice and treatment progress sections of the MTPS, less research has been conducted on the treatment target section. Specifically, no information exists on the validity of the treatment target section, which is a significant limitation of the measure and has implications for gaining a full picture of usual care treatment. The purpose of the current study is to assess the validity of the treatment target section of the MTPS. The study aims to evaluate whether clinicians’ self-reported treatment targets are consistent with their treatment objectives in session, as judged by trained coders reviewing audiotapes of therapy sessions. The current study also examines patterns of clinicians’ self-reported treatment targets in order to examine tendencies to over- or underreport targets. Data for this project are a set of 47 audio-recorded therapy sessions and the corresponding MTPS’s that were completed for each case, as part of a larger treatment effectiveness project (Weisz et al., 2012). The sessions are coded for therapist focus on treatment targets and the results will be compared to the therapist-endorsed targets on the MTPS. Although analyses are currently underway, preliminary analysis of interrater reliability among coders is high, with 79% of endorsed targets coded at a reliability of .60-1.0 (moderate to excellent). The findings of this study will add to our current knowledge of the validity of the MTPS as a measure of specific targets on which therapists focus in treatment. The need for valid, reliable, and pragmatic measures of mental health services is high (Glasgow & Riley, 2013), given the increased demand for accountability within mental healthcare settings. Although the MTPS has the potential to shed light on therapeutic practices and foci of usual care, it is important to understand therapists’ tendencies to over- or underreport specific treatment targets on such a measure. Thus, by providing insights into the accuracy of therapists’ self-reported treatment content, the results of this study could help to inform the structure and goals of future service delivery efforts.