Phase 5: Evaluating the Psychometric Characteristics of HCBS Measures
The purpose of phase #5 is to conduct a multi-site evaluation of reliability, validity, item discrimination, and sensitivity to change of prioritized HCBS measures. The psychometric requirements addressed in this study are based on the Blueprint for the CMS Measures Management System (CMS, 2016), NQF Scientific Acceptability of Measure Properties (NQF, 2013), and related scientific standards. Multiple aspects of reliability and validity will be evaluated. This study will also attempt to assess sensitivity to change of selected items. An additional goal of the study is to identify and test measures for which data can be collected from in- person, proxy, and/or administrative data sources.
This project phase will build on previous investigations conducted in RTC/OM, incorporating stakeholders’ feedback on NQF domains and subdomains in phase #1; items and measures mapped onto prioritized HCBS domains and subdomains in phase #2; procedures used to administer different measures will be based on evidence gathered in phase #3; and new measures developed and tested for usability and feasibility in phase #4.
A thorough psychometric evaluation of a multitude of measures across prioritized NQF domains and disability populations is a unique and much needed effort in the field of HCBS outcome measurement, including community inclusion or legal and human rights. To date, the large majority of work of this nature has focused on health and medical outcomes with the assessment of HCBS outcomes for people with disabilities quite limited with little evidence of the psychometric properties of such measures. Evaluations in the area that have been conducted thus far have been at the level of entire instruments or subscales, rather than at the measure level (Smith & Ashbaugh, 2001; Lakin et al., 2008). In spite of the paucity of evidence supporting the technical adequacy of existing measures, states and policy-makers are using them to make critical decisions that have a significant impact on the lives of people with disabilities.
Given the scope of this project, it is not possible to examine measures in all domains identified by the NQF. The measures will be prioritized based on stakeholder feedback as well as the following feasibility factors related to the extent that measures are: (1) commonly used across instruments; (2) relevant across populations; (3) possess greatest possibility of obtaining data from individual, proxy or administrative data sets; (4) have highest likelihood of being psychometrically sound; (5) demonstrate the highest utility across stakeholder groups; and (6) are person-centered. A special focus will be put on those domains where there is a dearth of measures, but are rated as of high priority.
The selected measures will be tested at a national level with participants from all five diasabilty groups. At the end of phase #5, the examined measures will be evaluated based on their usability, feasibility and psychometric properties and based on NQF standards recommended for NQF for endorsement. Those measures recommended will be: (1) usable and feasible to stakeholders; (2) have satisfactory established thresholds for reliability and validity; (3) able to discriminate between people; and (4) sensitive to change (for those measures for which this characteristic is applicable).