Evaluation for Health Policy and Health Care

Steven H. Sheingold, Ph.D. is the Director of the Division of Health Financing Policy, Office of the Assistant Secretary for Planning and Evaluation (ASPE), Department of Health and Human Services. His areas of responsibility include economic and policy analysis of Medicare's payment systems, evaluation strategies, analysis of competition in insurance and provider markets, and economic issues related to health care and pharmaceutical markets. Prior to joining ASPE, Dr. Sheingold held several managerial and senior analyst positions within the Centers for Medicare and Medicaid Services and the Congressional Budget Office. As an adjunct, he has taught classes in health policy, health services research, and statistics at George Mason University and George Washington University. He has published articles concerning reimbursement systems; technology assessment and cost effectiveness analyses; value based purchasing programs; the use of evidence for health policymaking; and the impact of social risk factors on quality of care in journals such as Health Affairs, New England Journal of Medicine, Medical Care, and the Journal of Health Policy, Politics and Law. He holds a Ph.D. in Economics from the Pennsylvania State University. Anupa Bir, ScD MPH is the Senior Director of the Center for Advanced Methods Development at RTI International. A health economist by training, much of her work has focused on the well-being of vulnerable populations and aligning incentives within various systems, including the welfare, child welfare, corrections, and health systems, to improve well-being. Dr. Bir currently leads several contracts to evaluate complex health and social policy interventions. These interventions include innovative workforce interventions to improve access to quality health care, interventions that offer financial incentives for asset development, and interventions that improve communication and family strength during stressful circumstances like incarceration and reentry. Within health care and health policy, she leads evaluations of State Innovation Models, efforts funded by the Centers for Medicare and Medicaid Innovation to accelerate the transition to value-based payment models in 11 states. She also leads meta-evaluation work to understand the lessons from state Medicaid demonstrations to improve service delivery for those with substance use disorders or serious mental illness. She holds an MPH from Yale University School of Public Health and a doctoral degree in international health economics from the Harvard TH Chan School of Public Health.

List of Figures and Tables Preface Acknowledgments About the Editors PART I. SETTING UP FOR EVALUATION Chapter 1. Introduction Background: Challenges and Opportunities Evaluation and Health Care Delivery System Transformation The Global Context for Considering Evaluation Methods and Evidence-Based Decision Making Book's Intent Chapter 2. Setting the Stage Typology for Program Evaluation Planning an Evaluation: How Are the Changes Expected to Occur? Developing Evaluations: Some Preliminary Methodological Thoughts Prospectively Planned and Integrated Program Evaluation Summary Chapter 3. Measurement and Data Guiding Principles Measure Types Measures of Structure Measures of Process Measures of Outcomes Selecting Appropriate Measures Data Sources Looking Ahead Summary PART II. EVALUATION METHODS Chapter 4. Causality and Real-World Evaluation Evaluating Program/Policy Effectiveness: The Basics of Inferring Causality Defining Causality Assignment Mechanisms Three Key Treatment Effects Statistical and Real-World Considerations for Estimating Treatment Effects Summary Chapter 5. Randomized Designs Randomized Controlled Trials Stratified Randomization Group Randomized Trials Randomized Designs for Health Care Summary Chapter 6. Quasi-experimental Methods: Propensity Score Techniques Dealing With Selection Bias Comparison Group Formation and Propensity Scores Regression and Regression on the Propensity Score to Estimate Treatment Effects Summary Chapter 7. Quasi-experimental Methods: Regression Modeling and Analysis Interrupted Time Series Designs Comparative Interrupted Time Series Difference-in-Difference Designs Confounded Designs Instrument Variables to Estimate Treatment Effects Regression Discontinuity to Estimate Treatment Effects Fuzzy Regression Discontinuity Design Additional Considerations: Dealing With Nonindependent Data Summary Chapter 8. Treatment Effect Variations Among the Treatment Group Context: Factors Internal to the Organization Evaluation Approaches and Data Sources to Incorporate Contextual Factors Context: External Factors That Affect the Delivery or Potential Effectiveness of the Treatment Individual-Level Factors That May Cause Treatment Effect to Vary Methods for Examining the Individual Level Heterogeneity of Treatment Effects Multilevel Factors Importance of Incorporating Contextual Factors Into an Evaluation Summary Chapter 9. The Impact of Organizational Context on Heterogeneity of Outcomes: Lessons for Implementation Science Context for the Evaluation: Some Examples From Centers for Medicare and Medicaid Innovation Evaluation for Complex Systems Change Frameworks for Implementation Research Organizational Assessment Tools Analyzing Implementation Characteristics Summary PART III. MAKING EVALUATION MORE RELEVANT TO POLICY Chapter 10. Evaluation Model Case Study: The Learning System at the Center for Medicare and Medicaid Innovation Step 1: Establish Clear Aims Step 2: Develop an Explicit Theory of Change Step 3: Create the Context Necessary for a Test of the Model Step 4: Develop the Change Strategy Step 5: Test the Changes Step 6: Measure Progress Toward Aim Step 7: Plan for Spread Summary Chapter 11. Program Monitoring: Aligning Decision Making With Evaluation Nature of Decisions Cases: Examples of Decisions Evidence Thresholds for Decision Making in Rapid-Cycle Evaluation Summary Chapter 12. Alternative Ways of Analyzing Data in Rapid-Cycle Evaluation Statistical Process Control Methods Regression Analysis for Rapid-Cycle Evaluation A Bayesian Approach to Program Evaluation Summary Chapter 13. Synthesizing Evaluation Findings Meta-analysis Meta-evaluation Development for Health Care Demonstrations Meta-regression Analysis Bayesian Meta-analysis Putting It Together Summary Chapter 14. Decision Making Using Evaluation Results Research, Evaluation, and Policymaking Program/Policy Decision Making Using Evidence: A Conceptual Model Multiple Alternatives for Decisions A Research Evidence/Policy Analysis Example: Socioeconomic Status and the Hospital Readmission Reduction Program Other Policy Factors Considered Advice for Researchers and Evaluators Chapter 15. Communicating Research and Evaluation Results to Policymakers Suggested Strategies for Addressing Communication Issues Other Considerations for Tailoring and Presenting Results Closing Thoughts on Communicating Research Results Appendix A: The Primer Measure Set Appendix B: Quasi-experimental Methods That Correct for Selection Bias: Further Comments and Mathematical Derivations Propensity Score Methods An Alternative to Propensity Score Methods Assessing Unconfoundedness Using Propensity Scores to Estimate Treatment Effects Unconfounded Design When Assignment Is at the Group Level Index

"This text offers a general introduction to the process and methods used to conduct rigorous and timely evaluations of health policies and programs using real-world examples. It would make an excellent text for a program evaluation course." -- Brad Wright "A must read for anyone interested in monitoring and evaluation! The text does a great job addressing the important ingredients for a successful evaluation." -- Sandra Schrouder "Evaluating health policies and programs can be a very challenging process because the evaluation itself is so often an afterthought, leading to a variety of data issues that can produce biased results and poor policy decisions. This book provides an outstanding-yet highly accessible-overview of a wide variety of methods that evaluators can use to minimize these biases and generate robust evidence for decision-makers." -- Larry R. Hearld