Evaluation Failures

Kylie Hutchinson took her first course in evaluation in 1988 and was hooked. She won the Canadian Evaluation Society's Student Competition in 1990, made the leap into evaluation consulting in 1998, and has never looked back. In 2005 she was asked to deliver the Canadian Evaluation Society's Essential Skills course in British Columbia which she did until 2011. This focus on training sparked a new interest in evaluation capacity-building and she enrolled in Vancouver Community College's Instructor Development Program. By 2007 approximately 40% of her work was evaluation capacity-building. She continues to this day providing both her own private and sponsored workshops and webinars for clients such as the American Evaluation Association, Canadian Evaluation Society, and U.S. Centers for Disease Control. Since 2008, she has become increasingly interested in evaluation reporting, and has run a pre-conference workshop on the topic at AEA. Kylie acknowledges her roots are firmly rooted in the muddy trenches of evaluation practice versus academia. She has a passion for taking evaluation theory and making it accessible for evaluators to use immediately in their practice: this is evidenced by some of the free evaluation resources she has produced over the years including one-page reference sheets, a podcast, two videos, an eLearning course, a mobile learning course for senior decision-makers, and an evaluation glossary mobile app. She is also the author of An Innovative Guide to Evaluation Reporting and Survive and Thrive: Three Steps to Securing Your Program Sustainability

Foreword - Michael Quinn Patton Acknowledgments Introduction PART I. MANAGE THE EVALUATION Chapter 1. It's Not Me, It's You: The Value of Addressing Conflict Head On - Corey Newhouse Chapter 2. The Scope Creep Train Wreck: How Responsive Evaluation Can Go Off the Rails - E. Jane Davidson Chapter 3. The Buffalo Jump: Lessons After the Fall - Gail Vallance Barrington Chapter 4. Evaluator Self-Evaluation: When Self-Flagellation Is Not Enough - Emma Williams PART II. ENGAGE STAKEHOLDERS Chapter 5. That Alien Feeling: Engaging All Stakeholders in the Universe - Hallie Preskill Chapter 6. Seeds of Failure: How the Evaluation of a West African Agricultural Scale Up Project Went Awry - Thomas Archibald Chapter 7. I Didn't Know I Would Be a Tightrope Walker Someday: Balancing Evaluator Responsiveness and Independence - Rakesh Mohan Chapter 8. When National Pride Is Beyond Facts: Navigating Conflicting Stakeholder Requirements - Felix Muramutsa PART III. BUILD EVALUATION CAPACITY Chapter 9. Stars in Our Eyes: What Happens When Things Are Too Good to Be True - Jara Dean-Coffey PART IV. DESCRIBE THE PROGRAM Chapter 10. A "Failed" Logic Model: How I Learned to Connect With All Stakeholders - Chris Lovato Chapter 11. Lost Without You: A Lesson in System Mapping and Engaging Stakeholders - Kylie Hutchinson PART V. FOCUS THE EVALUATION DESIGN Chapter 12. You Got to Know When to Hold 'Em: An Evaluation That Went From Bad to Worse - Robert P. Shepherd Chapter 13. The Evaluation From Hell: When Evaluators and Clients Don't Quite Fit - Marla Steinberg PART VI. GATHER CREDIBLE EVIDENCE Chapter 14. The Best Laid Plans of Mice and Evaluators: Dealing With Data Collection Surprises in the Field - Jan Noga Chapter 15. Are You My Amigo, or My Chero? The Importance of Cultural Competence in Data Collection and Evaluation - Isaac D. Castillo Chapter 16. OMG, Why Can't We Get the Data? A Lesson in Managing Evaluation Expectations - Jennifer Bisgard and Mary Pat Selvaggio Chapter 17. No, Actually, This Project Has to Stop Now: Learning When to Pull the Plug - Karen Snyder Chapter 18. Missing in Action: How Assumptions, Language, History, and Soft Skills Influenced a Cross-Cultural Participatory Evaluation - Susan Igras PART VII. JUSTIFY CONCLUSIONS Chapter 19. "This Is Highly Illogical": How a Spock Evaluator Learns That Context and Mixed Methods Are Everything - Benoit Gauthier Chapter 20. The Ripple That Became a Splash: The Importance of Context and Why I Now Do Data Parties - Diana Tindall Chapter 21. The Voldemort Evaluation: How I Learned to Survive Organizational Dysfunction, Confusion, and Distrust - Lisa O'Reilly PART VIII. REPORT AND ENSURE USE Chapter 22. The Only Way Out Is Through - Stephanie Evergreen Conclusion

"Real failures of real evaluators . . . no additives or preservatives." -- Robert Renaud "Evaluation Failures is an impressive evaluation guide that highlights several real-world evaluation pitfalls (as presented by experts in the field) and provides useful suggestions as to how these may be skillfully navigated. It gives both new and seasoned evaluators permission to laugh at themselves as they discover more about conducting a successful evaluation. The text introduces lessons from evaluation failures as important tools on the road to evaluation success!" -- Sandra Schrouder "This collection is useful for evaluators that want to reflect on their own work, successes and failures, and improve their craft. It is inspiring to see how the evaluators we all look up to have identified and addressed their challenges. We can all do the same to improve our craft! This collection of stories speaks volumes to the skills that we only learn from years working in the field of evaluation. This collection of stories offers new and seasoned evaluators the opportunity to learn from the mistakes and failures that are all too common in the field of evaluation." -- Beverly L. Peters "This is the evaluation book we've been waiting for! A must-read for all learning and working in the field." -- Amanda M. Olejarski