Standard Setting

Gregory J. Cizek is Professor of Educational Measurement at the University of North Carolina, Chapel Hill. His background in the field of educa-tional assessment includes five years as a manager of licensure and certification testing programs for American College Testing (ACT) in Iowa City, Iowa, and 15 years of teaching experience at the col-lege level, where his teaching assignments have consisted primarily of graduate courses in educational testing, research methods, and statistics. He is the author of over 200 books, chapters, articles, conference papers, and reports. His books include Handbook of Educational Policy (Academic Press, 1998); Cheating on Tests: How to Do It, Detect It, and Prevent It (Lawrence Erlbaum, 1999); Setting Performance Standards: Concepts, Methods, and Perspectives (Lawrence Erlbaum, 2001); and Detecting and Preventing Classroom Cheating (Corwin Press, 2003). Dr. Cizek has served as an elected member and vice president of a local school board in Ohio, and he currently works with several states, organizations, and the U.S. Department of Education on tech-nical and policy issues related to large-scale standards-based testing programs for students in grades K-12. He began his career as an ele-mentary school teacher in Michigan, where he taught second and fourth grades. Michael B. Bunch is Vice-President of Measurement Incorporated, a test development and scoring company serving the statewide assessment community. Dr. Bunch joined MI in 1982 and has built the research and development division of that organization into a team of more than 60 project directors, psychometricians, editors, content specialists, and support staff. Prior to joining MI, Dr. Bunch was a senior professional with NTS Research Corporation, where he provided ESEA Title I evaluation technical assistance to state education agencies. From 1976 to 1978, he was a research psychologist with the American College Testing Program (ACT). Dr. Bunch has authored dozens of professional and technical reports on a variety of topics--many on standard setting--and has presented his work at national conferences, including annual meetings of the American Educational Research Association (AERA), National Council on Measurement in Education (NCME), American Psychological Association (APA), Council of Chief State School Officers (CCSSO), Education Commission of the States (ECS) and Adult Education Research Conference (AERC). He received his Ph.D. in psychological measurement from the University of Georgia.

SECTION I: FUNDAMENTALS OF STANDARD SETTING 1. Contemporary Standard Setting: An Enduring Need The Need to Make Decisions The Benefits of Standard Setting General Approaches to Standard Setting Current Contexts for Standard Setting 2. What is Standard Setting? Kinds of Standards Definitions of Standard Setting Policy Issues and Standard Setting Item Scoring Criteria and Total-Test Performance Standards Conclusions 3. Common Elements in Setting Preformance Standards Purpose Choosing a Standard Setting Method Performance Level Labels Performance Level Descriptions Key Conceptualizations Selecting and Training Participants Evaluating Standard Setting Providing Feedback to Participants Professional Guidelines for Standard Setting Evaluating Standard Setting Conclusions SECTION II: STANDARD SETTING METHODS 4. The Nedelsky Method Procedures for the Nedelsky Method Alternative Procedures and Limitations 5. The Ebel Method Procedures for the Ebel Method Alternative Procedures and Limitations 6. The Angoff Method and Angoff Variations Procedures for the Angoff Method Procedures for Angoff Variations Alternative Procedures and Limitations 7. The Direct Consensus Method Procedures for the Direct Consensus Method Alternative Procedures and Limitations 8. The Contrasting Groups and Borderline Group Methods Procedures for the Contrasting Groups Method An Example Using the Contrasting Groups Method The Borderline Group Method Alternative Procedures and Limitations 9. The Body of Work and Other Holistic Methods The Judgmental Policy Capturing Method The Dominant Profile Method The Analytic Judgment Method Summary Analysis of Three Holistic Methods Overview of the Body of Work Method Procedures for the Body of Work Method An Application the Body of Work Method Alternative Procedures and Limitations 10. The Bookmark Method Overview of the Bookmark Method An Implementation of the Bookmark Method Alternative Procedures and Limitations 11. The Item-Descriptor Matching Method Procedures for the IDM Method Alternative Procedures and Limitations 12. The Hofstee and Beuk Methods The Hofstee Method Procedures for Implementing the Hofstee Method The Beuk Method Procedures for Implementing the Beuk Method Alternative Procedures and Limitations SECTION III: CHALLENGES AND FUTURE DIRECTIONS IN STANDARD SETTING 13. Scheduling Standard Setting Activities Scheduling Standard Setting for Educational Assessments Scheduling Standard Setting for Credentialing Programs Conclusions and Recommendations 14. Vertically-Moderated Standard Setting The Interrelated Challenges A Brief History of Vertically-Moderated Standard Setting What Is Vertically-Moderated Standard Setting? Approaches to VMSS Applications of VMSS An Illustration of VMSS Procedures Alternative Procedures and Limitations 15. Standard Setting on Alternate Assessments The Unique Challenges of Alternate Assessment Necessary Conditions for Alternate Assessment Systems A Generalized Holistic (GH) Method Limitations and Alternative Procedures 16. Special Topics and Next Steps Rounding Methods of Adjusting Cut Scores Deciding How to Incorporate Uncertainty Generalizability of Standards Using Multiple Methods of Standard Setting Improving Participant Training

A well-timed text, given the legislated requirement of performance standards in our K-12 educational system, as well as the continued need in the process of credentialing professionals. This book would be useful to individuals retained to produce or coordinate standard-setting activites. -- PsycCritiques It is abundantly apparent that the authors know standard setting and themselves have implemented numerous standard-setting procedures using different methods for different types of assessment programs.... Although the book is very practical, someone who has a fair amount of knowledge in standard setting will not feel shortchanged. There is a richness in the discussions of the methods that engages the reader, such as the different modifications of a method and the situational challenge that warranted each modification. Also, when a specific implementation of a method is discussed in detail, the authors are very conscientious in discussing the impetus for the adaptations. The historical perspective presented for some of the methods is quite intriguing. -- Journal of Educational Measurement