Evaluation in the face of uncertainty: anticipating surprise and responding to the inevitable / edited by Jonathan A. Morell

1 From Firefighting to Systematic Action
Adding "Surprise" to the Mix 1
Historical Roots: Evaluation, Planning, and System Behavior 2
From Explaining Surprise to Dealing with It 4
Development Path of This Book 6
Guiding Principles 8
How to Read This Book 12
In Sum 14
2 Structure of the Unexpected
Where Does Surprise Come From? 17
Beyond Simple Distinctions 21
In Sum 26
3 Placing Surprise in the Evaluation Landscape
When Is the Probability of Surprise High? 27
When Is Surprise Disruptive to Evaluation? 35
In Sum 44
4 Minimizing Foreseeable Surprise
Theory: Using Explanatory Power and Simplified Relationships 48
Exploiting Past Experience: Capitalizing on What We Already Know 58
Limiting Time Frames to Minimize the Opportunity for Surprise 63
In Sum 67
5 Shifting from Advance Planning to Early Detection
Leading Indicators 72
System-Based Logic Modeling 78
In Sum 81
6 Agile Evaluation
Data 83
Agile Methodology 87
Retooling Program Theory 95
Agility and Stakeholder Needs 98
In Sum 98
7 How Much Is Too Much?: Appreciating Trade-Offs and Managing
the Balance
A Framework for Appreciating Design Trade-Offs 103
Maximizing Choice, Minimizing Risk 107
Evaluation Design 112
In Sum 113
8 Applying the Examples to Categories of Cases: The Life Cycle View
"Unintended Consequences": Unity across Programs
and Their Evaluations 116
Interpreting Cases through a Life Cycle Perspective 117
In Sum 123
9 Applying the Examples to Categories of Cases:
The Social/Organizational View
Navigating through the Cases 125
Placement of Cases on the Social/Organizational Map 131
Categorizations Derived from the Data 147
In Sum 154
10 Lessons from Individual Cases: Tactics for Anticipating Surprise
In Sum 168
11 Lessons from Individual Cases: Responding to Surprise
The Middle 169
Leading Indicators and Agile Evaluation 172
In Sum 176
12 Unanticipated Program Outcomes
Case Descriptions 177
Applying the Cases to Unintended Program Outcomes 181
Comparing the Cases 183
Predicting the Need for Agile Evaluation 187
In Sum 191
13 Concluding Thoughts
Case 1 Grasping at Straws and Discovering a Different
Program Tfieory: An Exercise in Reengineering
Analysis Logic in a Cfiild Care Evaluation Setting
Dennis P. Afjholter
Case 2 Sfiifting Sands in a Training Evaluation Context
James W Altschuld and Phyllis M. Thomas
Case 3 Evaluating Programs Aimed at Promoting Cfiild Well-Being:
Tfie Case of Local Social Welfare Agencies in Jerusalem
Anat Zeira
Case 4 Assessing tfie Impact of Providing Laptop
Computers to Students
J. Dan Strahl, Deborah L. Lowther, and Steven M. Ross
Case 5 Quasi-Experimental Strategies Wfien Randomization Fails:
Propensity Score Matching and Sensitivity Analysis
in Whole-School Reform
Gary L. Bowen, Roderick A. Rose, and Shenyang Quo
Case 6 Unexpected Changes in Program Delivery:
The Perils of Overlooking Process Data When
Evaluating HIV Prevention
Bryce D. Smith, Danny Sprouse, and Kevin L DeWeaver
Case 7 Evaluating Costs and Benefits of Consumer-Operated Services:
Unexpected Resistance, Unanticipated Insights,
and Deja Vu All Over Again
Brian T. Yates
Case 8 Keep Up with the Program!: Adapting the Evaluation Focus
to Align with a College Access Program's Changing Goals
Kristine L. Chadwick and Jennifer Conner Blatz
Case 9 Assumptions about School Staff's Competencies
and Likely Program Impacts
Laura Hassler Lang, Christine E. Johnson, and Shana Goldwyn
Case 10 Mixed Method Evaluation of a Support Project
for Nonprofit Organizations
Riki Savaya and Mark Waysman
Case 11 Evaluating the Health Impacts of Central Heating
Jeremy Walker, Richard Mitchell, Stephen Platt,
and Mark Petticrew
Case 12 Recruiting Target Audience: When All Else Falls, Use the
Indirect Approach for Evaluating Substance Abuse Prevention
Molly Engle
Case 13 Unintended Consequences of Changing Funder Requirements
Midproject on Outcome Evaluation Design and Results
in HIV Outreach Services
Lena Lundgren, Therese Fitzgerald, and Deborah Chassler
Case 14 Generating and Using Evaluation Feedback for Providing
Countywide Family Support Services
Deborah L. Wasserman
Case 15 Trauma and Posttraumatic Stress Disorder among Female
Clients in Methadone Maintenance Treatment in Israel:
From Simple Assessment to Complex Intervention
Miriam Schiff and Shabtay Levit
Case 16 From Unintended to Undesirable Effects of Health Intervention:
The Case of User Fees Abolition in Niger, West Africa
Valery Ridde and Aissa Diarra
Case 17 Unintended Consequences and Adapting Evaluation:
Katrina Aid Today National Case Management Consortium
Amanda Janis and Kelly M. Stiefel
Case 18 Evaluation of the Integrated Services Pilot Program
from Western Australia
Susanne Therese Bahn, Peter Hancock, and Trudi Cooper1 From Firefighting to Systematic Action
Adding "Surprise" to the Mix 1
Historical Roots: Evaluation, Planning, and System Behavior 2
From Explaining Surprise to Dealing with It 4
Development Path of This Book 6
Guiding Principles 8
How to Read This Book 12
In Sum 14
2 Structure of the Unexpected
Where Does Surprise Come From? 17
Beyond Simple Distinctions 21
In Sum 26
3 Placing Surprise in the Evaluation Landscape
When Is the Probability of Surprise High? 27
When Is Surprise Disruptive to Evaluation? 35
In Sum 44
4 Minimizing Foreseeable Surprise
Theory: Using Explanatory Power and Simplified Relationships 48
Exploiting Past Experience: Capitalizing on What We Already Know 58
Limiting Time Frames to Minimize the Opportunity for Surprise 63
In Sum 67
5 Shifting from Advance Planning to Early Detection
Leading Indicators 72
System-Based Logic Modeling 78
In Sum 81
6 Agile Evaluation
Data 83
Agile Methodology 87
Retooling Program Theory 95
Agility and Stakeholder Needs 98
In Sum 98
7 How Much Is Too Much?: Appreciating Trade-Offs and Managing
the Balance
A Framework for Appreciating Design Trade-Offs 103
Maximizing Choice, Minimizing Risk 107
Evaluation Design 112
In Sum 113
8 Applying the Examples to Categories of Cases: The Life Cycle View
"Unintended Consequences": Unity across Programs
and Their Evaluations 116
Interpreting Cases through a Life Cycle Perspective 117
In Sum 123
9 Applying the Examples to Categories of Cases:
The Social/Organizational View
Navigating through the Cases 125
Placement of Cases on the Social/Organizational Map 131
Categorizations Derived from the Data 147
In Sum 154
10 Lessons from Individual Cases: Tactics for Anticipating Surprise
In Sum 168
11 Lessons from Individual Cases: Responding to Surprise
The Middle 169
Leading Indicators and Agile Evaluation 172
In Sum 176
12 Unanticipated Program Outcomes
Case Descriptions 177
Applying the Cases to Unintended Program Outcomes 181
Comparing the Cases 183
Predicting the Need for Agile Evaluation 187
In Sum 191
13 Concluding Thoughts
Case 1 Grasping at Straws and Discovering a Different
Program Tfieory: An Exercise in Reengineering
Analysis Logic in a Cfiild Care Evaluation Setting
Dennis P. Afjholter
Case 2 Sfiifting Sands in a Training Evaluation Context
James W Altschuld and Phyllis M. Thomas
Case 3 Evaluating Programs Aimed at Promoting Cfiild Well-Being:
Tfie Case of Local Social Welfare Agencies in Jerusalem
Anat Zeira
Case 4 Assessing tfie Impact of Providing Laptop
Computers to Students
J. Dan Strahl, Deborah L. Lowther, and Steven M. Ross
Case 5 Quasi-Experimental Strategies Wfien Randomization Fails:
Propensity Score Matching and Sensitivity Analysis
in Whole-School Reform
Gary L. Bowen, Roderick A. Rose, and Shenyang Quo
Case 6 Unexpected Changes in Program Delivery:
The Perils of Overlooking Process Data When
Evaluating HIV Prevention
Bryce D. Smith, Danny Sprouse, and Kevin L DeWeaver
Case 7 Evaluating Costs and Benefits of Consumer-Operated Services:
Unexpected Resistance, Unanticipated Insights,
and Deja Vu All Over Again
Brian T. Yates
Case 8 Keep Up with the Program!: Adapting the Evaluation Focus
to Align with a College Access Program's Changing Goals
Kristine L. Chadwick and Jennifer Conner Blatz
Case 9 Assumptions about School Staff's Competencies
and Likely Program Impacts
Laura Hassler Lang, Christine E. Johnson, and Shana Goldwyn
Case 10 Mixed Method Evaluation of a Support Project
for Nonprofit Organizations
Riki Savaya and Mark Waysman
Case 11 Evaluating the Health Impacts of Central Heating
Jeremy Walker, Richard Mitchell, Stephen Platt,
and Mark Petticrew
Case 12 Recruiting Target Audience: When All Else Falls, Use the
Indirect Approach for Evaluating Substance Abuse Prevention
Molly Engle
Case 13 Unintended Consequences of Changing Funder Requirements
Midproject on Outcome Evaluation Design and Results
in HIV Outreach Services
Lena Lundgren, Therese Fitzgerald, and Deborah Chassler
Case 14 Generating and Using Evaluation Feedback for Providing
Countywide Family Support Services
Deborah L. Wasserman
Case 15 Trauma and Posttraumatic Stress Disorder among Female
Clients in Methadone Maintenance Treatment in Israel:
From Simple Assessment to Complex Intervention
Miriam Schiff and Shabtay Levit
Case 16 From Unintended to Undesirable Effects of Health Intervention:
The Case of User Fees Abolition in Niger, West Africa
Valery Ridde and Aissa Diarra
Case 17 Unintended Consequences and Adapting Evaluation:
Katrina Aid Today National Case Management Consortium
Amanda Janis and Kelly M. Stiefel
Case 18 Evaluation of the Integrated Services Pilot Program
from Western Australia
Susanne Therese Bahn, Peter Hancock, and Trudi Cooper