1	Program Design, Monitoring and Evaluation Virginia Lamprecht, Senior Technical Advisor, USAID Presented at CCIH Annual Conference, May 28, 2005 Workshop to Strengthen the Design, Monitoring, and Evaluation of Family Planning Projects
2	What Is a Results Framework? A presentation of a whole view of a program including— The big goal Our ideas about the things that have to be in place to achieve success.
3	Why Use Results Frameworks? Because they help us focus on the END result(s) and the strategies that we can use to achieve them Because they force us to identify the logic and links behind our programs and to identify necessary and sufficient elements for success Because a Results Framework is a simple tool to present what we are trying to accomplish in a program or project
4	Elements of Results Frameworks Goal Statement—the change in health conditions that we hope to achieve Strategic (or Key) Objective (SO)—the main result that will help us achieve our goal and for which we can measure change (Intermediate) Results (IRs)—the things that need to be in place to ensure achievement of the SO Strategies & Activities—what a project does to achieve its intermediate results that contribute to the objective
5	Some Typical Health Goals and Related Strategic Objectives Goal: Stabilize population/Reduce fertility (we don’t measure) SO: Increased FP use or reduced unmet need for FP Goal: Reduce child mortality SO: Increased use of preventive health behaviors (immunization) Goal: Improved adolescent health SO: Increased use of risk reduction behaviors among adolescents
6	General Characteristics of an Intermediate Result Statement of a desired outcome or a situation that changed as a result of project intervention—not an activity or process Contraceptive services accessible to target population Community members are knowledgeable about and approve of FP. This outcome contributes to our ability to get to our SO (e.g., use of FP). The result is measurable.
7	Strategies and Activities These are things the project does in order to achieve the desired outcomes or changes in the situation. For example: Train and supply CBDs Strengthen FP logistic systems Advocate for contraceptive supply
8	Common Difficulties in Formulating Results Frameworks and Program Design Mixing up results, strategies, and activities Starting Project Design with a list of activities that may not logically lead to the desired objective Choosing indicators that truly measure the results (we will get to this later on)
9	GOAL: Improved Reproductive Health Status
10	A Complete Program Design Process Involves the project team and key counterparts Requires background research and data collection Uses data collected to identify gaps Creates a Results Framework Helps the team determine strategies that address gaps Develops an M&E plan linked to the RF/Project Design
11	Steps in the Design Process
12	Step 1—Situation Analysis (SA) Four Components of SA: Secondary Data Compilation and Review Policy Environment Scan Participatory Qualitative Assessment Health Service Delivery Assessment
13	What Is Situational Analysis? Situational analysis describes— Background setting Current environment Behaviors Service delivery quality, use, patterns. Situational analysis provides the basis for planning interventions, which— Address gaps Build on existing strengths within the community and the health system.
14	Tool for Synthesizing Situation Analysis Data (1 per IR)
15	Steps in the Design Process
16	Steps 2 and 3: Results Framework and Strategy Selection Step 2: Create a Results Framework with desired results from the preceding step Step 3: Articulate major strategies that will be used to achieve the results and/or address gaps/barriers as described in Step 1
17	Assumptions With any program design, there are key assumptions that we make that are critical to success—it is worthwhile to note these! General assumptions such as political stability, collaboration from MOH, and ability to hire competent staff Specific assumptions such as availability of contraceptives; collaboration between partners for different elements, such as training and supervision; and favorable policy environment Other examples
18	Project Summary Results Framework
19	Small Group Work What are some strategies you would use to address— Demand issues Husbands are concerned that their wives will become infertile if they use the pill. Quality problems Facility staff do not respect confidentiality—may share a woman’s FP use with her husband. Access problems FP services in the clinic 8 kilometers away. Policy problem IUD cannot be inserted without negative lab results of pregnancy test.
20	An Example RF for a FP Project
21	Once We Have Selected Results and Likely Strategies… Step 4: It’s time to select indicators that measure the results we want to achieve and fit the strategies we will use.
22	Steps in the Design Process
23	Monitoring and Evaluation Planning Matrix
24	What Is an Indicator? A variable That measures One aspect of a program/project For a specific population.
25	Let’s Take a Look at Indicators We choose indicators that will show us whether we achieved what we set out to do. We choose indicators that correspond to the stated results and specific strategies. We use existing resources to identify and monitor already tested indicators with good definitions.
26	What Is a Good Indicator? Valid: Measures the effect it is supposed to measure Reliable: Will give the same result if measured in the same way Precise: Is operationally defined so people are clear about what they are measuring Timely: Can be measured at an interval that is appropriate to the level of change expected Comparable: Can be compared across different target groups or project approaches
27	Criteria for Indicator Selection Consistent with project design—measure the desired result Useful—contributes to project design, management, and evaluation Available Affordable
28	Steps in the Design Process
29	Step 5: M&E Plan Development Fill in desired results (SOs and IRs). Fill in selected indicators for each result. For each indicator, determine source of information, frequency of data collection, and responsible agent.
30	M&E Plan Matrix
31	M&E Plan Creation Matrix should be accompanied by text that describes how data collected will be used by the team regularly to monitor progress, make course corrections, etc. Targets (if desired) will be set after baseline data are collected.
32	Steps in the Design Process
33	Step 6: Baseline Quantitative Data Collected Can be done through KPC survey or other methods (e.g., health service stats, complete community registration) Measures knowledge, use of services/current behaviors, or proxy of these (e.g., CYP) Provides baseline values for indicators selected
34	Steps in the Design Process
35	Situation Analysis: Secondary Data Review and Policy Scan Step 1 of the Design Process Components A and B of Situation Analysis
36	Steps in Design Process
37	What Are Secondary Data? Secondary data are information resources that someone else has collected or developed. They can provide current or recent information about health knowledge, practices and status of the population, types and distribution of health facilities, policies and protocols in place, quality of care, and community perceptions of health.
38	Advantages of Secondary Data Readily available (usually free) Provide an overview of the health situation, but they may not be exactly the same as the conditions in the specific project area May identify social or policy issues that could contribute to the success or failure of a project Offer a starting point for developing the background for a situation analysis.
39	Benefits and Limitations Benefits Inform health project design with data pertinent to knowledge, demand, access, quality, and policy Raise questions for further research. Limitations May not be specific to project area, interventions, or target groups May be outdated Often do not reflect community perspective Incomplete.
40	Tool for Synthesizing Situation Analysis Data (1 per IR)
41	Situation Analysis: Participatory Qualitative Assessment Step 1 of the Design Process Component C of Situation Analysis
42	Steps in Design Process
43	Information from Qualitative Research Answers: Why? How? When? What? Who? Explains Broad patterns of behavior Motivations for behaviors The how and why of knowledge and behavior Provides in-depth understanding Complements quantitative methods
44	What is Qualitative Research? Any research that captures information and produces findings not reached by means of quantitative procedures.
45	What Is Not Obtained? Qualitative research does not answer— How often? How many? How much? Does not produce— Frequencies, rates, averages, numbers of knowledge and behavior Statistically significant information.
46	Limitations of Qualitative Research Highly susceptible to subjective bias Questions asked by interviewers Interpretation of responses Often applied and analyzed inappropriately Flexible
47	Qualitative Methods, How Information Is Obtained Open-ended questions Techniques that encourage participation by all Flexible methodologies that allow facilitator to change techniques in order to improve dialog or observation Respecting local knowledge and capabilities Use of a flexible guide to help the process
48	Steps to Analyzing Qualitative Studies The challenge in qualitative analysis is to organize and categorize data effectively so they can be used, without compromising the data in the process. Reading/Immersion Identification of themes Categorizing data by theme Reducing Interpreting
49	Pitfalls in Qualitative Analysis Quantifying focus group information Overemphasizing marginal or minority data
50	Participatory Qualitative Data Collection Tools Observation In-depth interviews Focus group discussions Visual techniques such as community maps, transect walks, body mapping, social or network mapping Group ranking
51	In-depth Interview (IDI)—Definition Conversation during which the informant provides us with information she/he generally knows well.
52	Focus Group Discussion (FGD)—Definition Group discussion and exchange of ideas among participants to— Discover trends and patterns in perceptions Explore the range and variety of attitudes and practices Explore the variety of barriers and motivations Learn about social norms.
53	Participatory Learning and Action (PLA)—Definition PLA is a process that uses participatory approaches used to help communities by— Analyzing their needs Identifying solutions to address those needs Developing and implementing a plan of action.
54	Participatory Learning and Action (PLA)—Definition PLA is based on the assumption that community members are “experts” on their own health and social situations. By involving community members in the process, the assessment process also becomes an intervention for change. The evolution of the name also reflects this evolution: from Rapid Rural Appraisal to Participatory Rural Appraisal to Participatory Learning and Action
55	Positive Deviance Inquiry (PD) as a Qualitative Research Tool PD is a developmental approach that— Is based on the premise that solutions to community problems requiring behavioral or social change may already exist within the community Focuses on what is going right in communities rather than what is going wrong Like PLA, involves community members in every step.
56	Steps in PD: The Six Ds of PD Define—the problem to be addressed. Determine—who is successful in the community. Discover—what they are doing right. Design—an intervention around the positive behavior. Discern—evaluate the effectiveness. Disseminate—encourage others to adopt.
57	Tool for Synthesizing Situation Analysis Data—(1 per IR)
58	Completing the Situation Analysis: Health Service Delivery Assessment (HSDA) Step 1 of the Design Process Component D of Situation Analysis
59	Steps in Design Process
60	What Is HSDA? Qualitative and quantitative data collection about the health system and its services that— Examines what health services/supplies are available, their accessibility, quality, and current use pattern Uses a systems orientation to identify gaps and strengths in the health system Must be PARTICIPATORY to maximize ownership and interpretation of findings Covers different types of service delivery, e.g., formal, informal, private, and public sector services Requires a package of tools to cover the different components of the assessment Helps explain why provider performance is adequate or inadequate, so we address the right barriers to improved performance.
61	Brainstorm What kinds of information might we collect through a HSDA? What sources do we use to get this information?
62	Types and Sources of Information Collected During HSDA
63	Types and Sources of Information Collected During HSDA
64	Types and Sources of Information Collected During HSDA
65	GOAL: Improved Reproductive Health Status
66	Components of an HSDA Inventory Infrastructure Equipment and supplies Personnel and training Observations Client reception and patient flow Patient—provider interaction Hygiene and infection prevention Interviews—Providers and Clients Record Review Patient registers Stock cards
67	Overview of HSDA Process Define purpose of HSDA. Design process—who should participate, informants, types of information needed, etc. Identify pertinent HSDA components according to the purpose. Select and adapt tools. Train staff and interviewers. Select sample. Collect, analyze, and document data. Use findings to inform program design. Must be PARTICIPATORY.
68	Using HSDA for Program Design and M&E Conduct an HSDA during the Situation Analysis phase to determine whether and how quality, accessibility, and availability of services need to be improved. Often, you will want to repeat pieces of the HSDA to monitor changes in quality over time or as part of a supervision system. For example, client satisfaction, frequency of stock-outs, or improved provider performance. You may want to repeat the entire HSDA as part of the project evaluation to look for changes in quality, accessibility, and use of services.
69	Tool for Synthesizing Situation Analysis Data (1 per IR)
70	Tool for Synthesizing Situation Analysis Data—(1 per IR)
71	Completing Our Results Framework and Selecting Strategies Steps 2 and 3 of the Design Process
72	Steps in Design Process
73	Project Summary Results Framework
74	Strategies and Activities Strategies are HOW a team plans to reach the intermediate outcomes/results. Activities are components of strategies and reflect what a project DOES on a day-to-day basis to achieve its intermediate results.
75	Exercise—Synthesizing Situation Analysis
76	Project Summary Results Framework
77	Reproductive Health Preliminary Action Plan (To be filled out for each IR—e.g., IR #1 (Access/Availability)
78	Completing the Program Design and M&E Plan Process Step 4: Indicator Selection Step 5: Developing M&E Plan
79	Steps in Design Process
80	Why Do We Need M&E? We need evaluation to be sure— That we have reached our destination That we have achieved the desired results. We need monitoring to ensure— That we are on the right road That our activities are leading to the destination. We need monitoring for Management to be sure project activities are completed as planned.
81	Steps in Developing an M&E Plan Review the results framework. Select or develop appropriate indicators for the SO and IRs, using standard indicators where applicable. Make sure indicators are selected for monitoring as well as for evaluation. Determine the best sources of monitoring (periodic) and evaluation information for each indicator. Check work from previous design steps. Make sure indicator collection is feasible and that indicators reflect results and strategies. Consider who and how the information will be used Fill out the planning matrix.
82	Selecting or Developing Indicators Select appropriate core or standard indicators where applicable—usually at the SO and IR levels. Develop and/or select additional indicators for monitoring according to project strategies and activities.
83	Step 1: Review the Results Framework Review where you are heading (SO intermediate results and strategies). Remember that the M&E plan must include indicators that measure each element of the results framework (except the goal).
84	Project Summary Results Framework
85	What Is an Indicator? An indicator is— A variable That measures One aspect of a project For a specific population. A well-defined evaluation indicator typically has both a numerator and denominator (but not always).
86	Characteristics of a Good Indicator A good indicator is— Valid: Measures what it intends to measure Reliable: All persons collecting the information should obtain the same measure Precise: The indicator is clearly defined Timely: Provides information when needed.
87	Example of Well-Defined Indicators Neonatal Mortality Rate Number of infant deaths 1–28 days last year in Madagascar Number of live births last year in Madagascar Contraceptive Prevalence Number of married women of CBA using modern FP Number of married women of childbearing age
88	Criteria for Indicator selection Consistent with project design—measures the desired result and is connected to selected strategies Useful—contributes to project design, management, and evaluation Available Affordable
89	Standard Indicators Where possible, a project should select standard indicators. They have been tested for validity and reliability. They allow comparison between projects or sites. They tend to be available for SOs and some IRs.
90	How Many Indicators? Choose at least one or two indicators per intermediate result, as well as the SO for evaluation purposes. Choose one or two indicators per result for program monitoring. Choose indicators that may be able to “cover” more than one element. For management, think about basic activities that you need to monitor to judge if you are implementing activities as planned; include indicators that help you make decisions.
91	Question What high-level SO indicator is sometimes used to measure contraceptive use if the project is not doing a population-based survey?
92	M&E Planning Matrix
93	M&E Planning Matrix Carefully identify and define indicators. Describe sources for data. Determine frequency of collection. Determine person responsible for data collection. Fill in baseline information when available.
94	Making Sure Indicators are Useful for Your Project For each indicator ask: Does it help you judge whether you have reached your result and/or you are on the way? Are data easily obtained? Is the cost reasonable? Are adequate resources available to collect the information? Is the indicator compatible with internationally accepted indicators? Is it useful to the people who are collecting the information? Does it help them make decisions? Check to make sure that indicators required by the donor are included.
95	Using Information for Decision Making In order to interpret information, objectives, targets, or decision points may be established for the IRs and some strategies. These are numeric values that get attached to indicators and provide benchmarks against which a project can measure progress. These values can be determined for the life of the project for higher-level indicators (objectives) or they can be defined for shorter time periods or lower-level indicators providing intermediate targets for management. When progress deviates from defined expectations, it is important for project implementers to review the reasons for the deviation and consider adjustments.
96	M&E Planning Matrix
97	After M&E Plan Is Developed Collect baseline information. Where appropriate, set expected change (amount and direction) for indicators. Integrate the M&E plan into project implementation plan. Modify activities and strategies based on information obtained. Inform outside world about achievements.
98	Step 6: Baseline Data Collection Where Are We Now? How Far Do We Have to Go to Reach Our Destination?
99	Steps in Design Process
100	Process for a Household Survey Adapt questionnaire according to project design desired indicators. Define the target population from which to draw the sample. Identify the sampling method and select the sample. Identify and train interviewers. Interview respondents in selected households. Check questionnaires for completion. Tally and/or enter data. Analyze the data. Use the data for project adaptation and improvement.
101	Baseline Data Collection We collect baseline information in to— Determine the current level of our indicators Set targets that we expect to achieve through project efforts. Under additional conditions we may want to— Compare with levels measured during or at the end of the project to determine if we have achieved the desired results. Baseline data collection favors quantitative techniques.
102	Baseline Data Sources Can come from same sources as situation analysis or indicator monitoring information: HSDA Service delivery statistics Policy documents Project records Can also come from population surveys or census Done after the project design That validate information from situation analysis Are more likely to be quantitative.
103	Quantitative Techniques Through quantitative techniques you can measure— The level (coverage) of actions or behaviors Frequencies, rates, averages, and trends Change over time in our sample. Statistical rules help us make inferences about what really happens in the entire population: How precise are our estimates? Are changes statistically significant?
104	Methods for Collecting Quantitative Data Service delivery statistics HSDA Project records Census-based population registers Population-based surveys
105	Service Delivery Statistics They can provide baseline information by reviewing data collected during the month(s) before the beginning of the project. They offer a good way to quantify progress over time and to discuss results among partners. Examples include utilization data, contraceptive distribution for CYP, logistic system information, and identification of new or continuing FP users.
106	Project Records Although their primary purpose is for project monitoring, they may include some information for evaluation purposes. Examples are: Number of communities with community-based FP providers Number of condoms sold by community-based providers Number of community health committees that meet regularly Number of community health committees with revolving funds for contraceptives.
107	Policy/Protocol Documents Useful when policy/protocol change is a key program strategy (usually not QUANTITATIVE, but Yes/No type indicator—policy is in place or changed).
108	Census-Based Information Systems Require complete registration of catchment area population. Require regular updating of data regarding the population’s practices, use of services, etc. Require literate health workers and significant investment of community-level workers for house-to-house visits. Provide up-to-date accurate rates for coverage of services/practice of behaviors that are recorded in register.
109	Population-Based Surveys Surveys are the primary tool for measuring quantitative indicators for knowledge, attitudes, practices, and coverage at the household level.
110	Examples of Indicators Measured by Survey CPR—contraceptive prevalence rate Women of reproductive age who know at least three modern methods of FP Sexually active respondents who report discussing FP issues with their spouse or sexual partner within the past 12 months Respondents of reproductive age who report discussing FP with a health worker during the past 12 months Number of women who report being a new user of a modern FP method Number of women with an infant less than 6 months old who report using LAM Women/men who report that they or their partner used a condom during last intercourse with non regular partner
111	Quantitative vs. Qualitative Approaches Quantitative—Surveys etc. Rigorous sample selection Questions asked the same every time—Follow the questionnaire exactly No probing questions unless defined by the questionnaire Qualitative—(PLA, FGDs) Four different methods to select respondents Open and flexible questions Probing questions and exploration essential for richer information
112	Options for Sample Selection— Household Surveys Direct Random Sample (rarely done) Uses the smallest sample size (96) but tends to be difficult because it depends on identifying all the “eligible” households in the target area and requires visiting all sites that are selected, which could be widely dispersed Practical alternatives for population-based health surveys at district or subdistrict level: 30 Cluster sample LQAS sample