Credible evidence on complex change processes: key challenges in impact evaluation on agricultural value chains.

Ton, Giel and Vellema, Sietze and DeRuyterDeWildt, Marieke (2011): Credible evidence on complex change processes: key challenges in impact evaluation on agricultural value chains.

Abstract

Although a growing field of policy intervention, the effectiveness of public-private value chain support is regularly questioned in the policy realm. Partly resulting from stronger pressures on aid money to show its worth, convincing evidence is asked for the effect on poverty alleviation. However, impact evaluations of interventions are challenging: outcome indicators are often multi-dimensional, impact is generated in dynamic and open systems and the external validity of conclusions are often limited, due to contextual particularities. Therefore, there is a strong case for theory-based evaluation where logic models indicate how the intervention is expected to influence the incentives for people’s behaviour. The key assumptions inherent in these casual models can be tested through observation and measurement of specific outcome indicators, using mixed methods in triangulation. The mix of methods will have to anticipate the major threats to validity to the type of evaluative conclusion that the evaluation is expected to generate .Following the work of Shadish, Cook and Campbell (2002), validity threats relate to: 1) statistical conclusion validity; 2) internal validity; 3) construct validity; and, 4) external validity. The authors propose the combined use of data-set observations and causal-process observations in a comparative case-study design, based on critical realist concept of contextmechanism-outcome configurations. The use of a realist method to describe and analyze intervention pilots, facilitates the exchange of experiences between development agencies with evidence-based research. Its defined generalisation domain may prevent uncritical embracement of good practices. Certain value chain upgrading strategies may be viable and effective in a range of situations but are not the panacea, the standard solution, for creating market access; they all involve specific institutional arrangements that ‘fire’ specific mechanisms and incentives that depend on the institutional environment and social capital of stakeholders involved.

• Item Type: MPRA Paper
• Language: English
• Keywords: rural development, mixed methods, attribution, organisational performance, realist evaluation
• Subjects: D - Microeconomics > D2 - Production and Organizations > D23 - Organizational Behavior ; Transaction Costs ; Property Rights; A - General Economics and Teaching > A1 - General Economics > A12 - Relation of Economics to Other Disciplines; O - Economic Development, Innovation, Technological Change, and Growth > O1 - Economic Development > O12 - Microeconomic Analyses of Economic Development

References

Bamberger, M., 2007. Simply the Best? Understanding the Market for 'Good Practice' Advice from Government Research and Evaluations: a framework for assessing validity and utilization of evaluations. American Evaluation Association 2007, Baltimore.

Bamberger, M., J. Rugh, and L. Mabry, 2006. RealWorld evaluation: working under budget, time, data, and political constraints, Sage Publications Inc.

Begg, C., 1994. Publication bias. The Handbook of Research Synthesis, 399-409.

Bijman, J., R. Muradian, and A. Cechin, 2011. Agricultural Cooperatives and Value Chain Coordination. In: Helmsing, A.H.J., and S. Vellema (Eds.), Value Chains, Inclusion and Endogenous Development: Contrasting Theories and Realities, Routhledge.

Blaikie, N.W.H., 1991. A critique of the use of triangulation in social research. Quality and Quantity, 25 (2), 115-136.

Blaikie, N.W.H., 2000. Designing social research: The logic of anticipation, Polity.

Borgen, S.O., 2004. Rethinking incentive problems in cooperative organizations. Journal of Socio-Economics, 33 (4), 383-393.

Brady, H.E., and D. Collier, 2004. Rethinking social inquiry: diverse tools, shared standards, Rowman & Littlefield Publishers.

Brady, H.E., D. Collier, and J. Seawright, 2006. Toward a Pluralistic Vision of Methodology. Political Analysis, 14 (3), 353-368.

Byrne, D., and C. Ragin, 2009. The SAGE handbook of case-based methods, Sage Publications Ltd.

Chelimsky, E., and W.R. Shadish, 1997. Evaluation for the 21st century: A handbook, Sage.

Chen, H.T., 1994. Theory-Driven Evaluations, Sage Publications Inc.

Cook, M., 1994. The role of management behavior in agricultural cooperatives. Journal of Agricultural Cooperation, 9, 42-58.

Cornforth, C., 2004. The Governance of cooperatives and mutual associations: a paradox perspective. Annals of Public & Cooperative Economics, 75 (1), 11-32.

Creevey, L., and M. Ndiaye, 2008. Common Problems in Impact Assessment Research. Impact Assessment Primer Series #7, USAID, Washington.

Creevey, L., and G. Woller, 2006. Methodological Issues in Conducting Impact Assessments of Private Sector Development Programs. Impact Assessment Primer Series #2, USAID, Washington.

DAC, 2008. Evaluating Development Co-operation: summary of key norms and standards, OECD, Paris.

DCED, 2010. The DCED Standard for Measuring Achievements in Private Sector Development: Control Points and Compliance Criteria. Version V., Donor Committee for Enterprise Development.

Denzin, N.K., 1970. The research act, Aldine Chicago.

Donaldson, S., C. Christie, and M. Mark, 2008. What counts as credible evidence in applied research and evaluation practice?, Sage Publications, Inc.

Duflo, E., R. Glennerster, M. Kremer, L. Center, and T. Floor, 2006. Using Ramdomization in Development Economics Research: A Tool Kit. MIT Department of Economics Working Paper No. 06-36.

European Commission, 2008. Second Strategic Review of Better Regulation in the European Union European Commission, Brussels.

Farrington, D.P., 2003. Methodological Quality Standards for Evaluation Research. The ANNALS of the American Academy of Political and Social Science, 587 (1), 49-68.

Heckman, J.J., 2005. The scientific model of causality. Sociological Methodology, 35 (1), 1-98.

Helmsing, A.H.J., and S. Vellema, 2011. Value Chains, Inclusion and Endogenous Development: Contrasting Theories and Realities, Routhledge.

Hospes, O., 2008. Evaluation Evolution?- three approaches to evaluation. The Broker, 2008 (8), 24-26.

Humphrey, J., and L. Navas-Aleman, 2009. Multinational Value Chains, Small and Medium Enterprises, and 'Pro-Poor' Policies: A Review of Donor Practice. IDS Research Report 63, IDS, Brighton.

Khandker, S., B. Koolwal, and H. Samad, 2009. Handbook on Impact Evaluation, World Bank, Washington, 239 pp. Lawson, T., 2003. Reorienting Economics, Routledge, London & New York.

Lawson, T., 2009. Applied economics, contrast explanation and asymmetric information. Cambridge Journal of Economics, 33 (3), 405-419.

Mackie, J.L., 1965. Causes and conditions. American philosophical quarterly, 245-264.

Maredia, M.K., 2009. Improving the proof: Evolution of and emerging trends in impact assessment methods and approaches in agricultural development, Intl Food Policy Res Inst.

Mill, J.S.S., 2009 (1843). System of Logic Ratiocinative and Inductive: Being a Connected View of the Principles of Evidence and the Methods of Scientific Investigation, Cosimo Classics.

Olson, M., 1965. The Logic of Collective Action.

Ostrom, E., and I. Bloomington, 2009. Building Trust to Solve Commons Dilemmas: Taking Small Steps to Test an Evolving Theory of Collective Action. Games, Groups, and the Global Good, 207.

Pawson, R., 2002. Evidence-based Policy: The Promise of 'Realist Synthesis'. Evaluation, 8 (3), 340-358. Pawson, R., 2003. Nothing as practical as a good theory. Evaluation, 9 (4), 471-490.

Pawson, R., and N. Tilley, 1997. Realistic evaluation, Sage Publications Inc.

Ragin, C., 2000. Fuzzy-set social science, University of Chicago Press.

Ragin, C., 2008. Redesigning social inquiry: Fuzzy sets and beyond, University of Chicago Press.

Ravallion, M., 2009. Should the Randomistas Rule? The Economists' Voice, 6 (2), 6.

Rihoux, B., and C. Ragin, 2008. Configurational comparative methods. Qualitative Comparative Analysis (QCA) and related techniques (Applied Social Research Methods). Thousand Oaks and London: Sage.

Roche, C.J.R., and C. Roche, 1999. Impact assessment for development agencies: Learning to value change, Oxfam.

Rogers, P.J., 2009. Matching impact evaluation design to the nature of the intervention and the purpose of the evaluation. Journal of Development Effectiveness, 1 (3), 217 - 226.

Shadish, W.R., T.D. Cook, and D.T. Campbell, 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Houghton Mifflin Co. Boston, MA.

Snodgrass, D., 2006. Assessing the Impact of New Generation Private Sector Development Programs. Impact Assessment Primer Series #1, USAID, Washington.

Snowden, D., and M. Boone, 2007. A leader's framework for decision making. harvard business review, 85 (11), 68.

Tanburn, J., 2008. The 2008 Reader on Private Sector Development: measuring and reporting results. International Training Centre - ILO, Turin.

Ton, G., 2008. Challenges for smallholder market access: A review of literature on institutional arrangements in collective marketing. Stewart Postharvest Review, 4 (5).

Ton, G., 2010. Resolving the Challenges Of Collective Marketing: incentive structures that reduce the tensions between members and their group. ESFIM Policy Brief #4, ESFIM, Wageningen.

Ton, G., S. Taylor, J. Vlaming, and S. Hiller, 2009. Evaluating Poverty Impacts of Bottom-of-the-Pyramid Irrigation Technology Supply: IDE’s Rolling Baseline Approach to Household Income Impact Assessment International Conference on Impact of Base of the Pyramid Ventures, 16-18 November 2009, Delft.

Vaessen, J., 2010. Methodological and Conceptual Challenges in Evaluating the Impact of Development Interventions, Datawyse / Universitaire Pers Maastricht, Maastricht.

Vellema, S., 2011. Transformation and Sustainability in Agriculture: connecting practice with social theory, Wageningen Academic Publishers, Wageningen.

World Bank, 2007. World Development Report 2008: Agriculture for Development, World Bank, Washington.

Zandniapur, L., J. Sebstad, and D. Snodgrass, 2004. Review of Evaluations of Selected Enterprise development Projects. MicroREPORT #3, USAID, Washington.