Evidence, Understanding and Complexity
Evaluation in Non-Linear Systems
This article provides a critical analysis of the implementation of the Theory of Change Approach (Connell & Kubisch, 1998, The Aspen Institute). The authors analyze the challenges associated with the evaluation of complex policy initiatives, focusing on one element of the national evaluation of Health Action Zones in England: Building Capacity for Collaboration (grants to 26 communities to improve health delivery services). The authors argue that the application of randomized control trials could not assess all the changes that the project tried to address, which were at individual, community, service organization, and system levels. They analyze the multiple dimensions of complexity that characterized HAZ, which include the dynamics of both the internal organizational and external political environment. These dimensions are:
- Levels (structural complexity);
- Time (measurable impact);
- Scope (broad objective);
- Players (wide range players involved);
- Strategies and models;
- Rules and conditions; and
- The problem context.
Each of these dimensions is discussed comprehensively, in regards to the implication of TOC approach by contrasting its theory and practice. On their role as evaluators, they analyze it "through the lens of power relationships" so that they "can give attention to ‘phenomena such as value orientations, cultural habits and institutional arrangements which may be decisive for the success of key ideas in various programme theories." They conclude that "the complexity of such programmes cannot be captured within one overarching theory". In the process of developing TOC, they suggest to draw from different theories include social constructionism, complexity theory, and new institutional theory.