1 College of Nursing and Health Sciences, Flinders University, Adelaide, SA, Australia.
Find articles by Angela Lawless2 Southgate Institute for Health Society and Equity, Flinders University, Adelaide, SA, Australia.
Find articles by Fran Baum2 Southgate Institute for Health Society and Equity, Flinders University, Adelaide, SA, Australia.
Find articles by Toni Delany-Crowe3 College of Medicine and Public Health, Flinders University, Adelaide, SA, Australia.
Find articles by Colin MacDougall4 Department of Health and Ageing, Adelaide, SA, Australia.
Find articles by Carmel Williams5 The Poche Centre for Indigenous Health and Well-being, Flinders University, Adelaide, SA, Australia.
Find articles by Dennis McDermott2 Southgate Institute for Health Society and Equity, Flinders University, Adelaide, SA, Australia.
Find articles by Helen van Eyk 1 College of Nursing and Health Sciences, Flinders University, Adelaide, SA, Australia. 2 Southgate Institute for Health Society and Equity, Flinders University, Adelaide, SA, Australia. 3 College of Medicine and Public Health, Flinders University, Adelaide, SA, Australia. 4 Department of Health and Ageing, Adelaide, SA, Australia.5 The Poche Centre for Indigenous Health and Well-being, Flinders University, Adelaide, SA, Australia.
* Correspondence to: Angela Lawless Email: ua.ude.srednilf@sselwal.alegna Received 2016 Dec 1; Accepted 2017 Oct 7. Copyright © 2018 The Author(s); Published by Kerman University of Medical SciencesThis is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
See commentary "How Do We Evaluate Health in All Policies?" in volume 7 on page 758. See commentary "Evaluating Health in All Policies" in volume 7 on page 761.Background: The importance of evaluating policy processes to achieve health equity is well recognised but such evaluation encounters methodological, theoretical and political challenges. This paper describes how a program theorybased evaluation framework can be developed and tested, using the example of an evaluation of the South Australian Health in All Policies (HiAP) initiative.
Methods: A framework of the theorised components and relationships of the HiAP initiative was produced to guide evaluation. The framework was the product of a collaborative, iterative process underpinned by a policy-research partnership and drew on social and political science theory and relevant policy literature.
Results: The process engaged key stakeholders to capture both HiAP specific and broader bureaucratic knowledge and was informed by a number of social and political science theories. The framework provides a basis for exploring the interactions between framework components and how they shape policy-making and public policy. It also enables an assessment of HiAP’s success in integrating health and equity considerations in policies, thereby laying a foundation for predicting the impacts of resulting policies.
Conclusion: The use of a program theory-based evaluation framework developed through a consultative process and informed by social and political science theory has accommodated the complexity of public policy-making. The framework allows for examination of HiAP processes and impacts, and for the tracking of contribution towards distal outcomes through the explicit articulation of the underpinning program theory.
Keywords: Healthy Public Policy, Evaluation, Inter-sectoral Action, Health Equity, Social Determinants
Since 2008 the South Australian Government has implemented a Health in All Policies (HiAP) approach to facilitate the development of healthy public policy. A five year action research program was established to examine the changes in policy-making processes as a result of the HiAP initiative, and to assess the impacts of these processes. This paper examines the development of a program theory-based framework used to guide evaluation of the South Australian HiAP initiative over a five year period.
Evaluation of policy-making processes and outcomes presents methodological, theoretical and political challenges. 1,2 Policy and policy work is understood and described in diverse and sometimes conflicting ways 3 but these ‘multiple accounts’ can help make sense of the processes, each adding a particular perspective. 2,4 Such interpretive approaches to policy-making suggest that research should examine how institutions shape thinking and working, attend to the role of relationships and boundaries in the ‘doing’ of policy work, uncover the actual practices of policy-making and the role of ideas and ideation processes in reform efforts, problem identification and representation. 4-6 The activities of policy-making are broad and include the identification and conceptualisation of problems, gaining the attention of government, the formulation of policy alternatives, selection of policy solutions and implementation, evaluation and revision – not as discrete episodes but as an ongoing ‘juggling’ process. 2,3 Evaluators, as well as the policy-makers need to negotiate the political context which includes political cycles, key personnel changes and the waxing and waning of political support. 7 Practically there may be obstacles in gaining access to the diverse individuals and groups involved in policy-making as researchers encounter gatekeepers and seek to engage time-poor bureaucrats and politicians. 2,8,9 Program theory-based evaluation offers a means of evaluation that engages with these difficulties in a rigorous and systematic manner. 10
Program theory has been defined as “the process through which program components are presumed to affect outcomes and the conditions under which these processes are believed to operate.” 10 By uncovering the theories underlying policy-making processes, explicit links and pathways can be hypothesized between program components and outcomes. Program theory can also include identification of the assumptions that underpin an initiative and the risks to those assumptions 11 as well as how the initiative relates to the economic, social and political environments.
It is well understood that this broader context fundamentally shapes interventions but it is often not well accounted for in evaluation. 12 Consideration of context is essential in uncovering the reasons and conditions in which a particular intervention works and how well it works. 13 Disappointing results or failure may be the result of factors external to the program itself. Real-life policy-making is driven by a wide range of contextual factors, including power relations, which make it a ‘messy’ undertaking. 3 Politics, layers of administration, and non-government organisations all form part of the context of policy processes. 14 An exploration of possible implications of contextual factors for the program is central to development and testing of program theory. Context is also central to the success or otherwise of programs becoming embedded and routinized. 12 In constantly changing policy contexts adaptability of interventions is a key to their success. 15,16
Where outcomes are long-term, theory-based evaluation allows identification of interim indicators of progress toward those long-term outcomes. Such evaluations provide a means of elaborating program theory and provide a framework for decisions about which aspects of the program will be measured or evaluated. Evaluation can thus be structured to examine the theories thought to underpin the program, and data collection designed to permit investigation of and confirm that theory. 17 The program theory underpinning a program such as HiAP is developed and then the key activities and events of the program are examined through multiple data sources in various contexts (eg, in various instances of health lens analysis [HLA]) allowing testing of the program theory.
Whilst interim steps in the pathway between an intervention and the achievement of high level outcomes can be identified, establishing causality is particularly difficult in complex health initiatives. Contribution analysis offers a step-wise, structured approach to the examination and testing of causal claims. 18 “[Contribution Analysis] assesses causal chains from beginning to end, reports on whether the intended changes occurred or not, and identifies the main contributions to such changes, including the intervention under evaluation” (p. 281). 19 Where a causal chain can be verified with empirical evidence and other major external influencing factors accounted for, then credible conclusions can be made regarding the contribution of an intervention to specified outcomes. Policy-making entails many complex elements and interactions, therefore accounting for all the external influences affecting the outcomes may prove impractical if not impossible. A pragmatic but systematic approach to causal analysis - which tests the congruence of the program theory with the results, employs counterfactual comparisons where possible, reviews results critically and seeks to explain exceptions and identify alternative explanations - may be more appropriate. 20
Key actors can assist researchers in building a robust understanding of an intervention, its causal pathways and its context. Such involvement requires the engagement of policy actors in making sense of the data strategically and operationally. As well as knowledge about the specific intervention, policy actors also contribute broader bureaucratic knowledge - the experiential and tacit knowledge of processes, contexts and audiences, and perceptions of what constitutes viable policy options. 21 Colebatch maintains that ‘getting close’ to policy processes and policy actors is particularly important for research to trace the ‘evolution of policy work over time.’ 6
The process followed by the South Australian Government in developing and implementing HiAP is described in Box 1. The adoption of HiAP by the South Australian Government provides an opportunity to examine the “juggling” of resources and negotiation of barriers undertaken by policy actors in an authentic context, that is, where complex interactions between actors, strategies and institutions play out in attempts to deliver better government. 22
In 2008 the South Australian Government committed to the adoption of a HiAP approach. Considerable work had been undertaken to prepare the way for such an approach and, as with the European initiative, HiAP in South Australian built on a significant history of advocacy and innovation regarding healthy public policy. By 2010 a dedicated HiAP unit had been established within the Health department, governance for the initiative was linked to processes already in place for the State’s strategic plan 23 and a process termed ‘health lens analysis’ had been developed and implemented in collaboration with a number of sectors. 24
Source: The South Australian Model of Health in All Policies 2011 (p. 26).25
HiAP is based on the understanding that health is shaped by social, economic and environmental determinants, and that action to improve health requires the involvement of multiple sectors and actors. 26 HiAP has been described as an approach that seeks to use “all available measures in all policy fields” (p. xvi) 27 and is designed to encourage action to improve population health and health equity across government sectors. 26 It involves multiple partners with sometimes divergent agendas and is implemented in, and influenced by, changing political, organisational and economic contexts. 28 It requires negotiation of relationships, values and meaning across organisational boundaries. 14
HiAP has been characterised as an “instrumental process-based intervention” – instrumental in its potential to create healthier policy through a focus on government processes. 5,29 Carey et al suggest that HiAP is a “set of institutional arrangements for delivering JUG (joined up government)” which in the South Australian case operates within the Government bureaucracy in a top-down manner. 29 These institutional policy-making arrangements structure the decision-making and frame the policy action (and inaction). 22
Understanding the policy-making processes and contexts that support consideration of health and equity in the policies of non-health sectors is critical to the success and maintenance of HiAP. This paper describes development of a framework that has guided the evaluation of the South Australian HiAP initiative and discusses its benefits and limitations.
The evaluation approach described in this paper is rooted in the tradition of theory-based approaches to complex community-based initiatives (see for example 30,31 ). Whilst there are differences between theory-based approaches, Stame identifies four similarities that span theory-based approaches:
The evaluation is based on an account of what is thought likely to happen as a result of the initiative.
They take account of context. The approach utilises a range of methods. They are committed to internal validity and to looking for causal relationships. 32Theory-based evaluation makes the causal assumptions behind policy interventions explicit, ie, it explains how and why a program or policy is thought to work, which forms the logic that underpins an initiative. 33 As Leeuw and others note, program theory is often drawn from stakeholder knowledge and is considered distinct from substantive social science theory, which may nevertheless inform and enrich program theory. 30,33 A distinction can also be drawn between program theory and implementation theory. 31,34,35 Program theory is concerned with mechanisms leading to the desired changes rather than the activities per se. 35 Implementation theory sheds light on how a particular initiative is operating, and program theory seeks to understand how program effects are realized. 17
To develop the framework that is presented in this paper, the research team undertook an iterative, participatory process to develop and refine an evaluation framework. Analysis of South Australia (SA) Health in All Policies Unit documents was undertaken, involving examination of approximately 300 sources, including policy documents, HiAP project proposals, project reports, online materials, training materials and a book. The documents were analysed to determine how HiAP was described, what rationales for HiAP were given, what activities were being undertaken and what outcomes were hoped to be achieved.
Two research workshops were held to capture the ‘history, experience and intuition of key stakeholders.’ 36 The research team had considerable experience in use of theory-based evaluation and had used such workshops previously to explicate program theory. 37 They adapted the method from their previous experience to apply to HiAP in SA. The purpose of these workshops was to elaborate how HiAP was operationalised in SA and examine how and why HiAP was being implemented in a particular way and what outcomes were expected. The study employed theoretical sampling in seeking informants with specific experience in either the development of HiAP in SA and/or implementation of HiAP. Involving stakeholders in the articulation of program theory strengthens the utility, feasibility, propriety and accuracy of the study. 30 The first workshop was attended by 9 public servants (policy officers, middle managers and a director) from the education, planning, transport, local government, rural health and mining sectors of the SA Government. Participants had all been directly involved in a HiAP HLA project. The second workshop involved 16 employees from the SA Department of Health (policy officers, middle managers and directors) with specific knowledge of, and responsibilities for, HiAP. These participants included those who had been involved in the development of HiAP in SA as well as those who had participated in HLA projects.
During the workshops, participants also contributed bureaucratic knowledge 21 about administrative procedures and processes and the political milieu. The research team contributed knowledge of the initiative gained through observation and interaction with the HiAP initiative and formative evaluation of HiAP projects, 24 as well as insights from research regarding healthy public policy and inter-sectoral action. As Birckmayer and Weiss suggest, the researchers “cycled through” the various sources and synthesized the information to construct an agreed upon program theory. 38
At the stage of framework development, agenda setting theory provided by Kingdon was considered to be a useful theoretical lens for analysing the HiAP initiatives because it explains how evidence, theory and political processes respond to a range of influences – ideas, interests and institutions. A key characteristic of the SA case has been the early and proactive engagement of health with other sectors prior to a particular policy proposal being suggested. This early engagement means that HiAP can contribute to the agenda-setting phase of the policy process 39 which was seen as an important point of difference between the SA model and health impact assessment. 40 The utility of Kingdon’s framework in relation to HiAP has been recognized by others 41,42 and is especially pertinent to explanations of how and why the social determinants of health rarely reach the policy agenda 43 He proposes that policy action requires a ‘policy window’ to become available through linkage of three streams: problems, policy and politics.
The Kingdon framework informed the themes explored in the research workshops, with participants asked to consider the ways in which HiAP was able to establish itself on the agenda of multiple sectors and its strengths and weaknesses as a process of policy reform to enable action on determinants of health.
Each four hour workshop included an introduction to program theory-based modelling. Broad questions exploring the impetus and start-up phase of HiAP were posed. These included the conditions that facilitated or hindered the Government’s adoption and implementation of the approach. A guided discussion then drew out the assumptions underpinning the HiAP approach, its activities and processes and the shorter-term changes and long-term outcomes expected. Workshop proceedings were audio recorded and transcribed verbatim. The research team conducted a collaborative, thematic analysis of the transcripts using NVivo 10. Following an initial open coding, selective coding 44 was applied to the transcripts to explore the participants’ discussion of HiAP and the contextual factors, assumptions, activities and outcomes that they associate with it. This analysis, together with knowledge and insights from relevant literature and HiAP documentation was used to develop a framework of the South Australian HiAP initiative.
After the workshops, the draft framework was circulated to participants by email inviting comment and clarification. Seven participants took the opportunity to provide feedback. Feedback was considered by the research team, modifications made and the revised framework was re-circulated. The framework was discussed and endorsed at a meeting of the Project Advisory Group comprising project investigators and key stakeholders (including representatives from Indigenous health, the social services peak body and the Department of the Premier and Cabinet).
As the framework was applied to guide the subsequent evaluation we drew on other social and political science theories both to identify key questions and better understand the emerging findings. This will be further elaborated in the Discussion section.
The product of this process was a graphic framework of the South Australian HiAP initiative (see Figure 1 ). The framework incorporates the components of the HiAP initiative identified through the development process, that is to say HiAP’s strategies, mediating factors influencing its implementation and its activities, the contextual factors that influence HiAP and the theories of change that underpin it. This provides a framework for researchers to seek credible evidence of change at each link in the chain.
