The Research-Practice IMPACT Relationship AND THE MYTH OF - - PDF document

1

‘IMPACT’ AND THE MYTH OF RESEARCH-BASED POLICYMAKING AND PRACTICE

Martyn Hammersley

The Open University, UK

Seminar given at the Centre for Organizational Research, University of Roehampton, March 2014

The Research-Practice Relationship

• There is a long history of debate about

the role of research in relation to policymaking and practice.

• It has frequently been argued that there

is a ‘research-practice gap’, where instead there should be continuity: linear, dialogical, or interactive.

• This ‘gap’ has led to complaints from both

sides.

Complaints about research

• Not closely enough focused on the concerns
• f policymakers or practitioners;
• Fails to produce findings at the time they are

needed;

• Generates conflicting and confusing evidence;
• Provides evidence that is at odds with what is

well known to policymakers and practitioners, so that its validity seems weak;

• Produces conclusions that are inaccessible to

practitioners, for example because too elaborate and qualified, or jargon-ridden.

Complaints about policymakers and practitioners

• Closed-minded or set in their ways, and

therefore resistant to new perspectives;

• Committed to the dominant ideology and

unwilling even to consider radical challenges that research findings may imply;

• Untrained in the capacity to understand and

make use of research;

• Lacking in the motivation required to seek out

research evidence, and to reflect on their decisions in light of it.

Evidence-based Medicine

The evidence-based medicine movement, from the 1980s onwards (Pope 2003), required that:

• Clinicians must access research evidence

about ‘what works’, and use only what has been scientifically validated, rather than relying upon their own past experience or outdated training.

• Funding must be directed into research aimed

at discovering ‘what works best’. This research should use randomised controlled trials (RCTs), and funds must be allocated for systematic reviews designed to synthesise findings from multiple studies.

The rise of ‘research-based’ policymaking and practice

• Spread of the notion of evidence-based practice

to other areas: education, social work, etc. This fitted with the ‘new public management’ that became influential in the 1990s, aimed at making public sector professionals more ‘transparently’ accountable (Pollitt 1990).

• The promotion of ‘evidence-based

policymaking’, for example the ‘behavioural insights team’ (the ‘nudge unit’) in the Cabinet Office (Haynes et al 2012) advocating RCTs in many areas of Government policy.

2

Changes in the research ‘contract’

• Looking at the history of research funding,

we can see a shift from the assumption that research will be beneficial in unpredictable ways, and should not be subject to external strategic management, to the requirement that it be more closely directed towards bringing about practical results and evaluated in these terms (Guston 2000).

• ‘Strategic research’ (Irvine and Martin,

1984), Pascal’s quadrant (Stokes 1997)

• Mode A and Mode B (Gibbons et al 1994)

Pasteur’s quadrant

This is roughly what Hodgkinson et al (2001) refer to, in the context of management, as ‘pragmatic science’. I note that many of the same debates about Mode 1 and Mode 2, and various versions of Mode 3 that purportedly combine all the benefits of the other two, have been present in management science for some time, as in other fields: Huff 2000, Starkey and Madan 2001, Grey 2001, Hatchuell 2001, etc.

The investment model

• Over the past 30 years public sector

expenditure has increasingly been conceptualised in economic terms.

• More specifically, it has been viewed as

‘investment’ – with the implication that a ‘return’ should be expected and should be demonstrable.

• University teaching and research are among

those parts of the public sector where this conceptualisation is especially strained, the arts and sport would be other examples.

Strategic funding of science

• Application of the investment model to

research funding first arose with research funded directly by government departments

• However, there was an early attempt in the

1950s to apply this model to the funding of university science (Polanyi 1948; McGucken 1978). This was defeated, but later came to be applied, primarily as a result of the very large amounts of funding required by some areas of natural science. Relevant background here is the increasingly industrial and commercialised character of much science (Ziman 2000).

The impact agenda

• A central concept in application of the

investment model to research has been the notion of ‘impact’.

• Institutional requirements have been

introduced that researchers work to maximise the impact of their research, through dissemination and engagement activities.

• Also required is that these activities and their

‘impact’ be ‘evidenced’.

• Certain methods, notably RCTs, have been

viewed as intrinsically ‘high impact’ because they ‘demonstrate’ ‘what works’.

The impact metaphor

• The analogy is, of course, a physical one, and
• f a relatively simple kind.
• The model is of one object (say, a billiard ball)

hitting another.

• So, research is conceived as coming into

contact with policymaking or practice and sending it in a predetermined direction, in much the way that a cue ball can do this to an

• bject ball in billiards.
• This conception of the relationship between

research and practice is held not just by lay people but also by many researchers.

3

A literary example of the kind of ‘impact’ many researchers desire

‘This savage novel of the bestial conditions among the stockyards and slaughterhouses

• f Chicago in the early years of the twentieth

century is perhaps the most influential and harrowing of all Upton Sinclair’s writings. […] So great was the furore caused by the publication of this novel that the food laws of the United States were changed within six months’. (Back cover blurb for The Jungle: Sinclair 1906)

But is maximising ‘impact’ always desirable?

• Carol Weiss long ago raised doubts about this.

She argued that what is important is not so much ‘to increase the use of research, but to improve the contribution that research makes to the wisdom of social policy’ (Weiss 1979:431)

• The idea that maximising impact is always

desirable relies on an Enlightenment assumption about the progressive role of scientific knowledge. Yet this was questioned even in the eighteenth century, for example by Rousseau.

Questioning the impact metaphor

• The cue ball is struck so as to hit the object
• ball. The first ball thereby has directionality

and momentum: Is this true of research?

• The object ball is stationary, and only moves

as a result of direct and immediate contact by the cue ball. Is this true of policymaking or practice?

• Contact between billiard balls operates under

laws predicting repeatable patterns of action, given certain background conditions. Is this true of the relationship between research and policymaking or practice?

Myth of research-based practice

The myth = that research can tell us what is the best policy or practice. Three reasons why this is misleading: a) Research cannot validate value conclusions: the ambiguity of ‘what works’. b) It can only provide limited and fallible evidence about the effects of particular policies

• r practices; and this is not its main function.

c) Research evidence must always be combined with local knowledge in professional judgments about what is best in particular contexts.

Consequences of the impact model

• What will be the consequences of this model

for academic research? Is it, as Brewer (2011) claims, a ‘sheep in wolf’s clothing’ that enables social science to demonstrate its value?

• Will ‘demonstrating impact’ remain forever a

paper exercise, more bureaucratic nonsense with which one must appear to comply so as to obtain research funding? Or will it force the kind of change that RCUK and ESRC propose: the goal of achieving impact being built into the design of all research projects from the start (see Holmwood 2011)?

4

A dire warning

‘In line with those who think that the impact agenda can be shaped toward the ends of public social science, Nowotny et al (2001) set

• ut an attractive image of a new “public agora”,

drawing upon socially distributed expertise. Yet the reality seems to be […] the rise of privately- negotiated user-researcher relationships and the replacement of disciplinary hierarchies by those of government strategic priorities. […] There is very little of the “agora” about these relationships’ (Holmwood 2010:14-15). He suggests this is ‘a pathway to mediocrity’ (p16).

Types of inquiry

• Inquiry-subordinated-to-another-activity
• Practical research: concerned with providing

specific information of value for policymaking

• r practice
• Academic research: aimed at contributing to a

body of knowledge about particular broad topics: for example, types of organisational structure or the causes of financial crises. These types are very different in character, and serve divergent purposes, but they are all important and their distinctive value must be recognised (Hammersley 2002:ch6).

Conclusion

• I have sketched the history of debates about

the relationship between research and policymaking or practice, noting the recent rise

• f the idea of these should be ‘evidence-

based’.

• I argued that this proposal is founded on the

application of an investment model to research funding.

• Central to this is the demand that the ‘impact’
• f research findings be maximised.
• I examined this metaphor and outlined its

weaknesses, and the dangers surrounding it.

References

Brewer, J. (2011) ‘Viewpoint – From Public Impact to Public Value’ Methodological Innovations Online, 6, 1, pp9-12. Gibbons, M. (2000) ‘Mode 2 society and the emergence of context-sensitive science’, Science and Public Policy, 26, 5, pp159-63. Grey, C. (2001) Re-imagining relevance: a response to Starkey and Madan’, British Journal of Management, Special Issue, S27-S32. Guston, D. (2000) Between Politics and Science, Cambridge, Cambridge University Press. Hammersley, M. (1995) ‘Selling science short’, Times Higher Education Supplement, December 1st, 1995. Available at (accessed 29.1.14): http://www.timeshighereducation.co.uk/news/selling-science-short/95966.article Hammersley, M. (2002) Educational Research, Policymaking and Practice, London, Paul Chapman/Sage. Hammersley, M. (2005) ‘Is the evidence-based practice movement doing more good than harm? Reflections on Iain Chalmers’ case for research-based policymaking and practice’, Evidence and Policy, vol. 1, no. 1, pp1-16. Hammersley, M. (2013) The Myth of Research-Based Policy and Practice, London, Sage. Hatchuel, A. (2001) ‘The two pillars of new management research’, British Journal of Management, 12, Special issue, S33-S39.

.

Haynes, L., Service, O., Goldacre, B., and Torgerson, D. (2012) Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials, London, Behavioural Insights Team, Cabinet Office, UK Government. Hodgkinson, G. et al (2001) ‘Re-aligning the stakeholders in management research’, British Journal of Management, 12, S41-S48. Holmwood, J. (2011) ‘The impact of “impact” on UK social science’, Methodological Innovations Online, 6, 1, pp13-17. Hood, C. and Peters, G. (2004) ‘The Middle Aging of New Public Management: Into the Age of Paradox?’ Journal of Public Administration Research and Theory, 14: 267–82. Huff, A.S. (2000) ‘Changes in organizational knowledge production’, Academy of Management Review, 25, 2, pp288-293. Irvine, J., Martin, B.R. (1984) Foresight in Science: Picking the Winners, Frances Pinter, London. Lane, J-E. (2000) New Public Management, London: Routledge. Lenhard, J., Lücking, H., Schwechheimer, H. (2006) ‘Expert knowledge, Mode-2 and scientific disciplines: Two contrasting views’, Science and Public Policy, 33, 5, 50. MacGregor, S. (2011) ‘The impact of research on policy in the drugs field’, Methodological Innovations Online, 6, 1, pp41-57. McGucken, W. (1978) ‘On freedom and planning in science: the Society for Freedom in Science 1940-6’, Minerva XVI. 1, pp42-72.

Novotny, H., Scott, P. and Gibbons, M. (2001) Re-Thinking Science: Knowledge and the Public in an Age of Uncertainty, Cambridge: Polity. Polanyi, M. (1948) ‘Ought Science to Be Planned? The Case for Individualism’, The Listener (September 16, vol. 5, p17). Pollitt, C. (1990) Managerialism and the Public Services, Oxford, Blackwell. Pope, C. (2003) ‘Resisting evidence: evidence-based medicine as a contemporary social movement’, Health 7, 3, pp267–282. Sinclair, U. (1906) The Jungle, New York, Doubleday, Page and Co. Smith, K. (2013) Beyond Evidence-Based Policy in Public Health: The interplay

• f ideas, Basingstoke, Palgrave Macmillan

Starkey, K. and Madan, P. (2001) ‘Bridging the Relevance Gap: Aligning Stakeholders in the Future of Management Research’, British Journal of Management, 12, Special Issue, S3-S26. Stokes, D. E. Completing the Bush Model: Pasteur’s Quadrant, available at (accessed16.03.09): http://www.dcc.uchile.cl/~cgutierr/cursos/INV/Stokes.pdf) Weiss, C. (1979) ‘The many meanings of research utilization’, Public Administration Review, 39, 5, pp426-31 Ziman, J. (2000) Real Science, Cambridge, Cambridge University Press.

5

Advert

I have recently written a paper focusing

• n the impact metaphor. It is in draft

form, but can be found temporarily on the documents page of my wordpress website (search for: martynhammersley wordpress). Alternatively, you can go straight there with this web address: http://martynhammersley.wordpress.com/ documents/ You will also find these slides on the slides page of this website.