Sunday, July 27, 2014

The Methodological Debate

The aim of all research is to answer questions (Robson, 2011); but unfortunately the processes, as well as the answers, are not always that simple. There are different issues that can work to impede and/or complicate research within and between the hard sciences (biology, chemistry, physics, psychology and medicine) and the social sciences (criminology, sociology, social work and psychology[i]). Within social science orientated research, we are using social science in its broadest capacity here to discuss any and all research that examines human participants as well as the relationship between the self and society; and there is a vast array of different methodologies to choose from. I would firmly place sex offender research within this bracket. All research is about using tried and tested methodologies to answer specific questions, which means that you need to use the right methodology to answer the research question that you have to answer; this can sometimes be a bone of contention (Robson, 2011; Chambers, 1999).

Within any given research project there are often a diaspora of “interested” parties (for instance, funders, government bodies, other stakeholders) who directly or indirectly impact upon the research, when these “interested” parties also have a vested interest in the research (they are funding it, supplying data or access) they will get more vocal in promoting their viewpoint. All of these “interested” parties want the research to say and do different things, which can be quite frustrating (Robson, 2011; Davies, Francis & Jupp, 2011). Within sexual abuse research, these viewpoints can become difficult to navigate. There are a lot of “interested” parties; sexual abuse research does not exist in isolation and its outcomes can impact a swath of professions (police, probation, prisons, therapists, counsellors, community groups) and a number of issues (prevention, public protection, criminal justice, treatment, risk management), all of which can confine research practices.

In the main, when we talk about academic rigour and research with impact, especially in the Criminal Justice field, we are talking about inferential statistics and quantitative methodologies (NESTA). The prevailing view among both the academy and stakeholders, especially in government, internationally is that empirical, quantitative methodologies are the always the most appropriate approach to take when conducting research. Interestingly, there seems to be a step away from the traditional international governmental attitude to quantitative research and statistics within branches of government in Canada, where the emerging attitude seems to be that inferential statistics should not be used at all!

This international disparity throws up the real issue, that is that judgments are being made on the viability and impact of research based upon attitudes to methodology NOT, as it should be, attitudes to the whole research process and how the methodology sits in respect to the question. We should not play favorites with some methodologies, nor should we blacklist or ostracize others out of context. We need to make sure that the full methodological toolkit is available to us so that we can better develop and execute appropriate research; especially given that sexual abuse is a multi-disciplinary area and needs representation from different disciplines and methodologies. Ultimately, we should applaud all attempts to improve our knowledge rather than criticizing existing research because it couldn’t be of the highest quality. This is particularly true in an era when policy-makers are often unwilling to accept guidance from scientific findings. For instance, in the UK the Scottish (Chan et al, 2010) and English (Kemshall et al, 2010) of the limited disclosure scheme indicated a very low take up of the schemes, a lack of engagement with the schemes by the public and low levels of real world impact in policing terms all of which would indicate that the disclosure scheme was not doing what it was supposed to did not stop the government implementing it. A case of policy based evidence over evidence based policy maybe? There is no question that we should want the best methodologies; and we should want to best overall processes and options as well.

Not all methodologies suit all research questions, regardless of what some interested parties think, and using an inappropriate methodology will compromise the research study, impact upon the quality of the research as well as the outcomes (Robson, 2011). For instance,

-        Randomized Control Trials are often seen as the gold standard in evaluation research (Robson, 2011) and have been historically, and successfully used in social science research, and are now being used more often in criminal justice research (Duwe, 2012; Singer & Cooper, 2009), as criminology is starting to become more quantitative and experimental. However, from Larry Sherman and Richard  Berk’s domestic violence RCT, the first criminal justice RCT, to Grant Duwe’s (2012), both in Minnesota and both related to interpersonal violence interestingly, the use of RCTs is better able to polarize a group of researchers above and beyond any other methodology. The main concern with RCT’s is that they are inferentially powerful tools and should only be used accordingly, namely with a strong, variable and disperse sample. Therefore using an RCT to evaluate a well-established mainstream sex offender programme would be appropriate, relevant and useful; whereas using it to evaluate a pilot, specialized programme with 5 participants would not. This is not to say that RCT’s should not be used, but rather there are limits to the functionality, reliability, validity and results that it would produce; which would problematise the results and give a potential false negative.  In addition to the practical issues with RCTs there are also moral dilemmas as well as ethical issues relating to alternatives to treatment, public protection and offender care which although not limited to the sex offender field are very pertinent to it.

-         Using a purely quantitative approach to test the effectiveness and impact of an emerging paradigm. An example would be Circles of Support and Accountability (CoSA), where one would be better placed to use a multi-strategy design, a case study design, or even a purely qualitative approach to understand the processes and analysis the impact of CoSA, which, far from a simple treatment program is a complex social phenomenon (McCartan et al, 2014). That is to say, an RCT of CoSA should take place further down the line when a quantitative study with an appropriate (and sizeable) sample, but doing it with an unrepresentative or inappropriate sample would skew the results and problematise the research. Criticizing the methodology of existing studies too early in the research process (as has been the case in many spheres) misses the point and risks sending the wrong message to stakeholders.

The important thing to realize is that the choice to use a non-inferential methodology should not be seen as weakening the coherence of the research, its outcomes or impact just because you cannot give a level of significance to 0.05 or 0.01. If, through qualitative of case study research, you can demonstrate that a majority of your participants experienced a positive outcomes you are indicating that your research is having an impact, regardless of not having a level of significance. Instead you are showing that you are making an informed decision to use the correct methodology to answer your research question in the most appropriate light and that your results reflect the reality of what you are investigating. This means that the fringe benefits of research, like contributing to evidence based policy and practice, are realistic and grounded.

It is important to stay open minded to all forms of research methodology and use the one best suited to your research question, there are many ways to show success and impact so why decide to limit them.

Kieran McCartan, PhD
David S. Prescott, LICSW

References

Chambers, A. F. (1999). What is this thing called Science? 3rd Edition. Open University Press: Buckingham.

Chan, V., Homes, A., Murray, L., & Treanor, S. (2010). Evaluation of the Sex Offender Community Disclosure Pilot. Scottish Government Social Research. Retrieved December 6, 2010, from http://www.scotland.gov.uk/Resource/Doc/328113/0106001.pdf.

Davies, P., Francis, P., & Jupp, V. (2011). Doing Criminological Research, 2nd Edition. London: Sage.

Duwe, G. (2012) Can Circles of Support and Accountability (COSA) Work in the United States?  Preliminary Results From a Randomized Experiment in Minnesota.  Sexual Abuse: A Journal of Research and Treatment, online access August 1st 2012 at: http://sax.sagepub.com/content/early/2012/07/26/1079063212453942.abstract

Kemshall, H. & Wood, J., with Westwood, S., Stout, B., Wilkinson, B., Kelly, G., & Mackenzie, G. (2010). Child Sex Offender Review (CSOR) public disclosure pilots: A process evaluation. London: Home Office. Retrieved March 31, 2011, from http://rds.homeoffice.gov.uk/rds/pdfs10/horr32c.pdf.

Robson, C. (2011). Real world research, 3rd Ed. John Wiley.

McCartan, K., Kemshall, H., Westwood, S., Solle, J., MacKenize, G., Cattell, J., & Pollard, A. (2014) Circles of Support and Accountability (CoSA): A case file review of two pilots. Project Report. Ministry of Justice.

Singer, L., & Cooper, S. (2009). Improving Public Confidence in the Criminal Justice System: An Evaluation of a Communication Activity. Howard Journal of Criminal Justice, 48 (5), 485 -500.




[i] Please note the ambiguous labeling of psychology as “hard” (i.e., cognitive psychology, vision and perception) and “soft” science (i.e., social psychology), this is because psychology is a wide and varied discipline that in different topic areas it is both.

No comments:

Post a Comment