Challenges in Socio-Legal Empirical Research by Prof. Susan McVie (University of Edinburgh)
I am a ‘quantitative criminologist’ so my topic today is slanted towards the more quantitative elements of research but I hope to give you a balanced picture. I’ve been asked to talk about some of the methodological challenges in empirical research, particularly in law, but I hope I’m going to talk about some opportunities as well. This is a good time for developing research in a range of empirical methods and I’m going to talk about some of the current funding and other opportunities that are starting to emerge just at the moment. I’m also going to take a historical perspective so apologies to those of you who already know this story.
Let’s start with Baldwin and Davis because their description of ’empirical research and the law’ captures the kind of totality of what it is we’re trying to look at. So we’re looking at the study through direct methods rather than secondary sources of the institution’s rules, procedures and personnel of the law with a view to understanding how they operate and what affects they have. Empirical research really helps us to understand how the law works in practice. The impact that law and legal institutions and legal personnel and all the associated phenomenon, what impact they have on people, on communities and on wider organisations. And importantly as well, it’s important to understand what the wider social, political and economic factors that are going on in other disciplines, what impact those are having on law, on legal phenomenon and legal institutions.
What do we mean by direct methods? Empirical research in law should draw on the full spectrum of all research methods that are available and those fall broadly into three main domains. Firstly, we have quantitative design, so those designs that approach these social phenomenon through using quantifiable evidence, collecting quantitative data using a variety of different techniques and relying on statistical analysis of these data, data that often involves the collection of many, many cases and may involve some sort of experimental design (although not always). But essentially through quantitative design what we’re trying to do is create valid and reliable generalised claims about what’s happening in the world.
On the other hand, through qualitative design the emphasis shifts a little bit more to understanding the underlying social phenomenon and processes by using direct observation and communication and looking at our participants, analysing text which is a large part of empirical research in law, but stressing contextual and subjective accuracy over the more generality. So generality more falls to the quantitative design.
The desired or the ideal situation is to have a mixture of those two methods so we can approach our research questions using both quantitative and qualitative data, sometimes known as mixed methods. Some students often think mixed methods just means that you do a bit of both, but actually the ideal is to have a kind of integrated methodology where they are both building on each other.
In terms of engaging in empirical research within law, most of the research that is conducted at the moment falls within the qualitative domain rather than the quantitative domain. There is a lack of empirical skills within law to conduct some of the larger scale quantitative research and there’s a long history behind that. If we look at our typical research design model where we have our aims and objectives, we select our methods, we look at a range of methods hopefully building the two together, we have qualitative analysis, statistical analysis and other forms of advanced analysis. The empirical research within law has tended to focus on the qualitative rather than going down the quantitative route. There have been concerns within the academy going back to the 1960s about the lack of capacity in empirical research methods amongst those scholars who were engaging in socio-legal research.
The first kind of systematic work that was undertaken was way back in the early 1970s by the Nuffield Foundation which became so concerned about the lack of socio-legal researchers engaging in this type of research that they launched a scheme to try to encourage more people to participate in it. They didn’t put a huge amount of funding into it but they were at least trying to encourage people in this general direction. But twenty years later, 1994, there were still significant concerns about the lack of empirical legal research being undertaken and so at that time the SSRC (the predecessor of the ESRC) funded the Oxford Centre for Socio-Legal Research in an attempt to promote empirical legal research both within Oxford but also to spill this over to other institutions within the UK. However, the concerns continued so another twelve years later we had a demographic review of UK social sciences and that report by Mills and others found that there was still very poor capacity for socio-legal research using empirical research techniques. Just prior to that demographic review in 2004, the Nuffield Foundation, which I think have really been a leading light to draw attention to this problem and do something about it, launched a consultation amongst academics to establish the kind of key challenges there were, and, why this deficit was still in abundance. The consultation was led by Martin Partington, Hazel Genn & others who published a report in 2006 called ‘Law In The Real World’ which concluded that despite increasing demand (which incidently had come primarily from legal organisations and from the Government), there was still clear evidence of what they described as a ‘developing crisis’ in the capacity of UK universities to undertake empirical legal research. So you can see that the background here up to the mid-2000s is a real significant deficit in terms of empirical research skills amongst those scholars who were conducting these forms of research. ‘Law In The Real World’ was a real watershed moment and a number of significant changes have occurred since that report was published. The fact that there’s so many of you in the room today is evidence that there has been real change.
I want flick through some of the challenges that were highlighted by Genn and others in the Nuffield consultation because, although things have moved on a bit since then, some of these challenges still exist, and it’s important to be aware of them and to make sure that we don’t go back down that dark past that we’ve come from over the last thirty five years or so. Genn and others were critical of institutional factors or factors that related to the way in which law was taught. There were concerns expressed about i) the traditions and culture of legal scholarship, ii) the relative insularity from other social sciences, and iii) the fact that there was a degree of self-replication within law schools which were dominated by theoretical and text based doctrinal research.
Hence, there was a problem of impact of professional practice training requirements on undergraduate law curricula and the requirement for law students to have applied skills, thereby the focus being on law students going out into the world to practice law rather than thinking about the more kind of transferable skills that would be appropriate to them learning empirical research. Then, there was a lack of engagement between law and other areas of social sciences, and in part this was because there was a lack of inclination on the part of lawyers to engage with other disciplines, but there was also a real lack of opportunity. In many law schools across the country, lawyers or legal scholars had very heavy teaching loads which prohibited the development of some of these other areas of research skills. Moreover, there was a lack of clear definition of civil law in particular (with civil law being drawn out in the Genn report as a particular area of deficit in terms of empirical research skills and this was partly due to the sheer breadth of the subject), it covers such a broad range of different areas and that in itself inhibits the creation of a critical mass of researchers within that area, but there also a concern that there was no civilology as equivalent to criminology (which in turn had gone down a slightly different route).
Besides the criticisms around legal scholarship, Genn and others also focused their attention on funding bodies – not just the academic funding bodies that we typically think of but also the government itself. They talked about an absence of sustained and predictable sources of funding for empirical research work, particularly in the non-criminal related areas. Within criminology and criminal justice there had been more support for the development of empirical research but much less so on the civil side. There was a paucity of funding streams for people who wanted to conduct this sort of research, there was also a real paucity of data that was available for those scholars who wanted to engage in this sort of research and often what happened when legal scholars were trying to undertake the research was that they were having to undertake such tortuous processes of dragging data out of various organisations that it created real barriers to this research.
Linked to that was an absence of research training or any kind of mentoring for new recruits. There was some recognition that there were pockets of excellence across the UK in terms of legal scholars who were doing excellent work but there wasn’t a critical mass for empirical legal researchers to come together form any substantive stimulating and intellectual environment for researchers to get together, so it made it difficult to support early career scholars and PhD students.
Finally Genn and others were critical of institutions themselves. They found that in many institutions there were barriers that inhibited cross-disciplinary development between law and other subject disciplines. They talked about a kind of Victorian brigading of knowledge within law and in those institutions where law had worked in an inter-disciplinary way it tended to be more driven by managerial or financial considerations rather than through any kind of underlying desire to expand the subject discipline. Until relatively recently there hasn’t really been an incentive in terms of the RAE for legal scholars to work in an inter-disciplinary manner.
So those were some of the key issues that were highlighted by the Nuffield Foundation report and I think what they concluded essentially was that there was a need for integrated strategy of short, medium and long term initiatives that would address the needs of potential researchers at various different career stages and that incentives in training for those coming from different routes needed to be built up in terms of focusing on not just undergraduates and postgraduates but also putting in place training incentives for researchers at various stages of their career. And some of the changes that they recommended were targeted at all universities particularly in terms of trying to make universities think more broadly about developing their curricula. But most of the recommendations were targeted at those institutions where there were already kind of small hubs of centres of excellence and trying to focus them in developing a critical mass in order to kind of create an empirical legal research community that would become self-sustaining.
Now I want to switch tack and talk about parallel concerns within the UK social science about the deficit in quantitative methods – this shows a similar trend. The concerns about the lack of capacity to conduct quantitative research in the UK started to emerge in the 1960s. It wasn’t until 1987 that the ESRC commissioned their Horizons and Opportunities Report and identified a really wide discrepancy between what potential employers from a range of different areas (public, private and voluntary sector), what they said they were needing from social science scholars when they graduated from university and what they were being faced with in terms of the graduate population.
Again it was another twenty years later, with a number of reports published between the 1980s and the mid-2000s that highlighted again this deficit in quantitative methods within social science, that some action started to be taken. The Higher Education Funding Council for England made a statement about the strategically important and vulnerable subjects in the UK and highlighted that quantitative social science was of particular concern because its supply was seen as insufficient and particularly because quantitative social science underpins all social science disciplines.
The demographic review of the UK social sciences, mentioned earlier in this passage, had not only highlighted the deficit in empirical research skills amongst legal scholars also identified that advanced quantitative methods was a problem right across the UK. And following that report, funders such as the ESRC, the Nuffield Foundation and also the British Academy have started to pump a large amount of funding and investment into projects that are focused around quantitative methods. So there’s a large amount of funding going in there and things have started to happen.
But I think what is interesting about this is that if we go back to the Genn report, they specifically state in the report that those who come from legal backgrounds have no easy way to acquire the skills to do empirical research, data collection and that statistical and quantitative analysis skills are particularly hard to develop. So Genn had specifically identified in the report that this was an area of concern within empirical research and law, but in the report itself it makes no specific conclusions or recommendations about developing quantitative skills amongst legal scholars and only a very small number of the Nuffield funded projects involve any kind of quantitative analysis. Again, although there’s been an expansion in empirical research in law, the vast majority of it appears to be qualitative.
Now I think there are some peculiarities within law which are worth mentioning and which makes it difficult in some instances to conduct quantitative research and again that can be a barrier to people even considering that as a route that they want to take. I like Shari Diamond’s description of legal activity as a leaky funnel. When it comes to law and legal processes and institutions, we start off with a large number of cases and by the time we get to whatever we’re interested in we’ve lost half of them because the leaky funnel means that people or institutions drift out. So if you’re trying to study and measure processes that have some inherit element of attrition or potential bias in the way that the processes operate then obviously that has research design challenges.
There are many vagaries of legal decision making that make it difficult to conduct robust quantitative research because the way that people behave in legal processes and institutions don’t always conform to the kind of predictive models that us quantitative scholars like to develop. And then there are various selection effects that happen within some legal processes, particularly in terms of the way organisations and individuals behave that make it difficult to research, make it difficult to make claims about impact of for example legal processes when there are other things underlying that. These are all problems that are all faced by other subject disciplines as well though, but I think these have been perceived to be particular barriers and that somehow law is different to everything else and in fact when we start to look at some of the medical research models for example it’s not.
For the purposes of quantitative methods we were particularly interested in conducting causal inference and we again come across problems within law. The practicality for example of conducting controlled experiments within some legal research, there are concerns about small numbers of cases, about the cost of conducting these sorts of research projects or the time that it takes and more particularly often concerns about the ethics that are involved in conducting randomised control trials.
There are also particular problems in terms of the availability and suitability of collecting robust data within some empirical legal research fields. Criminal justice is much better served in terms of the amount of data that’s readily available. Those who are engaged in more civil justice areas have had to work much harder to try and collect their data to access it and of course particularly if you’re interested in private institutions or organisations, accessibility to data is a huge problem and concerns about ethics and privacy abound.
So what’s to be done in terms of building capacity in quantitative methods then within empirical legal research? I come back to Shari Diamond who about twenty years ago was saying that our understanding of legal phenomenon deepens when research take into account how scholars and other disciplines would approach the questions they’re asking and the evidence for the claims they’re making. So while legal scholars have many subject specific strengths I think the key to expanding empirical research in this area is really working in a disciplinary fashion.
I wanted to just say what lessons could be learned from other disciplines and there are two key things I think that certainly from my own perspective I have learned greatly from other disciplines, originally a biologist I know what it means to make that leap from one discipline to another. The first is greater exploitation of social science data. There are huge amounts of data out there that are underused and we have to capitalise on those data and demonstrate to a range of people including funders and those people that influence policy and practice how those data can be used. We can use some of the existing data sources and further develop them to collect a broader range of information particularly around law or justice and we need to put more pressure on funding bodies I think to release more funding for new sources of data around empirical research and the law. And I think there’s never been a better time to do that. The funding bodies are more receptive to listening to the legal academy now than they ever have been.
I think there’s also a huge amount that could be done in terms of greater exploitation of some of the administrative data sources that are available and some of the administrative data sources that aren’t available but could be made available. We need to put more pressure on organisations to improve the quantity and the quality of the data that they collect, and I’m not just talking about legal organisations, but I’m talking about governments as well. Some of the data that’s available particularly on the civil justice side is appallingly poor quality. Much more needs to be done to get them to improve that.
We need to persuade organisations of the benefits of sharing their data and to create opportunities for greater data linkage and again there’s never been a better time for demonstrating the value of linking different data sets together. The Scottish Health Informatics Programme has been forging the way ahead in this area in terms of health data. The Scottish Government have now created a data linkage framework which brings together a large number of organisations who want to share their data. The Justice Board within government which covers both criminal and civil justice is very keen to see more projects that involve linked data to the extent that funding could be made available if they were persuaded of the value of doing this. At the moment the government, the Justice Board currently has a dashboard, it’s much less impressive than the dashboard that we saw earlier, but it a dashboard that tells senior policy makers within government how justice is doing. It’s very simplistic, it’s a traffic light system with green, amber and red and it shows them if things are improving, staying the same or going down. There is only one little graph on that that reflects civil justice. So I think again there’s huge scope here for persuading government that far more measures around the value of civil justice need to be developed.
A quick example talking about exploitation of existing data on civil law – the Scottish Crime and Justice Survey which is one of the biggest social surveys that’s carried out. At the moment it’s biannual in Scotland but we’re hoping to get them to start doing it annually again. There are four years worth of data from 2009 up to 2012 which involve just about fourteen thousand cases and they have covered a huge number of questions around disputes and dispute resolution, around the home and family living arrangements, health and well-being, money and finance and fair treatment and there’s a range of follow up questions where people are asked about what the outcome of the situation was, whether they use a method of resolution and what that was, whether they were satisfied with that, what sources of help and advice were available to them, and if they haven’t taken resolution there are some questions about that too.
I’m sure there must be somebody who is interested in this data and nobody is using it. There are fourteen thousand cases over four years. I did a quick analysis of it yesterday and found that sixty percent of people in that survey reported that they had experienced some form of dispute over the last three years, and you can see the kind of range. Most of them were neighbourhood disputes but you can see there’s a range of different things there, faulty goods and services comes out quite highly. Now sixty percent of our population are experiencing some form of dispute in the last three years, surely somebody is interested in analysing that data. At the moment it’s sitting there in big data sets not being used.
There are many, many opportunities and I’ve outlined some of them today, but there are also some challenges that you and others will face. There’s still a significant hurdle in terms of trying to get socio-legal researchers to be convinced of the benefits of using quantitative data or even mixed methods and there’s also barriers in terms of training and getting people to even think about that area of research within their scholarly activities. And the barriers are cultural, there’s still many people that work within socio-legal areas that think that numbers are bad, but there are also practical as well as I said socio-legal areas of enquiry are often seriously lacking in data and there needs to be significant investment there.
The administrative data that you can collect from organisations is good but it’s insufficient to explain all of the processes as I’ve show with my example with the self-report data and the convictions so data linkage is really the key to understanding some of these complex mechanisms and as you all know interdisciplinary working is also key to the health of socio-legal research, not just in terms of substantive understanding but also in terms of methodological advancement and I encourage you all to learn from each other in terms of your methodological skills. Partnership working, data sharing, data linkage and in the future the development of more online resources are probably the keys.
Observations/ Comments/ Questions
Prof Martin Kretschmer: I think it was extremely valuable. I’ve got two observations relating to what you said to our research programme. So the first set on the traditional socio-legal approach which investigates law by its institutions, processes and actors. We’ve got some projects of that type in our programme whether it’s the small claims court, whether it’s enforcement, whether it’s the composition of the chambers of the ECJ, so that’s a traditional kind of socio-legal work. Most of the other things we do our unit is really the creative economy, so we’re looking at what creators do, what firms do and law plays a part somewhere so that’s much more obscure and tricky. But if you replace the work in our talk criminal with creator, everything you said applies okay. So studying the creator all the methodological questions you posed apply directly and I think one of the challenges we have is that we need to, it’s not enough to have self-reporting, it’s not good enough just to talk to the creator and to say tell me something, you need to find a different angle somewhere and the imagination is needed to find the source of data which gives the picture a good triangulation but there needs to be something else there, I think it’s a real challenge and I think you gave good examples of how this could be done. But you have to look for where you find the sources of data and that’s part of the creative process here.
Prof. Ruth Towse: I have a real question which is to what extent are universities providing general research methods courses to which anybody could go as it were. I worked in Erasmus University in Rotterdam for a long time, ten years and all students doing a PhD had to go to some externally organised research programme. I get the impression from some of the things that people are saying that these don’t exist. I’d just like to know if they do somewhere because why couldn’t legal scholars go to say a social science or an economics one or have a general one for everybody?
SM: I think it’s improving, the funding bodies such as ESRC have put a lot of pressure on institutions to ensure that their postgraduate students are getting a broad range of research skills training. There’s still quite a large degree of opportunity for those students to decide whether or not they do quantitative skills. If you go the US for example, all of your students will have some degree of quantitative training, not all of them will focus at the advanced end but they’ll all come out being able to do numbers to some extent. The same is not true in the UK. I think quantitative training is being made more available, it’s patchy across institutions, some institutions are better than others but there is a recognition that it is important and postgraduate students are being offered it. Whether they take it or not is another matter and again that comes down to quite often supervisors who are not quantitative themselves, may not appreciate the value of them having a kind of broad range of research skills. More recently the ESRC, the British Academy and the Nuffield Foundation have made available funding for a series of centres of excellence in teaching at undergraduate level at quantitative methods. They’re looking to fund I think around fifteen centres across the whole of the UK which will probably go to the usual suspects I suspect but the decision on that hasn’t been made, the applications are with the funders at the moment. But there is a recognition that even focusing your resources at postgraduate level it’s not encouraging enough social scientists to take advantage of the broad range of training opportunities.
Prof. Robin Williams: An observation and a question, in terms of the barriers to creating a socio-legal strand I remember when SCRIPT was first set up which was the research centre here on social property and technology law and Hector McQueen decided that he wouldn’t appoint….do what we did in science and technology studies which was to create a cadre of long-term inter-disciplinary research fellows. He wouldn’t do that because he felt that lawyers needed to get back into the field. So the best he could get was three months or six months and so they spent large sums of money on employing the senior staff to write good papers, and they wrote wonderful papers, and using junior staff as research assistants. So it seems to me that one of the issues is the occupational structures and skill structures that exist and what happens in law when the call of professional practice is so strong, so that’s an observation.
The other is a question I really liked your discussion about the difference between self-reporting and admin data but I wondered what was your strategy then for resolving that differences? How do you interpret and explain that? Because that seems to me to raise some of the issues about how large scale data and other forms of data might interact together.
SM: Well I didn’t want to talk too much about the research but there’s a whole other paper I could give about what we did. We used a variety of different techniques to bring these different datasets together and what we were able to demonstrate was that there was a systematic impact of institutional contact on children, so basically dragging more and more children into the youth justice system, recycling them around without doing anything productive with them was just basically setting them up to be chucked into the adult criminal justice system by the time they go to their sixteenth birthdays. So we were able to persuade government that more intervention with young people was bad, less intervention and more diversionary activities was good and what they did was completely restructure the youth justice system and introduced a system where they now keep far more kids out of the system and deal with them through multi-agency partnership working so our impact story is very good. We’re hoping for a four star on that one.