Extending the scope of enquiry across different settings and longitudinally

Extending the scope of enquiry across different settings and longitudinally – Prof. Robin Williams (University of Edinburgh)

Cliff Jones, a leading computer scientist in the UK said the only question about SAP is why people have bought the wrong software! It was an axiomatic principle, and, people say it’s marketing, planning, and other interesting stories. So this is an impossible project, a little bit like the physicist who calculated that bumblebees cannot fly. They do fly!

Now, this outcome is a total clash with the dominant tradition in social science. And the dominant tradition sees that organisations are unique, in fact, even within organisations, departments and individual workers will create their own methods of doing things, such that any standardised solution will require massive customisation of the standard solution which will be expensive, will invoke new, create new kinds of risks and vulnerabilities, or you have to redesign your organisation to meet the needs of the package, and that’s seen as being unhelpful and disruptive. And this is based upon work by a Lucy Suchman who was at Xerox Palo Alto Research Centre, an anthropologist, who studied why traditional approaches to system design failed, why engineers frequently failed to understand how organisational processes operated and she flagged the active role of organisation members in getting organisational procedures to work. She said organisational procedures are not the expression of a universal rule that can be captured in a computer, but they are improvisations and actually you can show quite well that on a day to day basis organisation members have to use their ingenuity, their interpretability, their adaptability to deal with ambiguities – things that computers aren’t particularly very good at doing. From this, so called interactionist tradition, the assumption was organisational practices in organisations are unique and standard systems must fail. As a result loads of people like me went and did detailed ethnographies of technology design and of technology implementation.

I now want to talk a little bit about the problems of those studies. And there are several kinds of short term studies which emerged, so for example, we found that there were about 600 studies of the implementation of this particular package software and enterprise systems. Often they’re before and after studies, often they’re impact studies, they’re studies typically associated with a consultant or a vendor themselves who say here is the organisation before, here it is after the technology, this is how the technology overcame those problems. And typically those studies are actually so closely associated with supplier views that they tell you very little, but they produce can-do accounts, they produce accounts of successful modernity, of how the organisation problems are captured and resolved by resorting to these package solutions.

Following in the wake of those things which typically came out of the business studies department or computer science departments, we see more serious social science studies which I will call implementation studies, but they are studies taken often in the immediate aftermath of the adoption of these technologies. Now these technologies cost at least £1 million to buy, you then need to spend between one or ten years to actually get them to work in your organisation – they are highly complex technologies. And if you go and look at one of the organisations that have adopted it, in the year after, or the two years after what you find is that people are very aware of the gulf between the promise that the technology was supposed to deliver and the actual experience.

So, these kinds of social science accounts produce a kind of no-can-do critique of the can-do vendor’s rhetoric. So you saw two kinds of literatures pointing entirely in the opposite direction and this literature always emphasises how we did things different as to how the package does things. And we wanted to do things differently. Interestingly the account that is produced by social scientists going in to the organisation, talking to junior organisation members and the people implementing the thing is of how SAP and the other enterprise system suppliers don’t care about the vendor. They just throw it over the wall, typical language. So you see an account of the uncaring vendor. Interesting, how does the uncaring vendor manage to sell to all the FTSE 100 companies? Not explained.

So we looked at the weakness of these kinds of studies and often they’re very empiricist, stuff coming out of the business studies department do not necessarily have a strong theoretical framework and often have a rather weak methodology. But also, even in social science departments we see a predominance of short term studies, so ethnographers go in and do a work place study, usually for a couple of years. But why two years? Because three years is the maximum we’re likely to get money from the SRC or three years is the maximum time you might have for a PhD.

So the timing framings or research grants condition the kind of studies we undertake. And there is an evident failure to reflect upon how the choice of research design might shape the findings. Different kinds of studies yield different kinds of knowledge. They’re better at producing different kinds of outputs. Post-implementation studies will highlight the gap between promise and delivery.

In addition then we saw the disciplinary fragmentation of research, and in particular most of the studies were of implementation, very few of design, and virtually nobody looked at design and implementation in tandem. Why? Well because design takes ten years, design and implementation then you’ve got minimum 13 years. How do you do that? So, there are pragmatic and disciplinary barriers to doing integrated studies.

We tried to overcome this, my colleague Knut Sorensen articulated the social learning perspective and said that these technologies are complex and are often associated with a kind of trial and error process described as learning by doing, as organisations try to make sense of technologies and integrate them with their purposes and practices. And he extends this to more general technology in other settings as well. And one feature of that is inaffusion. And here I refer to my colleague Jamie Fleck who coined this word ‘inaffusion’.

In the past there was invention, innovation and diffusion. Technology was developed in the lab, put into the market and diffused. Actually, Jamie Fleck showed that artefacts aren’t fixed as they emerge from technology supply but often reinvented in the struggle to get them to work and to be useful in their context of use. And there’s a process when you implement a technology of learning between the supplier and the adopter, in which the technology is often broken down and sometimes reinvented, and sometimes these local reinventions can produce ideas for future technologies which can be adopted into future technologies as well.

And in fact, if you look at work place technologies, most of them have a history. Most of them are re-implementations of technology developed earlier. This was particularly the case in CAP-M. CAP-M that we were looking at in late 1980s early 1990s could be traced back to stock control system being used by Ford and Boeing in the 1960s, and you could see the kernel of those old systems, other functions were added on to make them into the enterprise systems that we see today. So the technology has evolved over multiple cycles of being improved by suppliers, implemented, refined in the context of implementation. So we see the MRP evolves across many cycles of implementation, we talk about the spiral of innovation.

So, through this we said well we need to move away from these episodic studies of particular moments and locales whether of design or implementation, and have multi-locale, multi-temporal studies and propose in particular a need for more careful consideration about the choice of research design. We proposed what we call strategic ethnography, so theory should shape how you choose your research sites, and of course there’s an element of pragmatism as well, opportunism has to be exploited. But we want to extend the scope of enquiry, not just take for granted that a company study and implementation study or design study is sufficient. And in practice in our own personal biographies we’d assembled quite a wide range of studies, which I’ll briefly review to you, to give what we considered to be an account of the biography, the evolution of an artefact and the processes associated.

We looked at its design, we looked at its implementation, sorry, design, procurement, implementation, post-implementation support. And we also for example contrasted an established supplier with a new supplier and how they tried to understand their market more effectively. And in this process we found out that there were new kinds of players that we didn’t know about, new kinds of spaces. And in particular we found the role of industry analysts as people who helped vendors understand these complex packages and how they work.

So we concluded, SAP conquered the world, but it didn’t do it as the modernist story that the computer scientists might like of the victory of a universal solution. They conquered the world by one sector at a time. They moved from manufacturing to retail to banking to local authorities and universities. And they did that also through a strategy we describe as generification. So we saw for example how a new package was born, how a company which made accommodation management software for a university thought we can sell this on to other universities, and sell this on more generally. Now this company had one good solution for one company, it went on to a new customer and said well we want this, but we want these other functions as well, and very rapidly it produced a very baroque solution that become unduly complex, difficult to maintain and it also had a whole load of functionality that the first customer didn’t want. And it had new functionality from the first customer that the last customer didn’t want. So they immediately had to make some decisions if they wanted to make their solution a package solution, they needed to make some decisions about what functionality they were going to take on board. So that was a new supplier and then we looked at SAP and other enterprise solutions and we found that they segment the market, they decide to make alliances with certain kind of strategic partners who are particularly important and they design their product around certain selected partners are deemed to be a good enough representation of the whole market. And then they work very closely with their customers in developing a solution that meets their needs.

Time’s short so I won’t go into this in detail, but how do they do that? Do they do it by being in a room like this where a load of customers are saying what their needs are, and the company says what requirements do you need? Interestingly we saw in this process we saw they try and get their customers to align their needs. If several customers align their needs, then the company will support that. So the customers are actually beginning to align themselves to the anticipated functionalities that the system will involve. And I won’t give you the details.

I mention that they segment the market, the supplier has its pilot users around it, and if those users want it then the supplier will develop it. If you are only a transactional user, far away, you can have the functionality, but only if you pay for it. So they’re beginning to sort their customers and their business, the strategic importance and commercial importance of their customers in terms of what functionality they do. And in the end you get this kind of complex package which contains some things, a kernel of standard database processes, some functionality which maybe everybody uses, there are some where, this is the US model and we’ll have the US model in another version which is the UK model so they cater for a limited range of diversity. And in this way they produce packages which don’t represent every single organisation but produce a doable accommodation between most organisations in particular sectors and good enough that most organisations can buy into that package solution.

Then we came across another group and my colleague, Neil Pollock, managed to get access to the email systems of several partners and he could see this huge web of relationships that existed between suppliers and vendors and one of the vendor contacts emails a pilot site customer and says look Gartner have produced some more positive assessments of our products, and later, you see a plot emerging between the customer and the supplier to try and improve the reputation of this product. Why? Because the purchaser wants the product to continue to be supported and the people associated with that product, within the enterprise system organisation, they want to get more resources for their team as well, so there’s a complex alliance of interest between customer and vendor here that’s going on. And they’re trying to influence Gartner, produce these two by two matrices and they assess all the products on the market and they assess them in terms of the ability to execute and the completeness of vision. So there are two features they assess, the vendor and its products. And this is about not just the current capability of the products but the commitment of the vendor to that field and its future performance i.e. they’re assessing things that cannot be assessed, but they’re producing numerical, quantifiable accounts and clearly the thing you want to do is to be in the top right hand segment in terms of ability to execute i.e. technical capacity and completeness of vision i.e. where you’re going in the future.

So Gartner have had extraordinary authority over this market. Why have they become so powerful? Well user organisations buy these systems every ten years or so, so you can’t use customer experience as a way of assessing the vendor’s performance, it costs millions of pounds and takes many years to find out whether the technology works and whether it works for you. These are complex non-material products, their properties cannot be assessed by inspection, they can only really be assessed by use. By you using them and it costs you at least a million pounds and at least a year or two to use them. So this is a very hard system for caveat emptor to apply, this is a system where the market fails.

So what happens in the past was that potential adopters would talk to their pals, they would talk to their peers in the industry and see who worked, who was successful because that’s a very inefficient and reactionary way of assessing things, you get that knowledge, that community network reputation only comes to you slowly and you only look at parts of the market so that model is insufficient and Gartner has emerged as the organisation that maintains links with the adopters of technologies and the vendors of technologies and collects community knowledge and makes it available on a fee for service basis to technology adopter organisations.

So Gartner has arisen from the very asymmetry of access to information within the market for these complex products and we describe this as what we call the new knowledge infrastructures of the IT systems market. The IT systems market can only work effectively through the operation of these intermediaries who are able to through their links to customers, vendors and intermediaries and consultants to actually command, coordinate and exchange and supply knowledge.

In our next stage of our project it turns out there’s another group emerging: industry analysts. The role of the industry analyst who may be a consultant or somebody from marketing or sales or somebody from a vendor organisation is to study how Gartner works and how a product therefore needs to be presented to maximise its position in the magic quadrant. So there’s a whole thing dot-ology. A new technique of how you get your product to be in the top right hand quadrant. So the assessment process is now changing. We’re seeing the elaboration of increasingly complex divisions of expert labour in trying to get the market and trying to gain the market. When we started the study of course we had no idea there was this group called Gartner.

Conclusions: To understand this we needed to study different locales, the site of implementation, the site of development, we needed to decide the spaces where developers and users interacted together. We needed to look at the multiple cycles of interaction between technology as it went through cycles of supply, implementation, use from re-defined. And from this emerged our idea of the need to address the biographical aspect which isn’t to say that it’s just long term studies. This is sometimes seen to be akin to the Hutchins Cube which suggests that you might study something at the interactive moment of a few seconds to the long-now of centuries. But we need to think about temporal framings, when we’re studying things. We also need to look at contextual framings, for example we highlighted the importance of the technical field. A particular field, ARP, the existence of this field helps align expectations about what’s good about technology, where technology should go forward. As well as providing a space in which differential assessments of different products can take place.

So we propose, if you want to understand ERP you can’t understand it through particular moments. Cock said we need to look at ERP as a community, a community of designers, developers, consultants and artefacts. But we might make different studies look at different parts of the community over time. And we concluded what was needed was also a variable reset of geometry where you both were able to pull out and view particular contextual factors but also zoom in and capture some of the intricacies of organisational process that had taken place.

So what’s the orientation to theory that we have here? And, it takes us back to the original question, how does analytical framework shape findings? It seems to me that we’re not arguing a single methodology is right but that social scientists need to argue the adequacy of their methodology according to the scope and purposes of their enquiry and we’re offering the concept of biographical artefact as a template for analysis and as a better template than some of the rather simplistic templates that have emerged in many social sciences such as the single science study, the company case study, etc.

Is the new economy radically different from the old? And here you have visions of radical transformation, on the other hand, you’ve got to start from somewhere. And there are going to be elements of continuity as well as destruction. So then the question is can we provide some kind of balanced theorisation of these two? Rather than for example go along with the can-do philosophy or the no-can-do philosophy, can we find ways of theorising tension and contradiction and look at how these processes of tension and contradiction can produce complex vectors and complex outcomes.

And in particular of course there’s a tendency in social sciences to talk about localist studies or generalist studies, well actually that’s not helpful. Can we look, for example, at the way that local and global interact, and can we look at ways that people try to manage the relationship between local and global, universal and specific? But I suppose the conclusion is and maybe the relevance of this work for you is the need to look at emergence – that when we find new phenomena there is typically some embryonic form in existing phenomena but we cannot presume as we might have from 1990 that the embryonic form actually represents the future. So we need tools which can both capture processes of continuity but also begin to put dimensions around ways in which outcomes may differ from the original templates.

It also has consequences for how we do research. One conclusion is research is a team task that we need to move away from the idea that these things can be addressed with single PhD students or single RAs, we need to have multiple studies and get the benefits of bringing together multiple studies, maybe sometimes purposefully bringing together people with comparative studies, but also opportunistically finding people working in the same area and drawing their work into a single analytical framework. And of course that raises challenges about how different kinds of research can talk to each other, quite complex challenges that I won’t be able to talk to. We would suggest then the challenge is, if you really want to understand the technology, you need to build a research community around that technology and give yourself time to understand it effectively. So we would see research as a journey and hopefully CREATe will be an interesting journey for us to take part in and we can contribute to more effective methodologies but I’ll stop there.

Prof. John Street: Thank you very much Robin that befits the winner of Daniel’s prize. That was a brilliant combination of insight into methodology, insights into organisational change and insights into what CREATe itself might think of itself as doing.

Prof. Martin Kretschmer: Okay, so for CREATe, as I see it, the big change has happened since digitisation and the internet as a mass-medium with the World Wide Web and browsers mid-1990s. So we’ve got a three or fear year horizon, do you have any suggestion how we investigate the last 20 years? So we can’t have ethnographic studies going back, so we have to use historical material to track what happened. So how do we do that and how do we configure that into a biographical study of your type?

RW: I think there are very valuable resources from the past and at our annual retreat we had Paola Quattrone in helping us to understand the emergence of accountancy looking at the Jesuit Church in the 18th century and providing some very interesting ideas about how the readability of a church map could be an indicator to how modern accounting systems were thought through. So I think there are lessons from history and also, it seems to me that one can track things in real time but one also needs to have tools for thinking over a slightly longer time. So when I started this work there was a period when ICT and broadcast technology converged. And what was interesting there was everybody was saying in the 1990s what’s going to be the big thing that’s going to drive TV and interactive systems into the home? And it was interesting, we were still looking for the big driver and then somebody said “oh, there’s a thing called the internet”, so the future was already there but nobody had actually identified the key driver, the champion product that would develop things forward.

So I think there are quite important lessons. In terms of how we go forwards, there are opportunities for large scale and small scale studies. One of the problems we face is we need to draw some position between the rhetorics of transformation and the rhetorics of stability. And I think we need tools that will allow us to unpick those and give dimensions to make some judgement about how we’re going to move from, how future reality is going to position itself between there. When we were looking at this in the 1990s we saw a whole load of strategic manoeuvres – Sony joined up with one of the studios. So we saw this period of trial and error learning as the big corporations tried to adapt to it. It seems to me that we’re in a very similar process of learning now. You know various firms in the digital music sector are experimenting, and we don’t know what the solution’s going to be. And I think we have to say well, we don’t know. It’s not that there is a known future that we haven’t yet grasped, that future is being worked out and I think there are lessons we can draw from the working out process for what the future is going to look like.

Prof. Ruth Toswe: Well, what you’re talking about is also very much an economic history of technological development on which there is a very rich literature, I mean people have done studies of, for example, learning by doing in the cotton industry in the 1760s which are doing exactly the same thing that you’re talking about. I suppose probably stone age people taught each other how to chip the flint or something. But there are these very long term sort of general approaches to this, by people like Paul David or Richard Lipsey for example. Richard Lipsey, with colleagues has coined this phrase ‘general purpose technology’ and has identified big stages in industrial revolution type terms, electrification, steam, printing, those sort of big writing, those big developments and how they played out, and of course digitisation is one of them. So that might be a kind of interesting literature to look at.

RW: I have to say that we find the economic history literature extremely interesting, and the social learning perspective is very explicitly informed by Arrow and Rosenberg’s account of learning by doing. So one of the things is maybe to both ‘sociologise’ and economics and to ‘economatise’ sociology if we’re really to understand these processes, and I think there are learning opportunities there that we would like to exploit.