Stuck in cross-case analysis

I've got to rewrite chapters 2 and 6 - well edit them. They are the literature review and the cross-case analysis.

The literature review edit is fairly straight forward and supervisor has helped by reading it thoroughly and giving me over sixty comments in writing. Is that critical or helpful? I think it's helpful supervision because I know the problem and what I have to do to solve it.

The cross-case analysis on the other hand has been difficult for months, and despite listening to my supervisors for an hour and listening again on the recording, I don't know how to tackle it. Until I have the cross-case analysis in my head, I cannot write it down, so in the meantime supervisors can't comment constructively - just say that they don't understand the abstractness of my writing in chapter 6, and beat me up.

Supervisor #2 suggested I was filtering out stuff and to brainstorm, but brainstorming is a group technique, not a really a one-PhD woman job. However he did also suggest I use a technique called EIAG which stands for

• Experience
• Identify
• Analyse
• Generalise

The idea is that I know/see/feel and find the experiences from my research and then identify them. The next two steps seem easy enough but I'm still stuck on the experience bit. For instance, at supervision, I pointed out the cycles and feedbacks on the engagement model that I've developed, but I haven't actually written these loops down anywhere and explained them, because I filter them out and yet they are important - because lose the loops and you lose engagement and also lose the evidence that engagement is continual. It was filtering out this idea from writing yet bringing it up in talking that made my supervisor think of EIAG.

Now what do I do? Stuck again in cross-case analysis

I'll go and edit the literature review.

Getting somewhere

Post supervision:

1. Supervisors are happy (ish)
2. my framework could make a significant contribution
3. Supervisor # 1 is (again) looking for a case study through work contacts
4. As usual both supervisors are available together, even if they have to come in especially, which is what they might have done today - I appreciate that effort.
   
5. They gave me direction to work out how to finish my literature review with this proposed framework for analysis of the case studies, then "This suggests the framework is useful to test..."

The happiish bit is because of the paucity of rich case studies, hence supervisor #1 wants to widen outside the public sector. Without the data, I can't finish on time with a good thesis. The options are:

1. Finish but I may find the examiner passes the thesis only with major revisions or
   
2. Wait until case study materialises so I finish late
   

Both scenarios mean an unfunded delay.

For a rich case study, I need an IT project where I can speak to several people: users, project manager, contractors, developers and consultant. I have one otherwise good case study where I couldn't speak to the consultant; an examiner might argue that without the consultant's perspective I can't say enough about engagement.

So it's back to the word processor to work out how to write my arguments. And the drawing board to think who I can approach for case studies, in or out of the public sector.

Production

I've produced:

1. a rewritten paper on my second case study to my supervisors and
   
2. a review of a really useful new book by Sturdy et al about consultants

And I've marked a couple of early computing assignments.

Next task must be to work on another case study. And to keep looking for access to IT projects in the public sector.

Reviewing a conceptual paper

How do you review a conceptual paper? Our training is good on reviewing papers about research. For example, here's the first part of an assignment question from B852:

Critically review some journal article:
(a) Explain the purpose of the research, any research questions or propositions, and how it relates to existing theory and/or practice.
(b) Describe and discuss the appropriateness of the research methodology employed, highlighting any particular strengths or limitations.
(c) Summarise and critically assess the findings of the research and any conclusions drawn from them.

There are an awful lot of conceptual papers, and not so many research papers but I've got hold of an interesting and relevant paper that's about to be published. I'd like to share it with my supervisors, so need to prepare something in mind to tell them. If I write it, they'll read it because they're good like that. So how do I structure a review of a conceptual paper?

Systematic literature review again

When researching, you need to know what you don't know. "Ignorance is undervalued" was the starting point of a session on systematically reviewing the literature. And:

"Research is the process of going up alleys to see if they're blind"(Marston Bates)

Colin Pilbeam at the BAM conference set out a framework for a systematic review, explaining that

1. a critical review comes before
   
2. a scoping study that leads into
   
3. the systematic review.

I hadn't realised that sequence. They mentioned Wallace and Wray - a quick search gives me this recent overview from an enthusiastic student who recommends their framework too and gives a link to downloading questions for a critical review.

The overheads that Colin Pilbeam and Rob Briner displayed gave me a better sense of being systematic, and encourage me to keep records of for example:

• information sources
• citation databases
• searches I've done and the search strings

and to tabulate my findings. I'll keep the records; they'll be evidence.

Systematic review

The Cranfield Master of Research dissertations are not empirical research but systematic reviews of the literature, and as such are great examples of how to go about a systematic review. Consequently I'm low that I didn't tackle my literature review this way at the start of this academic year.

Now though I am imagining all the things that won't come together, the important literature I won't find and how to write it up coherently.

Work plan for research project

It's a bit hard to plan two years ahead, but since I don't need to worry about coordinating financial and people resources, it's not a great problem. So here's first draft starting July 2008, with thanks to my student colleagues who shared their approaches.

Literature review
Month Activity
1- 27 Continue to read and review literature on public sector, consultants, clients and relevant method such as social network theory

Skills audit
Month Activity
1 - 22 Attend courses, seminars & conferences as appropriate
Rewrite my OUBS web page when it’s out of date
2-4 Learn any specific analysis skills such as the maths needed for social network analysis
6-10 Learn necessary software such as NVivo and software for social network analysis
1-28 Adjust plans as life gets in the way
9-10 Write a paper on the pilot case study results

Methodology
Month Activity
1-4 Review and decide methodology & methods
Choose options for analysis
4 Write something about the design, method and methods
4 Get ethics approval (skill B2, B3

Data collection
Month Activity
1-4 Identify 5 or 6 case studies, one of which can be a pilot study. Five might suffice, but six will cover contingencies such as lack of access
5 Interview pilot study
3-15 Identify & interview about 6 interviewees per case study, these being the equivalent of chief exec or primary client, senior responsible owner, project manager, a consultant, a user, relevant politician. .
5-15 From interview data and documentation identify potential relationships in networks and send questionnaires to confirm relationships. There can be hundreds on a project, so think about how many to choose, how to choose them and why
6-15 Devise and sent questionnaires on relationships to identified participants
6-20 Transcribe interview data and input questionnaire data

Analysis
Month Activity
7-20 Analyse earlier cases while collecting data for later cases
Meet supervisors regularly

Writing
Month Activity
15-23 Write drafts for supervisors
Continue to meet supervisors regularly
26 Proof read dissertation
28 Submit

Probation looms

By the end of June I must submit a project report that includes:

• a viable research question
• a critical literature review which situates the proposed research
• a research proposal including an outline of proposed method(s) and a critical justification for them
• a work plan for the project.

And then I get a viva.

• I'm beginning to get the research question, something about engaging with consultants in order to get an effective contribution to a project.
• I've read a fair deal around the topic: consultants, clients, public sector including government literature, projects, IT projects, New Public Management, institutionalism, accountability, value engineering and risk. And I've written up on what I've read so my supervisors have some idea of what I've read and absorbed.
• Now I have to put together the research proposal, which I guess is to take case studies from public sector organisations and interview clients. It would be great if I could shadow a senior responsible officer. The critical justification for the method is in the theoretical framework.
• The work plan is not going to get much time from me. I wonder what they mean by it.
  

Supervision

Next task is to review literature on the management of consultants, which is quite difficult to find, possibly because

1. there is little such literature
2. it's hidden because the research terms 'management' AND 'consultant' could mean how consultants manage, or how they consult for managers rather than how managers select, use and manage them.

I also need to turn up at the next supervision meeting with around 10 possible PhD skills to pick three from and concentrate on.

Thirdly, I need to work out what, if anything I can submit to the student stream at the BAM conference in September.

Review of accountability

I'm to review the literature on accountability. I read literature on this for my MRes and attempted to make sense of the construct of accountability, writing and rewriting for months. If I'm to review the literature I suppose I go back to what I had first time round, and reread it. And add to it.

Searches
I've been search EBSCO for 'accountability' with other words:

• public accountability
• public accountability AND logic
• logic of accountability
• accountabilities AND logics
• public accountability AND review
• public accountability AND review AND literature- no recent review of the literature on accountability
• Accountability review NOT accounting
• participatory accountability - results seem to be about poverty & third world
• participatory theory AND accountability
• accountability AND risk-taking
• ethical accountability - the results seem to relate to adverts

Then I followed up some systems writing by Flood [1] and looked for

• public accountability AND structure - best is Pugh et al from years ago
  
• public accountability AND processes
• public accountability AND meaning - hits seem a bit old
  
• public accountability AND power AND knowledge - no hits here
• public accountability AND power
  

---

[1] Flood, Robert, L., Rethinking the Fifth Discipline: Learning Within the Unknowable (Paperback)at http://www.amazon.co.uk/Rethinking-Fifth-Discipline-Learning-Unknowable/dp/0415185300/ref=sr_1_1?ie=UTF8&s=books&qid=1196353306&sr=8-1

DTW Rigour and reviewing

How are you going to demonstrate rigour in your research?
In the doctoral training workshop, our small group suggested (after a discussion about the meaning and spelling of 'rigorous'):

• an obedience to rules
• following instructions rigoursly
• consistency
• validity
• reliability

The class suggestions were:
Before data collection: Address own bias, Be consistent with the research question, Be objective, Justify the methodology, Have awareness of the literature in the field

After data collection Triangulate, Be repeatable, Repeat experiments to make sure Use methods consistently, Compare with other’s work, Analyse from a different perspective, Know limitations, Find other explanations

Always Following the training rules, Be convincing, Demonstrate external validation

The answer taken from Research Skills for Policy and Development [1]:

1. To show that you have enough evidence to justify your conclusions
2. That evidence has been obtained properly and that contrary evidence has been sought, but either not found or found to be relatively unconvincing.

See slides that they'll post on the web site later.

Ways of doing research

Maybe have a section in the PhD dissertation about the way of researching, e.g.:

• observing,
• two-way interactions e.g. semi-structured interviews
• participation.

See rules for rigourous research in Glassich, Huber & Maeroff, 1997 [2]

Some questions that a reviewer might be required to ask of a paper:

1. whether there's evidence to support the claims
2. whether the claims are relevant and topical
3. accuracy of the numbers or quotations
4. ethics of publication
5. credible references
6. relevance of the topic (journal/field/fashion)
7. originality might be desirable but not required
8. limitations
9. clarity of ideas
10. findings justified
11. structure
12. abstract

---

[1] Research Skills for Policy and Development: How to Find Out Fast (Published in association with The Open University) by Alan Thomas and Giles Mohan (Paperback - 16 May 2007) at http://amazon.co.uk/s/ref=nb_ss_w_h_?url=search-alias%3Daps&field-keywords=Research+Skills+for+Policy+and+Development
[2] Scholarship Assessed: Evaluation of the Professoriate (Special Report) by Charles E. Glassick, Mary Taylor Huber, and Gene I. Maeroff (Paperback - 15 Aug 1997)

Supervision - literature review

All three of us managed to meet for about half an hour yesterday for the first time since I started on the PG programme.

Agenda included:

• PhD skills audit
• my scribbles
• ethics committee feedback on MRes, a summary and an issue that arose
  

PhD skills audit

My scribbles

These scribbles connected my ideas from reading on organisations and accountability. They linked the intangible recommendations of consultants to the invisibility of recommendations that were not taken up and the undiscussability of decisions to implement recommendations or not. These ideas built on readings from Mullins, Thornton and Argyris with reference to concepts of institutional logic and power.

Criticism is undiscussable, or consultants come up with recommendations that are ignored. I have stuff about how aspects of a consultancy and issues that arise are undiscussable so how does accountability work there, so single & double loop processes are being challenged. there are issue in my research about defining the context.

• What does it mean to use consultants and
  
• What might accountability mean in those contexts?
  
• What happens if we've hired consultants to change the way we think? which is what institutional logic & double loop learning is about, throwing up in the air the criteria used. There might be conflicting logics in play.
• what is legitimate, what's discussable?
  

There are ways of setting up and legitimising some accountability relationships and not others. You can have a culture that allows some and not others. The institutional context & culture shapes the micro-level of interaction between groups and organisations. The mental models take place within an institutional context. Supervisors suggest that where there are challenges to Argyris' double loop learning, then issues of context matter. There are systems of meaning in a context of accountability at a micro level with mental models. They recommend that I explore accountability more, e.g. what happens when organisational templates change? You import an institution to make an organisation out of. The organisation is the enactment of the model. you What does it look like when you have different institutional loyalties in play? what's the process of transition like? Thornton looked at something like this- a tradition called institutional theory.
Follow up Hinings & Royston Greenwood, 1998, Mayo & Rowan, DiMaggio & Powell, Michael Lounsbury.

Concepts in the accountability literature that I had read:

• institutional logic
• performance measurement - is an institutional kind of logic, imposed by audit & inspection regimes
• participation - pressure on councils to consult, part of the institutional environment
  

Literature review

This is to be on:

1. accountability
2. new institutionalism

My task before Christmas is to write something about accountability giving a sense of where the literature on accountability now is and then to move on to anything promising about institutionalism. Also I'm going to look at one or two theses.

DTW literature review

See web site at http://www.sagepub.co.uk/potter/and and chapter 7 of Doing Postgraduate Research

Why do a literature review?

• to know what's been said
  
• to find gaps
  
• to clarify the research question
  
• to 'locate' research in context
  
• to find who is doing what
  
• to facility your learning about the process about how people do research.
  

Compare Potter's slide on the purpose of a literature review, which includes using the literature review to choose a research method or approach.
We discussed Woodley's 1980's review for the literature on the success rate of mature students. (I hope his more recent work indicates that mature students are successful.) He writes a story of the emergence of information on the topic, not a list of the papers that he's read.

Sources of information that we thought of include:

• journals
• books
• grey literature: organisational reports, blogs
• technical reports
• web
• people

See Doing Post Graduate Research Chapter 7.3