Updated: 13/11/2007; 21:54:44.


on personal productivity in knowledge-intensive environments, weblog research, knowledge management, PhD, serendipity and lack of work-life balance...
If you search for mathemagenic that has nothing to do with weblogs try this

Earlier | Home | Later

  02 October 2007

  Methodology chapter: Quality criteria

[From draft version of methodology chapter for my dissertation, slightly adapted for the web]

In my reading of methodological literature I often felt lost, so I was very happy to find the work of Peregrine Schwartz-Shea on quality evaluation criteria for interpretive research (Schwartz-Shea, 2006). In her discussion on quality criteria as suggested by different authors she not only discusses how multiple terms and categories are used across and within different research paradigms without making parallel terms explicit, but also draws some of the missing parallels. I used her table (Schwartz-Shea, 2006: 94) that matches terms used in classic interpretive research texts (Lincoln & Guba, 1985; Miles & Huberman, 1994) to positivist research to suggest terms that I would like to use for my own research: authenticity, trustworthiness and impact (column 5 is added by me).

Criterion Terms used in methodological positivism Lincoln and Guba (1985): parallel terms Miles and Huberman (1994): parallel and new terms My simplified terms
Truth value Internal validity Credibility Internal validity / credibility / authenticity Authenticity
Applicability External validity / generalizability Transferability External validity / transferability / fittingness Trustworthiness
Consistency Reliability Dependability Reliability / dependability / auditability Trustworthiness
Neutrality Objectivity Confirmability Objectivity / confirmability Trustworthiness

Utilization / application / action Impact

I propose a simplified list of terms as a way to resolve the differences between terminology used in a variety of publications that I consulted. For example, a list of criteria suggested by Richardson (2000) to evaluate autoethnography provides an example of an alternative terminology that does not easily matches one of the classic texts, but addresses well specific issues for this type of research:

1. Substantive contribution: Does this piece contribute to our understanding of social life? Does the writer demonstrates a deeply grounded (if embedded) human world understanding and perspective? How has this perspective informed the construction of the text?

2. Aesthetic merit: Does this piece succeed aesthetically? Does the use of creative analytical practices open up the text, invite interpretive responses? Is the text artistically shaped, satisfying, complex, and not boring?

3. Reflexivity: How did the author came to write this text? How was the information gathered? Ethical issues? How has the author's subjectivity been both a producer and a product of this text? Is there an adequate self-awareness and self-exposure for the reader to make judgements about the point of view? Do authors hold themselves accountable to the standards of knowing and telling of the people they have studies?

4. Impact: Does this affect me? Emotionally? Intellectually? Generate new questions? Move me to write? Move me to try new research practices? Move me to actions?

5. Lived experience: Does this text embody a fleshed out, embodies sense of lived-experience? Does it seem "true" - a credible account of a cultural, social, individual, or communal sense of the "real"? (Richardson, 2000)

In proposing my own simplified criteria I tried to integrate those from publications that discussed evaluation criteria and/or corresponding quality verification strategies in a way applicable to my work. I define proposed criteria in the previous table by describing what is judged by each of them and how this could be translated into specific questions to ask about the research.

Evaluation criteria What is judged Specific questions (adapted from Miles & Huberman, 1994; Brower et al., 2000)
Authenticity Quality of representing real-life phenomenon Do the findings of the study make sense? Are they credible to the participants and readers of the report? Do the results provide an authentic portrait of phenomenon studied? Has the author been there in the field?
Trustworthiness Quality of the research

Do other researchers have enough information to judge:

  • Research process and methods used
  • Connections between data, interpretations and conclusions
  • Biases and influences of the researcher and measures to address those
  • Opportunities to transfer the results to other contexts, to generalise
  • Theoretical contribution
  • Impact Engaging the reader Relevance to practice (OR Changing the world, one reader at a time ;-) Does the text create unique impressions about the subject for readers? Does it stimulate them to re-examine taken for granted assumptions in their own worldviews? Does it affect them emotionally? Does the study provides insights relevant to the practice? Are there implications for actions?

    Comparing my research approach to those done in more traditional ways I expect most challenges in defending its trustworthiness, since I report explicitly about my personal involvement and certain degrees of subjectivity in doing it. A good example of what I could expect is presented by Holt (2003), who analyses the comments to his autoethnographic paper by journal reviewers. He identifies two groups of issues related to acceptance of his work: the use of self as the only data source and the use of verification strategies in autoethnographic studies. The first is applicable fully to only one of my studies, while for the dissertation as a whole I use my own case to complement other cases and include autoethnographic elements to add transparency to the research process. I address the second concern, difficulty of using common verification strategies to judge this type of research, in the following section by describing quality verification strategies that fit my research.

    However, next to the efforts to ensure and to defend trustworthiness of this research, I am always prepared to defend those choices that help me to provide a better overview of my topic and to make sure the results make a difference, even if they make my work weaker in the eyes of some researchers. Fortunately, weblogs supply not only challenges of studying them, but also alternative ways to provide transparency of the research and accountability of the researcher.


    Earlier | Home | Later

    © Copyright 2002-2007 Lilia Efimova.

    This weblog is my learning diary. Sometimes I write about things related to my work, but the views expressed here are personal and do not necessarily reflect the views of my employer.

    October 2007
    Sun Mon Tue Wed Thu Fri Sat
      1 2 3 4 5 6
    7 8 9 10 11 12 13
    14 15 16 17 18 19 20
    21 22 23 24 25 26 27
    28 29 30 31      
    Sep   Nov

    Edublog award 2004 as Best Research Based Blog. Click for more details...

    Click to see the XML version of this web page. Click here to send an email to the editor of this weblog. Please, make sure that I recognise your name or you have a nice autorisation message - I tend to decline calls from people I don't know ;)

    Locations of visitors to this page Technorati Profile