[From draft version of methodology chapter for my dissertation, slightly adapted for the web]
In my reading of methodological literature I often felt lost, so I was very happy to find the work of Peregrine Schwartz-Shea on quality evaluation criteria for interpretive research (Schwartz-Shea, 2006). In her discussion on quality criteria as suggested by different authors she not only discusses how multiple terms and categories are used across and within different research paradigms without making parallel terms explicit, but also draws some of the missing parallels. I used her table (Schwartz-Shea, 2006: 94) that matches terms used in classic interpretive research texts (Lincoln & Guba, 1985; Miles & Huberman, 1994) to positivist research to suggest terms that I would like to use for my own research: authenticity, trustworthiness and impact (column 5 is added by me).
| Criterion | Terms used in methodological positivism | Lincoln and Guba (1985): parallel terms | Miles and Huberman (1994): parallel and new terms | My simplified terms |
| Truth value | Internal validity | Credibility | Internal validity / credibility / authenticity | Authenticity |
| Applicability | External validity / generalizability | Transferability | External validity / transferability / fittingness | Trustworthiness |
| Consistency | Reliability | Dependability | Reliability / dependability / auditability | Trustworthiness |
| Neutrality | Objectivity | Confirmability | Objectivity / confirmability | Trustworthiness |
| Utilization / application / action | Impact |
I propose a simplified list of terms as a way to resolve the differences between terminology used in a variety of publications that I consulted. For example, a list of criteria suggested by Richardson (2000) to evaluate autoethnography provides an example of an alternative terminology that does not easily matches one of the classic texts, but addresses well specific issues for this type of research:
1. Substantive contribution: Does this piece contribute to our understanding of social life? Does the writer demonstrates a deeply grounded (if embedded) human world understanding and perspective? How has this perspective informed the construction of the text?
2. Aesthetic merit: Does this piece succeed aesthetically? Does the use of creative analytical practices open up the text, invite interpretive responses? Is the text artistically shaped, satisfying, complex, and not boring?
3. Reflexivity: How did the author came to write this text? How was the information gathered? Ethical issues? How has the author’s subjectivity been both a producer and a product of this text? Is there an adequate self-awareness and self-exposure for the reader to make judgements about the point of view? Do authors hold themselves accountable to the standards of knowing and telling of the people they have studies?
4. Impact: Does this affect me? Emotionally? Intellectually? Generate new questions? Move me to write? Move me to try new research practices? Move me to actions?
5. Lived experience: Does this text embody a fleshed out, embodies sense of lived-experience? Does it seem “true” – a credible account of a cultural, social, individual, or communal sense of the “real”? (Richardson, 2000)
In proposing my own simplified criteria I tried to integrate those from publications that discussed evaluation criteria and/or corresponding quality verification strategies in a way applicable to my work. I define proposed criteria in the previous table by describing what is judged by each of them and how this could be translated into specific questions to ask about the research.
| Evaluation criteria | What is judged | Specific questions (adapted from Miles & Huberman, 1994; Brower et al., 2000) |
| Authenticity | Quality of representing real-life phenomenon | Do the findings of the study make sense? Are they credible to the participants and readers of the report? Do the results provide an authentic portrait of phenomenon studied? Has the author been there in the field? |
| Trustworthiness | Quality of the research | Do other researchers have enough information to judge:
|
| Impact | Engaging the reader Relevance to practice (OR Changing the world, one reader at a time ;-) | Does the text create unique impressions about the subject for readers? Does it stimulate them to re-examine taken for granted assumptions in their own worldviews? Does it affect them emotionally? Does the study provides insights relevant to the practice? Are there implications for actions? |
Comparing my research approach to those done in more traditional ways I expect most challenges in defending its trustworthiness, since I report explicitly about my personal involvement and certain degrees of subjectivity in doing it. A good example of what I could expect is presented by Holt (2003), who analyses the comments to his autoethnographic paper by journal reviewers. He identifies two groups of issues related to acceptance of his work: the use of self as the only data source and the use of verification strategies in autoethnographic studies. The first is applicable fully to only one of my studies, while for the dissertation as a whole I use my own case to complement other cases and include autoethnographic elements to add transparency to the research process. I address the second concern, difficulty of using common verification strategies to judge this type of research, in the following section by describing quality verification strategies that fit my research.
However, next to the efforts to ensure and to defend trustworthiness of this research, I am always prepared to defend those choices that help me to provide a better overview of my topic and to make sure the results make a difference, even if they make my work weaker in the eyes of some researchers. Fortunately, weblogs supply not only challenges of studying them, but also alternative ways to provide transparency of the research and accountability of the researcher.
Tags: methodologyArchived version of this entry is available at http://blog.mathemagenic.com/2007/10/02.html#a1948; comments are here.
Related posts