13:51 11/06/2004 Mathemagenic: Mathemagenic
Mathemagenic
on personal productivity in knowledge-intensive environments, weblog research, knowledge management, PhD, serendipity and lack of work-life balance...
        

Mathemagenic

  25 October 2007


  Methodology chapter: Matching quality criteria and verification strategies

[From draft version of methodology chapter for my dissertation, slightly adapted for the web]

The table below presents an overview of verification strategies and their relevance to the quality criteria.

Verification strategy

Authenticity

Trustworthiness

Impact

Exposure
Being long enough in the right places talking to a variety of people to uncover important issues

Triangulation


 

Alternative interpretations are uncovered and represented Data source: rich picture, replicating findings across data sources
Study: replicating findings in other contexts with other methods
Researcher: decreasing subjectivity

Theorising     Teasing out the implicit Clear theoretical contribution by justifying research questions and positioning the results

Participants as co-researchers

 

Participants have a chance to make sure that important issues are uncovered and reported
Participants have an opportunity to shape research to have practical relevance
Gives power back

Reflexivity    

 


Uncovering and accounting for unexpected in the process of doing research Articulating subjectivity in writing Revealing dilemma's and uncertainties in research process engages readers
Ethics (no bad impact)'
Transparency Provides evidence of researcher's immersion in the field Allows alternative examination or replication of the study
Thick description "Transports" the reader to the field through quotes and contextualised descriptions Connection between data and conceptual categories is evident in the text
Readers have enough contextual information to decide how far results could be generalised
Engaging readers through storytelling
Purposeful confessional writing Provides a view onto researcher practices next of those studied Delineating between "objective" data and subjective interpretations Engaging readers through sharing personal experiences and uncertainties


  03 October 2007


  Methodology chapter: Quality verification strategies

[From draft version of methodology chapter for my dissertation, slightly adapted for the web]

In this section I propose several strategies to ensure quality of my research. I call them verification strategies, however similar to (Morse et al., 2002) I would like to emphasise that most of them should be used during research and not only for verifying quality of the outcomes.

Exposure

Prolonged exposure "in the field" ensures that a researcher had enough opportunities to encounter a variety of perspectives that would allow rich representation of the phenomenon under study. Yanow (2006) notes that prolonged exposure refers not only to the time, but to location as well. For me this means "being long enough in the right places talking to a variety of people to uncover important issues".

While doing research this means taking an effort to "map the territory" (Yanow, 2006) in a way that allows representing a variety of perspectives. For example, in case of my research this means talking to bloggers with diverse practices, including those in minority. In the study at Microsoft I strive for diversity by complementing snowball interview sampling with finding people "outside the network" by searching for "deviating weblogs", e.g. those written in another language or used in "unconventional" way (see 'Those that belong to the Emperor' for an example). For KM blogger study I use social network analysis based on linking between weblogs to define communal boundaries next to my (more subjective) personal knowledge of the actors and setting.

In reporting research exposure is reflected by describing the study settings, time and duration of being there; efforts made to define the field, to acquire representative data, to include multiple perspectives.

Triangulation

Triangulation refers to use of multiple sources and modes of evidence to make findings stronger by showing and agreement of independent measures or by exploring and explaining conflicting findings (Miles & Huberman, 1994; Schwartz-Shea, 2006). I employ several types of triangulation in my research:

  • Triangulating by study - studying blogging practices from three perspectives using a variety of methods.
  • Data triangulation - including in the analysis different types of data (e.g. text and statistics), data sources and data collection methods. In my research that means, for example, including non-elicited data (Pargman, 2000) from public sources (e.g. weblog text) next to the recorded interviews.
  • Involvement of multiple researchers in data collection and analysis in all studies except one.

Theorising

Next to being a starting point or target for a research, theory could be an instrument to make it stronger. In my research I use theory:

  • as a "sensitising device" (Klein & Myers, 1999) to inform research questions and conceptual categories, to tease out implicit nuances;
  • to explain and to position findings;
  • to "normalise the atypical" (Brower et al., 2000) by drawing parallels between my cases and conditions more familiar to the readers.

Participants as co-researchers

One of the strategies to ensure that research results represent the phenomena under study is informant feedback (Miles & Huberman, 1994; Schwartz-Shea, 2006), asking study participants to comment on the report. In my case I take it further, treating participants as co-researchers. This means not only asking for a feedback on finished reports, but also providing them opportunities to observe and to influence parts of the research process via my weblog.

Reflexivity

Reflexivity refers to the awareness and theorising about the role of self in all phases of the researcher process (Schwartz-Shea, 2006). I also like to think about it in terms of reflection-in-action and reflection-on-action (Schön, 1983). The first one is the source of what Yanow (2006) calls "improvisational character of interpretive research" - reflecting to address difficult to predict research circumstances (e.g. reacting to unanticipated turn in an interview). The second one refers to evaluation in retrospect, formulating the "lessons learnt" to guide actions that follow and to share in research report. In my research this strategy is used in several ways:

  • My weblog serves as a reflexive journal (Lincoln & Guba, 1985), which documents many of my reflexive moments, doubting and thinking aloud while doing research or evaluating it in retrospect. I also reference and quote those entries where relevant in the text.
  • I try to convey uncertainties, choices and lessons learnt while doing research in reporting the results by choosing not to present the research as clear path, but instead incorporating stories about reflexive moments in the text. I also discuss my own role, influences and mistakes while doing research. I write (in my academic publications, but also in my weblog) to think with the readers, not for them, as I believe that "the skill of writing is to create a context in which other people can think" (Edwin Schlossberg).

Transparency

At its extreme, making one’s research transparent means conducting an audit, where a detailed record of research processes and decisions as documented by a researcher is examined by an independent researcher to access research quality (Halpern, 1983; Akkerman et al., 2007). I prefer a broader definition of transparency as a set of practices to document research for an inspection by others (Schwartz-Shea, 2006). In my research I do it in the following ways:

  • I use my weblog not only to document my research, but also to carry out part of the research process. In this way it is available not only for auditing in retrospect, but also for a real-time feedback, which is more useful for a correction.
  • Since some of the data I use is publicly accessible, I provide readers of my research reports with references to it, so they have an opportunity to check my interpretations by examining it for themselves to the public data.
  • I provide descriptions of research processes and decisions made in research reports.

Thick description

Thick description (Geertz, 1973) refers to the style of reporting the research results aimed to "transporting the reader to the field" by providing detail-rich description of life of the research participants (Klein & Myers, 1999; Brower et al., 2000; Yanow, 2006; Schwartz-Shea, 2006). In the case of my research this means providing extensive quoting from weblogs and interviews, describing history and context of a particular setting, portraying the complexity and interrelations between different aspects of blogging practices. When quoting from weblogs I preserve linking in the text and provide direct link to the post, so those who read my work digitally can literally "transport themselves to the field" with one click.

Purposeful confessional writing

I use the term confessional writing to address different forms of bringing personal experiences in a publication (using personal pronouns, talking about my background or beliefs, including personal examples, etc.). Taken to an extreme, this could turn a research report into autobiography (Schultze, 1999; Duncan, 2004). This strategy aims to avoid that risk by making sure that confessional writing serves a research-related purpose. I do so by:

  • articulating the purpose of including confessional material (e.g. as an additional data source, to provide transparency of the research process, as a way to engage the readers);
  • drawing parallels between my personal experience and those of other bloggers (in my studies or as reported in a literature);
  • interlacing self-revealing writing with more traditional forms of academic writing (e.g. as a layered account Ronai, 1995);
  • separating between personal data available for others to examine (e.g. my weblog posts) and personal interpretations of it.

References


  02 October 2007


  Methodology chapter: Quality criteria

[From draft version of methodology chapter for my dissertation, slightly adapted for the web]

In my reading of methodological literature I often felt lost, so I was very happy to find the work of Peregrine Schwartz-Shea on quality evaluation criteria for interpretive research (Schwartz-Shea, 2006). In her discussion on quality criteria as suggested by different authors she not only discusses how multiple terms and categories are used across and within different research paradigms without making parallel terms explicit, but also draws some of the missing parallels. I used her table (Schwartz-Shea, 2006: 94) that matches terms used in classic interpretive research texts (Lincoln & Guba, 1985; Miles & Huberman, 1994) to positivist research to suggest terms that I would like to use for my own research: authenticity, trustworthiness and impact (column 5 is added by me).

Criterion Terms used in methodological positivism Lincoln and Guba (1985): parallel terms Miles and Huberman (1994): parallel and new terms My simplified terms
Truth value Internal validity Credibility Internal validity / credibility / authenticity Authenticity
Applicability External validity / generalizability Transferability External validity / transferability / fittingness Trustworthiness
Consistency Reliability Dependability Reliability / dependability / auditability Trustworthiness
Neutrality Objectivity Confirmability Objectivity / confirmability Trustworthiness



Utilization / application / action Impact

I propose a simplified list of terms as a way to resolve the differences between terminology used in a variety of publications that I consulted. For example, a list of criteria suggested by Richardson (2000) to evaluate autoethnography provides an example of an alternative terminology that does not easily matches one of the classic texts, but addresses well specific issues for this type of research:

1. Substantive contribution: Does this piece contribute to our understanding of social life? Does the writer demonstrates a deeply grounded (if embedded) human world understanding and perspective? How has this perspective informed the construction of the text?

2. Aesthetic merit: Does this piece succeed aesthetically? Does the use of creative analytical practices open up the text, invite interpretive responses? Is the text artistically shaped, satisfying, complex, and not boring?

3. Reflexivity: How did the author came to write this text? How was the information gathered? Ethical issues? How has the author's subjectivity been both a producer and a product of this text? Is there an adequate self-awareness and self-exposure for the reader to make judgements about the point of view? Do authors hold themselves accountable to the standards of knowing and telling of the people they have studies?

4. Impact: Does this affect me? Emotionally? Intellectually? Generate new questions? Move me to write? Move me to try new research practices? Move me to actions?

5. Lived experience: Does this text embody a fleshed out, embodies sense of lived-experience? Does it seem "true" - a credible account of a cultural, social, individual, or communal sense of the "real"? (Richardson, 2000)

In proposing my own simplified criteria I tried to integrate those from publications that discussed evaluation criteria and/or corresponding quality verification strategies in a way applicable to my work. I define proposed criteria in the previous table by describing what is judged by each of them and how this could be translated into specific questions to ask about the research.

Evaluation criteria What is judged Specific questions (adapted from Miles & Huberman, 1994; Brower et al., 2000)
Authenticity Quality of representing real-life phenomenon Do the findings of the study make sense? Are they credible to the participants and readers of the report? Do the results provide an authentic portrait of phenomenon studied? Has the author been there in the field?
Trustworthiness Quality of the research

Do other researchers have enough information to judge:

  • Research process and methods used
  • Connections between data, interpretations and conclusions
  • Biases and influences of the researcher and measures to address those
  • Opportunities to transfer the results to other contexts, to generalise
  • Theoretical contribution
  • Impact Engaging the reader Relevance to practice (OR Changing the world, one reader at a time ;-) Does the text create unique impressions about the subject for readers? Does it stimulate them to re-examine taken for granted assumptions in their own worldviews? Does it affect them emotionally? Does the study provides insights relevant to the practice? Are there implications for actions?

    Comparing my research approach to those done in more traditional ways I expect most challenges in defending its trustworthiness, since I report explicitly about my personal involvement and certain degrees of subjectivity in doing it. A good example of what I could expect is presented by Holt (2003), who analyses the comments to his autoethnographic paper by journal reviewers. He identifies two groups of issues related to acceptance of his work: the use of self as the only data source and the use of verification strategies in autoethnographic studies. The first is applicable fully to only one of my studies, while for the dissertation as a whole I use my own case to complement other cases and include autoethnographic elements to add transparency to the research process. I address the second concern, difficulty of using common verification strategies to judge this type of research, in the following section by describing quality verification strategies that fit my research.

    However, next to the efforts to ensure and to defend trustworthiness of this research, I am always prepared to defend those choices that help me to provide a better overview of my topic and to make sure the results make a difference, even if they make my work weaker in the eyes of some researchers. Fortunately, weblogs supply not only challenges of studying them, but also alternative ways to provide transparency of the research and accountability of the researcher.

    References


      01 October 2007


      Methodology chapter: Participation

    [From draft version of methodology chapter for my dissertation, slightly adapted for the web]

    Although being a detached outsider is often considered to be the best position for a researcher, soon after starting my research I found myself at the opposite end, doing research through active participation (Mortensen, 2003). When I submitted my first paper on knowledge work and blogging (Efimova, 2004b which eventually became my background chapter) I realised that my insights came as much from analysing replies to the questionnaire as from my experiences of being part of the KM blogger community: writing my own weblog, interacting with other bloggers and reflecting on those experiences. I started to look for methodologies that would allow accounting for personal experiences of the researcher and soon found that ethnography addressed many of my concerns. (Yanow (2006) suggests that '"ethnography" refers both to a set of research tools and to a mode of writing'. I would position my research as "ethnographically informed" as I use some of conceptual distinctions and research instruments of ethnography, while only partially adopting ethnographic writing mode.)

    Ethnography, originated as a method in sociology and anthropology, is increasingly used in research of technology-mediated practices. It includes studying particular culture by learning to live a life of its members (Hammersley & Atkinson, 1994). Next to informal interviewing, participant observation is a central way to generate ethnographic data:

    Anthropologists are trained to use a research method known as 'participant observation', which is essentially means participating in the life and culture of the people one is studying, to gain a true insider's perspective on their customs and behaviour, while simultaneously observing them as a detached, objective scientist. Well, that's the theory. In practice it often feels rather like that children's game where you try to pat your head and rub your tummy at the same time. (Fox, 2005: 3)

    Although weblogs, as many other online tools, provide an opportunity to observe unobtrusively by lurking and reading, passive observation was not a choice for me, since my beginning of my PhD coincided with my first blogging experiences. In my research I played two roles: a knowledge worker who blogs about her work and a researcher who studies knowledge worker blogs. The following quote, illustrates one of my first attempts to describe the effects of combining those two roles:

    I sketch an outline - main things that I want to say - about personal experience of blogging as a starting point that shapes my research questions, about drive of find out why others do not believe my blogging stories (they couldn't be fake even if there is evidence that they are not true for an average weblog - I can't throw away my own experience!), about my learnings from stories others share in reaction to my blog posts, about writing as participation, data collection, feedback on emergent interpretations and final publication (all melted into one), about hard choices of being blogger and researcher at the same time, about all things that make my research so fun and so insecure when I think how to frame it to be a "proper scientist"...

    Participating in the life and culture of my target group by blogging helped my research in a several ways as discussed below.

    Learning about blogging culture

    My personal blogging practices became an important source of learning about blogging, especially in respect to understanding the aspects of it that couldn’t be observed by reading weblogs. Those aspects include, for example, the effort that goes into fine-tuning a weblog tool to fit my needs, the surprises of receiving feedback on pieces that I never expected to be interesting to others, or the change of daily morning routines as a result of blogging. Next to substituting the state of the art literature study, personal blogging experiences helped me to follow weak signals of interesting issues that I might overlook otherwise.

    Reading weblogs, as "another blogger" and not only with the coding purposes in mind, was an important part of my personal blogging experience. It helped me to get to know people behind weblogs: "absorbing details of others' lives from their weblogs, sense of connectedness and somewhat intimate knowledge about them" turned into "interviews that could touch themes and go to the depths not possible otherwise" [quoting this post].

    I also learnt a lot by comparing my own blogging practices to those of others in my own community and outside it. I has been frequently surprised that even bloggers that I considered to be very close to me in blogging attitudes would make different choices from my own. From another side, trying to describe to sceptics some practices shared in my blogging community helped to shape my research questions and to position my own work in respect to other researchers.

    Blogger identity, relations and access

    Being a blogger gave me an identity between other bloggers and helped to develop trusted relations with others. In my own blogging community I didn't need introductions and could easily contact others for information or an advice, by email, instant messaging or phone. When I travelled, I stayed in the houses of some of my blogging friends and was able to peak into their private lives and have casual conversations about blogging on topics that would likely to escape more formal interviewing.

    Having a weblog also served me when approaching study participants outside of my own network. Arranging for the study at Microsoft was mediated via my weblog. Also, when I emailed introductions for bloggers I didn't know and asked for interviewing opportunity, I would include link to my own weblog next to other credentials. It’s difficult to measure how much closed doors it have opened, but I also felt that it provided more equality as participants of my research could check my background as easily as I could check theirs (see Mortensen & Walker, 2002, for similar example; Beaulieu, 2004, for a methodological discussion of it).

    Others as co-researchers

    I blogged on the progress of my research and my participants could easily follow those posts, creating influences and feedback loops that researchers learn to avoid to escape "contaminating their data". I have learnt to embrace and use them in my research, taking an advice from Hammersley and Atkins:

    Once we abandon the idea that the social character of research can be standardized out or avoided by becoming a ‘fly on the wall’ or a ‘full participant’, the role of the researcher as active participant in the research process becomes clear. He or she is the research instrument par excellence. The fact the behaviours and attitudes are often not stable across contexts and that the researcher may influence the context becomes central to the analysis. (Hammersley & Atkinson, 1994: 19)

    I tried to vary the degree of closeness to my informants between and within studies. For example, as I was aware of the feedback loops in my study of KM bloggers, I had an extended email discussion with Jonathan Grudin, my supervisor at Microsoft, on possible choices in respect to blogging about the study progress while being in the company. When making choices for interviewees I made an effort to talk to people more distant from myself (for example, based on knowledge that they don't read my weblog).

    Many of bloggers who participated in my studies could be described as lead users, those who shape emerging technology to address their needs (von Hippel, 1986). Often their own professional interests aligned well with my research quest to discover how weblogs could support knowledge work. Blogging about progress of my research helped to involve them as co-researchers. I had multiple occasions of feedback from fellow bloggers on shaping study methods, data collection instruments, emergent interpretations or drafts of my papers.

    Colliding worlds

    As my blogging served me in both roles, a researcher and a blogger, it was not always easy to separate them and to make choices in case of role conflicts. This is an example of one of this cases as documented in my weblog (it refers to the weblog conversation study done in my own blogging community):

    [Extended quote from Hard choices: researcher vs. blogger?]

    However, role conflicts appeared also where I did not expect them. Studying blogging practices of people outside of my own circle brought me to making choices between insider participation and outsider distance as well:

    [Extended quote from Studying weblogs at Microsoft: connecting the dots]

    In addition to the role conflicts, being a blogger made it difficult to draw a line between fieldwork and homework, participant observation and writing up, creating a risk of turning my research into an on-going endeavour (Beaulieu, 2004). In addressing this problem, publication deadlines served me well: as a deadline would approach I would not have much time to read other weblogs and to blog myself, thus creating natural withdrawal moment that would serve as a boundary between the field and home.

    Finally, the feedback loops resulting from blogging changed the way I would write up my research for a publication. Although sometimes study participants are expected to read final research report, they are not the intended audience for it. In my case thinking of the study participants as readers of the finished work was the choice I could not escape knowing how little effort it would take for them to access my published work.

    References





    © Copyright 2002-2007 Lilia Efimova Click here to send an email to the editor of this weblog.

    This weblog is my learning diary. Sometimes I write about things related to my work, but the views expressed here are personal and do not necessarily reflect the views of my employer.

    Last update: 15/11/2007; 01:21:07.