practical phd

After doing most of my qualitative data analysis solo during my time as a graduate student, I have spent the last 7 years readapting to coding in teams.  Unlike many of my fellow sociologists, I began as a qualitative researcher in a team-based setting.  In evaluation research, I worked with as few as 2 and as many as 5 other researchers with varying degrees of prior experience to code qualitative interviews, focus groups, observation notes, and supporting documents.  The experience of being on those teams and my more recent experience leading teams through the process have brought to light key insights about working with teams for qualitative coding.

Conducting qualitative coding with teams is challenging because of the subjective or interpretative nature of a lot of coding.  Most projects do include codes that are straight forward and explicit, capturing direct and explicit mentions of a topic, person, organization, program, idea, or feeling, but these codes are only part of most analyses.  Many of the questions researchers have involve codes that require interpreting what our documents say in order to put like passages together across documents where language may be quite varied. 

This interpretation process makes it challenging to produce consistent results in group coding.  Some researchers take the approach of reliability by having more than one team member code every document and produce a score that captures how similar their coding is as proof of consistent results.  But that approach is not always possible due to the resources required, particularly with larger qualitative datasets, varied access to Qualitative Data Analysis software, and differences in deadlines for final products.  Instead, I have approached the process of group coding with 5 strategies that ensure consistent coding results across my team.

1. Documenting the coding scheme.  All of my projects have a codebook.  These include the name of the code, a definition of the code, notes on any exceptions or universal application of the code, and sometimes a sample quote where the code would be applied.  In some cases, I include specific words or phrases that indicate that the code should be used.  Similarly, for exceptions or universal applications, I flag where the code should never be applied (e.g., never with X code, never when a Y person/place/thing is mentioned), and where the code should always be applied.  The codebook is a living document that is continuously updated based on feedback and discussion with the team, so I also keep a “date last updated” at the top of the document so everyone is aware of whether they are working with the latest version of the codebook.
2. Test coding as a team.  Along with the codebook, I train the team with test coding the same document or 2 together.  We then meet to discuss and compare our coding, as well as review any questions that surfaced while doing the coding.  The comparison part of this process always surfaces different interpretations of the same codes from either the lack of application of a code or the application of a code to very different passages.  These differences provide the fodder for a conversation about how team members understand the codes and lead to an agreement on how to interpret the codes.  These clarifications are written up in meeting notes and then added to the codebook as a reminder to the team.
3. Regular team meetings.  Once test coding is completed and each team member has their individual coding assignment, I hold weekly team meetings to review questions.  I ask team members to bring specific passages with questions to the meetings including what codes they applied to the passage that they didn’t have questions about, which we all discuss.  While we always discuss the specific question that the team member had about their passage, seeing what other codes they applied to the passage also opens up new discussions about when we are applying code X or whether code Y should also be applied to the passage.  These are always a little tentative at the beginning as team members get comfortable with being vulnerable with each other, but with time and maintaining a supportive environment, everyone becomes used to bringing questions and concerns to the meeting.  Key to these discussions is uncovering the assumptions that each team member is bringing to their coding and making sure that we all get on the same page.
4. Continuous review.  Because not all issues with coding will surface as questions, I also include paired coding assignments on a regular schedule such as every 10 documents where 2 team members code the same document and we discuss any differences in their coding.  This brings new interpretation differences to the table and can also highlight areas of the codebook that coders are overusing or underusing. 
5. Cleaning the coding.  Once each team member has completed their coding assignment, they are tasked with cleaning their coding by revisiting the documents they coded early on in the process until they find that they are not finding any corrections to the coding.  Because our interpretation and consensus with the codes is evolving throughout the coding process, team members generally find that their early coding needs to be corrected to be consistent with their latest understanding of the codebook and the codes.  Based on meeting notes, you can also create a list of common errors for your coding team to review for based on changes to the codebook including additions and changes to definitions, and issues that arose in the coding question shares and the paired coding reviews.

With these 5 components, I find that my coding teams can reach agreement in their coding that provides consistency in the results despite having multiple brains involved in the process.