That gap is why percent agreement alone shouldn't be the gatekeeper for coding quality, whether your coders are two humans, two AI runs, or one of each.
The workflow we recommend before any codeframe goes into production:
The 2026 twist: more teams now run this check with one human coder and one AI coder. Clear-cut, deductive codes (“mentions price,” “mentions delivery”) tend to hold up fine. Interpretive or inductive themes, the ones requiring judgment calls, are where agreement drops, sometimes well below the human-to-human baseline.
When kappa comes in low, don't just retrain the coder and hope. Rewrite the codebook definitions, add examples for the edge cases, or tighten the AI prompt, then re-test on a fresh subset before you commit the full dataset.
How do you currently check whether your open-end coding, human or AI, is actually reliable before you report the numbers? Tools like Tagger build this kind of check into the codeframe step, so you catch the drift before it reaches the topline.
Practical guides on questionnaire design, qualitative analysis and market entry. One email a month. Unsubscribe anytime.
We keep seeing the same split in conversations with exporters. Some run a handful of calls with distributors and call it done. Others commission a full conjoint study that takes two months and a five-figure invoice…
That's the real argument for MaxDiff. Instead of rating items in isolation, respondents see a small set, say four or five items, and pick the one that matters most and the one that matters least. Repeat that twelve…
China imported 133,744 metric tons of pistachios in 2024, making it the world's second-largest buyer after the EU. The US supplied 68% of that, Iran most of the rest. Now Iran's exports are down roughly 30% year-to-date…
