Paper review: in this series we summarize academic research on qualitative analysis.
A 2023 study by Xiao and colleagues tested two ways of asking GPT-3 to do deductive coding, applying a research team’s existing codes to new data. A codebook prompt performed better than one based around recommendations.
The paper: Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding, by Xiao, Yuan, Liao, Abdelghani and Oudeyer (IUI 2023).
Why it matters: A codebook is written by people and for people. Nobody designed it for an AI model, so there was no guarantee that it would work as a prompt. However, in this study, it did.
What they did: Researchers at Microsoft Research and Inria took 668 children’s questions that psychologists had already coded. They wrote two prompts from the psychologists’ codebook. One kept the codebook’s own layout: code, description, examples. The other was built around worked examples with the reasoning spelled out.
What the prompts looked like: Both prompts came from the same codebook. Only the way it was presented to GPT-3 changed. The codebook layout:
Code: HIGH
Description: the answer to this question is not a simple fact but requires explaining a mechanism, a relationship, etc.
Examples: Why were dinosaurs so big?
The example-led layout:
“Why were dinosaurs so big?” is an example of “HIGH” because the answer to this question is not a simple fact but requires explaining a mechanism, a relationship, etc.
By the numbers: The study measured inter-rater reliability between GPT-3 and the experts, using Cohen’s kappa.
With one example per code:
- 0.60 for the codebook layout versus 0.40 on the first code
- 0.38 versus 0.13 on the second
- With five examples the gap narrowed but held: 0.61 versus 0.52, and 0.35 versus 0.30
Between the lines: Definitions alone weren’t enough. Adding a single example per code was the biggest jump, but going from one to five changed little.
Yes, but: The model fell well short of the human experts and of the inter-rater scores a research team would accept, and it did so on a fairly simple codebook. The paper itself notes that AI has trouble with nuance, so a more interpretive codebook may well be harder. That said, this was a 2022 model. Models have improved a lot since, and we’ve seen AI get noticeably better at applying codes in that time.
What’s next: The authors ask whether a codebook written for people holds the model back, and whether codebooks should be written differently for AI. At least one later study takes that up directly: Halterman and Keith (2025) tested whether AI models faithfully follow real-world codebooks, and looked at how to prepare a codebook for both people and AI.
Delve’s take: If a codebook is a good way to tell AI how to apply your codes, then getting the codebook right matters a great deal. That matches what we’ve seen: investing in the codebook is the most reliable way to improve how AI applies it.
The study started from a strong one. Psychologists had developed the codebook, and two of them coding the same questions agreed at 0.88 and 0.90, which is high intercoder reliability. That work was done before AI saw a single question.
Our view is that analysis with AI will look a lot like analysis always has. It still comes down to discovering a codebook, refining it, and getting everyone who applies it to agree on what it means. The work hasn’t changed.
The bottom line: Codebooks were created so one researcher could explain their thinking to another. They may turn out to be a good way to explain it to AI as well.
Go deeper: Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding, Xiao, Yuan, Liao, Abdelghani and Oudeyer, IUI 2023. A free copy is on arXiv.
Related guides from Delve:
- Deductive and Inductive Coding in Qualitative Research
- How to Create a Qualitative Codebook
- How to Get More Accurate Results from AI in Qualitative Research
- AI Prompts for Qualitative Research
Try Delve AI for Qualitative Analysis | Start a Free Trial
Cite this article
Limpaecher, A. (2026, October 6). Your Codebook Is Already a Prompt. https://delvetool.com/blog/codebook-as-ai-prompt