Don't Assume AI Follows Your Codebook in Deductive Coding

Paper review: in this series we summarize academic research on qualitative analysis.

A 2025 paper by Halterman and Keith asks a question about using AI for deductive coding: once a model has your codebook, does it follow it?

The paper: Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts, by Andrew Halterman and Katherine A. Keith (Political Analysis, 2025).

Why it matters: Anyone using AI to apply a codebook should understand how it can fail. The training that makes a model powerful also leaves it with assumptions and probabilistic quirks.

An evaluative baseline: Recognizing those failures, the authors of the paper look to create evaluative tests to catch how well an AI can actually apply a qualitative codebook. They implemented these tests using small AI models and real political science codebooks. Here are five.

The AI overemphasizes the code name and underemphasizes the code description

Take “protest,” the authors’ example. A model already has an idea of what a protest is. A research field, or a particular framework, may define it quite differently, and the model may fall back on its own idea, whatever the codebook says.

To test this, the authors swapped each code’s name for a generic one, such as LABEL_1, and left the definitions alone. Scores fell, which suggests the models had been leaning on the names.

It fails the elephant test

A codebook is full of specific rules about what a code includes and leaves out. To see whether a model follows them, the authors laid a trap inside the code definition by adding the following line:

This category does not apply if the document discusses elephants.

They also added a line to the document being coded:

We support elephants.

The theory was that if the AI truly read the full code definition, it would understand not to code the document because it included elephants. However, the models would often still apply the code, meaning they did not act on a rule written into the definition.¹

The authors also found a real example, with no elephants involved. A passage about more funding for university students was coded by the AI as “welfare positive.” However, the “welfare positive” code explicitly said: “This category excludes education.” The correct code was “education positive.” The example could have been ambiguous if not for the explicit exclusion rule, which the AI seemed not to take into account.

It overemphasizes keywords

The authors found that the AI sometimes picked a code because its name appeared in the text, whether or not it was the correct one to apply.² For example, when coding a news story that used the word “rally,” the model applied the “rally” code, even though the story fit the codebook’s definition of a different code, “demonstration.”

The order of the codes changes the answer

If a model is reading the whole codebook, listing the codes in a different order should not matter. However, when the authors reversed and shuffled the codes, the answer changed for a large share of the documents. They suggest the models may not pay even attention across a long prompt.³

It makes up codes

The simplest test is seeing if the AI provides real codes or if they are hallucinated. On a 12-code codebook the models passed. On one with 142 codes, 45% of a sample of one model’s answers did not give a single valid code.

Yes, but: The authors chose small models so that others could repeat the work, and they expect newer models to do better. The paper was not meant as an evaluation of LLMs, but as a framework for evaluating LLMs.

Delve’s take: The authors found a number of ways AI can fail at deductive coding, and we have seen several of them while building Delve’s own AI coding. Some have faded as models improved: applying a code that isn’t in the codebook is much rarer now. Others are harder to spot. We still see AI lean on a code’s name over its description, and match keywords where the code calls for nuance.

There is no single fix. Some failures can be caught by the software around the model. Catching a code that isn’t in the codebook is easy, and Delve already does it. Others come from the model itself. How well a model follows inclusion and exclusion rules, or keeps track of the middle of a long codebook, will likely differ from model to model and from codebook to codebook.

Some of these tests need no human coding, which is an interesting idea and worth adopting selectively. But what a shuffled codebook reveals is not easy to interpret. The most reliable check, and the most intuitive for researchers, may be to compare the AI with human coding on your own data and your own codebook, the way teams check intercoder reliability with each other.

The bottom line: Teams have always checked that everyone applies a codebook the same way. AI needs that check too.

Stray thoughts:

A code can’t be judged on its own. The authors suggest a model needs the whole codebook, not one code at a time: shown only an “assassination” code, it might apply it to a drone strike that belongs under the “drone assassination” code. They did not see this happen in practice, but we have. A codebook should be considered as a whole, because a code’s boundary is often influenced by other codes.
A good codebook still comes first. The authors write that AI can scale up coding, but that preparing the codebook “cannot be bypassed or automated.” We agree. If you want good coding, you have to put the effort into a good codebook.

Go deeper: Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts, Halterman and Keith, Political Analysis, 2025. It is open access, and an earlier version is on arXiv.

Related guides from Delve:

Try Delve AI for Qualitative Analysis | Start a Free Trial

Cite this article

Limpaecher, A. (2026, October 6). Don’t Assume AI Follows Your Codebook. https://delvetool.com/blog/does-ai-follow-your-codebook

The authors adapted this test from Karpinska et al. (2024). ↩
This was a known issue before this paper. The authors call it a lexical overlap heuristic and cite Levy et al. (2023). ↩
Sensitivity to order was a known issue before this paper. The authors cite Liu et al. (2024), on models losing track of the middle of a long prompt, and Zhao et al. (2021), on how the order of examples in a prompt changes the answer. ↩