RESOURCE · COGNITIVE TESTS AT SCALE · Synthesis · October 2026 · work in progress

Cognitive tests at scale, for pennies: a short research article

About 220,000 logged calls and under four dollars. What a fast, cheap judgment model shares with people, where it differs, and what that is good for.

Thesis: A cheap, typed judgment model can support useful cognitive tests at a scale that was previously impractical, but it must be measured as its own system rather than treated as a small person.

• The surprise: in Asch-style trait lists, Jev trusts the last information it hears. Later negative traits lowered its rating, the reverse of the first-impression direction used for the human comparison.
• Linda depends on format: rated alone, the conjunction was higher on every tested profile; shown side by side, Jev chose the conjunction on none.
• It is a harsher but rank-faithful word-pair judge: on 999 SimLex-999 pairs, pooled r = 0.854 [0.841, 0.868], while the fitted slope was 0.636 and most ratings sat below equal human levels.
• It matches human sensorimotor word ratings best for touch (r = 0.783) and least for the inner body, interoception (r = 0.494).
• The cost: all of this took about 220,000 calls and about $3.80 in total.

AI developers should calibrate wording, order, and score scale before deployment; Jev users should average symmetric word orders and use ranks where levels are off; brain researchers should treat this as a cheap non-human comparison, not a brain model.

Limits: one pinned version, our own prompts, about a dozen items in many bias tests, and an unchecked Lancaster GitHub mirror. These results do not compare Jev with other models or show shared human mechanisms.

The article

Abstract

Jev is a typed judgment model. In these runs it was pinned to jev-1.13.0. It answered a question about text with a probability in roughly a fifth of a second. The project asks whether such a tool can make small, repeatable cognitive experiments cheap enough to run routinely.

We ran word-rating, judgment-bias, word-similarity, sensorimotor, and engineering tests. The latest two runs add a useful split. On 999 SimLex-999 word pairs, Jev ranked similarity much like people did (pooled Pearson r = 0.854 [0.841, 0.868]), but it gave systematically lower levels. On 600 sensorimotor words, it matched human ratings best for touch (r = 0.783 [0.745, 0.818]) and least for interoception, the inner-body sense (r = 0.494 [0.431, 0.557]). The most striking earlier result remains a reversal: in trait-list tests where people are usually described as weighting the first traits more, Jev weighted later traits more.

These are not claims that Jev thinks, feels, or models a brain. They are measurements of one version, with our own wording and small item sets. They show that a cheap judgment model can be treated as an experimental object and a calibrated component, provided its wording, order effects, and scale use are measured rather than assumed.

Introduction

Many AI systems need a small judgment before a larger action. Is a claim supported? Is a message urgent? Does this word fit this category? Usually these decisions are hidden inside a larger model or handled with a prompt that is never tested on its own. Jev makes the small judgment explicit: a typed question, text state, and a probability distribution over answers.

That makes a different kind of experiment possible. A test can ask the same question many ways, compare a known human norm with the model, reverse the input order, and preserve every call and its reported cost. The cost is small enough that a few thousand ratings are practical. The results are repeatable in the narrow sense that the exact version, item set, and wording can be named. They are not automatically general. A different model version, wording, or input format can change them.

Three audiences may benefit. AI developers can learn whether a judgment component is safe to use as a score, a ranker, or a gate. Jev users can learn practical rules for prompt design and calibration. Brain and cognitive researchers can use a cheap, inspectable non-human comparison point. That comparison point cannot explain human cognition by itself. It can, however, help separate patterns that follow from language and task structure from patterns that may depend on people.

Methods in brief

Every live run used the pinned version jev-1.13.0, logged each call in our logged results, and had a call and dollar cap. The new S1 run rated all 999 SimLex-999 pairs in two question wordings and both word orders: 3,996 live calls. The new L1 run rated 600 seeded words on eight Lancaster Sensorimotor dimensions in two wordings: 9,600 live calls. Human answers were not sent to Jev. Before each run, the project wrote guesses in our logged results; later scorecards distinguish those guesses from descriptive observations. Probe calls remain in the ledgers but were excluded from the S1 and L1 analyses. This article reports repo observations, not live calls made for this draft.

Results

Word pairs: good rank agreement, different levels

S1 gives the cleanest illustration of why one number is not enough. The pooled similarity correlation was 0.854. With the direct wording, “how similar in meaning,” which warned that related is not the same as similar, the forward-order Pearson correlation was 0.888. With a replacement wording, it was 0.766 by Pearson correlation, though its Spearman correlation was still 0.865. So the model largely preserved the ordering but changed the spacing of the ratings.

The scatter plot makes that visible. Almost all points lie below an equal-rating diagonal. The fitted slope is 0.636 and the intercept is 0.345. Jev was especially strict on some category, part-whole, and closely related pairs: alcohol/gin was 8.65 for people and 2.65 for Jev; other large under-ratings included think/rationalize, fever/illness, tobacco/cigarette, sheep/lamb, and book/literature. This list is descriptive and post hoc. It was not a pre-registered category test.

Word order is another practical warning. The two orders correlated about 0.97, but swapping the words still moved an individual answer by 0.337 levels with wording A and 0.254 with wording B. That is small relative to the whole scale, but too large to ignore when comparing close items. A symmetric relation should be asked in both orders and averaged.

Senses named in text

L1 compared Jev with human sensorimotor ratings. The model did not simply follow an expected text-only ordering. We guessed that vision and hearing would both reach 0.80; they reached 0.683 and 0.679. We guessed smell and taste would be lower than vision; they were higher, at 0.705 and 0.747. Touch was best at 0.783. Interoception, words tied to internal bodily states, was lowest at 0.494.

Across six perceptual senses, Jev and people selected the same highest-rated sense for 379 of 600 words, or 63.2%. The uniform six-way comparison value is 16.7%. This is substantial agreement but below the pre-registered 70% target. It should not be read as evidence that Jev has sensory experience. It shows only that a text-trained system’s word judgments align unevenly with human ratings.

The headline reversal

The bias battery provides the sharper behavioural contrast. In B3 and B4, the same positive and negative traits were presented in opposite orders. Jev’s rating was lower when the negative traits came last. The later information carried more weight. The later B5 run found the same direction across trait-list lengths and in job-candidate facts. This is the reverse of the first-impression direction used as the human comparison in the project’s Asch-style framing.

That reversal matters more than a vague claim that an AI “has biases.” It gives builders a testable input property: order belongs in the specification. It gives cognitive researchers a non-human system with a measurable, opposite pattern. We do not know why the reversal happens. It could be a property of list processing, bad-news placement, the wording, or the model’s learned text patterns. The experiments do not separate these explanations.

Discussion

The shared patterns are narrow but useful. Jev can order many word pairs similarly to people. It can track human word norms well for some concepts. It shows some familiar judgment effects, including separate-rating conjunction errors, anchoring where it lacks a known answer, and outcome bias. These results make it plausible that small judgment models can help construct stimuli, screen candidate items, or run controlled comparisons cheaply.

The non-shared patterns are just as important. It uses a different numeric scale for similarity. It does not consistently match human disagreement or arousal ratings. It is weakest on inner-body word ratings. It changes with wording and word order. The recency-versus-Asch reversal is the clearest example: a shared task format does not imply a shared cognitive process.

For AI developers, the message is practical. Calibrate the exact question you will ship. Use correlation or ranking when the absolute scale is off. Test alternative wordings. Average symmetric orders. Keep inactive inputs separate from active ones. For Jev users, the message is that low price does not remove the need for measurement; it makes measurement affordable. For brain researchers, the value is a controllable contrast class. It can show that a pattern appears without a person behind it, which makes "it follows from the task wording and language alone" a live alternative to test, but it cannot confirm a biological mechanism.

Limitations

1. Each bias test used about a dozen items, not a large representative sample.
2. The experimenters wrote the wording and many stimuli.
3. Every result is for one pinned model version.
4. Wording and rendering changed answers in earlier rounds and in S1.
5. The SimLex bibliographic record was checked, but the paper was not otherwise reviewed for this work.
6. The Lancaster data file was a GitHub mirror of unchecked origin.
7. The Lancaster human ratings were Mechanical Turk ratings, not a general population sample.
8. Some earlier human directions were read only from abstracts or were not rechecked in the underlying papers.
9. No results for other AI models or other methods were looked up here.
10. Correlation does not show equal scale use, shared representation, or shared mechanism.
11. The human norms cover English words and the selected samples, not all language.
12. The worm slice failed because the simple comparison model saturated; it cannot support a behavioural claim about the worm.

The scorecards are also a limit on our forecasting. B1 had 1 of 6 guesses right. B2 had 4 of 5. S1 had 1.5 of 4 if the mixed order result counts as half right. L1 had 2 of 4, treating its mixed first guess as no point.

What we would do next

1. Replicate S1 and L1 with pre-written alternative wordings and a held-out item sample.
2. Test whether averaging both word orders improves agreement on a separate similarity set.
3. Run the same small battery on another pinned judgment model before making any model-general claim.

Where the pieces are

• Eight classic bias tests
• Rating words the way people do
• Do word pairs feel as similar to Jev as to people?
• Which senses does a text-only AI hear in a word?
• Building with a gut-feeling model

References

1. Asch (1946), Forming impressions of personality (full text read; Experiment VI)
2. Tversky and Kahneman (1983), the conjunction fallacy (abstract read)
3. Baron and Hershey (1988), Outcome bias in decision evaluation (abstract and method read)
4. Binz and Schulz (2023), Using cognitive psychology to understand GPT-3 (full text read)
5. Pagliaro (2026), Cognitive Biases in Large Language Models: A Systematic Quantitative Assessment and Debiasing Analysis (full text read)
6. Dilution in legal decision making, Current Psychology (1998) (abstract read; it describes the dilution effect of Nisbett, Zukier and Lemley, 1981)
7. Hill, Reichart and Korhonen (2015), SimLex-999, Computational Linguistics (bibliographic record checked)
8. Lynott, Connell, Brysbaert, Brand and Carney (2020), The Lancaster Sensorimotor Norms, Behavior Research Methods (abstract read)

Independent work using Jev from TypeSafe AI; not affiliated with or endorsed by TypeSafe. Jev version: jev-1.13.0.