Alex Picoxanderpico

Threads / Cognitive tests at scale, for pennies

THREAD · IN DEVELOPMENT

Cognitive tests at scale, for pennies

Where do the quick judgments of a very cheap AI differ from ours, and what does running thousands of tests teach us about minds and about building with AI?

Jev, from TypeSafe AI, answers a typed question about a text with a probability in about a fifth of a second for about two thousandths of a cent, and when it is confident it gives the same answer each time. That makes psychology-style experiments practical at a scale that normally needs hundreds of volunteers: rating thousands of words, or running a battery of classic bias tests in minutes. What turns up matters to AI builders, Jev users and brain researchers: some human quirks appear, some do not, and on first impressions it does the opposite of people.

Judgment modelsCognitive scienceAI & machine learning

Resources

  • RESULTSEight classic bias tests, one cheap AI model, two centsA battery of classic psychology tests run on a fast, cheap AI judgment model: which human quirks it shows, which it does not, and the one it reverses. Includes the Linda problem and a first-impressions test that comes out the opposite way to people. Work in progress.
  • RESULTSRating words the way people doA cheap AI model rates 400 words each for concreteness, valence and arousal and is compared with published human norms: close on two, weaker on arousal, and where it disagrees. Work in progress.
  • GUIDEBuilding with a gut-feeling model: notes from 200,000 callsPractical rules for using Jev, a fast, cheap AI judgment model, from about 200,000 logged calls, including one failed attempt at a worm's nerve circuit.

RESEARCH RECORD

Records on this thread

8 records around this project.

  1. 2026-10-14

    Cognitive tests at scale, for pennies · Release · TypeSafe AI blog

    Jev: For the builders AND dreamers

    Announces Jev 1.14.0 and experimental image, box and long-context features behind danger-zone flags, noting that models perform worse on images than on pure text.

    Source ↗ Cognitive tests at scale, for pennies

  2. 2026-06-02

    Cognitive tests at scale, for pennies · Paper · Electronics

    Cognitive Biases in Large Language Models: A Systematic Quantitative Assessment and Debiasing Analysis

    Measures eleven cognitive biases in eight language models; reports framing among the strongest effects, sunk cost largely absent and anchoring weak, especially on factual questions.

    Source ↗ Cognitive tests at scale, for pennies

  3. 2023-02-02

    Cognitive tests at scale, for pennies · Paper · Proceedings of the National Academy of Sciences

    Using cognitive psychology to understand GPT-3

    Tests GPT-3 on classic experiments; it fell for the Linda conjunction problem, avoided a base-rate trap, and small changes to the prompts led it astray.

    Source ↗ Cognitive tests at scale, for pennies

  4. 2013-10-19

    Cognitive tests at scale, for pennies · Paper · Behavior Research Methods

    Concreteness ratings for 40 thousand generally known English word lemmas

    Human concreteness ratings for about 40 thousand generally known English word lemmas.

    Source ↗ Cognitive tests at scale, for pennies

  5. 2013-02-13

    Cognitive tests at scale, for pennies · Paper · Behavior Research Methods

    Norms of valence, arousal, and dominance for 13,915 English lemmas

    Human ratings of valence, arousal and dominance for 13,915 English lemmas.

    Source ↗ Cognitive tests at scale, for pennies

  6. 1988

    Cognitive tests at scale, for pennies · Paper · Journal of Personality and Social Psychology

    Outcome bias in decision evaluation

    Subjects rated decisions as better when the outcome was favorable, although they understood they had all the information available to the decision maker.

    Source ↗ Cognitive tests at scale, for pennies

  7. 1983-10

    Cognitive tests at scale, for pennies · Paper · Psychological Review

    Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment

    Reports that a conjunction can seem more probable than one of its parts, in judgments of lay people and experts, in between-subjects and within-subjects comparisons.

    Source ↗ Cognitive tests at scale, for pennies

  8. 1946-07

    Cognitive tests at scale, for pennies · Paper · The Journal of Abnormal and Social Psychology

    Forming impressions of personality

    In Experiment VI, the same six traits read forward or in reverse gave two groups different impressions; Asch concluded that the first terms set a direction that colors later ones.

    Source ↗ Cognitive tests at scale, for pennies

Related background

Selected publications that share a tag with this thread. They are background and do not shape it.