1 / 18
Session 17 · Decision Models

Can an AI tell you
when it is guessing?

RLCD, Reinforcement Learning for Calibrated Decisions, is the training method behind TypeSafe's Jev. Today: what calibration means, how RLCD rewards it, what we know about Jev's architecture, and how to use it in production.

01 The problem 02 Calibration 03 Training 04 Architecture 05 Production
What was missing

An LLM that is right 95% of the time
still cannot automate the job.

40 support tickets · LLM says "I'm sure" on every one · illustrative
All 40 look identical

Two of the 40 answers are wrong. The model sounded equally confident on all 40.

TypeSafe's launch post: "If a model can do a task 95% of the time but doesn't say when it's in the 5%, it can't automate that task." A human would have to re-check every answer.
In S03 you saw RLHF make GPT helpful. TypeSafe points out that RLHF "can also reward sycophancy and confident-sounding hallucinations." What we need now is a model whose confidence you can act on.
Meet Jev

A "System One" model: fast, intuitive decisions.
No sentences at all.

⚡

System 1: Jev

Fast and intuitive. "Is this email spam?" "Which team should get this ticket?" Picks an answer in about 70–500 ms and returns a probability.

🧠

System 2: GPT / Claude

Slow and deliberate. Writes code, drafts replies, reasons step by step. Takes seconds or minutes and produces free text.

15 Sep 2026
Released by TypeSafe AI (early access)
jev-1.13.0
Only published version so far
$0.042
per million input tokens · output is free
64k
tokens of context per request
The names come from Kahneman's Thinking, Fast and Slow (System 1 vs System 2) and from the economist William Stanley Jevons: when intelligence gets cheaper, people use far more of it. Founder Diogo Almeida previously worked at OpenAI on InstructGPT and ChatGPT, the RLHF research you studied in S03.

Sources: TypeSafe launch post · TypeSafe model docs · Wikipedia

What Jev returns

Unstructured state in.
Typed probabilities out.

TypeSafe calls it "a frontier-intelligence function call." You define the possible answers. Jev can only pick from your list, so it cannot invent a category you never offered.

noul · true / false
"Does this email need a reply?"
true
0.88
false
0.12
choice · pick from a menu
"Is this billing, bug, how-to or other?"
billing
0.82
bug
0.10
how-to
0.05
other
0.03
score · ordered scale
"How likely is this customer to cancel? (1–5)"
1 low
0.05
2
0.10
3
0.20
4
0.45
5 high
0.20
Probabilities shown are illustrative
Every answer is a full probability distribution, not just a label. That distribution is what RLCD trains to be honest.
TypeSafe says Jev "can't hallucinate." That means it can never return an option outside your list. It can still pick the wrong option from your list.
The one idea

Calibration: when it says 80%,
it is right 80% of the time.

perfect 50% 100% WHAT THE MODEL SAID (confidence) HOW OFTEN IT WAS RIGHT

🌦️ The weather forecaster test. Collect every day she said "70% chance of rain". If it rained on about 70% of those days, she is calibrated.

This chart is a reliability diagram. Bars on the dashed line mean the confidence is honest. Bars below the line mean the model is overconfident. In this example it said 95% but was right only 66% of the time.
Calibration is not accuracy. A model can be 60% accurate and perfectly calibrated, as long as it says 60%. What calibration gives you is knowing when to trust each answer.

Bar heights are illustrative, for teaching.

Where RLCD fits

Same pretrained model.
Three ways to reward it.

Pretrained language model RLHF Reward = what human raters prefer → Chat models risk: rewards sounding sure RLVR Reward = 1 if a program verifies the answer is correct → Reasoning models risk: confidently wrong RLCD Reward = how well probabilities match real outcomes → Decision models (Jev) goal: honest probabilities
TypeSafe's own wording: RLCD optimizes for "calibrated decisions: answers with epistemically honest probabilities on System One tasks." RLCD is a training method. It is not an architecture.
Why a new method was needed

Preference training squeezes the probabilities
into a confident spike.

POSSIBLE ANSWERS
Spread across several plausible answers.

Bar heights are illustrative.

Mode dropping. RLHF teaches the model to favour one style of answer, and the probability of every other answer drains away. What is left reflects what raters liked, not how often the answer is right.

RLVR pays 1 for a correct answer and 0 otherwise. The best strategy is to always bet everything on your top guess. The OpenJev-RLCD paper finds RLVR makes reasoning models "sharply overconfident."
RLCD pays for the whole distribution being honest. Being 100% sure about a wrong answer is the worst thing it can do.
The reward: a proper scoring rule

Brier score: penalty = (confidence − truth)²

Truth is 1 if the answer was right and 0 if it was wrong. Drag the slider and watch how being confidently wrong explodes.

CONFIDENCE THE ANSWER IS "BUG" PENALTY 0% 100% no bug really bug really there
Model's confidence: 95%
Brier penalty
0.003
Log-loss penalty (another proper rule)
0.05
"Proper" means the lowest average penalty comes from reporting what you actually believe. Bluffing never pays off over many examples.
TypeSafe has not published Jev's exact reward. Brier and log loss are what the open reproduction uses, and what outside observers suspect Jev uses.
How one training step works

Predict → score → penalize → update.
Repeated over many examples.

Task: "Does this code contain a security bug?" Pick an example and watch one step. This is the general idea of calibration training, not Jev's published recipe.

1 · INPUT
Task + true label
—
→
2 · MODEL OUTPUT
Decision + probability
—
↑
↓
4 · WEIGHT UPDATE
Gradient step
—
←
3 · SCORE
Brier penalty
—
CaseSaidTruthPenalty
A · right & surebug 95%bug0.003
B · wrong & surebug 98%clean0.960
C · unsurebug 52%either0.23–0.27
Case B is the main target. A 98%-sure mistake costs nearly 400× more than a 95%-sure correct answer earns. Hedging at 52% is cheap either way.
Why the model learns to be honest

Over many examples, the lowest penalty
comes from telling the truth.

On this kind of input the model is really right 70% of the time
Average Brier penalty if it always says…

Bar heights are stretched to make small differences visible. Read the numbers.

Average penalty = q·(1−p)² + (1−q)·p²

q = how often it is really right. p = what it says.

The green bar always lands where p = q. Saying 100% when you are right 70% of the time costs more. So does saying 50%. Gradient descent drifts the model toward honest numbers.
This is the whole trick: with the right reward, an honest model is the one that loses the least. That is how the training loop makes "80% sure" mean "right 80% of the time."
An open reproduction you can read

OpenJev-RLCD: calibrate first, then reinforce.

TypeSafe has not published its recipe. An independent team did, using Qwen3-1.7B, with open code. This is the best public picture of what RLCD can look like.

Base model Qwen3-1.7B STAGE 1 · CALIBRATE THE DECISION Model writes its reasoning, then we read its probabilities over the answer options after "Answer:" Train those probabilities with Brier or log score (reasoning not rewarded) STAGE 2 · REINFORCE THE REASONING Use the same proper score as an RL reward for the reasoning text A penalty keeps it close to the stage-1 calibrated model
GSM8K answer checking · share of items decided automatically at ≤ 5% error
RLCD
81%
GRPO
19%
RFT + KL
12%
SFT
6%
GRPO and RLCD had almost the same accuracy (91.8% vs 92.2%). The big difference is how many answers you can safely automate: GRPO's confidence could not separate its right answers from its wrong ones. Every method also got temperature scaling, a standard confidence correction applied after training.
This is an independent paper, not TypeSafe's method. Treat it as "RLCD is a real, workable idea", not "this is exactly how Jev was trained."

Source: Gao & Wang, OpenJev-RLCD (arXiv 2609.38850) · code on GitHub

Architecture · part 1

An LLM writes one token per step.
Jev returns every probability in parallel.

Autoregressive LLM (GPT, Claude)

0 GENERATION STEPS · one per token

Jev: parallel, non-autoregressive

billing
0.82
bug
0.10
how-to
0.05
other
0.03
0 QUERY · all answers in parallel
Ticket: "My card was charged twice."
LLM work grows with every word it writes. Jev writes no words: the answers are a fixed list you defined, so it can score all of them at once. TypeSafe credits this design for its claimed 70–500 ms responses, versus 3–329 s end to end for frontier LLMs.
Architecture · part 2 · be honest with students

What is inside Jev?
Known, unknown, and guessed.

Confirmed by TypeSafe

  • Transformer-based
  • Starts from a pretrained language model, then RLCD post-training
  • "New model architecture" with a parallel, hardware-aware sampler
  • No text generation: typed answers only, all questions in parallel
  • Not trained on customer requests or responses

Not disclosed

  • Base model and parameter count
  • Encoder, decoder, or something new?
  • The exact reward / loss function
  • Training data details and calibration curves
  • No paper yet. The CEO said the architecture is "close to the chest for now"

Popular guesses

  • BERT-style bidirectional encoder with classification heads
  • Built on top of an open-weight LLM
  • Brier or log loss inside the reward
  • Treat these as hypotheses, not facts
Common myth: "Jev has a separate confidence head." Jev returns one probability distribution per question. The API then computes choice and confidence from it with fixed formulas (next slides).

Sources: TypeSafe launch post · Learn Jev: what is known about RLCD · Wikipedia: Jev

Architecture · part 3 · a teaching model

A mental model you can draw on the whiteboard

Not TypeSafe's blueprint. A safe picture that matches everything they have published.

STATE "My card was charged twice for order A-104." text, JSON or list QUESTIONS (you define) Q1 choice: category? Q2 noul: urgent? Q3 score: anger 1–5? options fixed in advance TRANSFORMER pretrained, then RLCD-tuned layer layer … layer ONE parallel query internals undisclosed PROBABILITIES Q1 Q2 Q3 this is all Jev outputs PLAIN API MATH choice = top score = weighted average confidence = peakiness
Compare with a classic text classifier: same output idea (text → probabilities over labels). The difference is that the labels arrive in the request, zero-shot, so you never train your own model.
From probabilities to "how sure"

The confidence number is a formula,
not a second model.

A real request shape

# POST https://api.typesafe.ai/v1/systemone
{
  "model": "jev-latest",
  "state": "My card was charged twice.",
  "questions": {
    "category": {
      "type": "choice",
      "instructions": "What kind of request?",
      "criteria": {
        "billing": "charges, refunds",
        "bug": "something is broken",
        "how_to": "how do I…",
        "other": "none of these"
      }
    },
    "urgent": { "type": "noul",
      "instructions": "Is this urgent?" }
  }
}

Also on OpenRouter as typesafe/jev-1.13, and in Pydantic AI as typesafe:jev-latest.

Known limits: text input only (no images or audio) · up to 255 options per choice · questions are answered independently, so if one depends on another's answer you make two calls.

Worked example: choice with n = 4 options

billing
0.82
bug
0.10
how_to
0.05
other
0.03

confidence = (pmax − 1/n) / (1 − 1/n)
= (0.82 − 0.25) / 0.75 = 0.76

0 means a flat, random-guess distribution. 1 means all probability on one option. For score questions the answer is the probability-weighted average, and confidence measures how tightly the probabilities cluster.
By the formula, confidence only measures how concentrated the probabilities are. Whether high confidence really means "usually correct" is the calibration promise, so check it on your own labelled data.
Inference · the decision gate

Pick a threshold. Automate above it.
Send the rest to a human.

Confidence threshold: 70%
auto · correctauto · wrongsent to a human
—
automated
—
error rate on automated
—
human reviews

60 simulated tickets from a calibrated model. Number on each tile = confidence.

Incoming ticket / PR Jev decision + confidence · 70–500 ms confidence ≥ threshold? YES NO Automate route / tag / merge Escalate human or a bigger LLM
Higher threshold means fewer mistakes but more human work. Calibration is what makes this a dial you can tune instead of a guess.
The production pattern

Jev decides. The LLM writes.
A human sees only the hard cases.

📥 Every email your whole inbox JEV · 3 questions in parallel • needs a reply? (noul) • brand deal? (noul) • scam? (noul) cheap · fast · cannot hallucinate a label Needs reply + sure → Claude / GPT drafts it the expensive model only sees what matters Scam / no reply + sure → archive automatically no LLM tokens spent at all Low confidence → human inbox the few cases that need judgement
1
Give Jev a way outAdd "None" or "Other" so it never has to force a bad pick.
2
Send the unsure ones to a personStart at about 60% and tune it from real results.
3
Check before you trustRun it on a labelled set and compare accuracy at each confidence level.
4
Pin the versionUse jev-1.13.0, not jev-latest, once you've tuned a threshold.

Email pattern and habits 1–3: RoboNuggets "Jev use cases" guide. Habit 4: TypeSafe model docs.

Recap · and think like a skeptic

The story you should be able to retell

1
LLMs are smart but overconfidentYou can't automate a model that hides its mistakes.
2
Calibration = said 80%, right 80%The reliability diagram is the test.
3
RLCD rewards honest probabilitiesProper scoring rule (e.g. Brier) as the reward.
4
Confidently wrong = biggest penaltySo, over time, honesty is the cheapest strategy.
5
Jev: parallel, typed answers onlyArchitecture internals are still undisclosed.
6
Production = threshold gateAutomate the confident ones, escalate the rest.
Be a skeptic. TypeSafe's speed and cost figures (up to 193.6× faster, 444.6× cheaper) come from workflows built by its own team, scored against reference answers from OpenAI and Anthropic models. TypeSafe says so itself, and calls these "on the higher end of real world gains." Measure on your data before you trust them.
Name clash: a different 2023 paper, "RLCD: Reinforcement Learning from Contrastive Distillation" (Yang et al.), is an alignment method unrelated to Jev. Searching for "RLCD" will often find that paper first.
TypeSafe launch post Jev docs Learn Jev: RLCD OpenJev-RLCD paper OpenJev-RLCD code Pydantic AI + Jev The other RLCD (2023)
Same job · three products

Jev is first. It is not the only one.

In three weeks, three organisations shipped the same idea: typed answers, no generated text. Learn the pattern. Then pick the product that fits the hardware and the budget.

TypeSafe AI

Jev 1.13

15 Sep 2026 · hosted API
  • Closed model. You call it; you cannot open the weights.
  • Text in. Typed probabilities out. No images.
  • $0.042 per million input tokens. Output is free.
  • TypeSafe claims 70–500 ms end to end.
  • Architecture unpublished. Training method named RLCD.
AWS Strands Labs

Strands Decider 2B

1 Oct 2026 · open weights
  • Qwen3.5-2B torso. LM head removed. Pointer head of ~1M parameters. Rank-16 LoRA.
  • Runs on a local CPU, GPU, or Apple Silicon.
  • Median ~115 ms on an RTX 3090. ~153 ms for small tasks on an M3 MacBook.
  • Weights, training data, and scripts are published.
  • Same three question shapes: yes/no, choice, score.
OpenAI

Decisions API

DevDay 29 Sep 2026 · public beta
  • Hosted at POST /v1/decisions. Model: gpt-6-luna only.
  • Text and images in. Typed answers out.
  • Three types: predicate, choice, score.
  • OpenAI says about 10× faster than Luna on the Responses API.
  • Architecture unpublished. No RLCD claim.
Strands is the one you can open and inspect. OpenAI is the one that can look at a photo. Jev is the one this class uses. The skill that transfers is the same: define the options, read the probability, set a threshold.

Sources: TypeSafe launch post · Strands Decider blog · OpenAI Decisions docs · DevDay 2026 recap

Claims vs measurement

Those were their claims.
Here is what I measured.

Same three of our use cases, same states, same options, same gold labels, run against all three systems. 88 decisions each. Strands ran locally on an M2 Pro; Jev and the Decisions API were called over the internet from Dubai.

Read this before you quote a winner. McNemar exact test on the paired results: Jev vs Strands p=1.000, Jev vs Luna p=0.375, Strands vs Luna p=0.219. None of the accuracy gaps are statistically significant at 88 items. The three systems disagreed on only 7 of 74 choice decisions. The honest conclusion is that they are in the same accuracy band, and you should choose on latency, hosting and cost instead.

Latency is not like for like: Strands has no network hop but modest hardware; their own figure is ~115 ms on an RTX 3090. Jev cost uses TypeSafe's published $0.042/1M input; Luna uses OpenAI's $0.10/1M input, output unbilled on both.

The finding that actually matters

All three know when they are guessing.

Split every choice decision by whether the model reported 0.8 confidence or above. If the gate is real, accuracy inside the two buckets should differ sharply. It does, on all three.

systemdecisions at ≥0.8accuracy there decisions below 0.8accuracy thereseparation
Jev6698.5%8 37.5%61 points
Strands 2B4897.9%26 84.6%13 points
Luna59100.0%15 80.0%20 points

Above the gate · automate

Jev98.5% right across 66 decisions
Strands97.9% right across 48
Luna100% right across 59

Below the gate · send to a human

Jev37.5% right, almost a coin toss
Strands84.6% right
Luna80.0% right
Jev separates hardest, and that is the point of RLCD. A confidence number is only useful if being unsure actually predicts being wrong. Jev drops to 37.5% below the gate, so the threshold carries real information. The other two stay above 80% when unsure, which means their low-confidence bucket is safer but far less informative, so you would send work to humans that did not need it.

Choice decisions only (74 of 88). Yes/no questions carry no confidence field in any of the three APIs, by design. Run it yourself: 17_Decision_Models/comparison/benchmark.py