What is SemIf? Semantic ifs from open models, explained

TL;DR. SemIf is an MIT-licensed research project that makes an open language model answer a decision instead of writing a reply. You give it some unstructured state, a question, and 2–16 labelled options. It runs one forward pass of a frozen model (Qwen3.5-4B in the docs) and reads each option's probability straight from the next-token logits. Nothing is generated, so there is nothing to parse.¹ The trick itself is not new — Llama Guard 3, ShieldGemma and Qwen3-Reranker already score with it²³⁴ — what SemIf adds is making the question and options runtime-defined. On one RTX 3090 the author measured 21 decisions over a shared document in a median 1.023 s, against 5.332 s for the same model writing a compact JSON array.¹ The catch, stated by the project itself: the scores are not calibrated confidence.⁵

Updated 24 September 2026. Six independently-authored reimplementations of Jev's interface (SemIf among them) now exist, two outside analyses of it reached the Hacker News front page, and SemIf itself gained a CPU backend and a calibration layer. See Update — 24 September 2026, below, for what that changes.

Updated 26 September 2026. The pattern now has a local serving runtime, Ollaya, that answers the hosted API's wire format on your own machine. Two independent benchmarks have also measured the hosted model. See Update — 26 September 2026, below.

Updated 29 September 2026. Jeff, a set of 0.8B and 2B Jev-compatible decision models, was trained on one workstation GPU; its benchmark figures are its author's own. A preprint has also counted 2,170 Jev projects on GitHub. See Update — 29 September 2026, below.

What SemIf is

The repo's tagline is "Semantic ifs from open models, on a 3090 at home." Its starting observation:

"Most agent decisions are small: route this, retry that, does the evidence support X? A chat model can answer them, but it spends time generating text that software immediately parses back into an if statement."¹

SemIf is an open reproduction of an interface pattern — runtime-defined semantic decisions that return typed scores. The pattern comes from Jev, a closed hosted model from TypeSafe. SemIf was formerly called OpenJev, says it is "not affiliated with or endorsed by TypeSafe," and that it "does not reproduce Jev's undisclosed model or training."¹

TypeSafe describes Jev as "a frontier-intelligence function call: unstructured state in, typed probabilistic decisions out," and advertises an "end-to-end response time" of "70ms-500ms," support for "a cardinality up to 255" options, and input pricing of "$0.042 / MTok."⁶ Those are vendor claims; nothing in this post tests them (two independent benchmarks have since measured the hosted latency; see the 26 September update). Jev's launch drew a large Hacker News thread (1,937 points and 509 comments when fetched on 21 September 2026), including the pointed objection that "the speed comparison seems misleading… Jev can only generate structured output."⁷ Several open clones followed within days; Latent.Space rounded up six of them.⁸

One correction to that early coverage: Latent.Space described SemIf as a Qwen3.5 backbone with "a tiny three-class NLI classifier on the last token."⁸ The code on master (commit ca3ba65, 19 September 2026) has no classifier head. It reads letter-token logits directly, and docs/RESULTS.md lists "runtime-defined questions rather than a fixed task classifier head" as a point in its favour.⁵ The round-up's description fits a different, similarly named project that it linked in the same line: AlexWortega/openjev on Hugging Face, a Qwen3.5 sequence classifier with three NLI labels and last-token pooling.⁹ Two projects called OpenJev appeared in the same week; SemIf is the one with no trained head. Trust the code over the round-up.

In packaging terms SemIf is a small Python package (semif-phase1, v0.1.0) with one CLI, semif-score. It also ships an MLX backend for Apple Silicon (Qwen3.5 models only), a browser-only WebGPU demo built on wllama/llama.cpp with "no build step, API, database, telemetry, or server-side inference," and a frozen benchmark bundle with row-level predictions and SHA-256 checksums.¹¹⁰

How a decision is computed

From src/semif_phase1/core.py and direct.py:

1. Your options are relabelled A, B, C, … up to P — so 2–16 options.
2. The model gets a fixed system prompt: "Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. Respond with only its uppercase letter, with no explanation or reasoning." The user turn is a JSON payload of evidence, criterion, and the lettered option descriptions.
3. SemIf runs one forward pass and keeps only the last-position logits for the letter tokens.
4. A softmax over just those logits gives one score per option, mapped back to your option IDs.

flowchart TD
    A[State, question and 2-16 described options] --> B[Options relabelled A, B, C ... P]
    B --> C[Fixed system prompt plus JSON payload]
    C --> D[One forward pass of a frozen model]
    D --> E[Last-position logits for the letter tokens only]
    E --> F[Softmax over those logits]
    F --> G[One score per option ID - no token sampled]

Figure: how SemIf scores one decision. The state, question and described options are relabelled with letters, wrapped in a fixed prompt, and passed through a frozen model once; only the letter-token logits at the last position are kept, softmaxed, and mapped back to your option IDs. No token is ever sampled.

No token is sampled — docs/METHOD.md puts it as "It does not decode a token."¹¹ The code refuses to run if a letter isn't exactly one round-trip token, or if appending the letter would change how the prompt tokenizes. It also refuses to truncate: rows over --max-tokens (default 4096) raise an error.¹

This is the same scoring rule that the original MMLU benchmark used — it compares "the probabilities of the following letter answers" (A, B, C, D) — and it matters which rule you pick: Hugging Face showed one model scoring 0.636 under letter-probability scoring and 0.488 under a different full-answer scoring of the same benchmark.¹² Robinson et al. found that when a model reliably binds options to their letters, showing all options and reading the letter beats scoring each answer separately.¹³ SemIf takes the letter-reading side of that choice.

Four ways to run it

shared enforces its precondition: every row must carry exactly the same state, or it raises "Shared scoring requires one nonempty exact state".¹

How well it works: the author's numbers

Every figure in this section comes from the repo's README.md and docs/RESULTS.md. They are the author's measurements; they were cross-checked against the repo's files and its raw prediction bundle, but not re-run.

Speed. One RTX 3090, Qwen3.5-4B, one owned state, 21 yes/no criteria (median of 3):

The two paths agreed on 18 of 21 criteria, which the README frames as "a systems comparison rather than a claim that the two readouts are semantically equivalent."¹

On a larger 37-state × 21-criterion workload (777 decisions, states of about 8,000 characters):

Reusing the state gives about 8.6× the throughput of fresh scoring on the same GPU. The author labels the reuse paths experimental: BF16 execution changed 5–6 of the 777 argmax decisions relative to fresh scoring.⁵ The repo also warns that these numbers "must not be directly divided into TypeSafe's reported service latency" — a 3090 at home and a hosted service are different systems.⁵

Quality. Direct logits with Qwen3.5-4B were the strongest open baseline tested:⁵

WANLI is an independent natural-language-inference dataset of 107,885 examples built by GPT-3 generation plus crowdworker revision,¹⁴ so it is the least author-controlled number here — and also the lowest. The README compares 0.845 with "0.883 for published Jev," noting it covers "the 102 rows that could be aligned from public artifacts, not TypeSafe's reported 711-row aggregate," and that no live Jev endpoint was run.¹ Two things to know before reading that as a near-tie. First, the 0.883 figure could not be traced to a TypeSafe page for this post; it rests on SemIf's README. Second, TypeSafe's own eval dashboard defines its reference answers as "an average of the responses of GPT-6 Astra and Claude Fable 5.1, both at high thinking"¹⁵ — so "reference agreement" measures agreement with frontier models, not with human-verified truth.

Smaller models fall off fast: Qwen3-0.6B scored 0.440 mean family balanced accuracy on the authored set, MiniCPM5-2B 0.686.¹

What SemIf is for: use cases, with evidence

The useful way to think about SemIf is as one instance of a pattern: ask a closed question, read the answer's probability, threshold it. That pattern already runs in shipped models, which is the strongest evidence it works. For each use case below: the production precedent, what SemIf itself ships or measured, and how strong the evidence is.

1. Safety and content moderation gates

Precedent (confirmed). Meta's Llama Guard 3 model card: "we look at the probability for the first token, and use that as the 'unsafe' class probability. We can then apply score thresholding to make binary decisions."² Google's ShieldGemma does the same, taking the logits for Yes and No at the last position and softmaxing them into a policy-violation score.³

In SemIf. The browser demo's "Email triage" preset is really a moderation-style call: a fake payroll email classified as Legitimate / Spam / Phishing.¹⁰

Difference that matters. Llama Guard and ShieldGemma were trained on a safety taxonomy. SemIf asks a general model a question you wrote at runtime. You gain flexibility — any policy, no fine-tune — and give up the training that made those scores meaningful. Treat a SemIf moderation gate as a pre-filter in front of a trained classifier or a human, not a replacement.

2. Retrieval re-ranking

Precedent (confirmed). monoT5 ranks passages by training the model to "generate relevance labels as 'target words'" and reading "the underlying logits of these target words."¹⁶ Qwen3-Reranker-4B's reference code stacks the no/yes logits, log-softmaxes them, and uses P(yes) as the relevance score.⁴

In SemIf. Both the direct path and the reranker mode scored Recall@1 of 1.000 on a code-retrieval set (6 queries) and 0.929 on a company-knowledge set (7 queries).¹ Those samples are far too small to rank the two approaches. For re-ranking specifically, a purpose-trained reranker is the default; SemIf's direct path is interesting when you want to pick among described options rather than score query–document pairs.

3. Routing: queues, teams, tools

In SemIf. The shipped example route-1 routes "Customer asks to reset a forgotten password and says the reset email never arrived." across account_access / billing / sales; the demo's "Account support" preset routes a locked account between access support, billing, and "close as resolved."¹¹⁰

Alternatives worth weighing. Aurelio's semantic-router routes with embedding similarity instead of LLM generation ("we use the magic of semantic vector space"),¹⁷ which is faster still but needs example utterances per route. RouteLLM trains a router to pick between a strong and a weak model and reports cutting cost "by over 2 times in certain cases."¹⁸ SemIf sits between them: no training and no example set — just option descriptions — at the cost of a 4B-model forward pass.

4. Evidence checks with an "insufficient" escape hatch

In SemIf. support-1 asks whether a log shows the deployment succeeded, with yes / no / insufficient;¹ WANLI labels are mapped to supported / insufficient / contradicted.¹¹ The authored benchmark includes a 36-row missing-evidence set, built to test whether the model admits it doesn't know. It mostly does — 31 of the 36 rows picked insufficient in the committed raw predictions, by this post's count — but it still made one confident (>0.8) non-insufficient choice.⁵

Why the escape hatch matters. Without an insufficient option, a softmax over the remaining options must sum to 1, so the model is forced to pick something. Adding the option is the cheapest abstention mechanism available.

5. Policy application and agent action gating

In SemIf. policy-1 asks whether listing files in a production backup requires a change ticket under a deletion policy.¹ On Every's public parallel-judgment "action firewall" set — ask several yes/no risk questions per agent action (destructive? reversible? out of scope? sharing sensitive data? needs confirmation?) and compose the scores into allow, confirm or block¹ — SemIf scored 0.700 composed accuracy over 10 actions, tied with the reranker path.⁵

Outside evidence. One independent developer benchmarked Jev on 60 agent tool-call-risk cases and got 91.7% (55/60), noting that "every incorrect answer came with confidence below 1.000" — while flagging that 60 cases is too few and there were no frontier-model baselines.¹⁹ No production write-up of an open-model logit-based action firewall was found for this post. Read this use case as promising and unproven: 10 actions is a smoke test.

6. Judgment grids and LLM-as-judge

In SemIf. Many criteria over one document — a checklist review of a contract, a PR description, an incident report — is exactly the shape shared mode is built for: 21 criteria per state in the speed benchmark, and 0.806 on Every's 36-row judgment grid.¹

Precedent (confirmed). G-Eval weights a judge's score by token probabilities (score = Σ p(sᵢ)·sᵢ) because "LLMs usually only output integer scores… This leads to many ties."²⁰ Reading probabilities rather than a single sampled answer is an established fix for coarse judge outputs.

7. High-volume triage with a human fallback

Precedent (confirmed). OpenAI's logprobs cookbook describes the operational version of this: logprobs let you "set classification or confidence thresholds," so you can "automatically classify headlines that exceed a certain threshold and send less certain ones for manual review" — and it advises validating those thresholds "with task-specific evals."²¹

In SemIf / Jev. TypeSafe's public eval subset — which SemIf re-scored — is drawn from four workflows: invoice processing, agent-trace observability, customer service, and security incidents.¹ Those are the domains the pattern's own authors think it fits. The worked example below is one of them.

When not to use it

• When you need a trustworthy probability, not a ranking. SemIf stamps its direct, serial and shared output rows with "conditional option score; uncalibrated as decision confidence" (reranker rows say "relative option compatibility; uncalibrated as categorical probability"), and lists "calibrated probabilities suitable for operational thresholds" among things it has not established.⁵ Outside research agrees this is the weak spot: Kamath et al. found "abstention policies based solely on the model's softmax probabilities fare poorly, since models are overconfident on out-of-domain inputs";²² Tian et al. found verbalized confidences "typically better-calibrated than the model's conditional probabilities" in RLHF-tuned models.²³ (The counterpoint: Kadavath et al. found large models "well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format."²⁴ Format is doing a lot of work there.)
• When option order or wording might change. Reversing option order flipped 10 of 36 direct decisions in SemIf's perturbation test; rewording the criterion flipped 9.⁵ That matches the literature: Zheng et al. trace LLMs' preference for particular option IDs to "token bias";²⁵ Pezeshkpour & Hruschka measured "approximately 13% to 75%" performance gaps when options are reordered;²⁶ Wang et al. showed judge rankings "can be easily hacked by simply altering their order."²⁷ Zhao et al.'s contextual calibration — score a content-free input like "N/A" and correct for the bias — gained "up to 30.0% absolute" on few-shot classification²⁸ and is worth trying on top of SemIf.
• When the decision needs step-by-step reasoning. A single forward pass forbids chain-of-thought by design. Sprague et al. found CoT "gives strong performance benefits primarily on tasks involving math or logic, with much smaller gains on other types of tasks";²⁹ Liu et al. found tasks where reasoning hurts, by "up to 36.3% absolute accuracy."³⁰ So: fine for most language-judgment calls, a poor fit for "does this invoice total match the line items?"
• When someone needs to read the why. SemIf returns a letter's probability and nothing else. Anthropic's ticket-routing guide takes the opposite design — reasoning inside <reasoning> tags before the label — precisely because explainability is a requirement in many triage settings.³¹
• When the output is open-ended (a summary, a reply, an extracted field) or has more than 16 options (the CLI's cap; the browser demo allows 20).¹⁰
• When you plan to ship a quantized build you haven't evaluated. The repo notes quantization changes the probabilities, and that its owned scores are "not presented as measurements of the quantized artifacts."⁵
• When the action is irreversible. "A forced typed output can still be semantically wrong."¹¹ Money, deletions and account changes stay behind a human.

Applying it: a step-by-step worked example

Scenario. A support team receives a few thousand tickets a day and wants four decisions on each before a human sees it: which queue, whether it's urgent, whether the customer is asking for a refund, and whether a canned auto-reply is safe. All four are about the same ticket text — the textbook case for shared mode.

Step 1 — Install

Python 3.10+, CUDA, and one GPU that can hold a 4B model in BF16 (Apple Silicon: the MLX backend; the Qwen3.5-4B weights are about 9 GB on disk).¹

git clone https://github.com/TheoLeeCJ/SemIf.git && cd SemIf
python -m venv .venv
. .venv/bin/activate
export HF_HOME=/path/to/large-drive/huggingface
pip install -e '.[test]'          # Apple Silicon: pip install -e '.[test,mlx]'

The torch loader insists on exactly one visible CUDA GPU (CUDA_VISIBLE_DEVICES=0), and remote models need a pinned 40-character commit revision, not a branch name — deliberate, for reproducibility. (A CPU-only llama.cpp backend landed on 22 September 2026, after this post was first written — see the update below.)

Step 2 — Write the decisions as JSONL

One row per question: id, state, question, and options (each with an id and a description); 2–16 options, unique IDs. state can be a string, JSON object, or array. For shared mode, every row carries the identical ticket:

ticket-4812.jsonl

{"id":"t4812-queue","state":{"subject":"Charged twice this month","body":"Hi, I was billed twice for my March subscription (two charges on my card on 3 March). Please refund the duplicate. I also can't log in since resetting my password yesterday.","customer_tier":"pro"},"question":"Which queue should own this ticket first?","options":[{"id":"billing","description":"Billing, charges, invoices and refunds."},{"id":"account_access","description":"Login, password and authentication problems."},{"id":"technical","description":"Bugs and product malfunctions."},{"id":"sales","description":"Pricing questions and product evaluation."}]}
{"id":"t4812-urgent","state":{"subject":"Charged twice this month","body":"Hi, I was billed twice for my March subscription (two charges on my card on 3 March). Please refund the duplicate. I also can't log in since resetting my password yesterday.","customer_tier":"pro"},"question":"Does this ticket need a response within one hour?","options":[{"id":"urgent","description":"Yes: money was taken incorrectly or the customer is locked out."},{"id":"normal","description":"No: it can wait for the normal queue."},{"id":"insufficient","description":"The ticket does not contain enough information to decide."}]}
{"id":"t4812-refund","state":{"subject":"Charged twice this month","body":"Hi, I was billed twice for my March subscription (two charges on my card on 3 March). Please refund the duplicate. I also can't log in since resetting my password yesterday.","customer_tier":"pro"},"question":"Is the customer explicitly requesting a refund?","options":[{"id":"yes","description":"The customer explicitly asks for money back."},{"id":"no","description":"The customer does not ask for money back."}]}
{"id":"t4812-autoreply","state":{"subject":"Charged twice this month","body":"Hi, I was billed twice for my March subscription (two charges on my card on 3 March). Please refund the duplicate. I also can't log in since resetting my password yesterday.","customer_tier":"pro"},"question":"Can this ticket be fully resolved by a canned auto-reply with no human action?","options":[{"id":"yes","description":"A standard reply fully resolves it."},{"id":"no","description":"A human must act (e.g. issue money, unlock an account)."}]}

Two tips that follow from how SemIf works:

• Write option descriptions, not just labels. The model sees the descriptions, lettered A, B, C…, never your IDs.
• Add an insufficient option wherever missing evidence is possible (use case 4).

Step 3 — Score all four decisions in one pass

CUDA_VISIBLE_DEVICES=0 semif-score \
  --mode shared \
  --model Qwen/Qwen3.5-4B \
  --revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a \
  --input ticket-4812.jsonl \
  --output ticket-4812.results.jsonl

The output file must not already exist — the CLI opens it create-only. On a Mac, add --backend mlx.

Step 4 — Read the output

This ticket file was not run for this post. Here instead is a real recorded row: the author's CLI output for the repo's own route-1 example, captured on an Apple M5 Max with the MLX backend in direct mode on Qwen3.5-4B at the pinned revision (the run predates the rename, so it came from openjev-score). It is kept in the repo at docs/media/openjev-mlx-results.jsonl; the long model metadata object and two SHA-256 hash fields (input_ids_sha256, prompt_sha256) are trimmed here.³²

{"id": "route-1",
 "option_ids": ["account_access", "billing", "sales"],
 "probabilities": [0.9999798389588899, 1.4738902057685174e-05, 5.422139052461843e-06],
 "option_logits": [28.375, 17.25, 16.25],
 "answer_token_ids": [32, 33, 34],
 "input_tokens": 130,
 "prompt_version": "direct-options-v1",
 "readout": "native last-position logits restricted to declared answer slots; no generated tokens",
 "probability_status": "conditional option score; uncalibrated as decision confidence",
 "forward_seconds": 0.0577162504196167,
 "total_seconds": 0.05874616699293256}

Two things to read off it. The same run's policy-1 row is much less lopsided — not_required 0.938, required 0.060 — so the scores do spread when a question is genuinely closer.³² And probability_status travels with every row: these are scores conditional on the options you supplied. Your shared-mode rows carry the same core fields — option_ids, probabilities, option_logits, probability_status — but per-row forward_seconds/total_seconds are replaced by one shared_timing block for the batch (prefill and suffix-forward seconds, batch size); model.serving_config reads native-state-prefix-parallel-v1 on torch and mlx-shared-v1 on MLX.

Step 5 — Turn scores into ifs, with a human fallback

import json

rows = {r["id"]: dict(zip(r["option_ids"], r["probabilities"]))
        for r in map(json.loads, open("ticket-4812.results.jsonl"))}

def decide(scores, threshold):
    best = max(scores, key=scores.get)
    return best if scores[best] >= threshold else None   # None → human review

queue     = decide(rows["t4812-queue"], 0.60)
urgent    = decide(rows["t4812-urgent"], 0.70)
refund    = decide(rows["t4812-refund"], 0.80)
autoreply = decide(rows["t4812-autoreply"], 0.90)

if queue is None or urgent in (None, "insufficient"):
    send_to_human_triage("t4812")
else:
    assign("t4812", queue, priority="p1" if urgent == "urgent" else "p3")
    if refund == "yes":
        open_refund_review("t4812")          # a human still approves money
    if autoreply == "yes":
        send_canned_reply("t4812")

The thresholds are placeholders. The real route-1 row above shows why raw scores can't set them: an uncalibrated model will happily report 0.99998. That's the next step.

Step 6 — Calibrate on your own data before you trust it

The repo is blunt: "Returned probabilities are conditional on the supplied options. Calibrate and validate them on the workload where they will make decisions."¹ A practical loop:

1. Pull 200–500 past tickets that humans already labelled for each question.
2. Score them with the exact same prompts, model, revision and serving mode — the author's BF16 reuse paths changed a handful of close decisions, so the mode is part of the system.
3. Per question, pick the threshold that hits your precision target (say, 98% before an auto-reply fires); everything below goes to a human.
4. Score each question twice with the option order reversed; route disagreements to a human. With 10-in-36 order flips in the author's own test, this is cheap insurance.⁵
5. Re-check whenever you change option wording, order, model or quantization.
6. Keep irreversible actions behind a human. Use SemIf to route and prioritize, not to be the final authority.

Step 7 — Scale up

For a batch of tickets, group rows by ticket and run shared once per ticket, or run serial over a mixed file (it re-uses the prefill across consecutive identical states). To try the idea before touching a GPU, open the browser WebGPU demo: it runs Qwen3-0.6B (639 MB), MiniCPM5-2B (1.56 GB) or Qwen3.5-4B (3.01 GB) quantized models entirely in the browser. The repo calls the demo's timings "not portable performance claims," and its quantized scores need their own evaluation.¹⁰

Update — 24 September 2026

In the three days after this post was published, the pattern it describes stopped being one project and became an ecosystem. Six independently-authored reimplementations of Jev's interface now sit on GitHub, two outside analyses of it reached the Hacker News front page, and SemIf itself shipped a CPU backend and a calibration layer.

Star counts and Hacker News scores move. Every figure below was read from the GitHub REST API and the Hacker News API on 24 September 2026 and is stated as observed on that date.

Six reimplementations, six different authors

The strongest evidence that an interface is real is other people rebuilding it without permission. At least six separate GitHub accounts have now published their own take on Jev's typed-decision interface. The six below were all created between 16 and 21 September 2026:³³

Read the third column, not the second. Stars measure attention and decay; six distinct authors independently arriving at the same interface is a structural fact that does not decay. Two of them — simple-jev and AnyJev — describe themselves as adapters that turn any model into a Jev-style endpoint, which is the clearest possible statement that the authors believe the mechanism is generic rather than proprietary.

Two honest caveats. First, a large share of the GitHub activity around this term is curation, not implementation: at least seven distinct "awesome-jev" list repos appeared in the same window, and a list is evidence of attention, not of a working reimplementation.³³ Second, star counts at this age are dominated by a single front-page placement, so they rank visibility, not quality.

The mechanism, deflated in 25 lines

The fastest-climbing Jev item of the week argues the opposite of a moat. "Jev in 25 Lines of Python," published by Duarte O. Carmo on 22 September 2026, reached 631 points and 197 comments in roughly 18 hours on Hacker News, where it was posted the next morning — about 35 points an hour, the highest rate of any Jev submission in this period.³⁵ It reduces the whole thing to reading logits: "It classifies: it gets a prompt with choices and outputs probabilities. It's fast. It's local."³⁶

It is explicitly a joke. The piece labels itself "a parody blog post" and links to three open implementations it considers more complete — one of them SemIf's own site.³⁶ But a parody only lands if the audience already agrees, and 631 points is a large audience agreeing that the mechanism is ordinary. That is independent support for this post's opening claim — reading option probabilities from one forward pass is an old trick, and what is new is the interface around it, not the arithmetic.

An outside read on where the moat actually sits

The second front-page analysis reaches the same place by a different road. In "Will OpenAI eat Jev's lunch?" (316 points, 221 comments), John Berryman argues that "OpenAI has been using large language models implicitly as specialized classifiers since at least the introduction of tool calling."³⁷³⁵

His conclusion is blunt: "Architecture-wise, I don't think there's much of a moat for the very reasons stated above." His candidate for the real moat is not the readout but the data — "Perhaps the real moat is in the training data itself."³⁷

That distinction matters for anyone choosing between a hosted decision model and an open one. If the readout is commodity and the training data is the asset, then an open reimplementation gets you the interface immediately and the decision quality only as far as its base model already goes — which is exactly the gap SemIf's own quality tables show.

Calibration got an outside argument — and a shipped answer

This post's sharpest warning was that the scores are not calibrated confidence. Both halves of the ecosystem have now converged on it independently.

On the argument side, Alex Molas published "Jev Can't Be Calibrated" on 23 September 2026. His case is distributional rather than architectural: "calibration is not just a property of the model, but also of your data distribution," and "for the same input and prompt, Jev will provide the same probabilities to both companies, regardless of their different underlying data distributions."³⁸

His prescription matches this post's Step 6 almost line for line — "If you want calibrated probabilities you'll still need to recalibrate Jev's probabilities on your own data," and "A few hundred labeled examples from your actual data can be enough."³⁸ His summary judgement is worth keeping: treat these outputs "as good scores (they rank examples well) rather than good probabilities."³⁸

On the shipping side, SemIf added a per-workload temperature-scaling layer on 22 September 2026.³⁹ The documented result is narrower than the headline suggests, and the repo says so itself:⁴⁰

One clean win, on the least author-controlled workload — the same WANLI set that was the lowest score in the quality table above. The other two rows did not gain because scaling failed there; the repo's own reading is that on those workloads the model is "already close to calibrated," so there was little to fix.⁴⁰ Two things keep the win from overturning anything.

Calibration is monotone, so "accuracy, balanced accuracy, and every other decision metric are identical before and after. Only confidence changes."⁴⁰ And it is a separate labelled step: "the native scorer, its prompts, and every committed raw prediction are unchanged,"⁴⁰ while docs/RESULTS.md still lists "Calibrated probabilities suitable for operational thresholds" among what the project has not established.⁵

The net effect is that the caveat holds and the remedy got cheaper. You still cannot read a raw score as a probability. You can now fit one scalar per workload on labelled rows you already have — which is what Step 6 told you to do, now with the fitting code in the repo.

What changed in SemIf itself

Four repository-level changes since this post was written, all confirmed at commit 23cf1f3 (23 September 2026):³⁹

• The repo was renamed from TheoLeeCJ/SemIf to TheoLeeCJ/SemIf-OpenJev. Every SemIf URL in this post still resolves, because GitHub keeps a permanent redirect on renamed repositories.³³
• A CPU-only path exists. A llama.cpp backend scores the same prompts from a local GGUF with no CUDA device: --backend llamacpp --gguf /path/to/model.gguf, with --llama-threads capping threads. Prompt construction stays on the pinned reference tokenizer, so "prompt_sha256 matches the Torch backend row for row," and scores carry the GGUF checksum. A slower full-precision reference path is available as --device cpu --dtype float32.³⁹
• PyTorch/MPS scoring landed for Apple Silicon (--device mps), alongside the MLX backend this post already covered.³⁹
• A Qwen3.8-27B EXL3 bridge was contributed, scoring 0.958 balanced accuracy on the same 144 authored rows against the 4B baseline's 0.813. The repo is careful about what that is: "a system-level quality comparison rather than a controlled model-size or quantization ablation: model family, size, quantization, and runtime all differ," it "has not yet been run on the other quality workloads," and it agrees with the pinned 4B model on 84.43% of the 777-decision fixture.³⁹

What this changes in the post above

• The "Does SemIf need a GPU?" answer was too narrow. A CUDA device is no longer required. The FAQ below carries the correction; Step 1's install block describes the torch path specifically, which is still accurate for that backend.
• "Not calibrated" stands, with a route out. The native readout is unchanged and still uncalibrated. An optional, separately-fitted temperature layer now ships in the repo.
• Nothing in the mechanism, the CLI contract, the speed tables or the quality tables changed. The 1.023 s versus 5.332 s figures, the 777-decision throughput, the order-sensitivity flips and the option cap are all still the numbers on master.
• The Hacker News figures in the body are dated 21 September 2026 and were accurate then. The origin thread now reads 1,977 points and 513 comments.³⁵

Update — 26 September 2026

Since the last update, the pattern has gained a local server. Ollaya, an Apache-2.0 runtime created on 23 September 2026, pulls open decision models by name and serves them on your own machine behind an API that speaks TypeSafe's wire format. SemIf is unchanged. Code written for the hosted model can now point at a local one.

Figures below were read from ollaya.dev, the GitHub REST API and the Hacker News API on 26 September 2026, and are stated as observed on that date.

What Ollaya is, and what it serves

Ollaya's README pitches it as a way to "Run open decision models locally, the way Ollama runs LLMs."⁴¹ It serves /v1/systemone and /v1/models "with TypeSafe's request and response shapes," and its site says "The official TypeSafe Python SDK 0.7.1 works unchanged against a local server."⁴² Its FAQ says it is "not affiliated with TypeSafe" and "not affiliated with Ollama."⁴³ The repo had 175 stars; the Hacker News launch thread stood at 331 points and 97 comments.⁴¹⁴⁴

It does not serve SemIf. The catalogue is other authors' open models: Convai Innovations' laya, Mapika's decider, Moritz Laurer's NLI classifiers, Knowledgator's gliclass, Qwen3Guard, Victor Hugo Panisa's von, and kev, the Jared Palmer project in the table above.⁴¹ One of them uses SemIf's mechanism: Ollaya describes decider as decoders on Qwen3.5 where "the answer is read from option-letter logits in one forward pass."⁴²

The speed claim is Ollaya's, and the comparison is loose

Ollaya's headline is "On your own GPU, a five-question request to Laya takes about 10 ms, end to end through the HTTP API," charted as 8–10 ms on an RTX 4090 against 236–276 ms for hosted Jev.⁴² The Laya figure is the vendor measuring its own tool; it was not re-run here.

The Jev figure is not Ollaya's measurement at all. The chart's own note calls it the hosted API's latency "in third-party benchmarks," one "which includes the network. Setups differ, so read it as an order-of-magnitude comparison."⁴² So the chart sets a local GPU with no network hop against a hosted service reached over the internet. It measures deployment, not models.

What the two independent benchmarks measured

Both repos Ollaya cites tested hosted Jev. Neither tested Ollaya, SemIf, or any model Ollaya serves.

• nibzard/decision-model-benchmark (created 18 September) ran Jev against eight constrained LLMs and three deterministic baselines, with "$28.34 of measured provider spend, every raw log published." Jev's p50 was 264–276 ms, and it placed mid-pack (76.3%) on a 77-way banking-intent suite: "No class wins on quality."⁴⁵
• The same benchmark tested honesty under uncertainty. Every LLM admitted ignorance on at least 97.3% of forced-uncertainty items, while "jev admits on 49.7%, with the worst calibration error measured (ECE 0.246; the LLMs range 0.039-0.122)."⁴⁵
• AbdelStark/jev-benchmarks (created 17 September) is a 300-example pilot of Jev against GLiNER2.5 running on a local CPU. Jev led on AG News (0.910 vs 0.700) and Banking77 (0.870 vs 0.610); on DAIR Emotion the gap was unresolved and Jev was "substantially worse calibrated." Hosted Jev ran 236–256 ms p50 on the 4- and 6-label tasks, which its author calls "operational measurements, not normalized model throughput."⁴⁶

The 236–276 ms in Ollaya's chart spans those two ranges: 264–276 ms from nibzard's, 236–256 ms from AbdelStark's. No independent number yet compares a locally served open model with Jev on quality, and Ollaya's own FAQ does not claim parity: "The small encoders, such as laya, are the fastest but fall well below Jev on harder questions; decider is more accurate and slower."⁴³ One Hacker News commenter reported the same from use: "From my experience, Laya performs significantly worse."⁴⁴

"Calibrated," as Ollaya uses the word

Ollaya's homepage promises "calibrated answers in milliseconds."⁴² Its FAQ explains what that means: "temperature scaling per question type and number of options, shipped with each model. For thresholds you rely on, refit the temperatures on your own labelled data and bake them in with a Modelfile."⁴³ That is the Step 6 remedy and SemIf's temperature layer, packaged. It restates this post's warning rather than overturning it: your own labelled rows set the threshold.

What this changes in the post above

• Nothing in SemIf. master is still at 23cf1f3; the mechanism, CLI contract and every table stand.
• "Jev, a closed hosted model" still holds, but its interface is no longer hosted-only. Open models can now be served locally behind the same request shape. On that path, Ollaya says "The network is used only to pull models"; on the hosted API, the text you score goes to TypeSafe.⁴³
• The hosted latency claim now has outside measurements. TypeSafe's advertised "70ms-500ms" range contains the two benchmarks' p50 figures.

Update — 29 September 2026

The pattern now has decision models small enough to train at home. Jeff, published on 28 September 2026, fine-tunes Qwen3.5 (0.8B and 2B) and Gemma 4 E2B to answer Jev-format requests in one forward pass. Its author trained the 0.8B in about two hours on one workstation GPU. Every Jeff benchmark figure below is the author's own.

Figures below were read from the Jeff README, the GitHub REST API, the Hugging Face API and the Hacker News API on 29 September 2026, and are stated as observed on that date.

What Jeff is, and how it differs from SemIf

Jeff's README describes "small, fast decision models you slot into your code, with the same request format as Jev," which return "a calibrated probability for each option from a single forward pass."⁴⁷ The code is MIT-licensed and the weights Apache 2.0. It began as a fork of AutoJev, an open recipe for a 27B Jev-style model, and states it "is not affiliated with or endorsed by TypeSafe."⁴⁷

The repo had 319 stars a day after creation; the Hacker News thread stood at 265 points and 106 comments.⁴⁸⁴⁹

The difference from SemIf is training. SemIf reads option-letter logits from a frozen model; Jeff runs "full-weight fine-tuning, one epoch," with "cross-entropy over the option letters."⁴⁷

The README says the work was done on "one RTX PRO 6000 workstation GPU (the 0.8B trains in about 2 hours, the 2B in about 3.5)," with synthetic data from an open model: "No cloud GPUs, and no closed-model output in the training data." A closed model spot-checked a sample.⁴⁷ It reports a median 22 ms per decision for the 0.8B on that GPU, 28 ms on an Apple M4 Max and 463 ms on a 32-thread CPU.⁴⁷

The benchmark comparison is the author's, on a different sample

The 83.1 against 83.0 is not a head-to-head. The README says "The published Jev and AutoJev figures were measured on a different sample of the same benchmarks," and that on "the reasoning-heavy benchmarks (BBH, JudgeBench, JevBench) it stays well below them."⁴⁷

Nobody has re-run these figures independently. One Hacker News commenter reported the opposite of parity on their own tasks: "70% vs 94% . for classification, it's unacceptable."⁴⁹ The README's remedy for that is a short fine-tune on your own examples, which by its own figures took a voice-navigation task "from 31.7% to 95.8%."⁴⁷

"Calibrated" still means one fitted temperature

Jeff's recipe ends with "one fitted temperature for calibration."⁴⁷ That is the mechanism SemIf added on 22 September and Ollaya ships per model. The temperature was fitted during the author's training run, not on your data. This post's warning still holds: refit it on your own labelled rows before a threshold triggers anything.

A first count of Jev in use

A preprint, "Jev in the Wild" (Ling, Xue and Ye, submitted 24 September 2026 and listed on Hugging Face Papers), analyses "2,170 publicly available Jev projects collected from GitHub as of September 22, 2026."⁵⁰ It finds that "Attribute judgment and scoring are widely used," and that "public attention is concentrated in routing and interface agents and does not track project counts."⁵⁰ It counts projects built around Jev, not reimplementations, and it has not been peer-reviewed.

What this changes in the post above

• Nothing in SemIf. master is still at 23cf1f3; the mechanism, CLI contract and every table stand.⁴⁸
• Training a small decision model now has a published budget. kev and NanoJev in the table above already ship training pipelines; Jeff states the hardware and the hours. SemIf remains the no-training path.
• Jeff's figures await replication. Like SemIf's quality tables, they come from the people who built the model.

Frequently Asked Questions

What is SemIf?

SemIf is an MIT-licensed research project that turns an open language model into a decision function. You pass unstructured state, a question and 2–16 described options; it runs one forward pass of a frozen model and softmaxes the option letters' logits into one score per option. Nothing is generated, so there is nothing to parse.¹

Is SemIf the same as Jev?

No. Jev is TypeSafe's closed hosted model. SemIf reproduces Jev's interface pattern with open models and says it "does not reproduce Jev's undisclosed model or training." It was formerly called OpenJev and is not affiliated with or endorsed by TypeSafe.¹

How fast is SemIf compared with generating JSON?

On the author's single RTX 3090, 21 yes/no decisions over one shared document took a median 1.023 s with direct logits, versus 5.332 s for the same Qwen3.5-4B model writing a compact JSON array. Those are the author's measurements; they were not re-run for this post.¹

Are SemIf's probabilities calibrated?

No. Direct, serial and shared rows are stamped "conditional option score; uncalibrated as decision confidence," and the repo lists calibrated probabilities among the things it has not established. Calibrate thresholds on your own labelled data before a score triggers an action.⁵ Since 22 September 2026 the repo ships an optional per-workload temperature-scaling layer that fits that calibration for you; it leaves the native scorer and every committed prediction unchanged, and it never moves the selected option.⁴⁰

Does SemIf need a GPU?

No longer. The torch backend needs exactly one visible CUDA GPU that can hold a 4B model in BF16, and Apple Silicon can use MLX or PyTorch/MPS — but since 22 September 2026 a llama.cpp backend scores the same prompts from a local GGUF on CPU only (--backend llamacpp).¹³⁹ The browser demo runs quantized Qwen3-0.6B, MiniCPM5-2B or Qwen3.5-4B locally over WebGPU.¹⁰

Has SemIf changed since this post was published?

Yes. As of 24 September 2026 the repository is renamed TheoLeeCJ/SemIf-OpenJev, master is at commit 23cf1f3, and it has added a CPU llama.cpp backend, PyTorch/MPS scoring, a Qwen3.8-27B EXL3 bridge, and per-workload temperature calibration. The mechanism, CLI contract, speed tables and quality tables are unchanged.³⁹

Can I run a Jev-style decision model locally?

Yes. SemIf runs on your own GPU, CPU or browser, and since 23 September 2026 Ollaya serves open decision models such as laya, decider and kev locally behind a TypeSafe-compatible API. Since 28 September 2026, Jeff's 0.8B and 2B models run on a GPU, a Mac or a CPU with the same request format. Ollaya's and Jeff's speed and accuracy figures are their own; independent benchmarks so far measured only hosted Jev, and Ollaya's FAQ says its fastest models fall well below Jev on harder questions.^4243474546

When should I not use SemIf?

Skip it when you need a trustworthy probability rather than a ranking, when option order or wording will change, when the decision needs step-by-step reasoning, when someone must read the why, when the output is open-ended, and whenever an irreversible action would fire without a human check.

Verification status

What was checked for this post, and against what. Nothing was executed; this is desk research against the repo at commit ca3ba65 and the public sources below, read on 21 September 2026.

The repo is days old (created 16 September 2026) and moving fast. Commands and numbers reflect master as read on 21 September 2026.

Second pass, 24 September 2026. The update section above was verified the same way — desk research, nothing executed. Repository names, creation dates, licences and star counts were read from the GitHub REST API; Hacker News scores, comment counts and submission timestamps from the Hacker News API, cross-checked against the Algolia search API; article quotes were fetched and matched character for character against the published pages.

Third pass, 26 September 2026. Same method, nothing executed. Ollaya's site, FAQ and README were fetched and every quote matched against the published text; repository metadata came from the GitHub REST API, and Hacker News figures from the Hacker News API, cross-checked against Algolia.

Fourth pass, 29 September 2026. Same method, nothing executed. Jeff's README and the preprint abstract were fetched and every quote matched against the published text; repository and model metadata came from the GitHub REST and Hugging Face APIs, and Hacker News figures from the Hacker News API, cross-checked against Algolia.