Jev vs. Generative Models for Typed Software Decisions

Abstract. A five-ticket, provider-direct experiment compares TypeSafe AI's Jev with DeepSeek Flash and GPT-5.6 Luna on bounded support-ticket decisions. Jev is fastest in the committed run, DeepSeek Flash is cheapest, and the systems disagree often enough that no quality ranking is justified. The useful result is architectural: typed evaluation, inexpensive structured generation, and general-purpose reasoning expose different interfaces and trade-offs. An agent runtime should route each bounded operation to the inference primitive that meets its quality, latency, cost, and assurance requirements.

Bounded software decisions are routing points in a larger system, not automatically text-generation requests. Classifying a request, deciding whether to retry, scoring risk, and choosing a queue may require generated interpretation, but they may also be expressed as typed questions with explicit policies. The choice of inference primitive affects latency, cost, uncertainty handling, and the controls available to the application.

The comparison asks a systems question: how should an agent runtime route different cognitive operations to deterministic code, typed evaluation, structured generation, deeper reasoning, or human review?

The companion experiment is a small probe of that routing question, not a quality benchmark. Its committed run measures one five-ticket workload, one sequential execution, and three different provider paths. The result supports an architectural hypothesis, not a universal model ranking.

The inference primitives

General-purpose language models are often used for every bounded decision. Structured output makes that pattern practical: GPT-5.6 Luna supports structured outputs and can run with reasoning explicitly disabled.

Generation is not the only available primitive. TypeSafe AI's Jev accepts state and declared typed questions, returning Choice, Score, and Boolean answers with probabilities. Vercel describes those questions as being evaluated in parallel rather than produced through ordinary autoregressive text generation, while TypeSafe provides the underlying description of Jev and System One models.

The distinction changes the application contract. A generative path asks a model to interpret a rubric and produce a schema. A typed-evaluation path declares bounded questions and receives typed results, including an uncertainty signal that the application may use in policy. Neither interface is automatically more accurate; each exposes different failure modes and control points.

Comparison design

The committed run used three reasoning-off configurations:

Arm	Provider path	Model	Reasoning
Jev	Vercel AI Gateway evaluation protocol	typesafe-ai/jev	not applicable; evaluation model
DeepSeek Flash	DeepSeek direct	deepseek-flash	explicitly disabled
GPT-5.6 Luna	OpenAI direct	gpt-5.6-luna	explicitly none

DeepSeek's September 10, 2026 release notes identify V4.1 Flash and the deepseek-flash identifier as the current Flash path. The comparison uses that current first-party path directly. The retired deepseek-v4-flash identifier is not the current comparison model; retaining the model ID matters because aliases and provider paths are experimental variables.

The task contains five multi-message support threads. Each system returns five bounded decisions:

• department: billing, tech, or sales;
• urgent: boolean;
• severity: integer from 0 to 4;
• requires escalation: boolean; and
• estimated effort: categorical duration.

The two generative models receive the same canonical thread text and objective rubric. Luna uses strict JSON-schema structured output; DeepSeek Flash uses JSON-object structured output. Jev receives equivalent questions through its typed evaluation protocol.

Capability probes verified model IDs, reasoning-off behavior, structured-output support, usage accounting, and Jev reachability before measurement. The run has no retries. Fifteen calls ran sequentially in ticket order: each ticket was sent to OpenAI direct, DeepSeek direct, and then Jev through Vercel AI Gateway. OpenAI and DeepSeek therefore have first-party paths, while Jev has a Vercel path. These are useful controls, but not a perfectly controlled provider benchmark.

Measured result

The committed Markdown report and machine-readable JSON report record the run at 2026-09-20T12:23:23.033Z:

Configuration	Five-call latency	Computed cost
Jev	1,738 ms	$0.00043785
DeepSeek Flash	4,565 ms	$0.00037669
GPT-5.6 Luna	5,859 ms	$0.00073720

Two different winners appear in this run. Jev is approximately 3.4× faster than Luna and 2.6× faster than DeepSeek Flash. DeepSeek Flash is the least expensive. Jev is approximately 41% cheaper than Luna but 16% more expensive than DeepSeek Flash.

Jev also consumed more reported input tokens: 10,425, compared with 2,149 for DeepSeek Flash and 2,456 for Luna. The higher input-token count did not prevent the lower observed wall-clock time in this run. That observation is consistent with TypeSafe's description of Jev as a model designed for typed probabilistic decisions, but the experiment does not establish why the latency differed.

TypeSafe reports much larger results in its own workflow evaluations: up to 193.6x faster and 444.6x cheaper. Those are vendor-produced figures, not independent measurements. TypeSafe says they are toward the high end of expected real-workload gains and acknowledges possible workflow-construction bias. They should be treated as vendor evidence rather than as a result of this comparison. See TypeSafe's report for the source context.

These measurements establish interface trade-offs, not quality. The narrow measurement claim is therefore: Jev occupied a distinct latency regime in this sequential run, while cost depended on the competing model and provider pricing. Neither result generalizes to all tasks or deployments.

Agreement is not correctness

The timing and cost measurements establish trade-offs between interfaces; agreement asks a separate question. The experiment has no labeled ground-truth dataset, so it measures agreement among systems, not accuracy.

All three systems produced identical complete decisions on 2 of 5 tickets. Pairwise field agreement was:

Field	Agreements
urgency	15 / 15
department	13 / 15
escalation	13 / 15
effort	13 / 15
severity	11 / 15

The disagreements are useful because they expose policy ambiguity that a structured schema can hide.

Mixed issues: what does “department” mean?

The first thread combines an incorrect renewal charge with users being locked out of their workspace. Luna and DeepSeek route it to billing; Jev routes it to tech. All three agree that it is urgent, severity 4, requires escalation, and likely takes one to four hours.

Without an explicit routing policy, neither department choice is obviously wrong. Routing might mean root cause, the most urgent issue, the first-response owner, the primary customer request, or the eventual resolution owner. A production benchmark must choose one. Otherwise, policy ambiguity is misread as model error.

Severity boundaries need a rubric

On the production-webhook outage, DeepSeek and Jev assign severity 4; Luna assigns 3, while the other fields agree. A labeled benchmark needs an explicit boundary between high and critical severity before this disagreement can be interpreted as a quality difference.

Escalation exposes policy thresholds

The fifth case is an enterprise pre-sales request involving pricing, SAML, SCIM, a DPA, subprocessors, data residency, audit-log retention, and possible custom deployment. All three systems agree that it belongs to sales and is not urgent, but they differ on the remaining fields:

• Luna: severity 0, no escalation, 1–4 hours;
• DeepSeek Flash: severity 1, escalate, 1–2 days; and
• Jev: severity 0, escalate, 1–4 hours.

Jev returns a 0.61 probability for escalation. At a threshold of 0.5, the application escalates; at 0.7, it does not. That is an interface property, not evidence of calibration. Vercel documents Jev's typed answers and probabilities, but an application still has to validate whether those probabilities support its policy.

One possible policy is:

p(escalate) >= 0.9       -> escalate automatically
0.4 <= p(escalate) < 0.9 -> use a second evaluator or human review
p(escalate) < 0.4       -> continue automatically

The evaluator exposes a quantity that policy can use. The workflow defines the acceptable risk.

Calibration is a measurement problem

Emitting probabilities does not make Jev calibrated. A model that emits 0.8 should be correct about 80% of the time on comparable examples; that property must be measured against labels. Calibration is separate from classification accuracy. Guo et al.'s study of neural-network calibration provides a useful foundation for the distinction.

A serious evaluation of a typed evaluator should report:

• Brier score;
• log loss;
• expected calibration error;
• reliability curves;
• precision and recall at policy thresholds;
• autonomous-action rate;
• human-review rate; and
• failure cost at each threshold.

Those measurements would turn probability outputs into operational evidence. Until then, they are an uncertainty representation with unverified reliability. Calibration must be measured for the task and thresholds that will govern production actions.

From structured output to typed evaluation

Once uncertainty is treated as a measurement problem, the contract difference becomes architectural. The two approaches both return structured data, but they do not expose the same contract.

flowchart TD
    G1["state"] --> G2["prompt"] --> G3["generated structured output"] --> G4["parse"] --> G5["validate"] --> G6["decision"]
    T1["state × typed questions"] --> T2["probabilities / choices / scores"] --> T3["decision"]

The first path is flexible: the model can interpret a broad rubric and fill a schema. The second makes the bounded questions and their result types explicit. That difference affects uncertainty handling, threshold control, schema-failure modes, batching of related judgments, observability, calibration, retries, and escalation policy.

The distinction does not make Jev automatically better. DeepSeek Flash is cheaper in the committed run, and a generative model may be the simpler choice when flexible structured interpretation is the requirement. Typed evaluation deserves separate evaluation because its programming contract differs from “ask a language model for JSON.”

From model routing to inference routing

The usual economics question is: which model should handle this request? FrugalGPT shows how model cascades can reduce inference cost while preserving useful performance. RouteLLM treats routing as a learned choice between stronger and weaker models under quality-cost trade-offs.

The broader question is: what kind of computation should handle this step? An agent runtime may need an inference graph rather than a single-model agent:

flowchart TD
    B["bounded operation"] --> C["deterministic code"]
    B --> Q["database queries"]
    B --> S["search and retrieval"]
    B --> T["typed probabilistic evaluation"]
    B --> L["lightweight structured generation"]
    B --> E["cheap structured extraction"]
    B --> R["deep reasoning for genuine ambiguity"]
    B --> G["open-ended generation"]
    B --> H["human review above a defined risk threshold"]
    E --> T --> V["deterministic validation"] --> R --> H

One proposed hybrid for this task is:

1. DeepSeek Flash extracts structured information cheaply.
2. Jev evaluates a high-frequency bounded policy decision with low latency.
3. Deterministic code validates hard constraints.
4. GPT-5.6 Luna handles an ambiguous case requiring broader reasoning.
5. A human reviews decisions above a defined risk threshold.

The routing question is no longer Luna or DeepSeek? It is code, retrieval, evaluator, cheap model, reasoning model, or human? The experiment therefore treats model selection as one local decision inside a broader inference architecture.

This routing design remains a hypothesis. The next measurement must test whether the added stages improve assurance enough to justify their cost and latency.

Experimental limits and next measurement

Provider path is part of the result. OpenAI and DeepSeek ran directly against their first-party APIs; Jev ran through Vercel AI Gateway because that is how the evaluation-model path was accessed. Network paths and provider infrastructure differ. Calls were sequential, and a single run captures transient load. The reported wall-clock latency is an observed measurement, not a controlled benchmark.

The run also has no retries, uses only five tickets, and has no ground-truth labels. One sequential run cannot establish a general quality or latency ranking. Outputs and wall-clock latency can vary between runs, and the cost ratios may not generalize to other tasks.

The next experiment should prioritize ground truth and repeated measurement rather than adding another model. A credible benchmark would use 100–500 independently labeled support threads with explicit routing policies, adjudicated severity labels, escalation rules, effort labels, intentionally ambiguous cases, multi-issue threads, repeated calls per model, and randomized execution order.

It should report accuracy and macro-F1, confusion matrices, Brier score and log loss for probabilistic outputs, calibration curves, p50/p95/p99 latency, cost per correct decision, cost per autonomous decision, human escalation rate, and failure rate at each confidence threshold. A hybrid policy could then test:

Jev confidence high         -> act immediately
Jev confidence intermediate -> ask DeepSeek Flash
models disagree            -> ask Luna or a human

That design would measure which inference architecture reaches the required assurance level at the lowest total cost and latency.

Architectural conclusion

The committed run does not show that Jev is more accurate. It does not show that DeepSeek Flash is more accurate. It does not establish a universal latency ordering or general cost ratios. It does show that typed evaluation should be treated as a separate inference primitive rather than merely another model behind a prompt.

The practical design principle is conditional: use deterministic software when the rule is known; use typed evaluators when the output is bounded and measured uncertainty is operationally useful; use cheap generative models when flexible structured interpretation is needed; and use deeper reasoning when the task actually requires it.

An agent runtime built around these distinctions can route each operation to the cheapest and fastest primitive that satisfies its quality and assurance requirements. The result is a systems direction: model ranking is only one local decision inside it.

Practical Takeaways

1. Record model ID, provider path, timestamp, capability probes, and pricing provenance with every comparison. Rolling aliases and intermediary paths are experimental variables.
2. Treat agreement as a diagnostic until the task has labeled ground truth and explicit policies for routing, severity, escalation, and effort.
3. Do not treat returned probabilities as calibrated confidence. Measure calibration and failure cost at the thresholds that will drive production actions.
4. Keep inference routing broader than model routing: deterministic code, retrieval, typed evaluation, structured generation, reasoning, and human review should be available as separate primitives.
5. Use this five-ticket result to form a routing hypothesis, not to select a universal winner.

Positioning Note

This is an applied systems note, not an academic benchmark or a vendor evaluation. It records one committed experiment, separates its observations from vendor claims and architectural inference, and proposes the measurements needed for a stronger study. It is also not vendor documentation: the primary sources describe product behavior, while the companion report supplies the measurements and this note interprets their architectural significance.

Status & Scope

This note is exploratory personal lab work and is non-authoritative. It describes five support tickets in one sequential run with no ground-truth labels and no retries. Those conditions do not establish general quality, latency, or cost rankings. Provider paths differ, and the vendor figures are vendor evidence subject to workflow-construction bias, not independent measurements. Treat the result as a dated artifact and a design prompt; validate the relevant task, provider paths, pricing, calibration, and failure costs before using it to govern production actions.

References

1. rMax.ai, ai-provider-triage-comparison, provider-direct reasoning-off support-ticket experiment, September 20, 2026.
2. rMax.ai, committed comparison report.
3. rMax.ai, machine-readable comparison report.
4. Vercel, “TypeSafe AI's Jev now available on AI Gateway”, September 16, 2026.
5. TypeSafe AI, “Introducing System One Models & Jev”, September 15, 2026.
6. Vercel AI Gateway, “Jev — API, Pricing & Playground”.
7. OpenAI, “GPT-5.6 Luna Model”.
8. DeepSeek, “Change Log: V4.1 Flash Release”, September 10, 2026.
9. DeepSeek, “V4.1 Flash: Smarter, Faster, More Efficient”.
10. Guo et al., “On Calibration of Modern Neural Networks”, ICML 2017.
11. Chen, Zaharia & Zou, “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance”.
12. Ong et al., “RouteLLM: Learning to Route LLMs with Preference Data”.