> ## Content Index
> Fetch the complete content index at: https://vill.ar/llms.txt
> Use this file to discover other available public pages before exploring further.

# The AI model you need isn't always generative:  using a System One model in our projects
- URL: https://vill.ar/the-ai-you-need-doesnt-always-need-to-be-generative-implementing-a-system-one-model-in-our-projects/
- Published: 2026-10-22T01:46:06.000Z
- Updated: 2026-10-22T14:40:17.000Z
- Author: Mike Villar

I used to reach for a language model every time a product needed a bit of judgement. These days I mostly don't.

We run three AI-heavy projects: an internal content hub for an agency team, an AI assistant that reads a multitude of signals and decides how to act, and a prototype of AI-assisted search for an E-commerce store. All three are full of judgement calls, and almost none of those calls need a line of code or paragraph of prose. They need an answer: yes or no, this one or that one, how sure are you?

Early on we sent those questions to a large language model, because that's what was on hand. It worked. It was also slow, expensive and inconsistent. So we moved most of them to [**Jev**, a System One model from TypeSafe](https://typesafe.ai/?ref=vill.ar). This post covers what that means and where we use it. It also covers what we measured when we put Jev head to head with Claude Haiku, and the rule that makes it safe to depend on.

## What a System One model is

A language model writes. A System One model **decides**.

You send Jev a piece of state (an email, a draft, a search query) and a set of typed questions. It returns calibrated probabilities for all of them in a single call, in roughly 300 milliseconds. It never generates text. There are three question types:

- **noul**: a yes/no, returned as a probability from 0 to 1\. "Is this email asking me to do something?"
- **choice**: pick one of several options you supply, with the full probability distribution and a confidence. "Is this idea new, a partial overlap, or already covered?"
- **score**: an ordered rating.

Every question in a call runs in parallel against the same state. Fifty questions cost about the same wall-clock time as one.

The probabilities are what make it useful. A probability isn't a verdict. It's something you can put a threshold on, and thresholds are where the engineering happens.

The corollary is that Jev can't invent anything. It can only choose among options your code puts in front of it. That sounds like a limitation, and it is one. It also turns out to be a very good division of labor: code proposes candidates, Jev chooses, and anything that needs wording or arithmetic stays in code.

## The AI assistant: a bouncer in front of the expensive models

![](https://vill.ar/content/images/2026/10/image-7.png)

The assistant has plenty of ways to spend money and time: bigger models, live Gmail and Calendar context, a dozen tool groups. Jev decides how much each request deserves.

**Chat routing.** Every message gets one Jev call: which model tier, what kind of task, and which context and tools it needs. A simple question no longer pays for a heavyweight setup. We don't trust Jev's top pick, though. It is often honestly split between "easy" and "medium", and a `choice` flattens that into one label. So we read the distribution and apply our own cutoffs: the hard tier wins at 50% or more, the cheap tier at 60% or more, and anything between goes to the middle.

**The Today screen.** Before an email becomes a task, Jev asks whether the latest message is actually asking me to act (given To/Cc and whether I'm only copied) and whether it's automated. One production sweep screened 24 threads in about a second and kept 5\. The rest never reached a model.

**Skipping work that would be thrown away.** Several jobs used to make an LLM call whose usual answer was "nothing to do", such as fact extraction after every chat turn. Jev now asks first. The same pattern covers attachment triage, sender checks and bill detection. A bill is auto-marked paid only when Jev is confident **and** amount, card digits and date check out in plain code.

**Chat chips.** Quick-action buttons used to appear on 98% of turns, mostly noise. Now Jev answers one yes/no per chip kind and a model writes values only for those that pass. In a replay of 40 production turns, 65% now get no chips and none show fragments. "None" is the right answer most of the time, and a probability makes it easy to say.

## The content hub: judging relevance at scale

![](https://vill.ar/content/images/2026/10/image-4.png)

The content hub helps a team write articles that fit a client's existing site, which turns out to be a pile of pairwise judgements.

- **Retrieval.** Jev asks of each retrieved passage whether it would help write this article. Passages under 0.5 are dropped.
- **Internal links.** Jev scores each sentence-and-published-article pair: a natural link, or a distraction? One request scores the whole batch.
- **Overlap check.** Search can tell you two pieces share words. Jev judges whether a draft covers the same topic *and angle* as a published article. At 70 or more it reports "overlaps", and from 45 to 69 "partly overlaps".
- **Plan dedupe.** For up to 200 AI-proposed ideas, each gets a `choice` of new, overlap or covered, with criteria written in terms of search intent rather than topic.

## The head-to-head: Jev vs Haiku on product search

![](https://vill.ar/content/images/2026/10/image-3.png)

The assistant and the content hub use Jev to save money and time. The E-Commerce search prototype is where we tested whether it could replace an LLM outright.

The problem is familiar to anyone who runs any E-commerce store. Shoppers type `99 ford bronco oil filter` or `f150 2019 floor mats`. Magento's native search matches keywords with no idea what vehicle you drive, so in our tests **72% of the parts in the top ten didn't fit the vehicle that was typed**. The fix is to understand the query first (year, vehicle, part type), filter by fitment, then rank.

We built that as a four-stage pipeline and ran three engines through the same retriever, so only the "understand the query" step differed:

1. **Normalize (code, about 1 ms).** Pull out years, candidate vehicles, a shortlist of part types, SKUs, with typo tolerance.
2. **Interpret (model).** Either Claude Haiku with a structured-output call, or Jev with parallel Choice and Noul questions.
3. **Retrieve (code, about 5 ms).** Text search intersected with a hard fitment filter, and a relaxation ladder for vehicles the catalog doesn't cover.
4. **Re-rank (optional).** Haiku orders a list; Jev answers one yes/no per candidate product, in the same call.

We tested on 300 searches generated from the catalog: shorthand years, missing makes, reordered words, aliases, injected typos, synonyms. Labels came from the fitment data rather than from any model, so no LLM judged its own homework.

![](https://vill.ar/content/images/2026/10/image-5.png)

Accuracy was a tie at 81%. Jev was **4.7× faster and 3.4× cheaper**. Its query-understanding score was higher too (92% against 89%), and Jev put fewer non-fitting parts on the page (2% against 3%).

Two of those numbers need context.

**Cost counts every call.** Type-ahead means each search triggers around four extra lookups as the shopper types, so we priced the whole sequence rather than the final query alone. That is about 4.8 model calls per search. Jev's output tokens are free, which helps more than you'd expect: Haiku's output was 31% of its bill.

**The rules-only row is the honest baseline.** Plain code with no model at all reached 79%. The AI's whole contribution is the last couple of points, which is mostly typos ("escaladde", "challenegr") and unlisted synonyms ("grill", "trunk mat"). About 80% of test searches are unambiguous enough to skip the model entirely. That is the biggest single saving in the design, and it's the same principle as the assistant: the cheapest model call is the one you don't make.

### Jev started out behind

The first version of the Jev interpreter scored 78% against Haiku's 81%. We could have concluded that the fast model was a bit worse and moved on. The gap turned out to be question design, and fixing it taught us the most about how to use a System One model:

- **Ask one yes/no per option when more than one answer can be right.** "Floor mats" means two part types (all-weather and carpeted). A single `choice` can only return one, and it tended to pick the literal word "Mat". Up to 15 separate noul questions fixed that, all in the same call.
- **Keep arithmetic in code.** Jev reads dates as text. For `'10 ram 1500` it picked a truck generation that didn't exist until 2019\. Now code drops vehicle generations that don't cover the typed year before Jev ever sees the options.
- **Give Jev the options a human would consider.** A typo like "ratpor" produced no Raptor option, so Jev couldn't choose it. Offering sibling trims and fuzzy matches fixed it.
- **Don't take literal yeses to broad categories.** "Plug" scored high for spark plugs, and also for drain plugs. When a more specific type is chosen, code drops the broader one.
- **Let code write the text.** Jev can't produce search keywords. They're rebuilt in code after Jev picks the vehicle.

We also tried adding explicit true/false definitions to the yes/no questions, which is standard prompt-writing instinct. It made Jev stricter, dropped understanding from 96% to 91%, and cost 40% more. We reverted it. Writing for a System One model is not the same craft as writing for an LLM.

### What our data says about trusting confidence

Because the output is a probability, we could check whether it was honest. We bucketed Jev's reported confidence against how often it was actually right:

![](https://vill.ar/content/images/2026/10/image-6.png)

High confidence is dependable and low confidence is a real warning. So low-confidence searches can be routed to a slower path or answered with a "did you mean". Haiku also reports confidence, but its 0.70–0.85 bucket was right only 54% of the time, which makes it a much weaker signal. Jev's confidence is useful as a routing tool, and an LLM's self-reported confidence usually isn't.

### Where Jev did not win

Our latency budget was 300 ms end to end. Jev's p95 was 346 ms, so it **missed**. Haiku's was over two seconds. But a bare HTTPS round trip from our dev machine to either API costs about 220 ms before any model work happens. Subtract that and Jev's model-side time is around 45 ms against Haiku's roughly 1,060 ms. Host the service next to the API and Jev fits the budget. Haiku never does.

Jev also trailed on SKU lookups and part-first phrasing, though on small samples. It's strongest at picking from a defined set and weakest where free text needs interpreting. And this is a prototype on generated queries. The next step is to replay the store's own search log.

## The rule that keeps it safe

Every Jev call in all three systems follows the same rule: **Jev is an optimisation, never a dependency.**

- No key, a timeout, a rate limit or a malformed answer all raise one error, and every call site handles it.
- The assistant gives Jev a hard one-second budget on the chat path, because a slow answer is as good as no answer. When it fails, it falls back to the older classifier and offers every tool. Routing can also be switched to `llm` or `heuristic` with a single setting and no deploy.
- The content hub falls back to search ranking, word matches and approximate scores, and labels them as approximate in the UI.
- The search prototype falls back to plain native search. We were deliberate that Jev never falls back to Haiku, and Haiku never to Jev, so a failure shows up honestly in the results rather than being quietly papered over.
- Jev never surfaces an error to a user.

That rule is why we were comfortable putting a third-party model on the critical path. The worst case is that the product gets slower and a bit less smart. It never breaks.

## What we learned

1. **Use probabilities, not labels.** Thresholds encode your product's tolerance: recall-biased for what to load, strict for what to automate.
2. **Cheap first, expensive for survivors.** The biggest wins come from skipping work. In search, 80% of queries need no model at all.
3. **Code has the last word on anything irreversible or numeric.**
4. **Code proposes, Jev chooses.** Results improved when we sharpened the option list, never when we asked Jev to work harder.
5. **Multi-way choice is the weakest answer type.** Split it into yes/no questions where you can, and set a high confidence floor where you can't.
6. **The question text is the prompt, so test it.** Extra definitions made ours worse.
7. **Measure against a no-AI baseline.** The rules-only row changed how we thought about the whole project.

Language models are great at writing stuff (code, prose, etc.). A lot of what products need isn't writing, though. It's a fast, consistent, cheap opinion, with an honest measure of how sure it is. That's what a System One model is for.