10 · Use cases 9 & 10: the RAG relevance filter and the citation checker¶
Retrieval-augmented generation has two classic failures:
- Retrieval noise. The retriever returns 6 passages and only 1 is useful. The LLM gets confused, or uses the wrong one.
- Unsupported claims. The answer cites a passage that doesn't actually say that.
Jev is a good fit for both, because each is a yes/no judgement about a pair of texts:
question -> retrieve top-k (code) -> (9) Jev: "does passage i help answer this?" -> keep only the relevant ones
|
LLM answers from the kept passages, citing [ids]
|
(10) Jev: "is this claim supported by the passage it cites?" -> unsupported? -> regenerate
In [1]:
Copied!
# --- Provider config: ONE OpenRouter key drives both brains ------------------------------
# slow brain (LLM) -> OpenAI SDK -> https://openrouter.ai/api/v1 (chat completions + tools)
# fast brain (Jev) -> TypeSafe SDK -> https://openrouter.ai/api (POST /v1/systemone)
import os, json, time
# Load settings from a .env file if present (falls back to existing env vars).
try:
from dotenv import load_dotenv, find_dotenv
load_dotenv(find_dotenv(usecwd=True))
except Exception:
if os.path.exists(".env"):
for _line in open(".env"):
_line = _line.strip()
if _line and not _line.startswith("#") and "=" in _line:
_k, _v = _line.split("=", 1)
os.environ.setdefault(_k.strip(), _v.strip())
BASE_URL = os.environ.get("OPENAI_BASE_URL", "https://openrouter.ai/api/v1")
API_KEY = os.environ.get("OPENAI_API_KEY", "set-me")
MODEL = os.environ.get("MODEL", "openai/gpt-6-luna") # slow brain: any tool-capable chat model
JEV_BACKEND = os.environ.get("JEV_BACKEND", "typesafe").strip().lower() # typesafe | adapter
JEV_MODEL = os.environ.get("JEV_MODEL", "~typesafe/jev-latest")
JEV_BASE_URL = os.environ.get("TYPESAFE_BASE_URL", "https://openrouter.ai/api")
JEV_API_KEY = os.environ.get("TYPESAFE_API_KEY") or API_KEY # one key for both brains
# Behind a TLS-intercepting firewall/proxy, set VERIFY_SSL=false in .env (trusted networks only).
import httpx, httpx2
VERIFY_SSL = os.environ.get("VERIFY_SSL", "true").strip().lower() not in ("false", "0", "no")
if not VERIFY_SSL:
import warnings
warnings.filterwarnings("ignore")
print("\u26a0\ufe0f SSL verification DISABLED (VERIFY_SSL=false) \u2014 use only on a trusted network")
from openai import OpenAI
client = OpenAI(base_url=BASE_URL, api_key=API_KEY, http_client=httpx.Client(verify=VERIFY_SSL))
if JEV_BACKEND == "adapter":
# No Jev access yet? Same System One API, answered by your LLM (slower, pricier, but it runs).
from system_one_adapter import SystemOneAdapterClient, Noul, Choice, Score
from system_one_adapter.providers.openai import OpenAIProvider
jev = SystemOneAdapterClient(
structured_outputs=True, llm_answer_mode="probabilities", normalize_probabilities=True,
n_retry_malformed_structure=2,
model=OpenAIProvider(MODEL, base_url=BASE_URL, api_key=API_KEY, api="chat_completions"))
else:
from typesafe_sdk import TypeSafeClient, Noul, Choice, Score
jev = TypeSafeClient(api_key=JEV_API_KEY, base_url=JEV_BASE_URL, model=JEV_MODEL,
http_client=httpx2.Client(verify=VERIFY_SSL))
print(f"slow brain (LLM): {MODEL} @ {BASE_URL}")
print(f"fast brain (Jev): {JEV_MODEL if JEV_BACKEND != 'adapter' else MODEL + ' via adapter'} @ {JEV_BASE_URL if JEV_BACKEND != 'adapter' else BASE_URL}")
# --- Provider config: ONE OpenRouter key drives both brains ------------------------------
# slow brain (LLM) -> OpenAI SDK -> https://openrouter.ai/api/v1 (chat completions + tools)
# fast brain (Jev) -> TypeSafe SDK -> https://openrouter.ai/api (POST /v1/systemone)
import os, json, time
# Load settings from a .env file if present (falls back to existing env vars).
try:
from dotenv import load_dotenv, find_dotenv
load_dotenv(find_dotenv(usecwd=True))
except Exception:
if os.path.exists(".env"):
for _line in open(".env"):
_line = _line.strip()
if _line and not _line.startswith("#") and "=" in _line:
_k, _v = _line.split("=", 1)
os.environ.setdefault(_k.strip(), _v.strip())
BASE_URL = os.environ.get("OPENAI_BASE_URL", "https://openrouter.ai/api/v1")
API_KEY = os.environ.get("OPENAI_API_KEY", "set-me")
MODEL = os.environ.get("MODEL", "openai/gpt-6-luna") # slow brain: any tool-capable chat model
JEV_BACKEND = os.environ.get("JEV_BACKEND", "typesafe").strip().lower() # typesafe | adapter
JEV_MODEL = os.environ.get("JEV_MODEL", "~typesafe/jev-latest")
JEV_BASE_URL = os.environ.get("TYPESAFE_BASE_URL", "https://openrouter.ai/api")
JEV_API_KEY = os.environ.get("TYPESAFE_API_KEY") or API_KEY # one key for both brains
# Behind a TLS-intercepting firewall/proxy, set VERIFY_SSL=false in .env (trusted networks only).
import httpx, httpx2
VERIFY_SSL = os.environ.get("VERIFY_SSL", "true").strip().lower() not in ("false", "0", "no")
if not VERIFY_SSL:
import warnings
warnings.filterwarnings("ignore")
print("\u26a0\ufe0f SSL verification DISABLED (VERIFY_SSL=false) \u2014 use only on a trusted network")
from openai import OpenAI
client = OpenAI(base_url=BASE_URL, api_key=API_KEY, http_client=httpx.Client(verify=VERIFY_SSL))
if JEV_BACKEND == "adapter":
# No Jev access yet? Same System One API, answered by your LLM (slower, pricier, but it runs).
from system_one_adapter import SystemOneAdapterClient, Noul, Choice, Score
from system_one_adapter.providers.openai import OpenAIProvider
jev = SystemOneAdapterClient(
structured_outputs=True, llm_answer_mode="probabilities", normalize_probabilities=True,
n_retry_malformed_structure=2,
model=OpenAIProvider(MODEL, base_url=BASE_URL, api_key=API_KEY, api="chat_completions"))
else:
from typesafe_sdk import TypeSafeClient, Noul, Choice, Score
jev = TypeSafeClient(api_key=JEV_API_KEY, base_url=JEV_BASE_URL, model=JEV_MODEL,
http_client=httpx2.Client(verify=VERIFY_SSL))
print(f"slow brain (LLM): {MODEL} @ {BASE_URL}")
print(f"fast brain (Jev): {JEV_MODEL if JEV_BACKEND != 'adapter' else MODEL + ' via adapter'} @ {JEV_BASE_URL if JEV_BACKEND != 'adapter' else BASE_URL}")
slow brain (LLM): openai/gpt-6-luna @ https://openrouter.ai/api/v1 fast brain (Jev): ~typesafe/jev-latest @ https://openrouter.ai/api
In [2]:
Copied!
# --- Helpers used in every notebook: ask the fast brain, ask the slow brain, track spend ---
class Spend:
"""Running tally of what OpenRouter charged, per brain (it reports exact USD per call)."""
def __init__(self):
self.usd, self.calls = {"jev": 0.0, "llm": 0.0}, {"jev": 0, "llm": 0}
def add(self, brain, resp):
self.calls[brain] += 1
try: cost = resp.raw_http_response.json()["usage"].get("cost") # Jev response
except Exception: cost = getattr(getattr(resp, "usage", None), "cost", None) # LLM response
self.usd[brain] += cost or 0.0
def __repr__(self):
return (f"Jev: {self.calls['jev']} calls ${self.usd['jev']:.5f} | "
f"LLM: {self.calls['llm']} calls ${self.usd['llm']:.5f}")
SPEND = Spend()
def ask_jev(state, questions):
"""Fast brain. state (text or JSON) + typed questions -> typed, calibrated answers."""
r = jev.system_one(state, questions)
SPEND.add("jev", r)
return r
def chat(prompt, system="Be concise.", model=None):
"""Slow brain, single shot (no tools). Returns the reply text."""
resp = client.chat.completions.create(model=model or MODEL, messages=[
{"role": "system", "content": system}, {"role": "user", "content": prompt}])
SPEND.add("llm", resp)
return resp.choices[0].message.content
def show(r):
"""Print every answer in a Jev response on one line each."""
for k, a in r.answers.items():
if a.type == "noul":
print(f" {k:<18} noul P(yes)={a.noul:.2f}")
elif a.type == "choice":
top = sorted(a.probabilities.items(), key=lambda kv: -kv[1])[:3]
print(f" {k:<18} choice {a.choice!r:<22} conf={a.confidence:.2f} top={top}")
else:
level = {int(i): v for i, v in (a.legend or {}).items()}.get(round(a.score), "")
print(f" {k:<18} score {a.score:.2f} -> {level!r:<18} conf={a.confidence:.2f}")
# --- Helpers used in every notebook: ask the fast brain, ask the slow brain, track spend ---
class Spend:
"""Running tally of what OpenRouter charged, per brain (it reports exact USD per call)."""
def __init__(self):
self.usd, self.calls = {"jev": 0.0, "llm": 0.0}, {"jev": 0, "llm": 0}
def add(self, brain, resp):
self.calls[brain] += 1
try: cost = resp.raw_http_response.json()["usage"].get("cost") # Jev response
except Exception: cost = getattr(getattr(resp, "usage", None), "cost", None) # LLM response
self.usd[brain] += cost or 0.0
def __repr__(self):
return (f"Jev: {self.calls['jev']} calls ${self.usd['jev']:.5f} | "
f"LLM: {self.calls['llm']} calls ${self.usd['llm']:.5f}")
SPEND = Spend()
def ask_jev(state, questions):
"""Fast brain. state (text or JSON) + typed questions -> typed, calibrated answers."""
r = jev.system_one(state, questions)
SPEND.add("jev", r)
return r
def chat(prompt, system="Be concise.", model=None):
"""Slow brain, single shot (no tools). Returns the reply text."""
resp = client.chat.completions.create(model=model or MODEL, messages=[
{"role": "system", "content": system}, {"role": "user", "content": prompt}])
SPEND.add("llm", resp)
return resp.choices[0].message.content
def show(r):
"""Print every answer in a Jev response on one line each."""
for k, a in r.answers.items():
if a.type == "noul":
print(f" {k:<18} noul P(yes)={a.noul:.2f}")
elif a.type == "choice":
top = sorted(a.probabilities.items(), key=lambda kv: -kv[1])[:3]
print(f" {k:<18} choice {a.choice!r:<22} conf={a.confidence:.2f} top={top}")
else:
level = {int(i): v for i, v in (a.legend or {}).items()}.get(round(a.score), "")
print(f" {k:<18} score {a.score:.2f} -> {level!r:<18} conf={a.confidence:.2f}")
In [3]:
Copied!
from concurrent.futures import ThreadPoolExecutor
def jev_map(fn, items, workers=8):
"""Run fn over items in parallel threads. Jev allows 1,200 requests/min, so 8 workers is safe."""
with ThreadPoolExecutor(max_workers=workers) as pool:
return list(pool.map(fn, items))
from concurrent.futures import ThreadPoolExecutor
def jev_map(fn, items, workers=8):
"""Run fn over items in parallel threads. Jev allows 1,200 requests/min, so 8 workers is safe."""
with ThreadPoolExecutor(max_workers=workers) as pool:
return list(pool.map(fn, items))
1. The knowledge base and a deliberately simple retriever¶
In [4]:
Copied!
import re, math
from pathlib import Path
from collections import Counter
def load_passages(kb_dir="data/kb"):
"""Split every KB markdown file into one passage per '## ' section."""
out = []
for f in sorted(Path(kb_dir).glob("*.md")):
title = f.read_text().splitlines()[0].lstrip("# ")
for sec in f.read_text().split("\n## ")[1:]:
head, _, body = sec.partition("\n")
out.append({"id": f"{f.stem}#{head.strip().lower().replace(' ', '-')}",
"text": f"{title} > {head.strip()}: {body.strip()}"})
return out
PASSAGES = load_passages()
_tok = lambda s: re.findall(r"[a-z0-9]+", s.lower())
_df = Counter(t for p in PASSAGES for t in set(_tok(p["text"])))
def retrieve(query, k=6):
"""Plain keyword retrieval (BM25-flavoured TF-IDF). Cheap, dumb, and good enough to have a recall problem."""
q = _tok(query)
def score(p):
tf = Counter(_tok(p["text"]))
return sum(tf[t] / (tf[t] + 1.2) * math.log(1 + len(PASSAGES) / _df[t]) for t in q if t in tf)
return sorted(PASSAGES, key=score, reverse=True)[:k]
print(len(PASSAGES), "passages")
for p in retrieve("How many API requests per minute can a Pro plan workspace make?", k=4):
print(" ", p["id"])
import re, math
from pathlib import Path
from collections import Counter
def load_passages(kb_dir="data/kb"):
"""Split every KB markdown file into one passage per '## ' section."""
out = []
for f in sorted(Path(kb_dir).glob("*.md")):
title = f.read_text().splitlines()[0].lstrip("# ")
for sec in f.read_text().split("\n## ")[1:]:
head, _, body = sec.partition("\n")
out.append({"id": f"{f.stem}#{head.strip().lower().replace(' ', '-')}",
"text": f"{title} > {head.strip()}: {body.strip()}"})
return out
PASSAGES = load_passages()
_tok = lambda s: re.findall(r"[a-z0-9]+", s.lower())
_df = Counter(t for p in PASSAGES for t in set(_tok(p["text"])))
def retrieve(query, k=6):
"""Plain keyword retrieval (BM25-flavoured TF-IDF). Cheap, dumb, and good enough to have a recall problem."""
q = _tok(query)
def score(p):
tf = Counter(_tok(p["text"]))
return sum(tf[t] / (tf[t] + 1.2) * math.log(1 + len(PASSAGES) / _df[t]) for t in q if t in tf)
return sorted(PASSAGES, key=score, reverse=True)[:k]
print(len(PASSAGES), "passages")
for p in retrieve("How many API requests per minute can a Pro plan workspace make?", k=4):
print(" ", p["id"])
23 passages rate-limits#limits-by-plan rate-limits#what-happens-when-you-exceed-a-limit rate-limits#burst-allowance pricing#plans
2. Use case 9: the relevance filter¶
We make one small call per passage, in parallel. Each state is just question + one passage, following Jev's advice to keep the state small and relevant. Passages with P(relevant) < 0.5 are dropped before the LLM sees them.
In [5]:
Copied!
def relevant(question, passage):
r = ask_jev({"question": question, "passage": passage["text"]}, {"relevant": Noul(
instructions="Does the passage state a fact that answers the question, or at least one part of it?",
criteria={"true": "it contains a fact the answer needs, even for only one part of the question",
"false": "it is about something else, or only shares keywords with the question"})})
return r.nouls["relevant"].noul
def filtered_retrieve(question, k=6, threshold=0.5):
cands = retrieve(question, k)
scores = jev_map(lambda p: relevant(question, p), cands)
return [dict(p, p_relevant=round(s, 2)) for p, s in zip(cands, scores) if s >= threshold], \
[(p["id"], round(s, 2)) for p, s in zip(cands, scores)]
q = "What's the retention period for automated backups, and can I restore to a point in time?"
kept, all_scores = filtered_retrieve(q)
print("retrieved:", all_scores)
print("kept :", [p["id"] for p in kept])
def relevant(question, passage):
r = ask_jev({"question": question, "passage": passage["text"]}, {"relevant": Noul(
instructions="Does the passage state a fact that answers the question, or at least one part of it?",
criteria={"true": "it contains a fact the answer needs, even for only one part of the question",
"false": "it is about something else, or only shares keywords with the question"})})
return r.nouls["relevant"].noul
def filtered_retrieve(question, k=6, threshold=0.5):
cands = retrieve(question, k)
scores = jev_map(lambda p: relevant(question, p), cands)
return [dict(p, p_relevant=round(s, 2)) for p, s in zip(cands, scores) if s >= threshold], \
[(p["id"], round(s, 2)) for p, s in zip(cands, scores)]
q = "What's the retention period for automated backups, and can I restore to a point in time?"
kept, all_scores = filtered_retrieve(q)
print("retrieved:", all_scores)
print("kept :", [p["id"] for p in kept])
retrieved: [('backups#point-in-time-recovery', 0.88), ('backups#automated-backups', 0.97), ('rate-limits#what-happens-when-you-exceed-a-limit', 0.01), ('deploys#rolling-back', 0.02), ('sso#enabling-saml', 0.01), ('regions#eu-data-residency', 0.03)]
kept : ['backups#point-in-time-recovery', 'backups#automated-backups']
3. Answer from the kept passages only, with citations¶
In [6]:
Copied!
ANSWER_SYSTEM = ("Answer ONLY from the passages. After each sentence cite its source like [backups#automated-backups]. "
"If the passages don't contain the answer, say exactly: I don't know based on the docs.")
def answer(question, passages, feedback=""):
ctx = "\n\n".join(f"[{p['id']}] {p['text']}" for p in passages) or "(no relevant passages)"
return chat(f"Passages:\n{ctx}\n\nQuestion: {question}{feedback}", system=ANSWER_SYSTEM)
print(answer(q, kept))
ANSWER_SYSTEM = ("Answer ONLY from the passages. After each sentence cite its source like [backups#automated-backups]. "
"If the passages don't contain the answer, say exactly: I don't know based on the docs.")
def answer(question, passages, feedback=""):
ctx = "\n\n".join(f"[{p['id']}] {p['text']}" for p in passages) or "(no relevant passages)"
return chat(f"Passages:\n{ctx}\n\nQuestion: {question}{feedback}", system=ANSWER_SYSTEM)
print(answer(q, kept))
Automated backups are retained for 14 days on Pro and 35 days on Enterprise. [backups#automated-backups] Enterprise databases can be restored to any second within the retention window, while Pro databases can be restored only to a daily snapshot. [backups#point-in-time-recovery]
4. Use case 10: the citation checker¶
We split the answer into sentences (code). For every citation, Jev checks whether that passage supports that sentence. We demonstrate on a deliberately wrong answer first, then on the real one.
In [7]:
Copied!
BY_ID = {p["id"]: p for p in PASSAGES}
CITE = re.compile(r"\[([a-z0-9-]+#[a-z0-9-]+)\]")
def check_citations(ans):
ans = re.sub(r"([.!?])\s*((?:\[[^\]]+\]\s*)+)", r" \2\1 ", ans) # "fact. [id]" -> "fact [id]."
claims = [s.strip() for s in re.split(r"(?<=[.!?])\s+", ans) if CITE.search(s)]
pairs = [(c, pid) for c in claims for pid in CITE.findall(c) if pid in BY_ID]
def supported(pair):
claim, pid = pair
r = ask_jev({"claim": CITE.sub("", claim).strip(), "cited_passage": BY_ID[pid]["text"]}, {"supported": Noul(
instructions="Is every fact in the claim stated in the cited passage?",
criteria={"true": "the passage states it (paraphrase is fine)",
"false": "the passage contradicts it, or doesn't mention part of it"})})
return r.nouls["supported"].noul
return [(c, pid, round(s, 2)) for (c, pid), s in zip(pairs, jev_map(supported, pairs))]
wrong = ("Pro workspaces can make 6,000 API requests per minute [rate-limits#limits-by-plan]. "
"Requests over the limit receive HTTP 429 [rate-limits#what-happens-when-you-exceed-a-limit].")
for claim, pid, s in check_citations(wrong):
print(f"{'SUPPORTED ' if s >= 0.5 else 'UNSUPPORTED'} P={s:.2f} {claim[:80]}")
BY_ID = {p["id"]: p for p in PASSAGES}
CITE = re.compile(r"\[([a-z0-9-]+#[a-z0-9-]+)\]")
def check_citations(ans):
ans = re.sub(r"([.!?])\s*((?:\[[^\]]+\]\s*)+)", r" \2\1 ", ans) # "fact. [id]" -> "fact [id]."
claims = [s.strip() for s in re.split(r"(?<=[.!?])\s+", ans) if CITE.search(s)]
pairs = [(c, pid) for c in claims for pid in CITE.findall(c) if pid in BY_ID]
def supported(pair):
claim, pid = pair
r = ask_jev({"claim": CITE.sub("", claim).strip(), "cited_passage": BY_ID[pid]["text"]}, {"supported": Noul(
instructions="Is every fact in the claim stated in the cited passage?",
criteria={"true": "the passage states it (paraphrase is fine)",
"false": "the passage contradicts it, or doesn't mention part of it"})})
return r.nouls["supported"].noul
return [(c, pid, round(s, 2)) for (c, pid), s in zip(pairs, jev_map(supported, pairs))]
wrong = ("Pro workspaces can make 6,000 API requests per minute [rate-limits#limits-by-plan]. "
"Requests over the limit receive HTTP 429 [rate-limits#what-happens-when-you-exceed-a-limit].")
for claim, pid, s in check_citations(wrong):
print(f"{'SUPPORTED ' if s >= 0.5 else 'UNSUPPORTED'} P={s:.2f} {claim[:80]}")
UNSUPPORTED P=0.02 Pro workspaces can make 6,000 API requests per minute [rate-limits#limits-by-pla SUPPORTED P=0.98 Requests over the limit receive HTTP 429 [rate-limits#what-happens-when-you-exce
5. Put it together: retrieve, filter, answer, verify, and regenerate once if needed¶
In [8]:
Copied!
def ask_docs(question):
kept, _ = filtered_retrieve(question)
ans = answer(question, kept)
checks = check_citations(ans)
bad = [(c, pid) for c, pid, s in checks if s < 0.5]
if bad:
fb = "\n\nYour previous answer had unsupported claims: " + "; ".join(c for c, _ in bad) + \
". Rewrite using only what the passages state."
ans, checks = answer(question, kept, fb), None
checks = check_citations(ans)
return {"answer": ans, "kept": [p["id"] for p in kept],
"supported": all(s >= 0.5 for _, _, s in checks) if checks else None}
for question in ["How many API requests per minute can a Pro plan workspace make before getting 429s?",
"How do I enable SAML single sign-on for my organization?",
"Which regions support data residency in the EU?",
"Does Acme support IPv6-only VPCs?"]: # not in the docs: should say "I don't know"
r = ask_docs(question)
print(f"Q: {question}\n kept={r['kept']} all citations supported={r['supported']}\n A: {r['answer']}\n")
def ask_docs(question):
kept, _ = filtered_retrieve(question)
ans = answer(question, kept)
checks = check_citations(ans)
bad = [(c, pid) for c, pid, s in checks if s < 0.5]
if bad:
fb = "\n\nYour previous answer had unsupported claims: " + "; ".join(c for c, _ in bad) + \
". Rewrite using only what the passages state."
ans, checks = answer(question, kept, fb), None
checks = check_citations(ans)
return {"answer": ans, "kept": [p["id"] for p in kept],
"supported": all(s >= 0.5 for _, _, s in checks) if checks else None}
for question in ["How many API requests per minute can a Pro plan workspace make before getting 429s?",
"How do I enable SAML single sign-on for my organization?",
"Which regions support data residency in the EU?",
"Does Acme support IPv6-only VPCs?"]: # not in the docs: should say "I don't know"
r = ask_docs(question)
print(f"Q: {question}\n kept={r['kept']} all citations supported={r['supported']}\n A: {r['answer']}\n")
Q: How many API requests per minute can a Pro plan workspace make before getting 429s? kept=['rate-limits#limits-by-plan'] all citations supported=True A: A Pro workspace can make 600 API requests per minute [rate-limits#limits-by-plan].
Q: How do I enable SAML single sign-on for my organization? kept=['sso#enabling-saml', 'sso#supported-protocols'] all citations supported=True A: An organization owner should open **Settings > Security > Single sign-on**, choose **SAML 2.0**, upload the identity provider metadata XML, and map the email attribute. After a successful test login, the owner can enforce SSO for all members. [sso#enabling-saml]
Q: Which regions support data residency in the EU? kept=['regions#eu-data-residency', 'regions#available-regions'] all citations supported=True A: Workspaces pinned to eu-central or eu-west keep customer data, backups, and logs inside the European Union. [regions#eu-data-residency]
Q: Does Acme support IPv6-only VPCs? kept=[] all citations supported=None A: I don't know based on the docs.
In [9]:
Copied!
print(SPEND)
print(SPEND)
Jev: 38 calls $0.00064 | LLM: 7 calls $0.00031
Recap¶
- Relevance filter (9): judge each question + passage pair. The LLM reads fewer, better passages, and when nothing relevant comes back it says "I don't know" instead of improvising.
- Citation checker (10): judge each claim + cited passage pair. Unsupported claims trigger a rewrite before the user ever sees them.
- Both are "pairwise yes/no" jobs. Jev does them in parallel, in milliseconds, for a fraction of a cent, which makes it cheap enough to verify every answer.
Next: 11_jev_as_judge_and_evals.ipynb, where Jev grades your agent, in CI and at runtime.