Your semantic cache is probably wrong more often than your dashboards show
CacheVerifier is a semantic cache verification platform for LLM apps. A similarity threshold can't tell a real cache hit from a confident-looking near-miss. Serve one of those, and if an agent treats it as fact, it's a hallucination you never chose.
Query: "When will my order arrive?"
"In-stock items ship within 48 hours and usually arrive 2–3 days after shipping."
"In-stock items ship within 48 hours."
It says when the order ships, not when it arrives.
Both candidates score high on similarity — a plain threshold can't separate them. Illustrative example, not a live API call; try /v1/verify on your own pairs for free.
Architecture
Where CacheVerifier sits in your system
You keep your cache backend and decide what counts as gray-zone. CacheVerifier only runs on the candidates you route to it, and the learning loop runs asynchronously — neither ever blocks a request.
Request flow · synchronous
A user asks a question.
GPTCache, LangChain, a Redis lookup — you set the gray zone.
Gray-zone hits only — a confident hit skips this and serves straight from cache.
approved → serve the cached answer
rejected → call your LLM, then re-cache
Async learning loop · never on the request path
Was the served answer actually correct?
Fine-tune job retrains your verifier · drift monitor watches its stability.
Flows back into /v1/verify — the next gray-zone hit uses the updated model.
One line in a pipeline you already have
It implements GPTCache's own SimilarityEvaluation extension point — no fork, no PR. On LangChain, VerifiedCache wraps the cache you already use. A self-built Redis lookup? /v1/verify is a plain REST call.
from cacheverifier.integrations.gptcache import CacheVerifierEvaluation
cache.init(
similarity_evaluation=CacheVerifierEvaluation(api_key="cv_..."),
...
)Works with the cache you already run
CacheVerifier sits behind your cache; it doesn't replace it.
Capabilities
What CacheVerifier gives you
Synchronous verification
Send a borderline hit, the question and the cached answer, and get approve or reject with a score in tens of milliseconds, whatever cache produced it.
Self-serve fine-tuning
Send feedback on verdicts and train a verifier on your own traffic in minutes. It's free, and it's what makes the verifier accurate.
Drift monitoring
Flags when your traffic shifts away from what your verifier was trained on, the failure that quietly turns a fine-tune harmful.
Certified risk budgets
Certify a threshold that bounds expected false-reuse risk while traffic looks like your calibration data; randomized audits keep that sample fair.
ROI, not just a hit rate
Your dashboard turns real verify volume and held-out accuracy into calls avoided, wrong hits caught and, given your LLM cost, dollars saved.
Gray-zone tuning
Replays your feedback to recommend where your cache's own cutoff should sit, so verify only runs on the hits that are genuinely uncertain.
Proof
We tried to break this before shipping it
Three questions the underlying research didn't answer — how noisy feedback can get, how little data is enough, whether gains decay over time — checked on three public benchmarks and real production traffic, not just one dataset.
Fine-tuning gain (AUC delta) vs. label noise — all three datasets cross zero in the same range
~130K
benchmark-labeled gray-zone examples across multiple datasets
~30%
label-noise threshold, consistent across datasets
1,000
feedback rows before fine-tuning reliably helps everywhere
Both limits are enforced for you automatically — Health Check fails closed below the AUC bar, and fine-tune jobs hold back instead of activating once label noise clears the danger zone.
~6×
more answers reused
Live API test: where verification pays off
When 90% of reused answers had to be correct, a fine-tuned verifier reused 44% of the reusable cached answers. A tuned similarity threshold managed 7%. At looser risk levels, a good threshold did just as well.
LMArena public benchmark, 500 held-out pairs through the production API, single run, best threshold for each picked on the same data. Full results and caveats
Fit
Is this your traffic?
Verification helps a specific shape of workload, not every semantic cache. Four things that need to be true — and the honest read if they aren't.
The same question reaches you worded many different ways.
FAQ deflection, support assistants, high-volume RAG — enough of your hits are genuine paraphrases, not near-duplicates, that a similarity score has real work to do.
A wrong reused answer costs several times more than one extra model call.
The wrong refund policy, cancellation step, or dosage is expensive; a miss just costs a regeneration. With a verifier fine-tuned on your traffic, the break-even on our strongest benchmark is about 3.5×; untuned, it takes tens of times. If the two costs are close, raising your threshold for maximum hit rate is the better move.
The gray zone is a real fraction of your hits.
Not near-zero — then there's nothing to verify. Not near-everything — then your threshold is just set wrong. Somewhere in between, a verify call actually changes the outcome.
The verifier can tell your gray-zone hits apart — on your data.
Not something you can eyeball. The free Health Check measures whether the stock model, or a fine-tune on your feedback, is actually more trustworthy than raw similarity in that band.
If a Health Check shows your gray zone is Quora-shaped — where the task's own score ceiling, not your threshold, is the limit — raising the threshold is the right call and you don't need this. We'd rather tell you that before you integrate.
FAQ
Common questions
Data handling, self-hosting, SLA, fine-tuning requirements and the rest are on the full FAQ.
+What is semantic cache verification?
+Do I have to replace my existing cache?
+Is it really free?
+When is verification worth adding — and when isn’t it?
+Is this the same as prompt caching?
+How much latency does a verify call add?
Get started
From first call to a managed loop
Start free
Register gets you a tenant, a dashboard, and an API key — no card required.
Point your gray zone at /v1/verify
Wherever your cache backend flags an uncertain hit — GPTCache, LangChain, a self-built Redis lookup.
Run a health check
See baseline vs. fine-tuned precision on a sample of your real traffic before committing to anything.
Let us run the loop
Move to managed calibration + drift monitoring once you know it's worth automating.
Self-serve is free forever.
Verify and fine-tuning stay free, and so does the Health Check Report. Don't want to own the retraining loop? Managed runs recalibration on a schedule, watches for drift and alerts you, from $149/mo.