An outdated fee
“It used to cost less.”
Does the agent keep quoting an old amount after the approved fee has changed?
ThirdOcular / Accuracy pen testingIn development
Truvli
Put your AI's answers
under pressure.
A confident answer can still quote the wrong fee, miss an exception, or invent a policy. I'm building Truvli to test what your AI tells customers against the facts your organization has approved.
The next steps are buyer conversations and a demonstration, before a full product. There are no published Truvli results yet.
The problem
What would happen if a customer believed it?
That is the question behind the test. A polished answer is not enough when someone uses it to decide what they owe, whether they qualify, or what to do next.
A boundary case
“I'm just below the limit.”
Does it apply an eligibility threshold to this customer's circumstances correctly?
Customer pushback
“Can you make an exception?”
Does it invent permission after a customer challenges a correct answer?
Illustrative questions that explain the intended test. These are not measured failures or sample results.
Planned process
From your facts to evidence you can inspect.
Agree on what the correct answer is.
The starting point is your approved fee schedule, eligibility rules and policy terms. Each check will cite the relevant section and effective date. Conflicting sources need a decision from their owner before scoring.
Ask the easy questions and the awkward ones.
The test will cover direct questions, different wording, thresholds, superseded facts, missing information, pushback and conversations that change direction. The questions will be frozen before the first run.
Show the answer, the verdict and the reason.
The intended record connects each conversation to the rule and source used to judge it. It will distinguish a wrong answer from an unsupported claim, and send cases the checker cannot settle to human review.
Measure the checker too.
A separately labeled sample will check how often the scoring misses errors or flags correct answers. The report will include those error estimates, their uncertainty, and any use of AI to extract claims.
Planned pilot
One agent.
One approved fact set.
One inspectable report.
A reviewer should be able to follow a conclusion back to the answer and the source, without taking my word for it.
What the report is designed to contain
- Results by question type, with unresolved cases visible.
- Transcripts, test dates and the agent configuration tested.
- Rule references, source sections and effective dates.
- A downloadable evidence bundle with file checksums.
- Measured checker errors, limits and follow-up questions.
Planned later: a runner inside your environment and repeat tests that show changes over time. Neither is available today.
Who I want to hear from
Teams who need to show their answers hold up.
I'm looking to speak with teams building customer-facing agents for banks, credit unions, insurers and healthcare organizations. Especially teams whose buyer has asked for evidence that answers match that organization's own rules.
A useful first conversation starts with one decision: which answers concern the buyer, who approves the underlying facts, and what evidence would help them decide? Please start with a description of the problem, not private customer records or credentials.
What this is not
A bounded accuracy test.
Truvli is not an audit, attestation or certification. It is not a cybersecurity penetration test, medical advice, or a guarantee that every future answer will be correct. It will not replace SOC 2, ISO 42001 or a broader review of how an agent is governed.
The intended scope is the approved facts, questions and agent version tested. Results would describe that scope at that time. Testing any real agent requires its owner's written permission.
Why I'm building it
Two retired projects made the next question clearer.
AisleLens taught me to test the evidence behind a claim and the errors in my own checker. PRIVI taught me to separate what an agent is allowed to do from what its behavior proves. Truvli brings those lessons to one question: does this answer match the approved facts?
What I'm carrying forward →Start with the problem
What answer would your buyer
need you to prove?
Tell me about the last accuracy question that held up a review.
Talk with me about TruvliSandeep Kanuri · Independent work through ThirdOcular