The evaluation lab for AI in technical support
Turn your support conversations and playbooks into tests. Your experts approve what good support looks like. Run every new version of your AI agent against them before it reaches customers.
NavishAI evaluates your support AI. It never replies to your customers.
Runs / Run 14
Replay readiness · candidate v3
| Result | Case | Importance | Checks |
|---|---|---|---|
| Fail | Webhook replay writes duplicate invoices | Critical | 2 checks failed |
| Abstain | SSO stopped after certificate rotation | High | Judge below threshold |
| Pass | REST quota retry headers | Normal | 3 of 3 checks |
| Pass | REST quota retry timing | Normal | 2 of 2 checks |
| Pass | Invoice shows duplicate seat charges | High | 4 of 4 checks |
Failed check
Escalate to Engineering on repeated ACS 500s
Company evidence
“Valid IdP metadata plus repeated ACS 500 responses: escalate to Engineering.”
SSO playbook, section 4.2
Agent response
“Please re-upload the metadata and try again.”
Generic benchmarks don't know your product
The right answer to an SSO failure, an API error or a billing dispute depends on your playbooks, plans and escalation rules. A polite reply can still be the wrong decision.
- Built from your evidence
- Every scenario links to the exact conversation or document it came from.
- Judged by your experts
- Experts approve each expected behaviour. Machine proposals stay drafts until they do.
- Kept as regressions
- A reviewed failure becomes a test for the next version of your agent.
From support history to regression suite
Six steps. Each one keeps its sources, its versions and the human decisions behind it. A historical answer is evidence, not automatically the right answer.
-
Build a source-backed corpus
Upload conversation exports and text or Markdown documentation. Fixed snapshots let every scenario point to the exact record it came from.
-
Find cases worth testing
Review proposed issue families, examples and risk signals, and see why each case was picked. Proposals need expert review; selection counts are not verified coverage.
-
Let experts define good support
Correct the situation, facts, expected outcome and escalation rules, then approve an exact version. Controlled variants keep their parent and need their own review.
-
Compile the behaviour into checks
Bind each requirement to a fixed grader and a source excerpt. Deterministic checks inspect reported fields and actions; versioned rubric judges handle context.
-
Calibrate against human judgment
Compare graders with expert labels, inspect false alarms, missed failures and disagreement, and keep held-out evidence separate. Confidence alone is not accuracy.
-
Inspect failures. Keep regressions.
Run a suite against a configured target or a stored output. Read each check beside the response and the evidence, then keep reviewed failures for the next version.
Test the decision, not just the wording
One SSO incident can test diagnosis, evidence collection and escalation. This authored example shows the method—not a customer result or a prebuilt SSO integration.
SAML login fails after an IdP change
- Starting context
- The customer reports a login failure after updating their identity provider.
- Knowledge for the target
- The expert chooses which SSO documentation excerpts the agent may see.
- Expectation evidence
- The company playbook supports the diagnostic steps and handoff rule. Hidden expectations stay out of target input.
Checks for this reviewed situation
- Collect evidence
- Request the certificate expiry date before suggesting a configuration change.
- Respect the boundary
- Do not claim to have changed the customer's IdP configuration.
- Escalate on new facts
- When valid metadata still yields repeated ACS 500 responses, hand off to Engineering with the evidence.
- Grade with the right tool
- A field check can inspect a reported expiry value. A calibrated rubric judge can assess the troubleshooting explanation. Neither proves external actions occurred.
What changes in a controlled variant?
Change one named fact: an expired certificate becomes a valid certificate. Keep the parent scenario and the exact before/after values. An expert explains why the expected behaviour changes, reviews the new version and compiles its checks. A synthetic variant does not replace real company evidence.
Make the next failure a test
Support changes. Keep the test history without freezing your assumptions.
- Bring in a production failure
- Upload a structured trace and its reported correction. Inspect suggested scenario matches or create a draft. A match or imported correction cannot approve the expected behaviour.
- Review, compile, retain
- An expert checks the evidence and approves a version. Grade the output, then deliberately add a failed case to regression. Stored replay grades an existing response; it does not rerun the production agent.
- Compare like with like
- Compare saved runs on the same fixed case and visible input. Changed definitions stay unmatched. Source-change views flag linked evidence; they cannot discover every unrecorded assumption.
Your corpus is private input, not training material
Company support data can contain customer details, secrets and internal policies. Know what stays in the deployment and what a requested operation sends.
- Local by default
- Intake, local analysis, review and deterministic grading stay in your deployment. Workspace-scoped access separates company data. NavishAI does not train models on your corpus.
- Explicit external disclosure
- Model-assisted source processing, HTTP targets and rubric judges need operator-approved endpoints and human disclosure confirmation. Targets receive visible context and permitted knowledge—not hidden expectations. Judges receive the fixed requirement, relevant evidence and output.
- Masking has limits
- Email masking or authored exact-text rules do not remove all PII. Review retained titles, text and context before approving disclosure. Masking new input does not rewrite earlier snapshots.
- Retention and deletion
- Source retention limits access. Source deletion also clears local derived scenarios, calibration and run records in that corpus. It cannot recall data already sent to an endpoint; remote retention belongs to that operator.
Before you open the lab
A focused evaluation workflow, with clear limits.
Is NavishAI a helpdesk or support agent?
No. NavishAI evaluates AI support systems. It does not run an inbox, draft customer replies or send customer messages. Historical conversations enter as evidence for tests.
What data can I start with?
Upload supported Intercom or Zendesk JSON conversation exports, the lab's bounded conversation JSONL format, and text or Markdown documents. Structured production traces use the lab's trace schema. These are upload paths, not live vendor connections.
How do I connect the system under test?
Use the lab's generic HTTPS target contract through an operator-approved public endpoint, including the incremental conversation contract when needed. Compatible stored outputs can be graded locally. The scripted adapter tests the machinery; it is not a live support agent. There are no built-in vendor agent integrations.
Does a passing suite mean the agent is safe to ship?
It means the recorded outputs passed those fixed checks. It does not establish coverage of all company issues, grader accuracy or deployment readiness. Check expert disagreement and held-out evidence. Abstentions and execution errors stay distinct from behavioural failures; unknown cost stays unknown.
Where does NavishAI run?
NavishAI supports a self-hosted, customer-controlled deployment. Operators own the database, TLS, secrets, jobs, backups and endpoint policy. Sign in to an existing deployment; first-Owner setup needs an active operator-issued bootstrap token. This page does not offer public account registration or a managed service.
Start with the support decisions your AI must get right
Bring a slice of your support history and the agent you want to test. Your experts set the expectations; the failures worth keeping become your regression suite.
Built by a Support leader with 13+ years in B2B SaaS post-sales. Questions? Write to [email protected].