Capable isn't consistent.
The same agent that nails a task once can quietly drift over a long session or across repeated runs. The Harness closes that loop automatically.
Closes the Reliability Gap
Capable agents still degrade over long sessions and across runs. The Harness turns passive quality signals into automatic self-correction — not just observability.
Evidence-Gated Learning
Self-authored rules don't get trusted on reflection alone. Every rule proves itself by replaying a real failure or surviving a forward trial before it's promoted to active.
Never Blocks Your Agent
Evaluation runs as a detached, non-blocking background task after each turn. Nothing on the healing path can stall or interrupt your host loop.
Pull, Not Push
Diagnostic notices post to a filesystem mailbox. Nothing is pushed into the conversation — the agent pulls notices on its own initiative.
Framework-Agnostic
Drop-in adapters for LangGraph, LangChain, DeepAgents, CrewAI, Claude Agent SDK, and OpenAI Agents — or wrap any custom loop directly.
Zero-Dependency Core
A pure standard-library core that talks to the PandaProbe platform exclusively through the pandaprobe CLI — a clean, explicit trust boundary.
Detect, notice, heal, validate.
A closed loop that turns quality signals into proven, promoted fixes — without ever touching your agent's conversation until it opts in.
Detect
A detached, non-blocking background task scores every turn on agent reliability and agent consistency. Breaches and declining trends are both caught.
Notice
Breaches post a structured diagnostic notice to a filesystem mailbox. Nothing is pushed into the conversation — the agent pulls notices on its own initiative.
Heal
A standing protocol in the system prompt guides the agent to read the notice, inspect its flagged traces, and record a mitigation rule with the self-diagnostic toolset.
Validate
The rule enters as a candidate. It's promoted to active only after replaying the captured failure or surviving a forward trial — and retired if it ever regresses.
Rules earn trust — they aren't granted it.
Self-authored fixes are validated against real evidence before they ever run in production, and re-checked every time the rule set changes.
Replay Validation
Prove it on the failure that triggered it.
When a replay function is wired in, the candidate rule is re-rendered against up to three of the newest failing cases plus two protected wins, then scored on the result.
- Up to 3 newest failing cases, 2 protected wins
- Promotes only with no regression on any metric
- 3 inconclusive rounds fall back to a forward trial
Forward-Trial Validation
Prove it live, when there's no replay harness.
Without a replay function, the candidate is trialed over live sessions and promoted once its breach rate hits zero or clears its own baseline by a real margin.
- Automatic fallback — no setup required
- Enrolls sessions matching the rule's metric family
- A statistical bar, not a single anecdote
Regression Guard & Calibration
Keep the gate honest as the rule set grows.
Every eval case captured on a breach — plus every protected win — gets replayed against the current rule set on demand, and breach thresholds are measured against labeled outcomes instead of guessed at.
- pandaprobe-harness-eval replays the eval set for regressions
- pandaprobe-harness-calibrate tunes thresholds against labeled outcomes
Drop it around what you already run.
First-class adapters for the major agent frameworks, a raw API for custom loops, and a self-diagnostic toolset the agent calls like any other tool.
Framework Adapters
Self-Diagnostic Toolset
14 operations across 4 categories, exposed as native tool-calling functions your agent can invoke directly.
Wrap your agent loop in minutes.
Install the package, then wrap a single turn boundary — no framework migration required.
Install & configure
| 1 | pip install pandaprobe-harness |
| 2 | |
| 3 | # point it at your PandaProbe project |
| 4 | export PANDAPROBE_API_KEY="pp_live_..." |
| 5 | export PANDAPROBE_PROJECT_NAME="your-project" |
Zero runtime dependencies in the base install — Python 3.13+.
Wrap your loop
| 1 | from pandaprobe_harness import Harness |
| 2 | from pandaprobe_harness.agent_tools.native import as_anthropic_tools |
| 3 | |
| 4 | harness = Harness.create() |
| 5 | system_prompt = harness.system_context() + MY_PROMPT |
| 6 | specs, dispatch = as_anthropic_tools(harness.toolset) |
| 7 | tools = my_tools + specs |
| 8 | |
| 9 | async def one_turn(session_id: str, user_input: str) -> str: |
| 10 | async with harness.turn(session_id): |
| 11 | return await my_agent_step(system_prompt, tools, user_input) |
One block around your existing turn boundary — for_langgraph(), for_crewai(), and the other framework adapters need even less.
Frequently asked questions
Everything you need to know about self-healing your agents with the PandaProbe Harness.