Evidence package
What the evaluation’s evidence is, and how to check it.
For the technical evaluator and the security reviewer. It covers the Claude Code path the evaluation exercises: the formats, the verification procedure, the architecture, where enforcement starts and stops, the threat model, and the limits. Where a control is partial, the limit is stated next to it.
Architecture
- Claude Code is about to run a tool. Its
PreToolUsehook startsprae hook claude-code, one process per call, with the call on stdin. - The hook loads the policy pack (and, optionally, an organisation’s signed packs), maps the call to the facts rules read (tool, capability, arguments, paths relative to the session’s directory) and asks the policy kernel. The kernel is deterministic: no model is in the decision.
- It appends a row to the ledger under a lock, chained to the previous row and signed with the operator’s key, then answers Claude Code: nothing for allow,
denyoraskwith the reason otherwise. A clamp is a deny that names the permitted value. - For a held call,
PostToolUserecords that it ran (approved) andSessionEndrecords the holds that never ran (declined). - Everything is local: the pack, the ledger and the key stay on the machine. Nothing is sent to PRAE.
The ledger row
One JSON object per line. The fields an evaluator reads:
- seq, prev, hash
- Position in the chain, the previous row’s hash, and this row’s sha256 over its canonical JSON (keys sorted at every depth) including seq and prev. The first row’s prev is 64 zeros.
- sig
- On an installed hook: an Ed25519 signature over hash, made with a key the operator holds. Absent on the evaluation page’s rows.
- at, session, cwd
- When, in which Claude Code session, and in which working directory the call was judged.
- via
- The enforcement point: claude-code:PreToolUse on an install, the evaluation page on this site.
- tool, subject, capability
- The tool, the one bounded string it was aimed at (a path or URL, never file contents or a whole command line), and its capability class.
- verdict, rule, reason, trace
- allow, deny, ask or correct; the deciding clause and its rationale; every rule that matched, strongest first.
- pack
- The policy in force: name, version and the sha256 of its exact bytes.
- corrected
- For a clamp: each argument’s requested and permitted value.
- micros
- The decision’s own cost, in microseconds.
The evidence file
What the evaluation exports, one JSON document:
- schema
- prae.evaluation-evidence/1
- generated, source
- When the file was made, and the page that made it.
- pack
- Name, version, sha256 and the full source of the policy, so the hash and every decision can be checked from the file alone.
- decisions[]
- Per call: the scenario, the call, the expected and actual verdict, the clause, the trace, the permitted value for a clamp, the exact answer Claude Code would receive, the latency, the timestamp, the ledger row it produced, and the prae explain command that reproduces it.
- ledger
- The rows themselves, the entry count, the head hash and its short seal, and the integrity result computed when the file was made.
- verify
- The commands below.
- limits
- What the file does and does not show, carried with it.
Verification procedure
With Node 20 or later and jq, from the directory holding the evidence file:
# 1. Integrity: rebuild the ledger from the file and verify the chain jq -c '.ledger.rows[]' prae-<session>.evidence.json > evaluation.ledger.jsonl npx prae-cli verify evaluation.ledger.jsonl # → "<n> entries · chain intact · seal …"; the seal equals .ledger.seal # 2. The policy: extract the pack and check its hash jq -j '.pack.source' prae-<session>.evidence.json > evaluation.prae.json shasum -a 256 evaluation.prae.json # equals .pack.sha256 and every row's pack.sha256 # 3. Correctness: re-run each decision against the pack jq -r '.decisions[].reproduce' prae-<session>.evidence.json # run each line; the verdict and clause match .decisions[].verdict and .rule
- A tampered file is identified. Change any field of any row, or delete or reorder a row, and step 1 prints
CHAIN BROKENwith the first line that does not match. The page’s tampered download is one such file. - Integrity is not correctness. Step 1 shows the rows were not changed after they were written. It cannot show a decision was right; step 3 is that check, made independently of the ledger.
- On an install, add the operator’s public key:
prae verify <ledger> --key <public.pem>also checks every row’s signature, andprae checkpointpins a head so a truncation shows.
Threat model and limits
Assets: the developer’s credentials and files, the systems the agent can reach through tools, and the integrity of the record. Out of scope: a malicious process already running as the developer, and anyone holding the signing key who rewrites the whole ledger.
| Risk | How | Control | Residual |
|---|---|---|---|
| An agent runs a tool the policy forbids | Prompt injection, a confused agent, a poisoned file | The hook judges every tool call before it runs and answers deny; matcher * covers built-in and MCP tools | Shell commands are judged by their text, not their effect: cat .env in Bash is not a Read of .env. Write command rules, or deny proc:spawn. |
| An agent edits PRAE’s own pack, ledger or settings | The agent runs as the developer | PRAE-SELF refuses writes to them, reads of the key, and shell lines that name them | A determined shell can spell a path no string check sees. An edited ledger breaks its chain and signatures. |
| Someone edits the record after the fact | Anyone with write access to the file | Hash chain: an edited, removed or reordered row breaks it. Rows signed with the operator’s key | A whole-file rewrite by someone holding the key, and truncation, need an external checkpoint (prae checkpoint, then prae verify --checkpoint). |
| The gate fails open | A bad pack, an unwritable ledger, a crash, a timeout | Enforce mode refuses when the hook runs and fails, and writes onFailure: "block" | Claude Code before 2.1.295 ignores onFailure: a hook that cannot start at all, or hits the 10 s timeout, lets the call through. |
| A path rule is walked around | Symbolic links, .. segments | .. is resolved first; a path through a symlink is judged at its real target, and the stricter verdict stands | A link swapped between the check and the tool’s own open (a race) is not covered. |
| An approval is misattributed | Who answered the prompt | Ask uses Claude Code’s own permission prompt; PRAE records that the held call ran | Claude Code does not say who approved. The ledger records the outcome, not an attested approver. |
- Arguments are never rewritten in place. A clamp refuses the call and names the permitted value; the agent retries with it, so the call the agent and the developer see is the call that runs.
- Tool output is not screened in Claude Code in this release.
- No detection of prompt injection or goal hijack. Their effects are bounded by the rules at the execution boundary.
- Commands you type after
!run in your own shell, without the agent or any hook.
What the evaluation page is, and is not
It runs the real policy kernel in your browser with the Claude Code hook’s mapping, and an automated test
holds it to the hook: every scenario is put through prae hook claude-code,
and the verdict, the answer to Claude Code and the ledger row must match. It does not run an agent, read a
file, resolve a symbolic link, ask a person or sign a row, and its latency is the browser’s. The figures on
the evaluation page come from the real hook.