What the evaluation’s evidence is, and how to check it.

For the technical evaluator and the security reviewer. It covers the Claude Code path the evaluation exercises: the formats, the verification procedure, the architecture, where enforcement starts and stops, the threat model, and the limits. Where a control is partial, the limit is stated next to it.

Architecture

  1. Claude Code is about to run a tool. Its PreToolUse hook starts prae hook claude-code, one process per call, with the call on stdin.
  2. The hook loads the policy pack (and, optionally, an organisation’s signed packs), maps the call to the facts rules read (tool, capability, arguments, paths relative to the session’s directory) and asks the policy kernel. The kernel is deterministic: no model is in the decision.
  3. It appends a row to the ledger under a lock, chained to the previous row and signed with the operator’s key, then answers Claude Code: nothing for allow, deny or ask with the reason otherwise. A clamp is a deny that names the permitted value.
  4. For a held call, PostToolUse records that it ran (approved) and SessionEnd records the holds that never ran (declined).
  5. Everything is local: the pack, the ledger and the key stay on the machine. Nothing is sent to PRAE.

The ledger row

One JSON object per line. The fields an evaluator reads:

seq, prev, hash
Position in the chain, the previous row’s hash, and this row’s sha256 over its canonical JSON (keys sorted at every depth) including seq and prev. The first row’s prev is 64 zeros.
sig
On an installed hook: an Ed25519 signature over hash, made with a key the operator holds. Absent on the evaluation page’s rows.
at, session, cwd
When, in which Claude Code session, and in which working directory the call was judged.
via
The enforcement point: claude-code:PreToolUse on an install, the evaluation page on this site.
tool, subject, capability
The tool, the one bounded string it was aimed at (a path or URL, never file contents or a whole command line), and its capability class.
verdict, rule, reason, trace
allow, deny, ask or correct; the deciding clause and its rationale; every rule that matched, strongest first.
pack
The policy in force: name, version and the sha256 of its exact bytes.
corrected
For a clamp: each argument’s requested and permitted value.
micros
The decision’s own cost, in microseconds.

The evidence file

What the evaluation exports, one JSON document:

schema
prae.evaluation-evidence/1
generated, source
When the file was made, and the page that made it.
pack
Name, version, sha256 and the full source of the policy, so the hash and every decision can be checked from the file alone.
decisions[]
Per call: the scenario, the call, the expected and actual verdict, the clause, the trace, the permitted value for a clamp, the exact answer Claude Code would receive, the latency, the timestamp, the ledger row it produced, and the prae explain command that reproduces it.
ledger
The rows themselves, the entry count, the head hash and its short seal, and the integrity result computed when the file was made.
verify
The commands below.
limits
What the file does and does not show, carried with it.

Verification procedure

With Node 20 or later and jq, from the directory holding the evidence file:

# 1. Integrity: rebuild the ledger from the file and verify the chain
jq -c '.ledger.rows[]' prae-<session>.evidence.json > evaluation.ledger.jsonl
npx prae-cli verify evaluation.ledger.jsonl
#    → "<n> entries · chain intact · seal …"; the seal equals .ledger.seal

# 2. The policy: extract the pack and check its hash
jq -j '.pack.source' prae-<session>.evidence.json > evaluation.prae.json
shasum -a 256 evaluation.prae.json      # equals .pack.sha256 and every row's pack.sha256

# 3. Correctness: re-run each decision against the pack
jq -r '.decisions[].reproduce' prae-<session>.evidence.json
#    run each line; the verdict and clause match .decisions[].verdict and .rule
  • A tampered file is identified. Change any field of any row, or delete or reorder a row, and step 1 prints CHAIN BROKEN with the first line that does not match. The page’s tampered download is one such file.
  • Integrity is not correctness. Step 1 shows the rows were not changed after they were written. It cannot show a decision was right; step 3 is that check, made independently of the ledger.
  • On an install, add the operator’s public key: prae verify <ledger> --key <public.pem> also checks every row’s signature, and prae checkpoint pins a head so a truncation shows.

Threat model and limits

Assets: the developer’s credentials and files, the systems the agent can reach through tools, and the integrity of the record. Out of scope: a malicious process already running as the developer, and anyone holding the signing key who rewrites the whole ledger.

Risk How Control Residual
An agent runs a tool the policy forbids Prompt injection, a confused agent, a poisoned file The hook judges every tool call before it runs and answers deny; matcher * covers built-in and MCP tools Shell commands are judged by their text, not their effect: cat .env in Bash is not a Read of .env. Write command rules, or deny proc:spawn.
An agent edits PRAE’s own pack, ledger or settings The agent runs as the developer PRAE-SELF refuses writes to them, reads of the key, and shell lines that name them A determined shell can spell a path no string check sees. An edited ledger breaks its chain and signatures.
Someone edits the record after the fact Anyone with write access to the file Hash chain: an edited, removed or reordered row breaks it. Rows signed with the operator’s key A whole-file rewrite by someone holding the key, and truncation, need an external checkpoint (prae checkpoint, then prae verify --checkpoint).
The gate fails open A bad pack, an unwritable ledger, a crash, a timeout Enforce mode refuses when the hook runs and fails, and writes onFailure: "block" Claude Code before 2.1.295 ignores onFailure: a hook that cannot start at all, or hits the 10 s timeout, lets the call through.
A path rule is walked around Symbolic links, .. segments .. is resolved first; a path through a symlink is judged at its real target, and the stricter verdict stands A link swapped between the check and the tool’s own open (a race) is not covered.
An approval is misattributed Who answered the prompt Ask uses Claude Code’s own permission prompt; PRAE records that the held call ran Claude Code does not say who approved. The ledger records the outcome, not an attested approver.
  • Arguments are never rewritten in place. A clamp refuses the call and names the permitted value; the agent retries with it, so the call the agent and the developer see is the call that runs.
  • Tool output is not screened in Claude Code in this release.
  • No detection of prompt injection or goal hijack. Their effects are bounded by the rules at the execution boundary.
  • Commands you type after ! run in your own shell, without the agent or any hook.

What the evaluation page is, and is not

It runs the real policy kernel in your browser with the Claude Code hook’s mapping, and an automated test holds it to the hook: every scenario is put through prae hook claude-code, and the verdict, the answer to Claude Code and the ledger row must match. It does not run an agent, read a file, resolve a symbolic link, ask a person or sign a row, and its latency is the browser’s. The figures on the evaluation page come from the real hook.