How good are the detectors?
A hit rate is a rate of the detector until somebody says what the detector should have
found. So a labelled corpus ships inside the package, at ruleprobe/corpus/: ten synthetic
sessions in all three transcript shapes, every interesting event labelled by hand with the
detectors that ought to fire on it, and a deliberate near-miss beside each one - a cat of
a line range, a find narrowed by -name, a git push after the gate ran, a heredoc with
rm -rf in its body as text rather than as a command.
ruleprobe corpusdetector pos neg tp fp fn prec recall f1 note
-------------------------------------------------------------------------------------------
cache-hygiene/compact 5 6 5 0 0 1.00 1.00 1.00
cache-hygiene/model-switch 5 15 5 0 0 1.00 1.00 1.00
commits/non-conventional-subject 5 9 5 0 0 1.00 1.00 1.00
git-safety/force-push-default 5 7 5 0 0 1.00 1.00 1.00
package-manager/pip-install 5 7 5 0 0 1.00 1.00 1.00
secrets/secret-file-add 5 6 5 0 0 1.00 1.00 1.00
secrets/secret-in-write 15 13 15 0 0 1.00 1.00 1.00
testing/test-after-change 11 6 11 0 0 1.00 1.00 1.00
transcript-hygiene/unfiltered-find 10 15 10 0 0 1.00 1.00 1.00
transcript-hygiene/whole-file-cat 10 14 10 0 0 1.00 1.00 1.00
verification/no-verify 11 13 11 0 0 1.00 1.00 1.00
-------------------------------------------------------------------------------------------
total 87 111 87 0 0 1.00 1.00 1.00 floor 0.90The six shipped detectors and every catalog entry are scored over the corpus. A detector
the corpus labels is scored by those labels alone, so a catalog entry’s own examples: score
its row only when the corpus labels none of it, and the row’s source in --json then says
examples. The test suite runs every restating entry’s examples against the shipped
detector, so they stay true of it.
pos and neg are what the labels asked for; tp, fp and fn are what happened.
ruleprobe corpus --floor 0.9 exits non-zero when a scored detector falls under the floor,
and CI in this repository runs exactly that. It is a gate on the repository, not on a run:
nothing in ruleprobe report reads the floor, and no report of yours will ever fail because
a detector scored badly. --json prints the same numbers as data.
ruleprobe report --validity puts each detector’s p= and r= beside its row. It is off
by default because the report is meant to be read in a minute and an eight-column table is
not, and because the same two numbers apply to every run - they belong to the detector, not
to your transcripts.
Read the number for what it is. The corpus is synthetic and hand-labelled: no real
transcript content, no home paths, no personal names. A 1.00 says the detector is right
about the shapes somebody thought to write down, which is a weaker claim than it looks - the
false positives a detector meets in the wild are the ones nobody anticipated. It is a floor
under an obvious mistake and a place to put the next surprising transcript, not a measured
field accuracy. Growing it is the cheapest contribution this repository takes: add a session
under ruleprobe/corpus/sessions/, label it in ruleprobe/corpus/labels.yaml, and the
table above moves.
A wrong hit of yours becomes a labelled negative. When ruleprobe explain shows a hit that
should not have fired, ruleprobe label records it in a corpus directory you name:
mkdir -p ~/ruleprobe-corpus
ruleprobe label --session claude-code:sess-1 --detector verification/no-verify \
--key 1:toolu_2 --corpus ~/ruleprobe-corpus --name no-verify-in-a-message
ruleprobe corpus --corpus ~/ruleprobe-corpusIt reruns the detector over your transcripts, takes the one event behind the hit, and writes it,
redacted, as sessions/<name>.events.jsonl - one event per line, read back without a runtime
reader - with one near label for it in labels.yaml, which it creates or appends to so a
hand-written file keeps its comments. It is the only command that writes, and it writes nothing
outside that directory. It refuses, says why and writes nothing when the hit is a session hit
(<turn>:-), which one event cannot reproduce; when the written event no longer produces the
hit, because redaction changed what the detector matched or the hit needs the events around
it, since that negative would pass trivially; when a secret shape would survive into the
written bytes; and when the session file exists already or labels.yaml would not read back
as before plus one entry. It takes report’s --root, --runtime and --since, and
--rules, --detectors and --no-config; --name is a plain file stem. It takes no
--stance, because a detector gated on one is refused, since the corpus runs with none, and
no --plugins, because ruleprobe corpus loads none to score the negative with. A Claude Code
subagent’s own transcript is its own session, keyed as <parent session id>/<agent id>, and a
transcript that yields no event is no session at all. Transcripts that share a session id, as
copies do, are one session in report, explain and label: the one with the most events is
read, the first in path order on a tie, and the rest are reported as copies set aside. When
the session read has no hit at the key and copies of it were set aside, the refusal names
them with their folders, so --root can point at the one that holds the hit. ruleprobe corpus is the exception: a label names its session by file, so it reads each labelled file
as its own session, copies included. It prints the
ruleprobe corpus command that scores the new negative, with the same --rules, --detectors
and --no-config it was given. The new negative scores as a false positive until the detector
stops firing on it, which is the point.
In a corpus’s sessions/, the suffix .events.jsonl is reserved: a file ending in it is read
as one event per line, never through a runtime reader, so a native transcript must not be
named that way.
A detector of your own scores itself. Rather than a corpus, a declarative detector may
carry an examples: block of minimal cases, and ruleprobe corpus --rules ./docs/rules
scores those:
- id: house-style/sudo-install
rule: house-style
event: tool_use
when:
command: {starts_with: [sudo, pip]}
examples:
fire:
- bash: sudo pip install ruff
skip:
- bash: uv pip install ruff
note: the tool the rule asks forfire is a list of cases the detector should fire on and skip a list it should not. A
case is bash: <command>, or event: <one event>, or events: [...] for a session
detector, with an optional note. Nothing runs them at report time. A detector with no
examples and no corpus label prints no examples rather than a number, and the floor steps
over it: an unmeasured detector is a gap to see, not a failure to fix.