Detectors

Sixty seconds on a rule of your own

The six above are generic. Your rules are not, and measuring one takes no Python: write a detector beside them as data. This is a real run over the example transcript and the example rules in this repository, so it is reproducible from a clone:

cat docs/rules/house-style.md

---
rule: house-style
detector:
  id: house-style/sudo-install
  event: tool_use
  when:
    command: {starts_with: [sudo, pip]}
---

# House style

Install with `uv`, never with `sudo pip`.

ruleprobe report --root docs --rules docs/rules

detector                                 hits  sessions    of   share  note
---------------------------------------------------------------------------
cache-hygiene/compact                       0         0     1       0%
cache-hygiene/model-switch                  0         0     1       0%
house-style/sudo-install                    2         1     1     100%
secrets/secret-in-write                     0         0     1       0%
transcript-hygiene/unfiltered-find          0         0     1       0%
transcript-hygiene/whole-file-cat           0         0     1       0%
verification/no-test-run                    1         1     1     100%
verification/no-verify                      0         0     1       0%

rules: 2 measured, 1 dark, 1 unmeasured
  measured   house-style                 docs/rules/house-style.md
  dark       secrets                     docs/rules/secrets.md: a credential that never reaches a file leaves no shape in a transcript
  measured   verification                docs/rules/verification.md
  unmeasured working-style               docs/rules/working-style.md

Three lines of that report are the point. house-style/sudo-install is a rule of the reader’s own, firing. secrets is dark by choice: its front matter carries opt_out: <reason>, because the rule is about a credential that never reaches a file and a transcript only shows what did. working-style is unmeasured: it has neither a detector nor an opt-out, and saying so is the only way an author sees the gap. Nothing fails; a report is evidence, not a gate.

Point --rules at whatever directory your own rules live in, and drop the same entries into .ruleprobe/detectors.yaml at the root of a repository, or into ~/.config/ruleprobe/detectors.yaml for the ones you want everywhere. Both are found without a flag; --no-config skips them.

Writing a detector

An entry is id, rule, event, when, and an optional gate. event is one of tool_use, assistant_text and session, and it says what a hit is counted against. when is a matcher: a mapping in which every key must hold, composed with any, all and not.

A tool use. The shape is a command, a tool name, or an argument:

- id: house-style/wide-grep
  rule: house-style
  event: tool_use
  when:
    command:
      name: grep
      none_of: [-n, --include, --exclude]
      arg_count: {max: 1}

An assistant message. One hit per message the pattern matches:

- id: voice/hedged-verdict
  rule: voice
  event: assistant_text
  when:
    message:
      role: assistant
      final: true
      regex: "(?i)(should (now )?work|I think it works)"

A whole session. absent counts a session in which something never happened, which is the only way to measure a rule that asks for something to be done; order counts one event followed by another; change counts a field that differs from the event before it:

- id: verification/no-test-run
  rule: verification
  event: session
  when:
    absent:
      of:
        command: {name: [pytest, tox, nox]}
      scope: session

The matchers, in one list: tool (name, glob), arg (field, regex, path_glob, contains, equals, exists), command (name, starts_with, contains, none_of, arg_count, sole_segment, redirect, unparsed, regex), git (subcommand, args_any, args_none, token_prefix), env (name, command), text (source, regex, contains), message (role, final, regex, contains), kind, and the three session matchers order, absent and change. ruleprobe/matchers.py documents each in one line. The shipped six in ruleprobe/detectors/common.yaml use ten of them - tool, arg, command, git, env, text, kind, change, any and all - because that is what those six observables need; message, order, absent and not are exercised by the examples on this page and in tests/, not by a shipped detector.

Three rules about the format worth knowing before you hit them. Every key inside one command block is read against the same pipeline segment, so two constraints on one command belong in one block rather than in an all of two. A session matcher may only be the whole of a session detector’s when, because a hit it produces is not a hit on an event in hand. And a list is always alternatives: regex, contains and path_glob hold when any one of their patterns does. contains is a substring, so contains: no-verify finds the token --no-verify; path_glob is a path, so * stops at a /, ** crosses one, and src/*.py matches the absolute path a transcript actually carries.

The format is a YAML subset, and JSON is the same thing. YAML is not in the standard library and this package takes no dependencies, so ruleprobe/declarative.py implements the subset a detector needs - block and flow mappings and sequences, scalars, quoted strings, comments - and refuses everything else by name and line: anchors, aliases, tags, block scalars, directives, and more than one document in a file. A file named .json is read by the standard library’s JSON parser into exactly the same objects, so a generator can write JSON and a person can write YAML.

A malformed entry is a finding, not a crash. It is printed with its file, its line and its reason, that entry is skipped, and every other detector in the file still runs.

How it is put together

• ruleprobe/events.py - the event schema every reader emits: assistant_text, tool_use, tool_result, user_prompt, compact.
• ruleprobe/readers/ - one module per runtime, turning a transcript into that schema. Claude Code and Codex today; a reader is ROOT, transcripts() and read().
• ruleprobe/shell.py - the Bash decomposition every shell detector shares. Compounds, pipelines, heredocs, substitutions and continuations, parsed once per command.
• ruleprobe/registry.py - Detector, Registry, run(). Third-party detectors arrive through the ruleprobe.detectors entry point group or through Registry.add.
• ruleprobe/declarative.py - the YAML subset and the front-matter split, with a line number on every refusal.
• ruleprobe/matchers.py - one entry compiled into the same Detector a Python one builds.
• ruleprobe/rules.py - where detector files live, and which rule files nothing measures.
• ruleprobe/report.py - rows in, text out. A row is a small dict, so a report can be taken over rows you stored months ago rather than over transcripts you still have.

The public API is six names:

iter_sessions(root=None, runtime="auto", since=None, errors=None)  # -> Session(.id .repo .events)
run(events, stances=None, *, registry=DEFAULT, strict=False, errors=None)
Registry.add(Detector(id, rule, event, fn, gate=None))
Registry.from_entry_points("ruleprobe.detectors")
report(rows, by="rule", min_sessions=20, promote_share=0.30)
report_data(rows, by="rule", ...)                       # the same numbers as a dict; --json prints it
validity(registry=DEFAULT, directory=None)              # -> {detector_id: Score(.precision .recall .f1)}

plus two for the declarative half:

load_bundle(paths=None, rules_dir=None, cwd=None, config=True)  # -> Bundle(.detectors .rules .findings)
compile_detector(spec, path="<spec>", lines=None)               # -> Detector