Detectors

Sixty seconds on a rule of your own

The six above are generic. Your rules are not, but a rule in a common shape needs no detector at all. Point --rules at your rule files and a section that no front matter binds is matched against a small shipped catalog by what it says, with no model: its heading and each sentence of its text are tested against one anchored pattern per entry, and a section that matches exactly one entry is measured by that entry’s detector. In a clone:

ruleprobe report --root docs --rules tests/fixtures/catalog

binds a file holding one section in each shape, and lists each section as measured with the note catalog-bound, <detector id>. The shapes are: run the tests before finishing (testing/test-after-change), never skip pre-commit hooks (verification/no-verify), never force-push the default branch (git-safety/force-push-default), use uv, not pip (package-manager/pip-install), do not read a whole file (transcript-hygiene/whole-file-cat), Conventional Commit subjects (commits/non-conventional-subject), and never commit a secret-shaped file (secrets/secret-file-add).

One row reads the other way round. Every other row counts violations, so fewer hits is better; testing/test-after-change counts compliance - a hit is a file change that a later test run followed - so more is better, and its opportunities and followed count are the measure.

The binding under-counts on purpose. A section matching none, or more than one, stays unmeasured, and the coverage block names the entries when it matched several. A pattern never reaches across a clause break (;, ,, :, a run of hyphens between spaces such as - or --, a dash), except the comma of its own contrast (“use uv, not pip”). A rule carrying an exception or a permission anywhere, its heading included - except (excepted, exception), unless, other than, apart from, excluding, allowed, fine, okay, ok - binds nothing, so “Never force-push to main. Hotfixes excepted.” is left unmeasured rather than read as a rule it is not. A condition or a contrast - if, when, but, however, without - unbinds a rule only in the sentence the pattern matched: “Never force-push to main when others share it” binds nothing, while “Run the tests before finishing. If one fails, fix it.” binds. only is not a marker: it intensifies as often as it narrows. A catalog detector joins the report only when a rule binds it, a detector of your own with a catalog id replaces it, ruleprobe corpus scores every entry, and ruleprobe detectors lists every entry marked catalog.

One place it over-counts, knowingly. A file with no heading, or none of whose sections is a rule, is one rule and binds by its whole text: one matching sentence marks the whole file measured, and its other sentences may say “if” or “when” as they like, so the measured share counts it as one measured rule while the detector measures one sentence of it. The exception and permission words still apply to the whole text, so such a file that says “fine” or “unless” anywhere stays unmeasured - in a long file, the likelier outcome.

A catalog entry measures the shape it names and no more. “Never commit secrets” binds secrets/secret-file-add, which counts git add of a secret-shaped file; a secret written inline into a file or a command is not what it counts.

Any other rule takes no Python either: write a detector beside it as data. This is a real run over the example transcript and the example rules in this repository, so it is reproducible from a clone:

cat docs/rules/house-style.md

---
rule: house-style
detector:
  id: house-style/sudo-install
  event: tool_use
  when:
    command: {starts_with: [sudo, pip]}
---

# House style

Install with `uv`, never with `sudo pip`.

ruleprobe report --root docs --rules docs/rules

detector                                 hits  sessions    of    share  note
----------------------------------------------------------------------------
cache-hygiene/compact                       0         0     1       0%
cache-hygiene/model-switch                  0         0     1       0%
house-style/sudo-install                    2         1     1     100%
secrets/secret-in-write                     0         0     1       0%
transcript-hygiene/unfiltered-find          0         0     1       0%
transcript-hygiene/whole-file-cat           0         0     1       0%
verification/no-test-run                    1         1     1     100%
verification/no-verify                      0         0     1       0%

rules: 2 measured, 1 dark, 1 unmeasured (50% measured)
  measured   house-style                    docs/rules/house-style.md
  dark       secrets                        docs/rules/secrets.md: a credential that never reaches a file leaves no shape in a transcript
  measured   verification                   docs/rules/verification.md
  unmeasured working-style.md#working-style docs/rules/working-style.md

Three lines of that report are the point. house-style/sudo-install is a rule of the reader’s own, firing. secrets is dark by choice: its front matter carries opt_out: <reason>, because the rule is about a credential that never reaches a file and a transcript only shows what did. working-style is unmeasured: it has neither a detector nor an opt-out, so it is split at its headings and its one section is listed by its id, working-style.md#working-style; saying so is the only way an author sees the gap. Nothing fails; a report is evidence, not a gate.

Point --rules at whatever directory your own rules live in, and drop the same entries into .ruleprobe/detectors.yaml at the root of a repository, or into ~/.config/ruleprobe/detectors.yaml for the ones you want everywhere. Both are found without a flag; --no-config skips them.

Writing a detector

An entry is id, rule, event, when, and an optional gate. event is one of tool_use, assistant_text and session, and it says what a hit is counted against. when is a matcher: a mapping in which every key must hold, composed with any, all and not.

An entry may also carry schema_version, the integer schema it was written under; this release reads 1 and 2. An entry without one takes its detectors file’s top-level version, and a file without one is schema 1. Only a detectors file has a top-level version; a detector: entry in rule-file front matter has no file-level default, so without its own key it is schema 1. In a detectors file the file-level key is version: a top-level schema_version is a finding, as a bad version is. An entry’s own key wins over the file’s. A value this release does not know - newer than 2, below 1, or not an integer ("2", 2.0 and true included) - is a finding with its file and line, and only the entries that would be read under it are skipped, so an entry written for a later schema is refused rather than counted wrong.

version: 2
detectors:
  - id: house-style/sudo-install
    when:
      command: {starts_with: [sudo, pip]}
  - id: house-style/old-entry
    schema_version: 1
    when:
      command: {name: npm}

A tool use. The shape is a command, a tool name, or an argument:

- id: house-style/wide-grep
  rule: house-style
  event: tool_use
  when:
    command:
      name: grep
      none_of: [-n, --include, --exclude]
      arg_count: {max: 1}

An assistant message. One hit per message the pattern matches:

- id: voice/hedged-verdict
  rule: voice
  event: assistant_text
  when:
    message:
      role: assistant
      final: true
      regex: "(?i)(should (now )?work|I think it works)"

A whole session. absent counts a session in which something never happened, which is the only way to measure a rule that asks for something to be done; order counts one event followed by another; change counts a field that differs from the event before it:

- id: verification/no-test-run
  rule: verification
  event: session
  when:
    absent:
      of:
        command: {name: [pytest, tox, nox]}
      scope: session

Over a Bash command the shell parse skipped - empty or missing, longer than 16,384 characters, or one that does not tokenize or parse - the segment keys of command, git and env, and a text read of source: heredocs, are undecided rather than false, so not and absent do not count it. Any one such command in a session leaves the session-scope absent above undecided, and so no hit, whether or not that command was the test run. command: {unparsed: true} on its own matches those commands; combined with a segment key it can never fire.

The matchers, in one list: tool (name, glob), arg (field, regex, path_glob, contains, equals, exists), command (name, starts_with, contains, none_of, arg_count, sole_segment, redirect, unparsed, regex, program, first_operand), git (subcommand, args_any, args_none, token_prefix, arg_regex, config_regex, message_regex), env (name, command), text (source, regex, contains), message (role, final, regex, contains), kind, and the three session matchers order, absent and change. ruleprobe/matchers.py documents each in one line. The shipped six in ruleprobe/detectors/common.yaml use ten of them - tool, arg, command, git, env, text, kind, change, any and all - because that is what those six observables need; message, order, absent and not are exercised by the examples on this page and in tests/, not by a shipped detector.

Three rules about the format worth knowing before you hit them. Every key inside one command block is read against the same pipeline segment, so two constraints on one command belong in one block rather than in an all of two. A session matcher may only be the whole of a session detector’s when, because a hit it produces is not a hit on an event in hand. And a list is always alternatives: regex, contains and path_glob hold when any one of their patterns does. contains is a substring, so contains: no-verify finds the token --no-verify; path_glob is a path, so * stops at a /, ** crosses one, and src/*.py matches the absolute path a transcript actually carries.

The format is a YAML subset, and JSON is the same thing. YAML is not in the standard library and this package takes no dependencies, so ruleprobe/declarative.py implements the subset a detector needs - block and flow mappings and sequences, scalars, quoted strings, comments - and refuses everything else by name and line: anchors, aliases, tags, block scalars, directives, and more than one document in a file. A file named .json is read by the standard library’s JSON parser into exactly the same objects, so a generator can write JSON and a person can write YAML.

A malformed entry is a finding, not a crash. It is printed with its file, its line and its reason, that entry is skipped, and every other detector in the file still runs.

How it is put together

• ruleprobe/events.py - the event schema every reader emits: assistant_text, tool_use, tool_result, user_prompt, compact.
• ruleprobe/readers/ - one module per runtime, turning a transcript into that schema. Claude Code, Codex and Gemini CLI today; a reader is ROOT, transcripts() and read().
• ruleprobe/shell.py - the Bash decomposition every shell detector shares. Compounds, pipelines, heredocs, substitutions and continuations, parsed once per command.
• ruleprobe/registry.py - Detector, Registry, run(). Third-party detectors arrive through the ruleprobe.detectors entry point group or through Registry.add.
• ruleprobe/declarative.py - the YAML subset and the front-matter split, with a line number on every refusal.
• ruleprobe/matchers.py - one entry compiled into the same Detector a Python one builds.
• ruleprobe/detectors/catalog.py - the shipped catalog: common rule shapes as Python literals, each an anchored pattern and a declarative detector with its examples:.
• ruleprobe/rules.py - where detector files live, and which rule files nothing measures.
• ruleprobe/report.py - rows in, text out. A row is a small dict, so a report can be taken over rows you stored months ago rather than over transcripts you still have.

The public API

The declared API is these names, at these import paths. tests/test_contract.py imports and exercises every one, so removing a name or changing a call shape below fails the suite, and a name joins the list only with its test. Every other name in ruleprobe.__all__ is importable but not declared, so it may change in any release.

• ruleprobe: Registry, analyse, compile_detector, counts, is_undecided, iter_sessions, load_bundle, report, report_data, run, validity
• ruleprobe.declarative: load
• ruleprobe.detectors.common: DETECTORS, SECRET_PATTERNS
• ruleprobe.events: hit, input_of, text_of
• ruleprobe.registry: Detector, Registry
• ruleprobe.shell: Context, MARKER_RE, MAX_COMMAND, Parsed, SUB_PLACEHOLDER, git_calls, has_redirect, normalise, operands, pipelines, strip_heredocs
• ruleprobe.validity: CorpusError, DEFAULT_FLOOR, Score, below_floor, score_corpus, scores_as_dict, validity_table

Measuring and reporting:

iter_sessions(root=None, runtime="auto", since=None, errors=None)  # -> Session(.id .repo .events)
                                          # one per runtime and id; errors also holds each copy
                                          # set aside: error == readers.COPY, with kept
run(events, stances=None, *, registry=DEFAULT, strict=False, errors=None)
                                          # -> {detector_id: [Hit(id, turn, tool_use_id)]},
                                          # detectors with no hit omitted
Registry([Detector, ...]); Registry.add(Detector(id, rule, event, fn, gate=None))
Registry.from_entry_points("ruleprobe.detectors")
report(rows, by="rule", min_sessions=20, frequent_share=0.30, registry=DEFAULT,
       validity=None, min_opportunities=20)
report_data(rows, by="rule", ...)                       # the same numbers as a dict; --json prints it
validity(registry=DEFAULT, directory=None)              # -> {detector_id: Score(.precision .recall)}
load_bundle(paths=None, rules_dir=None, cwd=None, config=True)  # -> Bundle(.detectors .rules .findings)
compile_detector(spec, path="<spec>", lines=None)               # -> Detector

Writing a detector in Python:

Detector(id, rule, event, fn, gate)       # positional, gate optional; a subclass may add
                                          # __slots__ = () and properties named kind, stance
fn(events, ctx)                           # -> [(turn, tool_use_id), ...]
Detector(..., opportunities=count)        # keyword only; count(events, ctx) ->
                                          # [(turn, tool_use_id, followed), ...], followed
                                          # True, False or None when undecided
ctx.events, ctx.bash, ctx.finals          # a Context: every event, each Bash call as a
                                          # Parsed(.event .command .heredocs .skipped
                                          # .pipelines), final messages
hit(event); hit(event, tool_use_id=False) # -> (turn, tool_use_id or None)
git_calls(parsed, ("commit",))            # yields (segment, subcommand, args)
MARKER_RE.match(token).group(1)           # an index into parsed.heredocs
input_of(event); text_of(value); normalise(command)
pipelines(command)                        # -> [[[token, ...], ...], ...]; [] on an
                                          # unterminated quote
operands(segment); has_redirect(segment); strip_heredocs(command)  # -> (text, bodies)

A gate is None or a (dimension, variants_or_None) pair. A detector’s event is one of bash, write, agent-brief, assistant-final, session, tool_use or assistant_text; a command longer than MAX_COMMAND is skipped and never tokenized. A detector reads the event fields kind, turn, id, name, input, text, final, tool_use_id and tool_name, each on the kinds ruleprobe/events.py documents it for.

Scoring against the corpus:

score_corpus(registry=DEFAULT, directory=None)  # -> {detector_id: Score}; raises CorpusError
Score(detector_id)                              # .detector .scored .precision .recall .add(other)
below_floor(scores, floor); scores_as_dict(scores, floor); validity_table(scores, floor)
declarative.load(path)                          # -> (document, lines)

A predicate from compile_matcher returns true, false, or a falsy undecided value when it could not read its input. A Python caller who negates one itself reads undecided as false and can over-count: compose through the declarative not, any and all, or test the result with is_undecided(value) before negating it. compile_matcher itself is not declared: it needs arguments the package does not declare, so it is not yet a declared way to build a predicate, and is_undecided is declared ahead of it.

These promises are not names:

• SECRET_PATTERNS stays a plain module-level assignment of a literal list in ruleprobe/detectors/common.py, so it can be read by syntax tree without importing, from the source tree or from that member of the wheel. Every pattern compiles with re.
• DETECTORS in ruleprobe.detectors.common is exactly the six generic detectors: cache-hygiene/compact, cache-hygiene/model-switch, secrets/secret-in-write, transcript-hygiene/unfiltered-find, transcript-hygiene/whole-file-cat and verification/no-verify.
• The wheel is pure Python and named ruleprobe-<version>-py3-none-any.whl.
• The package imports and runs from that wheel placed on sys.path as a zip, uninstalled.
• The wheel carries the corpus at ruleprobe/corpus/, beside ruleprobe.__file__. A caller importing from a zip unpacks it, or points RULEPROBE_CORPUS at a copy.
• Its labels.yaml writes every fire and near as a flow list, so a caller can read the labelled ids by pattern. score_corpus(directory=) scores a caller’s own corpus in the same layout, ignoring a top-level key it does not read.

Versioning

0.2.0 versions the contract: the declared API above, the row schema (schema_version on each row and on the report_data result) and the detector-entry schema (schema_version on an entry, version on a detectors file). CHANGELOG.md lists every part of that break and its migration.

• Within a minor series, nothing declared breaks. No declared name, call shape, row schema or entry schema changes incompatibly between 0.2.0 and any 0.2.x. A patch may add; it may not remove or reshape. A detector renamed in a patch ships its fold entry, so rows stored under the old id still count under the new one, and a test fails when a shipped id is neither registered nor folded. The release preflight refuses a patch release whose contract test no longer declares a name its series’ .0 tag declared.
• A later 0.x minor may break the declared surface, with notice. Its changelog section opens with a Breaking heading naming the migration, and the contract test changes in the same release. A tool pinning a minor should treat each minor bump as a possible break: read that heading and re-run its own contract tests before bumping.
• From 1.0, a break is a major.