Command filter rules

rtok run (and rtok filter --stdin, and the PreToolUse Bash rewrite that calls rtok run) shortens command output with per-family rules before the model sees it. The raw output is always archived first, so every shortened result stays retrievable with rtok expand <id> (lossless by default, D4). Rules never redact: a line carrying an error, warning or secret passes through unchanged.

Three layers merge in this order; a later layer wins per command family:

1. Built-ins — rules/default.toml in the repo (grep, rg, sed, cat, make, curl, npm, pnpm, node, plus the cargo/git/test/tree formatters in code).
2. User file — [plugins.cmd] rules (default ~/.rtok/rules.toml).
3. Drop-ins — every *.toml in [plugins.cmd] rules_dir (default ~/.rtok/rules.d/), merged in file-name order.

A missing file or dir is not an error — that layer is skipped. A malformed file is skipped at runtime (fail open) and reported by rtok config validate, which names every bad file.

Schema

One TOML table per command family. The table name is the command's first word ([grep], [pytest]); unknown families fall back to the default rule (max_lines = 40, head = 10, tail = 10, dedupe on). One exception: a single Bash string that chains 2+ DISTINCT programs with &&, ;, or a newline (T177) is not one family's output — it is matched against [script] instead. A same-program chain (cargo build && cargo test, or with a leading cd/export) keeps using that program's own formatter/rule.

The same engine cuts foreign MCP results behind rtok mcp -- <server argv> (T59.4): a [mcp] section is the rule for every wrapped server's tools/call text block, the default rule when there is none. Only text blocks longer than max_lines of a non-isError result are touched; the raw block is archived first and the cut ends with the same [rtok <id> · N lines · expand: rtok expand <id>] trailer rtok run prints. The saving lands as a cmd / wrap measurement with ref_id = <server>/<tool>:<id>.

Any other field, a wrong type, or broken TOML is malformed. A non-zero exit ignores all of this: the last [plugins.cmd] fail_tail_lines lines (default 80) print verbatim, because a failed command's error is the payload.

Worked example

~/.rtok/rules.d/pytest.toml — keep pytest quiet runs to a pointer, but never lose a failure:

[pytest]
max_lines = 30
head = 5
tail = 15
drop = ["PASSED", "passed"]
keep = ["FAILED|failed|ERROR|error"]
dedupe = true

Before (212 lines of dots and passes), after (12 lines):

tests/test_auth.py::test_login PASSED
tests/test_auth.py::test_refresh FAILED
… 200 lines omitted (expand abc123)
E   assert False is True

Check it before relying on it:

rtok config validate   # names rules.d/*.toml files that do not parse

validate checks the file named on its command line (or the user config) plus the single rules file and every rules.d/*.toml it resolves to. A reported file is skipped at runtime until fixed; everything else keeps filtering.

Families (T50.1)

Measured on the golden fixtures in tests/cmd_golden/ (rtok filter --stdin, Rule vs Rule::default() on the same body; est tokens = bytes/4).

Formatters (T58.5)

Measured on the same golden bodies (formatters::compress, Measurement.kind = formatter vs Rule::default() on the same fixture; est tokens = bytes/4). Each formatter keeps one row per object and returns None on unrecognized output so the rule path stays the fallback.

Grouping (T64.1)

Measured on the golden fixtures in tests/cmd_golden/ (formatters::compress with the built-in rule vs the same rule with group off; est tokens = bytes/4). A family is switched on only when the grouped body is smaller. tree, git status, pytest, and python stay as they are: tree art and the git/pytest formatters are not path-per-line or coded-diagnostic streams, and the T50.1 python traceback does not shrink as NameError ×1.

JSON (T65.2)

A body that serde_json parses as an object or array is rewritten after grouping and before the head/tail cut: null / empty-string / empty-container fields dropped, arrays beyond json_items shown as … +K more, object keys kept, strings longer than json_string cut with their length, one line per top-level key. Unparseable bodies are untouched. Table formatters (kubectl get, docker ps) stand down when the body is JSON so kubectl get -o json reaches this pass. toon stays off the hook path. Raw bytes stay in the archive; a shortened result still carries the expand trailer.

Measured on tests/cmd_golden/{gh,aws,kubectl}_json.in (formatters::compress vs a head/tail cut of the same pretty body; est tokens = bytes/4).