Benchmark Strategy

Status

Accepted; revised for source-backed regex tasks and separate scanner workloads.

Context

Comparing regex engines requires the same observable work. Extra syntax does not automatically make a useful workload. The earlier suite mixes demonstration expressions, generated documents, flattened grammar scans and scanner-call replays. Its retired reports are available in Git history and cannot establish performance of the revised corpus.

Decision

General regex comparison

production_regex_bench is the source-backed publication candidate. It separates basic expressions, complex expressions and advanced Oniguruma features, without requiring support for every feature to participate in other cases. Selection and input construction precede examination of performance winners.

The catalog records pinned sources/hashes, original patterns, dialect recipes, operation boundaries, fixtures, original-step oracles, expected output and licenses. The corpus guide records retained/replaced cases and coverage limits. Source regression fixtures establish semantics. Production traffic distributions and large-document coverage require additional separately identified inputs.

The core comparators are Ferroni, C Oniguruma, pinned Ruby Onigmo, PCRE2 JIT and interpreter, fancy-regex with seek off/on, and Rust regex. Ferroni's baseline uses the standard public Regex methods. A separate profile uses fallible *_with searches/iteration with SearchOptions::new(); both use standard replacement because there is no fallible replacement API. Both profiles must reproduce the source oracles, and the baseline also receives a fallible preflight check. Each profile's flags, Unicode policy, contexts, required processing and output allocation are explicit. C Oniguruma is compiled at -O3. Other language runtimes can be added as separately identified profiles.

Each task defines its required output: boolean, first span, captures, all captures, replacement or chunk strings. It does not run every operation against every pattern. Each engine must reproduce every source-oracle fixture before timing. Pending candidates, rejected syntax, semantic differences, execution errors and timeouts are distinct; none becomes an ordinary negative match. Unset and empty captures remain distinguishable.

One documented dialect recipe applies to all engines in the main case. Future equivalent-task alternatives have separate IDs/profiles, preserve the output contract, and include preprocessing/postprocessing. Unsupported patterns are not silently simplified for one engine.

Compile, compile-plus-first-use and steady reuse have separate lanes. The first two create fresh patterns in a warmed process, including contexts/JIT, with destruction outside timing. Steady reuse executes the complete fixture batch using a reused object, including output assembly/destruction. These lanes do not measure OS-process startup. Runtime is primary; optional memory rows use a named metric on selected tasks. Whole-process peak RSS is not compiled-pattern size.

TextMate Scanner and full highlighting

TextMate Scanner is a separate application workload for the API Ferroni provides. Its replays keep complete captured document calls, including failed calls and original pattern order. Publication candidates are four of them, one per complexity tier: JSON, Rust, TSX and HTML. Ferroni is a regex engine first; this lane shows its scanner against the other engines and guards against regressions, while highlighting breadth belongs to Ferriki. The C, Java, PHP, C++ and SCSS replays and the hot scanner groups remain local diagnostics. The Astro replay remains a local regression replay for issue #327: no comparison engine reproduces it. Each replay has a document lane, which reuses scanners compiled before timing, and a cold_document lane, which compiles every scanner of the trace and replays the document once, with the compiled scanners dropped outside timing.

Every engine has the call boundary of Ferroni's find_next_match_utf16. Input preparation stays outside timing: trace parsing, validation, the subjects with their UTF-16/UTF-8 offset tables (OnigString for Ferroni, the same tables for the comparators) and fresh string identities for each replay. Inside each timed call the engine converts the UTF-16 start through those tables, searches, and materializes the winning pattern index with every capture in UTF-16 code units, inline for up to eight groups as in ScannerMatch. Engines that search pattern by pattern keep each candidate's captures in their own match storage and read only the winner's, as vscode-oniguruma does. The result is consumed and dropped inside timing.

The remaining differences are each engine's own search, named in every report: Ferroni's DFA pre-filter and position-lead set search; the vscode-oniguruma C scanner's RegSet search, which every captured line takes because none reaches 1000 bytes, so its per-pattern string cache does not apply; an earliest-match loop over the patterns for Onigmo, PCRE2 and fancy-regex; and fancy-regex's RegexSet. Each engine is called as its client calls it: Ferroni without a caller cache ID, as Ferriki does, the C scanner with vscode-oniguruma's string cache IDs. Shiki's JavaScript engine runs in Node under another harness and remains a local diagnostic.

An engine is timed only where it reproduces every captured result, including the difference between an empty and an unset capture; otherwise the report gives the reason. Ferroni and C Oniguruma must reproduce every call. compare-scanner-replays.py follows the measurement rules below with one process per case and engine, and the manual Blacksmith workflow runs it as a separate lane. Its per-platform, per-case reports have no cross-case aggregate and do not enter the regex ranking or the website.

Full grammar execution and highlighting belong to Ferriki's separate benchmark. So do effects of the consumer's build: Ferriki's TypeScript/TSX regression with 2.2.1 came from register allocation in its addon and did not show in the TSX replay. A flattened expression list with deduplication or compile failures filtered out is not full TextMate execution. Its old measurements remain engine diagnostics.

Internal diagnostics

regression_bench and compiler/executor/scanner microbenchmarks remain internal regression tools. battle_bench remains an internal diagnostic tool. Synthetic demonstrations can test mechanisms without being presented as production workloads. Superseded measurement pages and stored result artifacts are removed from the current tree; Git history retains them.

Measurement and publication

Use the existing Criterion harness and thin LTO release/bench profiles. The compare-production.py runner starts with at least three independent processes per engine/case/platform, 30 samples per lane and three seconds measurement. It uses seeded case order and rotated/reversed engine order. Retain logs, raw samples, intervals, eligibility reasons, corpus, Cargo lock, C source pins, binary hash, flags, toolchain and host context. Short runs require --diagnostic. Expand or repeat unstable comparisons before publication.

Every ratio/aggregate names its common task set. Families have equal weight regardless of variant or fixture count. Different supported subsets are not an overall ranking. ARM and x86 are separate reports. Inconsistent run intervals are inconclusive; this conservative screening rule is not a cross-process statistical significance test. Review host load, power and thermal conditions and raw evidence before publication.

Published results name the measured commit. Their checks verify the retained source snapshot, raw samples, and generated tables after later engine changes; they do not certify the current revision's speed. Publishing a new run still requires its library source to match the current tree. Local improvements require new runner measurements before they change public Blacksmith numbers.

Consequences

• New general-purpose claims require revised contracts and retained repeated-run evidence. Retired numbers cannot stand in for the new corpus.
• Native C build inputs remain pinned in benches/battle_inputs.toml and prepared by the existing scripts; ffi/onigmo require a C compiler.
• Scanner and highlighter measurements have their own scope and publication.
• Homepage, README and result pages describe the current workload methodology. New claims require new reviewed evidence.