Skip to content

Benchmarks

This page records the current PyBetterleaks benchmark results and the commands used to reproduce them. The numbers are synthetic and local, but they are useful as a baseline for release checks and future performance work.

Current Generated Snapshot

The checked-in table below is generated from benchmarks/bench.py output. Do not edit the marked block by hand.

The benchmark workflow refreshes this same section inside CI artifacts on every pull request, main push, and release/** push. To commit a fresh snapshot to a branch, run the workflow manually with update_docs=true.

PyBetterleaks Benchmark Results

  • Betterleaks: v1.6.1
  • Python: CPython 3.13.9
  • Platform: macOS-15.0.1-arm64-arm-64bit-Mach-O
  • Warmups: 1
  • Rounds: 3
Case Fixture Mean Median Min Max
pybetterleaks.scan_text one synthetic secret 0.40 ms 0.24 ms 0.15 ms 0.82 ms
pybetterleaks.scan_dir 25 files, 2 secrets per file 1.39 ms 1.35 ms 1.25 ms 1.57 ms
python subprocess scan_text one synthetic secret 62.31 ms 62.27 ms 61.57 ms 63.09 ms
python subprocess scan_dir 25 files, 2 secrets per file 63.14 ms 62.40 ms 62.11 ms 64.89 ms

Benchmark Fixture

Synthetic Rule

The benchmark uses a typed config with one custom rule:

Rule(
    id="pybetterleaks-bench",
    description="Synthetic PyBetterleaks benchmark rule",
    regex=r"PYBETTERLEAKS_BENCH_[A-Z0-9]{16}",
    keywords=["PYBETTERLEAKS_BENCH_"],
)

The fixture secret is PYBETTERLEAKS_BENCH_0123456789ABCDEF.

Reproduce

Build the native bridge before running benchmarks:

uv sync --all-extras --dev
uv run python scripts/build_native.py

Run the same synthetic benchmark used by CI:

uv run python benchmarks/bench.py \
  --subprocess-wrapper \
  --rounds 3 \
  --warmups 1 \
  --files 25 \
  --secrets-per-file 2 \
  --json-output benchmark-results/checkout-benchmark.json \
  --markdown-output benchmark-results/checkout-benchmark.md

uv run python scripts/update_benchmark_docs.py \
  benchmark-results/checkout-benchmark.md \
  --docs docs/benchmarks.md

Expected output shape:

files=25 secrets_per_file=2
warmups=1 rounds=3
pybetterleaks.scan_text: mean=...
pybetterleaks.scan_dir: mean=...
python subprocess scan_text: mean=...
python subprocess scan_dir: mean=...

To compare against a local Betterleaks CLI, install the CLI and add --cli:

brew install betterleaks
uv run python benchmarks/bench.py --cli --cli-path /path/to/betterleaks

CI uploads JSON, Markdown, and a refreshed docs/benchmarks.md snapshot from benchmark-results/. It also builds a local wheel and runs a tiny wheel-installed benchmark smoke test from a temporary virtual environment.

The workflow does not commit benchmark changes on every push because benchmark numbers naturally move a little. To intentionally update the checked-in docs, run the Benchmarks workflow manually on a release branch with update_docs=true, then review the resulting commit or pull request.

How To Read These Numbers

These results should be read as SDK integration measurements, not as proof that PyBetterleaks makes the Betterleaks engine itself faster.

PyBetterleaks runs inside the already-started Python process:

Python process
  -> ctypes call into already-loaded Go shared library
  -> Betterleaks scan
  -> JSON response
  -> Python dataclasses

The subprocess-wrapper baseline starts a new Python process for each measured scan:

Python benchmark process
  -> subprocess starts python
  -> Python imports pybetterleaks
  -> Betterleaks scan
  -> Python parses results
  -> process exits

For small scans, process startup and import/setup work are a large share of total time. That is the core PyBetterleaks use case: Python services, CI helpers, notebooks, agent tools, and repeated scans that should call an importable SDK instead of spawning a fresh command each time.

scan_text mostly measures Python-to-native boundary overhead, request JSON serialization, config handoff, Betterleaks setup, and result parsing for one small input.

scan_dir is the more useful SDK-level benchmark. It covers fixture creation outside the measured path, then measures scanning a directory with synthetic files and parsing the returned JSON into typed Python dataclasses.

When --cli is enabled, betterleaks dir is the upstream CLI baseline. It includes process startup and CLI output generation, but it does not include Python JSON parsing or dataclass construction. The checked-in CI-sized snapshot uses the Python subprocess wrapper because the Betterleaks CLI is not installed by default in the benchmark workflow.

Prefer medians when comparing these small, sub-100ms command timings; single samples can move around on busy developer and CI machines.

The earlier 500-file benchmark exposed a separate SDK bug: when callers used a typed BetterleaksConfig, PyBetterleaks generated the TOML config inside a brand-new temporary directory for every scan. Profiling showed that path was much slower than scanning with a stable config_path. The SDK now sends typed configs as inline TOML over the JSON ABI and the Go bridge parses them directly with Betterleaks' config parser.

As scan work grows, PyBetterleaks still pays real costs for Go-to-C JSON serialization, crossing the C ABI boundary, Python JSON parsing, and dataclass construction. Keep using these numbers as a release baseline, not as a universal claim about all repositories or all rule sets.

These numbers are not a security-engine benchmark and should not be presented as a universal Betterleaks performance claim. They are a PyBetterleaks release baseline: the public API stays importable, the bundled bridge stays fast enough for CI and service use, and future changes have a simple regression check.

Next Measurements

  • Add cold-start versus warm-call measurements.
  • Add larger repository-shaped fixtures with mixed file types.
  • Track results across supported wheel platforms once the data is stable.