Skip to content

v0.3 Plan

v0.3 combines the selected future tracks:

  1. Git-first workflows
  2. Streaming scan results
  3. Betterleaks config coverage expansion
  4. Release hardening

The release theme:

Make PyBetterleaks feel production-ready for real repositories, large scans, and maintainable releases.

Release Goals

  • Add repository-aware scanning without requiring users to shell out to Git or the Betterleaks CLI at runtime.
  • Add a streaming result API so large scans can yield findings incrementally.
  • Expand typed config coverage to more stable Betterleaks TOML fields.
  • Improve generated docs, wheel smoke coverage, release artifacts, and benchmark tracking.
  • Keep the no-runtime-subprocess promise for supported platforms.

Non-Goals

  • Do not add musllinux/Alpine support in v0.3. It remains unsupported while Go shared libraries fail under musl with initial-exec TLS resolves to dynamic definition.
  • Do not add provider source wrappers for GitHub, GitLab, Hugging Face, or S3 unless they naturally fall out of the local Git work.
  • Do not replace Betterleaks config semantics with a Python-only rules engine.
  • Do not introduce a runtime dependency unless it removes meaningful maintenance risk.
  • Do not publish source distributions until source builds have a clean, documented Go toolchain story.

1. Git-First Workflows

Proposed API

from typing import Literal

GitScope = Literal["worktree"]
SUPPORTED_GIT_SCOPES = ("worktree",)

from pybetterleaks import scan_git

result = scan_git(
    ".",
    scope="worktree",
    config_path=".betterleaks.toml",
)

Candidate scopes:

  • tracked: scan files tracked by Git.
  • worktree: scan the current working tree, including untracked files.
  • staged: scan staged changes only.
  • diff: scan a commit range or ref pair.

Initial Scope

Start with local repositories:

  • scan_git(path, scope="worktree")
  • clear structured errors for non-repositories and invalid scopes
  • no runtime call to the git executable

Stretch goals:

  • scope="tracked"
  • scope="staged"
  • scope="diff" with base_ref and head_ref
  • remote Git URL scans only if Betterleaks exposes a stable internal source API that avoids a runtime subprocess

Implementation Notes

  • Prefer Betterleaks' own Go source packages where they provide stable Git behavior.
  • If upstream Git APIs are too CLI-shaped or unstable, implement local Git file discovery conservatively in Go and feed file fragments into the existing detector.
  • Preserve scan_dir behavior. Git scanning should be additive.
  • Add fixture repositories under tests/fixtures or e2e/fixtures.

2. Streaming Results

Proposed API

from pybetterleaks import iter_scan_dir

for finding in iter_scan_dir("src", config_path=".betterleaks.toml"):
    print(finding.file, finding.rule_id)

Async shape:

from pybetterleaks import iter_scan_dir_async

async for finding in iter_scan_dir_async("src"):
    print(finding.file, finding.rule_id)

Bridge Shape

Avoid Python callbacks from Go. Use a pull-based native scan handle:

BetterleaksScanStartJSON(requestJSON *C.char) *C.char
BetterleaksScanNextJSON(scanID *C.char) *C.char
BetterleaksScanClose(scanID *C.char) *C.char

The bridge owns a bounded Go channel per active scan. Python repeatedly calls ScanNextJSON, which returns one event:

{
  "type": "finding",
  "finding": {}
}

or:

{
  "type": "done"
}

or:

{
  "type": "error",
  "errors": []
}

Rules

  • scan_text, scan_dir, and scan_git remain collecting APIs.
  • Streaming APIs must close native handles in finally.
  • Cancellation must close the native handle and cancel the Go context.
  • Backpressure is controlled by the bounded channel.
  • If streaming adds too much risk, ship sync iterators first and async iterators second.

3. Config Coverage Expansion

v0.2 intentionally modeled the highest-value config fields first. v0.3 should review upstream docs/config.md again and fill stable gaps.

Candidates:

  • global allowlists if they are still part of the modern Betterleaks config model
  • per-rule allowlists if not deprecated
  • entropy-related fields
  • match-context fields
  • archive/decode/file-size tuning if upstream exposes stable config fields
  • helper constructors for common rule shapes

Acceptance criteria:

  • every added typed field round-trips to Betterleaks TOML
  • invalid combinations fail in Python before native scan where practical
  • native smoke tests prove Betterleaks accepts the generated TOML
  • docs map every Python field to its TOML spelling

4. Release Hardening

Documentation

  • Expand generated API docs with examples for every public function.
  • Add a dedicated Git scanning guide.
  • Add a streaming guide with cancellation examples.
  • Keep benchmark results in docs/benchmarks.md.

CI And Packaging

  • Add wheel-installed benchmark smoke tests for at least one Linux and one macOS job.
  • Attach benchmark output as CI artifacts.
  • Keep CIBW_SKIP: "*-musllinux_*" until Alpine is genuinely supported.
  • Add release-note generation or a release template.
  • Consider SBOM generation and artifact signing after wheel publishing is stable.

Quality Gates

Before tagging v0.3:

  • uv run pytest
  • uv run ruff check .
  • uv run mypy python
  • GOCACHE=/private/tmp/go-cache-pybetterleaks go test ./...
  • GOCACHE=/private/tmp/go-cache-pybetterleaks go vet ./...
  • uv run --group docs mkdocs build --strict
  • Docker E2E on a glibc Python image
  • wheel install smoke tests from built artifacts

Proposed Milestones

Phase 1: Design Spike

  • Inspect Betterleaks' Git source APIs and detector flow.
  • Prototype pull-based streaming in the Go bridge.
  • Decide whether Git scanning can share streaming internals.

Phase 2: Git MVP

  • Add request mode git.
  • Add scan_git.
  • Add local repository fixtures.
  • Add structured errors for invalid repos/scopes.

Phase 3: Streaming MVP

  • Add native scan handles.
  • Add iter_scan_dir and iter_scan_git.
  • Add cancellation and cleanup tests.
  • Keep collecting APIs implemented on top of the stable path, unless streaming becomes clearly better internally.

Phase 4: Config Expansion

  • Re-read upstream config docs.
  • Add the safest missing typed fields.
  • Add docs and native smoke tests for each field family.

Phase 5: Release Hardening

  • Add Git and streaming guides.
  • Add wheel-installed benchmark artifacts.
  • Prepare release notes and checksum artifacts.
  • Confirm musllinux remains skipped and documented.

Open Decisions

  • Should scan_git default to tracked or worktree?
  • Should scan_git(..., scope="diff") be v0.3 or v0.4?
  • Should async streaming ship in v0.3, or only sync streaming plus existing async collect APIs?
  • How much config expansion is worth doing before waiting for real users?
  • Should SBOM/signing be v0.3 acceptance criteria or post-release hardening?