Skip to content

Batch Vale invocations so a timeout costs one batch, not the whole run #324

Description

@thecodedrift

runVale spawns Vale once over every target. Vale buffers its entire result and prints one JSON document at the end, so killing it on timeout loses every finding the run had produced. A size cap (#321) prevents the one cause we can predict; batching would bound the damage from the ones we cannot.

Not urgent. #321 ships the prospective guard, and this is the better long-term shape rather than a fix for a live defect.

Vale emits nothing until it finishes

Measured directly. 600 markdown files, one existence rule, killed 400ms into a run that takes longer than that:

killed mid-run after 400ms
  stdout chunks received: 0
  bytes buffered: 0
  partial output does NOT parse -> nothing emitted at all

Not truncated JSON — nothing at all. So there is no partial result to salvage, and a timeout can never preserve findings. It is only a ceiling on damage, which is what VALE_TIMEOUT_MS's own docblock already says: "a ceiling on damage, not a performance target."

Why the size cap is not the whole answer

#321 caps a single file at 128KB because size is the only signal available BEFORE running. That works, and it is the right prospective control, but it has a real limit: the cap is machine-independent while the cost is not. Measured on an M-series laptop, one rule:

size duration
128KB 0.77s
256KB 3.30s
640KB 14.3s
1MB 48.5s

A CI runner several times slower changes what that byte number buys, and the cap cannot adapt. It also only predicts the ONE cause it was written for. A runaway script check, a pathological rule, or a hung binary still costs the whole run, and no size cap sees those coming.

The shape

Run Vale over chunks of the target set rather than all of it, so a timeout costs one chunk's findings instead of everything.

  • A huge file costs its batch; a batch of one costs only itself, which is what the size cap approximates but adaptively and without a magic number.
  • It degrades on any hardware rather than against a fixed byte count.
  • It covers causes a size cap cannot predict, since any slow batch is bounded the same way.

Cost is one extra spawn per batch. Measured, a Vale spawn on a small corpus is ~46ms, so 600 files in batches of 100 adds roughly 300ms. Batched throughput is otherwise good: 400 files of ~2KB run in 190ms total.

Questions to settle before building it

  • Batch by count, by total bytes, or by predicted cost? Given the quadratic, total bytes is a poor predictor and one large file dominates its batch regardless. Sum-of-squares would model it better; count is simplest.
  • What does a timed-out batch report? Each of its files needs to surface as unchecked, which is the same reporting seam One large markdown file blocks the whole Vale run, because Vale is quadratic in file size #321 added for oversized files (notices, not findings, because a preemptive exclusion cannot confirm rule scope).
  • Does the per-batch budget divide the existing 60s, or does each batch get its own? Dividing keeps the total bounded; per-batch does not, and a pathological corpus could then run for a long time.
  • Interaction with One large markdown file blocks the whole Vale run, because Vale is quadratic in file size #321's cap. If batching lands, is the size cap still needed? Probably yes as a fast path, since it avoids spending the time at all rather than spending and discarding it — but that is worth re-deciding with the numbers rather than assuming.
  • Ordering and determinism. Findings must come back in a stable order regardless of how they were batched, or output churns between runs.

Not this

Running one Vale invocation per file. Measured at ~46ms of spawn overhead each, 400 files becomes ~18s against 190ms batched. The batching has to be coarse to be worth doing.

Refs #321

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CLIRelated to the taskless CLI

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions