bench: qualify bleeding-edge local inference backends on real agent work

[ngalluzzo]

Problem

Buzz can route agents to arbitrary OpenAI-compatible endpoints, and the Harbor Buzz Orchestra benchmark can swap endpoint deployment configuration. It does not yet produce a backend qualification receipt that makes rapidly changing local inference engines comparable and upstream-debuggable.

On Apple Silicon, backend/model changes can materially improve throughput while also regressing batching isolation, streaming tool calls, reasoning controls, cancellation, prefix caches, or long-running server health. Tokens per second alone is not a sufficient promotion gate for an agent workforce.

Proposed first slice

Add a benchmark-only qualification path that:

• records exact backend, model, drafter, dependency, OS, hardware, and launch configuration provenance;
• exercises an OpenAI-compatible endpoint with focused correctness/concurrency probes before agent work;
• emits a machine-readable, append-only result receipt with raw evidence paths;
• keeps endpoint/backend selection outside Buzz product semantics;
• establishes a Qwen3.8-27B autoregressive MLX baseline before comparing MTP and other serving engines;
• can feed surviving conditions into the existing Harbor Buzz Orchestra workload harness.

Initial acceptance criteria

• A local endpoint condition can be described without hard-coding MLX concepts into Buzz runtime code.
• The qualifier checks basic generation, streaming, reasoning controls, tool-call continuation, cancellation, concurrency isolation, and liveness after failures where supported.
• Unsupported capabilities are recorded explicitly rather than treated as passes.
• Every run emits a JSON receipt pinning revisions and configuration.
• Unit tests cover manifest validation and receipt serialization.
• The first checked-in example documents the source-pinned mlx-vlm autoregressive baseline procedure for Qwen3.8-27B-8bit.

Non-goals

• Selecting a permanent inference engine.
• Adding MLX-specific concepts to the Buzz agent protocol.
• Publishing benchmark claims before repeated real-workload validation.
• Modifying or redistributing model weights.