Make BGE reranking bounded and portable

[tandede]

Summary

• enforce the documented k contract after sorting BGE reranker scores
• batch query-document inference with configurable batch_size and max_length
• make device selection portable across CUDA, Apple Silicon MPS, and CPU
• retain full precision on CPU/MPS while using FP16 only on CUDA
• fail fast on invalid limits/configuration and on model-output cardinality mismatches
• replace the machine-specific model test with deterministic, hermetic coverage

Why

BgeReranker.rerank() accepted k (default: 5) but returned the complete ranked list. This affects the README's RAG flow directly: it retrieves 10 candidates and calls the reranker with its default limit, so all 10 documents previously continued into generation instead of the best five.

The inference path also tokenized every candidate in one call and unconditionally converted the model to FP16. Large retrieval sets could therefore cause avoidable memory spikes, while CPU execution used a precision mode that is not broadly supported. This is especially relevant because the README documents a local Python installation rather than a CUDA-only deployment.

This change makes resource use bounded and explicit:

• candidates are processed in batches while preserving global score ordering
• automatic selection prefers CUDA, then MPS, then CPU
• callers can still override the device
• only CUDA models are converted to FP16
• an unexpected number of model scores raises an error instead of silently losing documents through zip() truncation

When is_sorted=False, the existing score-only behavior is preserved: every document is returned in input order.

Compatibility

The original positional arguments (model_name_or_path, api_key, and url) keep their order. New options are appended with conservative defaults (batch_size=32, max_length=512, automatic device selection).

Validation

• provisioned Python 3.12 and followed the README source-install path with platform-appropriate PyTorch
• pip check — no broken requirements
• python -m pytest -q tests/units/test_reranker.py — 18 passed
• flake8 tests/units/test_reranker.py — passed
• real CPU smoke test with cross-encoder/ms-marco-TinyBERT-L2-v2 — correct top-1 result with batch_size=1
• full repository suite — 18 passed, 1 skipped, 1 pre-existing failure in RewriterApp because HyDE_rewriter is undefined

The tests cover exact top-k ordering, score-only behavior, empty input, batch boundaries, token length forwarding, configuration validation, CUDA/MPS/CPU precision behavior, MPS auto-selection, and model-output shape mismatches without downloading a model during CI.

[tandede]

Hi maintainers, just following up on this PR in case it was missed. It is ready for review, and I’d be happy to address any feedback or make further adjustments. Thank you!