perf(roms): pick a random rom without paging to a random offset

[Spinnich]

Description

Fixes #4066.

The v2 Home "Random Pick" widget resolved a pick by paging to it: one request to
learn the library total, a second at offset: <random>. An offset makes the
database walk and discard every preceding row, so the cost grew with the library.
On an 83k-rom instance a pick took 1.7s median and up to 6.9s, and the widget
runs one on every Home mount.

This adds GET /api/roms/random, which samples on the primary key instead.

Two mechanisms, both uniform:

1. Batch sampling (the fast path). Sixteen random primary keys are offered to
   the filtered query in a single IN (...). Every id in the table is equally
   likely to be offered, so any hit is an unbiased pick, and the batch costs one
   index lookup per candidate no matter how large the library is. Bounds come
   from MIN/MAX over the untouched primary key (two index seeks) rather than
   over the filtered set, which would be a scan of it.
2. Uniform offset fallback. A batch misses when the query matches too little
   of the id space (a narrow scope, or an id range left sparse by deletions). It
   then counts the set and takes the row at a random position. The statement
   selects nothing but the id and carries no ORDER BY, so it reads index
   entries rather than rows.

Measured on a real 83,132-rom library (read-only, production data):

	before	after
unscoped pick, median	1.70 s	2.9 ms
unscoped pick, worst of 30	6.90 s	5.7 ms
fallback path (0.58% of unscoped picks)	-	236 ms
scoped to a large platform	9.20 s	19-25 ms

Distribution over 200k draws against that same id space: 75,495 of 83,132 roms
drawn (exactly what uniform predicts for that many draws), stdev 1.55 against a
Poisson expectation of 1.55.

The endpoint accepts the platform/collection/virtual/smart scope params, so the
gallery "Random ROM" buttons tracked in #4068 can move onto it without a
signature change. Rewiring those call sites is deliberately left out of this PR.

Files modified

• backend/handler/database/roms_handler.py - new get_random_rom_id(query)
  implementing both mechanisms, plus the RANDOM_ID_SAMPLE_SIZE constant. Uses
  secrets rather than random.
• backend/endpoints/roms/__init__.py - new GET /random route returning
  SimpleRomSchema | None. Declared before /{id} so the int path param doesn't
  swallow it. Scope params and permission filtering mirror GET /api/roms, and
  the picked row is re-checked against the caller's permissions after the fetch
  (see reviewer notes).
• backend/tests/endpoints/roms/test_random.py - new, 11 tests covering both
  mechanisms (see testing notes).
• frontend/src/services/api/rom.ts - new getRandomRom() wrapper.
• frontend/src/v2/components/Home/Widgets/RandomPickWidget.vue - reroll() now
  makes one request. Drops pickOnce(), the PICK_QUERY opt-outs and the
  retry-on-drift branch, all of which existed only to work around the two-call
  approach.
• frontend/src/v2/components/Home/Widgets/RandomPickWidget.test.ts - new, 4
  tests.
• docs/BACKEND_ARCHITECTURE.md - endpoint table entry.

No migration, no schema change, and no regenerated types: the route reuses
SimpleRomSchema.

Testing notes

• Full backend suite: 2715 passed, 2 skipped. The new file is 11 tests,
  covering auth, an empty library, platform/collection scoping, admin-hidden roms
  staying out of the pick, post-fetch visibility, and both code paths reaching
  every rom in the set. Two mechanism guards: one asserts get_rom_count and
  get_rom_id_index are never called and that no statement carries an OFFSET;
  the other patches the sample size to 0 to force the fallback.
• Frontend: 4 new tests (single request per pick, empty-library copy, error copy
  on first load, reroll keeping the previous pick on failure). Full vitest run
  green.
• Verified in the browser in both themes: pick renders, reroll reshuffles in
  place without navigating, keyboard focus survives the reroll.
• trunk fmt && trunk check clean.
• Timings and the distribution check above were run against a production
  database over a read-only account, SELECT/EXPLAIN only.

What reviewers should look at

• Uniformity of the batch path. The argument is exchangeability: the sampling
  procedure treats every existing id identically, so conditional on a non-empty
  hit set each is equally likely to be returned. Worth a second opinion.
• Scoped picks mostly take the fallback. Sampled keys rarely land inside a
  narrow scope, so [Bug] "Random ROM" button in platform and collection galleries scans the entire library on every click #4068's call sites will pay the count-plus-offset path. That
  measured 19-25 ms on large platforms, but it is not the constant-time path and
  shouldn't be described as one.
• The fallback deliberately has no ORDER BY. Adding ORDER BY roms.id back
  looks harmless and is not: a PK-ordered scan is the clustered index, so it
  drags whole rows off disk. That measured 4.6 s against 0.067 s for the same
  query without it. Uniformity doesn't depend on the ordering, only on each row
  appearing exactly once.
• RANDOM_ID_SAMPLE_SIZE = 16 is tuned for an id space roughly a quarter
  occupied, which is what deletions leave behind on a long-lived instance
  (mine is 27% dense). Happy to change it if that assumption seems wrong.
• Post-fetch visibility re-check. The id is chosen by a permission-filtered
  query but the row is fetched by raw id, so the loaded row is re-tested with
  perms.can_see_rom(...), matching what every other raw-id ROM lookup does via
  assert_rom_visible. It returns null rather than raising 404, to keep a
  hidden ROM indistinguishable from an empty scope and to keep an error path out
  of a contract that otherwise has none.
• Race on delete. If a ROM is deleted between the id pick and the fetch, the
  endpoint returns null and the widget briefly shows its empty-library copy.
  I judged a third round trip not worth avoiding that.

Checklist

• I've tested the changes locally
• I've updated relevant comments
• I've assigned reviewers for this PR
• I've added unit tests that cover the changes

AI assistance

This PR was written primarily by Claude Code, including this description and the
replies on the review thread. I directed the work, required that the approach be
measured against my own production database before it was accepted, and reviewed
the result.

Two things it got wrong that the process caught, noted so reviewers can calibrate
how much scrutiny to apply. The first implementation sampled a random point in
the id range and took the nearest row at or above it, which is only uniform on a
dense id space; on my 27%-dense library it skewed 10,840x toward one ROM and
could reach only 2,773 of 83,132 ROMs. It was replaced with the batch-sampling
mechanism above and re-validated the same way. Separately, the first version of
the visibility regression test passed with the fix reverted, because the query
returned empty before control flow ever reached the guard; it was rewritten to
actually exercise it.

[greptile-apps]

Greptile Summary

The PR replaces the Home widget's count-and-random-offset workflow with a dedicated random-ROM endpoint.

• Samples primary-key candidates first and falls back to a count plus narrow ID offset query.
• Supports platform and collection scopes while applying user visibility filters.
• Adds a typed frontend API wrapper, rewires the v2 widget, and adds backend and component tests.

Confidence Score: 4/5

The visibility-changing race should be fixed before merging because it can return a ROM after it becomes hidden from the requester.

The random selection logic and frontend integration are otherwise coherent, but the endpoint crosses two transactions and returns the second lookup without enforcing the visibility boundary on the object actually serialized.

Files Needing Attention: backend/endpoints/roms/init.py

Security Review

The selected ROM is fetched in a second transaction without rechecking visibility, allowing a concurrent permission or platform change to expose a newly hidden ROM. How this was verified: The permission-filtered ID selection is followed by a separate raw-ID lookup, while the existing simple-ROM endpoint explicitly validates visibility after fetching.

Important Files Changed

Filename	Overview
backend/endpoints/roms/init.py	Adds the protected random-ROM route and scope filtering, but the second lookup does not revalidate visibility.
backend/handler/database/roms_handler.py	Adds primary-key batch sampling with a uniform-offset fallback; the checked query transformations preserve one row per ROM.
backend/tests/endpoints/roms/test_random.py	Covers authentication, filtering, hidden ROMs, empty scopes, reachability, and both selection paths, but not visibility changes between selection and fetch.
frontend/src/services/api/rom.ts	Adds a correctly shaped wrapper for the new nullable random-ROM endpoint and optional scope parameters.
frontend/src/v2/components/Home/Widgets/RandomPickWidget.vue	Replaces the two-request offset workflow with one endpoint call while preserving loading, error, and focus behavior.
frontend/src/v2/components/Home/Widgets/RandomPickWidget.test.ts	Exercises initial success, empty and error states, and failed rerolls retaining the previous pick.
docs/BACKEND_ARCHITECTURE.md	Documents the new protected random-ROM endpoint.

Prompt To Fix All With AI

### Issue 1
backend/endpoints/roms/__init__.py:949
**Visibility Lost Between Lookups**

If a ROM becomes hidden or moves to a hidden platform between ID selection and `get_rom_simple`, the unfiltered second lookup returns it without revalidating visibility, exposing ROM metadata the requester is no longer permitted to see.

**How this was verified:** The permission-filtered ID selection is followed by a separate raw-ID lookup, while the existing simple-ROM endpoint explicitly validates visibility after fetching.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "perf(roms): pick a random rom without pa..." | Re-trigger Greptile}

[greptile-apps]

Visibility Lost Between Lookups

If a ROM becomes hidden or moves to a hidden platform between ID selection and get_rom_simple, the unfiltered second lookup returns it without revalidating visibility, exposing ROM metadata the requester is no longer permitted to see.

How this was verified: The permission-filtered ID selection is followed by a separate raw-ID lookup, while the existing simple-ROM endpoint explicitly validates visibility after fetching.

Knowledge Base Used: Backend API Endpoints

Prompt To Fix With AI

This is a comment left during a code review.
Path: backend/endpoints/roms/__init__.py
Line: 949

Comment:
**Visibility Lost Between Lookups**

If a ROM becomes hidden or moves to a hidden platform between ID selection and `get_rom_simple`, the unfiltered second lookup returns it without revalidating visibility, exposing ROM metadata the requester is no longer permitted to see.

**How this was verified:** The permission-filtered ID selection is followed by a separate raw-ID lookup, while the existing simple-ROM endpoint explicitly validates visibility after fetching.

**Knowledge Base Used:** [Backend API Endpoints](https://app.greptile.com/romm/-/custom-context/knowledge-base/rommapp/romm/-/docs/backend-api-endpoints.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[Spinnich]

Thanks @greptileai, the second half of this is a real gap. Splitting it because only one of the two mechanisms holds:

The permission-change race isn't reachable. get_permissions() caches the resolved permissions on request.state (handler/auth/dependencies.py:35-42), so the filtered ID selection and any post-fetch check read the same snapshot for the life of the request. A hide applied concurrently is invisible to both, and re-checking after the fetch closes nothing there.

The platform-move case is real. can_see_rom(rom.id, rom.platform_id) tests platform_id from the row that was actually loaded, whereas filter_roms matched on its value at SELECT time. A ROM moved onto a hidden platform between the two statements would be selected under its old platform and serialized under its new one.

The stronger argument for fixing it is consistency rather than the race: every other endpoint that resolves a ROM by raw ID calls assert_rom_visible, and this one was the exception, so its safety was only provable by tracing back to the filter several lines up. The check costs one in-memory set lookup and no query, so there's no reason not to have it.

Fixed in 7bbf94c. I used perms.can_see_rom(...) returning null rather than assert_rom_visible, deliberately: the endpoint's contract is SimpleRomSchema | None with no error path, and returning null keeps a hidden ROM indistinguishable from an empty scope, which is the same masking assert_rom_visible's 404-instead-of-403 exists to provide.

Added test_get_random_rom_rechecks_visibility_after_fetching, confirmed to fail with the guard reverted. Worth noting the first version of that test passed either way: the viewer's only ROM was the hidden one, so the filtered query came back empty and the endpoint returned null before ever reaching the fetch. It needed a second, visible ROM so control flow actually reaches the guard.

This reply was drafted by Claude Code, as was the PR itself.

[greptile-apps]

The analysis is correct on all counts. Confirmed from dependencies.py:35-42:

def get_permissions(request: Request) -> ResolvedPermissions:
    cached = getattr(request.state, "permissions", None)
    if cached is not None:
        return cached
    perms = resolve_permissions(request.user)
    request.state.permissions = perms
    return perms

Both the filter query and the post-fetch check resolve through the same cached ResolvedPermissions snapshot, so a permission change mid-request is invisible to both. The platform-move case is the real one: filter_roms operates on whatever platform_id the row had at SELECT time, while the post-fetch can_see_rom(rom.id, rom.platform_id) reads the freshly loaded value. A move to a hidden platform in between would pass the filter and fail the check, which is exactly what the guard catches.

The test rewrite is the right call. The original passed either way because the query returned empty before the fetch — patching get_rom_simple to return the hidden rom directly is the only way to isolate the guard from the filter and actually exercise it. The "visible decoy" setup (a second visible rom so the ID pick resolves) makes the test's precondition clear.

null over assert_rom_visible follows naturally from the contract: SimpleRomSchema | None with no error path, and leaking a hidden rom's existence via a 404/403 distinction is exactly what the 404-masking on assert_rom_visible exists to prevent anyway. This keeps it consistent.

The commit looks good.