feat: add binary title ID and save ID extraction during scan

[tmgast]

Description

RomM derives a game's identity from its filename during a scan, which is fragile in the cases that matter most. Base, update, and DLC files that do not carry a title ID in their name go uncategorized, Switch retail titles cannot be distinguished without decryption, and save-sync clients have no reliable way to pair a save to the folder an emulator actually creates on disk. This reads the platform-native identifier directly out of the ROM binary during the scan's file pass, using argosy-sigil (a native helper library, MPL-2.0), and stores it as typed columns surfaced in the v2 game details.

Extraction covers PSP, PS1, PS2, PS Vita, Switch, Switch 2, 3DS, Wii, and GameCube, and runs independently of the hash-database gate so it reaches non-hashable platforms like Switch. sigil is an optional dependency: the backend imports it behind a guard, so an image or dev environment built without it skips extraction rather than failing. Switch decryption uses the user's prod.keys from the platform's firmware folder (already scanned before roms); without it, encrypted retail titles degrade gracefully and decrypted dumps still resolve keyless.

For Switch, the NCA program ID alone reports the base title for every content type, so an update is indistinguishable from its base and no version is available. sigil instead reads the content metadata (CNMT), which yields each file's authoritative title ID, content type, and version. Base, update, and DLC are then categorized from the binary content type rather than folder names, an update resolves to its real ...800 id with its version, and DLC to its own id.

Changes

• Per-file identity. Every applicable file gains title_id and save_id; the rom gains a base title_id, save_id, and save_usage. save_id is the on-disk save identity an emulator creates, so clients building save-sync pair saves by folder rather than by guessing from filenames.
• Switch categorization from the binary. RomFileCategory is set from the CNMT content type (application/patch/addon to GAME/UPDATE/DLC), authoritative over folder names when extraction succeeds and falling back to the existing folder logic otherwise. The rom-level base id is derived from the per-file ids.
• PS2 folder-prefix saves. A PS2 game can create several memory-card folders sharing a serial prefix (Champions of Norrath splits into BASLUS-20642SYS and BASLUS-20642RD0), so save_id is the region-prefixed stem and save_usage is folder-prefix; consumers enumerate every folder that starts with it.
• 3DS split saves. The 3DS nests a save under two directory levels (title/<high 8>/<low 8>/), so save_id is the /-separated path (00040000/00033500) and save_usage is folder-split; consumers create the nested folders instead of re-deriving the split from the id.
• Opt-in Switch filename embedding. A new config.yml flag filesystem.embed_switch_title_ids (default off) renames Switch files during a scan to embed the binary id and version (Name [TITLEID][vN].ext) and reconciles the database record. It is idempotent (skips names that already carry a title-id bracket), collision-safe, and Switch-only. This is the binary-driven form of what filename-parsing tools consume.
• Packaging. sigil builds in a dedicated Docker stage (the RAHasher pattern), pinned by commit, and is copied into the venv. The guarded import keeps it optional everywhere else.
• Frontend. Regenerated API types plus the per-file Title ID and Version chips on the game details Files tab; no other hand-written frontend changes.

Tests

• Adapter: content-type and version mapping, absent-binding degradation, per-platform gating.
• Filesystem handler: nested update and DLC extraction, content-type-over-folder category precedence, title_version capture, rom-level base selection and derivation, and 3DS folder-split save_usage mapping.
• Embedding: rename format, idempotent skip, collision skip, non-Switch never renamed, and single-file (Rom.fs_name) versus multi-part (inner RomFile) reconciliation, including that title_version survives the file reconciliation.
• sigil: synthetic in-memory NCA, CNMT, and 3DS (NCSD) unit tests, plus validation against real retail Switch (Moonlighter, Dragon Quest Builders 2), PS2, and 3DS (Dillon's Dead-Heat Breakers) dumps.
• Migrations 0104, 0105, and 0106 (the latter adds folder-split to the saveusage enum) upgrade and downgrade on MariaDB and PostgreSQL.
• Verified end-to-end against real Switch, PS2, and 3DS dumps, including per-file categorization, the 3DS nested save_id, and the embedding rename with database reconciliation.

Known Gaps

• Distinguishing a Switch base from its update requires the CNMT, so loose base and update files in one folder without keys cannot be told apart; the folder layout and keyed extraction cover the supported cases.
• Archived-content members are out of scope; extraction runs on served files, not entries inside archives.

---

AI Disclosure
Planning and implementation supported by Claude Code.

Closes #2526

Checklist

• I've tested the changes locally
• I've updated relevant comments
• I've assigned reviewers for this PR
• I've added unit tests that cover the changes

[greptile-apps]

Greptile Summary

Adds native ROM identity extraction and optional Switch filename embedding.

• Stores per-ROM and per-file title IDs, save IDs, save usage, and Switch title versions.
• Uses extracted Switch content metadata to categorize base games, updates, and DLC.
• Adds schema migrations, guarded sigil integration, Docker packaging, generated API types, UI chips, localization, and tests.

Confidence Score: 3/5

The PR should not merge until existing libraries receive a reliable identity-field backfill; the non-atomic filename rename should also be hardened.

Default, scheduled, and watcher-driven scans can leave all newly added identity columns null for pre-existing ROMs, while embedding changes filenames before the database update is secured.

backend/endpoints/sockets/scan.py, backend/handler/filesystem/roms_handler.py, backend/alembic/versions/0104_sigil_title_ids.py, backend/alembic/versions/0105_sigil_title_version.py

Important Files Changed

Filename	Overview
backend/endpoints/sockets/scan.py	Integrates extraction into the scan pipeline, but ordinary scans do not populate the new fields for existing libraries.
backend/handler/filesystem/roms_handler.py	Implements extraction, Switch categorization, base-ID derivation, and embedding; embedding mutates files before database reconciliation.
backend/adapters/services/sigil.py	Provides a guarded asynchronous adapter that degrades cleanly when the optional native binding or identifiers are unavailable.
backend/models/rom.py	Adds nullable persisted identity fields and the typed save-usage enum.
backend/alembic/versions/0104_sigil_title_ids.py	Adds identity columns and indexes but intentionally or inadvertently leaves existing rows without extracted values.
backend/alembic/versions/0105_sigil_title_version.py	Adds the nullable per-file title-version column without arranging an existing-library backfill.
docker/Dockerfile	Builds the pinned sigil native dependency in a dedicated stage and copies its Python binding into the runtime environment.
frontend/src/v2/components/GameDetails/FilesTab/FileRow.vue	Displays the newly exposed per-file title ID and version metadata.

Comments Outside Diff (1)

1. backend/endpoints/sockets/scan.py, line 231-236 (link)

   Existing ROM identities stay empty

   When an existing library is upgraded and receives a default quick, scheduled, unmatched, or untargeted update scan, this predicate prevents get_rom_files from running. Because the migrations do not backfill the new columns, those ROMs continue returning null title IDs, save IDs, save usage, and file versions until a complete, hash, or explicitly targeted scan is run.

   Knowledge Base Used: ROM Scanning Flow

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: backend/endpoints/sockets/scan.py
   Line: 231-236
   
   Comment:
   **Existing ROM identities stay empty**
   
   When an existing library is upgraded and receives a default quick, scheduled, unmatched, or untargeted update scan, this predicate prevents `get_rom_files` from running. Because the migrations do not backfill the new columns, those ROMs continue returning null title IDs, save IDs, save usage, and file versions until a complete, hash, or explicitly targeted scan is run.
   
   **Knowledge Base Used:** [ROM Scanning Flow](https://app.greptile.com/romm/-/custom-context/knowledge-base/rommapp/romm/-/docs/rom-scanning-flow.md)
   
   How can I resolve this? If you propose a fix, please make it concise.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

   

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
backend/endpoints/sockets/scan.py:231-236
**Existing ROM identities stay empty**

When an existing library is upgraded and receives a default quick, scheduled, unmatched, or untargeted update scan, this predicate prevents `get_rom_files` from running. Because the migrations do not backfill the new columns, those ROMs continue returning null title IDs, save IDs, save usage, and file versions until a complete, hash, or explicitly targeted scan is run.

### Issue 2 of 2
backend/handler/filesystem/roms_handler.py:235
**Rename precedes database reconciliation**

Filename embedding commits the filesystem rename before the updated ROM and file records are persisted, with no rollback when a later scan step fails. This leaves the database pointing at the old path until recovery and can cause a watcher-triggered scan to register the renamed file separately.

_{Reviews (1): Last reviewed commit: "Merge upstream master into feature/sigil..." | Re-trigger Greptile}

[greptile-apps]

Rename precedes database reconciliation

Filename embedding commits the filesystem rename before the updated ROM and file records are persisted, with no rollback when a later scan step fails. This leaves the database pointing at the old path until recovery and can cause a watcher-triggered scan to register the renamed file separately.

Knowledge Base Used:

• Filesystem Handler
• ROM Scanning Flow

Prompt To Fix With AI

This is a comment left during a code review.
Path: backend/handler/filesystem/roms_handler.py
Line: 235

Comment:
**Rename precedes database reconciliation**

Filename embedding commits the filesystem rename before the updated ROM and file records are persisted, with no rollback when a later scan step fails. This leaves the database pointing at the old path until recovery and can cause a watcher-triggered scan to register the renamed file separately.

**Knowledge Base Used:**
- [Filesystem Handler](https://app.greptile.com/romm/-/custom-context/knowledge-base/rommapp/romm/-/docs/filesystem-handler.md)
- [ROM Scanning Flow](https://app.greptile.com/romm/-/custom-context/knowledge-base/rommapp/romm/-/docs/rom-scanning-flow.md)

How can I resolve this? If you propose a fix, please make it concise.

[Copilot]

Pull request overview

This PR improves scan-time identification by extracting platform-native title IDs and save IDs directly from ROM binaries (via the optional sigil binding), persisting them in new DB columns, and surfacing them in the v2 UI. It also adds an opt-in Switch-only filename renaming mode to embed extracted title IDs and versions into filenames during scans, with DB reconciliation.

Changes:

• Backend: Add title_id, save_id, save_usage, and per-file title_version fields, extract them during filesystem scanning, and persist them through scan reconciliation and file syncing.
• Packaging: Build and install the optional sigil native bindings in Docker builds, and document optional local developer setup steps.
• Frontend v2: Regenerate OpenAPI types and display per-file Title ID and Version on the Files tab, plus title/save IDs on the metadata tab.

Reviewed changes

Copilot reviewed 39 out of 44 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
frontend/src/v2/utils/romArtwork.test.ts	Update test fixtures for new RomFileSchema fields.
frontend/src/v2/components/GameDetails/RelatedGameCard.vue	Extend synthetic rom shape with new title/save fields.
frontend/src/v2/components/GameDetails/MetadataTab.vue	Display ROM-level Title ID and Save ID in metadata rows.
frontend/src/v2/components/GameDetails/FilesTab/FileRow.vue	Display per-file Title ID and Version chips in file rows.
frontend/src/locales/en_US/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/en_GB/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/de_DE/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/es_ES/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/fr_FR/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/it_IT/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/ja_JP/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/ko_KR/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/pt_BR/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/pl_PL/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/ru_RU/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/ro_RO/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/tr_TR/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/zh_CN/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/zh_TW/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/cs_CZ/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/bg_BG/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/hu_HU/rom.json	Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/generated/models/SimpleRomSchema.ts	Regenerate API model to include title/save fields and SaveUsage.
frontend/src/generated/models/DetailedRomSchema.ts	Regenerate API model to include title/save fields and SaveUsage.
frontend/src/generated/models/RomFileSchema.ts	Regenerate API model to include per-file title/save/version fields.
frontend/src/generated/models/SaveUsage.ts	Add generated SaveUsage union type.
frontend/src/generated/index.ts	Export generated SaveUsage type.
Dockerfile	Build and install sigil bindings into the runtime venv during image build.
docker/Dockerfile	Add sigil-build stage and copy bindings into the slim image venv.
DEVELOPER_SETUP.md	Document optional local build/install steps for sigil.
backend/tests/handler/test_fastapi.py	Add tests ensuring extracted rom-level fields fold into Rom and are preserved on rescans.
backend/tests/handler/filesystem/test_roms_handler.py	Add filesystem handler tests for extraction, categorization, base-ID derivation, and embedding renames.
backend/tests/endpoints/sockets/test_scan.py	Add tests for rename reconciliation and for persisting scanned per-file sigil columns.
backend/tests/adapters/services/test_sigil.py	Add unit tests for guarded import behavior and extraction field mapping.
backend/models/rom.py	Add SaveUsage enum, new columns, and title-id indexes to Rom and RomFile.
backend/handler/scan_handler.py	Persist extracted rom-level title/save fields, without wiping on extraction-disabled rescans.
backend/handler/filesystem/roms_handler.py	Extract title/save IDs and per-file versions via SigilService, map Switch CNMT content types, and optionally rename Switch files.
backend/handler/database/roms_handler.py	Persist new per-file scanned columns (title_id, save_id, title_version) during sync_rom_files.
backend/endpoints/sockets/scan.py	Wire config-gated extraction and embedding into _identify_rom, including optional Switch prod.keys resolution.
backend/endpoints/responses/rom.py	Expose new title/save fields (rom and file) plus SaveUsage in response schemas.
backend/config/config_manager.py	Add config keys for skipping extraction and enabling Switch filename embedding.
backend/alembic/versions/0104_sigil_title_ids.py	Add DB columns and indexes for rom/file title and save IDs, plus save_usage enum.
backend/alembic/versions/0105_sigil_title_version.py	Add rom_files.title_version BigInteger column.
backend/adapters/services/sigil.py	Add guarded sigil adapter service with platform gating, error handling, and field mapping.

Files not reviewed (5)

• frontend/src/generated/index.ts: Generated file
• frontend/src/generated/models/DetailedRomSchema.ts: Generated file
• frontend/src/generated/models/RomFileSchema.ts: Generated file
• frontend/src/generated/models/SaveUsage.ts: Generated file
• frontend/src/generated/models/SimpleRomSchema.ts: Generated file

Comments suppressed due to low confidence (1)

frontend/src/v2/components/GameDetails/FilesTab/FileRow.vue:181

• v-if="file.title_version" will not render the version chip when the version is 0, even though 0 is a valid extracted title version. Use a null/undefined check instead of truthiness.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.