Refactor: define TracePcap's layers, enforce the seams, and group functions by deterministic / LLM / human-annotated

[NotYuSheng]

Description

TracePcap is a modular wrapper around a growing suite of functions — the architecture exists so new capability is added by dropping a module into a stage, not by editing a core. The seams for this exist (analysis/spi/, DeviceClassificationSignal, story detectors, HostServiceLogExtractor); the code has drifted from them. This epic defines the target model, enforces its rules, and refactors toward it slice by slice.

Design — SETTLED (2026-07-15)

The target model is documented in docs/architecture/layers.rst and enforced by backend/src/test/java/com/tracepcap/architecture/LayerDependencyTest.java. Summary:

Five stages (data changes shape: packets → facts → insights → conclusions → narrative):

Ingest → Extract → Scan → Adjudicate → Narrate     Present = read surface over all

• Extract — observations with witnesses: append-only, conflict-preserving (no winner-picking at write time), graded MEASURED / REPORTED / INFERRED (schema-level, per fact type), plus a run manifest (extractor status + version → enables backfill). Two sub-kinds: wire extractors (read pcap) and reference extractors (GeoIP/OUI — backfillable without the capture). Extraction may be lazy (session reconstruction, carving).
• Scan — additive scanners; read facts, never the pcap; judgment is the boundary (no judgment = a view, lives with Present). Contradiction scanners are first-class (fact conflicts are findings: spoofing, impersonation). Scope (per-capture vs cross-snapshot) is an attribute. Loop rule: scanners consuming adjudications treat them as INFERRED unless human-confirmed (Research: learn and document organisational addressing conventions across and within networks #507 circularity guard, structural).
• Adjudicate — exclusive: one adjudicator per question; consumes insights + human annotations ranked first; weighs by grade (measurement beats testimony); winner-with-confidence or contested (legal, displayable). Versioned; staleness = re-adjudication. ScoreBoard is the embedded prototype; [Enhancement] Conflicting device information: nodeType, deviceType and role can contradict each other on one host #499 = missing host-identity adjudicator; [Enhancement] Surface classification uncertainty on the graph instead of only in the click-through popup #498 = exposing the margin.
• Narrate — evidence citations mandatory; deterministic citation checker; offline LLM only.
• Annotation is the envelope, not a stage — origin/llmSuggested/confirmedByHuman/staleSince on every fact and conclusion (NodeRoleEntity is the reference implementation); human authority enters at Adjudicate.

Dependency rules (all enforced, frozen):

1. Feature → analysis, never the reverse ([Epic] Decompose the analysis package into feature modules via dependency inversion #416) — holds, 0 violations
2. Cross the analysis boundary via analysis.spi/dto, never repositories/entities — 755 frozen violations
3. No module cycles — 71 frozen (monitor ↔ insights, analysis ↔ file)
4. Raw-capture access confined to Ingest plumbing / Extract (eager or lazy) / evidence export — 6 frozen (all FilterService, the known grey case)

The frozen store (backend/src/test/resources/archunit_store) may only shrink; a new violation fails the build naming the exact class (verified with a planted probe).

Evidence of drift (measured)

• SPI bypassed 755:1 — modules import analysis repositories/entities directly.
• Two cycles: monitor ↔ insights (decoupling event NodeRoleChangedEvent already exists, unevenly applied), analysis ↔ file (single edge: AsyncAnalysisService).
• HostnameResolverService adjudicates at write time and destroys competing claims (the [Enhancement] Expand hostname resolution — wire in nDPI SNI/Host, TLS cert subject, LLMNR #511 spoofing case is undetectable today); IpMacObservationEntity does it right.
• WebServerLogExtractor mixes Extract + Scan (the api/web role decision [Enhancement] Server/client classification: TLS participation and low ports wrongly force a Web Server label #496 flags).
• No run manifest: UnknownAppDetector can't distinguish "nDPI didn't run" from "unknown" (Research: how unknown/unidentified protocols are flagged and handled (incl. proprietary payloads) #501).
• The frontend scans and adjudicates (networkService.ts nodeType; getNodeColor precedence) — client-side, on a truncated node set; [Enhancement] Server/client classification: TLS participation and low ports wrongly force a Web Server label #496/[Enhancement] Conflicting device information: nodeType, deviceType and role can contradict each other on one host #499 are the symptoms.
• No adjudicators outside ScoreBoard's prototype; conclusions ship un-reconciled ([Enhancement] Conflicting device information: nodeType, deviceType and role can contradict each other on one host #499).
• Mass concentration: analysis 6,774 LOC (2× next); god classes SessionReconstructionService (1,416), ReportService (1,407).

Acceptance Criteria

1. Layer model defined and documented ✅ docs/architecture/layers.rst (five stages, three grades, contracts, worked example, divergence list) + C4 diagram.
2. Boundaries enforced, not just documented ✅ Four frozen ArchUnit rules; store committed; only-shrink policy; verified to catch new violations.
3. monitor ↔ insights and analysis ↔ file cycles broken (events or shared contracts; NodeRoleChangedEvent shows the pattern). Cycle store → 0. (Note from slice 1: fixing a cycle can surface previously-unreported ones — monitor ↔ subnets appeared only after analysis ↔ file was broken.)
4. Cross-module access to analysis migrates onto analysis.spi/DTOs; the 755-entry store trends to 0, slice by slice.
5. Run manifest: extractors report status/version per file; UnknownAppDetector (and peers) consult it — fixes the Research: how unknown/unidentified protocols are flagged and handled (incl. proprietary payloads) #501 conflation.
6. Conflict-preserving extraction: HostnameResolverService stores all claims (à la IpMacObservationEntity); winner-picking moves to a hostname adjudicator.
7. First real adjudicator: host identity ([Enhancement] Conflicting device information: nodeType, deviceType and role can contradict each other on one host #499) — consumes nodeType/deviceType/role insights + human overrides, outputs winner-or-contested; frontend consumes it instead of computing its own (networkService.ts scanning/getNodeColor precedence retired).
8. WebServerLogExtractor split: extraction stays, the api/web role decision becomes a scanner ([Enhancement] Server/client classification: TLS participation and low ports wrongly force a Web Server label #496 fix lands there).
9. The 7-stage AnalysisService pipeline expressed as composable units, not comments in one method.
10. Annotation envelope generalised beyond node roles (device type next — [Enhancement] Let users override device classification and contribute weighted evidence #506).
11. No behaviour change per slice; existing tests stay green.

Slices (one PR each, roughly ordered)

• 0. Map + enforcement — layers.rst, C4, four frozen ArchUnit rules, baseline committed.
• 1. Break analysis ↔ file — single edge (AsyncAnalysisService); event or port. Smallest structural win; proves the store shrinks.
• 2. Run manifest — extractor status/version table + UnknownAppDetector consults it (Research: how unknown/unidentified protocols are flagged and handled (incl. proprietary payloads) #501's immediate bug fixed here).
• 3. Break monitor ↔ insights and monitor ↔ subnets (PR Break monitor ↔ insights and monitor ↔ subnets — module cycles to zero (#512 slice 3) #514, cycles → 0) — extend the NodeRoleChangedEvent pattern; the subnets edge is ChangeDetectionService:64.
• 4. Conflict-preserving hostnames (claims table V33 + HostnameAdjudicator) — claims table + hostname adjudicator ([Enhancement] Expand hostname resolution — wire in nDPI SNI/Host, TLS cert subject, LLMNR #511 prerequisite).
• 5a. Host-identity adjudicator — backend (PR Host-identity adjudicator — one voice per host, contested is legal (#512 slice 5a) #516: HostIdentityService, contested outcomes, AnalysisCompletedEvent, HostClassificationLookup port)
• 5b. Frontend consumes adjudicated identity (PR Frontend consumes adjudicated host identities — contested renders as contested (#512 slice 5b) #517: contested amber rings, human labels win, NodeDetails identity row — Playwright-verified) — [Enhancement] Conflicting device information: nodeType, deviceType and role can contradict each other on one host #499/[Enhancement] Surface classification uncertainty on the graph instead of only in the click-through popup #498; frontend consumes adjudications.
• 6. SPI migration — the 755, module by module (biggest consumers first: measure per-module counts at slice start).
• 7. Pipeline decomposition — stages as units.
• Follow-ups filed as the map exposes them (god-class splits, FilterService grey case, frontend repatriation beyond identity).

Notes

Related: #416 (predecessor epic — its invariant holds at 0 violations and is now frozen), #496/#498/#499/#501/#506/#507/#511 (each maps to a slice above — the model carves them at their joints), #492 (frontend), #502/#503/#504/#505/#510 (new modules that should land on the target seams, not the old ones).

Key files: docs/architecture/layers.rst, docs/architecture/enforcement.rst, backend/src/test/java/com/tracepcap/architecture/LayerDependencyTest.java, backend/src/test/resources/archunit_store/ (the shrink-only baseline).