feat(ingest): data ingestion layer + SOSA/SSN sensor layer, from workshop Appendix A

[Sulstice]

What

Unstructured material (workshop notes, slide decks, transcripts) now enters Supabase through a data ingestion layer, and the KG is built from the resulting rows rather than from a checked-in file. The first source is the workshop's Appendix A sensor and device registry. The exporter turns it into W3C SOSA/SSN: which project deploys which sensor, what that sensor observes, and on whose subjects (Janowicz et al. 2018).

raw document → ingestion_sources → ingestion_records → promote_ingestion_record() → domain tables → export()

ingestion_records holds one payload per record, validated against ingest/kinds/<kind>.schema.json, along with its review flags and status.

Docs:

• ingest/README.md: the ingestion architecture
• kg/sosa/README.md: the SOSA mapping and the remaining phases

Changes

Migration 20261006120000_data_ingestion.sql

• Generic intake tables: ingestion_sources and ingestion_records.
• promote_ingestion_record() and promote_ingestion_source(). Both are idempotent, and anything the promoter can't resolve becomes a review flag instead of a guess.
• Domain tables for the first record kind, sensor_deployment:
  • observable_properties
  • sensor_deployments
  • sensor_deployment_devices
  • sensor_deployment_properties
  • sensor_deployment_presenters (curator-only)
• Two SOSA invariants are enforced as CHECKs:
  • grant is unresolved if and only if it has no grant_id
  • a row with no presenter must carry a verify_note
• A trigger checks that each property's kind matches its role (an actuatable property can't be "observed").
• Every new table gets the audit trigger. ingestion_records is excluded from cell provenance because it is intake state.
• Access: anon can read the domain tables, but not payloads or names.

Migration …120100_ingest_workshop_appendix_a.sql

• Generated by ingest/to_sql.py.
• Seeds the 53-key vocabulary, the source, and 60 records with review flags, then promotes them.

Exporter

• export() section 4 reads the new tables through kg/sosa_layer.emit_sosa().
• Until the migration is applied it does nothing.

kg/sosa/build_sosa.py

• Runs the real exporter over simulated tables: the staged records, promoted the same way the SQL function promotes them.
• That leaves one implementation. Its output matches the previous build exactly, apart from one predicate renamed from workshop_row to source_locator.

CI

• New steps: ingest/to_sql.py --check (records are valid and the seed is current) and the SOSA shapes check.
• The kg job now also runs for changes under ingest/ and to the ingestion migrations.

Verified

• Both migrations were applied to a scratch Postgres 16 with stubs for the Supabase objects they depend on.
  • All 60 records promote. A prefixed grant number (1R34…) resolves to its grant. Re-running the schema and seed files is idempotent.
  • Audit rows are attributed to the migration's actor.
  • Each negative test is refused as intended: both CHECKs, the property-kind trigger, promoting a rejected record, promoting a record with an unknown grant, and promoting as a non-curator.
  • Anon is denied ingestion_records and sensor_deployment_presenters.
• ingest/to_sql.py --check goes RED on a stale seed and on a no-presenter record that has no verify_note.
• kg/ci_check.py passes, with the baseline unchanged.
• npm run test:guards: 44/44 pass.
• tsc was not run: there are no TypeScript changes and node_modules is not installed in this environment.

Not done here

• Migrations not applied. Run them by hand in the SQL editor: schema first, then seed. After that, regenerate kg/export/bbqs.ttl and move the SOSA shapes into kg/shapes/.
• Presenter names are deliberately not in the repo, because it is public. A separate SQL file loads them into the curator-only table.
• Award attribution for 9 rows needs a human. They are listed in kg/sosa/README.md.
• LLM extraction (ingest-document) and a review screen are the next steps; see ingest/README.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_014VR8fVc4t6stw4vYKkhSDp

[github-actions]

🤖 Triage: Class C → spec:C

Touches a danger layer. Requires Spec Kit artifacts (spec/plan/tasks/qa-itinerary), a Constitution Check, and human sign-off. Never auto-merges.

Blast radius: HIGH

• ci-workflow: CI/CD workflow — this is the pipeline that ships everything else, including this gate.
• migration: Schema/RLS/view migration — applied MANUALLY to production and hard to reverse. RLS policy and CREATE OR REPLACE VIEW changes live here too.

Complexity: HIGH

• 18 files changed (> 4).
• 3835 lines changed (> 80).
• 16 new files (> 2).

Files that set the class:

• .github/workflows/kg.yml → ci-workflow (hard)
• supabase/migrations/20261006120000_data_ingestion.sql → migration (hard)
• supabase/migrations/20261006120100_ingest_workshop_appendix_a.sql → migration (hard)

_{Deterministic classifier — computed from the diff, not from the issue text. Change the rules in scripts/triage/classify.mjs (itself a Class-C path).}

[github-actions]

🛑 Class C — this touches a danger layer

Per the constitution (Development Workflow) danger-layer changes are spec-driven. Before this can merge, a maintainer must:

1. Create the Spec Kit artifacts in bbqs-agent/specs/<NNN-feature>/ — spec.md, plan.md (with a Constitution Check), tasks.md, qa-itinerary.md.
2. Point bbqs-agent/.specify/feature.json at that directory.
3. Run the Find → Fix → Test → Verify-Live loop, incl. the sandbox QA workflow.
4. Add the spec-signoff label to this PR to release the triage/spec-gate check.

This gate is here because a small-looking diff to .github/workflows/kg.yml reaches far beyond the file it edits.