Add experimental PostgreSQL data store adapter (#702) - #708
Conversation
429c028 to
35d5ec2
Compare
87adea2 to
fc2d8da
Compare
Add a first-class PostgreSQL-backed DataStore adapter (issue bolshakov#702), so apps already running Postgres can use Stoplight without Redis. Architecture mirrors the Redis adapter, including its Lua-script approach: the atomic operations live in installed pgSQL FUNCTIONS (record_failure/success, recovery-probe record, get_metrics, transition_to_{red,green,yellow}, release_lock) under infrastructure/postgres/data_store/functions/*.sql — the direct analogue of redis/data_store/lua_scripts/*.lua. The Ruby adapter calls them via SELECT stoplight_*(...); it issues no mutation SQL inline. Functions are installed by the generator's migration (logidze-style), with a `--update` mode to refresh them. The generator embeds the gem's canonical Schema SQL (text / timestamptz columns) so generated tables always match the function signatures. Backed by raw `pg` (no new core runtime deps; the user supplies a PG::Connection or a ConnectionPool). Plugs into LightFactory#create_data_store and is wrapped by FailSafe::DataStore for automatic failover to the in-memory store. First-writer-wins transitions use a guarded UPDATE inside the function (row lock serializes 50-thread races to exactly one winner). Five tables map 1:1 to the memory adapter's five stores. Retention is a per-write prune folded into the record functions. All time-sensitive operations use the application Ruby clock (Timecop-compatible, matching memory/Redis); the recovery lock uses the database clock for TTL expiry. Includes: - Stoplight::DataStore::Postgres public config object - full data-store contract (16 methods) + recovery lock store/token - 9 pgSQL functions + Functions installer; Schema.create! installs tables+functions - wiring into LightFactory + FailSafe recovery-lock token routing - Rails generator `stoplight:postgres:install` (+ `--update`) embedding schema+functions - RBS signatures (wiring + public API type-checked; impl steep:ignored, as Redis) - shared contract examples incl. windowed AND non-windowed paths, 50-thread transition/lock race specs, schema/function type-contract guard, window-counting property test, generator specs - README (experimental section, incl. schema_format = :sql note) + UPGRADING note Refs bolshakov#702
fc2d8da to
f0697bf
Compare
|
Hey @Halvanhelv, thanks a lot for this - it's a serious chunk of work and the effort is genuinely appreciated. I want to be upfront about where things are, because a couple of issues mean we can't take it in as-is. First, we'd actually started discussing a SQL backend internally around the same time your feature request landed, and the direction we're leaning is fairly different from this implementation. Second, the PR builds on interfaces that we're in the middle of removing, so a lot of it would need reworking against the new storage layer before it could merge. There's also a practical problem: at 2k+ lines, it's very hard to give the review the quality it deserves. Splitting it up would help both of us. So I'd like to take a step back and align on a plan first. Let's use #702 to coordinate with @Lokideos and agree upfront:
Every PR should keep unit tests green on CI. Admin-panel support and feature tests don't need to land in every PR, but they do need to land eventually. Let's continue in #702 and figure out a plan together |
Summary
Adds a first-class PostgreSQL
DataStoreadapter, so apps already running Postgres can use Stoplight without operating Redis. Refs #702.Configuration mirrors the Redis adapter:
Approach
The adapter mirrors the existing Redis adapter, including its Lua-script model. The atomic operations live as installed pgSQL functions (
infrastructure/postgres/data_store/functions/*.sql) — the direct analogue ofredis/data_store/lua_scripts/*.lua. The Ruby adapter calls them viaSELECT stoplight_*(...)and issues no mutation SQL inline.Atomicity is achieved with native SQL rather than Lua:
HSETNX(first-writer-wins)UPDATE ... WHERE col IS NULL RETURNING(row lock + re-check under READ COMMITTED)COUNT(*) WHERE occurred_at >= window_startSET NX PX(recovery lock)INSERT ... ON CONFLICT DO UPDATE ... WHERE expires_at < now() RETURNINGDELETE ... occurred_at < cutoff)Other decisions:
pg; the user supplies aPG::Connectionor aConnectionPool(exactly like the Redis client).LightFactory#create_data_store; wrapped by the existingFailSafe::DataStorefor automatic in-memory failover.What's included
Stoplight::DataStore::Postgresconfig object + RBS signatures.Infrastructure::Postgres::DataStore(full data-store contract) + recovery lock store/token.Functionsinstaller;Schema.create!installs tables + functions.LightFactory+ Postgres token routing inFailSafe::DataStore.stoplight:postgres:install(+--updateto refresh functions), embedding the canonical schema/function SQL.Schema / installation
bin/rails g stoplight:postgres:install && bin/rails db:migratecreates 5 tables + 9 functions. Since pgSQL functions are not representable in the Ruby schema dumper, apps must either useschema_format = :sqlor add thefxgem (which dumps the functions intoschema.rb) — same options matrix aslogidze. Documented in the README.Testing
DataStorecontract examples (the same ones the Redis/Memory adapters use):#names,#get_metrics,#get_recovery_metrics,#set_state,#transition_to_color(incl. the 50-thread "thread safe" example).window_size: nil) coverage.steepclean,standardrbclean.Known limitations (experimental)
schema_format: see above (:sqlorfx).eventsare pruned;metadata/statesrows for unused lights persist untildelete_light.metadatarow (per-light row lock) — fine across different lights.set_stateoperation may silently fail or create inconsistent state during failover #543).Open questions / coordination
@bolshakov @Lokideos — #702 mentions plan to build this in the next major, so I'd like to align before going further:
PG::Connectiononly, or alsoConnectionPool(current impl supports both viarespond_to?(:with))?fxsupport be first-class in the generator, or docs-only (current)?Happy to adjust naming/structure to match your intended design.