Skip to content

Add experimental PostgreSQL data store adapter (#702) - #708

Draft
Halvanhelv wants to merge 1 commit into
bolshakov:developfrom
Halvanhelv:feat/postgres-data-store
Draft

Add experimental PostgreSQL data store adapter (#702)#708
Halvanhelv wants to merge 1 commit into
bolshakov:developfrom
Halvanhelv:feat/postgres-data-store

Conversation

@Halvanhelv

@Halvanhelv Halvanhelv commented Jun 28, 2026

Copy link
Copy Markdown

Summary

Adds a first-class PostgreSQL DataStore adapter, so apps already running Postgres can use Stoplight without operating Redis. Refs #702.

Configuration mirrors the Redis adapter:

Stoplight.configure do |config|
  config.data_store = Stoplight::DataStore::Postgres.new(PG.connect(ENV["DATABASE_URL"]))
  # or a ConnectionPool<PG::Connection> for multi-threaded apps
end

Marked experimental, per the direction in #702.

Approach

The adapter mirrors the existing Redis adapter, including its Lua-script model. The atomic operations live as installed pgSQL functions (infrastructure/postgres/data_store/functions/*.sql) — the direct analogue of redis/data_store/lua_scripts/*.lua. The Ruby adapter calls them via SELECT stoplight_*(...) and issues no mutation SQL inline.

Atomicity is achieved with native SQL rather than Lua:

Redis Postgres
Lua single-threaded atomicity single-statement atomicity + function body in one txn
HSETNX (first-writer-wins) guarded UPDATE ... WHERE col IS NULL RETURNING (row lock + re-check under READ COMMITTED)
ZSET + ZCOUNT (sliding window) timestamped rows + COUNT(*) WHERE occurred_at >= window_start
SET NX PX (recovery lock) INSERT ... ON CONFLICT DO UPDATE ... WHERE expires_at < now() RETURNING
key TTL lazy per-write prune (DELETE ... occurred_at < cutoff)

Other decisions:

  • No new runtime deps — raw pg; the user supplies a PG::Connection or a ConnectionPool (exactly like the Redis client).
  • Plugs into LightFactory#create_data_store; wrapped by the existing FailSafe::DataStore for automatic in-memory failover.
  • Time comes from the application clock (Timecop-compatible, matching the memory/Redis adapters), except the recovery-lock TTL, which uses the DB clock.

What's included

  • Stoplight::DataStore::Postgres config object + RBS signatures.
  • Infrastructure::Postgres::DataStore (full data-store contract) + recovery lock store/token.
  • 9 pgSQL functions + a Functions installer; Schema.create! installs tables + functions.
  • Wiring into LightFactory + Postgres token routing in FailSafe::DataStore.
  • Rails generator stoplight:postgres:install (+ --update to refresh functions), embedding the canonical schema/function SQL.
  • README + UPGRADING docs.

Schema / installation

bin/rails g stoplight:postgres:install && bin/rails db:migrate creates 5 tables + 9 functions. Since pgSQL functions are not representable in the Ruby schema dumper, apps must either use schema_format = :sql or add the fx gem (which dumps the functions into schema.rb) — same options matrix as logidze. Documented in the README.

Testing

  • All shared DataStore contract examples (the same ones the Redis/Memory adapters use): #names, #get_metrics, #get_recovery_metrics, #set_state, #transition_to_color (incl. the 50-thread "thread safe" example).
  • 50-thread transition + recovery-lock races (exactly one winner).
  • Window-counting property test; schema/function type-contract guard; non-windowed (window_size: nil) coverage.
  • Generator specs.
  • Full suite green (910 examples, 0 failures), steep clean, standardrb clean.

Known limitations (experimental)

  • schema_format: see above (:sql or fx).
  • Stale lights: only events are pruned; metadata/states rows for unused lights persist until delete_light.
  • Very hot single lights serialize on their one metadata row (per-light row lock) — fine across different lights.
  • During a Postgres outage, FailSafe falls back to per-process memory (same semantics as the Redis adapter; see set_state operation may silently fail or create inconsistent state during failover #543).

Open questions / coordination

@bolshakov @Lokideos#702 mentions plan to build this in the next major, so I'd like to align before going further:

  • Is a contributed adapter welcome here, or are you already mid-flight?
  • Connection contract: bare PG::Connection only, or also ConnectionPool (current impl supports both via respond_to?(:with))?
  • Should fx support be first-class in the generator, or docs-only (current)?
  • Retention: lazy per-write prune (current) vs a dedicated cleanup task vs a trigger?

Happy to adjust naming/structure to match your intended design.

@Halvanhelv
Halvanhelv changed the base branch from main to develop June 28, 2026 13:29
@Halvanhelv
Halvanhelv force-pushed the feat/postgres-data-store branch from 429c028 to 35d5ec2 Compare June 28, 2026 13:43
@Halvanhelv
Halvanhelv marked this pull request as draft June 28, 2026 13:48
@Halvanhelv
Halvanhelv force-pushed the feat/postgres-data-store branch 5 times, most recently from 87adea2 to fc2d8da Compare June 28, 2026 14:00
Add a first-class PostgreSQL-backed DataStore adapter (issue bolshakov#702), so apps
already running Postgres can use Stoplight without Redis.

Architecture mirrors the Redis adapter, including its Lua-script approach: the
atomic operations live in installed pgSQL FUNCTIONS (record_failure/success,
recovery-probe record, get_metrics, transition_to_{red,green,yellow},
release_lock) under infrastructure/postgres/data_store/functions/*.sql — the
direct analogue of redis/data_store/lua_scripts/*.lua. The Ruby adapter calls
them via SELECT stoplight_*(...); it issues no mutation SQL inline. Functions
are installed by the generator's migration (logidze-style), with a `--update`
mode to refresh them. The generator embeds the gem's canonical Schema SQL
(text / timestamptz columns) so generated tables always match the function
signatures.

Backed by raw `pg` (no new core runtime deps; the user supplies a PG::Connection
or a ConnectionPool). Plugs into LightFactory#create_data_store and is wrapped by
FailSafe::DataStore for automatic failover to the in-memory store.

First-writer-wins transitions use a guarded UPDATE inside the function (row lock
serializes 50-thread races to exactly one winner). Five tables map 1:1 to the
memory adapter's five stores. Retention is a per-write prune folded into the
record functions. All time-sensitive operations use the application Ruby clock
(Timecop-compatible, matching memory/Redis); the recovery lock uses the database
clock for TTL expiry.

Includes:
- Stoplight::DataStore::Postgres public config object
- full data-store contract (16 methods) + recovery lock store/token
- 9 pgSQL functions + Functions installer; Schema.create! installs tables+functions
- wiring into LightFactory + FailSafe recovery-lock token routing
- Rails generator `stoplight:postgres:install` (+ `--update`) embedding schema+functions
- RBS signatures (wiring + public API type-checked; impl steep:ignored, as Redis)
- shared contract examples incl. windowed AND non-windowed paths, 50-thread
  transition/lock race specs, schema/function type-contract guard, window-counting
  property test, generator specs
- README (experimental section, incl. schema_format = :sql note) + UPGRADING note

Refs bolshakov#702
@Halvanhelv
Halvanhelv force-pushed the feat/postgres-data-store branch from fc2d8da to f0697bf Compare June 28, 2026 14:09
@bolshakov

Copy link
Copy Markdown
Owner

Hey @Halvanhelv, thanks a lot for this - it's a serious chunk of work and the effort is genuinely appreciated.

I want to be upfront about where things are, because a couple of issues mean we can't take it in as-is.

First, we'd actually started discussing a SQL backend internally around the same time your feature request landed, and the direction we're leaning is fairly different from this implementation. Second, the PR builds on interfaces that we're in the middle of removing, so a lot of it would need reworking against the new storage layer before it could merge.

There's also a practical problem: at 2k+ lines, it's very hard to give the review the quality it deserves. Splitting it up would help both of us.

So I'd like to take a step back and align on a plan first. Let's use #702 to coordinate with @Lokideos and agree upfront:

  • which technologies we're committing to
  • how schema migrations get delivered to users
  • how to break the work into small, independent, reviewable PRs

Every PR should keep unit tests green on CI. Admin-panel support and feature tests don't need to land in every PR, but they do need to land eventually.

Let's continue in #702 and figure out a plan together

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants