Skip to content

Add extension update test via pgxntool test-build - #11

Merged
jnasbyupgrade merged 17 commits into
pgxntool-updatefrom
extension-update-test
Jul 24, 2026
Merged

Add extension update test via pgxntool test-build#11
jnasbyupgrade merged 17 commits into
pgxntool-updatefrom
extension-update-test

Conversation

@jnasbyupgrade

@jnasbyupgrade jnasbyupgrade commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Add full update+upgrade testing to the suite: test/deps.sql now selects between three install modes via a TEST_LOAD_SOURCE GUC - fresh (CREATE EXTENSION at current), update (installs 0.9.6, then ALTER EXTENSION UPDATE, exercised by make test-update), and existing (asserts an already-installed extension is present and current, touching nothing else, for use against a real pg_upgrade'd or out-of-band-updated database). A second, independent TEST_SCHEMA switch runs the whole suite both WITHOUT specifying a schema at all (CREATE EXTENSION with no targeting, landing wherever the session's own ambient search_path resolves) and WITH one explicitly targeted, using a name that requires SQL identifier quoting - both legs run in CI, neither redundant with the other. See test/README.md for the full breakdown of both switches and exactly what drives the need for the two remaining expected/extension_tests*.out variants.

A dependency-guard technique (bin/test_existing.sql/*.sql: a view with a hard pg_depend edge to a stable, never-dropped extension member) proves an install/update/pg_upgrade sequence didn't silently drop and reinstall the extension it's meant to be testing, which would otherwise let a regression hide behind a fresh reinstall; bin/test_existing factors install -> plant guard -> update/upgrade -> assert -> drop guard -> run suite into one committed, parameterized script instead of duplicating it as inline YAML. Three new CI jobs exercise all of this on every push: extension-update-test (0.9.6 -> current, across a PostgreSQL x schema matrix), pg-upgrade-test (a real binary pg_upgrade across PostgreSQL majors, old_pg/new_pg x schema legs), and a docs-only gate that also derives the PostgreSQL-major matrix list from a single set of constants (so a new major is a one-line change, not an edit in several jobs) feeding a stable all-checks-passed required check. ci.yml now leads with a top-of-file summary comment covering the whole matrix strategy.

Removed test/sql/simple.sql and test/sql/sanity.sql. simple.sql existed to exercise "the other" schema before TEST_SCHEMA existed, dressed up as an on-search-path test, but the extension's own functions are all schema-qualified via ncs() regardless of search_path, so its search_path manipulation never actually changed anything - the schema coverage it stood in for is now done properly (and more thoroughly) by the TEST_SCHEMA matrix on extension_tests.sql directly. sanity.sql's two direct null_count() calls turned out to be a strict subset of what test__functionality's bag_eq comparisons already cover (the same 8-row test_data set exercises every null-count combination those two calls checked, and more). Confirmed no coverage was lost either time before removing.

Adapted from Postgres-Extensions/cat_tools's own update+upgrade testing where it differs: no bridge-update leg is needed, since count_nulls has no pg_upgrade-unsafe old version to work around (a real, not rare, risk for any extension - just not one count_nulls happens to have); the dependency guard is dropped before the suite runs, rather than left standing through it, since count_nulls's suite has its own legitimate (harmless, rolled-back) DROP EXTENSION test that would otherwise collide with it; and no every-major stepwise pg_upgrade climb, since count_nulls has nothing catalog-touching that would benefit from it. The TEST_SCHEMA/quoting design was arrived at independently here, before noticing cat_tools has its own open PR doing the same thing for test role names - worth noting as a broadly useful pattern rather than a one-off.

Also simplified bin/test_existing.sql/plant_guard.sql (a single quote_ident()-built prefix variable instead of branching the whole view definition) and test/sql/extension_tests.sql's two schema-aware test__* functions (schema passed as a function parameter default instead of a current_setting() workaround inside the plpgsql body, since psql substitution doesn't reach inside dollar-quoted bodies but does reach a parameter default).

Deliberately NOT doing right now, but captured as a comment in test/core/functions.sql for next time that file is substantially touched: migrating the main suite's install to pgxntool's test/install feature (cat_tools's own pattern for centralizing fresh/update/existing switching, an independent argument for it regardless of the schema question), and revisiting the debuggability tradeoff of this file's shared/loop-driven design versus interactively \i-ing a traditional, self-contained test file in psql - the actual reason count_nulls originally had separate smoke-test files like the two just removed, worth an intentional replacement if this ever becomes a real problem rather than just letting it stay absent.

test/expected/extension_tests.out was regenerated via make results for the new default (TEST_SCHEMA empty). Two pg_regress alternate-expected-output files cover the two other cases that produce genuinely different (not wrong) text: _1.out for TEST_SCHEMA=Quoted, and _2.out for the empty-schema case as exercised by a bare standalone psql session (the extension-update-test/pg-upgrade-test jobs) rather than the in-suite pg_regress session, which lands in a different real schema than the in-suite case does.

Verified locally against live PG12/PG17 clusters, including a real binary pg_upgrade PG12->PG17 run (extension carried at 0.9.6, updated to 1.0.0, dependency guard proven through every step) before trusting any of this as CI YAML; make test and make test-update pass cleanly on both PG17 and PG12, with and without an explicit TEST_SCHEMA.

Stacked on #10 (now merged).

@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: b58bf997-1ae5-4cf0-8a2f-9b17332c29f3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch extension-update-test

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

test/build/upgrade.sql is a fast smoke check (via pgxntool 2.1.0's
test-build feature): install 0.9.6 (oldest version we ship a full script
for) and ALTER EXTENSION UPDATE, rolled back.

That alone doesn't prove much — it never runs the actual test suite
against the upgraded state. test/deps.sql now selects between installing
fresh (current version) or via the upgrade path, based on a
count_nulls.test_load_mode GUC the Makefile sets via PGOPTIONS
(TEST_LOAD_SOURCE=fresh|upgrade). Since every test file loads deps.sql,
`make test-update` reruns the whole existing suite (extension_tests,
sanity, simple) against the upgraded install, comparing against the exact
same expected/*.out as `make test` — proving the upgrade behaves
identically to a fresh install rather than just completing without error.

Modeled on cat_tools's test/install/load.sql approach (see their
new_functions-pgxntool-2.1.0 branch), adapted to count_nulls's simpler
single-file test/deps.sql structure instead of introducing a separate
test/install fixture, since count_nulls's suite already installs the
extension per test file (into different schemas) rather than once
globally.

sql/count_nulls--0.9.6.sql isn't picked up by base.mk's DATA wildcard
(which only covers upgrade scripts and the current version file), so it's
added to DATA explicitly. Filed as a pgxntool bug:
Postgres-Extensions/pgxntool#48.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jnasbyupgrade
jnasbyupgrade force-pushed the extension-update-test branch from b11893d to d2ff70e Compare July 16, 2026 21:49
jnasbyupgrade and others added 5 commits July 16, 2026 18:25
Per the terminology convention: 'update' is extension-level (ALTER
EXTENSION UPDATE), 'upgrade' is cluster-level (pg_upgrade) — a separate,
not-yet-covered axis. The wrapper target was already called test-update;
the mode value and GUC name were the odd one out.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test/deps.sql gains a third TEST_LOAD_SOURCE mode ('existing'): asserts the
extension is already installed and current, without dropping/creating/
updating anything, for use against a real pg_upgrade'd or out-of-band
updated database.

Replaces the per-test-file hardcoded schema literals
(schema_to_load_count_nulls / public) - a stand-in for real
schema-qualification coverage that only ever tested two fixed, always-
lowercase names - with a single TEST_SCHEMA make var, propagated the same
way as TEST_LOAD_SOURCE via a GUC, applied uniformly for the whole test run.
This also resolves an architectural conflict between the two: a real
pg_upgrade leaves the extension in exactly one schema, which cannot
simultaneously match two different per-file hardcoded literals, so
'existing' mode needs every file installing into the same run-wide schema
to produce comparable output.

sanity.sql previously bypassed test/deps.sql entirely (a bare, unconditional
CREATE EXTENSION), which broke outright under 'existing' mode
("extension already exists"); it now respects both TEST_LOAD_SOURCE and
TEST_SCHEMA like the rest of the suite.

Locally: `make test TEST_SCHEMA=Quoted` exercises a schema requiring SQL
identifier quoting (mixed case - unquoted would fold to lowercase and
silently test 'public' again).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test/pg_upgrade/{plant,assert,drop}_guard.sql: a view with a hard pg_depend
dependency on a stable, never-dropped/redefined extension member
(null_count(anyarray), unchanged since 0.9.0), so a non-CASCADE
DROP EXTENSION count_nulls is blocked. Proves an install/update/pg_upgrade
sequence didn't silently drop+reinstall the extension it's meant to be
testing - a stray CASCADE, a logic bug, or a bad CI step would otherwise
fall through to a silent fresh reinstall and still report green.

bin/test_existing factors the install -> plant guard -> update/upgrade ->
assert -> drop guard -> run suite sequence (needed by both the
extension-update-test and pg-upgrade-test CI jobs) into one committed,
parameterized script instead of duplicating it as inline YAML. Modeled on
Postgres-Extensions/cat_tools's script of the same name, with two
differences: count_nulls has no pg_upgrade-unsafe old version to bridge past
(it has always been pure SQL functions, no SELECT-*-over-catalog views), and
its suite has its own legitimate (harmless - always rolled back) DROP
EXTENSION test, so here the guard is dropped before run-suite instead of
surviving through it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- test: unchanged fresh-install job, now also matrixed over TEST_SCHEMA
  (public, Quoted) via the environment (Make imports it automatically).
- extension-update-test: installs 0.9.6, plants + proves the dependency
  guard, ALTER EXTENSION UPDATEs to current, runs the suite against the
  result in existing mode - across the PG x schema matrix.
- pg-upgrade-test: installs 0.9.6 on an old cluster, plants the guard,
  binary pg_upgrades to a newer cluster, updates to current, runs the suite
  against the real migrated objects in existing mode. A small old_pg/new_pg
  matrix (10->17, 12->17) x schema, not the full PG matrix, to keep cost
  reasonable given this is the most expensive job (installs two full
  PostgreSQL majors and runs the real pg_upgrade binary per leg).
- changes: cheap docs-only gate so the three heavy jobs above can skip
  themselves on doc-only pushes without leaving the required check stuck
  Pending.
- all-checks-passed: single stable required-status-check name; asserts its
  own needs list matches the actual job set so a new job can't be silently
  omitted from the gate.

Locally validated end to end on this container's PG12/PG17 clusters,
including a real binary pg_upgrade (PG12 -> PG17, extension carried at
0.9.6, updated to 1.0.0, guard proven through every step) before trusting
this as CI YAML.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test/expected/extension_tests.out: regenerated via `make results` against
the new default (TEST_SCHEMA=public) - the only change is the schema name
throughout (schema_to_load_count_nulls -> public), consistent with deps.sql
now installing every test file into the same run-wide schema instead of a
different literal per file. simple.out/sanity.out are byte-identical to
before (they already defaulted to 'public').

test/expected/{extension_tests,simple}_1.out: pg_regress's alternate-
expected-file convention, for the TEST_SCHEMA=Quoted CI legs (a schema name
requiring SQL identifier quoting - mixed case, so unquoted would fold to
lowercase and silently retest 'public'). Captured from a real
`make test TEST_SCHEMA=Quoted` run with zero TAP-level failures (`grep -c
'^not ok' test/results/*.out` = 0 in every file) - the only difference from
the default expected output is the schema name. sanity.out needs no
alternate: its assertions are all unqualified, so its output doesn't vary by
schema.

Verified after regenerating: `make test`, `make test-update`,
`make test TEST_SCHEMA=Quoted` and `make test-update TEST_SCHEMA=Quoted`
all report "All 3 tests passed" with zero regression.diffs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jnasbyupgrade
jnasbyupgrade force-pushed the extension-update-test branch from 35a0b7f to 8292c56 Compare July 22, 2026 23:30
jnasbyupgrade and others added 11 commits July 22, 2026 18:46
Taken near-verbatim from Postgres-Extensions/cat_tools PR #48: the changes
job now derives the PG matrix list from two constants (NEWEST, FLOOR) and
emits it as a job output, consumed by both the `test` and
`extension-update-test` matrices via fromJSON. Previously both jobs
hardcoded the same [17..10] list independently - exactly the drift risk this
fixes (a new PG major meant editing two places, and forgetting one would
silently undertest it).

count_nulls needs only ONE floor, unlike cat_tools's three-constant version
(NEWEST/CURRENT_FLOOR/LEGACY_FLOOR): 0.9.6 is pure SQL over anyarray/json/
jsonb with no catalog-version sensitivity, so it installs on every supported
major - there's no PG10-only legacy leg to carve out the way cat_tools's
pre-0.2.2 scripts need.

Also documents, at pg-upgrade-test, the deliberate decision NOT to add
cat_tools's companion pg-upgrade-stepwise job (one cluster climbing every
major in sequence): that job protects against a regression at one specific
major-to-major boundary, relevant to cat_tools's catalog-touching views but
not to count_nulls's pure SQL functions.

Pure refactor - the derived list resolves to the same PG majors as before
([17,16,15,14,13,12,11,10]), confirmed by running the bash derivation
standalone.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
deps.sql previously required a non-empty schema, defaulting to 'public' -
meaning the suite only ever exercised CREATE EXTENSION with an explicitly
targeted schema, never the plain, untouched default-install code path a
brand-new user actually gets. TEST_SCHEMA is now optional: empty means don't
CREATE SCHEMA/SET search_path at all, landing wherever the session's own
ambient search_path resolves; non-empty explicitly targets that schema as
before. Both legs matter and run in CI - genuinely different code paths.

Where the extension lands when nothing targets it is not always 'public' -
this suite's own pgTAP setup already puts its own schema first on
search_path before deps.sql runs, so the empty leg lands there instead.
That's not a bug to paper over (it's actually useful: it proves nothing is
hardcoded to 'public'), so assertions that need a known expected schema
(test__check_ncs, test__shutdown__drop_all) fall back to discovering the
real location dynamically (ncs()) or skip when there's no fixed expectation,
rather than assuming one.

bin/test_existing's create_extension_in_schema helper and
bin/test_existing_sql/plant_guard.sql both gained the same empty/non-empty
branch (a quoted empty identifier is a real Postgres syntax error, so the
empty case can't just always emit `CREATE SCHEMA "" `).

Also moves test/pg_upgrade/*.sql to bin/test_existing_sql/ (they're helpers
for bin/test_existing, not part of the pg_regress suite, so they belong
under bin/ alongside the script that uses them), and removes
test/sql/simple.sql: it existed to exercise "the other" schema before
TEST_SCHEMA existed, dressed up as an on-search-path test, but
functions.sql's own calls are all %I-qualified via ncs() regardless of
search_path - so its search_path manipulation never actually changed
anything, and the schema coverage it stood in for is now done properly (and
more thoroughly - two legs, not one) by the TEST_SCHEMA matrix on
extension_tests.sql itself. Confirmed no coverage is lost: everything
simple.sql exercised (test__definition, test__functionality, via
functions.sql) already runs identically inside extension_tests.sql.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
extension_tests.out: regenerated via make results for the new default
(TEST_SCHEMA empty) - the only change is the schema name throughout (now
'tap', where this suite's own pgTAP setup lands it when nothing targets a
schema, replacing the previous hardcoded 'public' default).

extension_tests_2.out: a third pg_regress alternate-expected-output variant
(existing _1.out already covers TEST_SCHEMA=Quoted), for TEST_SCHEMA=empty
runs that go through bin/test_existing's standalone psql sessions (the
extension-update-test/pg-upgrade-test CI jobs) rather than the in-suite
pg_regress session - those land in 'public' (a bare session's real ambient
default, unaffected by the pgTAP setup that only runs inside the suite),
genuinely different text from the in-suite empty-schema case. Captured from
real runs with zero TAP-level failures in both cases.

test/expected/simple.out, simple_1.out: removed along with
test/sql/simple.sql.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…t context

test/extension-update-test/pg-upgrade-test matrices: schema axis is now
["", Quoted] (without/with an explicit schema) instead of [public, Quoted],
matching TEST_SCHEMA's new empty-mode support. pg-upgrade-test's new_pg
targets bumped 17->18 to match the derived PG list's new ceiling (confirmed
working: a real cat_tools CI run on PostgreSQL 18 is green).

NEWEST bumped 17->18 in the single-source PG-major derivation.

Removed comments that leaned on cat_tools context to justify a design
choice in this repo's own CI (e.g. "unlike cat_tools's pg-upgrade-test,
which has to dig itself out of...") - every comment here should stand on
its own reasoning; a reader of this repo has no reason to know cat_tools's
history. Reworded the "worth doing anyway" phrasing into something that
actually reads coherently. Also hardened the docs-only-changes step: it now
writes docs_only=false as its literal first action, so any early exit or
error further down leaves the fail-safe default in place instead of relying
on every branch remembering to set it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same reasoning as the earlier move out of test/pg_upgrade/ - these files
are helpers for bin/test_existing specifically, so the directory name
reads as "test_existing's sql" alongside the script itself.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bin/test_existing.sql/plant_guard.sql: replaced the \if/\else branch with a
single schema_prefix variable (empty, or the quoted schema name plus a
literal '.'), computed once via quote_ident() and spliced in as plain text -
one CREATE VIEW statement instead of two branches of one.

test/sql/extension_tests.sql: test__check_ncs and test__shutdown__drop_all
now take the schema as a function parameter (schema_hint name DEFAULT
NULLIF(:'schema', '')::name) instead of calling current_setting() inside the
plpgsql body. psql doesn't interpolate variables inside dollar-quoted
bodies, which is why the previous version needed a runtime GUC read at all -
a parameter default is plain top-level SQL, so the psql substitution just
works there directly, and runtests() calling every test__* function with no
arguments always gets the default.

Added a top-of-file "Test strategy" comment to .github/workflows/ci.yml
summarizing what each job covers before the per-job comments get into
specifics, and test/README.md documenting the two independent mode switches
(TEST_LOAD_SOURCE, TEST_SCHEMA) and - concretely, with a scenario table -
exactly what drives the need for multiple expected/extension_tests*.out
variants, not just that they exist.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Captured per discussion: test/install (cat_tools's pattern) has a real,
independent argument for centralizing fresh/update/existing load-mode
switching in one place, separate from the schema-targeting question. And the
actual debuggability cost of this file's shared/loop-driven design is that
interactively \i-ing a traditional, self-contained test file into psql is
much more direct than poking at this file's generated names and shared
ncs() lookup - not really about automated TAP output specificity, which
mostly localizes failures fine already. Not urgent, not changing now, but
worth revisiting rather than forgetting next time this file is substantially
touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CREATE FUNCTION name( / args / ) RETURNS ..., matching the codebase's
established convention (e.g. sql/count_nulls.sql), instead of putting the
name and the parenthesized arg list on separate lines.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Convert a two-line -- comment to the /* */ block style used by every other
multi-line comment in this file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Its behavioral coverage (null_count over a jsonb value with one null key,
over a variadic anyarray with one null) is a strict subset of what
test__functionality's bag_eq comparisons already exercise (core/functions.sql -
the same 8-row test_data set covers every null-count combination, not just
these two). The one remaining argument for keeping it - that it's the only
test independent of pgTAP itself working - was judged not worth it on its
own. Updated test/README.md and the TODO note in core/functions.sql
accordingly; if that file's shared/loop-driven-design debuggability tradeoff
becomes a real problem later, it's worth an intentional replacement then,
not resurrecting this by default.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Front-load the thing a reader coming from typical pgTAP extension testing
would find surprising: core/functions.sql defines a shared library of
test__* functions rather than each test/sql/*.sql file being independent.
Also dropped the expected/*.out bullet - what that directory is for is
standard pgxntool knowledge, not specific to this suite.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jnasbyupgrade
jnasbyupgrade marked this pull request as ready for review July 24, 2026 20:17
@jnasbyupgrade
jnasbyupgrade merged commit 9f088ac into pgxntool-update Jul 24, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant