Add extension update test via pgxntool test-build - #11
Merged
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
jnasbyupgrade
force-pushed
the
pgxntool-update
branch
from
July 16, 2026 21:47
738cf44 to
04fa85a
Compare
test/build/upgrade.sql is a fast smoke check (via pgxntool 2.1.0's test-build feature): install 0.9.6 (oldest version we ship a full script for) and ALTER EXTENSION UPDATE, rolled back. That alone doesn't prove much — it never runs the actual test suite against the upgraded state. test/deps.sql now selects between installing fresh (current version) or via the upgrade path, based on a count_nulls.test_load_mode GUC the Makefile sets via PGOPTIONS (TEST_LOAD_SOURCE=fresh|upgrade). Since every test file loads deps.sql, `make test-update` reruns the whole existing suite (extension_tests, sanity, simple) against the upgraded install, comparing against the exact same expected/*.out as `make test` — proving the upgrade behaves identically to a fresh install rather than just completing without error. Modeled on cat_tools's test/install/load.sql approach (see their new_functions-pgxntool-2.1.0 branch), adapted to count_nulls's simpler single-file test/deps.sql structure instead of introducing a separate test/install fixture, since count_nulls's suite already installs the extension per test file (into different schemas) rather than once globally. sql/count_nulls--0.9.6.sql isn't picked up by base.mk's DATA wildcard (which only covers upgrade scripts and the current version file), so it's added to DATA explicitly. Filed as a pgxntool bug: Postgres-Extensions/pgxntool#48. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jnasbyupgrade
force-pushed
the
extension-update-test
branch
from
July 16, 2026 21:49
b11893d to
d2ff70e
Compare
Per the terminology convention: 'update' is extension-level (ALTER EXTENSION UPDATE), 'upgrade' is cluster-level (pg_upgrade) — a separate, not-yet-covered axis. The wrapper target was already called test-update; the mode value and GUC name were the odd one out. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test/deps.sql gains a third TEST_LOAD_SOURCE mode ('existing'): asserts the
extension is already installed and current, without dropping/creating/
updating anything, for use against a real pg_upgrade'd or out-of-band
updated database.
Replaces the per-test-file hardcoded schema literals
(schema_to_load_count_nulls / public) - a stand-in for real
schema-qualification coverage that only ever tested two fixed, always-
lowercase names - with a single TEST_SCHEMA make var, propagated the same
way as TEST_LOAD_SOURCE via a GUC, applied uniformly for the whole test run.
This also resolves an architectural conflict between the two: a real
pg_upgrade leaves the extension in exactly one schema, which cannot
simultaneously match two different per-file hardcoded literals, so
'existing' mode needs every file installing into the same run-wide schema
to produce comparable output.
sanity.sql previously bypassed test/deps.sql entirely (a bare, unconditional
CREATE EXTENSION), which broke outright under 'existing' mode
("extension already exists"); it now respects both TEST_LOAD_SOURCE and
TEST_SCHEMA like the rest of the suite.
Locally: `make test TEST_SCHEMA=Quoted` exercises a schema requiring SQL
identifier quoting (mixed case - unquoted would fold to lowercase and
silently test 'public' again).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test/pg_upgrade/{plant,assert,drop}_guard.sql: a view with a hard pg_depend
dependency on a stable, never-dropped/redefined extension member
(null_count(anyarray), unchanged since 0.9.0), so a non-CASCADE
DROP EXTENSION count_nulls is blocked. Proves an install/update/pg_upgrade
sequence didn't silently drop+reinstall the extension it's meant to be
testing - a stray CASCADE, a logic bug, or a bad CI step would otherwise
fall through to a silent fresh reinstall and still report green.
bin/test_existing factors the install -> plant guard -> update/upgrade ->
assert -> drop guard -> run suite sequence (needed by both the
extension-update-test and pg-upgrade-test CI jobs) into one committed,
parameterized script instead of duplicating it as inline YAML. Modeled on
Postgres-Extensions/cat_tools's script of the same name, with two
differences: count_nulls has no pg_upgrade-unsafe old version to bridge past
(it has always been pure SQL functions, no SELECT-*-over-catalog views), and
its suite has its own legitimate (harmless - always rolled back) DROP
EXTENSION test, so here the guard is dropped before run-suite instead of
surviving through it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- test: unchanged fresh-install job, now also matrixed over TEST_SCHEMA (public, Quoted) via the environment (Make imports it automatically). - extension-update-test: installs 0.9.6, plants + proves the dependency guard, ALTER EXTENSION UPDATEs to current, runs the suite against the result in existing mode - across the PG x schema matrix. - pg-upgrade-test: installs 0.9.6 on an old cluster, plants the guard, binary pg_upgrades to a newer cluster, updates to current, runs the suite against the real migrated objects in existing mode. A small old_pg/new_pg matrix (10->17, 12->17) x schema, not the full PG matrix, to keep cost reasonable given this is the most expensive job (installs two full PostgreSQL majors and runs the real pg_upgrade binary per leg). - changes: cheap docs-only gate so the three heavy jobs above can skip themselves on doc-only pushes without leaving the required check stuck Pending. - all-checks-passed: single stable required-status-check name; asserts its own needs list matches the actual job set so a new job can't be silently omitted from the gate. Locally validated end to end on this container's PG12/PG17 clusters, including a real binary pg_upgrade (PG12 -> PG17, extension carried at 0.9.6, updated to 1.0.0, guard proven through every step) before trusting this as CI YAML. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test/expected/extension_tests.out: regenerated via `make results` against
the new default (TEST_SCHEMA=public) - the only change is the schema name
throughout (schema_to_load_count_nulls -> public), consistent with deps.sql
now installing every test file into the same run-wide schema instead of a
different literal per file. simple.out/sanity.out are byte-identical to
before (they already defaulted to 'public').
test/expected/{extension_tests,simple}_1.out: pg_regress's alternate-
expected-file convention, for the TEST_SCHEMA=Quoted CI legs (a schema name
requiring SQL identifier quoting - mixed case, so unquoted would fold to
lowercase and silently retest 'public'). Captured from a real
`make test TEST_SCHEMA=Quoted` run with zero TAP-level failures (`grep -c
'^not ok' test/results/*.out` = 0 in every file) - the only difference from
the default expected output is the schema name. sanity.out needs no
alternate: its assertions are all unqualified, so its output doesn't vary by
schema.
Verified after regenerating: `make test`, `make test-update`,
`make test TEST_SCHEMA=Quoted` and `make test-update TEST_SCHEMA=Quoted`
all report "All 3 tests passed" with zero regression.diffs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jnasbyupgrade
force-pushed
the
extension-update-test
branch
from
July 22, 2026 23:30
35a0b7f to
8292c56
Compare
Taken near-verbatim from Postgres-Extensions/cat_tools PR #48: the changes job now derives the PG matrix list from two constants (NEWEST, FLOOR) and emits it as a job output, consumed by both the `test` and `extension-update-test` matrices via fromJSON. Previously both jobs hardcoded the same [17..10] list independently - exactly the drift risk this fixes (a new PG major meant editing two places, and forgetting one would silently undertest it). count_nulls needs only ONE floor, unlike cat_tools's three-constant version (NEWEST/CURRENT_FLOOR/LEGACY_FLOOR): 0.9.6 is pure SQL over anyarray/json/ jsonb with no catalog-version sensitivity, so it installs on every supported major - there's no PG10-only legacy leg to carve out the way cat_tools's pre-0.2.2 scripts need. Also documents, at pg-upgrade-test, the deliberate decision NOT to add cat_tools's companion pg-upgrade-stepwise job (one cluster climbing every major in sequence): that job protects against a regression at one specific major-to-major boundary, relevant to cat_tools's catalog-touching views but not to count_nulls's pure SQL functions. Pure refactor - the derived list resolves to the same PG majors as before ([17,16,15,14,13,12,11,10]), confirmed by running the bash derivation standalone. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
deps.sql previously required a non-empty schema, defaulting to 'public' - meaning the suite only ever exercised CREATE EXTENSION with an explicitly targeted schema, never the plain, untouched default-install code path a brand-new user actually gets. TEST_SCHEMA is now optional: empty means don't CREATE SCHEMA/SET search_path at all, landing wherever the session's own ambient search_path resolves; non-empty explicitly targets that schema as before. Both legs matter and run in CI - genuinely different code paths. Where the extension lands when nothing targets it is not always 'public' - this suite's own pgTAP setup already puts its own schema first on search_path before deps.sql runs, so the empty leg lands there instead. That's not a bug to paper over (it's actually useful: it proves nothing is hardcoded to 'public'), so assertions that need a known expected schema (test__check_ncs, test__shutdown__drop_all) fall back to discovering the real location dynamically (ncs()) or skip when there's no fixed expectation, rather than assuming one. bin/test_existing's create_extension_in_schema helper and bin/test_existing_sql/plant_guard.sql both gained the same empty/non-empty branch (a quoted empty identifier is a real Postgres syntax error, so the empty case can't just always emit `CREATE SCHEMA "" `). Also moves test/pg_upgrade/*.sql to bin/test_existing_sql/ (they're helpers for bin/test_existing, not part of the pg_regress suite, so they belong under bin/ alongside the script that uses them), and removes test/sql/simple.sql: it existed to exercise "the other" schema before TEST_SCHEMA existed, dressed up as an on-search-path test, but functions.sql's own calls are all %I-qualified via ncs() regardless of search_path - so its search_path manipulation never actually changed anything, and the schema coverage it stood in for is now done properly (and more thoroughly - two legs, not one) by the TEST_SCHEMA matrix on extension_tests.sql itself. Confirmed no coverage is lost: everything simple.sql exercised (test__definition, test__functionality, via functions.sql) already runs identically inside extension_tests.sql. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
extension_tests.out: regenerated via make results for the new default (TEST_SCHEMA empty) - the only change is the schema name throughout (now 'tap', where this suite's own pgTAP setup lands it when nothing targets a schema, replacing the previous hardcoded 'public' default). extension_tests_2.out: a third pg_regress alternate-expected-output variant (existing _1.out already covers TEST_SCHEMA=Quoted), for TEST_SCHEMA=empty runs that go through bin/test_existing's standalone psql sessions (the extension-update-test/pg-upgrade-test CI jobs) rather than the in-suite pg_regress session - those land in 'public' (a bare session's real ambient default, unaffected by the pgTAP setup that only runs inside the suite), genuinely different text from the in-suite empty-schema case. Captured from real runs with zero TAP-level failures in both cases. test/expected/simple.out, simple_1.out: removed along with test/sql/simple.sql. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…t context test/extension-update-test/pg-upgrade-test matrices: schema axis is now ["", Quoted] (without/with an explicit schema) instead of [public, Quoted], matching TEST_SCHEMA's new empty-mode support. pg-upgrade-test's new_pg targets bumped 17->18 to match the derived PG list's new ceiling (confirmed working: a real cat_tools CI run on PostgreSQL 18 is green). NEWEST bumped 17->18 in the single-source PG-major derivation. Removed comments that leaned on cat_tools context to justify a design choice in this repo's own CI (e.g. "unlike cat_tools's pg-upgrade-test, which has to dig itself out of...") - every comment here should stand on its own reasoning; a reader of this repo has no reason to know cat_tools's history. Reworded the "worth doing anyway" phrasing into something that actually reads coherently. Also hardened the docs-only-changes step: it now writes docs_only=false as its literal first action, so any early exit or error further down leaves the fail-safe default in place instead of relying on every branch remembering to set it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same reasoning as the earlier move out of test/pg_upgrade/ - these files are helpers for bin/test_existing specifically, so the directory name reads as "test_existing's sql" alongside the script itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bin/test_existing.sql/plant_guard.sql: replaced the \if/\else branch with a single schema_prefix variable (empty, or the quoted schema name plus a literal '.'), computed once via quote_ident() and spliced in as plain text - one CREATE VIEW statement instead of two branches of one. test/sql/extension_tests.sql: test__check_ncs and test__shutdown__drop_all now take the schema as a function parameter (schema_hint name DEFAULT NULLIF(:'schema', '')::name) instead of calling current_setting() inside the plpgsql body. psql doesn't interpolate variables inside dollar-quoted bodies, which is why the previous version needed a runtime GUC read at all - a parameter default is plain top-level SQL, so the psql substitution just works there directly, and runtests() calling every test__* function with no arguments always gets the default. Added a top-of-file "Test strategy" comment to .github/workflows/ci.yml summarizing what each job covers before the per-job comments get into specifics, and test/README.md documenting the two independent mode switches (TEST_LOAD_SOURCE, TEST_SCHEMA) and - concretely, with a scenario table - exactly what drives the need for multiple expected/extension_tests*.out variants, not just that they exist. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Captured per discussion: test/install (cat_tools's pattern) has a real, independent argument for centralizing fresh/update/existing load-mode switching in one place, separate from the schema-targeting question. And the actual debuggability cost of this file's shared/loop-driven design is that interactively \i-ing a traditional, self-contained test file into psql is much more direct than poking at this file's generated names and shared ncs() lookup - not really about automated TAP output specificity, which mostly localizes failures fine already. Not urgent, not changing now, but worth revisiting rather than forgetting next time this file is substantially touched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CREATE FUNCTION name( / args / ) RETURNS ..., matching the codebase's established convention (e.g. sql/count_nulls.sql), instead of putting the name and the parenthesized arg list on separate lines. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Convert a two-line -- comment to the /* */ block style used by every other multi-line comment in this file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Its behavioral coverage (null_count over a jsonb value with one null key, over a variadic anyarray with one null) is a strict subset of what test__functionality's bag_eq comparisons already exercise (core/functions.sql - the same 8-row test_data set covers every null-count combination, not just these two). The one remaining argument for keeping it - that it's the only test independent of pgTAP itself working - was judged not worth it on its own. Updated test/README.md and the TODO note in core/functions.sql accordingly; if that file's shared/loop-driven-design debuggability tradeoff becomes a real problem later, it's worth an intentional replacement then, not resurrecting this by default. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Front-load the thing a reader coming from typical pgTAP extension testing would find surprising: core/functions.sql defines a shared library of test__* functions rather than each test/sql/*.sql file being independent. Also dropped the expected/*.out bullet - what that directory is for is standard pgxntool knowledge, not specific to this suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jnasbyupgrade
marked this pull request as ready for review
July 24, 2026 20:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add full update+upgrade testing to the suite: test/deps.sql now selects between three install modes via a TEST_LOAD_SOURCE GUC - fresh (CREATE EXTENSION at current), update (installs 0.9.6, then ALTER EXTENSION UPDATE, exercised by
make test-update), and existing (asserts an already-installed extension is present and current, touching nothing else, for use against a real pg_upgrade'd or out-of-band-updated database). A second, independent TEST_SCHEMA switch runs the whole suite both WITHOUT specifying a schema at all (CREATE EXTENSION with no targeting, landing wherever the session's own ambient search_path resolves) and WITH one explicitly targeted, using a name that requires SQL identifier quoting - both legs run in CI, neither redundant with the other. See test/README.md for the full breakdown of both switches and exactly what drives the need for the two remaining expected/extension_tests*.out variants.A dependency-guard technique (bin/test_existing.sql/*.sql: a view with a hard pg_depend edge to a stable, never-dropped extension member) proves an install/update/pg_upgrade sequence didn't silently drop and reinstall the extension it's meant to be testing, which would otherwise let a regression hide behind a fresh reinstall; bin/test_existing factors install -> plant guard -> update/upgrade -> assert -> drop guard -> run suite into one committed, parameterized script instead of duplicating it as inline YAML. Three new CI jobs exercise all of this on every push: extension-update-test (0.9.6 -> current, across a PostgreSQL x schema matrix), pg-upgrade-test (a real binary pg_upgrade across PostgreSQL majors, old_pg/new_pg x schema legs), and a docs-only gate that also derives the PostgreSQL-major matrix list from a single set of constants (so a new major is a one-line change, not an edit in several jobs) feeding a stable all-checks-passed required check. ci.yml now leads with a top-of-file summary comment covering the whole matrix strategy.
Removed test/sql/simple.sql and test/sql/sanity.sql. simple.sql existed to exercise "the other" schema before TEST_SCHEMA existed, dressed up as an on-search-path test, but the extension's own functions are all schema-qualified via ncs() regardless of search_path, so its search_path manipulation never actually changed anything - the schema coverage it stood in for is now done properly (and more thoroughly) by the TEST_SCHEMA matrix on extension_tests.sql directly. sanity.sql's two direct null_count() calls turned out to be a strict subset of what test__functionality's bag_eq comparisons already cover (the same 8-row test_data set exercises every null-count combination those two calls checked, and more). Confirmed no coverage was lost either time before removing.
Adapted from Postgres-Extensions/cat_tools's own update+upgrade testing where it differs: no bridge-update leg is needed, since count_nulls has no pg_upgrade-unsafe old version to work around (a real, not rare, risk for any extension - just not one count_nulls happens to have); the dependency guard is dropped before the suite runs, rather than left standing through it, since count_nulls's suite has its own legitimate (harmless, rolled-back) DROP EXTENSION test that would otherwise collide with it; and no every-major stepwise pg_upgrade climb, since count_nulls has nothing catalog-touching that would benefit from it. The TEST_SCHEMA/quoting design was arrived at independently here, before noticing cat_tools has its own open PR doing the same thing for test role names - worth noting as a broadly useful pattern rather than a one-off.
Also simplified bin/test_existing.sql/plant_guard.sql (a single quote_ident()-built prefix variable instead of branching the whole view definition) and test/sql/extension_tests.sql's two schema-aware test__* functions (schema passed as a function parameter default instead of a current_setting() workaround inside the plpgsql body, since psql substitution doesn't reach inside dollar-quoted bodies but does reach a parameter default).
Deliberately NOT doing right now, but captured as a comment in test/core/functions.sql for next time that file is substantially touched: migrating the main suite's install to pgxntool's test/install feature (cat_tools's own pattern for centralizing fresh/update/existing switching, an independent argument for it regardless of the schema question), and revisiting the debuggability tradeoff of this file's shared/loop-driven design versus interactively \i-ing a traditional, self-contained test file in psql - the actual reason count_nulls originally had separate smoke-test files like the two just removed, worth an intentional replacement if this ever becomes a real problem rather than just letting it stay absent.
test/expected/extension_tests.out was regenerated via
make resultsfor the new default (TEST_SCHEMA empty). Two pg_regress alternate-expected-output files cover the two other cases that produce genuinely different (not wrong) text:_1.outfor TEST_SCHEMA=Quoted, and_2.outfor the empty-schema case as exercised by a bare standalone psql session (the extension-update-test/pg-upgrade-test jobs) rather than the in-suite pg_regress session, which lands in a different real schema than the in-suite case does.Verified locally against live PG12/PG17 clusters, including a real binary pg_upgrade PG12->PG17 run (extension carried at 0.9.6, updated to 1.0.0, dependency guard proven through every step) before trusting any of this as CI YAML;
make testandmake test-updatepass cleanly on both PG17 and PG12, with and without an explicit TEST_SCHEMA.Stacked on #10 (now merged).