priority lanes for rebase-cascade pushes, escalate on stack promotion #327

Workflow file for this run

	# ===========================================================================
	# Test strategy
	#
	# count_nulls can be arrived at several ways, each of which can break
	# differently, so each is exercised by its own job below:
	#
	# test -- FRESH install: CREATE EXTENSION at the current
	# version, across every supported PostgreSQL
	# major. Also proves the IN-PLACE extension
	# update path (CREATE EXTENSION at 0.9.6, then
	# ALTER EXTENSION UPDATE - same PostgreSQL, no
	# pg_upgrade) in the same job/matrix, rather
	# than a dedicated job: a load mode is just an
	# input the same assertions run against, not a
	# real environment difference, so giving it its
	# own job would only duplicate this job's own
	# per-PG-version container/checkout setup for
	# no added confidence.
	# pg-upgrade-test -- BINARY pg_upgrade: install 0.9.6 on the OLD
	# PostgreSQL floor (the same META.json-derived
	# floor the `changes` job's supported_pg uses -
	# see its floor_pg output), then binary-upgrade
	# the cluster to the newest supported major
	# (changes job's newest_pg). A single leg (not
	# the full PG matrix - by far the most
	# expensive job here, installing two full
	# PostgreSQL majors and running the real
	# pg_upgrade binary), which via twin databases
	# migrated by the SAME binary pg_upgrade call
	# compares BOTH update-vs-pg_upgrade orderings
	# a real user could hit - see the job's own
	# comment for why one floor is enough to cover
	# both.
	# pg-tle-test -- pg_tle DEPLOYMENT: fresh install AND the
	# 0.9.6 -> current update path, both registered
	# through AWS pg_tle's database-backed catalog
	# instead of a filesystem .control file.
	#
	# During testing we always install into a freshly, randomly generated
	# schema (see test/install/load.sql; test/README.md covers how
	# test/expected/extension_tests.out stays schema-invariant).
	#
	# `changes` is a cheap gate that lets the heavy jobs above skip themselves on
	# doc-only pushes, and also derives the shared PostgreSQL-major list those
	# jobs consume from a single set of constants. `all-checks-passed` is the
	# single stable required-status-check name.
	#
	# Draft PRs get a further reduction, independent of `changes`/docs_only,
	# aimed at cutting shared-runner load while a PR is still being iterated on
	# (this repo's org-wide Actions queue backs up easily): `lint` always runs
	# in full; `test`'s matrix drops to just the newest supported PostgreSQL
	# major (see its own comment) instead of running full or being skipped
	# outright, since it's cheap per-leg and a draft author still wants signal
	# on every push; every other heavy job (`pg-tle-test`, `pg-upgrade-test`)
	# is skipped entirely via an added `&& github.event.pull_request.draft !=
	# true` on its existing `if:`. None of this applies to a `push` event (e.g.
	# the post-merge run on master) or a non-draft PR, both of which always run
	# the full suite exactly as before. `pull_request.types` below includes
	# `ready_for_review` specifically so marking a draft PR ready retriggers
	# immediately at full scope (draft is already false by the time that event
	# fires) instead of leaving the reduced draft-time result on the PR's last
	# commit as its current status until some later real push.
	#
	# CI PRIORITY: a `gh stack rebase` cascade pushes to every PR above the one
	# actually being changed, purely to move it onto a new base - the PR's own
	# diff (what it actually contributes) is unchanged. Running that at the
	# same priority as real new commits means a burst of cascade pushes can
	# crowd out the runner capacity a genuinely active push needs for fast
	# feedback. `changes` tells the two apart via a base-independent content
	# hash (bin/patch_id_hash, robust to rebase-induced context shifts - see
	# its own comment - persisted per-PR across pushes via actions/cache since
	# GitHub Actions has no other cross-run memory): a synchronize push whose
	# hash matches the last one observed for this PR is routed to a
	# LOW-priority lane (`ci-<job>-lowprio-<0..2>`, a small fixed set shared
	# across ALL PRs, capping rebase noise to at most 3 concurrent runs
	# repo-wide instead of letting it consume the account's whole concurrency
	# budget) rather than the per-PR high-priority lane real pushes use. It
	# still gets the FULL matrix either way, just possibly later - a rebase CAN
	# break something the diff itself didn't touch, and this repo won't merge
	# without a clean run regardless. `edited` is in `pull_request.types` so a
	# base retarget (the signal gh stack sends when this PR is promoted to the
	# bottom of its stack - the next one due to merge) can be caught by the
	# `escalate` step and bumped to an immediate high-priority run bypassing
	# whatever lane its last push landed in, UNLESS a full run already exists
	# for that exact head SHA (checked via the Checks API), in which case
	# there's nothing to gain by re-running it. `ready_for_review` gets the
	# same escalation unconditionally (no existing-run check): its prior run,
	# if any, was necessarily the reduced draft-time one above, which doesn't
	# count as "already had one".
	# ===========================================================================
	name: CI
	on:
	push:
	branches:
	- master
	pull_request:
	# Explicit (not the GitHub default of [opened, synchronize, reopened]):
	# `ready_for_review` retriggers immediately when a draft PR is marked
	# ready (see the top-of-file comment); `edited` catches a base retarget
	# for the priority-escalation check in the `changes` job below.
	types: [opened, synchronize, reopened, edited, ready_for_review]
	jobs:
	# Cheap gate that lets the heavy jobs below skip themselves on commits that
	# touch only docs. Must run on every push/pull_request (no paths-ignore on
	# the workflow itself), otherwise the required all-checks-passed check
	# would never report on doc-only pushes and get stuck Pending in branch
	# protection.
	#
	# Also derives the supported-PostgreSQL-major list every other job reads,
	# from a single NEWEST constant plus META.json's own declared minimum (not
	# a second hand-maintained floor constant) - every job that cares which
	# majors are supported reads the SAME values, so they can't silently drift
	# onto different sets, and adding a new major is a one-line change here
	# instead of an edit in several jobs.
	# - supported_pg: the full descending list (NEWEST down to the
	# META.json-derived floor), as a JSON array, for jobs that need every
	# supported major.
	# - newest_pg: just the single newest major, as a bare scalar, for
	# anything that needs one value instead of the full array.
	# - floor_pg: just the single oldest (META.json-derived) major, as a
	# bare scalar, for the same reason.
	changes:
	name: 🔍 Detect docs-only changes & derive PG matrix
	runs-on: ubuntu-latest
	permissions:
	contents: read
	checks: read # escalate step: has this SHA already got a full run?
	outputs:
	docs_only: ${{ steps.diff.outputs.docs_only }}
	supported_pg: ${{ steps.pg.outputs.supported_pg }}
	newest_pg: ${{ steps.pg.outputs.newest_pg }}
	floor_pg: ${{ steps.pg.outputs.floor_pg }}
	priority: ${{ steps.diff.outputs.priority }}
	lane: ${{ steps.lane.outputs.lane }}
	steps:
	- name: Check out the repo
	uses: actions/checkout@v4
	with:
	# Full history needed so BASE and HEAD below are both reachable
	# for `git diff`.
	fetch-depth: 0

	- name: Base retargeted onto a new branch? Check whether this SHA already has a full run
	# Fires only for `edited` WITH a base change - the gh-stack-promotion
	# signal described at the top of this file. A plain title/body edit
	# also fires `edited` but leaves `github.event.changes.base` absent,
	# so it's excluded here and falls through to the diff step's
	# edited-with-nothing-to-do branch below.
	id: escalate
	if: >-
	github.event_name == 'pull_request' &&
	github.event.action == 'edited' &&
	github.event.changes.base != null
	env:
	GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	REPO: ${{ github.repository }}
	SHA: ${{ github.event.pull_request.head.sha }}
	run: \|
	existing=$(gh api "repos/$REPO/commits/$SHA/check-runs" --paginate \
	--jq '[.check_runs[] \| select(.name == "all-checks-passed")]' 2>/dev/null \|\| echo '[]')
	count=$(echo "$existing" \| jq 'length')
	if [ "$count" -gt 0 ]; then
	echo "escalate=false" >> "$GITHUB_OUTPUT"
	echo "SHA $SHA already has an all-checks-passed run - nothing to escalate"
	else
	echo "escalate=true" >> "$GITHUB_OUTPUT"
	fi

	- name: Restore this PR's previously observed content patch-id
	id: patchid-restore
	if: github.event_name == 'pull_request' && github.event.action != 'edited'
	uses: actions/cache/restore@v4
	with:
	path: /tmp/prev_patch_id
	# A key guaranteed not to already exist, so this always falls
	# through to restore-keys (prefix match, picks the MOST RECENT
	# save) instead of ever hitting its own key directly.
	key: unused-${{ github.run_id }}
	restore-keys: \|
	ci-patchid-pr-${{ github.event.pull_request.number }}-

	- name: Compute per-push changed files, docs-only status, and CI priority
	id: diff
	run: \|
	# Fail safe to running the full matrix at high priority: default
	# both immediately, before anything below has a chance to compute
	# or fail. Writing the same GITHUB_OUTPUT key twice is fine (the
	# last write wins), so the only way this step ends with
	# docs_only=true or priority=low is by genuinely proving it
	# further down - never by skipping past an edge case with a
	# default.
	echo "docs_only=false" >> "$GITHUB_OUTPUT"
	echo "priority=high" >> "$GITHUB_OUTPUT"

	if [ "${{ github.event.action }}" = "edited" ]; then
	# Either a base retarget with nothing outstanding to run
	# (escalate above said false - an all-checks-passed run already
	# covers this SHA), or a plain title/description edit (no base
	# change at all, so escalate didn't even run). Either way,
	# there's no code to test right now.
	if [ "${{ steps.escalate.outputs.escalate }}" != "true" ]; then
	echo "docs_only=true" >> "$GITHUB_OUTPUT"
	fi
	exit 0
	fi

	if [ "${{ github.event_name }}" = "pull_request" ] && \
	[ "${{ github.event.action }}" = "synchronize" ] && \
	[ -n "${{ github.event.before }}" ]; then
	# A push to an already-open PR: before/after give the true
	# per-push diff, same as for a branch push.
	BASE="${{ github.event.before }}"
	HEAD="${{ github.event.after }}"
	elif [ "${{ github.event_name }}" = "pull_request" ]; then
	# First run for this PR (opened/reopened/etc, or synchronize
	# without a usable before): fall back to the whole base...head
	# diff.
	BASE="${{ github.event.pull_request.base.sha }}"
	HEAD="${{ github.event.pull_request.head.sha }}"
	else
	BASE="${{ github.event.before }}"
	HEAD="${{ github.event.after }}"
	fi

	echo "base=$BASE"
	echo "head=$HEAD"

	# A missing HEAD, or an all-zeros BASE (e.g. a new branch's first
	# push, where GitHub reports no prior commit), means we can't
	# compute a real diff. docs_only is already false from above;
	# just stop here rather than risk skipping tests.
	if [ -z "$HEAD" ] \|\| [ -z "$BASE" ] \|\| [[ "$BASE" =~ ^0+$ ]]; then
	exit 0
	fi

	CHANGED=$(git diff --name-only "$BASE" "$HEAD" \|\| echo __DIFF_FAILED__)

	DOCS_ONLY=true
	if [ "$CHANGED" = "__DIFF_FAILED__" ] \|\| [ -z "$CHANGED" ]; then
	DOCS_ONLY=false
	else
	while IFS= read -r f; do
	if ! [[ "$f" =~ \.(md\|asc)$ ]]; then
	DOCS_ONLY=false
	break
	fi
	done <<< "$CHANGED"
	fi

	echo "changed files:"
	echo "$CHANGED"
	echo "docs_only=$DOCS_ONLY" >> "$GITHUB_OUTPUT"

	# Priority: a synchronize push whose own patch-id (the PR's
	# actual contribution, independent of what it's based on) matches
	# the last one observed for this PR is a pure gh-stack rebase
	# cascade, not new work - safe to route to the low-priority lane.
	# Real new commits, and this PR's first appearance (nothing to
	# compare against yet), keep the high priority set above.
	if [ "${{ github.event_name }}" = "pull_request" ]; then
	NEW_PATCH_ID=$(bin/patch_id_hash "${{ github.event.pull_request.base.sha }}" "${{ github.event.pull_request.head.sha }}")
	echo "content patch-id: $NEW_PATCH_ID"
	if [ "${{ github.event.action }}" = "synchronize" ] && \
	[ -f /tmp/prev_patch_id ] && \
	[ "$(cat /tmp/prev_patch_id)" = "$NEW_PATCH_ID" ]; then
	echo "priority=low" >> "$GITHUB_OUTPUT"
	fi
	echo "$NEW_PATCH_ID" > /tmp/prev_patch_id
	fi

	- name: Save this push's content patch-id for the next push to compare against
	if: github.event_name == 'pull_request' && github.event.action != 'edited'
	uses: actions/cache/save@v4
	with:
	path: /tmp/prev_patch_id
	key: ci-patchid-pr-${{ github.event.pull_request.number }}-${{ github.run_id }}

	- name: Assign a low-priority lane number
	id: lane
	# A fixed small number of shared lanes (not one lane per PR) bounds
	# how much of the account's concurrent-runner budget rebase-cascade
	# noise can ever occupy at once, so it can't crowd out real-work
	# runs waiting for a runner - at the cost of not deduping a stale
	# rebase run against a newer one for the SAME PR if they land in the
	# same lane as an unrelated PR's job (acceptable: a stale completed
	# check on an old SHA doesn't block merging the new SHA).
	if: github.event_name == 'pull_request'
	run: echo "lane=$(( ${{ github.event.pull_request.number }} % 3 ))" >> "$GITHUB_OUTPUT"

	- name: Derive the supported-PostgreSQL-major list
	id: pg
	run: \|
	# A dozen-odd lines to replace what looks like a handful of version
	# references, but it buys CONSISTENCY: every job that needs the
	# supported-major list derives it from this ONE source, so none of
	# them can silently drift onto a different list. Adding a new
	# major is a one-line NEWEST bump here, not an edit in N places.
	#
	# The floor itself is NOT a second hardcoded constant here: it's
	# read straight from META.json's own build prereq (see
	# META.in.json's comment: "Depends on JSONB, created in 9.4"),
	# which is the actual source of truth for count_nulls' minimum
	# supported major. Hand-maintaining a duplicate floor constant in
	# this workflow is exactly how it ended up testing down to 10 while
	# META.json claimed 9.4 - 9.4/9.5/9.6 went untested for a while
	# even though pg-start genuinely installs them fine on current
	# Actions infra, and nobody noticed the gap.
	NEWEST=18

	DECLARED_FLOOR=$(jq -r '.prereqs.build.requires.PostgreSQL' META.json) # e.g. "9.4.0"
	FLOOR_MAJOR=$(echo "$DECLARED_FLOOR" \| cut -d. -f1)
	if [ "$FLOOR_MAJOR" -lt 10 ]; then
	# PostgreSQL's pre-10 versioning used X.Y as the major identifier
	# (10+ switched to a single integer) - e.g. "9.4.0" -> major
	# "9.4", not "9".
	FLOOR=$(echo "$DECLARED_FLOOR" \| cut -d. -f1,2)
	else
	FLOOR="$FLOOR_MAJOR"
	fi

	# A sub-10 floor needs two `seq` runs stitched together: one for
	# the integer majors (10..NEWEST) and one for the X.Y majors
	# (9.FLOOR_MINOR..9.6, since 9.6 was the last 9.x release).
	if [[ "$FLOOR" == 9.* ]]; then
	FLOOR_MINOR=$(echo "$FLOOR" \| cut -d. -f2)
	supported="$(seq "$NEWEST" -1 10) $(seq 6 -1 "$FLOOR_MINOR" \| sed 's/^/9./')"
	else
	supported=$(seq "$NEWEST" -1 "$FLOOR")
	fi

	# Emit a JSON array of STRINGS, not bare numbers - "9.4" isn't a
	# valid bare integer token, and keeping every leg (10+ and 9.x
	# alike) as a string keeps matrix.pg comparisons/interpolations
	# consistent regardless of which leg they're in. $supported is
	# deliberately unquoted below: it word-splits (on both the spaces
	# AND the newlines `seq`/the concatenation above produce) into
	# individual version tokens for printf to repeat over.
	json() { printf '"%s",' $supported \| sed 's/,$//; s/^/[/; s/$/]/'; }

	echo "supported_pg=$(json)" >> "$GITHUB_OUTPUT"

	# Also emitted as a bare scalar (not a JSON array), for anything
	# that needs just the single newest major instead of the full
	# array - so NEWEST still only needs to change in one place.
	echo "newest_pg=$NEWEST" >> "$GITHUB_OUTPUT"

	# Same idea for the OLDEST supported major: emit the exact $FLOOR
	# value supported_pg above already derived from META.json, as a
	# bare scalar, for anything that needs just the single oldest
	# major instead of the full array - rather than hand-maintaining a
	# THIRD copy of this number.
	echo "floor_pg=$FLOOR" >> "$GITHUB_OUTPUT"

	lint:
	name: 🧹 SQL lint
	runs-on: ubuntu-latest
	steps:
	- name: Check out the repo
	uses: actions/checkout@v4
	- name: make lint
	# Deliberately not pre-initializing the .vendor/linter submodule via
	# `submodules:` above -- letting `make lint` self-init it (lint.mk)
	# is what actually proves that works from a plain clone.
	run: make lint

	# Fresh install, then the in-place extension update path, both across the
	# PG matrix. The update leg CREATE EXTENSIONs at the oldest version we
	# still ship a full install script for (0.9.6), then ALTER EXTENSION
	# UPDATEs to current (no pg_upgrade, same PostgreSQL) and reruns the
	# suite - a single job rather than a dedicated one, since a load mode is
	# just an input the same assertions run against, not a real environment
	# difference, and the per-version container/checkout setup would
	# otherwise be duplicated across two jobs with the same PG matrix. Each
	# install (both legs) always targets its own freshly, randomly generated
	# schema (see test/install/load.sql), so both legs pass against the SAME
	# test/expected/extension_tests.out (see test/README.md for how the
	# suite keeps its output schema-invariant).
	test:
	needs: [changes]
	if: needs.changes.outputs.docs_only != 'true'
	# "ci-test-" prefix, distinct from pg-upgrade-test's/pg-tle-test's below:
	# those run CONCURRENTLY with this job within the same PR's workflow run,
	# so sharing a group name across job types would make them cancel/queue
	# behind EACH OTHER instead of behind their own prior runs for this job.
	# See the top-of-file CI PRIORITY comment for the low-priority lane
	# scheme this implements.
	#
	# The high-priority group includes ${{ matrix.pg }}: without it, every
	# leg of THIS SAME push's own matrix would collide on one shared group
	# name and cancel-in-progress would cancel siblings, not just a prior
	# push's run - the exact opposite of "always runs the full matrix
	# either way". Confirmed live: before this split, most legs of a normal
	# (non-rebase) push came back cancelled instead of completing.
	#
	# KNOWN OPEN ISSUE, not yet resolved: the low-priority group deliberately
	# has NO such per-leg split (that's the point - it's the shared,
	# capped-concurrency lane), but GitHub Actions' default concurrency
	# behavior only keeps ONE job PENDING per group at a time and CANCELS
	# the previous pending one when a new one arrives (no `queue:` support
	# for an expression, and `queue: max` can't combine with a
	# conditionally-true `cancel-in-progress`) - confirmed live: pushing an
	# unchanged-patch-id follow-up here cancelled most of this job's own
	# matrix legs instead of queueing them, contradicting "still gets the
	# FULL matrix either way" above for any push whose matrix has more legs
	# than fit through instantaneously. Needs a real decision (accept a
	# weaker/no concurrency cap, split low-priority into its own dedicated
	# job with `queue: max`, or something else) before relying on this path.
	concurrency:
	group: >-
	${{ needs.changes.outputs.priority == 'low'
	&& format('ci-test-lowprio-{0}', needs.changes.outputs.lane)
	\|\| format('ci-test-{0}-{1}', github.event.pull_request.number \|\| github.sha, matrix.pg) }}
	cancel-in-progress: ${{ needs.changes.outputs.priority != 'low' }}
	strategy:
	matrix:
	# From the single source in the changes job. On a draft PR, reduced
	# to just the newest supported major (never skipped outright, unlike
	# the other heavy jobs below - this is the one signal a draft author
	# still wants on every push): `github.event.pull_request.draft` is
	# null/falsy for a push event (e.g. the post-merge run on master), so
	# this expression falls through to the full list there with no extra
	# guard needed.
	pg: ${{ github.event.pull_request.draft && fromJSON(format('[{0}]', needs.changes.outputs.newest_pg)) \|\| fromJSON(needs.changes.outputs.supported_pg) }}
	name: 🐘 PostgreSQL ${{ matrix.pg }}
	runs-on: ubuntu-latest
	container: pgxn/pgxn-tools
	steps:
	- name: Start PostgreSQL ${{ matrix.pg }}
	run: pg-start ${{ matrix.pg }}
	- name: Check out the repo
	uses: actions/checkout@v4
	- name: Test on PostgreSQL ${{ matrix.pg }}
	run: make test
	- name: Install count_nulls
	run: make install
	- name: Update 0.9.6 -> current and run the suite
	run: make verify-results TEST_LOAD_SOURCE=update
	- name: Structurally compare the updated objects against a fresh install
	# A fixed pgTAP suite only proves the specific behaviors it asserts
	# still hold; it can't catch an update script that leaves some
	# definition/comment/ACL subtly different from what a fresh install
	# of the same version produces. bin/compare_fresh_vs_update installs
	# both ways itself, in its own scratch databases - each an
	# unqualified `CREATE EXTENSION`, so both land in the same default
	# schema by construction, which is all the diff needs to isolate
	# real update-vs-fresh divergence rather than a schema-name
	# difference - and diffs every object the extension owns; any
	# nonempty diff fails the step. Not a make target (it's a
	# standalone script, not `make test`), so invoked directly here.
	run: bin/compare_fresh_vs_update 0.9.6

	# Proves count_nulls survives a BINARY pg_upgrade (in-place catalog
	# migration to a newer PostgreSQL major). Installs 0.9.6 on the oldest
	# supported PostgreSQL floor, plants a dependency guard, binary-
	# pg_upgrades to the newest supported major, then runs the suite against
	# the REAL migrated objects in existing mode.
	#
	# A single old_pg/new_pg leg (old_pg from the changes job's floor_pg
	# output, new_pg from its newest_pg output - see that job's comment; NOT
	# a second/third hardcoded copy of either number), which via TWIN
	# databases in the SAME cluster, migrated by a SINGLE pg_upgrade call,
	# compares BOTH update-vs-pg_upgrade orderings a real user could hit:
	# upgrade_oldest_first -- stays at 0.9.6 through the
	# binary pg_upgrade, updated to current AFTER it (on the NEW
	# cluster). Proves pg_upgrade itself correctly preserves objects
	# that are STILL at the oldest supported version at the moment of
	# the upgrade - a real-world scenario (a user upgrading their
	# PostgreSQL major while still running an old extension version).
	# upgrade_current_first -- updated to current BEFORE the
	# binary pg_upgrade (on the OLD cluster). Proves pg_upgrade
	# correctly migrates the objects count_nulls' CURRENT code actually
	# creates - migrating 0.9.6's objects and updating afterward would
	# instead test whether pg_upgrade can migrate a legacy structure
	# frozen in the past, which isn't actionable on its own (that
	# version already shipped; nothing to fix if it turned out
	# fragile) - which is exactly why this leg exists alongside
	# oldest_first rather than instead of it.
	# The ordering comparison is a property of pg_upgrade itself, not of
	# which PG floor it runs from, so a single floor is enough to cover
	# both orderings - there was never a technical reason for a second
	# old_pg floor here (the previous old_pg=12 leg predated this ordering
	# comparison and had no purpose once it existed).
	#
	# Both databases still end up at the current version by the time
	# run-suite runs, so nothing downstream (the pgTAP suite, the
	# dependency guard check) needs to differ - only which side(s) of the
	# pg_upgrade call each database's update happens on. No bridge-update
	# step first: count_nulls has always been pure SQL functions with no
	# SELECT-*-over-catalog views, so it has no known pg_upgrade-unsafe old
	# version to bridge past.
	#
	# Deliberately not doing a stepwise every-major-in-sequence climb (one
	# cluster walking 10->11->12->...->newest, vs. the single big jump
	# here): that would catch a regression specific to one particular
	# major-to-major boundary, which would matter if count_nulls had views/
	# functions touching catalog internals, but it doesn't - pure SQL
	# functions over anyarray/json/jsonb, nothing version-sensitive to
	# break at a specific boundary. Revisit if count_nulls ever grows
	# something catalog-touching.
	pg-upgrade-test:
	needs: [changes]
	# Skipped outright (not just matrix-reduced like `test`) on a draft PR:
	# this is a heavy job, and a draft author doesn't need a real binary
	# pg_upgrade re-proven on every push while still iterating.
	if: needs.changes.outputs.docs_only != 'true' && github.event.pull_request.draft != true
	# "ci-pgupgrade-" prefix, matrix.old_pg/new_pg split - see the `test`
	# job's concurrency comment above (currently a single-combination
	# matrix, but split the same way regardless in case that ever changes).
	concurrency:
	group: >-
	${{ needs.changes.outputs.priority == 'low'
	&& format('ci-pgupgrade-lowprio-{0}', needs.changes.outputs.lane)
	\|\| format('ci-pgupgrade-{0}-{1}-{2}', github.event.pull_request.number \|\| github.sha, matrix.old_pg, matrix.new_pg) }}
	cancel-in-progress: ${{ needs.changes.outputs.priority != 'low' }}
	strategy:
	matrix:
	# Single-element lists built from the changes job's floor_pg/
	# newest_pg scalar outputs (see that job's comment), NOT a second/
	# third hardcoded copy of either number - keeping this a `matrix:`
	# (rather than plain `env:`) preserves the ${{ matrix.old_pg }}/
	# ${{ matrix.new_pg }} interpolations used throughout the steps
	# below unchanged.
	old_pg: ["${{ needs.changes.outputs.floor_pg }}"]
	new_pg: ["${{ needs.changes.outputs.newest_pg }}"]
	name: 🔄 Binary pg_upgrade ${{ matrix.old_pg }} → ${{ matrix.new_pg }}
	runs-on: ubuntu-latest
	container: pgxn/pgxn-tools
	env:
	# Both clusters must use the same initdb options or pg_upgrade
	# refuses to run.
	INITDB_OPTS: --data-checksums --auth trust
	steps:
	- name: Install PostgreSQL ${{ matrix.old_pg }} (cluster created next, with the right initdb options from the start)
	run: NO_CLUSTER=1 pg-start ${{ matrix.old_pg }}
	- name: Create old cluster with data checksums enabled
	run: \|
	# -p 5432: pg_createcluster would otherwise assign the next
	# available port; force 5432 so subsequent psql/createdb calls
	# connect without -p.
	pg_createcluster -p 5432 ${{ matrix.old_pg }} test -- $INITDB_OPTS
	pg_ctlcluster ${{ matrix.old_pg }} test start
	pg_isready -t 30
	- name: Check out the repo
	uses: actions/checkout@v4
	- name: Install count_nulls into old cluster
	run: make install
	- name: Prepare the old cluster (install + dependency guard), twin databases
	# Two separate databases (distinct names, one per ordering) so
	# both exist in the SAME cluster ahead of the single pg_upgrade
	# call below - that one binary upgrade migrates both at once.
	# prepare-old installs count_nulls at 0.9.6 in each, then plants +
	# proves the dependency guard, so a later accidental CASCADE drop
	# anywhere in this job cannot silently make the eventual
	# existing-mode run test a fresh install instead.
	run: \|
	bin/test_existing prepare-old upgrade_oldest_first 0.9.6
	bin/test_existing prepare-old upgrade_current_first 0.9.6
	- name: Update to current before pg_upgrade (proves pg_upgrade preserves CURRENT objects)
	# Updates ONLY upgrade_current_first
	# (upgrade_oldest_first must NOT be touched here - it
	# stays at 0.9.6 until AFTER pg_upgrade, below). Exercises ALTER
	# EXTENSION UPDATE on the OLD cluster, BEFORE the binary pg_upgrade
	# below, running the 0.9.6->stable update script, so pg_upgrade
	# runs against already-current objects, not 0.9.6 ones. `make
	# install` above already installed the current version's update
	# scripts/control file into this (old) cluster's sharedir, so
	# they're in place for this ALTER EXTENSION UPDATE to use.
	run: bin/test_existing update upgrade_current_first
	- name: Install PostgreSQL ${{ matrix.new_pg }}
	run: apt-get install -y postgresql-${{ matrix.new_pg }} postgresql-server-dev-${{ matrix.new_pg }}
	- name: Install count_nulls into new cluster
	# PG_CONFIG must be specified explicitly: at this point both old
	# and new PostgreSQL are installed, and the default pg_config on
	# PATH may not be the new version's.
	run: make install PG_CONFIG=/usr/lib/postgresql/${{ matrix.new_pg }}/bin/pg_config
	- name: Stop old cluster, binary pg_upgrade to PostgreSQL ${{ matrix.new_pg }}, start new cluster
	run: \|
	pg_ctlcluster ${{ matrix.old_pg }} test stop
	pg_createcluster -p 5432 ${{ matrix.new_pg }} test -- $INITDB_OPTS
	# PG17+ writes logs to $new_datadir/pg_upgrade_output.d/; older
	# versions write to CWD. Search both on failure.
	mkdir -p /tmp/pg_upgrade_logs
	chown postgres:postgres /tmp/pg_upgrade_logs
	su -c "cd /tmp/pg_upgrade_logs && /usr/lib/postgresql/${{ matrix.new_pg }}/bin/pg_upgrade \
	-b /usr/lib/postgresql/${{ matrix.old_pg }}/bin \
	-B /usr/lib/postgresql/${{ matrix.new_pg }}/bin \
	-d /var/lib/postgresql/${{ matrix.old_pg }}/test \
	-D /var/lib/postgresql/${{ matrix.new_pg }}/test \
	-o '-c config_file=/etc/postgresql/${{ matrix.old_pg }}/test/postgresql.conf' \
	-O '-c config_file=/etc/postgresql/${{ matrix.new_pg }}/test/postgresql.conf'" postgres \
	\|\| { find /tmp/pg_upgrade_logs \
	/var/lib/postgresql/${{ matrix.new_pg }}/test/pg_upgrade_output.d \
	-name '*.log' 2>/dev/null \| sort \| xargs -r tail -n +1; exit 1; }
	pg_ctlcluster ${{ matrix.new_pg }} test start
	- name: Update to current after pg_upgrade (proves pg_upgrade preserves OLDEST-version objects)
	# Exercises ALTER EXTENSION UPDATE AFTER the binary pg_upgrade
	# above, against the NEW cluster (already started by the previous
	# step, and the only cluster listening on 5432 at this point,
	# since the old one was stopped in that same step), on
	# upgrade_oldest_first ONLY (upgrade_current_first
	# was already updated before the upgrade, above, and must not be
	# updated again here) - the 0.9.6 objects prepare-old planted into
	# that database are thus still at 0.9.6 at the moment pg_upgrade
	# itself runs, so this proves pg_upgrade correctly preserves
	# objects still frozen at the oldest supported version, not just
	# already-current ones. `make install` into the new cluster above
	# already installed the current version's update scripts/control
	# file into ITS sharedir, so they're in place for this ALTER
	# EXTENSION UPDATE to use.
	run: bin/test_existing update upgrade_oldest_first
	- name: Run the suite against the pg_upgraded database(s) (existing mode), twin databases
	# run-suite asserts the version, re-proves the dependency guard
	# still blocks a non-CASCADE drop (i.e. it survived both the update
	# and pg_upgrade), drops the guard, then runs the suite against the
	# REAL pg_upgraded database via --use-existing (so pg_regress does
	# not drop/recreate it) - a plain fresh `make test` would silently
	# test a fresh install instead of the migrated objects. Once per
	# database, since each holds an independent ordering's result.
	run: \|
	bin/test_existing run-suite upgrade_oldest_first
	bin/test_existing run-suite upgrade_current_first
	- name: Structurally compare the pg_upgraded databases against a fresh install
	# Same rationale as the test job's own update leg's use of this tool
	# (see above), but here the "other side" is a REAL database a binary
	# pg_upgrade + ALTER EXTENSION UPDATE (in one order or the other)
	# just produced, not a scratch database this tool created itself -
	# passed as EXISTING_DB so the script queries it in place instead of
	# re-deriving it, discovering that database's OWN randomly generated
	# schema (each of the twin databases got an independent one from
	# bin/test_existing prepare-old) rather than generating a new one, so
	# both sides of each comparison still land in the same schema. Once
	# per database, since each holds an independent ordering's result.
	# Catches a divergence class the fixed pgTAP suite above doesn't: an
	# object left subtly different (body, comment, ACL) by surviving a
	# real catalog migration, as opposed to only an in-place update.
	run: \|
	bin/compare_fresh_vs_update 0.9.6 upgrade_oldest_first
	bin/compare_fresh_vs_update 0.9.6 upgrade_current_first

	# Covers both a fresh install AND the 0.9.6 -> current update path, both
	# purely via pg_tle. pgxntool 2.3.0's fix for installcheck's ordering bug
	# (Postgres-Extensions/pgxntool#83) made `installcheck` (and so `make
	# test`) unconditionally depend on `install`, which writes a real
	# .control file to disk - defeating the entire point of proving a pg_tle
	# deployment never touches the filesystem. There's still no upstream fix
	# for that (Postgres-Extensions/pgxntool#90, open) that would let
	# bin/test_existing's real pgTAP suite run without it, so the update-path
	# steps below use TEST_EXISTING_DEPLOY=pgtle (see bin/test_existing),
	# which instead sandboxes `make test`'s install step behind a scratch
	# DESTDIR - harmless here since a pg_tle-deployed database never needs
	# those files.
	pg-tle-test:
	needs: [changes]
	# Skipped outright (not just matrix-reduced like `test` above) on a
	# draft PR: this is a heavy job, and a draft author doesn't need the
	# pg_tle deployment path re-proven on every push while still iterating.
	if: needs.changes.outputs.docs_only != 'true' && github.event.pull_request.draft != true
	# "ci-pgtle-" prefix, matrix.pg split - see the `test` job's concurrency
	# comment above.
	concurrency:
	group: >-
	${{ needs.changes.outputs.priority == 'low'
	&& format('ci-pgtle-lowprio-{0}', needs.changes.outputs.lane)
	\|\| format('ci-pgtle-{0}-{1}', github.event.pull_request.number \|\| github.sha, matrix.pg) }}
	cancel-in-progress: ${{ needs.changes.outputs.priority != 'low' }}
	strategy:
	matrix:
	# Intersection of count_nulls' own supported range (10-18, see the
	# `test` job above / #21) with pg_tle 1.5.2's supported range (12-18,
	# see pgxntool/pgtle_versions.md): drop 10 and 11 since pg_tle doesn't
	# support them.
	pg: [18, 17, 16, 15, 14, 13, 12]
	name: 🧩 pg_tle ${{ matrix.pg }}
	runs-on: ubuntu-latest
	container: pgxn/pgxn-tools
	env:
	# NOT named PGTLE_VERSION: on the pgxntool version this job was
	# written against, that collided with pgxntool's own Makefile variable
	# of the same name (`pgtle:`'s
	# `$(if $(PGTLE_VERSION),--pgtle-version $(PGTLE_VERSION))`), which Make
	# auto-imports from the environment - `make run-pgtle` would silently
	# generate into pg_tle/1.5.2/ (the literal version) instead of the
	# correct range directory pg_tle/1.5.0+/, then fail when --run looked
	# for the range directory and didn't find it. Filed as
	# Postgres-Extensions/pgxntool#78, fixed upstream in pgxntool 2.3.0
	# (renamed to PGXNTOOL_PGTLE_VERSION - see this repo's pgxntool bump).
	# Keeping our own name distinct from EITHER of those regardless -
	# still the simplest way to guarantee no future collision.
	PG_TLE_BUILD_VERSION: "1.5.2"
	steps:
	# A dedicated cluster, never shared with the other jobs in this
	# workflow: pg_tle requires shared_preload_libraries and mixing
	# pg_tle/non-pg_tle extension installs on one cluster can misbehave.
	- name: Start PostgreSQL ${{ matrix.pg }}
	run: pg-start ${{ matrix.pg }}
	- name: Check out the repo
	uses: actions/checkout@v4
	- name: Install rsync
	run: apt-get install -y rsync
	- name: Snapshot filesystem extension control files (pre-pgtap baseline)
	# Whatever ships on disk by default (e.g. contrib), before installing
	# pgTAP. Lets the next step prove pgTAP is the ONLY thing `make pgtap`
	# puts on disk, instead of trusting that and folding whatever it did
	# into the pre-pg_tle baseline unexamined -- a future pgxntool change
	# to `make pgtap` that also happened to touch count_nulls' own files
	# would otherwise be silently absorbed into that baseline and never
	# get flagged by any later check.
	run: bin/assert_fs_clean snapshot ${{ matrix.pg }} /tmp/pre_pgtap_baseline.txt
	- name: Install pgtap (test harness dependency)
	# pgTAP is a filesystem-installed dependency of the TEST HARNESS, not
	# part of what this job proves is pg_tle-only -- it's not being
	# deployed via pg_tle here, and never will be.
	run: make pgtap
	- name: Verify make pgtap installed exactly pgtap.control, nothing else
	run: bin/assert_fs_clean verify ${{ matrix.pg }} /tmp/pre_pgtap_baseline.txt pgtap.control
	- name: Snapshot filesystem extension control files (pre-pg_tle baseline)
	# Everything on disk now that pgtap is confirmed the only addition
	# (contrib, pgtap). bin/assert_fs_clean's later checks diff against
	# this, so they flag ANY extension that lands on disk instead of
	# being registered via pg_tle -- not just count_nulls -- without
	# hardcoding contrib/pgtap names.
	run: bin/assert_fs_clean snapshot ${{ matrix.pg }} /tmp/control_baseline.txt
	- name: Build and install pg_tle ${{ env.PG_TLE_BUILD_VERSION }}
	# flex/bison/libkrb5-dev aren't in the pgxn-tools image; pg_tle's build
	# needs them (guc-file.l, and clientauth.c includes gssapi.h).
	run: \|
	apt-get install -y flex bison libkrb5-dev
	git clone --branch v${{ env.PG_TLE_BUILD_VERSION }} --depth 1 https://github.com/aws/pg_tle.git /tmp/pg_tle
	make -C /tmp/pg_tle install
	- name: Enable pg_tle and restart PostgreSQL ${{ matrix.pg }}
	run: \|
	echo "shared_preload_libraries = 'pg_tle'" >> /etc/postgresql/${{ matrix.pg }}/test/postgresql.conf
	pg_ctlcluster ${{ matrix.pg }} test restart
	pg_isready -t 30
	- name: Register pg_tle + count_nulls against template1
	# template1, not the ambient default db: pg_tle's registration catalog
	# is per-database, and `createdb` only inherits it because it copies
	# template1 by default. Every count_nulls database used below (the
	# smoke-test db) is created AFTER this step specifically so it
	# inherits both registrations.
	run: \|
	psql -d template1 -c "CREATE EXTENSION pg_tle"
	PGDATABASE=template1 make run-pgtle
	- name: Verify no stray extension control files landed on the filesystem
	# CRITICAL, and intentionally redundant with the count_nulls-specific
	# check in the next step: a filesystem control file silently wins
	# over a pg_tle-registered extension of the same name, which would
	# make this whole job a false pass without ever raising an error. Run
	# again after every step below that could plausibly write extension
	# files to disk -- never trust a single check to catch everything.
	run: bin/assert_fs_clean verify ${{ matrix.pg }} /tmp/control_baseline.txt pg_tle.control
	- name: Install count_nulls purely via pg_tle (fresh install, no filesystem trace)
	# count_nulls is never `make install`ed in this job, so a successful
	# CREATE EXTENSION here can only be resolving through pg_tle's
	# registration, not a control file on disk. Checked explicitly here
	# too (not just via the comprehensive check above) as a guard
	# specifically for the extension under test, in case that check's
	# exclude-list logic has a bug.
	run: \|
	test ! -e /usr/share/postgresql/${{ matrix.pg }}/extension/count_nulls.control
	createdb count_nulls_smoke
	psql -d count_nulls_smoke -c "CREATE EXTENSION count_nulls"
	- name: Verify count_nulls works when deployed via pg_tle
	run: \|
	INSTALLED=$(psql -d count_nulls_smoke -tAc "SELECT extversion FROM pg_extension WHERE extname = 'count_nulls'")
	# EXTENSION_count_nulls_VERSION (the .control file's default_version),
	# NOT PGXNVERSION (the PGXN distribution version, from META.in.json)
	# -- a version-less CREATE EXTENSION installs whatever the control
	# file's default_version says, and count_nulls' is currently the
	# 'stable' pseudo-version, not the last real release. See
	# RELEASE.md's note on distribution vs. extension versions.
	EXPECTED=$(make -s print-EXTENSION_count_nulls_VERSION 2>/dev/null \| sed -n 's/.set to "$.$"$/\1/p')
	echo "installed=$INSTALLED expected=$EXPECTED"
	if [ -z "$INSTALLED" ] \|\| [ -z "$EXPECTED" ] \|\| [ "$INSTALLED" != "$EXPECTED" ]; then
	echo "FAIL: installed='$INSTALLED' expected='$EXPECTED'"; exit 1
	fi
	RESULT=$(psql -d count_nulls_smoke -v ON_ERROR_STOP=1 -tAc "SELECT null_count(1, NULL, 2)")
	echo "null_count(1, NULL, 2)=$RESULT"
	if [ "$RESULT" != "1" ]; then
	echo "FAIL: expected null_count(1, NULL, 2) = 1, got '$RESULT'"; exit 1
	fi
	- name: Verify no stray extension control files after the fresh-install smoke test
	run: bin/assert_fs_clean verify ${{ matrix.pg }} /tmp/control_baseline.txt pg_tle.control
	- name: Install count_nulls at 0.9.6, purely via pg_tle (update-path prep)
	# A SECOND, separate scratch database, created after the template1
	# registration above so it inherits both registrations too (same
	# reasoning as count_nulls_smoke). prepare-old creates the database,
	# CREATE EXTENSIONs at 0.9.6 (pure SQL - resolves through pg_tle's
	# catalog, no `make install` call, which would defeat the whole
	# point), then plants + proves the dependency guard so a stray
	# CASCADE drop anywhere below can't silently turn the eventual
	# existing-mode run into a fresh install instead.
	run: \|
	test ! -e /usr/share/postgresql/${{ matrix.pg }}/extension/count_nulls.control
	bin/test_existing prepare-old count_nulls_pgtle_update 0.9.6
	- name: Verify no stray extension control files after installing 0.9.6 via pg_tle
	run: bin/assert_fs_clean verify ${{ matrix.pg }} /tmp/control_baseline.txt pg_tle.control
	- name: Update 0.9.6 -> current, purely via pg_tle
	# Pure SQL (ALTER EXTENSION ... UPDATE), no filesystem write either.
	run: bin/test_existing update count_nulls_pgtle_update
	- name: Verify no stray extension control files after the pg_tle update
	run: bin/assert_fs_clean verify ${{ matrix.pg }} /tmp/control_baseline.txt pg_tle.control
	- name: Run the real pgTAP suite against the pg_tle-updated database (existing mode)
	# run-suite re-proves the dependency guard, drops it, then runs the
	# FULL suite via --use-existing against the real pg_tle-deployed +
	# updated database - the same suite/expected-output as every other
	# leg (see test/README.md). TEST_EXISTING_DEPLOY=pgtle makes
	# run-suite sandbox `make test`'s otherwise-unavoidable `install`
	# step behind a scratch DESTDIR instead of writing to the real
	# extension directory (see bin/test_existing's TEST_EXISTING_DEPLOY
	# comment), and makes test/install/load.sql's existing-mode
	# assertion cross-check pgtle.available_extensions() instead of
	# pg_available_extensions (which never sees pg_tle registrations -
	# see the Makefile's TEST_EXISTING_DEPLOY comment).
	run: TEST_EXISTING_DEPLOY=pgtle bin/test_existing run-suite count_nulls_pgtle_update
	- name: Verify no stray extension control files after the pgTAP suite
	# THE step that actually proves the DESTDIR sandboxing worked: the
	# real extension directory must still be clean after `make test`
	# ran (with its otherwise-unavoidable `install` step) sandboxed
	# behind a scratch DESTDIR.
	run: bin/assert_fs_clean verify ${{ matrix.pg }} /tmp/control_baseline.txt pg_tle.control

	# A single stable check name for use as a required status check in branch
	# protection rules. Matrix jobs produce check names like
	# "🐘 PostgreSQL 14" (one per supported major) which would all need to be
	# listed individually and updated whenever the matrix changes. This job
	# passes if all others passed or were skipped (e.g. the heavy jobs gated
	# off by the `changes` job on a docs-only push), and fails if any failed
	# or were cancelled.
	all-checks-passed:
	needs: [changes, lint, test, pg-upgrade-test, pg-tle-test]
	if: always()
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@v4
	- name: Verify all jobs are listed in needs
	# Ensures this job won't silently ignore a newly-added job that was
	# omitted from the needs list above.
	run: \|
	DEFINED=$(python3 -c "
	import yaml
	with open('.github/workflows/ci.yml') as f:
	w = yaml.safe_load(f)
	print('\n'.join(sorted(j for j in w['jobs'] if j != 'all-checks-passed')))
	")
	NEEDED=$(echo '${{ toJson(needs) }}' \| python3 -c "
	import json, sys
	print('\n'.join(sorted(json.load(sys.stdin))))
	")
	if [ "$DEFINED" != "$NEEDED" ]; then
	echo "Some jobs are missing from all-checks-passed needs:"
	diff <(echo "$DEFINED") <(echo "$NEEDED")
	exit 1
	fi
	- name: Check all jobs passed or were skipped
	run: \|
	if [[ "${{ contains(needs..result, 'failure') \|\| contains(needs..result, 'cancelled') }}" == "true" ]]; then
	echo "One or more jobs failed or were cancelled"
	exit 1
	fi
	# vi: expandtab ts=2 sw=2

Navigation Menu

Uh oh!

priority lanes for rebase-cascade pushes, escalate on stack promotion #327

Workflow file

priority lanes for rebase-cascade pushes, escalate on stack promotion #327

Uh oh!

Workflow file for this run