Skip to content

ci: reduce the PR run time - #1063

Open
mbrandenburger wants to merge 10 commits into
hyperledger:mainfrom
mbrandenburger:ci/phase3-polish
Open

mbrandenburger wants to merge 10 commits into
hyperledger:mainfrom
mbrandenburger:ci/phase3-polish

Conversation

@mbrandenburger

@mbrandenburger mbrandenburger commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:
The median FPC test run over the last week was 12.8 min (32 successful runs). Its critical path is the build job (about 4 min), then the slowest container suite (go_echo or irb, about 7.5 min). In those suites the time went to Go compiles the cache should have saved, to downloading and loading a 1.3 GB artifact (920 MB of it images), to pulling Fabric images the suite doesn't use, and to fixed waits. Ten commits, best reviewed one by one:

  1. ci: keep a Go cache per test suite. The test jobs restored the build job's Go cache, which holds only what make build compiles; the suites compile other packages with other flags (-race, fpc_cgo, coverage) and started cold, e.g. 103s building fsccli with -race in go_echo. Each test job now has its own cache, keyed go-<suite>-<hash of all go.sum>. Only push runs save it (upstream that's main) and PR runs restore main's, so PR branches don't fill the repo's 10 GB cache quota, which it is already near.
  2. ci: run only what needs the SGX toolchain in the dev container. go_simulation and test_network run their chaincode without SGX. go_simulation already ran natively too; it now runs only there, and test_network moves to the native matrix (ubuntu-latest has the yq 4 and jq its scripts use). The container unit job runs only cgo-unit-test (the C/C++ code, plus internal and ercc with fpc_cgo) and the C++ linter; the Go unit tests, Go linter and license check already ran in native (unit). make test keeps its meaning: unit tests only natively, everything with FPC_CGO=1.
  3. test(fsc): set the FPC log spec in EnableFPC. Four FSC topologies set the same Fabric log spec (gRPC at error), echo's none. EnableFPC now sets it as fpctopo.LogSpec, unless a topology set its own (the simple testing network sample does). irb's go test also exports it in FABRIC_LOGGING_SPEC, like the other FSC suites, as FSC doesn't pass the topology spec to the discover CLI calls it makes.
  4. test: pull only the images a suite uses. The FSC suites and irb pulled fabric-ccenv/fabric-baseos:2.5. FSC needs them only for chaincode the peers build themselves, which FPC topologies don't have, as FPC chaincode runs as its own container; the pulls are gone. The simple testing network sample (not in CI) moves to ccenv:3.1. Of the container jobs, only deployment still pulls the Fabric 3.1.4 images: its peer builds a plain Go chaincode in fabric-ccenv, the other suites run the Fabric binaries from the build outputs.
  5. build(docker): shallow-clone Fabric in the dev image. The dev image cloned Fabric's whole history; make -C fabric patch needs only the release tag. The clone takes about a third of the time (5.7s against 16.9s) and 155 MB less; the dev image builds in about 9s instead of 17s. The apt step is skipped when APT_ADD_PKGS is empty.
  6. ci: pull the base images in the test jobs instead of shipping them. Unless utils/docker/** changed, the build job pulls ccenv and base-dev from ghcr.io; the test jobs now do the same and build the dev image on top, instead of downloading and docker loading 920 MB. When utils/docker/** changed, the images are rebuilt and shipped as before, as their own artifact, which publish also uses.
  7. test(sgx): cut the C++ suites' orderer batch timeout to 1s. Each invocation the C++ suites wait for waited up to the 2s batch timeout of sgx/integration/config/configtx.yaml; 1s is what the FSC test networks use.
  8. test(fabric): drop the fixed sleeps in peer.sh's ercc install. Six sleep 3, 18s per ledger setup in every C++ suite. None waits for anything the CLI or the peer doesn't already: install and the queries are synchronous, approveformyorg and commit wait for their commit (--waitForEvent defaults to true in Fabric 3.x), and the peer holds the first call to ercc until it has launched. The sleep 3 after channel join in ledger_init stays.
  9. test(fabric): log gRPC at error in the C++ suites. The C++ suites never set FABRIC_LOGGING_SPEC, so every Fabric process they start (peer, orderer, each peer CLI call, the client SDK test) logged each gRPC connection step at info: about 23,000 lines per run, 16,000 of them in client_sdk. fabric/bin/lib/common_utils.sh, which every suite and peer.sh source, now defaults it to gRPC at error; a caller's own spec is kept. Not covered: about 190 lines in test_network from Fabric's own test-network scripts, and 49 from a gRPC server in one client SDK unit test.
  10. test(fsc): wait for conditions instead of fixed sleeps when FPC chaincode deploys. The FPC NWO extension (extension/fsc/integration/nwo/fabric/fpc/fpc.go) slept 10s after committing each FPC chaincode, 3s after starting each chaincode container, and 3s at the end of PostRun: about 25 to 40s per FSC test network, in all 9 FSC jobs. As in FSC's PostRun (test(nwo): wait for conditions instead of fixed sleeps when a Fabric network starts hyperledger-labs/fabric-smart-client#2321, already in the FSC v0.27.0 FPC uses), it now waits for the conditions instead. CommitChaincode already returns once every peer has the definition committed, so the 10s sleep goes. After starting a container, it polls a TCP dial to the chaincode's port, bounded by EventuallyTimeout. The final sleep goes, as initEnclave has already invoked the enclave by then.

Which issue(s) this PR fixes:
Refs #1043. Part of #1044.

Special notes for your reviewer:

  • Commit 5 changes utils/docker/**, so this PR's own runs rebuild the base images and take the "ship the images" path of commit 6. The pull path and main's Go caches only take effect after the merge, once publish has pushed the new images and a push to main has saved the caches. I'll measure both on my fork (a run that fills the caches, then one that uses them) and add the numbers here.
  • Commit 1, measured on my fork (push run 42124772037, attempt 1 without the cache, attempt 2 with it): go_echo 8.0 to 4.2 min, irb 6.6 to 4.5, unit 5.3 to 2.8, go_kv 5.4 to 2.9. The C++-only suites don't change. The 13 caches take about 3.2 GB in total (go_echo 608 MB, irb 533 MB, unit 459 MB), against 187 MB for the build job's cache before; when the quota is full, GitHub evicts the oldest first, mostly stale setup-go caches of PR branches.
  • Commits 4, 7 and 8 change what the suites rely on, so only the suites themselves can check them; all suites passed on the first CI run of this PR with them. Their test steps got shorter: echo 67s to 38s, client_sdk 173s to 130s. If a suite ever misses an image (4), the pull comes back with 3.1; if the ercc install races (8), the sleeps become polling on querycommitted rather than coming back. The same goes for commit 10: if an FSC suite fails right after a chaincode commit, QueryInstalledReferences gets polled rather than the 10s sleep coming back.
  • Known gap, marked in the workflow: the test jobs in commit 6 pull :main, not the build job's digest, so a publish on main between the two pulls would hand a test job the newer base-dev.
  • client_sdk stays in the dev container: it deploys the C++ samples (fpc-c), so it needs the C++ enclaves and the SGX SDK runtime.

The test jobs restored the build job's Go cache, which only holds what
make build compiles. The suites compile other packages with other flags
(-race, fpc_cgo, coverage), so each run downloaded the integration
modules and rebuilt them, e.g. fsccli with -race for 103s in go_echo.

Each test job now has its own cache, keyed on the suite and go.sum, so
it caches exactly what it compiles. Only push runs save it (upstream
that is main); PR runs restore main's. The repo's caches are already at
the 10 GB quota, so PR branches saving 13 copies each would evict them.

Refs hyperledger#1043

Signed-off-by: Marcus Brandenburger <bur@zurich.ibm.com>
@mbrandenburger mbrandenburger self-assigned this Nov 6, 2026
@mbrandenburger mbrandenburger changed the title ci: keep a Go cache per test suite ci: cut the PR run time (Go cache per suite, fewer container jobs, smaller artifact, fewer waits) Nov 6, 2026
@mbrandenburger mbrandenburger changed the title ci: cut the PR run time (Go cache per suite, fewer container jobs, smaller artifact, fewer waits) ci: reduce the PR run time Nov 6, 2026
go_simulation and test_network run their chaincode without SGX, so
they need only Go and Docker. go_simulation already ran natively as well;
it now runs only there, and test_network moves to the native matrix.

The container unit job runs only cgo-unit-test (the C/C++ code, and
internal and ercc with fpc_cgo) and the C++ linter. The Go unit tests,
Go linter and license check already ran in native (unit).

make suites lists the suites CI runs on each side; make test still runs
unit tests only natively, and everything with FPC_CGO=1.

Refs hyperledger#1043

Signed-off-by: Marcus Brandenburger <bur@zurich.ibm.com>
Four FSC topologies set the same Fabric log spec (gRPC at error), echo's
none. EnableFPC, which every FPC topology goes through, now sets it as
fpctopo.LogSpec, unless the topology set its own spec (as the simple
testing network sample does). The topologies drop their copies.

Signed-off-by: Marcus Brandenburger <bur@zurich.ibm.com>
The FSC suites and irb pulled fabric-ccenv and fabric-baseos 2.5 and
tagged them latest. FSC needs them only for chaincode the peers build
themselves, which these topologies don't have: FPC chaincode runs as its
own container. irb keeps pulling redis, which its experimenter test uses.
The simple testing network sample, not run in CI, moves to ccenv 3.1.

Each container job pulled the Fabric 3.1.4 images; only deployment uses
them (its peer builds a plain Go chaincode in fabric-ccenv), the others
run the Fabric binaries from the build outputs.

irb's go test now gets the FPC log spec in FABRIC_LOGGING_SPEC, like the
other FSC suites, so the peer CLI calls FSC makes log gRPC at error.

Signed-off-by: Marcus Brandenburger <bur@zurich.ibm.com>
The dev image cloned Fabric's whole history; make -C fabric patch only
needs the release tag and its tree. A shallow clone takes about a third
of the time and 155 MB less. The apt step is skipped when APT_ADD_PKGS
is empty, as in CI.

Signed-off-by: Marcus Brandenburger <bur@zurich.ibm.com>
Every test job downloaded and loaded the 920 MB of images the build job
saved (the dev image, ccenv and base-dev). Unless utils/docker changed,
the build job pulls ccenv and base-dev from ghcr.io anyway; the test jobs
now do the same and build the thin dev image on top (about 50s, against
the image download plus about 40s of docker load).

When utils/docker changed, the build job rebuilds the images and ships
them as before, now as their own artifact, which publish also uses.

Signed-off-by: Marcus Brandenburger <bur@zurich.ibm.com>
Every invocation the C++ suites wait for (--waitForEvent, and the client
SDK's submits) waits for the orderer to cut a block: up to the 2s batch
timeout of the test network config. 1s is what the FSC test networks use;
a block still holds every transaction that arrives within it.

Signed-off-by: Marcus Brandenburger <bur@zurich.ibm.com>
peer.sh installed ercc with a sleep 3 after each lifecycle step, 18s per
ledger setup, which every C++ suite runs once. None waits for anything
the CLI or the peer doesn't already: install and the queries are
synchronous, approveformyorg and commit wait for their commit
(--waitForEvent defaults to true in Fabric 3.x), and the peer holds the
first call to ercc until it has launched. querycommitted at the end still
checks that ercc is live.

The sleep after channel join in ledger_init stays: there the peer has to
catch up on the new channel first.

Signed-off-by: Marcus Brandenburger <bur@zurich.ibm.com>
The C++ suites never set FABRIC_LOGGING_SPEC, so every Fabric process
they start (peer, orderer, each peer CLI call, the client SDK test) logged
each gRPC connection step at info: about 23,000 lines per run, 16,000 of
them in client_sdk. common_utils.sh, which every suite and peer.sh source,
now defaults it to gRPC at error; a caller's own spec is kept.

Signed-off-by: Marcus Brandenburger <bur@zurich.ibm.com>
…code deploys

The FPC NWO extension slept 10s after committing each FPC chaincode, 3s
after starting each chaincode container, and 3s at the end of PostRun:
about 25 to 40s per FSC test network, in every FSC suite. As in FSC's
PostRun (hyperledger-labs/fabric-smart-client#2321), it now waits for
what the sleeps stood in for:

- CommitChaincode already returns only once every peer has the
  definition committed, so the 10s sleep after it is gone.
- After starting a chaincode container, it polls until the chaincode
  server accepts connections on its port, bounded by EventuallyTimeout.
  The peer dials it only on the first invocation.
- The final sleep is gone: initEnclave has already invoked the enclave
  by then.

Signed-off-by: Marcus Brandenburger <bur@zurich.ibm.com>
@mbrandenburger
mbrandenburger marked this pull request as ready for review November 6, 2026 02:17
@mbrandenburger
mbrandenburger requested a review from a team as a code owner November 6, 2026 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant