Repository navigation
chore: improve integration test runs in the CI (build once, reuse across matrix) - #2932
mojtaba-esk wants to merge 40 commits into
Conversation
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is
❌ Your project status has failed because the head coverage (0.00%) is below the target coverage (40.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #2932 +/- ##
===========================================
+ Coverage 57.83% 66.59% +8.76%
===========================================
Files 2110 6 -2104
Lines 174010 470 -173540
===========================================
- Hits 100633 313 -100320
+ Misses 64425 127 -64298
+ Partials 8952 30 -8922
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
| RUN curl -L https://go.dev/dl/go1.25.6.linux-amd64.tar.gz | tar xvzf - -C /usr/local/ | ||
| RUN curl -L https://foundry.paradigm.xyz | bash |
There was a problem hiding this comment.
nit: may be worthwhile to pin this to a specific release. As is, this will take whatever that request returns, which could vary from run to run. Can also validate the checksum for integrity. Reproducibility in practice being the more concerning point to me.
There was a problem hiding this comment.
Very good point. I applied the changes and will be pushing it ASAP. Pinned the installer to the v1.5.1 tag and run foundryup -i v1.5.1; skipping checksum verification for now since we're pulling it from the official github tag over https.
| DOCKER_BUILD_CACHE_TO ?= | ||
| LOCALNODE_BASE_TAG := sei-chain/localnode-base:go1.25.6 | ||
|
|
||
| # Base image (Go, Foundry, Node). Rebuild only when tool versions or Dockerfile.base change. |
There was a problem hiding this comment.
Just a matter of preference: this seems a little misleading on first read. I initially interpreted this as the target enforcing this diff check.
There was a problem hiding this comment.
I see, made it clear now.
| fi | ||
| .PHONY: build-docker-node-base | ||
|
|
||
| # Build only the app image (requires sei-chain/localnode-base:go1.25.6 to exist locally; used in CI when base is pulled or pre-built) |
There was a problem hiding this comment.
Instead of commenting this dependency, consider codifying it and making the build of the localnode-base a prerequisite or adding a log to make the outcome clear.
There was a problem hiding this comment.
Nice catch! added.
| @if [ ! -f build/seid ] || [ ! -f build/price-feeder ]; then \ | ||
| echo "ERROR: build/seid and build/price-feeder must exist. Run the build step first."; exit 1; \ | ||
| fi |
| .PHONY: build-docker-node-app | ||
|
|
||
| # Build app image with prebuilt seid binary (context = repo root; requires build/seid and build/price-feeder to exist). | ||
| # Used in CI: build seid in a container first, then run this target so the image contains the binary. |
There was a problem hiding this comment.
nit: we're building seid in the runner I believe.
There was a problem hiding this comment.
oh yeah the comment needs to be updated or removed. good catch!
| # Build docker image for detected platform (depends on base; app layer is quick) | ||
| build-docker-node: build-docker-node-base |
There was a problem hiding this comment.
This is better since you're explicitly orchestrating dependencies rather than them being in a comment most people (or agents) won't immediately see.
Also, not immediately clear why you'd use a sub-make target for build-docker-node-app. If you do: build-docker-node: build-docker-node-base build-docker-node-app then Makefile analysis tools will be able to better understand make target relationships.
Let me know if I've missed something.
There was a problem hiding this comment.
Good point on the sub-make. I've switched to listing both as prerequisites:
build-docker-node: build-docker-node-base build-docker-node-app
so the dependency graph is explicit and tools (and readers) can see it without running the recipe. No need for a submake here since the target only needs those two up to date.
masih
left a comment
There was a problem hiding this comment.
Although wall-clock time is reduced, compared to the current head of main, this work shaves off about 2m of wait time. I really wish it was saving more time, but I don't think we need to block on it.
I left a set of comments, none are blockers. But I recommend taking a look, specially the build push image per commit. This is a big footprint given the e2e gain is -2m of wait time, and it could fill up the GHCR pretty quickly if too many PRs are being iterated on at the same time.
Happy to discuss further at standup. Thank you for coping with the integration tests! They are in much need of TLC 🙌
| shell: bash | ||
|
|
||
| env: | ||
| GHCR_BASE_IMAGE: ghcr.io/${{ github.repository_owner }}/sei-chain-localnode-base:go1.25.6 |
There was a problem hiding this comment.
Which container registry used to host the image is probably a minor detail, right? In that spirit, I would rename this env to be more representative of what it does. Maybe IT_BASE_IMAGE or INTEGRATION_TEST_BASE_IMAGE?
There was a problem hiding this comment.
delicate point! ofc, renamed it to INTEGRATION_TEST_BASE_IMAGE so the name reflects what it's for rather than the registry.
| - name: Push app image to GHCR | ||
| run: | | ||
| docker tag sei-chain/localnode ${{ steps.image_name.outputs.image_name }} | ||
| docker push ${{ steps.image_name.outputs.image_name }} |
There was a problem hiding this comment.
This builds and pushes the image per commit. We would be hitting the max size limit on GHCR pretty quickly? But then these images are very short-lived. Right?
Non blocker but I wonder if we should use ECR with a propepr lifecycle policy instead. Or build the image as a tar ball, upload it to GHA as an artifact, then download it in other jobs. This would save the push and pull dance.
There was a problem hiding this comment.
Yes, it's one image per commit. I originally used ttl.sh (images expire after a time window, e.g. 24h), but push time was ~50s there vs ~12–15s on GHCR, so I switched to GHCR for speed.
ECR with a lifecycle policy makes sense. Another option I'm considering is deleting the image from GHCR after the workflow run finishes (on success or failure), so we don't accumulate tags. WDYT?
There was a problem hiding this comment.
A cleanup job is added.
| - name: Pull localnode image | ||
| run: | | ||
| docker pull ${{ needs.build-localnode.outputs.image_name }} | ||
| docker tag ${{ needs.build-localnode.outputs.image_name }} sei-chain/localnode |
There was a problem hiding this comment.
Why tag again? we know what we just pulled right?
There was a problem hiding this comment.
it is b/c downstream steps expect the image as sei-chain/localnode, not the one with GHCR name.
| docker tag ${{ needs.build-localnode.outputs.image_name }} sei-chain/localnode | ||
| - name: Start 4 node docker cluster | ||
| run: make clean && DOCKER_DETACH=true INVARIANT_CHECK_INTERVAL=10 ${{matrix.test.env}} make docker-cluster-start | ||
| run: DOCKER_DETACH=true INVARIANT_CHECK_INTERVAL=10 ${{matrix.test.env}} make docker-cluster-up |
There was a problem hiding this comment.
Move env vars to step env ?
There was a problem hiding this comment.
you're right. nice one. organized it.
| # Application image: base + scripts. Rebuild when scripts or base tag change. | ||
|
|
||
| ARG GHCR_ORG=sei-protocol | ||
| FROM ghcr.io/${GHCR_ORG}/sei-chain-localnode-base:go1.25.6 |
There was a problem hiding this comment.
This might be misleading, in that the local docker cluster start now always pulls from remote, regardless of make target executed. Right?
There was a problem hiding this comment.
Nice catch. if user tries to build it the image tag might be different and if this image does not exist in the registry, it cannot pull it. as we already tag it in the ci once we pull it, we can simply rename the tag to a local name. so in that case if user runs local cluster start, base and app are built on the machine; nothing is pulled from GHCR.
| SHELL ["/bin/bash", "-c"] | ||
| # Application image: base + scripts. Rebuild when scripts or base tag change. | ||
|
|
||
| ARG GHCR_ORG=sei-protocol |
There was a problem hiding this comment.
I doubt we would ever use any other org. Better to set an ARG for FROM image?
There was a problem hiding this comment.
sure. keep it simple :)
Describe your changes and provide context
This PR proposes to build the localnode image once in a dedicated job (base from GHCR or build+push, app with GHA cache, then push to ttl.sh with commit-SHA tag). All 14 matrix test jobs pull that image and run the cluster/tests instead of building. Makefile adds build-docker-node-app and drops proxy/local-only logic; Dockerfiles use ubuntu:25.04.
Testing performed to validate your change (CI Performance Improvement Summary)
Comparison Scope
This compares the Baseline CI architecture vs the *Run after improvements applied (Latest run) *.
Wall Clock Time
Total reduction: 50m 37s (~43.5% faster)
Critical Path (Normalized Comparison)
Since baseline jobs each built their own image, we compare:
Baseline bottleneck:
EVM Interoperability = 14m 17sLatest bottleneck (Build + EVM Interoperability):
2m 17s + 10m 19s = 12m 36sNet improvement on true per-job cost: 1m 41s (~12% faster)
Resource Usage Impact
Baseline:
Latest:
This reduces total runner-minute consumption while also improving wall-clock time.
Summary
While the per-job runtime gain is modest (~1m 41s), the overall CI latency and resource efficiency improvements are substantial.
Note: Timings were collected from the GitHub Actions usage page for this PR only and may vary due to runner load, caching state, and other environmental factors.