Skip to content

chore: improve integration test runs in the CI (build once, reuse across matrix) - #2932

Closed
mojtaba-esk wants to merge 40 commits into
mainfrom
mojtaba/use-cache-to-improve-integration-test-in-ci
Closed

mojtaba-esk wants to merge 40 commits into
mainfrom
mojtaba/use-cache-to-improve-integration-test-in-ci

Conversation

@mojtaba-esk

@mojtaba-esk mojtaba-esk commented Feb 19, 2026 •

Copy link
Copy Markdown
Contributor

Describe your changes and provide context

This PR proposes to build the localnode image once in a dedicated job (base from GHCR or build+push, app with GHA cache, then push to ttl.sh with commit-SHA tag). All 14 matrix test jobs pull that image and run the cluster/tests instead of building. Makefile adds build-docker-node-app and drops proxy/local-only logic; Dockerfiles use ubuntu:25.04.

Testing performed to validate your change (CI Performance Improvement Summary)

Comparison Scope

This compares the Baseline CI architecture vs the *Run after improvements applied (Latest run) *.

  • Baseline: Each of the 14 jobs builds its own image.
  • Latest: One shared image build, then 14 jobs run in parallel.
  • Total time is measured as wall-clock duration (critical path).

Wall Clock Time

Run Total Time
Baseline 1h 56m 27s
Latest 1h 05m 50s

Total reduction: 50m 37s (~43.5% faster)


Critical Path (Normalized Comparison)

Since baseline jobs each built their own image, we compare:

  • Baseline bottleneck:
    EVM Interoperability = 14m 17s

  • Latest bottleneck (Build + EVM Interoperability):
    2m 17s + 10m 19s = 12m 36s

Metric Baseline Latest
Critical Path 14m 17s 12m 36s

Net improvement on true per-job cost: 1m 41s (~12% faster)


Resource Usage Impact

Baseline:

  • 14 jobs each built their own image.
  • Build step duplicated 14 times.
  • Significant redundant compute usage.

Latest:

  • Image built once (2m 17s).
  • Eliminates 13 redundant builds.
  • Reduces overall runner CPU time and resource contention.

This reduces total runner-minute consumption while also improving wall-clock time.


Summary

  • CI wall-clock time reduced by ~50 minutes.
  • ~43% faster pipeline execution.
  • Shared image build removes redundant compute work.
  • Critical path is now ~12% faster than baseline.
  • Lower compute waste and more scalable architecture.

While the per-job runtime gain is modest (~1m 41s), the overall CI latency and resource efficiency improvements are substantial.

Note: Timings were collected from the GitHub Actions usage page for this PR only and may vary due to runner load, caching state, and other environmental factors.

@github-actions

github-actions Bot commented Feb 19, 2026 •

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedFeb 23, 2026, 3:42 PM

@github-actions

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedFeb 19, 2026, 10:20 AM

@codecov

codecov Bot commented Feb 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.59%. Comparing base (f23d05a) to head (3a140e1).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
giga/executor/lib/evmlib.go 0.00% 8 Missing ⚠️

❌ Your project status has failed because the head coverage (0.00%) is below the target coverage (40.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##             main    #2932       +/-   ##
===========================================
+ Coverage   57.83%   66.59%    +8.76%     
===========================================
  Files        2110        6     -2104     
  Lines      174010      470   -173540     
===========================================
- Hits       100633      313   -100320     
+ Misses      64425      127    -64298     
+ Partials     8952       30     -8922     
Flag Coverage Δ
sei-chain 0.00% <0.00%> (-57.81%) ⬇️
sei-db 68.49% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
giga/executor/lib/evmlib.go 0.00% <0.00%> (-55.56%) ⬇️

... and 2104 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mojtaba-esk
mojtaba-esk requested a review from masih February 19, 2026 13:52
@mojtaba-esk mojtaba-esk changed the title chore: add cache to imrpve integration test runs in the CI (build once, reuse across matrix) chore: improve integration test runs in the CI (build once, reuse across matrix) Feb 20, 2026
@mojtaba-esk
mojtaba-esk marked this pull request as ready for review February 20, 2026 16:36
Comment thread docker/localnode/Dockerfile.base Outdated
Comment on lines +8 to +9
RUN curl -L https://go.dev/dl/go1.25.6.linux-amd64.tar.gz | tar xvzf - -C /usr/local/
RUN curl -L https://foundry.paradigm.xyz | bash

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: may be worthwhile to pin this to a specific release. As is, this will take whatever that request returns, which could vary from run to run. Can also validate the checksum for integrity. Reproducibility in practice being the more concerning point to me.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good point. I applied the changes and will be pushing it ASAP. Pinned the installer to the v1.5.1 tag and run foundryup -i v1.5.1; skipping checksum verification for now since we're pulling it from the official github tag over https.

Comment thread Makefile Outdated
DOCKER_BUILD_CACHE_TO ?=
LOCALNODE_BASE_TAG := sei-chain/localnode-base:go1.25.6

# Base image (Go, Foundry, Node). Rebuild only when tool versions or Dockerfile.base change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a matter of preference: this seems a little misleading on first read. I initially interpreted this as the target enforcing this diff check.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, made it clear now.

Comment thread Makefile Outdated
fi
.PHONY: build-docker-node-base

# Build only the app image (requires sei-chain/localnode-base:go1.25.6 to exist locally; used in CI when base is pulled or pre-built)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of commenting this dependency, consider codifying it and making the build of the localnode-base a prerequisite or adding a log to make the outcome clear.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch! added.

Comment thread Makefile
Comment on lines +259 to +261
@if [ ! -f build/seid ] || [ ! -f build/price-feeder ]; then \
echo "ERROR: build/seid and build/price-feeder must exist. Run the build step first."; exit 1; \
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Comment thread Makefile Outdated
.PHONY: build-docker-node-app

# Build app image with prebuilt seid binary (context = repo root; requires build/seid and build/price-feeder to exist).
# Used in CI: build seid in a container first, then run this target so the image contains the binary.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we're building seid in the runner I believe.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yeah the comment needs to be updated or removed. good catch!

Comment thread Makefile Outdated
Comment on lines +276 to +277
# Build docker image for detected platform (depends on base; app layer is quick)
build-docker-node: build-docker-node-base

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is better since you're explicitly orchestrating dependencies rather than them being in a comment most people (or agents) won't immediately see.

Also, not immediately clear why you'd use a sub-make target for build-docker-node-app. If you do: build-docker-node: build-docker-node-base build-docker-node-app then Makefile analysis tools will be able to better understand make target relationships.

Let me know if I've missed something.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point on the sub-make. I've switched to listing both as prerequisites:
build-docker-node: build-docker-node-base build-docker-node-app
so the dependency graph is explicit and tools (and readers) can see it without running the recipe. No need for a submake here since the target only needs those two up to date.

@masih masih left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although wall-clock time is reduced, compared to the current head of main, this work shaves off about 2m of wait time. I really wish it was saving more time, but I don't think we need to block on it.

I left a set of comments, none are blockers. But I recommend taking a look, specially the build push image per commit. This is a big footprint given the e2e gain is -2m of wait time, and it could fill up the GHCR pretty quickly if too many PRs are being iterated on at the same time.

Happy to discuss further at standup. Thank you for coping with the integration tests! They are in much need of TLC 🙌

Comment thread .github/workflows/integration-test.yml Outdated
shell: bash

env:
GHCR_BASE_IMAGE: ghcr.io/${{ github.repository_owner }}/sei-chain-localnode-base:go1.25.6

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which container registry used to host the image is probably a minor detail, right? In that spirit, I would rename this env to be more representative of what it does. Maybe IT_BASE_IMAGE or INTEGRATION_TEST_BASE_IMAGE?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delicate point! ofc, renamed it to INTEGRATION_TEST_BASE_IMAGE so the name reflects what it's for rather than the registry.

- name: Push app image to GHCR
run: |
docker tag sei-chain/localnode ${{ steps.image_name.outputs.image_name }}
docker push ${{ steps.image_name.outputs.image_name }}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This builds and pushes the image per commit. We would be hitting the max size limit on GHCR pretty quickly? But then these images are very short-lived. Right?

Non blocker but I wonder if we should use ECR with a propepr lifecycle policy instead. Or build the image as a tar ball, upload it to GHA as an artifact, then download it in other jobs. This would save the push and pull dance.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's one image per commit. I originally used ttl.sh (images expire after a time window, e.g. 24h), but push time was ~50s there vs ~12–15s on GHCR, so I switched to GHCR for speed.
ECR with a lifecycle policy makes sense. Another option I'm considering is deleting the image from GHCR after the workflow run finishes (on success or failure), so we don't accumulate tags. WDYT?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A cleanup job is added.

- name: Pull localnode image
run: |
docker pull ${{ needs.build-localnode.outputs.image_name }}
docker tag ${{ needs.build-localnode.outputs.image_name }} sei-chain/localnode

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why tag again? we know what we just pulled right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is b/c downstream steps expect the image as sei-chain/localnode, not the one with GHCR name.

Comment thread .github/workflows/integration-test.yml Outdated
docker tag ${{ needs.build-localnode.outputs.image_name }} sei-chain/localnode
- name: Start 4 node docker cluster
run: make clean && DOCKER_DETACH=true INVARIANT_CHECK_INTERVAL=10 ${{matrix.test.env}} make docker-cluster-start
run: DOCKER_DETACH=true INVARIANT_CHECK_INTERVAL=10 ${{matrix.test.env}} make docker-cluster-up

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move env vars to step env ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're right. nice one. organized it.

Comment thread docker/localnode/Dockerfile Outdated
# Application image: base + scripts. Rebuild when scripts or base tag change.

ARG GHCR_ORG=sei-protocol
FROM ghcr.io/${GHCR_ORG}/sei-chain-localnode-base:go1.25.6

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be misleading, in that the local docker cluster start now always pulls from remote, regardless of make target executed. Right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch. if user tries to build it the image tag might be different and if this image does not exist in the registry, it cannot pull it. as we already tag it in the ci once we pull it, we can simply rename the tag to a local name. so in that case if user runs local cluster start, base and app are built on the machine; nothing is pulled from GHCR.

Comment thread docker/localnode/Dockerfile Outdated
SHELL ["/bin/bash", "-c"]
# Application image: base + scripts. Rebuild when scripts or base tag change.

ARG GHCR_ORG=sei-protocol

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I doubt we would ever use any other org. Better to set an ARG for FROM image?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure. keep it simple :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants