Add docker layer caching (#3318) by gqvz · Pull Request #3358 · intelowlproject/IntelOwl

gqvz · 2026-02-24T06:44:37Z

Improvement #3318 (this issue may have multiple PRs)

Description

Add docker layer caches to ghcr. Docker image is built and its layers are stored on push to develop. These layer caches are used when running tests from PR CI

Type of change

Bug fix (non-breaking change which fixes an issue) (is this technically a bugfix?).

Checklist

I have read and understood the rules about how to Contribute to this project
The pull request is for the branch develop
No new plugin was added.
I have inserted the copyright banner at the start of the file: # This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl # See the file 'LICENSE' for copying permission.
No new libraries were added.
Linters (Ruff) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
No test needed.
No GUI edits
After you had submitted the PR, if DeepSource, Django Doctors or other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.

gqvz · 2026-02-24T06:45:35Z

I have inserted the copyright banner at the start of the file: # This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl # See the file 'LICENSE' for copying permission.

None of the workflows have the copyright banner, should i add it anyway?

gqvz · 2026-02-24T06:51:25Z

saves ~3m when only backend changes and ~1m when both backend and frontend changes

Copilot

Pull request overview

This PR implements Docker layer caching for GitHub Actions to speed up CI builds. The caching strategy builds and stores Docker image layers when pushing to the develop branch, which are then reused during PR CI runs. This optimization addresses issue #3318 by significantly reducing build times when dependencies haven't changed.

Changes:

Added new workflow .github/workflows/docker-build-cache.yml that builds Docker images on push to develop and stores layers in GitHub Actions cache
Modified .github/workflows/pull_request_automation.yml to use cached layers during PR builds and removed the --build flag from the startup script since images are now pre-built
Optimized docker/Dockerfile layer ordering by copying package files first, running installations, then copying source code to maximize cache effectiveness

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
`.github/workflows/docker-build-cache.yml`	New workflow that builds and caches Docker images on develop branch pushes
`.github/workflows/pull_request_automation.yml`	Updated to leverage cached layers and use pre-built images instead of building during startup
`docker/Dockerfile`	Restructured to optimize layer caching by reordering COPY and RUN commands

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mlodic · 2026-02-24T14:10:06Z

please address copilot review. no need about the copyright banner there

mlodic · 2026-02-24T14:10:33Z

@@ -0,0 +1,55 @@
+permissions: read-all


this file needs to be tracked in the dependabot configuration

IntelOwl/.github/dependabot.yml

Lines 194 to 202 in 0b4e564

- package-ecosystem: "github-actions"

directory: "/"

schedule:

interval: "weekly"

day: "tuesday"

target-branch: "develop"

ignore:

- dependency-name: "*"

update-types: ["version-update:semver-patch"]

already tracked?

wait nvm, does this ignore all action updates? why so?

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Removed cache-to configuration for Docker builds.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (2)

.github/workflows/docker-build-cache.yml:33

This workflow builds images only to populate the GHA layer cache, but load: true also imports the image into the runner’s local Docker daemon, increasing runtime and disk usage without being needed here. Consider removing load: true (keep cache-to/cache-from) unless subsequent steps in this workflow actually run containers from these images.

      - name: Build
        uses: docker/build-push-action@v6
        with:
          context: .
          file: docker/Dockerfile
          push: false
          load: true
          tags: intelowlproject/intelowl:ci
          build-args: |

.github/workflows/docker-build-cache.yml:55

Same as the main-image job: load: true is unnecessary if the workflow’s only goal is to export BuildKit cache layers. Dropping load will typically reduce time and disk consumption on the runner.

      - name: Build
        uses: docker/build-push-action@v6
        with:
          context: .
          file: docker/Dockerfile_nginx
          push: false
          load: true
          tags: intelowlproject/intelowl_nginx:ci

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

gqvz · 2026-02-25T04:39:55Z

I would like to test these changes on my fork again
Also is there any specific reason --no-cache-dir and --compile is used here?

IntelOwl/docker/Dockerfile

Lines 54 to 56 in 8419ace

    
           RUN pip3 install --no-cache-dir --use-pep517 --compile -r project-requirements.txt \ 
        
               && pip3 install --no-cache-dir pycti==${PYCTI_VERSION} \ 
        
               && pip3 install --no-cache-dir --compile -r certego-requirements.txt

I would like to add cache mounts https://docs.docker.com/build/ci/github-actions/cache/#cache-mounts

mlodic

the problem about integrating this change is that it is difficult to evaluate whether this works properly or not because that github action would work only when this change is merged into master.

So, to merge this PR, I need that you first merge this into your own fork and share the github action results that prove that this works as intended

mlodic · 2026-02-25T14:36:30Z


 COPY requirements/project-requirements.txt $PYTHONPATH/project-requirements.txt
 COPY requirements/certego-requirements.txt $PYTHONPATH/certego-requirements.txt
+COPY requirements/django-server-requirements.txt $PYTHONPATH/requirements/django-server-requirements.txt


no, this must be installed only on local environments otherwise it would install watchman also in production environments

this change only copies the file, pip install is run on this from watchman install script, which is only run when watchman isnt false, which is only set for tests. even without this change, this file is copied, just with the rest of the source code

gqvz · 2026-02-25T14:48:57Z

the problem about integrating this change is that it is difficult to evaluate whether this works properly or not because that github action would work only when this change is merged into master.

So, to merge this PR, I need that you first merge this into your own fork and share the github action results that prove that this works as intended

I tested before making this PR, will test again after making review changes
everything is cached -> https://github.com/gqvz/IntelOwl/actions/runs/22339203724/job/64638576292?pr=3
npm build wasnt cached in this one -> https://github.com/gqvz/IntelOwl/actions/runs/22338834316/job/64637464430?pr=3
no caching (develop branch currently) -> https://github.com/intelowlproject/IntelOwl/actions/runs/22397462530/job/64834761356

when nothing is cached it takes approximately the same time as before -> https://github.com/intelowlproject/IntelOwl/actions/runs/22382367884/job/64785856226?pr=3358
^ i would like to improve this time further using cache mounts
should i go ahead with this? #3358 (comment)

mlodic · 2026-02-25T18:47:11Z

--compile serves to reduce time of compilation when starting up the image
--no-cache-dir this prevents creation of useless cache which increase the size of the image

thanks for sharing all the info, it's appreciated

please go on experimenting

gqvz · 2026-03-07T20:27:16Z

Cache mounts didnt offer much improvement so i removed them
using buildx bake with registry cache now
https://github.com/gqvz/IntelOwl/actions/runs/22805585413/job/66154081670?pr=7 saves ~3 minutes

gqvz · 2026-03-07T20:30:33Z

i will test more scenarios (partially cached) and do some last changes before this is ready for review
sorry for taking so much time on this PR, had exams

mlodic · 2026-03-08T06:59:36Z

@gqvz that's fine as long as you are keep constant communication, thank you.

I am curious about the additional tests

gqvz · 2026-03-08T10:52:53Z

only backend changes - https://github.com/gqvz/IntelOwl/actions/runs/22805585413/job/66154081670?pr=7
frontend and backend changes - https://github.com/gqvz/IntelOwl/actions/runs/22819237911/job/66189327344
only frontend changes - basically same as above, backend changes is just copying source, frontend changes will trigger build which runs in parallel and takes more time than copying backend source
changes to pip requirements - https://github.com/gqvz/IntelOwl/actions/runs/22819320844/job/66189568622?pr=7
changes to pip and npm packages - basically same as above, npm install runs parallel to pip install and takes less time
changes to only npm packages - https://github.com/gqvz/IntelOwl/actions/runs/22819402348/job/66189722492?pr=7

worst case is when any changes are made to pip requirements, time taken for that case is around 20-30s more than what it normally takes (without any caching anywhere)

one question - for backend tests, why are we building the frontend and copying the files?, can easily skip all the frontend stuff

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

mlodic · 2026-03-08T19:29:40Z

one question - for backend tests, why are we building the frontend and copying the files?, can easily skip all the frontend stuff

to make the final image lighter

gqvz · 2026-03-08T19:36:30Z

to make the final image lighter

no like i meant, for backend tests ci we can completely skip everything frontend related, we are building and copying but it isnt being used for ci tests

mlodic

I am asking some questions to understand the choices made, please answer accordingly. thank you

mlodic · 2026-03-08T19:33:20Z

+      - name: Set image repo
+        run: echo "IMAGE_REPO=ghcr.io/${GITHUB_REPOSITORY,,}" >> "$GITHUB_ENV"
+
+      - name: Set up Docker Buildx


the order of this and login to ghcr is different between the 2 yml files, is there a reason?

sorry that wasnt intentional, i'll fix this rn

mlodic · 2026-03-08T19:34:47Z

+  default = "false"
+}
+
+variable "CACHE_REPO" {


can you explain how this variable is used and why it is empty?

this is so i dont have to hardcode the repo intelowlproject/intelowl into the code, because that wouldnt work for forks, it is set from env vars

mlodic · 2026-03-08T19:41:39Z

+target "redis" {
+  dockerfile-inline = "FROM redis:6.2.7-alpine"
+  tags              = ["redis:6.2.7-alpine"]
+}


I am not sure about these declarations. I mean, this duplicates the code that already exists in the docker compose files. This could pose possible problems in terms of maintainance. Is this file and all of these configuration necessary?

broadly speaking, we should build the images in CI like we do in dev and prod, using the same docker composes, to ensure that they actually reproduce the right environments.

i couldnt find any other way to run build and pulls in parallel
compose up builds and and pulls images in parallel, but it doesnt store cache anywhere, so to store cache we need to build the image separately, but then if we dont use bake to build and pull, the build and pulls will be sequential and the whole process takes more time
i dont think there will be any difference in the final image depending on how its built, as long as the repodownloader and watchman and othe env vars are set the same

about the redis and postgres declaration, i undestand that these may be problematic to maintain, i'll try to find alternatives

mlodic · 2026-03-08T19:47:48Z

no like i meant, for backend tests ci we can completely skip everything frontend related, we are building and copying but it isnt being used for ci tests

ok but the solution shouldn't make things too complex. Some speed improvs can really make the future maintainance problematic, so adding too much additional stuff can be counter productive. Please compromise for simpler solutions

gqvz · 2026-03-09T12:54:07Z

runtime is about the same +- 10s with this method
for local dev the cache-from line fails with error message and compose falls back to local layer caches

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

mlodic · 2026-03-13T14:06:18Z

I prefer this solution, thanks.

or local dev the cache-from line fails with error message and compose falls back to local layer caches

Can you show the error you mentioned?

And can you link the very last tests?

Then I think we are gtg

gqvz · 2026-03-13T14:18:43Z

=> [internal] load local bake definitions                                                                                                                                                        0.0s
 => => reading from stdin 1.12kB                                                                                                                                                                  0.0s
 => [nginx internal] load build definition from Dockerfile_nginx                                                                                                                                  0.0s
 => => transferring dockerfile: 432B                                                                                                                                                              0.0s
 => [uwsgi internal] load build definition from Dockerfile                                                                                                                                        0.0s
 => => transferring dockerfile: 3.81kB                                                                                                                                                            0.0s
 => [uwsgi internal] load metadata for docker.io/library/node:lts-alpine3.21                                                                                                                      0.0s
 => [uwsgi internal] load metadata for docker.io/library/python:3.11.7                                                                                                                            1.9s
 => [nginx internal] load metadata for docker.io/library/nginx:1.29.2-alpine                                                                                                                      1.9s
 => [uwsgi internal] load .dockerignore                                                                                                                                                           0.0s
 => => transferring context: 306B                                                                                                                                                                 0.0s
 => [nginx 1/2] FROM docker.io/library/nginx:1.29.2-alpine@sha256:61e01287e546aac28a3f56839c136b31f590273f3b41187a36f46f6a03bbfe22                                                                0.0s
 => => resolve docker.io/library/nginx:1.29.2-alpine@sha256:61e01287e546aac28a3f56839c136b31f590273f3b41187a36f46f6a03bbfe22                                                                      0.0s
--------------------------------------HERE------------------------------------------------------------------------------
 => ERROR [uwsgi] importing cache manifest from ghcr.io/intelowlproject/intelowl:cache-main                                                                                                       1.1s
--------------------------------------HERE ^^^^^^^^^^^^^^^^^^----------------------------------------------
 => [uwsgi internal] load build context                                                                                                                                                           4.7s
 => => transferring context: 7.06MB                                                                                                                                                               4.4s
 => [uwsgi backend-build  1/13] FROM docker.io/library/python:3.11.7@sha256:63bec515ae23ef6b4563d29e547e81c15d80bf41eff5969cb43d034d333b63b8                                                      0.0s
 => => resolve docker.io/library/python:3.11.7@sha256:63bec515ae23ef6b4563d29e547e81c15d80bf41eff5969cb43d034d333b63b8                                                                            0.0s
 => [uwsgi frontend-build 1/7] FROM docker.io/library/node:lts-alpine3.21@sha256:b8f7c9056af700568c1ce76173f1c93743fb64ca1343e18cdf3a6ded8985ad3d                                                 0.0s
 => => resolve docker.io/library/node:lts-alpine3.21@sha256:b8f7c9056af700568c1ce76173f1c93743fb64ca1343e18cdf3a6ded8985ad3d                                                                      0.0s
 => CACHED [nginx 2/2] RUN rm /var/log/nginx/access.log /var/log/nginx/error.log && touch /var/log/nginx/access.log /var/log/nginx/error.log                                                      0.0s
 => [nginx] exporting to image                                                                                                                                                                    0.1s
 => => exporting layers                                                                                                                                                                           0.0s
 => => exporting manifest sha256:5a8471a3b6d08ffb39da009c32eebc318e005e42c56f6b67f391372f399def58                                                                                                 0.0s
 => => exporting config sha256:48c889efd7baa013a571de90937782c93c6329083cd9a6f9da6c5f5ef27206f8                                                                                                   0.0s
 => => exporting attestation manifest sha256:81ee136ccc7904ffababe8a6fe7c4c09504ba7bc1fb7d3a0392bd2f692ddbbb6                                                                                     0.0s
 => => exporting manifest list sha256:32f92a9c4073f2252e03d9ba3f5735ffa4a2d2b808d7d4867627608e6974d24c                                                                                            0.0s
 => => naming to docker.io/intelowlproject/intelowl_nginx:ci                                                                                                                                      0.0s
 => => unpacking to docker.io/intelowlproject/intelowl_nginx:ci                                                                                                                                   0.0s
 => [nginx] resolving provenance for metadata file                                                                                                                                                0.0s
 => CACHED [uwsgi frontend-build 2/7] COPY frontend/package.json frontend/package-lock.json ./                                                                                                    0.0s
 => CACHED [uwsgi frontend-build 3/7] RUN npm install npm@11.11.0 --location=global && npm install                                                                                                0.0s
 => CACHED [uwsgi frontend-build 4/7] COPY frontend/ .                                                                                                                                            0.0s
 => CACHED [uwsgi frontend-build 5/7] COPY docker/.env .env.local                                                                                                                                 0.0s

^ falls back to local cache

everything is cached (no changes to any packages) - https://github.com/gqvz/IntelOwl/actions/runs/22853788781/job/66288738273?pr=8

i can rerun the tests for changes to packages but the results will be almost the same as the tests before

feat: optimize Dockerfile and add build caching for CI

95f22eb

Merge branch 'develop' into speedup_ci

2278eee

gqvz marked this pull request as ready for review February 24, 2026 06:49

Copilot AI review requested due to automatic review settings February 24, 2026 06:49

Copilot started reviewing on behalf of gqvz February 24, 2026 06:50 View session

gqvz mentioned this pull request Feb 24, 2026

Speedup CI #3318

Open

Copilot AI reviewed Feb 24, 2026

View reviewed changes

mlodic reviewed Feb 24, 2026

View reviewed changes

Update .github/workflows/docker-build-cache.yml

65e5bb3

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings February 25, 2026 04:19

Update .github/workflows/pull_request_automation.yml

7198eb9

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot started reviewing on behalf of gqvz February 25, 2026 04:19 View session

gqvz and others added 2 commits February 25, 2026 09:50

Apply suggestion from @Copilot

1f42edd

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Remove cache-to settings from Docker build steps

42f61e5

Removed cache-to configuration for Docker builds.

Copilot AI reviewed Feb 25, 2026

View reviewed changes

Comment thread docker/Dockerfile Outdated

Comment thread .github/workflows/pull_request_automation.yml Outdated

Comment thread .github/workflows/pull_request_automation.yml Outdated

Comment thread docker/Dockerfile Outdated

Add django-server-requirements.txt to Dockerfile

6c0fe90

gqvz marked this pull request as draft February 25, 2026 04:37

mlodic reviewed Feb 25, 2026

View reviewed changes

gqvz added 2 commits March 5, 2026 17:44

Add cache mounts for npm, apt, and pip installations

a53d280

Use registry cache

9e96dea

gqvz marked this pull request as ready for review March 8, 2026 18:52

gqvz requested review from Copilot and mlodic March 8, 2026 18:52

Copilot started reviewing on behalf of gqvz March 8, 2026 18:53 View session

Copilot AI reviewed Mar 8, 2026

View reviewed changes

Comment thread .github/workflows/docker-build-cache.yml Outdated

Comment thread docker/docker-bake.ci.hcl Outdated

Comment thread .github/workflows/pull_request_automation.yml Outdated

Comment thread .github/workflows/docker-build-cache.yml Outdated

Copilot AI review requested due to automatic review settings March 8, 2026 19:02

Copilot started reviewing on behalf of gqvz March 8, 2026 19:03 View session

Copilot AI reviewed Mar 8, 2026

View reviewed changes

Comment thread docker/Dockerfile

mlodic reviewed Mar 8, 2026

View reviewed changes

gqvz marked this pull request as draft March 9, 2026 11:14

gqvz force-pushed the speedup_ci branch from 5e56296 to 2926906 Compare March 9, 2026 12:49

Remove bake

1f0b907

gqvz force-pushed the speedup_ci branch from 2926906 to 1f0b907 Compare March 9, 2026 12:51

Discard changes to start

97b887b

gqvz marked this pull request as ready for review March 9, 2026 12:54

Copilot AI review requested due to automatic review settings March 9, 2026 12:54

Copilot started reviewing on behalf of gqvz March 9, 2026 12:55 View session

gqvz requested a review from mlodic March 9, 2026 12:55

Copilot AI reviewed Mar 9, 2026

View reviewed changes

mlodic approved these changes Mar 13, 2026

View reviewed changes

mlodic merged commit d81c62c into intelowlproject:develop Mar 13, 2026
12 of 13 checks passed

	- package-ecosystem: "github-actions"
	directory: "/"
	schedule:
	interval: "weekly"
	day: "tuesday"
	target-branch: "develop"
	ignore:
	- dependency-name: "*"
	update-types: ["version-update:semver-patch"]

Uh oh!

Conversation

gqvz commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Checklist

Uh oh!

gqvz commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gqvz commented Feb 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mlodic commented Feb 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gqvz commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mlodic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gqvz commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mlodic commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gqvz commented Mar 7, 2026

Uh oh!

gqvz commented Mar 7, 2026

Uh oh!

mlodic commented Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gqvz commented Mar 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

gqvz commented Feb 24, 2026 •

edited

Loading

gqvz commented Feb 24, 2026 •

edited

Loading

gqvz commented Feb 25, 2026 •

edited

Loading

gqvz commented Feb 25, 2026 •

edited

Loading

mlodic commented Feb 25, 2026 •

edited

Loading

mlodic commented Mar 8, 2026 •

edited

Loading

gqvz commented Mar 8, 2026 •

edited

Loading