Bug: Using kubernetes-novolume mode breaks composite actions — runner cannot find action.yml under RUNNER_WORKSPACE

[talsh-oasis]

Summary

After migrating our GitHub Actions Runner Scale Set from containerMode: kubernetes (with kubernetesModeWorkVolumeClaim) to the new containerMode: kubernetes-novolume implementation, composite actions can no longer resolve their local paths.

We receive the following error:
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/runner/_work/cloudburst/cloudburst/.github/workflows/playwright-test'. Did you forget to run actions/checkout before running your local action?

✔ Previous (Working) Configuration — kubernetes mode (with PVC)

RunnerScaleSet values:

runner-e2e:
  containerMode:
    type: "kubernetes"
    kubernetesModeWorkVolumeClaim:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "github-runners-sc"
      resources:
        requests:
          storage: 10Gi
  template:
    spec:
      containers:
      - name: runner
        env:
        - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE
          value: /home/runner/pod-templates/default.yaml
      volumes:
      - name: hook-extension
        configMap:
          name: runner-e2e-hook-extension

Hook extension (ConfigMap):

data:
  default.yaml: |
    apiVersion: v1
    kind: PodTemplate
    spec:
      serviceAccountName: runner-e2e
      resources:
        requests:
          memory: "8Gi"
          cpu: "3000m"
      containers:
      - name: "$job"
        imagePullPolicy: Always

➡ This configuration works.
Local composite actions are found under:
/home/runner/_work///.github/...

❌ New (Failing) Configuration — kubernetes-novolume mode

runner-e2e:
  containerMode:
    type: "kubernetes-novolume"
  template:
    metadata:
      labels:
        pod-type: runner
    spec:
      containers:
      - name: runner
        env:
        - name: ACTIONS_RUNNER_CONTAINER_HOOKS
          value: /home/runner/k8s-novolume/index.js
        - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE
          value: /home/runner/k8s-novolume/pod-templates/default.yaml
        volumeMounts:
        - name: hook-extension
          mountPath: /home/runner/k8s-novolume/pod-templates

Hook extension (ConfigMap):

data:
  default.yaml: |
    apiVersion: v1
    kind: PodTemplate
    spec:
      serviceAccountName: runner-e2e
      nodeSelector:
        karpenter.sh/nodepool: github-arc
      containers:
      - name: "$job"
        env:
        - name: RUNNER_WORKSPACE
          value: /__w
        - name: GITHUB_WORKSPACE
          value: /__w/cloudburst

❗ Observed Behavior

The job container tries to locate composite action files under:

/home/runner/_work/cloudburst/cloudburst/.github/workflows/playwright-test

But with kubernetes-novolume, the actual workspace appears to be:

/__w/cloudburst/cloudburst

Additionally, /home/runner/_work does not exist at all inside the job container.

This mismatch causes composite actions to fail with:

Can't find 'action.yml', 'action.yaml' or 'Dockerfile'

[zarko-a]

I experienced the same problem.

@talsh-oasis can you clarify does overriding RUNNER_WORKSPACE and GITHUB_WORKSPACE with pod template help resolve this problem? I don't have those specified on my end.

[talsh-oasis]

I experienced the same problem.

@talsh-oasis can you clarify does overriding RUNNER_WORKSPACE and GITHUB_WORKSPACE with pod template help resolve this problem? I don't have those specified on my end.

Unfortunately, it didn’t resolve the issue. I’m still experiencing the same problem on my end.

[zarko-a]

@nikola-jokic can you help us with this issue? I tried digging a bit, and I can see shell scripts in _work/_temp/*.sh mention GITHUB_ACTION_PATH which sounds like what we are looking for here. These files are generated by the runner, right, not container hook?

Looking at the runner, /home/runner/_work/<repo>/<repo> directory is empty. With my limited understanding, runner needs to generate those shell scripts and for that it needs a copy of the repo that contains the composite action. I'm not sure how kubernetes-novolume mode differs from kuberentes [with volume mode] not to pull the repo at this stage.

[zarko-a]

After chatting with an LLM to help me understand better how runner works and how does it interact with container hooks and composite actions - I think have an explanation and an acceptable workaround.

So, with kubernetes-novolume when there are no composite actions. Everything works and this is what happens.

• Runner pod receives job message from GitHub servers containing parsed workflow steps.
• Steps are converted to shell scripts and stored in _work/_temp/*.sh
• Container hook is invoked by runner pod and it copies shell scripts from runner pod to workflow pod
• Container hook execute scripts inside workflow pod
• Scripts run successfully because they reference paths at /__w/ which exist in workflow pod

With kubernetes-novolume when we have composite actions things break because:

• Runner pod receives job message from GitHub servers containing parsed workflow steps.
• Runner creates the shell scripts for each step, copies and executes until it gets to a composite action.
• Runner sees composite action reference like uses: ./.github/actions/my-action
• Runner does NOT download this action (it assumes it's in the checked-out repository)
• However repository is checked out at /__w/repo/repo/ in workflow pod only
• Runner tries to read action.yml from /home/runner/_work/<repo>/<repo>/.github/actions/my-action/ in runner pod which never got the files checked out.
• Job fails with "Can't find 'action.yml'" error because repository files only exist in workflow pod, not runner pod.

An acceptable workaround is not to use local references for composite actions. Instead reference the org/repository as if it was an external repo.
For example: uses: myorg/myrepo/.github/actions/my-action@main instead of uses: ./.github/actions/my-action

• This works because runner treats this as an external action, so it calls GitHub API to get download URL for myorg/myrepo@main.
• Runner downloads entire repository archive to runner pod at _work/_actions/myorg/myrepo/main/
• Runner then reads action.yml from _work/_actions/myorg/myrepo/main/.github/actions/my-action/action.yml in runner pod
• Composite action now exists in runner pod and it works fine.

Proper solution would probably involve copying the composite action yaml from the workflow pod to the runner so it can convert it to shell scripts, to be then copied back to workflow pod and executed 😅. This would have to happen after the [user configured] workflow checkout step. Seems kinda complex and definitely more than I can tackle.

... or maybe the answer is, locally referenced composite actions are not supported with kubernetes-novolume mode 🤷 .

[wd-hopkins]

From my experience, the error is misleading because it is checking for the existence of the action on the runner pod, not the workflow pod. It doesn't properly translate the path if using a container. Overwriting the workspace would change the path to the action once it's copied back to the runner, so it won't be able to find it. Changing GITHUB_WORKSPACE doesn't affect the context value github.workspace which is used to determine the path here.