Workflow start message fails to send: "Submitted message retention period (86398s) cannot be more than 1s..."

[hazzadous]

I get this when using start(). Not every time just sometimes, it appears to be when trying to create a large number of workflow runs.

Error [BadRequestError]: {"error":"Submitted message retention period (86398s) cannot be more than 1s shorter than the current queue's retention period (86400s). Messages in a queue must have non-decreasing retention periods with 1s clock skew tolerance."}
    at q (.next/server/chunks/[root-of-the-server]__4ef27106._.js:7:6746)
    at X.sendMessage (.next/server/chunks/[root-of-the-server]__4ef27106._.js:7:9086)
    at async ea (.next/server/chunks/[root-of-the-server]__4ef27106._.js:7:16994)
    at async Object.o [as queue] (.next/server/chunks/[root-of-the-server]__4ef27106._.js:84:124998)
    at async (.next/server/chunks/[root-of-the-server]__4ef27106._.js:88:9773)
    at async (.next/server/chunks/[root-of-the-server]__4ef27106._.js:4:10981)

I see this in the logs:

POST
vercel-workflow.com/api/v1/runs/create
200

POST
vercel-queue.com/api/v2/messages
400

It looks as though then the workflow is in a Pending state in the observability UI. It looks like the runtime is creating the run then sending a message I presume to trigger that the workflow should start.

It seems unusual for the client to be doing this rather than the vercel-workflow service. I would expect that the workflow service would be responsible for this guarantee. Is there a workaround for this?

I see #451 also highlights some issue around workflows being in pending state, although it looks like there are maybe multiple causes in that issue for the issue.

Using workflow 4.0.1-beta.41

[pranaygp]

workflow is intentionally implemented as a client library - so "vercel-worklow.com" doesn't do any special compute but is just a CRUD app that implements a storage world (like the postgres world)

workflow relies on a separate queue (here: vercel-queue) to actually enqueue work.

You can read more about the worlds abstraction here: https://useworkflow.dev/docs/deploying (although this doc could use more info - so asking claude to explain the source code is also a good option)

---

anyway, regarding your original question - this issue seems relevant to the vercel queue implementation which we're investigating right now

[hazzadous]

Thanks! Yes I had Claude have a rummage through my node modules, was quite handy.

Ill add some resilience my side to handle the queue issues in the meantime 🙏

[pranaygp]

we just fixed an upstream issue in queue that should resolve this. might take a bit to rollout but please lmk if this is still happening next week

[hazzadous]

Great! I don't have the reproducible code to test this any more so I'll just close for now to clean up your list.