How do I backfill a partitioned job one partition at a time, in order?

[rob-apella]

Hi folks,

We have a job with a daily partition. We would like to backfill this job over history, one partition at a time in order.

Is this possible with Dagster?

Thanks!

[zyd14]

Simplest thing that comes to mind would be to just write a script that submits backfills iteratively over the list of partitions you want to backfill (via the GraphQL API). The script could just wait for the current submission to finish before moving to the next one, or you could set the concurrency for the job to 1 and push all the backfills onto the queue.

[HynekBlaha]

or you could set the concurrency for the job to 1 and push all the backfills onto the queue

Tried this some time ago, does not work well if one partition fails.

[zyd14]

@HynekBlaha can you explain more what "does not work well" means? The partitions should be independent of one another, not sure how one partition failing would cause the other partitions to fail. I've definitely done backfills where one or more partitions failed and the others succeed (though not with a concurrency of 1).

[HynekBlaha]

@zyd14 The important missing part is "one at a time in order". The only reason, why "in order" would matter is that the current partition depends on the previous one(s).
As you mentioned, if one fails, the following will independently materialize with incorrect inputs.

[zyd14]

@HynekBlaha Ah okay sure that makes sense, thanks for clarifying