CONFIGURATION.md

Batch Queue Configuration

Configuration Parameters

Parameter	Purpose	Valid Range	Typical Values
calc_interval	How often to recalculate all priorities	> 0	5m - 30m
max_age	Maximum age for highest adjustment	> 0	24h - 72h
half_life	Exponential decay rate for usage	> 0	1h - 24h
max_size	Maximum job size for highest adjustment	> 0	1000 - 10000
age_weight	Priority boost for waiting jobs	any	10 - 50
size_weight	Priority boost for job size preference	any	10 - 50
usage_weight	Priority boost for resource fairness	any	20 - 60

Understanding the Priority Formula

The Dynamic Priority Queue calculates priority as:

priority = base_priority 
         + (age_adjustment × age_weight)
         + (usage_adjustment × usage_weight) 
         + (size_adjustment × size_weight)

Usage, Decay, and Garbage Collection

tbd

Common Configurations

First-In-First-Out (FIFO)

You want jobs to run in submission order with minimal overhead.

• No fairness requirements
• Jobs run in submission order
• Minimal computational overhead

Configuration

server {
  batch_queue {
    type        = "fifo"
  }
}

Warnings:

• No protection against monopolization, because there are no tenants or priority

---

Resource Fairness

Enforce strict fair sharing based on resource consumption.

• Tenants who have used more resources get lower priority
• Usage decays exponentially with according to half-life
• Jobs from light users almost always run first
• Age provides escape valve (example: max 48h wait)
• With default batch_eval_gc_threshold=24h: Usage fully removed after 24h (6.25% of original remains at GC time)

Use when:

• You have multiple tenants with equal rights, where you need to prevent resource monopolization
• Resource consumption varies significantly between tenants
• Fairness is more important than urgency

Configuration

server {
  batch_queue {
    type        = "dynamic_priority"
    ...

    config = {
      calc_interval = "10m"
      half_life     = "6h"
      max_age       = "48h"
      max_size      = 5000
      age_weight    = 10
      usage_weight  = 80
      size_weight   = 0
    }
  }
}

Warnings

• Cold start: All tenants at zero resource usage means that this queue behaves like FIFO initially
• Urgent jobs from heavy users may wait too long
• New tenants flood the queue before building history
• When a tenant submits one large job vs many tiny jobs there are no penalty differences
• When a heavy user submits critical high-priority job, it'll wait behind light users
• If half_life is too long (>12h), penalties persist and then disappear suddenly at GC threshold
• If usage_weight is too high and age_weight is too low, jobs can wait indefinitely

---

Priority Lanes

Respect job priority over fairness.

• Base priority (ex. 0-100) dominates
• Light fairness touch (usage_weight=20)
• High-priority jobs run quickly even from heavy users
• Usage penalties forgive slowly (half_life=12h)

Use when:

• Urgency matters more than fairness

Configuration

server {
  batch_queue {
    type        = "dynamic_priority"
    ...

    config = {
      calc_interval = "10m"
      half_life     = "12h"
      max_age       = "72h"
      max_size      = 5000
      age_weight    = 20
      usage_weight  = 20
      size_weight   = 0
    }
  }
}

Warnings

• The light fairness enforcement means that heavy users can monopolize the queue
  • If usage_weight too low, monopolization is possible
• Requires policy enforcement on job priorities outside of the queue (Gaming is possible if users can set their own priorities)

---

Backfill Small Jobs

Maximize cluster utilization by preferring small jobs that can fit in gaps.

• Small jobs get priority boost, large jobs get deprioritized
• Still enforces fairness (usage_weight=30)
• Age_weight=20 prevents indefinite starvation

Use when:

• Mix of large and small jobs
• Want to maximize throughput based on cluster resources

Configuration

server {
  batch_queue {
    type        = "dynamic_priority"
    ...

    config = {
      calc_interval = "5m"
      half_life     = "3h"
      max_age       = "24h"
      max_size      = 5000
      age_weight    = 20
      usage_weight  = 30
      size_weight   = 30
    }
  }
}

Warnings

• Large jobs may wait a very long time, while small jobs can flood the queue
  • age_weight is set, but it might not be able to overcome size penalty fast enough
• Doesn't guarantee backfill placement (scheduler still makes placement decisions)
• If all jobs are large, size_weight has no effect
• If max_size is misconfigured (too small), all jobs look "large"

---