Problem
I run Ascender in an environment with several isolated network zones (think datacenters or security zones). Each zone has its own execution nodes grouped in an instance group, because the firewalls only allow the execution nodes of a zone to reach the hosts of that same zone.
The playbooks are the same everywhere: baseline stuff like sysctl, users, limits, patching. But a job runs on a single instance group, so today I have to duplicate every job template per zone (same project, same playbook, only the limit and the instance group change), or split the inventory into one inventory per zone and still duplicate the templates. With N baseline templates and M zones that turns into N x M templates that only differ in two fields, and every new zone means another copy of everything.
Proposal
Let a job template fan out by instance group using inventory data, the same way sliced jobs fan out by count.
- The inventory already knows where each host lives. Add an opt in field on the job template, something like
instance_group_routing_var, holding the name of a host variable (in practice usually inherited from a group var). Example: the template sets instance_group_routing_var: dc_instance_group, and in the inventory the group datacenter1 has dc_instance_group: dc1-nodes while datacenter2 has dc_instance_group: dc2-nodes.
- At launch, when the field is set, hosts get bucketed by the value of that variable. The launch creates the same implicit workflow job that sliced jobs already create, with one node per bucket. Each node runs the job with
limit restricted to the hosts of its bucket and with the instance group named by the value.
- Hosts that don't define the variable go into one extra bucket that runs with the normal instance group resolution of the template (job template, then inventory, then organization, then default).
- If everything ends up in a single bucket, launch a plain job, like slicing already does when the effective slice count is 1.
Security
Instance groups are global objects protected by RBAC (use_role), and inventory variables can be edited by people who have no permission at all on those instance groups, so the variable can't be trusted blindly. What I'd propose:
- The feature does nothing unless it's explicitly enabled on the template, and setting the routing field requires admin on the template like any other template edit.
- At launch, every instance group referenced by the inventory data is checked against the launching user's use permission, exactly the same rule that
JobLaunchConfigAccess already applies to instance groups prompted at launch. If any referenced group fails the check, or simply doesn't exist, the launch is rejected with a clear error naming the group. No silent fallback, that would hide routing mistakes and run jobs somewhere unexpected.
No new scoping is invented here, it's the same use_role check that already protects prompted instance groups.
Why on top of the slicing machinery
The implicit WorkflowJob that sliced jobs create (create_unified_job building a workflow with is_sliced_job and one WorkflowJobNode per slice) already solves relaunch, status rollup and UI presentation for "one launch that fans out into several jobs". Federated inventories in this repo reuse the same pattern, with each node overriding the inventory so the job inherits the instance groups of the source inventory. This RFE is basically the per host variable version of that same idea, for setups where all the zones live in one inventory.
Open questions
- Interaction with
job_slice_count: I'd make them mutually exclusive for now (validation error if both are set) and consider slicing inside each bucket later if someone needs it.
- Whether an installation wide setting to disable the feature is wanted, on top of the per template opt in.
- Name of the field, suggestions welcome.
I'm happy to implement this and send a PR like the previous ones, but I'd rather agree on the approach first, especially the security part.
Problem
I run Ascender in an environment with several isolated network zones (think datacenters or security zones). Each zone has its own execution nodes grouped in an instance group, because the firewalls only allow the execution nodes of a zone to reach the hosts of that same zone.
The playbooks are the same everywhere: baseline stuff like sysctl, users, limits, patching. But a job runs on a single instance group, so today I have to duplicate every job template per zone (same project, same playbook, only the limit and the instance group change), or split the inventory into one inventory per zone and still duplicate the templates. With N baseline templates and M zones that turns into N x M templates that only differ in two fields, and every new zone means another copy of everything.
Proposal
Let a job template fan out by instance group using inventory data, the same way sliced jobs fan out by count.
instance_group_routing_var, holding the name of a host variable (in practice usually inherited from a group var). Example: the template setsinstance_group_routing_var: dc_instance_group, and in the inventory the groupdatacenter1hasdc_instance_group: dc1-nodeswhiledatacenter2hasdc_instance_group: dc2-nodes.limitrestricted to the hosts of its bucket and with the instance group named by the value.Security
Instance groups are global objects protected by RBAC (
use_role), and inventory variables can be edited by people who have no permission at all on those instance groups, so the variable can't be trusted blindly. What I'd propose:JobLaunchConfigAccessalready applies to instance groups prompted at launch. If any referenced group fails the check, or simply doesn't exist, the launch is rejected with a clear error naming the group. No silent fallback, that would hide routing mistakes and run jobs somewhere unexpected.No new scoping is invented here, it's the same
use_rolecheck that already protects prompted instance groups.Why on top of the slicing machinery
The implicit WorkflowJob that sliced jobs create (
create_unified_jobbuilding a workflow withis_sliced_joband oneWorkflowJobNodeper slice) already solves relaunch, status rollup and UI presentation for "one launch that fans out into several jobs". Federated inventories in this repo reuse the same pattern, with each node overriding the inventory so the job inherits the instance groups of the source inventory. This RFE is basically the per host variable version of that same idea, for setups where all the zones live in one inventory.Open questions
job_slice_count: I'd make them mutually exclusive for now (validation error if both are set) and consider slicing inside each bucket later if someone needs it.I'm happy to implement this and send a PR like the previous ones, but I'd rather agree on the approach first, especially the security part.