[WIP] - Upstream 16530 - Optimize HostList API: conditional DISTINCT + composite index on JobHostSummary by cigamit · Pull Request #560 · ctrliq/ascender

cigamit · 2026-07-02T22:03:42Z

Upstream Summary

Make .distinct() conditional on host_filter query parameter being set — without it, the RBAC IN subquery on a direct FK (inventory_id) cannot produce duplicates, so DISTINCT is pure overhead forcing PostgreSQL to sort/hash-dedup the entire result set
Add composite index (host_id, id DESC) on main_jobhostsummary so the with_latest_summary_id() correlated subquery can use an index-only top-1 scan instead of scanning and sorting per row

The host_list_rbac query pattern is the #2 DB time consumer in Scale Lab testing at 2,871 seconds total, and is worsening (+15%). Unlike AAP-81082, the RBAC subquery itself is clean (simple IN on a direct FK) — the cost is dominated by the unconditional DISTINCT and the correlated subquery lacking a composite index.

Why `.distinct()` is safe to make conditional

SmartFilter.query_from_string() can filter through M2M relationships (e.g., groups__name=...), which produce duplicate rows via JOINs. .distinct() is only needed when host_filter is set. Without host_filter, the queryset flows through HostAccess.filtered_queryset() which filters on inventory_id IN (RBAC subquery) — a direct FK that cannot produce duplicates.

Why the composite index helps

with_latest_summary_id() adds a correlated subquery:

SELECT id FROM main_jobhostsummary WHERE host_id = <outer.id> ORDER BY id DESC LIMIT 1

This runs for every result row. The existing auto-index on host_id alone requires scanning all matching rows then sorting. The composite (host_id, id DESC) index enables a single backward index scan to fetch the top-1 result directly.

EXPLAIN analysis (Scale Lab, Jul 2)

Ran on the aap26-next read replica (37,824 hosts, 16.7M job host summaries, 320K role evaluations).

The correlated subquery is the smoking gun — it scans the entire 367MB PK index backward with a filter instead of using a targeted index:

Index Scan Backward using main_jobhostsummary_pkey (cost=0.43..624,118)
      Filter: (host_id = $0)

Cost 624,118 per host row. With ~23K summaries per host on average (max 92K), each probe is extremely expensive.

Query variant	Estimated cost	Delta
RBAC subquery alone	3,113	—
Minimal (no DISTINCT, no subquery, no JOINs)	3,131	baseline
Full query WITHOUT DISTINCT	7,376	+4,245
Full query WITH DISTINCT	7,384	+4,253

The correlated subquery adds +4,167 cost (56% of total). DISTINCT adds +8 in planner estimate but forces Unique + Sort on 138-byte rows.

Test plan

All existing tests pass (210 host-related tests verified locally)
EXPLAIN on Scale Lab confirms correlated subquery uses full PK scan (cost 624,118/row)
EXPLAIN confirms no composite (host_id, id DESC) index exists
Verify composite index is picked up by the correlated subquery after deployment
Measure host_list_rbac mean query time improvement on Scale Lab

Classification

New or Enhanced Feature

Fixes: https://issues.redhat.com/browse/AAP-81517

…ite index on JobHostSummary

Copilot

Pull request overview

Optimizes the HostList API query plan by reducing unnecessary DISTINCT usage and improving the correlated “latest JobHostSummary per host” lookup via a composite index on JobHostSummary.

Changes:

Makes HostList.get_queryset() apply .distinct() conditionally instead of unconditionally.
Adds a composite DB index on JobHostSummary (host_id, id DESC) to speed up the with_latest_summary_id() correlated subquery.
Adds the corresponding Django migration for the new index.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
awx/api/views/init.py	Makes `distinct()` conditional in `HostList.get_queryset()` before annotating latest summary id.
awx/main/models/jobs.py	Adds a composite `(host, -id)` index on `JobHostSummary` to accelerate “latest summary per host” queries.
awx/main/migrations/0201_jobhostsummary_main_jobhostsumm_host_id_desc.py	Introduces the migration that adds the new composite index.

* Add artifact-based conditional workflow connectors (backend) Implements the backend for #505: a fourth workflow connector type that only activates the downstream node when the parent job finishes with the selected outcome AND a condition over its artifacts (set_stats data) evaluates to true. - New condition_nodes m2m on template and job workflow nodes, backed by through models storing trigger (success/failure/always, default success), artifact_key, operator (eq, ne) and expected_value - Workflow DAG engine evaluates conditions against the parent node's ancestor_artifacts merged with its own job artifacts (same semantics as artifact propagation to downstream extra_vars); the edge only fires when the parent outcome matches the trigger; nodes whose unified job template was deleted are treated as failures for condition routing - A condition edge only counts as an error handling path for a failed node when it will actually fire (trigger covers the failure and the condition holds); otherwise the workflow is marked failed, matching failure_nodes semantics - Artifact comparison parses expected_value as JSON with a bool/int cross-type guard (True does not match 1), falling back to string comparison - API sublists /workflow_job_template_nodes/N/condition_nodes/ and /workflow_job_nodes/N/condition_nodes/; association POST accepts trigger, artifact_key, operator and expected_value; re-posting updates the condition; node serializers expose condition_nodes and condition_edges (through rows prefetched to avoid N+1 on node lists) - Condition data is preserved on template-to-job instantiation, relaunch (including relaunch-from-failed carry-forward) and WFJT deep copy; the deep-copy through-row clone is idempotent and direction-aware - RBAC: connection management requires WFJT admin, same as other edges * Add On Condition workflow connector support to the UI - Link add/edit modal offers an On Condition run type with an Evaluate on selector (On Success by default, On Failure, Always) plus artifact key, operator (equals / not equals) and expected value fields - Add Node wizard Run step gains an On Condition card with the same fields; the in-flight condition values survive the launch-config resetForm, and wizard nav can no longer jump past a step whose Next button is disabled - Visualizer loads condition edges from condition_nodes/condition_edges, saves them through the condition_nodes association endpoint (re-posting updates the stored condition) and disassociates them like other links - Condition links render in warning color in the visualizer and the workflow job output view; the link tooltip shows the trigger and the condition; the legend documents the new connector - Node-level fields (identifier, nodeType, nodeResource) are stripped from promptValues when adding a node, fixing a pre-existing 400 from posting a blank identifier on node creation - New strings extracted to the i18n catalogs with Spanish translations * Add awxkit support for conditional workflow connectors Register the condition_nodes endpoints so awx export includes the fourth edge type instead of silently dropping it (an imported workflow would otherwise run condition children unconditionally as root nodes). Each exported edge keeps the target node natural key under 'id' plus its trigger, artifact_key, operator and expected_value, and the import posts the full association. Re-importing an existing link updates its condition in place, so export/import round-trips stay idempotent. Also adds WorkflowJobTemplateNode.add_condition_node for parity with the other add_*_node helpers.

…d host lists) (#562) * Add automatic ascender_stats_* job artifacts derived from playbook stats Every finished playbook job (jobs launched from job templates; project updates, inventory syncs and ad hoc commands are not affected) now automatically contributes a set of ascender_stats_* keys to its artifacts, computed from the final playbook_on_stats event without requiring set_stats in the playbook: - ascender_stats_changed / ascender_stats_failed booleans (failed includes unreachable hosts, matching the job event semantics) - ascender_stats_changed_hosts / ascender_stats_non_changed_hosts / ascender_stats_failed_hosts / ascender_stats_non_failed_hosts sorted host lists - ascender_stats_hosts_truncated, set when the play involved more hosts than ASCENDER_AUTO_STATS_MAX_HOSTS (default 100), in which case the host lists are omitted so artifacts stay small as they propagate through workflows; the boolean flags are always kept Since these are regular artifacts, they propagate to descendant workflow nodes as extra vars and can drive conditional workflow connectors (requested in #505: traverse a path only when the previous job reported changes). Keys set by the playbook via set_stats win over the automatic ones on collision, and _ansible_no_log redaction is unaffected. The feature is gated by the ASCENDER_AUTO_STATS_ENABLED setting (default true); both settings are exposed in the settings API (Jobs category) and can be overridden per job or per workflow with extra variables of the same name, resolved at the single per-job stats event so the event handler hot path is untouched. A negative max hosts override through extra vars is ignored in favor of the setting, which is itself validated with min_value=0. * Merge ascender_stats_* keys when aggregating artifacts of sibling jobs Every playbook job now emits the same ascender_stats_* key names, so aggregating artifacts with plain dict.update() let the last finished sibling mask the others: with a sliced job template, a failed host reported by slice 1 disappeared if slice 2 finished clean afterwards, so a conditional connector on ascender_stats_failed would not fire and the per-host lists passed downstream were wrong. The same applied to nested workflows and to several parents converging into one child. The ascender_stats_* keys are now merged where sibling artifacts meet (WorkflowJob.get_effective_artifacts and the ancestor artifacts built in get_job_kwargs): booleans are OR-ed and the host lists unioned, with truncation on any side dropping the merged lists. Everything else keeps last-writer-wins, and within a single branch a node's own output still supersedes what it inherited.

* Add webhook support to Projects to sync on repository push A push to the configured GitHub/GitLab/Bitbucket DC repository now triggers the same update the Sync button does, so the local copy of the project follows the repo without polling schedules or paying the update-on-launch penalty on every job. Projects reuse the existing webhook receivers and signature checks. Only push and tag push events start a sync, everything else is acknowledged and ignored. An optional fnmatch ref filter (webhook_ref_filter) limits which refs cause a sync, and project updates record webhook_service and webhook_guid so duplicate deliveries are dropped, same as jobs. The webhook credential handling stays on the template mixin; projects only take the service/key part (WebhookKeyTemplateMixin) since a sync has no status to post back. * Allow supplying your own webhook key Until now the webhook key could only be generated by the server, which does not play well with configuration as code: every time the resource is re-applied the key changes and the repository webhook has to be updated by hand. The webhook_key field is now also writable (write only) on projects, job templates and workflow job templates. When a key is supplied it is kept as is, so the same secret can be stored in a vault and applied to both the repository and the resource by automation. When the field is left blank the previous behavior remains: a new key is generated whenever the webhook service is set or changed, and blanking the key of an active webhook generates a fresh one. Keys are never returned on the resource itself, reading them still requires the webhook_key endpoint, and copies of a resource always get their own key. The key field in the UI is now an editable input with the same semantics. * Renumber webhook migration to 0202 to follow merged 0201 from PR #561 The conditional connectors PR (#561) already claimed migration 0201, so this moves the project webhooks migration to 0202 and updates its dependency chain accordingly.

We removed react-router-dom and added react-router, so also need to ensure the license files are properly add / removed.

Co-authored-by: test <test@test.com>

Workflow nodes get a max_retries setting (0-100, default 0). While a failed or errored node has retries left the scheduler treats it as still running instead of following its failure paths, and respawns its job on the next cycle with the same launch configuration. The node tracks the attempt count in retry_attempts and keeps superseded attempts in retried_jobs, which stay protected from deletion while the workflow runs. Canceled jobs, approval nodes, deleted templates and jobs that never started are not retried. The field lives on both the template node and the job node and is copied at launch, editable through the API, awxkit and the node form in the workflow visualizer.

* Allow pinning hosts to every slice of a sliced job Job slicing distributes the inventory evenly across slices, which breaks playbooks where a play targets a host the rest of the run depends on. The usual victim is a preparatory play against localhost: after slicing, only the slice that happened to receive localhost runs it, and the other slices miss whatever it set up. This adds a job_slice_pinned_hosts field to job templates: a comma separated list of inventory host names that are taken out of the round robin distribution and included in every slice instead, keeping their group memberships and variables. Names that do not match an inventory host are ignored and the field does nothing when the job is not sliced. Pinned hosts do not add to the work worth distributing, so they no longer count when capping the slice count to the number of hosts: an inventory of 4 hosts with one of them pinned yields at most 3 slices. The docs spell out the tradeoff: plays targeting a pinned host run once per slice, concurrently, so this is meant for idempotent coordination plays (localhost being the typical case). * Fix title case for Job Slice Pinned Hosts form label The form label used sentence case while the detail view and surrounding fields use title case. Align them for consistency. * Fix task impact counting and waitFor in pinned hosts test Use inventory query instead of len() for pinned host count in _get_task_impact so duplicates and unknown names are excluded. Fix waitFor in the test to throw when element is absent so it actually waits for the re-render.

#574)

…ad of failing (#573) When exporting resources (e.g. job_templates), the awxkit export code iterates over POST fields from the OPTIONS response. For write_only fields like webhook_key, the key appears in both post_fields and _page.related (added by the serializer's get_related), but the resolved endpoint (e.g. /api/v2/job_templates/N/webhook_key/) is served by a view whose page class (Base) has no NATURAL_KEY defined. Previously this caused the export to log an ERROR, set _has_error=True, and return None for every affected resource, producing empty asset lists. Fix: 1. Phase A (direct fields): When the resolved related endpoint has no NATURAL_KEY attribute, log a warning and skip the field instead of erroring out. 2. Documentation: Add webhook_key to DEPENDENT_NONEXPORT for Project, JobTemplate, and WorkflowJobTemplate. Co-authored-by: test <test@test.com>

…bes (#576) The task pod readiness probe runs awx-manage check_instance_ready when task_readiness_period is greater than 0, but the command was missing from this repo, so task pods never became ready. Bring in the command as it exists in upstream AWX (identical in 24.6.1 and devel): exit non-zero unless the local instance node_state is ready. Fixes #571 Co-authored-by: test <test@test.com>

#575) Commit c8eeb60 (#559) added user=root to the [supervisord] section of tools/docker-compose/supervisor.conf, but the docker-compose development container runs as the invoking user (uid 1000), not root. supervisord refuses to start with "Error: Can't drop privilege as nonroot user", so the awx container exits right after bootstrap and make docker-compose is broken on main. Removing the line restores the previous behavior: supervisord runs as whatever user launched it, which is what the development container expects. Co-authored-by: test <test@test.com>

…aries (#577) The job_host_summaries API endpoint for hosts and groups in constructed inventories was returning empty results because it queried the host FK (which points to the source inventory host) instead of the constructed_host FK. This matches the fix already present in the serializer for recent_jobs. Also adds a relink step after inventory sync that reconnects orphaned JobHostSummary records (where the host FK became NULL after the host was deleted by an overwrite sync) to the new host object by matching on host_name, scoped to the same inventory to prevent cross-inventory mis-linking.

* Add context_template to workflow approval nodes for approver visibility Approval nodes can now include a Jinja2 context_template that gets rendered with upstream set_stats artifacts when the approval is created. The rendered result is stored as context_message on the WorkflowApproval instance and displayed to the approver in the UI detail view and in the pending approval notification. This lets workflow authors surface relevant context (dry-run output, planned changes, terraform plan summaries) directly to the person who needs to approve, without them having to go find the previous job output. * Harden approval context_template rendering against resource exhaustion The Jinja2 sandbox blocks attribute escapes but not templates that burn CPU or memory, so rendering now happens in a short lived forked process with RLIMIT_CPU and RLIMIT_AS applied, and the task manager abandons and kills it after a hard timeout instead of blocking the scheduler loop. The rendered output is capped at 64KB. Also render with the artifacts passed as a context dict instead of **kwargs, so set_stats keys that are not valid identifiers cannot raise TypeError, and allow static templates to render when a node has no upstream artifacts. Any child process failure is logged and the approval is simply created without a context message. Adds functional tests covering rendering, static templates, weird artifact keys, template errors, the render timeout, the CPU and memory limits and output truncation. * Keep context_message out of the workflow approval list serializer The rendered context can be large, so only the detail endpoint returns it. The UI detail view already reads it from the detail endpoint. Adds an API test asserting the field shows up on detail and not on the list.

Copilot

Pull request overview

Copilot reviewed 107 out of 123 changed files in this pull request and generated 10 comments.

                        endpoint.post(post_data)
                        log.error("endpoint: %s, id: %s", endpoint.endpoint, rel_page['id'])
                        self._has_error = True


    additional_galaxy_env:
      # These paths control where ansible-galaxy installs collections and roles on top the filesystem
-      ANSIBLE_COLLECTIONS_PATHS: "{{ projects_root }}/.__awx_cache/{{ local_path }}/stage/requirements_collections"
+      ANSIBLE_COLLECTIONS_PATH: "{{ projects_root }}/.__awx_cache/{{ local_path }}/stage/requirements_collections"


+
+
+class Command(BaseCommand):
+    help = 'Check if the task manager instance is ready throw error if not ready, can be use as readiness probe for k8s.'


    remainingValues.webhook_credential = webhook_credential?.id || null;
+    if (webhook_key) {
+      remainingValues.webhook_key = webhook_key;
+    }


    remainingValues.project = project.id;
    remainingValues.webhook_credential = webhook_credential?.id;
+    if (webhook_key) {
+      remainingValues.webhook_key = webhook_key;
+    }


    templatePayload.webhook_credential = webhook_credential?.id || null;
+    if (webhook_key) {
+      templatePayload.webhook_key = webhook_key;
+    }


    templatePayload.webhook_credential = webhook_credential?.id;
+    if (webhook_key) {
+      templatePayload.webhook_key = webhook_key;
+    }


+    if (webhook_key) {
+      values.webhook_key = webhook_key;
+    }


+    if (webhook_key) {
+      values.webhook_key = webhook_key;
+    }


    def get_queryset(self):
        qs = super(HostList, self).get_queryset()
        filter_string = self.request.query_params.get('host_filter', None)
        if filter_string:
            filter_qs = SmartFilter.query_from_string(filter_string)
            qs &= filter_qs
-        return qs.distinct().with_latest_summary_id()
+            qs = qs.distinct()
+        return qs.with_latest_summary_id()


Upstream 16530 - Optimize HostList API: conditional DISTINCT + compos…

433524d

…ite index on JobHostSummary

cigamit self-assigned this Jul 2, 2026

Copilot AI review requested due to automatic review settings July 2, 2026 22:03

cigamit added the enhancement New feature or request label Jul 2, 2026

Copilot started reviewing on behalf of cigamit July 2, 2026 22:04 View session

cigamit changed the title ~~Upstream 16530 - Optimize HostList API: conditional DISTINCT + composite index on JobHostSummary~~ [WIP] - Upstream 16530 - Optimize HostList API: conditional DISTINCT + composite index on JobHostSummary Jul 2, 2026

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread awx/api/views/__init__.py

Comment thread awx/main/migrations/0206_jobhostsummary_main_jobhostsumm_host_id_desc.py

fernandorocagonzalez and others added 17 commits July 17, 2026 10:27

Fix python tests by correcting license file issues (#569)

0ea9887

We removed react-router-dom and added react-router, so also need to ensure the license files are properly add / removed.

Fix: use singular form ANSIBLE_COLLECTIONS_PATH (#570)

030c82e

Co-authored-by: test <test@test.com>

Update locale strings (#572)

a43ab85

Upgrade django to resolve CVE-2026-53877 CVE-2026-53878 CVE-2026-48588 (

74ecab2

#574)

Upgrade pyasn1 to resolve CVE-2026-59886 (#580)

04affdf

Upgrade setuptools to resolve CVE-2026-59890 (#581)

743c594

Rebase and move migration

54ab597

Copilot AI review requested due to automatic review settings July 17, 2026 15:28

Copilot started reviewing on behalf of cigamit July 17, 2026 15:28 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

cigamit closed this Jul 17, 2026

cigamit deleted the upstream16530 branch July 20, 2026 16:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] - Upstream 16530 - Optimize HostList API: conditional DISTINCT + composite index on JobHostSummary#560

[WIP] - Upstream 16530 - Optimize HostList API: conditional DISTINCT + composite index on JobHostSummary#560
cigamit wants to merge 18 commits into
mainfrom
upstream16530

cigamit commented Jul 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants



		class Command(BaseCommand):
		help = 'Check if the task manager instance is ready throw error if not ready, can be use as readiness probe for k8s.'

Uh oh!

Conversation

cigamit commented Jul 2, 2026

Upstream Summary

Why .distinct() is safe to make conditional

Why the composite index helps

EXPLAIN analysis (Scale Lab, Jul 2)

Test plan

Classification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

Why `.distinct()` is safe to make conditional