Skip to content

Add AQL pagination and dynamic include fields to avoid query timeouts#178

Open
christianwaldmann wants to merge 4 commits intodevopshq:masterfrom
christianwaldmann:fix-aql-performance
Open

Add AQL pagination and dynamic include fields to avoid query timeouts#178
christianwaldmann wants to merge 4 commits intodevopshq:masterfrom
christianwaldmann:fix-aql-performance

Conversation

@christianwaldmann
Copy link
Copy Markdown
Contributor

@christianwaldmann christianwaldmann commented Apr 22, 2026

Motivation

AQL queries can timeout out on large repositories. Two causes:

  1. Fetching all results in one AQL request can hit the query timeout threshold on large repos (default 900 s). Paginating in batches keeps each query within it.
  2. Always fetching property and stat triggers unnecessary SQL joins even when the active rules don't need those fields, inflating every query

What does this PR do?

  • Add optional page_size parameter to fetch artifacts in pages instead of a single request. When not set, behavior is unchanged
  • Add aql_add_include() hook so each rule declares only the AQL fields it actually needs, replacing the hardcoded .include("*", "property", "stat") on every query

Usage

policies:
  - name: Remove old files
    page_size: 5000
    rules:
      - rule: ...

@christianwaldmann christianwaldmann marked this pull request as ready for review April 24, 2026 07:03
@christianwaldmann
Copy link
Copy Markdown
Contributor Author

hi @allburov, can you have a look at this PR please and include it in a new release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant