-
Notifications
You must be signed in to change notification settings - Fork 518
Offload leaf search work to AWS Lambda functions #6157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+3,664
−246
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
c335f75
Offload leaf search work to AWS Lambda functions
fulmicoton-dd 7808c54
CR comment, checking sha256 of downloaded file
fulmicoton-dd 6627ae7
Changes in the lambda offloading
fulmicoton-dd ad71de4
CR comments
fulmicoton-dd c7be288
CR comment: using binary heap in greedy_batch_split
fulmicoton-dd 1127e12
Adding assert on search permit provider unit test
fulmicoton-dd 5006df2
Merge branch 'main' into lambda3
fulmicoton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| # This workflow creates a new release for a quickwit search aws lambda. | ||
| # The artifact is a zip file containing a binary for ARM 64, | ||
| # ready to be deployed by the deployer. | ||
| # | ||
| # See quickwit-lambda-client/README.md | ||
| name: Release Lambda binary | ||
|
|
||
| on: | ||
| push: | ||
| tags: | ||
| - 'lambda-*' | ||
| workflow_dispatch: | ||
| inputs: | ||
| version: | ||
| description: 'Version tag (e.g., v0.8.0)' | ||
| required: false | ||
| default: 'dev' | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| build-lambda: | ||
| name: Build Lambda ARM64 | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: write | ||
| actions: write | ||
| steps: | ||
| - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 | ||
|
|
||
| - name: Set version | ||
| run: | | ||
| if [ "${{ github.ref_type }}" = "tag" ]; then | ||
| # Extract version from tag (e.g., lambda-v0.8.0 -> v0.8.0) | ||
| echo "ASSET_VERSION=${GITHUB_REF_NAME#lambda-}" >> $GITHUB_ENV | ||
| elif [ -n "${{ github.event.inputs.version }}" ] && [ "${{ github.event.inputs.version }}" != "dev" ]; then | ||
| echo "ASSET_VERSION=${{ github.event.inputs.version }}" >> $GITHUB_ENV | ||
| else | ||
| echo "ASSET_VERSION=dev-$(git rev-parse --short HEAD)" >> $GITHUB_ENV | ||
| fi | ||
| - name: Install rustup | ||
| run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain none -y | ||
|
|
||
| - name: Install cross | ||
| run: cargo install cross | ||
|
|
||
| - name: Retrieve and export commit date, hash, and tags | ||
| run: | | ||
| echo "QW_COMMIT_DATE=$(TZ=UTC0 git log -1 --format=%cd --date=format-local:%Y-%m-%dT%H:%M:%SZ)" >> $GITHUB_ENV | ||
| echo "QW_COMMIT_HASH=$(git rev-parse HEAD)" >> $GITHUB_ENV | ||
| echo "QW_COMMIT_TAGS=$(git tag --points-at HEAD | tr '\n' ',')" >> $GITHUB_ENV | ||
| - name: Build Lambda binary | ||
| run: cross build --release --features lambda-release --target aarch64-unknown-linux-gnu -p quickwit-lambda-server --bin quickwit-aws-lambda-leaf-search | ||
| env: | ||
| QW_COMMIT_DATE: ${{ env.QW_COMMIT_DATE }} | ||
| QW_COMMIT_HASH: ${{ env.QW_COMMIT_HASH }} | ||
| QW_COMMIT_TAGS: ${{ env.QW_COMMIT_TAGS }} | ||
| working-directory: ./quickwit | ||
|
|
||
| - name: Create Lambda zip | ||
| run: | | ||
| cd quickwit/target/aarch64-unknown-linux-gnu/release | ||
| cp quickwit-aws-lambda-leaf-search bootstrap | ||
| zip quickwit-aws-lambda-${{ env.ASSET_VERSION }}-aarch64.zip bootstrap | ||
| mv quickwit-aws-lambda-${{ env.ASSET_VERSION }}-aarch64.zip ../../../../ | ||
| - name: Upload to GitHub release | ||
| uses: quickwit-inc/upload-to-github-release@9b2c40fba23bf8dea05b7d2eece24cbc95d4a190 | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| with: | ||
| file: quickwit-aws-lambda-${{ env.ASSET_VERSION }}-aarch64.zip | ||
| overwrite: true | ||
| draft: true | ||
| tag_name: ${{ env.ASSET_VERSION }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,240 @@ | ||
| --- | ||
| title: Lambda configuration | ||
| sidebar_position: 6 | ||
| --- | ||
|
|
||
| Quickwit supports offloading leaf search operations to AWS Lambda for horizontal scaling. When the local search queue becomes saturated, overflow splits are automatically sent to Lambda functions for processing. | ||
|
|
||
| :::note | ||
| Lambda offloading is currently only supported on AWS. | ||
| ::: | ||
|
|
||
| ## How it works | ||
|
|
||
| Lambda offloading is **only active when a `lambda` configuration section is present** under `searcher` in your node configuration. When configured: | ||
|
|
||
| 1. Quickwit monitors the local search queue depth | ||
| 2. When pending searches exceed the `offload_threshold`, new splits are sent to Lambda instead of being queued locally | ||
| 3. Lambda returns per-split search results that are cached and merged with local results | ||
|
|
||
| This allows Quickwit to handle traffic spikes without provisioning additional searcher nodes. | ||
|
|
||
| ## Startup validation | ||
|
|
||
| When a `lambda` configuration is defined, Quickwit performs a **dry run invocation** at startup to verify that: | ||
| - The Lambda function exists | ||
| - The function version matches the embedded binary | ||
| - The invoker has permission to call the function | ||
|
|
||
| If this validation fails, **Quickwit will fail to start**. This ensures that Lambda offloading works correctly before the node begins serving traffic. | ||
|
|
||
| ## Configuration | ||
|
|
||
| Add a `lambda` section under `searcher` in your node configuration: | ||
|
|
||
| ```yaml | ||
| searcher: | ||
| lambda: | ||
| offload_threshold: 100 | ||
| auto_deploy: | ||
| execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role | ||
| memory_size: 5 GiB | ||
| invocation_timeout_secs: 15 | ||
| ``` | ||
| ### Lambda configuration options | ||
| | Property | Description | Default value | | ||
| | --- | --- | --- | | ||
| | `function_name` | Name of the AWS Lambda function to invoke. | `quickwit-lambda-search` | | ||
| | `max_splits_per_invocation` | Maximum number of splits to send in a single Lambda invocation. Must be at least 1. | `10` | | ||
| | `offload_threshold` | Number of pending local searches before offloading to Lambda. A value of `0` offloads everything to Lambda. | `100` | | ||
| | `auto_deploy` | Auto-deployment configuration. If set, Quickwit automatically deploys or updates the Lambda function at startup. | (none) | | ||
|
|
||
| ### Auto-deploy configuration options | ||
|
|
||
| | Property | Description | Default value | | ||
| | --- | --- | --- | | ||
| | `execution_role_arn` | **Required.** IAM role ARN for the Lambda function's execution role. | | | ||
| | `memory_size` | Memory allocated to the Lambda function. More memory provides more CPU. | `5 GiB` | | ||
| | `invocation_timeout_secs` | Timeout for Lambda invocations in seconds. | `15` | | ||
|
|
||
| ## Deployment options | ||
|
|
||
| ### Automatic deployment (recommended) | ||
|
|
||
| With `auto_deploy` configured, Quickwit automatically: | ||
| 1. Creates the Lambda function if it doesn't exist | ||
| 2. Updates the function code if the embedded binary has changed | ||
| 3. Publishes a new version with a unique identifier | ||
| 4. Garbage collects old versions (keeps current + 5 most recent) | ||
|
|
||
| This is the recommended approach as it ensures the Lambda function always matches the Quickwit binary version. | ||
|
|
||
| ### Manual deployment | ||
|
|
||
| You can deploy the Lambda function manually without `auto_deploy`: | ||
| 1. Download the Lambda zip from [GitHub releases](https://github.com/quickwit-oss/quickwit/releases) | ||
| 2. Create or update the Lambda function using AWS CLI, Terraform, or the AWS Console | ||
| 3. Publish a version with description format `quickwit_{version}_{sha256}_{timeout}_{deploy_config}"` (e.g., `quickwit_0_8_0_fa940f44_5120_60s_6c3b2`) | ||
|
|
||
| The description must match the format Quickwit expects, or it won't find the function version. | ||
|
|
||
| ## IAM permissions | ||
|
|
||
| ### Permissions for the Quickwit node | ||
|
|
||
| The IAM role or user running Quickwit needs the following permissions to invoke Lambda: | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "lambda:InvokeFunction" | ||
| ], | ||
| "Resource": "arn:aws:lambda:*:*:function:quickwit-lambda-search:*" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| If using `auto_deploy`, additional permissions are required for deployment: | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "lambda:CreateFunction", | ||
| "lambda:GetFunction", | ||
| "lambda:UpdateFunctionCode", | ||
| "lambda:PublishVersion", | ||
| "lambda:ListVersionsByFunction", | ||
| "lambda:DeleteFunction" | ||
| ], | ||
| "Resource": "arn:aws:lambda:*:*:function:quickwit-lambda-search" | ||
| }, | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": "iam:PassRole", | ||
| "Resource": "arn:aws:iam::*:role/quickwit-lambda-role", | ||
| "Condition": { | ||
| "StringEquals": { | ||
| "iam:PassedToService": "lambda.amazonaws.com" | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ### Lambda execution role | ||
|
|
||
| The Lambda function requires an execution role with S3 read access to your index data. | ||
|
|
||
| Example policy: | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": "s3:GetObject", | ||
| "Resource": "arn:aws:s3:::your-index-bucket/*" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| The execution role must also have a trust policy allowing Lambda to assume it: | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Principal": { | ||
| "Service": "lambda.amazonaws.com" | ||
| }, | ||
| "Action": "sts:AssumeRole" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ## CloudWatch logging | ||
|
|
||
| The Lambda function emits structured logs (JSON) to stdout. To have these logs captured by CloudWatch, add the following iam permissions to the Lambda execution role: | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "logs:CreateLogGroup", | ||
| "logs:CreateLogStream", | ||
| "logs:PutLogEvents" | ||
| ], | ||
| "Resource": "arn:aws:logs:*:*:*" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| No additional configuration is needed on the Quickwit side. | ||
|
|
||
| ## Versioning | ||
|
|
||
| Quickwit uses content-based versioning for Lambda: | ||
| - A SHA256 hash of the Lambda binary is computed at build time | ||
| - This hash is embedded in the Lambda function description as `quickwit:{version}-{sha256_short}` | ||
| - When Quickwit starts, it searches for a version matching this description | ||
| - Different Quickwit builds with the same Lambda binary share the same Lambda version | ||
| - Updating the Lambda binary automatically triggers a new deployment | ||
|
|
||
| ## Example configuration | ||
|
|
||
|
|
||
| Minimal configuration (with auto-deployment): | ||
|
|
||
| ```yaml | ||
| searcher: | ||
| lambda: | ||
| auto_deploy: | ||
| execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role | ||
| ``` | ||
|
|
||
|
|
||
| Full configuration (auto-deployment): | ||
|
|
||
| ```yaml | ||
| searcher: | ||
| lambda: | ||
| function_name: quickwit-lambda-search | ||
| max_splits_per_invocation: 10 | ||
| offload_threshold: 10 | ||
| auto_deploy: | ||
| execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role | ||
| memory_size: 5 GiB | ||
| invocation_timeout_secs: 15 | ||
| ``` | ||
|
|
||
| Aggressive offloading (send everything to Lambda): | ||
|
|
||
| ```yaml | ||
| searcher: | ||
| lambda: | ||
| function_name: quickwit-lambda-search | ||
| offload_threshold: 0 | ||
| auto_deploy: | ||
| execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role | ||
| ``` | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.