Add migrate from variable support#1162
Open
Supplementing wants to merge 5 commits into
Open
Conversation
📝 WalkthroughWalkthroughThis PR adds a per-variable ChangesVariable-Level migrate_from Feature
🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…Subject to change during fleet implementation
Supplementing
force-pushed
the
add-migrate-from-variable-support
branch
from
60eb085 to
352ee88
Compare
…, and updated description and shorthand ability
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
test/packages/bad_var_migrate_from/_dev/build/docs/README.md (1)
1-102:⚠️ Potential issue | 🟠 Major | ⚡ Quick winTemplate syntax is likely breaking
check-static(merge blocker).This file introduces many markdownlint violations (MD041/MD022/MD037), which aligns with the recurring
Run check-staticCI failure. Please make this fixture README lint-safe (either suppress lint for this template file or adjust template comments/heading spacing).Suggested minimal suppression (if this fixture is intentionally template-heavy)
+<!-- markdownlint-disable MD041 MD022 MD037 --> {{- generatedHeader }} {{/* This template can be used as a starting point for writing documentation for your new integration. For each section, fill in the details described in the comments. @@ These APIs are used with this integration: * ... +<!-- markdownlint-enable MD041 MD022 MD037 -->🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/packages/bad_var_migrate_from/_dev/build/docs/README.md` around lines 1 - 102, The README.md template is triggering markdownlint errors (MD041/MD022/MD037) and failing check-static; fix it by adding a lint suppression at the top of the template (e.g., a markdownlint disable comment for MD041, MD022, MD037) or by adjusting the template comment/heading spacing so headings are separated by a blank line and inline HTML/templating comments (like "{{- generatedHeader }}" and the large "{{/* ... */}}" comment blocks) do not break markdown rules; locate the template start around "{{- generatedHeader }}" and the trailing "{{ inputDocs }}" section and either insert a markdownlint disable directive or reflow the comment blocks/headings to comply with markdownlint.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@spec/changelog.yml`:
- Around line 14-16: Add a per-entry pending marker directly above the changelog
entry whose description is "Add `migrate_from` object to variable definitions to
declare that a variable moved between input-level and stream-level scope within
the same input type." by inserting a comment line "# Pending on <url>"
immediately above that description line (replace <url> with the upstream Kibana
PR/issue URL cited in the review) so the entry itself is clearly marked pending
as required by the changelog policy.
In
`@test/packages/bad_var_migrate_from_scope/data_stream/logs/elasticsearch/ingest_pipeline/default.yml`:
- Around line 6-8: The data_stream.dataset value in the
bad_var_migrate_from_scope ingest pipeline is pointing to the wrong package
fixture; update the value under the field data_stream.dataset (currently
"good_migrate_from.logs") to the correct dataset for this fixture (e.g.,
"bad_var_migrate_from.logs") so the ingest pipeline and the
bad_var_migrate_from_scope fixture reference the matching dataset name.
In
`@test/packages/bad_var_migrate_from/data_stream/logs/elasticsearch/ingest_pipeline/default.yml`:
- Line 7: The dataset identifier value is incorrect: the ingest pipeline sets
value: "good_migrate_from.logs" inside the bad_var_migrate_from fixture; update
the identifier to the correct package fixture name (e.g.,
"bad_var_migrate_from.logs") so the dataset matches the package (modify the
value field in the ingest_pipeline/default.yml to use the bad_var_migrate_from
dataset identifier instead of good_migrate_from.logs).
In `@test/packages/bad_var_migrate_from/docs/README.md`:
- Line 20: The README has three spelling/grammar typos: replace "recieving" with
"receiving" at the high-level overview line, change "will be replace" to "will
be replaced" around line 87, and change "relevent" to "relevant" around line 99;
update those exact words in the README.md so the text reads correctly and
removes static-check noise.
In `@test/packages/good_var_migrate_from/_dev/build/docs/README.md`:
- Line 99: Replace the typo "relevent" with "relevant" in the source README
template that generates the documentation (the string literal "relevent" appears
in the generated README), update the rendered docs/README.md entry to the
corrected spelling, and re-run the docs build to regenerate the artifact so the
change is propagated to the `_dev/build/docs/` output.
In
`@test/packages/good_var_migrate_from/data_stream/logs/elasticsearch/ingest_pipeline/default.yml`:
- Line 7: Update the data_stream.dataset value in the ingest pipeline YAML to
match the package name from manifest.yml: change the current
"good_migrate_from.logs" to "good_var_migrate_from.logs" so it follows the
"<package>.<data_stream>" convention; locate the key named data_stream.dataset
in default.yml and replace the package segment to match the manifest package
name (good_var_migrate_from).
In `@test/packages/good_var_migrate_from/docs/README.md`:
- Line 99: Replace the misspelled word "relevent" with the correct spelling
"relevant" in the README content string "{{/* For integrations that use APIs to
collect data, document all the APIs that are used, and link to relevent
information */}}"; update that comment text so it reads "...link to relevant
information".
---
Outside diff comments:
In `@test/packages/bad_var_migrate_from/_dev/build/docs/README.md`:
- Around line 1-102: The README.md template is triggering markdownlint errors
(MD041/MD022/MD037) and failing check-static; fix it by adding a lint
suppression at the top of the template (e.g., a markdownlint disable comment for
MD041, MD022, MD037) or by adjusting the template comment/heading spacing so
headings are separated by a blank line and inline HTML/templating comments (like
"{{- generatedHeader }}" and the large "{{/* ... */}}" comment blocks) do not
break markdown rules; locate the template start around "{{- generatedHeader }}"
and the trailing "{{ inputDocs }}" section and either insert a markdownlint
disable directive or reflow the comment blocks/headings to comply with
markdownlint.
🪄 Autofix (Beta)
❌ Autofix failed (check again to retry)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 2bafd3e7-40c3-47c6-be9c-3afbfc22dc42
⛔ Files ignored due to path filters (6)
test/packages/bad_var_migrate_from/img/sample-logo.svgis excluded by!**/*.svgtest/packages/bad_var_migrate_from/img/sample-screenshot.pngis excluded by!**/*.pngtest/packages/bad_var_migrate_from_scope/img/sample-logo.svgis excluded by!**/*.svgtest/packages/bad_var_migrate_from_scope/img/sample-screenshot.pngis excluded by!**/*.pngtest/packages/good_var_migrate_from/img/sample-logo.svgis excluded by!**/*.svgtest/packages/good_var_migrate_from/img/sample-screenshot.pngis excluded by!**/*.png
📒 Files selected for processing (36)
code/go/pkg/validator/validator_test.gospec/changelog.ymlspec/integration/data_stream/manifest.spec.ymltest/packages/bad_var_migrate_from/LICENSE.txttest/packages/bad_var_migrate_from/_dev/build/docs/README.mdtest/packages/bad_var_migrate_from/changelog.ymltest/packages/bad_var_migrate_from/data_stream/logs/agent/stream/filestream.yml.hbstest/packages/bad_var_migrate_from/data_stream/logs/agent/stream/stream.yml.hbstest/packages/bad_var_migrate_from/data_stream/logs/elasticsearch/ingest_pipeline/default.ymltest/packages/bad_var_migrate_from/data_stream/logs/fields/base-fields.ymltest/packages/bad_var_migrate_from/data_stream/logs/manifest.ymltest/packages/bad_var_migrate_from/docs/README.mdtest/packages/bad_var_migrate_from/manifest.ymltest/packages/bad_var_migrate_from/sample_event.jsontest/packages/bad_var_migrate_from_scope/LICENSE.txttest/packages/bad_var_migrate_from_scope/_dev/build/docs/README.mdtest/packages/bad_var_migrate_from_scope/changelog.ymltest/packages/bad_var_migrate_from_scope/data_stream/logs/agent/stream/filestream.yml.hbstest/packages/bad_var_migrate_from_scope/data_stream/logs/agent/stream/stream.yml.hbstest/packages/bad_var_migrate_from_scope/data_stream/logs/elasticsearch/ingest_pipeline/default.ymltest/packages/bad_var_migrate_from_scope/data_stream/logs/fields/base-fields.ymltest/packages/bad_var_migrate_from_scope/data_stream/logs/manifest.ymltest/packages/bad_var_migrate_from_scope/docs/README.mdtest/packages/bad_var_migrate_from_scope/manifest.ymltest/packages/bad_var_migrate_from_scope/sample_event.jsontest/packages/good_var_migrate_from/LICENSE.txttest/packages/good_var_migrate_from/_dev/build/docs/README.mdtest/packages/good_var_migrate_from/changelog.ymltest/packages/good_var_migrate_from/data_stream/logs/agent/stream/filestream.yml.hbstest/packages/good_var_migrate_from/data_stream/logs/agent/stream/stream.yml.hbstest/packages/good_var_migrate_from/data_stream/logs/elasticsearch/ingest_pipeline/default.ymltest/packages/good_var_migrate_from/data_stream/logs/fields/base-fields.ymltest/packages/good_var_migrate_from/data_stream/logs/manifest.ymltest/packages/good_var_migrate_from/docs/README.mdtest/packages/good_var_migrate_from/manifest.ymltest/packages/good_var_migrate_from/sample_event.json
Merged
10 tasks
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. An unexpected error occurred while generating fixes: Not Found - https://docs.github.com/rest/git/refs#get-a-reference |
Contributor
There was a problem hiding this comment.
hi! regarding this change i have couple questions:
-
will the variables coming from an input package (composables) be in the scope of this change? when an integration uses packages on its data-streams, the variables from the input package are merged into the integration, and whatever variable is declared at the integration overrides the one coming from the input. With this said, how is this change affecting this cases?
-
i understand fleet reads this field from the manifest an uses the migrated value instead of the "original". How does this affect elastic-package? I believe system tests would need to mirror the behavior if, for example, some system test has variables declared?
Contributor
Author
This wasnt in scope for this change. Input packages themselves cant declare
By my understanding, no elastic-package changes are needed here, the package-spec release step will resolve any issues. Once the spec release happens, elastic-package will pick up the changes required. System tests install one version, the migration is upgrade-only. |
💚 Build Succeeded
History
|
| - description: Add optional `release` field to agentless deployment mode to explicitly declare its release stage. | ||
| type: enhancement | ||
| link: https://github.com/elastic/package-spec/pull/1130 | ||
| - description: Add `migrate_from` object to variable definitions to declare that a variable moved between input-level and stream-level scope within the same input type. |
Contributor
There was a problem hiding this comment.
will this wait until 3.7 or shall it be 3.6.3 ??? cc @jsoriano if we need a quick release of this maybe we can add it to next elastic-package release?
Member
There was a problem hiding this comment.
This needs to go in 3.7, as this requires changes in Kibana.
teresaromero
approved these changes
May 15, 2026
Contributor
teresaromero
left a comment
teresaromero
left a comment
There was a problem hiding this comment.
its fine for 3.7 if packages from this version forward could include this field. I dont see how this is blocking kibana, as the spec will affect packages from version 3.7 kibana should be able to check if the field is there and act on it 🤔 so until there are no packages with version 3.7 kibana wont be seeing this? is this correct?
the spec goes to elastic-package so integration devs can update their packages with the new spec fields, and bump the version. but i spec is not injected at fleet (as far as i know)
Contributor
Author
|
Quick update so theres a trail, this is no longer a blocker for the Kibana side! It can (and likely will) merge before this is ready. It has been verified to pass CI and will work without these changes. |
Supplementing
added a commit
to elastic/kibana
that referenced
this pull request
May 18, 2026
…es (#268789) ## Summary Closes #266831. Builds on the input/stream-level `migrate_from` mechanism (#242934) and the var-rename feature (#265810) to handle the remaining gap: same-input-type upgrades where a variable moves between input and stream scope. Previously, when a package author moved a var from input-level to stream-level (or vice versa) inside the same input type, Fleet silently dropped the user's configured value during updatePackageInputs() because removeStaleVars() discarded any var no longer at its original scope. The existing migrate_from mechanisms only fired on cross-input-type upgrades. ## What this adds - Var-level migrate_from is now an object (`{ name?, scope?, stream? }`) — folding the existing string-form rename into the same field. scope: 'input' | 'stream' declares the var moved scope; name declares it was renamed; both can be combined. Backed by elastic/package-spec#1162 - New applyVarScopeMigration() helper in package_policy_migration_helpers.ts — seeds old-scope values into the new scope's slot before deepMergeVars runs, then the existing removeStaleVars cycle drops the old scope automatically. Mirrors the merge-then-prune pattern used elsewhere. - Wired into the existing-input branch of updatePackageInputs() — fires only on same-input-type upgrades, so cross-input-type migrations continue to use applyInputLevelMigration / applyStreamLevelMigration unchanged. - Multi-stream safety: migrate_from.stream accepts either the fully-qualified dataset (mypackage.logs) or the bare folder name (logs). When omitted on multi-stream inputs, Fleet falls back to the package default rather than silently picking an arbitrary stream. - Secret-typed vars preserve their secret references across scope migration — manual end-to-end test confirmed input ↔ stream movement of a password-typed var carries the existing {id, isSecretRef: true} reference without re-creating or orphaning the underlying secret. - Backwards compatible with the original string-form migrate_from shorthand (#265810): normalized at read time to { name: <string> } so any package built outside the validator still migrates cleanly. - UI tooltip: extends the existing MigrationTooltip pattern with a sibling VarMigrationTooltip that renders an info icon next to migrated var labels during the upgrade flow, with messaging that adapts to whether the var was renamed, moved, or both. Before (v1.0.0) ``` { "items": [ { "id": "b49a95e5-6a57-485c-836c-1605cb027390", "version": "WzI0NCwxXQ==", "spaceIds": ["default"], "name": "good_var_migrate_from-1", "namespace": "", "description": "", "package": { "name": "good_var_migrate_from", "title": "Good Var Migrate From", "version": "1.0.0" }, "enabled": true, "policy_id": "a617c87f-5de4-42f5-846f-667412cd0e7f", "policy_ids": ["a617c87f-5de4-42f5-846f-667412cd0e7f"], "inputs": [ { "type": "filestream", "policy_template": "sample", "enabled": true, "streams": [ { "enabled": true, "data_stream": { "type": "logs", "dataset": "good_var_migrate_from.logs" }, "vars": { "paths": { "value": ["/var/log/*.log"], "type": "text" }, "shared_credential": { // <- Notice that this is in the streams array before "type": "password", "value": { "id": "QAUUGZ4Bllp68onmINYA", "isSecretRef": true } } }, "id": "filestream-good_var_migrate_from.logs-b49a95e5-6a57-485c-836c-1605cb027390", "compiled_stream": { "paths": ["/var/log/*.log"] } } ], "vars": { "api_url": { // <- Notice that this is in the vars "type": "url", "value": "https://www.example.com" } }, "id": "filestream-sample-b49a95e5-6a57-485c-836c-1605cb027390" } ], "secret_references": [ { "id": "QAUUGZ4Bllp68onmINYA" } ], "revision": 1, "created_at": "2026-05-11T22:06:49.897Z", "created_by": "system", "updated_at": "2026-05-11T22:06:49.897Z", "updated_by": "system", "package_agent_version_condition": "^8.9.0" } ], "total": 1, "page": 1, "perPage": 20 } ``` After (v1.0.1 — same data, scopes swapped) ``` { "items": [ { "id": "b49a95e5-6a57-485c-836c-1605cb027390", "version": "WzI2NSwxXQ==", "spaceIds": ["default"], "name": "good_var_migrate_from-1", "namespace": "", "description": "", "package": { "name": "good_var_migrate_from", "title": "Good Var Migrate From", "version": "1.0.1" }, "enabled": true, "policy_id": "a617c87f-5de4-42f5-846f-667412cd0e7f", "policy_ids": ["a617c87f-5de4-42f5-846f-667412cd0e7f"], "inputs": [ { "type": "filestream", "policy_template": "sample", "enabled": true, "streams": [ { "enabled": true, "data_stream": { "type": "logs", "dataset": "good_var_migrate_from.logs" }, "vars": { "paths": { "value": ["/var/log/*.log"], "type": "text" }, "api_url": { // <- Notice that this has now swapped places and is in the streams as expected "type": "url", "value": "https://www.example.com" } }, "id": "filestream-good_var_migrate_from.logs-b49a95e5-6a57-485c-836c-1605cb027390", "compiled_stream": { "paths": ["/var/log/*.log"] } } ], "vars": { "shared_credential": { // <- Notice that this has now moved to the vars "type": "password", "value": { "id": "QAUUGZ4Bllp68onmINYA", "isSecretRef": true } } }, "id": "filestream-sample-b49a95e5-6a57-485c-836c-1605cb027390" } ], "secret_references": [ { "id": "QAUUGZ4Bllp68onmINYA" } ], "revision": 2, "created_at": "2026-05-11T22:06:49.897Z", "created_by": "system", "updated_at": "2026-05-11T22:07:33.934Z", "updated_by": "elastic", "package_agent_version_condition": "^8.9.0" } ], "total": 1, "page": 1, "perPage": 20 } ``` ## Testing Locally These steps reproduce the same end-to-end test used to verify the feature. Assumes you have elastic-package, package-registry, and a local Kibana checkout. 1. Create two versions of a test package Pick any small package (or copy test/packages/good_var_migrate_from from the package-spec branch). You need: - v1.0.0 (before): a var at one scope (e.g. api_url at input level, shared_credential at stream level), no migrate_from. - v1.0.1 (after): the same vars moved to the opposite scope, each declaring migrate_from: ### Input-level (in top-level manifest.yml) - name: shared_credential type: password secret: true migrate_from: scope: stream #### Stream-level (in data_stream/<ds>/manifest.yml) - name: api_url type: url migrate_from: scope: input Same internal name: in both versions, version-bumped in manifest.yml and changelog.yml. Use version: 1.0.0 and up so the registry doesn't tag them release: beta. 2. Build both versions #### elastic-package needs to know about format_version 3.7.0, which adds var-level migrate_from. #### Rebuild from the package-spec branch that adds it: ``` cd /path/to/package-spec # branch: add-migrate-from-variable-support ``` #### Confirm go.mod has `replace github.com/elastic/package-spec/v3 => /path/to/package-spec` ``` cd /path/to/elastic-package go mod tidy && go install ./ ``` #### Build each version ``` cd /path/to/packages/my_test_package_v1 elastic-package build --skip-validation cd /path/to/packages/my_test_package_v2 elastic-package build --skip-validation ``` #### zips land in /path/to/package-spec/build/packages/ (or your repo root's build/packages/) 3. Serve them locally ``` mkdir -p /tmp/fleet-test-packages cp /path/to/build/packages/my_test_package-1.0.0.zip /tmp/fleet-test-packages/ cp /path/to/build/packages/my_test_package-1.0.1.zip /tmp/fleet-test-packages/ cat > /tmp/fleet-test-packages/registry-config.yml <<EOF package_paths: - /tmp/fleet-test-packages EOF cd /path/to/package-registry go build -o package-registry ./ ./package-registry \ --address localhost:8080 \ --feature-storage-indexer=false \ --disable-package-validation \ --config /tmp/fleet-test-packages/registry-config.yml ``` Sanity-check: ``` curl -s 'http://localhost:8080/search?package=my_test_package&all=true' \ | python3 -c "import sys,json; [print(p['version']) for p in json.load(sys.stdin)]" ``` #### expect: 1.0.0 and 1.0.1 4. Run Kibana pointed at the local registry ``` yarn es snapshot # terminal A yarn start --no-base-path --xpack.fleet.registryUrl=http://localhost:8080 # terminal B ``` 5. Exercise the upgrade 1. Integrations → install v1.0.0, attach to an agent policy, set non-default values for both vars. 2. Confirm pre-upgrade scopes with: `GET kbn:/api/fleet/package_policies?perPage=20` 3. Install v1.0.1 and upgrade the policy: `POST kbn:/api/fleet/epm/packages/my_test_package/1.0.1 {}` Then upgrade with `POST kbn:/api/fleet/package_policies/upgrade { "packagePolicyIds": ["<policy-id>"] }` 4. Re-fetch the policy. Expected: - Values are present at the new scopes. - Both old-scope entries are gone. - secret_references array still tracks any moved secret-typed vars (no orphan, no re-create). - Upgrade form shows the info-icon tooltip next to migrated var labels. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
jcger
pushed a commit
to elastic/kibana
that referenced
this pull request
May 26, 2026
…es (#268789) ## Summary Closes #266831. Builds on the input/stream-level `migrate_from` mechanism (#242934) and the var-rename feature (#265810) to handle the remaining gap: same-input-type upgrades where a variable moves between input and stream scope. Previously, when a package author moved a var from input-level to stream-level (or vice versa) inside the same input type, Fleet silently dropped the user's configured value during updatePackageInputs() because removeStaleVars() discarded any var no longer at its original scope. The existing migrate_from mechanisms only fired on cross-input-type upgrades. ## What this adds - Var-level migrate_from is now an object (`{ name?, scope?, stream? }`) — folding the existing string-form rename into the same field. scope: 'input' | 'stream' declares the var moved scope; name declares it was renamed; both can be combined. Backed by elastic/package-spec#1162 - New applyVarScopeMigration() helper in package_policy_migration_helpers.ts — seeds old-scope values into the new scope's slot before deepMergeVars runs, then the existing removeStaleVars cycle drops the old scope automatically. Mirrors the merge-then-prune pattern used elsewhere. - Wired into the existing-input branch of updatePackageInputs() — fires only on same-input-type upgrades, so cross-input-type migrations continue to use applyInputLevelMigration / applyStreamLevelMigration unchanged. - Multi-stream safety: migrate_from.stream accepts either the fully-qualified dataset (mypackage.logs) or the bare folder name (logs). When omitted on multi-stream inputs, Fleet falls back to the package default rather than silently picking an arbitrary stream. - Secret-typed vars preserve their secret references across scope migration — manual end-to-end test confirmed input ↔ stream movement of a password-typed var carries the existing {id, isSecretRef: true} reference without re-creating or orphaning the underlying secret. - Backwards compatible with the original string-form migrate_from shorthand (#265810): normalized at read time to { name: <string> } so any package built outside the validator still migrates cleanly. - UI tooltip: extends the existing MigrationTooltip pattern with a sibling VarMigrationTooltip that renders an info icon next to migrated var labels during the upgrade flow, with messaging that adapts to whether the var was renamed, moved, or both. Before (v1.0.0) ``` { "items": [ { "id": "b49a95e5-6a57-485c-836c-1605cb027390", "version": "WzI0NCwxXQ==", "spaceIds": ["default"], "name": "good_var_migrate_from-1", "namespace": "", "description": "", "package": { "name": "good_var_migrate_from", "title": "Good Var Migrate From", "version": "1.0.0" }, "enabled": true, "policy_id": "a617c87f-5de4-42f5-846f-667412cd0e7f", "policy_ids": ["a617c87f-5de4-42f5-846f-667412cd0e7f"], "inputs": [ { "type": "filestream", "policy_template": "sample", "enabled": true, "streams": [ { "enabled": true, "data_stream": { "type": "logs", "dataset": "good_var_migrate_from.logs" }, "vars": { "paths": { "value": ["/var/log/*.log"], "type": "text" }, "shared_credential": { // <- Notice that this is in the streams array before "type": "password", "value": { "id": "QAUUGZ4Bllp68onmINYA", "isSecretRef": true } } }, "id": "filestream-good_var_migrate_from.logs-b49a95e5-6a57-485c-836c-1605cb027390", "compiled_stream": { "paths": ["/var/log/*.log"] } } ], "vars": { "api_url": { // <- Notice that this is in the vars "type": "url", "value": "https://www.example.com" } }, "id": "filestream-sample-b49a95e5-6a57-485c-836c-1605cb027390" } ], "secret_references": [ { "id": "QAUUGZ4Bllp68onmINYA" } ], "revision": 1, "created_at": "2026-05-11T22:06:49.897Z", "created_by": "system", "updated_at": "2026-05-11T22:06:49.897Z", "updated_by": "system", "package_agent_version_condition": "^8.9.0" } ], "total": 1, "page": 1, "perPage": 20 } ``` After (v1.0.1 — same data, scopes swapped) ``` { "items": [ { "id": "b49a95e5-6a57-485c-836c-1605cb027390", "version": "WzI2NSwxXQ==", "spaceIds": ["default"], "name": "good_var_migrate_from-1", "namespace": "", "description": "", "package": { "name": "good_var_migrate_from", "title": "Good Var Migrate From", "version": "1.0.1" }, "enabled": true, "policy_id": "a617c87f-5de4-42f5-846f-667412cd0e7f", "policy_ids": ["a617c87f-5de4-42f5-846f-667412cd0e7f"], "inputs": [ { "type": "filestream", "policy_template": "sample", "enabled": true, "streams": [ { "enabled": true, "data_stream": { "type": "logs", "dataset": "good_var_migrate_from.logs" }, "vars": { "paths": { "value": ["/var/log/*.log"], "type": "text" }, "api_url": { // <- Notice that this has now swapped places and is in the streams as expected "type": "url", "value": "https://www.example.com" } }, "id": "filestream-good_var_migrate_from.logs-b49a95e5-6a57-485c-836c-1605cb027390", "compiled_stream": { "paths": ["/var/log/*.log"] } } ], "vars": { "shared_credential": { // <- Notice that this has now moved to the vars "type": "password", "value": { "id": "QAUUGZ4Bllp68onmINYA", "isSecretRef": true } } }, "id": "filestream-sample-b49a95e5-6a57-485c-836c-1605cb027390" } ], "secret_references": [ { "id": "QAUUGZ4Bllp68onmINYA" } ], "revision": 2, "created_at": "2026-05-11T22:06:49.897Z", "created_by": "system", "updated_at": "2026-05-11T22:07:33.934Z", "updated_by": "elastic", "package_agent_version_condition": "^8.9.0" } ], "total": 1, "page": 1, "perPage": 20 } ``` ## Testing Locally These steps reproduce the same end-to-end test used to verify the feature. Assumes you have elastic-package, package-registry, and a local Kibana checkout. 1. Create two versions of a test package Pick any small package (or copy test/packages/good_var_migrate_from from the package-spec branch). You need: - v1.0.0 (before): a var at one scope (e.g. api_url at input level, shared_credential at stream level), no migrate_from. - v1.0.1 (after): the same vars moved to the opposite scope, each declaring migrate_from: ### Input-level (in top-level manifest.yml) - name: shared_credential type: password secret: true migrate_from: scope: stream #### Stream-level (in data_stream/<ds>/manifest.yml) - name: api_url type: url migrate_from: scope: input Same internal name: in both versions, version-bumped in manifest.yml and changelog.yml. Use version: 1.0.0 and up so the registry doesn't tag them release: beta. 2. Build both versions #### elastic-package needs to know about format_version 3.7.0, which adds var-level migrate_from. #### Rebuild from the package-spec branch that adds it: ``` cd /path/to/package-spec # branch: add-migrate-from-variable-support ``` #### Confirm go.mod has `replace github.com/elastic/package-spec/v3 => /path/to/package-spec` ``` cd /path/to/elastic-package go mod tidy && go install ./ ``` #### Build each version ``` cd /path/to/packages/my_test_package_v1 elastic-package build --skip-validation cd /path/to/packages/my_test_package_v2 elastic-package build --skip-validation ``` #### zips land in /path/to/package-spec/build/packages/ (or your repo root's build/packages/) 3. Serve them locally ``` mkdir -p /tmp/fleet-test-packages cp /path/to/build/packages/my_test_package-1.0.0.zip /tmp/fleet-test-packages/ cp /path/to/build/packages/my_test_package-1.0.1.zip /tmp/fleet-test-packages/ cat > /tmp/fleet-test-packages/registry-config.yml <<EOF package_paths: - /tmp/fleet-test-packages EOF cd /path/to/package-registry go build -o package-registry ./ ./package-registry \ --address localhost:8080 \ --feature-storage-indexer=false \ --disable-package-validation \ --config /tmp/fleet-test-packages/registry-config.yml ``` Sanity-check: ``` curl -s 'http://localhost:8080/search?package=my_test_package&all=true' \ | python3 -c "import sys,json; [print(p['version']) for p in json.load(sys.stdin)]" ``` #### expect: 1.0.0 and 1.0.1 4. Run Kibana pointed at the local registry ``` yarn es snapshot # terminal A yarn start --no-base-path --xpack.fleet.registryUrl=http://localhost:8080 # terminal B ``` 5. Exercise the upgrade 1. Integrations → install v1.0.0, attach to an agent policy, set non-default values for both vars. 2. Confirm pre-upgrade scopes with: `GET kbn:/api/fleet/package_policies?perPage=20` 3. Install v1.0.1 and upgrade the policy: `POST kbn:/api/fleet/epm/packages/my_test_package/1.0.1 {}` Then upgrade with `POST kbn:/api/fleet/package_policies/upgrade { "packagePolicyIds": ["<policy-id>"] }` 4. Re-fetch the policy. Expected: - Values are present at the new scopes. - Both old-scope entries are gone. - secret_references array still tracks any moved secret-typed vars (no orphan, no re-create). - Upgrade form shows the info-icon tooltip next to migrated var labels. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Closes elastic/kibana#266829
This PR adds support for
migrate_fromat the variable level allowing package maintainers to move variables between scopes (for instance input <-> stream).As we already has an existing field named
migrate_from, it was rolled into an object form. You now must useanyOfname or scope in both places you usemigrate_from.Why is it important?
We needed this in order to stop vars from being dropped when maintainers moved them in scope. See elastic/kibana#264085
Checklist
test/packagesthat prove my change is effective.spec/changelog.yml.Related issues
Summary by CodeRabbit
New Features
Tests
Documentation