Skip to content

CBG-5413: supportability enhancements around metadata migration#8320

Merged
torcolvin merged 6 commits into
mainfrom
CBG-5413
Jun 5, 2026
Merged

CBG-5413: supportability enhancements around metadata migration#8320
torcolvin merged 6 commits into
mainfrom
CBG-5413

Conversation

@gregns1

@gregns1 gregns1 commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

CBG-5413

  • Add migration abandoned stat that tracks the amount of times migration gets to the end of its max tries and still hasn't finished
  • Adds cached bootstrap targets to the _cluster_info endpoint - this tells us the cached bootstrap-doc location for each bucket
  • Adds MetadataMigrationStatus to the _cluster_info endpoint - tracks the per-bucket metadata-migration lifecycle
  • Adds MetadataStoreMode to _status endpoint - this is to tell us if fallback reads are active or not on the metadata store. Left empty if MetadataStore is not used

This is all info I think will be useful in sgcollect for debugging purposes.

Pre-review checklist

  • Removed debug logging (fmt.Print, log.Print, ...)
  • Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
  • Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Dependencies (if applicable)

  • Link upstream PRs
  • Update Go module dependencies when merged

Integration Tests

Copilot AI review requested due to automatic review settings June 1, 2026 18:13
@github-actions

github-actions Bot commented Jun 1, 2026

Copy link
Copy Markdown

Redocly previews

@gregns1 gregns1 changed the title CBG-5413 - CBG-5413 - supportability enhancements around metadata migration Jun 1, 2026
@gregns1 gregns1 changed the title CBG-5413 - supportability enhancements around metadata migration CBG-5413: supportability enhancements around metadata migration Jun 1, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds supportability/observability signals around metadata migration by exposing additional runtime state via admin endpoints, adding a new migration “abandoned runs” stat, and updating OpenAPI + tests to cover the new fields.

Changes:

  • Add metadata_store_mode to /_status to indicate whether dual-store fallback reads are active/inactive.
  • Extend /_cluster_info bucket payload with cached bootstrap target and bucket-level metadata migration status.
  • Add abandoned_runs metadata migration stat for “hit retry ceiling” failures, plus supporting tests/docs.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
rest/utilities_testing_resttester.go Adds a RestTester helper to wait for /_metadata_migration to reach a target state.
rest/metadatamigrationtest/metadata_migration_test.go Asserts /_cluster_info includes bootstrap_target for the bucket after migration flow.
rest/adminapitest/admin_api_test.go Adds admin API tests for migration_status in /_cluster_info and metadata_store_mode in /_status.
rest/admin_api.go Implements new /_status + /_cluster_info response fields and wires them into handlers.
docs/api/components/schemas.yaml Updates OpenAPI schemas for new /_status and /_cluster_info fields and adds new migration status schemas.
db/background_mgr_metadata_migration.go Increments new “abandoned runs” stat when the bounded pass loop gives up.
db/background_mgr_metadata_migration_test.go Updates test setup to include DbStats needed for the new stat path.
base/stats.go Adds abandoned_runs to migration stats and registers/unregisters it.
base/stats_descriptions.go Adds description text for the new abandoned_runs stat.
base/rosmar_cluster.go Implements CachedBootstrapTargets() for Rosmar bootstrap connection.
base/dual_metadata_store.go Introduces MetadataStoreMode and helper to classify metadata store mode for /_status.
base/bootstrap.go Extends BootstrapConnection interface and implements cached bootstrap target exposure for CouchbaseCluster.

Comment thread db/background_mgr_metadata_migration.go
Comment thread rest/admin_api.go Outdated
Comment thread base/bootstrap.go Outdated
Comment thread base/bootstrap.go Outdated
Comment thread base/bootstrap.go Outdated
func (cc *CouchbaseCluster) CachedBootstrapTargets() map[string]string {
targets := make(map[string]string)
cc.bucketBootstrapTargets.Range(func(key, value any) bool {
if s := value.(bucketBootstrapTarget).String(); s != "" {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should type check on s, ok := value.(bucketBootstrapTarget)

Also, do you think we should throw an assertion error if s == ""? Should we return unknown, so we know it isn't just missing?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will return unknown for missing. Updates docs to reflect this too. I think the type asserting everywhere is a bit messy and not even sure what to do in event of type assert failing (although that shouldn't happen). So i made a SyncMap wrapper to allow generics to make it type safe.

Comment thread base/rosmar_cluster.go Outdated
Comment thread docs/api/components/schemas.yaml
Comment thread docs/api/components/schemas.yaml
rest.RequireStatus(t, resp, http.StatusOK)
err := base.JSONUnmarshal(resp.BodyBytes(), &clusterInfoResponse)
require.NoError(t, err)
assert.Equal(t, "system_mobile", clusterInfoResponse.Buckets[rt.Bucket().GetName()].BootstrapTarget)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can add this into existing tests but I'd love to see a case where it returns either "" (or omited) and also "default"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add assertion to test for _default case. Separate test made for omitted case.

Comment thread base/bootstrap.go Outdated
Comment thread rest/admin_api.go Outdated
Comment thread rest/admin_api.go Outdated
@torcolvin torcolvin assigned gregns1 and unassigned torcolvin Jun 3, 2026
@torcolvin torcolvin merged commit 2e5d0ad into main Jun 5, 2026
55 checks passed
@torcolvin torcolvin deleted the CBG-5413 branch June 5, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants