Catalog is recreated if storage_root contains trailing slash

[jjpass00]

Describe the issue

Change in workspace access settings of a catalog created with PyDABs triggers resource recreation in next deployment. At the same time, for YAML defined catalogs, changes to workspace settings are ignored (not updated) in deployment.

Behaviour should be identical.

In my opinion PyDABs should match existing YAML behaviour until isolation_mode -field discussed in #5183 is fully supported (i.e. you can define which workspaces have access) since otherwise single metastore multi-workspace setups cannot manage catalogs using DABs when isolation between workspaces in needed.

Configuration

from databricks.bundles.catalogs import Catalog

pydabs_catalog = Catalog.from_dict({"name": "pydabs_catalog", "storage_root": "<storage_url>"})

resources:
  catalogs:
    yaml_catalog:
      name: yaml_catalog
      storage_root: <storage_url>

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Run databricks bundle deploy ...
2. By default catalogs are accessible from all workspaces → change workspace access so that only the workspace used for deployment has access.
3. Redeploy.

Expected Behavior

Either both catalogs are:

1. recreated.
2. updated so that workspace access settings match initial "All workspaces have access" state.
3. left untouched (changes to workspace access settings are ignored).

Actual Behavior

DABs reports that the PyDABs catalog was recreated and 0 changed (i.e. YAML catalog was left unchanged)

Recreated catalogs.pydabs_catalog
Files: 0 uploaded, 0 deleted
Resources: 1 created, 0 changed, 1 deleted, 1 unchanged

OS and CLI version

Windows 11, Databricks CLI v1.16.0

Is this a regression?

Haven't experimented with other versions.

Debug Logs

Na

[janniklasrose]

I was unable to reproduce this. Could you please share the output of databricks bundle plan --output json to help understand why the two resources are planned differently?

[jjpass00]

Did a bit more digging of the output myself and found that the culprit was a trailing / in the value of storage_root afterall. So mistake on my end, sorry for the inconvenience!

This closes the original issue but opens a new one:

For me it seems like the trailing slash is dropped during deployment but this is done after comparing the two states. This results in the deployment thinking there has been a change in the resource and because the change is in the storage_root field, a recreation is triggered.

This should be fixed i.e. the deployment should either:

1. Fail (invalid value for root parameter, trailing /)
2. Cleanup the variable before comparing

I'd go with the first option.

[janniklasrose]

Good find! Indeed, we already have this kind of normalisation in place for schemas and volumes and it seems catalogs were simply missed.