PyDABs: support interface{} fields (cluster_policies, dashboards, genie_spaces)

.agents/rules/pydabs-acceptance-tests.md

@@ -5,4 +5,4 @@ paths:
   - "acceptance/bundle/python/**"
 ---
 
-**RULE: Before adding a PyDABs resource acceptance test, read `acceptance/bundle/python/README.md`.** It covers the `<plural>-support/` fixture layout, how to source and adapt realistic field values, the version/engine `test.toml` knobs, and the determinism re-run. Every PyDABs resource needs one (enforced by `test_python_support_coverage`).
+**RULE: Before adding a PyDABs resource acceptance test, read `acceptance/bundle/python/README.md`.** It covers the `<plural>-support/` fixture layout, how to source and adapt realistic field values, the version/engine `test.toml` knobs, the determinism re-run, and how to cover interface (`Any`) fields in both authoring forms. Every PyDABs resource needs one (enforced by `test_python_support_coverage`).

.nextchanges/bundles/pydabs-interface-resources.md

@@ -0,0 +1 @@
+* Added PyDABs (Python) support for cluster policies, dashboards, and Genie spaces. ([#6585](https://github.com/databricks/cli/pull/6585))

acceptance/bundle/python/README.md

@@ -43,5 +43,23 @@ files:
    test that only passes with `-update` is nondeterministic (usually a `$VAR` or a
    volatile field left in); fix it before finishing.
 
+## Interface (`Any`) fields
+
+A field generated as `VariableOrOptional[Any]` comes from a Go `interface{}` (schema ref
+`#/$defs/interface`) — an untyped blob (e.g. a serialized JSON document) that a Go config
+mutator normalizes to a string at deploy time. From Python it accepts two forms — an inline
+`dict` and a serialized JSON `str` — and the two take different `_transform` code paths, so
+a fixture must exercise **both through `add_<singular>`**. Covering only one leaves the
+other path silently brittle, so add two Python instances:
+
+- one sets the field as a **serialized JSON string** (`my_<name>_2`), and
+- one sets it as an **inline dict** (`my_<name>_3`).
+
+The YAML instance (`my_<name>_1`) may use either form, but it exercises the YAML loader, not
+the Python `Any` path, so it does not substitute for either Python instance. The golden
+shows all of them converging: a dict is marshalled to a compact JSON string, a string passes
+through unchanged. See `cluster_policies-support/` for an example. Apply this to every
+resource with an interface field.
+
 Note: `bundle validate` normalizes the `python:` key to `experimental.python` in the
 output — that's expected.

acceptance/bundle/python/cluster_policies-support/databricks.yml

@@ -0,0 +1,21 @@
+bundle:
+  name: my_project
+
+sync: {paths: []} # don't need to copy files
+
+python:
+  resources:
+    - "resources:load_resources"
+  mutators:
+    - "mutators:update_cluster_policy"
+
+resources:
+  cluster_policies:
+    my_cluster_policy_1:
+      name: "my_cluster_policy_1"
+      description: "My cluster policy"
+      # interface{} field authored inline as a YAML map
+      definition:
+        spark_version:
+          type: fixed
+          value: "13.3.x-scala2.12"

acceptance/bundle/python/cluster_policies-support/mutators.py

@@ -0,0 +1,11 @@
+from dataclasses import replace
+
+from databricks.bundles.cluster_policies import ClusterPolicy
+from databricks.bundles.core import cluster_policy_mutator
+
+
+@cluster_policy_mutator
+def update_cluster_policy(cluster_policy: ClusterPolicy) -> ClusterPolicy:
+    assert isinstance(cluster_policy.description, str)
+
+    return replace(cluster_policy, description=f"{cluster_policy.description} (updated)")

acceptance/bundle/python/cluster_policies-support/output.txt

@@ -0,0 +1,33 @@
+
+>>> uv run [UV_ARGS] -q [CLI] bundle validate --output json
+{
+  "experimental": {
+    "python": {
+      "mutators": [
+        "mutators:update_cluster_policy"
+      ],
+      "resources": [
+        "resources:load_resources"
+      ]
+    }
+  },
+  "resources": {
+    "cluster_policies": {
+      "my_cluster_policy_1": {
+        "definition": "{\"spark_version\":{\"type\":\"fixed\",\"value\":\"13.3.x-scala2.12\"}}",
+        "description": "My cluster policy (updated)",
+        "name": "my_cluster_policy_1"
+      },
+      "my_cluster_policy_2": {
+        "definition": "{\"spark_version\": {\"type\": \"fixed\", \"value\": \"13.3.x-scala2.12\"}}",
+        "description": "My cluster policy (2) (updated)",
+        "name": "my_cluster_policy_2"
+      },
+      "my_cluster_policy_3": {
+        "definition": "{\"spark_version\":{\"type\":\"fixed\",\"value\":\"13.3.x-scala2.12\"}}",
+        "description": "My cluster policy (3) (updated)",
+        "name": "my_cluster_policy_3"
+      }
+    }
+  }
+}

acceptance/bundle/python/cluster_policies-support/resources.py

@@ -0,0 +1,32 @@
+from databricks.bundles.core import Resources
+
+
+def load_resources() -> Resources:
+    resources = Resources()
+
+    # interface{} field authored as a serialized JSON string
+    resources.add_cluster_policy(
+        "my_cluster_policy_2",
+        {
+            "name": "my_cluster_policy_2",
+            "description": "My cluster policy (2)",
+            "definition": '{"spark_version": {"type": "fixed", "value": "13.3.x-scala2.12"}}',
+        },
+    )
+
+    # interface{} field authored as an inline dict
+    resources.add_cluster_policy(
+        "my_cluster_policy_3",
+        {
+            "name": "my_cluster_policy_3",
+            "description": "My cluster policy (3)",
+            "definition": {
+                "spark_version": {
+                    "type": "fixed",
+                    "value": "13.3.x-scala2.12",
+                },
+            },
+        },
+    )
+
+    return resources

acceptance/bundle/python/cluster_policies-support/script

@@ -0,0 +1,5 @@
+
+trace uv run $UV_ARGS -q $CLI bundle validate --output json | \
+  jq "pick(.experimental.python, .resources)"
+
+rm -fr .databricks __pycache__

acceptance/bundle/python/cluster_policies-support/test.toml

@@ -0,0 +1,7 @@
+Cloud = false # tests don't interact with APIs
+
+# cluster_policies are only supported in the current version of the wheel
+EnvMatrix.PYDAB_VERSION = ["current"]
+
+# cluster_policies are only supported on the direct deployment engine
+EnvMatrix.DATABRICKS_BUNDLE_ENGINE = ["direct"]

acceptance/bundle/python/dashboards-support/databricks.yml

@@ -0,0 +1,22 @@
+bundle:
+  name: my_project
+
+sync: {paths: []} # don't need to copy files
+
+python:
+  resources:
+    - "resources:load_resources"
+  mutators:
+    - "mutators:update_dashboard"
+
+resources:
+  dashboards:
+    my_dashboard_1:
+      display_name: "my_dashboard_1"
+      warehouse_id: "abc123"
+      parent_path: "/Workspace/Users/me"
+      # interface{} field authored inline as a YAML map
+      serialized_dashboard:
+        pages:
+          - name: main
+            displayName: "Main"

acceptance/bundle/python/dashboards-support/mutators.py

@@ -0,0 +1,11 @@
+from dataclasses import replace
+
+from databricks.bundles.core import dashboard_mutator
+from databricks.bundles.dashboards import Dashboard
+
+
+@dashboard_mutator
+def update_dashboard(dashboard: Dashboard) -> Dashboard:
+    assert isinstance(dashboard.display_name, str)
+
+    return replace(dashboard, display_name=f"{dashboard.display_name} (updated)")

acceptance/bundle/python/dashboards-support/output.txt

@@ -0,0 +1,39 @@
+
+>>> uv run [UV_ARGS] -q [CLI] bundle validate --output json
+{
+  "experimental": {
+    "python": {
+      "mutators": [
+        "mutators:update_dashboard"
+      ],
+      "resources": [
+        "resources:load_resources"
+      ]
+    }
+  },
+  "resources": {
+    "dashboards": {
+      "my_dashboard_1": {
+        "display_name": "my_dashboard_1 (updated)",
+        "embed_credentials": false,
+        "parent_path": "/Workspace/Users/me",
+        "serialized_dashboard": "{\"pages\":[{\"displayName\":\"Main\",\"name\":\"main\"}]}",
+        "warehouse_id": "abc123"
+      },
+      "my_dashboard_2": {
+        "display_name": "my_dashboard_2 (updated)",
+        "embed_credentials": false,
+        "parent_path": "/Workspace/Users/me",
+        "serialized_dashboard": "{\"pages\": [{\"name\": \"main\", \"displayName\": \"Main\"}]}",
+        "warehouse_id": "abc123"
+      },
+      "my_dashboard_3": {
+        "display_name": "my_dashboard_3 (updated)",
+        "embed_credentials": false,
+        "parent_path": "/Workspace/Users/me",
+        "serialized_dashboard": "{\"pages\":[{\"displayName\":\"Main\",\"name\":\"main\"}]}",
+        "warehouse_id": "abc123"
+      }
+    }
+  }
+}

acceptance/bundle/python/dashboards-support/resources.py

@@ -0,0 +1,36 @@
+from databricks.bundles.core import Resources
+
+
+def load_resources() -> Resources:
+    resources = Resources()
+
+    # interface{} field authored as a serialized JSON string
+    resources.add_dashboard(
+        "my_dashboard_2",
+        {
+            "display_name": "my_dashboard_2",
+            "warehouse_id": "abc123",
+            "parent_path": "/Workspace/Users/me",
+            "serialized_dashboard": '{"pages": [{"name": "main", "displayName": "Main"}]}',
+        },
+    )
+
+    # interface{} field authored as an inline dict
+    resources.add_dashboard(
+        "my_dashboard_3",
+        {
+            "display_name": "my_dashboard_3",
+            "warehouse_id": "abc123",
+            "parent_path": "/Workspace/Users/me",
+            "serialized_dashboard": {
+                "pages": [
+                    {
+                        "name": "main",
+                        "displayName": "Main",
+                    },
+                ],
+            },
+        },
+    )
+
+    return resources

acceptance/bundle/python/dashboards-support/script

@@ -0,0 +1,5 @@
+
+trace uv run $UV_ARGS -q $CLI bundle validate --output json | \
+  jq "pick(.experimental.python, .resources)"
+
+rm -fr .databricks __pycache__

acceptance/bundle/python/dashboards-support/test.toml

@@ -0,0 +1,4 @@
+Cloud = false # tests don't interact with APIs
+
+# dashboards are only supported in the current version of the wheel
+EnvMatrix.PYDAB_VERSION = ["current"]

acceptance/bundle/python/genie_spaces-support/databricks.yml

@@ -0,0 +1,22 @@
+bundle:
+  name: my_project
+
+sync: {paths: []} # don't need to copy files
+
+python:
+  resources:
+    - "resources:load_resources"
+  mutators:
+    - "mutators:update_genie_space"
+
+resources:
+  genie_spaces:
+    my_genie_space_1:
+      title: "my_genie_space_1"
+      description: "My genie space"
+      warehouse_id: "abc123"
+      parent_path: "/Workspace/Users/me"
+      # interface{} field authored inline as a YAML map
+      serialized_space:
+        instructions:
+          - "Answer questions about sales data"

acceptance/bundle/python/genie_spaces-support/mutators.py

@@ -0,0 +1,11 @@
+from dataclasses import replace
+
+from databricks.bundles.core import genie_space_mutator
+from databricks.bundles.genie_spaces import GenieSpace
+
+
+@genie_space_mutator
+def update_genie_space(genie_space: GenieSpace) -> GenieSpace:
+    assert isinstance(genie_space.description, str)
+
+    return replace(genie_space, description=f"{genie_space.description} (updated)")

acceptance/bundle/python/genie_spaces-support/output.txt

@@ -0,0 +1,39 @@
+
+>>> uv run [UV_ARGS] -q [CLI] bundle validate --output json
+{
+  "experimental": {
+    "python": {
+      "mutators": [
+        "mutators:update_genie_space"
+      ],
+      "resources": [
+        "resources:load_resources"
+      ]
+    }
+  },
+  "resources": {
+    "genie_spaces": {
+      "my_genie_space_1": {
+        "description": "My genie space (updated)",
+        "parent_path": "/Workspace/Users/me",
+        "serialized_space": "{\"instructions\":[\"Answer questions about sales data\"]}",
+        "title": "my_genie_space_1",
+        "warehouse_id": "abc123"
+      },
+      "my_genie_space_2": {
+        "description": "My genie space (2) (updated)",
+        "parent_path": "/Workspace/Users/me",
+        "serialized_space": "{\"instructions\": [\"Answer questions about sales data\"]}",
+        "title": "my_genie_space_2",
+        "warehouse_id": "abc123"
+      },
+      "my_genie_space_3": {
+        "description": "My genie space (3) (updated)",
+        "parent_path": "/Workspace/Users/me",
+        "serialized_space": "{\"instructions\":[\"Answer questions about sales data\"]}",
+        "title": "my_genie_space_3",
+        "warehouse_id": "abc123"
+      }
+    }
+  }
+}

acceptance/bundle/python/genie_spaces-support/resources.py

@@ -0,0 +1,35 @@
+from databricks.bundles.core import Resources
+
+
+def load_resources() -> Resources:
+    resources = Resources()
+
+    # interface{} field authored as a serialized JSON string
+    resources.add_genie_space(
+        "my_genie_space_2",
+        {
+            "title": "my_genie_space_2",
+            "description": "My genie space (2)",
+            "warehouse_id": "abc123",
+            "parent_path": "/Workspace/Users/me",
+            "serialized_space": '{"instructions": ["Answer questions about sales data"]}',
+        },
+    )
+
+    # interface{} field authored as an inline dict
+    resources.add_genie_space(
+        "my_genie_space_3",
+        {
+            "title": "my_genie_space_3",
+            "description": "My genie space (3)",
+            "warehouse_id": "abc123",
+            "parent_path": "/Workspace/Users/me",
+            "serialized_space": {
+                "instructions": [
+                    "Answer questions about sales data",
+                ],
+            },
+        },
+    )
+
+    return resources