feat: database level default connection configuration

[dantengsky]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Two New Database-Level Options

DEFAULT_STORAGE_CONNECTION, DEFAULT_STORAGE_PATH

These options define the default storage connection and path for the database. Any table created in the database without explicit storage settings will inherit these database-level defaults.

You can set the options when creating the database, or adjust them later with ALTER DATABASE.

Example

CREATE CONNECTION IF NOT EXISTS prod_s3
  STORAGE_TYPE = 's3'
  ACCESS_KEY_ID = '***'
  SECRET_ACCESS_KEY = '***';

CREATE DATABASE analytics OPTIONS (
  DEFAULT_STORAGE_CONNECTION = 'prod_s3',
  DEFAULT_STORAGE_PATH = 's3://mybucket/analytics/'
);

SHOW CREATE DATABASE analytics;

+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| Database  | Create Database                                                                                                                                   |
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| analytics | CREATE DATABASE `analytics` ENGINE=DEFAULT OPTIONS (DEFAULT_STORAGE_CONNECTION='prod_s3', DEFAULT_STORAGE_PATH='s3://mybucket/analytics/') |
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------+

USE analytics;

CREATE TABLE t AS SELECT 1;

SELECT storage_param
FROM system.tables
WHERE database = 'analytics' AND name = 't';

+-------------------------------------------------------------------------------------------------------+
| storage_param                                                                                         |
+-------------------------------------------------------------------------------------------------------+
| s3 | bucket=mybucket,root=/analytics/,endpoint=https://s3.amazonaws.com,ak=******,iam_role=           |
+-------------------------------------------------------------------------------------------------------+

ALTER DATABASE analytics SET OPTIONS (
  DEFAULT_STORAGE_PATH = 's3://mybucket/analytics_v2/'
);

CREATE TABLE t_v2 AS SELECT 1;

SELECT storage_param
FROM system.tables
WHERE database = 'analytics' AND name = 't_v2';

+----------------------------------------------------------------------------------------------------------+
| storage_param                                                                                            |
+----------------------------------------------------------------------------------------------------------+
| s3 | bucket=mybucket,root=/analytics_v2/,endpoint=https://s3.amazonaws.com,ak=******,iam_role=           |
+----------------------------------------------------------------------------------------------------------+

Notes:

• When creating a database you cannot supply only one of DEFAULT_STORAGE_CONNECTION or DEFAULT_STORAGE_PATH; they must appear together or both be omitted.

• When altering an existing database you may change one option at a time as long as the other option already exists; otherwise the command fails.

• During CREATE DATABASE/ALTER DATABASE, connection reachability and storage path permissions are validated.

• Updating these database-level options does not change storage parameters of existing tables; only tables created afterwards use the new defaults.

• fixes: Feature Request: Database level default connection configuration #18825

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

---

This change is 

[dantengsky]

@codex review

[dantengsky]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Hooray!

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[drmingdrmer]

@drmingdrmer reviewed 7 of 27 files at r1, all commit messages.
Reviewable status: 7 of 27 files reviewed, 1 unresolved discussion (waiting on @dantengsky and @SkyFan2002)

---

src/meta/api/src/database_api.rs line 545 at r1 (raw file):

                .await?
                .map(|v| v.seq())
                .unwrap_or_default();

re-fetching it may not get the correct version of the existing record. transistion contains the existing value. Use the following instead:

transition.prev.map(|seq_v| seq_v.seq)

see:

databend/src/meta/types/src/change.rs

Line 32 in 96108c4

pub struct Change<T, ID = u64>

Code quote:

        if !transition.is_changed() {
            let curr_seq = self
                .get_pb(&db_key)
                .await?
                .map(|v| v.seq())
                .unwrap_or_default();

[drmingdrmer]

@drmingdrmer reviewed 2 of 2 files at r2, all commit messages.
Reviewable status: 7 of 27 files reviewed, 1 unresolved discussion (waiting on @dantengsky and @SkyFan2002)