diff --git a/skills/external-volume/LICENSE b/skills/external-volume/LICENSE deleted file mode 100644 index 937dc0de..00000000 --- a/skills/external-volume/LICENSE +++ /dev/null @@ -1,22 +0,0 @@ -Snowflake Skills License - -© 2026 Snowflake Inc. All rights reserved. - -LICENSE: Use of these materials (including all code, prompts, assets, files, and other components of these skills (collectively, "Skills")) is governed by your agreement with Snowflake for the Service. If no separate agreement exists, use is governed by Snowflake's Terms of Service (available at: https://www.snowflake.com/en/legal/terms-of-service/). - -Your applicable agreement is referred to as the "Agreement." "Service" is as defined in the Agreement. - -ADDITIONAL RESTRICTIONS: Notwithstanding anything in the Agreement to the contrary, you may not: - -- Extract from the Service or retain copies of the Skills outside use with the Service; -- Reproduce or copy the Skills, except for temporary copies created automatically during authorized use of the Service; -- Create derivative works based on the Skills; -- Distribute, sublicense, or transfer the Skills to any third party; -- Make, offer to sell, sell, or import any inventions embodied in the Skills; nor, -- Reverse engineer, decompile, or disassemble the Skills. - -The receipt, viewing, or possession of the Skills does not convey or imply any license or right beyond those expressly granted above. - -Snowflake retains all rights, title, and interest in the Skills, including all copyrights, trademarks, patents, and all other applicable intellectual property rights. - -THE SKILLS ARE PROVIDED "AS IS," WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SKILLS OR THE USE OR OTHER DEALINGS IN THE SKILLS. diff --git a/skills/external-volume/SKILL.md b/skills/external-volume/SKILL.md deleted file mode 100644 index c694194e..00000000 --- a/skills/external-volume/SKILL.md +++ /dev/null @@ -1,443 +0,0 @@ ---- -name: external-volume -title: Setup Snowflake External Volumes -summary: Create and manage Snowflake external volumes with cloud storage (AWS S3) for Iceberg tables and data lake access. -description: >- - Create Snowflake external volumes with cloud storage. - Use for ALL requests that mention: setting up external storage, creating - external volume, configuring S3 for Snowflake, Iceberg tables, unloading - data, cloud storage, COPY INTO unload, external stage storage, data lake - storage, IAM trust policy, volume verification. - Triggers: sfutils-extvolumes, setup-external-volume, external volume, - create external volume, s3 snowflake, iceberg storage, data lake storage, - COPY INTO location, external stage, unload to S3, replay volumes, - replay volume manifest, recreate external volume, replay all manifests, - replay all sfutils, export manifest for sharing, setup from shared manifest, - replay from shared manifest, setup from manifest URL, replay from URL, - use manifest from URL, azure, gcs, blob storage, cloud storage, - multiple volumes, list volumes, setup connection, validate manifest, - migrate manifest, second external volume. - Do NOT use for: network rules, PAT creation, general SQL queries, - or non-storage tasks. -aliases: - - setup-external-volume - - external-volumes - - setup-external-volumes -tools: - - Bash - - Read - - Edit - - Glob - - Grep - - ask_user_question -prompt: "$external-volume Create an external volume backed by my S3 bucket for Iceberg tables" -language: en -status: stable -author: Kamesh Sampath -type: snowflake ---- - -# External Volume Setup - -Creates cloud storage resources and a Snowflake external volume for storage access. **Today only the AWS S3 path is implemented**; other backends are roadmap-only (see Step 4). - -**Supported now:** AWS S3 - -**Roadmap (not enabled):** Azure Blob Storage, Google Cloud Storage — do not offer as multiple-choice options; mention in prose only until workflows ship. - -**Use cases:** Iceberg tables, data lake access, COPY INTO unload, external stages. - -## Workflow - -**⚠️ CONNECTION USAGE:** This skill reads the Snowflake connection from `[snowflake].connection` in `manifest.toml`. The CLI passes `-c ` to all `snow sql` calls automatically — no manual env export needed. Admin_role defaults to `[snowflake].admin_role` or ACCOUNTADMIN. - -**📌 CONNECTION NOTE:** The CLI reads connection from `manifest.toml` and auto-injects `-c ` to all `snow sql` calls. No `.env` sourcing needed. Always run `vol setup-connection` to set the connection in `manifest.toml` before any other command. - -**📋 NO PREREQUISITE:** This skill does NOT require sfutils-pat. It operates independently. - -> **📋 MANIFEST AS SOURCE OF TRUTH** -> -> **📍 Location:** `.sfutils/manifest.toml` (ALWAYS this exact path — TOML format, NOT the legacy `.sfutils/sfutils-manifest.md`) -> -> **⛔ DO NOT hand-edit manifests.** Manifests are machine-managed by Cortex Code. Manual edits can corrupt the format and break replay, cleanup, and export flows. Use skill commands to modify resources instead. -> -> **🔒 Security:** Secured like `.ssh` (chmod 700 directory, chmod 600 files) -> -> **Skill-Scoped Admin Roles:** -> -> - Volumes default: `ACCOUNTADMIN` (CREATE EXTERNAL VOLUME privilege) -> - **External volumes are account-level objects — no database or schema setup needed** - -**🚫 FORBIDDEN ACTIONS - NEVER DO THESE:** - -- NEVER run SQL queries to discover/find/check values (no SHOW ROLES, SHOW DATABASES, SHOW EXTERNAL VOLUMES) -- NEVER reference SF_UTILS_DB, SNOW_UTILS_DB, or sf_utils_db — external volumes are account-level objects with no database dependency -- NEVER use `source .env`, `set -a && source .env && set +a`, or `load_dotenv()` for any `vol` command -- NEVER read `BUCKET`, `EXTVOLUME_PREFIX`, `EXTERNAL_VOLUME_NAME`, or `AWS_REGION` from `.env` to pass to `vol` commands — all values MUST be explicit CLI flags or derived from `manifest.toml` -- NEVER auto-populate empty manifest values by querying Snowflake -- NEVER use flags that bypass user interaction: `--auto-setup`, `--auto-approve`, `--quiet`, `--non-interactive` -- **`--yes` / `-y` is REQUIRED** when executing commands after user has approved the dry-run -- NEVER assume user consent - always ask and wait for explicit confirmation -- NEVER skip SQL/JSON in dry-run output - always show BOTH summary AND full SQL/JSON -- NEVER hardcode admin roles - get admin_role from manifest -- NEVER skip manifest - always update manifest IMMEDIATELY after user input -- NEVER leave .sfutils unsecured - always chmod 700/600 -- NEVER delete .sfutils directory or manifest file - preserve for audit/cleanup/replay -- **NEVER guess or invent CLI options** - ONLY use options from the CLI Reference tables -- **NEVER use sed/awk/bash to edit manifest files** — use the file editing tool (Edit/StrReplace) -- **NEVER run raw SQL for cleanup** — ALWAYS use `vol delete` -- **NEVER run `vol create` without first running `vol ... create --dry-run`** — mandatory, non-negotiable -- **NEVER show Step 4 as only** "Only AWS S3 is supported" without the full three-provider list below -- **NEVER assume cwd is the project directory without first confirming project root** - -**✅ REQUIRED ACTIONS - ALWAYS DO THESE:** - -- **ALWAYS run `--dry-run` before any `vol create --yes`. Non-negotiable even if user asks to skip.** -- **ALWAYS run the [Volume Manifest Gate](#volume-manifest-gate) before any manifest-dependent operation (replay, remove, manage-existing).** -- **ALWAYS use `ask_user_question` tool (not plain text output) to collect destructive confirmations.** - -**✅ INTERACTIVE PRINCIPLE:** This skill is designed to be interactive. At every decision point, ASK the user and WAIT for their response before proceeding. - -> **📋 `.env` SCOPE:** `.env` is read **only** by `vol migrate` to extract legacy Snowflake connection values (`SNOWFLAKE_DEFAULT_CONNECTION_NAME`, etc.). No other `vol` command reads `.env`. All operational values — bucket, region, prefix, volume name — come from `manifest.toml` or explicit CLI flags. If a `.env` file exists in the project after migration, **ignore it entirely** for all `vol` commands. - -> **📋 `.env` SCOPE:** `.env` is read **only** by `vol migrate` to extract legacy Snowflake connection values (`SNOWFLAKE_DEFAULT_CONNECTION_NAME`, etc.). No other `vol` command reads `.env`. All operational values — bucket, region, prefix, volume name — come from `manifest.toml` or explicit CLI flags. If a `.env` file exists in the project after migration, **ignore it entirely** for all `vol` commands. - ---- - -## Volume Manifest Gate - -**Non-negotiable precondition for ALL manifest-dependent flows: replay, remove, manage-existing.** - -Run before ANY of those operations: - -```bash -/vol validate-manifest -``` - -**If validation passes:** proceed immediately. - -**If validation fails:** - -```bash -/vol validate-manifest --fix -``` - -Re-run `validate-manifest`. If still failing after `--fix`: - -- These are **non-structural** issues (e.g. empty `connection`) that `--fix` cannot auto-repair -- Show the errors to the user, **STOP**, do not proceed -- Provide actionable next steps (e.g. "run `vol setup-connection` to set connection") - -**If `[snowflake].connection` is empty after fix:** redirect to Step 1 (connection picker) before continuing. - ---- - -### Locate Project Directory - -**Run this before any other step — never skip or assume you are already in the right place.** - -1. **If the user provided a project directory path** → create it if needed and `cd` to it. -2. Check if `.sfutils/manifest.toml` exists in the current directory: - - ```bash - ls -la .sfutils/manifest.toml 2>/dev/null && echo "FOUND" || echo "NOT_FOUND" - ``` - -3. **If found** → current directory is your project root. Proceed to Step 0. -4. **If not found** — check for legacy manifest: - - ```bash - ls -la .sfutils/sfutils-manifest.md 2>/dev/null && echo "LEGACY" || echo "NOT_FOUND" - ``` - - - **Legacy found** → run `vol migrate` (Step 0, Case A) - - **Neither found** and user mentioned existing project → ask for project directory path - - **Fresh setup** → use current directory, proceed to Step 0 - ---- - -### Step 0: Migration / Prerequisite Detection - -Check project state: - -```bash -ls .sfutils/sfutils-manifest.md 2>/dev/null && echo "LEGACY" || true -ls .sfutils/manifest.toml 2>/dev/null && echo "TOML" || true -``` - -**Case A — legacy manifest found, no TOML:** - -```bash -/vol migrate --dry-run -``` - -Review dry-run output, confirm with user, then: - -```bash -/vol migrate -``` - -After migrate, always run `vol check-setup` (Step 2) since `tools_verified` will be empty. - -**Case B — TOML manifest found:** - -Run Volume Manifest Gate. If issues → `vol validate-manifest --fix`. Proceed to Step 1. - -**Case C — neither found (new project):** - -Proceed directly to Step 1 to set up connection and create `manifest.toml`. - -> **✅ FAST-PATH:** If manifest exists and `tools_verified` has a date → skip Step 2 check-setup entirely. - ---- - -### Step 1: Connection Setup - -Check manifest for existing connection: - -```bash -cat .sfutils/manifest.toml 2>/dev/null | grep "^connection" -``` - -**If connection is set and non-empty:** use it, skip to Step 2. - -**If connection is empty or manifest doesn't exist:** - -```bash -snow connection list --format json -``` - -Use `ask_user_question` tool to present connection options to the user. - -Then test and cache to manifest: - -```bash -/vol setup-connection -c -``` - -This writes `[snowflake].connection`, `account`, `user`, and `account_url` to `manifest.toml`. - -**If connection test fails:** show error, ask user to check `snow connection list` and try again. - ---- - -### Step 2: Prerequisites Check (Tool Verification) - -Read prereqs from manifest: - -```bash -cat .sfutils/manifest.toml 2>/dev/null | grep "tools_verified" -``` - -**If `tools_verified` has a date:** Skip to Step 2c — tools already verified. - -**If `tools_verified` is empty or missing:** - -```bash -/vol check-setup --provider s3 -``` - -This checks: -- `snow` CLI available -- `aws` CLI available (for S3 provider) -- AWS credential env signal (profile, static keys, web identity) - -On success, sets `[prereqs].tools_verified` to today's date in `manifest.toml`. - -**If `csp_tools_ready: false`** in the output: the required CSP CLI tool is missing. Show what's missing and stop. - -**⚠️ STOP:** Do not proceed until tools are verified. - ---- - -### Step 2a: Admin Role from Manifest - -Admin role is read from `manifest.toml [snowflake].admin_role` (default: ACCOUNTADMIN). - -Check manifest: - -```bash -cat .sfutils/manifest.toml 2>/dev/null | grep "admin_role" -``` - -**If admin_role is set:** use it. - -**If NOT set**, prompt user: - -``` -External volume creation requires CREATE EXTERNAL VOLUME privilege. - -Snowflake recommends: ACCOUNTADMIN (has this privilege by default) - -Enter admin role for volumes [ACCOUNTADMIN]: -``` - -**⚠️ STOP:** Wait for user input. Then cache to manifest via `vol setup-connection --admin-role `. - ---- - -### Step 2c: Multi-Volume Selection - -```bash -/vol list -``` - -**If no volumes exist:** proceed to Step 3 (new volume). - -**Ask user:** -- Add a new external volume → proceed to Step 3 -- Manage an existing volume → show describe / update-trust / remove options - -**Multi-volume note:** After each volume creation fully completes Steps 4–8 (dry-run → create → IAM propagation → verify → `vol validate-manifest`), return here to offer creating another. Do NOT proceed to the next volume until the current one is COMPLETE in the manifest. - ---- - -### Step 3: Check Existing External Volume - -If user is managing an existing volume or provided a volume name: - -```bash -/vol describe --volume-name -``` - -**If volume exists:** Ask user: -1. Use existing volume (done — provide connection details) -2. Delete and recreate -3. Create new volume with different name - -**If volume doesn't exist or new volume requested:** Continue to Step 4. - ---- - -### Step 4: Storage provider (S3 only today) - -> **📝 NOTE:** This limitation is about **creating Storage Provider resources** (cloud-side infrastructure). Snowflake supports S3, Azure Blob Storage, and GCS as external volume backends. This skill currently only automates provisioning of **AWS S3** cloud resources. - -**Only AWS S3 is implemented.** Always show all three providers — never compress to a one-line question. - -> **✅ SELF-CHECK before sending your response:** Does your message contain all three provider lines? If not, insert the full block below. - -**MANDATORY — paste the following to the user as one unit:** - -``` -Storage provider: - - (•) AWS S3 — supported in this skill and CLI today - ( ) Azure Blob Storage — unavailable (planned / work in progress; not in sfutils-extvolumes yet) - ( ) Google Cloud Storage — unavailable (planned / work in progress; not in sfutils-extvolumes yet) - -Only AWS S3 is supported right now. Proceed with AWS S3 for this external volume? -``` - -**⚠️ STOP:** Wait for confirmation unless user already named S3 as provider. - -**Then:** - -1. Collect S3-specific inputs (bucket, region, aws-profile if applicable) -2. Run S3 provider prereq checks (from [S3 Workflow](workflow-s3.md)) -3. Execute [S3 Workflow](workflow-s3.md) Steps 4-6 - -**After workflow-s3.md Step 6 completes, return here for Step 7.** - ---- - -### Step 7: Verify - -```bash -/vol verify --volume-name -``` - -The CLI automatically retries with exponential backoff on IAM propagation lag — no manual `sleep` needed. If verification still fails after all retries, the error is shown with the last `storageLocationSelectionResult`. - -**Present** verification result and continue to Step 8. - ---- - -### Step 8: Write Success Summary and Validate Manifest - -The CLI automatically writes the volume entry to `.sfutils/manifest.toml` after creation. - -Run manifest validation to confirm it is well-formed: - -```bash -/vol validate-manifest -``` - -If validation fails: run `vol validate-manifest --fix` and re-check. - -See [Manifest Flows](manifest-flows.md) for the full manifest template format and example Iceberg DDL. - -> **Iceberg users:** If you encounter `Access Denied`, trust policy errors, or `SYSTEM$VERIFY_EXTERNAL_VOLUME` failures after the volume is created, load the **`iceberg/external-volume`** sub-skill for cloud-provider-specific troubleshooting. - ---- - -## Maintenance Operations - -Use these workflows when the user asks to inspect or repair an existing external volume. - -### List All Volumes - -```bash -/vol list -``` - -Shows: LABEL / VOLUME_NAME / TYPE / STATUS for all volumes in manifest.toml. - -### Describe Volume - -```bash -/vol describe --volume-name -``` - -Presents: storage location, IAM user ARN, external ID, and status. - -### Update Trust Policy - -When the user reports IAM trust policy mismatch: - -```bash -/vol update-trust --bucket -``` - -After update completes, run `vol verify --volume-name ` to confirm access is restored. - -### Remove Volume - -**Always run Volume Manifest Gate first.** - -```bash -/vol list # confirm label/name -/vol validate-manifest # gate check -/vol delete --bucket --yes --output json # remove resources -``` - -After delete: volume entry in manifest.toml is marked `status = "REMOVED"`, entry preserved. - ---- - -## Reference - -- [CLI Reference](cli-reference.md) — all commands, options, and usage -- [Manifest Flows](manifest-flows.md) — TOML manifest template, progressive write, remove flow -- [Prereqs](prereqs.md) — initialization, migration detection, tool verification -- [Replay Flows](replay-flows.md) — export for sharing, replay flows -- [Supplemental](supplemental.md) — SQL reference, troubleshooting, privilege escalation hints -- [S3 Workflow](workflow-s3.md) — self-contained AWS S3 provider workflow (Steps 4-6) - -## Stopping Points - -1. Step 0: Migration detected — confirm with user before `vol migrate` -2. Step 1: If connection test fails -3. Step 2: If CSP tools missing -4. Step 2a: If admin_role not set (prompts user) -5. Step 2c: Ask user Add new / Manage existing -6. Step 3: If volume exists (ask user what to do) -7. Step 4: Show S3 vs Azure/GCS (Azure/GCS disabled); optional "Proceed with S3?" confirmation -8. Step 4 (S3 workflow): Provider prereqs, requirements gathering, dry-run approval - -## Output - -- Cloud storage bucket/container with appropriate settings -- IAM policy/role or service principal for access -- Snowflake external volume -- Updated `.sfutils/manifest.toml` with volume entry diff --git a/skills/external-volume/cli-reference.md b/skills/external-volume/cli-reference.md deleted file mode 100644 index 9367ebbc..00000000 --- a/skills/external-volume/cli-reference.md +++ /dev/null @@ -1,250 +0,0 @@ -## CLI Invocation - -This skill provides the `vol` wrapper script for simplified CLI access: - -```bash -/vol [options] # main CLI (sfutils-extvolumes) -/vol check-setup [options] # pre-flight tool check -``` - -> Equivalent to `uv run --project sfutils-extvolumes ...` and `uv run --project check-setup ...` respectively. - -**Connection:** The CLI reads `[snowflake].connection` from `manifest.toml` and passes `-c ` to all `snow sql` calls automatically. No `.env` sourcing needed. - -**`.env` scope:** `.env` is read **only** by `vol migrate` to extract legacy connection values (`SNOWFLAKE_DEFAULT_CONNECTION_NAME`, etc.). All other `vol` commands derive values exclusively from `manifest.toml` and explicit CLI flags. Never set `BUCKET`, `EXTVOLUME_PREFIX`, or `EXTERNAL_VOLUME_NAME` in `.env` for operational use. - -**`.env` scope:** `.env` is read **only** by `vol migrate` to extract legacy connection values (`SNOWFLAKE_DEFAULT_CONNECTION_NAME`, etc.). All other `vol` commands derive values exclusively from `manifest.toml` and explicit CLI flags. Never set `BUCKET`, `EXTVOLUME_PREFIX`, or `EXTERNAL_VOLUME_NAME` in `.env` for operational use. - -## Tools - -### check-setup (bundled in the sfutils-extvolumes package) - -**Description:** Pre-flight check for snow CLI and CSP CLI tools (aws/az/gcloud) plus -credential-related environment variables. External volumes are account-level Snowflake -objects — no database setup is required. - -On success, writes `[prereqs].tools_verified` to `manifest.toml`. - -**Usage:** - -```bash -/vol check-setup -/vol check-setup --suggest -/vol check-setup --suggest --provider s3 -``` - -**Options:** - -| Option | Short | Required | Default | Description | -|--------|-------|----------|---------|-------------| -| `--suggest` | - | No | false | Output tool-readiness as JSON | -| `--provider` | - | No | `s3` | Which storage provider's CSP CLIs to check: `s3`, `azure`, or `gcs` | -| `--admin-role` | - | No | ACCOUNTADMIN | Admin role to cache in manifest | -| `--manifest-path` | - | No | `.sfutils/manifest.toml` | Path to manifest.toml | - -**`--suggest` JSON fields:** - -| Field | Description | -|-------|-------------| -| `ready` | True if all required CSP CLI tools are on PATH | -| `csp_tools_ready` | All required CSP CLI executables for `--provider` are on `PATH` | -| `csp_cli_tools` | List of `{ "provider", "tool", "available" }` for that provider | -| `supported_storage_providers` | Backends implemented in sfutils today (e.g. `S3`) | -| `planned_storage_providers` | Roadmap backends not yet wired in the skill/CLI | -| `csp_credential_env` | List of `{ "name", "set" }` for watched credential-related vars (never values) | -| `csp_credential_env_signal` | True if **any one** OR-branch matched for that provider | -| `csp_credential_env_satisfied_by` | Which branch matched first (`static_keys`, `profile`, `web_identity`, etc.) or `null` | -| `credential_env_note` | Neutral hint when no env signal; `null` when signal is true | - -For AWS, **one** satisfied branch is enough — do not treat unset watched vars as required. - ---- - -### sfutils-extvolumes CLI - -**Description:** Creates and manages Snowflake external volumes. - -> **Note:** The CLI currently supports **AWS S3 only**. Azure Blob Storage and GCS support are planned. - -**🔴 COMMAND NAMES (exact — do NOT substitute):** - -- `create` — NOT "setup", "make", "provision", "init" -- `delete` — NOT "remove", "destroy", "cleanup", "drop" -- `verify` — NOT "check", "test", "validate", "ping" -- `describe` — NOT "show", "get", "info", "status" -- `update-trust` — NOT "sync-trust", "refresh-trust" -- `setup-connection` — NOT "configure", "set-connection", "init-connection" -- `validate-manifest` — NOT "check-manifest", "verify-manifest" -- `list` — NOT "ls", "show-volumes" -- `migrate` — NOT "import", "convert", "upgrade" - -**🔴 OPTION NAMES (NEVER guess or invent options):** - -> ONLY use options listed in the tables below. -> If a command fails with "No such option", run `/vol --help` and use ONLY those options. - -**Global Options (pass BEFORE the command):** - -| Option | Short | Env Var | Default | Description | -|--------|-------|---------|---------|-------------| -| `--region` | `-r` | `AWS_REGION` | `us-west-2` | AWS region | -| `--prefix` | `-p` | - | current username | Prefix for AWS resources | -| `--no-prefix` | - | - | false | Disable username prefix | -| `--verbose` | `-v` | - | false | Enable verbose output | -| `--debug` | - | - | false | Enable debug output (shows SQL) | -| `--comment` | `-c` | - | auto | Comment for external volume | -| `--manifest-path` | - | - | `.sfutils/manifest.toml` | Path to manifest.toml | - ---- - -### `create` - -| Option | Short | Env Var | Required | Default | Description | -|--------|-------|---------|----------|---------|-------------| -| `--bucket` | `-b` | - | Yes | - | S3 bucket base name | -| `--role-name` | - | - | No | `{prefix}-{bucket}-snowflake-role` | IAM role name | -| `--policy-name` | - | - | No | `{prefix}-{bucket}-snowflake-policy` | IAM policy name | -| `--volume-name` | - | - | No | `{PREFIX}_{BUCKET}_EXTERNAL_VOLUME` | Snowflake external volume name | -| `--storage-location-name` | - | - | No | `{prefix}-{bucket}-s3-{region}` | Storage location name | -| `--external-id` | - | - | No | auto-generated | External ID for trust relationship | -| `--aws-profile` | - | `AWS_PROFILE` | No | - | AWS profile name for boto3 session | -| `--no-writes` | - | - | No | false | Create read-only external volume | -| `--skip-verify` | - | - | No | false | Skip external volume verification | -| `--dry-run` | - | - | No | false | Preview what would be created | -| `--force` | `-f` | - | No | false | Overwrite existing volume (CREATE OR REPLACE) | -| `--yes` | `-y` | - | No | false | Accepted for scripting compatibility (no interactive prompt on create) | - ---- - -### `delete` - -| Option | Short | Env Var | Required | Default | Description | -|--------|-------|---------|----------|---------|-------------| -| `--bucket` | `-b` | - | Yes | - | S3 bucket base name | -| `--role-name` | - | - | No | `{prefix}-{bucket}-snowflake-role` | IAM role name | -| `--policy-name` | - | - | No | `{prefix}-{bucket}-snowflake-policy` | IAM policy name | -| `--volume-name` | - | - | No | `{PREFIX}_{BUCKET}_EXTERNAL_VOLUME` | Snowflake volume name | -| `--delete-bucket` | - | - | No | false | Also delete the S3 bucket | -| `--force` | - | - | No | false | Force delete bucket even if not empty | -| `--yes` | `-y` | - | No | false | Skip confirmation prompt | -| `--output` | `-o` | - | No | `text` | Output format: `text` or `json` | - -After delete, the CLI marks the volume `status = "REMOVED"` in `manifest.toml`. - ---- - -### `verify` - -| Option | Short | Env Var | Required | Default | Description | -|--------|-------|---------|----------|---------|-------------| -| `--volume-name` | `-v` | - | Yes | - | Snowflake external volume name | -| `--retry` | - | - | No | false | Retry with exponential backoff on failure (for IAM propagation lag) | - ---- - -### `describe` - -| Option | Short | Env Var | Required | Default | Description | -|--------|-------|---------|----------|---------|-------------| -| `--volume-name` | `-v` | - | Yes | - | Snowflake external volume name | - ---- - -### `update-trust` - -| Option | Short | Env Var | Required | Default | Description | -|--------|-------|---------|----------|---------|-------------| -| `--bucket` | `-b` | - | No | - | S3 bucket base name (to derive role/volume names) | -| `--role-name` | `-r` | - | No | - | IAM role name to update | -| `--volume-name` | `-v` | - | No | - | Snowflake external volume name | - -> At least `--bucket` or both `--role-name` and `--volume-name` must be provided. - ---- - -### `setup-connection` - -Test a Snowflake connection and cache it to `manifest.toml`. - -```bash -/vol setup-connection -c -/vol setup-connection -c local-oauth --admin-role ACCOUNTADMIN -``` - -| Option | Short | Required | Default | Description | -|--------|-------|----------|---------|-------------| -| `--connection` | `-c` | Yes | - | Snowflake connection name (from `snow connection list`) | -| `--admin-role` | - | No | ACCOUNTADMIN | Admin role to cache in manifest | - -Writes `[snowflake].connection`, `account`, `user`, `account_url`, and `admin_role` to `manifest.toml`. - ---- - -### `validate-manifest` - -Validate `manifest.toml` structure and report issues. - -```bash -/vol validate-manifest -/vol validate-manifest --fix -``` - -| Option | Short | Required | Default | Description | -|--------|-------|----------|---------|-------------| -| `--fix` | - | No | false | Auto-repair structural gaps (adds missing sections with defaults) | - -Exit code 0 = valid, 1 = issues found. - ---- - -### `list` - -List all external volumes recorded in `manifest.toml`. - -```bash -/vol list -``` - -Output columns: `LABEL` / `VOLUME_NAME` / `TYPE` / `STATUS` - -No options. - ---- - -### `migrate` - -Migrate legacy `sfutils-manifest.md` + `.env` to `manifest.toml`. - -```bash -/vol migrate --dry-run -/vol migrate -``` - -| Option | Short | Required | Default | Description | -|--------|-------|----------|---------|-------------| -| `--env-path` | - | No | `.env` | `.env` file to read connection info from | -| `--manifest-md` | - | No | `.sfutils/sfutils-manifest.md` | Legacy markdown manifest to read | -| `--dry-run` | - | No | false | Show what would be written without writing | - -- Reads `sfutils-manifest.md` as **primary source** (volume_name, bucket_url, status, admin_role) -- Reads `.env` for connection info only (`SNOWFLAKE_DEFAULT_CONNECTION_NAME`, etc.) -- Always sets `infra_ready = false` — run `vol check-setup` afterwards -- Status defaults to `REMOVED` when unresolvable from markdown - ---- - -## Correct Command Structure - -```bash -vol [GLOBAL OPTIONS] [COMMAND OPTIONS] - -# Examples -vol --region us-east-1 create --bucket iceberg-data --dry-run -vol --no-prefix create --bucket iceberg-data --aws-profile prod -vol delete --bucket iceberg-data --yes -vol verify --volume-name MY_EXTERNAL_VOLUME -vol setup-connection -c local-oauth -vol validate-manifest --fix -vol list -vol migrate --dry-run -``` diff --git a/skills/external-volume/manifest-flows.md b/skills/external-volume/manifest-flows.md deleted file mode 100644 index e5a4666a..00000000 --- a/skills/external-volume/manifest-flows.md +++ /dev/null @@ -1,228 +0,0 @@ -## TOML Manifest Flow (manifest.toml — current) - -### Progressive Volume Write - -After successful `vol create`, the CLI automatically writes to `.sfutils/manifest.toml`. -The entry looks like this: - -```toml -# Machine-managed by Cortex Code. Do not hand-edit. -schema_version = "1" -project_name = "my-demo" -created_at = "2026-05-02T10:00:00Z" - -# ── Shared Snowflake connection (captured once, reused by all volumes) ──────── -[snowflake] -connection = "local-oauth" -account = "ABC12345" -user = "KAMESHS" -account_url = "https://abc12345.snowflakecomputing.com" -admin_role = "ACCOUNTADMIN" - -# ── Tool / infra pre-flight cache ───────────────────────────────────────────── -[prereqs] -tools_verified = "2026-05-02" -infra_ready = true - -# ── Volume: my-s3-volume ────────────────────────────────────────────────────── -[volume.my-s3-volume] -status = "COMPLETE" -created_at = "2026-05-02T10:15:00Z" -updated_at = "2026-05-02T10:15:00Z" -volume_name = "MY_S3_VOLUME" -storage_type = "s3" -bucket_url = "s3://my-bucket/prefix" -aws_region = "us-east-1" -aws_profile = "default" -storage_aws_role_arn = "arn:aws:iam::123456789012:role/MySnowflakeRole" -external_id = "abc123xyz" -admin_role = "ACCOUNTADMIN" - -[volume.my-s3-volume.cleanup] -volume_name = "MY_S3_VOLUME" -``` - -**TOML key derivation:** `volume_name.lower().replace("_", "-")` -Example: `MY_S3_VOLUME` → `my-s3-volume` - -**No `db` field in `[cleanup]`** — external volumes are account-level objects; cleanup is -S3 bucket + IAM role/policy + `DROP EXTERNAL VOLUME` only. - ---- - -### Multiple Volumes - -Each additional volume gets its own `[volume.