diff --git a/.claude/qc-judge/config.json b/.claude/qc-judge/config.json index 6c68c6491..5944866df 100644 --- a/.claude/qc-judge/config.json +++ b/.claude/qc-judge/config.json @@ -1,5 +1,5 @@ { - "release_notes_path": "docs/docs/beta-release-notes.mdx", + "release_notes_path": "docs/docs/release-notes.mdx", "internals_extra_patterns": [ "wave\\s*\\d+", "@databricks\\.com", @@ -30,6 +30,24 @@ "expect_exit": 0, "timeout_seconds": 30 }, + "param-name-parity": { + "name": "Parameter name consistency and arity parity across tiers", + "type": "command", + "severity": "warn", + "enabled": true, + "cmd": "[ -f docs/scripts/check-param-names.py ] || { echo 'param-names script absent; skip'; exit 0; }; python3 docs/scripts/check-param-names.py", + "expect_exit": 0, + "timeout_seconds": 30 + }, + "docs-examples": { + "name": "Docs example output tables well-formed + all declared tabs complete", + "type": "command", + "severity": "warn", + "enabled": true, + "cmd": "[ -f docs/scripts/check-docs-examples.py ] || { echo 'absent; skip'; exit 0; }; python3 docs/scripts/check-docs-examples.py", + "expect_exit": 0, + "timeout_seconds": 30 + }, "doc-coverage": { "name": "Every registered function documented + no placeholder example outputs", "type": "command", diff --git a/.github/CODECOV.md b/.github/CODECOV.md deleted file mode 100644 index b8e083e83..000000000 --- a/.github/CODECOV.md +++ /dev/null @@ -1,32 +0,0 @@ -# Codecov badge and coverage uploads - -## Badge (shield) - -The README Codecov badge is: - -- **Badge image:** `https://codecov.io/gh/databrickslabs/geobrix/branch/main/graph/badge.svg` -- **Project page:** https://codecov.io/gh/databrickslabs/geobrix - -The badge updates when Codecov receives a new coverage upload for this repo. This repo is public, so the default badge works. - -**Private repos (e.g. forks):** If the README badge does not update after a successful upload, the repo may be private. Codecov’s default badge is for public repos; for private repos you may need to install the [Codecov GitHub App](https://docs.codecov.com/docs/github-app) or use a [token in the badge URL](https://docs.codecov.com/docs/adding-the-codecov-badge). The upload itself can still succeed; only the public badge display may be restricted. - -## How coverage gets uploaded (tests run once) - -1. **build main** (on push/PR): Runs Scala and Python tests **once** with coverage on the `larger-runners` runner (the heavy build job), uploads coverage as workflow artifacts, then a separate **codecov** job (on the lighter `databrickslabs-protected-runner-group`) downloads those artifacts and uploads to Codecov. So the slow Codecov step does not block the build job, and tests are never run twice. -2. **Upload coverage to Codecov** (manual): Actions → “Upload coverage to Codecov” → Run workflow. Use only to refresh the badge without a new push; it runs the full test suite again. - -Both require the repo secret **CODECOV_TOKEN**. If you see **"Token required - not valid tokenless upload"** in the Codecov step, the secret is missing or empty. - -**To fix:** In [Codecov](https://codecov.io), open this repo → Settings → General → copy the **Upload token** (or use the GitHub App). In GitHub: repo **Settings → Secrets and variables → Actions → New repository secret** → name `CODECOV_TOKEN`, value = the Codecov token. Save. Re-run the workflow or push a commit. - -## Report paths (where coverage is produced and uploaded) - -| Source | Path(s) | -|---------------|--------| -| Scala (scoverage) | `target/scoverage.xml`, `target/scoverage-report/scoverage.xml` | -| Python (pytest-cov) | `python/geobrix/coverage.xml` | - -The build job stages files into `coverage-reports/`; the codecov job downloads that directory and passes it to Codecov (`directory: coverage-reports`). - -If the badge does not update: (1) Confirm `CODECOV_TOKEN` is set (see above); (2) Confirm the "Upload to Codecov" job succeeded (no "Token required" error); (3) For private repos, the badge may still need the [Codecov GitHub App](https://docs.codecov.com/docs/github-app) or a [badge token](https://docs.codecov.com/docs/adding-the-codecov-badge). diff --git a/.github/actions/pyrx_build/action.yml b/.github/actions/pyrx_build/action.yml index 1c2a76576..29cd2722c 100644 --- a/.github/actions/pyrx_build/action.yml +++ b/.github/actions/pyrx_build/action.yml @@ -40,10 +40,10 @@ runs: # Hash-pinned supply chain (matches requirements-ci.txt / dev-container # discipline; see docs/docs/security.mdx): every wheel is sha256-verified, # so a compromised mirror serving same-version-different-bytes fails closed. - # Regenerate via: uv pip compile --generate-hashes python/geobrix/requirements-pyrx-ci.in + # Regenerate via: uv pip compile --generate-hashes python/geobrix/requirements-light-ci.in # No GDAL exception here — rasterio's bundled-GDAL binary wheel is pinned # in the lock too (the lightweight API installs no native GDAL, no JAR). - pip install --require-hashes -r python/geobrix/requirements-pyrx-ci.txt + pip install --require-hashes -r python/geobrix/requirements-light-ci.txt # Install the in-repo package itself with --no-deps; every dependency is # already in the hash-pinned closure above. pip install --no-deps python/geobrix @@ -68,4 +68,4 @@ runs: # bench (bench-harness spec/compare/fingerprint/runner unit tests -- guards # the light bench FnSpec registry + cross-tier count invariants). # See test/conftest.py for the maintained condition. - pytest test/pyrx test/ds test/pyvx test/pygx test/pmtiles_light test/stac test/earthdata test/vizx test/sample test/bench -m "not integration" -v + pytest test/pyrx test/ds test/pyvx test/pygx test/pmtiles_light test/stac test/earthdata test/vizx test/sample test/bench test/core -m "not integration" -v diff --git a/.github/actions/python_build/action.yml b/.github/actions/python_build/action.yml index ee885ea68..2221c114c 100644 --- a/.github/actions/python_build/action.yml +++ b/.github/actions/python_build/action.yml @@ -6,7 +6,7 @@ inputs: description: 'Run Python lint (isort, black, flake8) and fail build on errors? true or false. Typically true only for PRs targeting main.' default: 'false' enable_coverage: - description: 'Run pytest with coverage (--cov) and emit coverage.xml for Codecov? true or false.' + description: 'Run pytest with coverage (--cov) and emit coverage.xml? true or false.' default: 'false' runs: using: "composite" @@ -117,14 +117,14 @@ runs: # Heavyweight must not test lightweight. The lightweight tiers (pyrx, pyvx, # pygx, ds, pmtiles_light, stac, earthdata, vizx, sample) need # rasterio/shapely/pandas/pyarrow etc. that are NOT in requirements-ci.txt; - # they run in their own 'pyrx' CI job against requirements-pyrx-ci.txt. + # they run in their own 'pyrx' CI job against requirements-light-ci.txt. # test/conftest.py collect_ignore already skips these when rasterio is # absent, but we ALSO pass --ignore explicitly so a light dir can never run # in the heavy phase even if the env drifts (gains rasterio) or a caller # invokes a light dir directly. This list MUST stay in sync with # _LIGHT_TEST_DIRS in test/conftest.py. # pytest-cov is already locked in requirements-ci.txt; no extra install needed. - LIGHT_IGNORES="--ignore=test/pyrx --ignore=test/pyvx --ignore=test/pygx --ignore=test/ds --ignore=test/pmtiles_light --ignore=test/stac --ignore=test/earthdata --ignore=test/vizx --ignore=test/sample --ignore=test/bench" + LIGHT_IGNORES="--ignore=test/pyrx --ignore=test/pyvx --ignore=test/pygx --ignore=test/ds --ignore=test/pmtiles_light --ignore=test/stac --ignore=test/earthdata --ignore=test/vizx --ignore=test/sample --ignore=test/bench --ignore=test/core" if [ "${{ inputs.enable_coverage }}" = "true" ]; then pytest -m "not integration" $LIGHT_IGNORES --cov=src --cov-report=xml --cov-report=term --cov-config=pyproject.toml -q else diff --git a/.github/docs/ci-runner-and-cache.md b/.github/docs/ci-runner-and-cache.md index 1cc1b4d6f..921a836a6 100644 --- a/.github/docs/ci-runner-and-cache.md +++ b/.github/docs/ci-runner-and-cache.md @@ -20,7 +20,7 @@ runs-on: Currently used by: `build_main` build, `build_python` build, `build_scala` build, `build_scala_by_package` test-package, `codecov-scala-parallel` -coverage-package, `codecov-upload` coverage, `codeql-analysis` analyze. +coverage-package, `codeql-analysis` analyze. ### Light jobs — `databrickslabs-protected-runner-group` / `linux-ubuntu-latest` @@ -33,7 +33,7 @@ runs-on: labels: linux-ubuntu-latest ``` -Currently used by: `build_main` update-doc-inventory + codecov, +Currently used by: `build_main` update-doc-inventory, `codecov-scala-parallel` merge-and-upload, `verify-maven-pgp` verify, `deploy-docs` build + deploy. diff --git a/.github/workflows/build_main.yml b/.github/workflows/build_main.yml index 06d3bebbb..b5a2e2511 100644 --- a/.github/workflows/build_main.yml +++ b/.github/workflows/build_main.yml @@ -112,7 +112,7 @@ jobs: - name: build scala uses: ./.github/actions/scala_build with: - # Disable scoverage in main build to keep CI ~10–15 min (full Scala coverage is ~30+ min). Run "Upload coverage to Codecov" workflow manually for Scala+Python coverage. + # Disable scoverage in main build to keep CI ~10–15 min (full Scala coverage is ~30+ min). Run coverage locally via the gbx:coverage:* commands. enable_coverage: "false" fail_on_scalastyle: ${{ github.event_name == 'pull_request' && github.base_ref == 'main' }} - name: build python @@ -130,44 +130,6 @@ jobs: run: | mkdir -p "$GBX_SAMPLE_DATA_ROOT" sudo -E mvn -C -q test -Dsuites='tests.docs.scala.*,docs.tests.scala.*' -DfailIfNoTests=false -Dscoverage.skip - - name: Stage coverage for upload job - run: | - mkdir -p coverage-reports - cp -f target/scoverage.xml coverage-reports/ 2>/dev/null || true - cp -f target/scoverage-report/scoverage.xml coverage-reports/scoverage-report.xml 2>/dev/null || true - cp -f python/geobrix/coverage.xml coverage-reports/ 2>/dev/null || true - if [ -z "$(ls -A coverage-reports 2>/dev/null)" ]; then - echo "No coverage files found. Expected Scala: target/scoverage.xml or target/scoverage-report/scoverage.xml; Python: python/geobrix/coverage.xml" - exit 1 - fi - ls -la coverage-reports/ - - name: Upload coverage artifacts - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 - with: - name: coverage-reports - path: coverage-reports - if-no-files-found: warn - name: upload artifacts uses: ./.github/actions/upload_artifacts - codecov: - name: Upload to Codecov - runs-on: - group: databrickslabs-protected-runner-group - labels: linux-ubuntu-latest - needs: build - if: always() && needs.build.result == 'success' - permissions: - contents: read - steps: - - name: Download coverage artifacts - uses: actions/download-artifact@70fc10c6e5e1ce46ad2ea6f2b72d43f7d47b13c3 # v8.0.0 - with: - name: coverage-reports - path: coverage-reports - - name: Upload to Codecov - uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2 - with: - token: ${{ secrets.CODECOV_TOKEN }} - directory: coverage-reports - fail_ci_if_error: false diff --git a/.github/workflows/build_scala.yml b/.github/workflows/build_scala.yml index 7b6d73088..2170a3f1c 100644 --- a/.github/workflows/build_scala.yml +++ b/.github/workflows/build_scala.yml @@ -45,9 +45,3 @@ jobs: enable_coverage: "true" - name: upload artifacts uses: ./.github/actions/upload_artifacts - - name: Publish test coverage to Codecov - uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2 - with: - token: ${{ secrets.CODECOV_TOKEN }} - files: target/scoverage.xml,target/scoverage-report/scoverage.xml - fail_ci_if_error: false diff --git a/.github/workflows/codecov-scala-parallel.yml b/.github/workflows/codecov-scala-parallel.yml index db10c18b8..09875aa8d 100644 --- a/.github/workflows/codecov-scala-parallel.yml +++ b/.github/workflows/codecov-scala-parallel.yml @@ -1,8 +1,8 @@ -# Optional: Run Scala coverage by package in parallel, merge, then upload to Codecov. +# Optional: Run Scala coverage by package in parallel and merge into one report. # Faster wall-clock time than a single job running all tests (~10–15 min vs ~30–45 min). # Trigger: workflow_dispatch only (or add schedule/on push if desired). +# The merged scoverage.xml is published as a workflow artifact (scoverage-merged). # -# Requires: CODECOV_TOKEN secret. # Merge script: scripts/ci/merge_scoverage.py # Design: .github/docs/scala-coverage-parallel-design.md @@ -108,9 +108,9 @@ jobs: echo "==============================================================================" fi - - name: Upload to Codecov - uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2 + - name: Upload merged coverage artifact + uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 with: - token: ${{ secrets.CODECOV_TOKEN }} - files: merged/scoverage.xml - fail_ci_if_error: false + name: scoverage-merged + path: merged/scoverage.xml + if-no-files-found: warn diff --git a/.github/workflows/codecov-upload.yml b/.github/workflows/codecov-upload.yml deleted file mode 100644 index ad77589bd..000000000 --- a/.github/workflows/codecov-upload.yml +++ /dev/null @@ -1,75 +0,0 @@ -# Manual workflow: run tests with coverage and upload to Codecov. -# Use only when you need to refresh the badge without a new push; build_main already -# runs tests once and uploads to Codecov in a separate job (no double test run). -# -# Trigger: Actions → "Upload coverage to Codecov" → Run workflow. -# -# Requirements: -# - Repo secret CODECOV_TOKEN (from https://codecov.io/gh/databrickslabs/geobrix → Settings). -# -# Badge: README uses https://codecov.io/gh/databrickslabs/geobrix/branch/main/graph/badge.svg - -name: Upload coverage to Codecov - -on: - workflow_dispatch: {} - -permissions: - contents: read - -jobs: - coverage: - name: Build, test with coverage, upload - # Heavy: full Scala scoverage + Python pytest --cov + GDAL native install (manual coverage-refresh path). Same workload as build_main, so same runner class. - runs-on: - group: larger-runners - labels: larger - # Checkout uses REPO_ACCESS_TOKEN (non-exempt secret), so gate behind the protected env. - environment: runtime - permissions: - contents: read - # Required by .github/actions/jfrog-auth: GitHub OIDC token exchange for pip/Maven via JFrog. - id-token: write - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - strategy: - matrix: - python: [ 3.12.3 ] - pytest: [8.4.2] - numpy: [ 2.1.3 ] - gdal: [ 3.11.4 ] - spark: [ 4.0.0 ] - steps: - - name: Checkout code - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - with: - token: ${{ secrets.REPO_ACCESS_TOKEN || secrets.GITHUB_TOKEN }} - - - name: Create pip cache key file - run: | - echo "${{ github.ref }}-${{ matrix.python }}-${{ matrix.numpy }}-${{ matrix.spark }}-${{ matrix.gdal }}" > .ci-pip-cache-key - - - name: Cache apt packages - uses: actions/cache@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3 - with: - path: .cache/apt-archives - key: apt-${{ runner.os }}-${{ hashFiles('.github/actions/scala_build/action.yml', '.github/actions/python_build/action.yml') }} - - - name: Build Scala (with coverage) - uses: ./.github/actions/scala_build - with: - enable_coverage: "true" - fail_on_scalastyle: "false" - - - name: Build Python (with coverage) - uses: ./.github/actions/python_build - with: - run_lint: "false" - enable_coverage: "true" - - - name: Upload coverage to Codecov - uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2 - with: - token: ${{ secrets.CODECOV_TOKEN }} - files: target/scoverage.xml,target/scoverage-report/scoverage.xml,python/geobrix/coverage.xml - fail_ci_if_error: false diff --git a/.github/workflows/deploy-docs.yml b/.github/workflows/deploy-docs.yml index 5fa19b775..ba70223d8 100644 --- a/.github/workflows/deploy-docs.yml +++ b/.github/workflows/deploy-docs.yml @@ -57,8 +57,19 @@ jobs: - name: Install dependencies run: cd docs && npm ci - - name: Build - run: cd docs && npm run build + - name: Build (strict — fail on broken links) + run: | + cd docs + set -o pipefail + npm run build 2>&1 | tee /tmp/docs-build.log + # docusaurus.config.js keeps onBrokenLinks='warn' (permissive for local + # dev / gbx:docs:dev), so `npm run build` exits 0 even with broken + # internal links. This CI gate greps the build output for Docusaurus's + # broken-link report and fails explicitly — the green validation step. + if grep -qiE "found broken links|broken links found|Broken link on source" /tmp/docs-build.log; then + echo "::error::Docusaurus reported broken links — failing the strict docs gate (see the build log above)." + exit 1 + fi env: NODE_ENV: production # Force public GitHub Pages baseUrl (https://databrickslabs.github.io/geobrix/) diff --git a/.gitignore b/.gitignore index dcc0b2c30..335809ff8 100644 --- a/.gitignore +++ b/.gitignore @@ -53,3 +53,17 @@ test-logs/bench/ # Lakeflow DAB local override files (real per-workspace values; template committed as *.override.yml.example) *.override.yml .databricks/ + +# Agent/session scratch + local tooling (never committed). All internal planning +# (specs, plans, prompts, input, SDD ledgers) is consolidated under /.superpowers/; +# the public representation of decisions lives in docs/docs/. The legacy root +# locations (/prompts/, /input/, /docs/superpowers/) stay ignored as guards. +/.superpowers/ +/docs/superpowers/ +/.isaac/ +/.tmp +/scratchpad/ +# pytest cache (appears in repo root and under docs/, python/, notebooks/, etc.) +.pytest_cache/ +# Locally-built docs bundle (only the LFS platform tarball in resources/static/ is committed) +/resources/static/geobrix-docs-*.zip diff --git a/CHANGELOG.md b/CHANGELOG.md index d3e3bee31..e2b613cd4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,10 @@ ## v0.2.0 -Version bump with notable fixes and improvements. See [Beta Release Notes](docs/docs/beta-release-notes.mdx) for API and naming changes. +Version bump with notable fixes and improvements. See [Release Notes](docs/docs/release-notes.mdx) for API and naming changes. ### Notable changes - BNG aggregators (`bng_cellunion_agg`, `bng_cellintersection_agg`): fixed shared aggregation buffer bug (fresh buffer per partition); chip field resolution by type/name in union agg. -- Reader renames and other API changes documented in Beta Release Notes. +- Reader renames and other API changes documented in Release Notes. --- diff --git a/CLAUDE.md b/CLAUDE.md index 2198dfe5e..be1a2e578 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ This file is the entry point for any Claude (or Cursor) session in this repo. Us ## Project -**GeoBrix** is a high-performance spatial processing library — a modern successor to [DBLabs Mosaic](https://databrickslabs.github.io/mosaic/), targeting Databricks Runtime (DBR 17.3 LTS or 18 LTS). Current version **0.4.3** (beta). APIs may break to stabilize, and there are **no function aliases** — one canonical name per function. See `docs/docs/beta-release-notes.mdx` for breaking changes. +**GeoBrix** is a high-performance spatial processing library — a modern successor to [DBLabs Mosaic](https://databrickslabs.github.io/mosaic/), targeting Databricks Runtime (DBR 17.3 LTS or 18 LTS). Current version **0.5.0** (beta). APIs may break to stabilize, and there are **no function aliases** — one canonical name per function. See `docs/docs/release-notes.mdx` for breaking changes. Heavy code is Scala/Spark (JAR); lightweight bindings are Python (wheel) and SQL, both wrapping the Scala columnar expressions via Spark Connect. @@ -15,10 +15,11 @@ Current branch: `beta/0.4.0`. Repo: `databrickslabs/geobrix`. These are the geobrix-specific translations of user-global preferences (`~/.claude/CLAUDE.md`): - **`gbx:*` commands are authoritative.** They are the canonical entry points for tests, coverage, docs, lint, Docker, data, CI, and security in this repo. If a `gbx:*` command doesn't do what you need, **fix the command** — don't work around it with ad-hoc shell, and don't paper over it by augmenting with extra inline logic. The "Adding or fixing a `gbx:*` command" section below has the procedure. The whole point of the palette is that everyone (you, me, future contributors, CI) runs the same code path. -- **Orchestrator-master + per-task subagents** — Never run a `gbx:*` command inline if it touches the docker container, Maven, or the doc-test suite. Dispatch a Task subagent with the full task text and let it handle the long-running work in isolation. Test suites often take minutes; running inline blocks the main session. +- **Orchestrator-master + per-task subagents** — Never run a `gbx:*` command inline if it touches the docker container, Maven, or the doc-test suite. Dispatch a Task subagent with the full task text and let it handle the long-running work in isolation. Test suites often take minutes; running inline blocks the main session. **Orient every subagent with the "Subagent orientation" section below** — an un-oriented subagent burns a run rediscovering repo basics, or reports a repo invariant as if it were a finding. +- **Check Databricks auth BEFORE dispatching, not after a browser tab appears.** The main agent owns auth readiness. Run `bash ~/.claude/hooks/databricks-auth-status.sh PreDispatch` (read-only, never opens a browser) before each dispatch block, and confirm the profiles the work needs are `VALID`. Most geobrix work is local (Docker/Maven/pytest/docs/git) and needs **no** profile — don't ask the user to re-auth a profile the work doesn't touch. Subagents must never fix auth; a `databricks auth login` is hook-blocked and only the user can run it. - **Skills first** — Useful for adjacent work: `databricks-query` for SQL against the workspace, `databricks-workspace-files` for browsing notebooks, `databricks-lakeview-dashboard` for visualization, `databricks-authentication` before any databricks operation. The Field Engineering skills (`fevm`, `sage-context-catalog`) are unrelated to geobrix and shouldn't be invoked here. - **Runtime judge** — Has already learned the common `gbx:*` scripts (`gbx-test-scala.sh`, `gbx-test-python.sh`, `gbx-docker-exec.sh`, etc.) from prior sessions. New patterns pay a 10-20s warmup; learned patterns are instant. Don't disable. -- **QC judge** — Project config at `.claude/qc-judge/config.json`. Wave-number regex (`wave\s*\d+`) blocks any user-facing doc that leaks the internal planning vocabulary (see "User-facing docs voice" below). `release_notes_path` points at `docs/docs/beta-release-notes.mdx` for the release-notes-current check. +- **QC judge** — Project config at `.claude/qc-judge/config.json`. Wave-number regex (`wave\s*\d+`) blocks any user-facing doc that leaks the internal planning vocabulary (see "User-facing docs voice" below). `release_notes_path` points at `docs/docs/release-notes.mdx` for the release-notes-current check. - **gh account switch** — `gh auth switch --user mjohns-databricks` before **any** push, PR creation, PR comment, or `gh api` write to `databrickslabs/geobrix`. The default `mjohns_data` returns 403 for write operations on this repo. - **Progress feedback on long-running ops** — Scala test suites, Maven builds, full doc tests, and coverage runs routinely take 1-10+ minutes. When you dispatch one of these, give the user a one-line progress update roughly every 30 seconds (tail the log, report the suite/file currently running). Don't go silent for minutes. @@ -57,6 +58,23 @@ Use `gbx:docker:start` / `gbx:docker:exec` rather than `docker run` directly. Th Default Maven profile is **`skipScoverage`** for fast compile/test (`mvn clean package -DskipTests`). Coverage commands explicitly trigger the `standard` profile. +## Running notebooks on Databricks (staging, install, runners) — READ BEFORE hand-rolling anything + +These facts have been painfully rediscovered by multiple agents. Follow them; don't reinvent. + +- **Use the canonical commands — do NOT hand-roll `jobs.submit`/`workspace.upload` drivers.** + - `gbx:test:notebooks-serverless` — imports a local `.ipynb` to the workspace, **strips `%pip`/`%restart_python` cells** (they fail in Serverless JOB compute), injects deps via the **Serverless environment spec** (`--extras`, `--wheel`, `--env-version`, `--profile`), submits via `jobs.submit`, polls. This sidesteps the whole notebook-install saga. + - `gbx:test:notebooks` — runs notebooks **cell-by-cell inside the `geobrix-dev` Docker container** (fully local, `/Volumes` mounted, no workspace). Best for a quick "does it render/run" check. + - If a command lacks a capability, **fix the command** (add an option) — don't write a one-off script. +- **Staging on dogfood** (a non-account-admin identity is assumed): wheels/data → the Volume **`/Volumes/geospatial_docs/geobrix/sample-data/`** via SDK `files.upload` (streaming). The configured `GBX_ARTIFACT_VOLUME` default (`…/gdal_artifacts/noble/geobrix`) **does not exist on dogfood** and returns a *misleading* `PermissionDenied: … not account admin` — that's a missing-schema error, not a real block. Notebooks → WSFS **`/Users//GeoBrix/`** (dogfood aggressively GCs old notebooks). **Import a notebook with `w.workspace.import_(path, format=ImportFormat.JUPYTER, content=base64(nb_bytes), overwrite=True)` after `w.workspace.mkdirs(parent)`** — this classic `/api/2.0/workspace/import` endpoint WORKS for a non-admin (it errors `ResourceDoesNotExist` if the parent folder is absent — hence the mkdirs). Do **NOT** use the `w.workspace.upload()` SDK mixin — *that* routes through the gated path and returns the misleading "not account admin" error (the mistake that made past sessions conclude "import is impossible"). `gbx:test:notebooks-serverless` already does the `import_` for you. `files.download`/`files.list` are gated — read job results via `jobs.get_run_output(task_run_id).notebook_output.result`, never `files.download`. +- **Notebook `%pip` install of the wheel** — ALWAYS `@ file:///Volumes/…/geobrix--py3-none-any.whl` **with the extra** (`light_dbr19` on dogfood incl. its serverless env v6; `light` on e2/oauth-fe serverless env v5 / classic DBR ≤18): + - **INTERACTIVE** (refresh a live session) — two steps: `--no-deps --force-reinstall "geobrix[EXTRA] @ file://…"` then the same line with no flags, then `restartPython()`. (`--force-reinstall` is needed to swap fresh bytes of an already-installed *same-version* wheel — `--no-cache-dir` alone won't; `--no-deps` is what keeps force-reinstall from touching pyspark/preinstalled deps.) + - **JOB** (non-interactive, fresh kernel) — a single **plain** install, NO flags. + - **Never `--force-reinstall` WITHOUT `--no-deps`** — that reinstalls pyspark/other preinstalled packages (serverless hard-fails `violate preinstalled package pyspark==…`; classic `Failure starting repl`). **Never a bare `geobrix[EXTRA]`** without `@ file://` — it resolves from PyPI and downgrades idna/protobuf → the kernel won't restart. +- **Don't auto-retry a failed job.** Report `result_state` / `state_message` / `run_page_url` and stop; a failure is almost always structural. +- **h3 mosaic rendering (`plot_mosaic`):** match `gridResolution` to the scene scale or the hexes render mostly empty (a 2 km scene at res-5 ≈ 8.5 km hexes → one ~1%-filled hex; res-8 ≈ 0.46 km edge suits a ~4 km scene; edge lengths res-6≈3.7 km, 7≈1.2 km, 8≈0.46 km, 9≈0.17 km). `plot_mosaic` is a **static** matplotlib render — pan/zoom is `plot_interactive`/`plot_cog`. +- **Sample raster data:** the usable one under `sample-data/Volumes/.../london/` is `sentinel2/london_sentinel2_red.tif` (388×385 @10 m, EPSG:32630); `elevation/srtm_n51w001.tif` is a degenerate 2×3-px placeholder. No NYC *raster*. For bigger/fresher scenes use `gbx:data:download`, the STAC light API, or the DEM-3DEP / NAIP / NASANEX / TROPOMI / EMIT downloaders. + ## Commands (the `gbx:*` palette) The repo has **50 `gbx:*` commands** in `scripts/commands/` (each is a `.md` registration + a `.sh` implementation). They handle Docker setup, env vars, log paths (`--log filename` → `test-logs/filename`), and profile selection. Originally registered for Cursor's command palette (hence the `.md` files), they're now invoked directly from any shell or via the Task tool. @@ -108,6 +126,7 @@ Only **integer indices ±1..±6** (1=100km, 2=10km, 3=1km, 4=100m, 5=10m, 6=1m; ### GDAL resource management +- **Serverless-safe materialize policy (REQUIRED):** any new code that reads a whole file or tile into executor RAM (a materialize) MUST route through `materialize_decision` in `ds/file_gbx.py` — never a raw unbounded `.read()` or `materialize_to_bytes` without it (Serverless per-task RAM ~1 GB; a mis-sized read silently OOMs). - **Prefer `rst_fromcontent` with `binaryFile` reader** over `rst_fromfile` when you already have bytes — avoids temp-file races on executors. - `GetNoDataValue` requires an output array (returns void otherwise). - `GetStatistics` only works on the MDArray, **not on `Band` directly**. @@ -135,6 +154,137 @@ Single-source pattern: doc SQL examples in `docs/tests/python/api/{rasterx,gridx - Run regeneration via `gbx:docs:function-info` or `gbx:test:function-info` (which also runs pytest). - Tests assert every function in `registered_functions.txt` has a non-empty example in `function-info.json`. If coverage fails, fix upstream — never add placeholder/empty usage. +#### Code examples are GENERATED — never hand-edit the JSON + +`function-info.json` is a **build artifact**. Hand-editing it works until the next +`gbx:docs:function-info`, which silently overwrites your change. To fix what +`DESCRIBE FUNCTION EXTENDED` prints, edit the **source**, then regenerate: + +| To change... | Edit this | Not this | +|---|---|---| +| the `Examples:` block | `docs/tests/python/api/*_functions_sql.py` → the function's `*_sql_example()` | ❌ `function-info.json` | +| `Usage:` / `Extended Usage:` | see "signature metadata" below | ❌ `function-info.json` | + +How the example is extracted (`docs/scripts/generate-function-info.py`) — these +mechanics surprise people, so check them before wondering why your text vanished: + +- Only the **first SQL statement** containing the package prefix is taken + (`first_statement_containing`). A second query in the same `*_sql_example()` is + ignored by `DESCRIBE FUNCTION` (it still renders in the docs page). +- `--` comments are **stripped**. Explanatory comments in the example never reach + `DESCRIBE FUNCTION`; put that prose in the description metadata instead. +- One example can fill **several** functions: every registered name appearing in the + statement inherits it, EXCEPT a name that has its own dedicated `*_sql_example()` + (so `gbx_st_asmvt` and `gbx_st_asmvt_pyramid` don't cross-contaminate). +- Keys beginning `_` (e.g. `_package_rasterx`) are section markers, not functions. + +#### Canonical `usageArgs` style + +`DESCRIBE FUNCTION` prints `name() - `, describing the **SQL** surface. + +- **Optional arguments use Style B: `[param]`** — brackets wrap only the parameter name, the + comma stays outside. `geom, attrs_struct, min_z, max_z, layer_name, [extent]`. Multiple + trailing optionals: `a, b, [c], [d]`. Do **not** use `geom, target_crs [, source_crs]` + (comma inside) — that form is being retired. +- **Parameter names are snake_case**, matching SQL — `geom`, `resolution`, `size_in_mb`. Not + the Scala camelCase (`geomWkb`, `cellId`) and not the internal `*Expr` field names. + **Exception — `cellid`/`cellid1`/`cellid2`**: bare cell-id parameters use the single + lowercase token `cellid` (not `cell_id`). This matches the chip-struct internal field, + Databricks product naming, and Mosaic convention, and is intentional. Chip-struct + parameters remain `left_chip`/`right_chip`/`input_chip` (snake_case, not affected by + this exception). +- An argument is optional exactly when `builder()` has a shorter `case N =>` branch that + injects a `Literal(...)` default. **34 functions** have optional args; rendering one as + required is a bug, not a style nit. +- Don't parse the docs `**Signature:**` lines as truth — 63 of 173 use camelCase and at least + one function has two conflicting lines. Validate against `builder()` arity instead. + +#### Signature metadata derivation (automated from Scala) + +As of v0.5.0, `usageArgs` and `description` are **derived from Scala case-class fields and builder +arity patterns**, not hand-maintained in `function-info.json`. This eliminates drift: parameter +names stay in sync with the actual Scala source, and optional parameter detection is validated +against real `builder()` branches. + +**How it works:** + +1. **`docs/scripts/extend-function-metadata.py`** (the parser): + - Reads all Scala expression files under `src/main/scala/com/databricks/labs/gbx/{rasterx,vectorx,gridx}`. + - For each function's case class, extracts field names and filters out internal state (e.g., `exprConfExpr`, aggregation buffer offsets). + - Strips the `Expr` suffix from each field and converts to snake_case. + - Inspects the `builder()` method: if `case N =>` and `case N+K =>` branches exist with `Literal(...)` defaults in the longer branch, marks args N+1…N+K as optional. + - Outputs parsed metadata as JSON. + +2. **`docs/scripts/generate-function-info.py`** (the generator): + - Calls the parser to fetch `usage_args` for each function. + - Merges parsed metadata into the JSON alongside examples (from `*_sql_example()` in docs). + - Writes `src/main/resources/com/databricks/labs/gbx/function-info.json`. + +3. **`WithExpressionInfo`** (the Scala consumer): + - `getUsageArgs()` and `getDescription()` prefer JSON values (via `FunctionInfoLoader.get(name)`). + - Fall back to Scala `usageArgs` / `description` overrides only if JSON is absent. + - This allows legacy Scala overrides to coexist with generated metadata during migration. + +**When adding or changing a function:** + +- Update the **Scala case class** field names and `builder()` arity — the parser feeds from there. +- Run `gbx:docs:function-info` to regenerate the JSON (no manual edits needed). +- No Scala `usageArgs` override is normally required (it is derived). `description` still is — see below. +- If you must override (e.g., a builder arity is too irregular to parse), add `override def usageArgs` or `override def description` in the companion — the JSON loader respects it as a fallback, and the no-regression check will hold the derived value to it. + +**Guardrails (these exist and are mutation-verified):** + +- The parser **fails loudly** — it raises `SystemExit` rather than warning, and + `generate-function-info.py` treats a parser failure as fatal instead of writing `{}`. A silent + fallback is how an optional argument got published as required. +- **No-regression check** — a derived `usage_args` is compared against every hand-written + `override def usageArgs`. Losing a bracket, or dropping a parameter the override listed, is a + hard failure. Verified by mutating the bracket logic: the check caught all 5 override-backed + functions and exited non-zero. +- **Multi-companion files are reported, not guessed.** When several companions share one SQL name + (`ST_TransformCrs` + `ST_TransformCrs3` both register `gbx_st_transformcrs`), the parser + describes the WIDEST case class so trailing optionals stay visible, and prints a note. +- Brace style must not matter: both `=> c.length match {` and `=> {` newline `c.length match {` + are in use and parse identically. + +Not yet wired: `gbx:test:function-info` does not assert usage coverage, and no lint checks bracket +syntax. `check-binding-parity.py` still compares **names only** — it cannot see a parameter list. + +Currently **177 of 180** registered functions have derived `usage_args`. The 3 without +(`gbx_rst_fromfile`, `gbx_st_legacyaswkb`, `gbx_pmtiles_agg`) have irregular shapes and are left +absent so the Scala fallback applies. **`description` is still empty for all 180** — `DESCRIBE +FUNCTION` currently renders `name(args) - ` with a trailing dash. Populating descriptions, and +resolving whether derived parameter names should be published while R1/N9 naming debt is open +(the parser faithfully emits `points_array` where the docs say `points_geom`), are deferred. + +#### Signature metadata (`usageArgs` / `description`) — a known drift area + +`Usage:` is assembled in `WithExpressionInfo` as `name(usageArgs) - description`. +Historically each companion overrode these, but the convention was dropped along the +way: for a long stretch only 8 of ~179 companions had them, so most functions printed +`gbx_rst_foo() - ` — empty parens, no description. Treat blank metadata as a bug, not +a default. See `.superpowers/prompts/refactoring/2026-08-06-describe-function-metadata-drift-inventory.md`. + +**A signature change must move every surface together.** Changing arity or a parameter's +meaning touches up to seven places, and the ones that fail *silently* are the dangerous +ones — SQL binds **positionally**, so a wrapper passing an arg the `builder()` doesn't +accept is discarded with no error (this is exactly how `rst_maketiles` advertised +`(tile, tileWidth, tileHeight)` while really taking `(tile, sizeInMB)` — callers set a +megabyte budget believing they set pixel dimensions): + +1. the expression case-class fields + `builder()` arity +2. the public Scala wrapper overloads in `/functions.scala` — **arg count must match `builder()`** +3. the heavy Python shim (`python/geobrix/src/databricks/labs/gbx//functions.py`) +4. the light Python binding (`.../pyrx|pyvx|pygx/functions.py`) + its registered UDF arity +5. the `**Signature:**` line in `docs/docs/api/*-functions.mdx` +6. the doc-test `*_sql_example()` (the generated example) — and its expected-output constant +7. signature metadata (`usageArgs`/`description`), then regenerate + +Cross-check before declaring done: wrapper arg count vs `builder()` accepted range, and +whether each wrapper param name still denotes the quantity of the field it lands on +positionally. `check-binding-parity.py` compares **names only** and cannot see parameter +lists, so none of this is caught by CI today. + ### Doc tests are the documentation source (single source of truth) Tests ARE the documentation source, not validators of it. Docs import code from tests via webpack raw-loader. @@ -156,10 +306,75 @@ Anything under `docs/docs/` is read by end users — release notes, package page | "the Wave 1 aggregator" | "the aggregator" or `gbx_st_asmvt` | | references to internal subagents or dispatch sequencing | reference behavior, not the process | -**Wave numbers** are legitimate only in: `prompts/features/*.md` (internal plans), dispatch prompts (internal), git commit messages (internal), `input/` scoping drafts (gitignored). +**Wave numbers** are legitimate only in: `.superpowers/prompts/features/*.md` (internal plans), dispatch prompts (internal), git commit messages (internal), `.superpowers/input/` scoping drafts (gitignored). Quick check before merging: `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/ 2>/dev/null` should print nothing. The QC judge enforces this automatically via the `internals-leak` check. +## Subagent orientation (paste the relevant parts into every dispatch) + +A subagent starts with no repo knowledge. Left un-oriented it will rediscover basics on +your budget, work around a `gbx:*` command instead of fixing it, or — worst — report a +**repo invariant as a finding**. Include the applicable items below in the dispatch prompt +itself; don't tell an agent to "go read CLAUDE.md" when you can hand it the slice. + +**Facts that are NOT findings.** Every one of these has been reported as a discovery by +some agent. State the relevant ones up front so the agent doesn't burn a run on them: + +- **The heavy tier needs a built, staged JAR.** `mvn ... -DskipTests` leaves `target/classes/` + but **no `*.jar`** unless `package` ran. If no JAR is present, heavy SQL registration + cannot work and integration/parity tests fail with mass `UNRESOLVED_ROUTINE`. That is a + missing build artifact, **not** a code defect — build/stage first, then test. +- **The light tier is pure Python and needs no JAR.** `pyrx`/`pyvx`/`pygx` never require the + JAR; the wheel is always JAR-less. +- **Both tiers register the same `gbx_*` SQL names** and the last registration wins. Function + metadata + builder are written to the registry as one atomic triple, so implementation and + metadata cannot desync. +- **SQL binds positionally.** Heavy expressions register as plain Catalyst expressions with no + named-argument support, so an extra wrapper argument is silently dropped rather than erroring. +- **Doc tests only run in Docker** (they need the full env + sample data under `/Volumes`). + Corpus tests skip unless the container was started with the sample-data mounts. +- **`.superpowers/` is gitignored** scratch — all internal planning (specs, plans, prompts, input, SDD ledgers); the public representation of decisions lives in `docs/docs/`. +- Non-EPSG / authority-less CRS may render as different-but-equivalent strings across tiers. + Parity means CRS-equivalence, not string equality. + +**Standing instructions for any implementation subagent:** + +1. Use `gbx:*` commands, never ad-hoc `docker`/`mvn`/`pytest`. If a command is broken, **fix + the command** and say how it broke — never route around it. +2. Run **only the affected suites**; a full run is the orchestrator's call. +3. Never run `databricks auth login` (hook-blocked) and never try to fix auth. +4. Don't commit unless explicitly told to. +5. **Verify before reporting.** Read the source behind every claim. Regex sweeps over Scala + produce false positives (`case Seq(...)`, `c.head`, overload chains that delegate) — mark + findings CONFIRMED vs SUSPECTED and quote real source, never paraphrase a signature from + memory. A fabricated parameter list is worse than no report. +6. If a precondition for a **scoped** check is missing (no JAR, no sample data, stale staged + artifact), emit **one clear line** — `PRECONDITION MISSING: ; not run` — and + stop that check. Do **not** report the consequence as a defect, do not silently substitute a + weaker test, and do **not** narrate a confusing half-state (e.g. "CANNOT VERIFY (no JAR)") + about a tier — either it was in scope (then it's a clean PRECONDITION-MISSING line) or it was + never in scope (then don't mention it at all). +7. Exclude build artifacts from every search: `docs/build-static-zip/`, + `docs/tests/coverage-report/`, `docs/tests/.pytest_cache/`, `target/`, `scripts/docker/m2/`, + `*.pyc`. A naive grep for a Scala symbol otherwise hits minified JS in the docs build. + +**Lead-agent responsibility (do NOT push this onto the subagent):** decide the tier/JAR +strategy *before* dispatching and state it in the prompt. Check `ls target/*.jar` yourself; a +pyrx/package-source-only change usually means there is **no fresh JAR**. Then the dispatch must +say, explicitly: which tiers to exercise, whether a staged JAR exists, and what to do if a +precondition is absent. When heavy verification is wanted but no fresh JAR is staged, either +(a) build+stage the JAR first, or (b) hand the subagent a **JAR-free isolation path** — e.g. +"register the pyrx UDF directly via `spark.udf.register(name, _pyrx_udf)`; do NOT call +`rasterx.register()` (it loads the JAR via `register_ds` and will wall you)." If neither is +possible, tell the subagent heavy is **out of scope** for this run. A subagent hitting a missing +precondition it was never briefed on is a dispatch failure, not a subagent failure. + +**Package-source changes need the unit suite, not just doc-tests.** A change to +`python/geobrix/src/.../{pyrx,pyvx,pygx}/functions.py` (or any package source) must be verified +with `gbx:test:pyrx` (etc.) on the affected `python/geobrix/test/**` files. Doc-tests exercise +the example surface, not the committed unit tests — a behavior change can leave the doc-tests +green while breaking `test/pyrx/*`. + ## Adding or fixing a `gbx:*` command When adding a new `gbx::` command (or fixing an existing one — don't work around failures, fix the command): @@ -177,12 +392,42 @@ When adding a new `gbx::` command (or fixing an existing one 4. **Make executable**: `chmod +x scripts/commands/gbx--.sh`. 5. **Fixing a broken command**: reproduce the failure, fix the script (or its `.md`), re-run to confirm, commit. Don't add fallback ad-hoc shell invocations elsewhere. +## Databricks authentication + +Work that touches a workspace (staging the wheel/JAR to a Volume, running Serverless jobs, `databricks-query`) needs a valid profile. **Never auto-select one** — pass `--profile ` explicitly and let the user choose. In Claude Code each Bash call is a separate shell, so `export DATABRICKS_CONFIG_PROFILE=…` on its own line does NOT carry to the next command; use `--profile`, or chain with `&&`. + +Profiles in `~/.databrickscfg` (check live status with `databricks auth profiles`): + +| Profile | Workspace | Use for | +|---|---|---| +| `oauth-fe` | `e2-demo-field-eng` | The usual one for geobrix — Volumes, jobs, warehouses | +| `logfood` | `adb-2548836972759138` (Azure) | Internal metrics/logfood queries | +| `oauth` | `fevm-serverless-stable-vqr02h` | FEVM serverless workspace | +| `genie-map-env` | `fevm-serverless-stable-genie-map` | Genie Map app workspace | +| `DEFAULT` | `e2-demo-field-eng` | PAT-based; prefer `oauth-fe` instead | + +**Why you get re-prompted, and what actually helps.** All the `oauth*` profiles use `auth_type = databricks-cli` — U2M OAuth. Access tokens last ~1 hour, but the CLI holds a **refresh token** and renews silently, so an expired access token is normal and not by itself a reason to log in again. Repeated browser prompts almost always mean one of: + +- **The refresh token itself expired** (idle too long for that workspace). Fix: `databricks auth login --host --profile ` for that ONE profile. Re-authenticating every profile is unnecessary. +- **A `DATABRICKS_HOST` / `DATABRICKS_TOKEN` env var is shadowing the profile** — these take precedence over `--profile` and silently bypass cached OAuth. Check with `env | grep -i databricks`. +- **Genuinely idle-aged credentials across many workspaces.** Only fix the profile you need. + +**Do not diagnose from `~/.databricks/token-cache.json`.** On macOS, CLI v1.10.0 keeps OAuth tokens in the **system keychain**; that JSON file is a stale leftover from an older CLI. Its timestamps do not update on login and reading them will tell you a profile is expired when it is actually valid. `databricks auth profiles` (the `Valid` column) plus a real call like `databricks current-user me --profile ` are the only trustworthy signals. + +Token lifetimes are workspace/account-level policy and are **not** configurable per-profile from the CLI, so there is no local setting that extends them. Diagnose before re-authenticating: `databricks auth profiles` shows `Valid YES/NO` per profile, and only the `NO` ones need attention. A `Valid NO` on a profile you aren't using is harmless — don't fix it preemptively. + +For unattended/CI work, U2M is the wrong credential: use an OAuth **M2M service principal** (client ID + secret, no browser). That's a separate identity, so it needs its own UC grants on the geobrix catalogs/Volumes/warehouses, and the secret belongs in a secrets manager or env var — never in `~/.databrickscfg` and never committed. Don't use PATs: they expire (~90 days) and are long-lived plaintext bearer secrets. + ## Session artifacts -Two locations, by artifact class: +All internal planning artifacts live under the **gitignored `.superpowers/` tree** — consolidated here to keep the project root uncluttered and internal planning out of the public repo. The public representation of decisions is `docs/docs/` (release notes, package pages), not the planning tree. By class: + +- **Design specs** (brainstorming-skill output, the `*-design.md` files) → `.superpowers/specs/YYYY-MM-DD--design.md`. +- **Implementation plans** (writing-plans-skill output) → `.superpowers/plans/YYYY-MM-DD-.md`. +- **Everything else** (session summaries, analyses, progress notes) → `.superpowers/prompts//YYYY-MM-DD-.md`. Categories include `features/`, `documentation/`, `refactoring/`, `testing/`, `bugfixes/`. +- **Scoping drafts / raw input** → `.superpowers/input/`. **SDD ledgers/workspaces** → `.superpowers/sdd//`. -- **Design specs and implementation plans** (the `superpowers` workflow outputs) live under `docs/superpowers/` — specs (brainstorming-skill output, the `*-design.md` files) under `docs/superpowers/specs/YYYY-MM-DD--design.md`, and plans (writing-plans-skill output) under `docs/superpowers/plans/YYYY-MM-DD-.md`. This tree is **version-controlled** — specs and plans are committed alongside the work they describe. -- **Everything else** (session summaries, analyses, progress notes, scoping drafts) goes under `prompts//YYYY-MM-DD-.md`. Categories include `features/`, `documentation/`, `refactoring/`, `testing/`, `bugfixes/`. **`/prompts/` is gitignored** — local scratch, not committed. +This **overrides the brainstorming/writing-plans skills' default `docs/superpowers/` location** — write specs/plans under `.superpowers/` instead. The whole tree is gitignored (kept locally across sessions, never committed). ## What used to live under `.cursor/` diff --git a/README.md b/README.md index 8d4abb2b4..ce93ccb68 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@ [![build](https://github.com/databrickslabs/geobrix/actions/workflows/build_main.yml/badge.svg)](https://github.com/databrickslabs/geobrix/actions/workflows/build_main.yml) -[![codecov](https://codecov.io/gh/databrickslabs/geobrix/branch/main/graph/badge.svg)](https://codecov.io/gh/databrickslabs/geobrix) [![documentation](https://img.shields.io/badge/docs-latest-brightgreen.svg)](https://databrickslabs.github.io/geobrix/) [![scala](https://img.shields.io/badge/scala-2.13-red.svg)](https://www.scala-lang.org/) [![python](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/) @@ -16,10 +15,10 @@ python/geobrix/src/databricks/labs/gbx/vizx/__init__.py __all__, and is excluded from the total. Update these badges if functions are added or removed. --> -![Functions](https://img.shields.io/badge/functions-174-2e7d32) -![RasterX](https://img.shields.io/badge/RasterX-126-1565c0) +![Functions](https://img.shields.io/badge/functions-180-2e7d32) +![RasterX](https://img.shields.io/badge/RasterX-129-1565c0) ![GridX](https://img.shields.io/badge/GridX-41-1565c0) -![VectorX](https://img.shields.io/badge/VectorX-6-1565c0) +![VectorX](https://img.shields.io/badge/VectorX-9-1565c0) ![VizX](https://img.shields.io/badge/VizX-19-6a1b9a) ![PMTiles](https://img.shields.io/badge/PMTiles-1-1565c0) @@ -52,12 +51,13 @@ GeoBrix supports both current Databricks Runtime LTS releases: |---|---|---|---|---|---|---|---| | **17.3 LTS** | 24.04 | 4.0.0 | 3.12.3 | 2.13.16 | 17 | **5+** (Py 3.12) | ✅ Supported | | **18 LTS** | 24.04 | 4.1.0 | 3.12.3 | 2.13.16 | 21 | **5+** (Py 3.12) | ✅ Supported | +| **19 LTS** | 26.04 | 4.2.0 | 3.12.3 | 2.13.16 | 21 | (n/a yet) | ✅ Supported (light tier) | -A **single wheel + single JAR** runs on both: Scala 2.13.16 matches both runtimes, the JAR is compiled to Java-17 bytecode so it loads on both JVMs, and Spark is a `provided` dependency. +A **single wheel + single JAR** runs on both 17.3 and 18 LTS: Scala 2.13.16 matches both runtimes, the JAR is compiled to Java-17 bytecode so it loads on both JVMs, and Spark is a `provided` dependency. The **Serverless env** column is the minimum Serverless environment version for the lightweight tier: **version 5+** provides Python 3.12, which the `[light]` dependencies require (Python ≥ 3.11). Older environment versions (Python 3.10) can't install `geobrix[light]`. Env v5 release notes: [AWS](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/five) · [Azure](https://learn.microsoft.com/azure/databricks/release-notes/serverless/environment-version/five) · [GCP](https://docs.databricks.com/gcp/en/release-notes/serverless/environment-version/five). -> **DBR 19 LTS is coming soon**, built on **Ubuntu 26.04**. The **lightweight** tier (pure-Python, rasterio's bundled GDAL) will be unaffected; the **heavyweight** tier's native GDAL/OGR libraries are compiled against the cluster OS, so they will need to be rebuilt for the new base image. +> **DBR 19 LTS** (Ubuntu 26.04) is now supported by the **lightweight** tier (✅ in the table above). The **heavyweight** tier's native GDAL/OGR libraries are compiled against the cluster OS, so heavyweight on DBR 19 requires a GDAL rebuild for the new base image — not yet available. ## Quick start (lightweight) diff --git a/docs/docs/_partials/_virtual-tile-overrides.mdx b/docs/docs/_partials/_virtual-tile-overrides.mdx new file mode 100644 index 000000000..17d00efa8 --- /dev/null +++ b/docs/docs/_partials/_virtual-tile-overrides.mdx @@ -0,0 +1,30 @@ +:::info Virtual-tile force-output params — lightweight tier only + +Tile-returning `rst_*` functions in the **lightweight tier** accept three optional keyword arguments +that control whether the produced tile carries raster bytes or a **bytes-free virtual reference** +(a path + window instead of in-memory pixels): + +| Argument | Type | Meaning | +|---|---|---| +| `virtualize_dir` | `str` (durable path) | Write the produced tile to `/[_].tif` and return a **virtual tile** (path + window, no bytes). The directory must be a durable, executor-readable location such as a Unity Catalog Volume path. Cannot be set together with `materialize=True`. | +| `virtualize_prefix` | `str` | Optional filename prefix added before the provenance-based filename — use this to deconflict when two different function outputs share the same `virtualize_dir`. | +| `materialize` | `bool` | `True` — ensure the produced tile carries bytes. `False` — no-op. Default (unset) — auto: **reference/passthrough** ops (header reads, `rst_clip`, `rst_setsrid`, identity `rst_transform`) belong to the no-new-pixels class and incur no pixel computation; **pixel-producing** ops (slope, focal, mapalgebra, reproject, merge, combineavg, frombands, …) materialize. | + +**Default (auto) behavior:** reference/passthrough operations (`rst_clip`, `rst_setsrid`, identity +`rst_transform`) produce no new pixels — they describe a region of or annotation on existing backing +data rather than computing fresh values, so they are the low-cost class. Pixel-producing operations +(slope, focal, mapalgebra, reproject, merge, combineavg, frombands, …) materialize and return bytes; +for these, `virtualize_dir` is the **only** way to get a virtual tile back — it writes the computed +result to a durable path and returns a light virtual row. + +**The lightweight tier is for light (virtual) raster tiles; the heavyweight tier is for heavy +(binary) raster tiles.** The heavyweight (`rasterx`) tier does not accept `virtualize_dir`, +`virtualize_prefix`, or `materialize`, and accepts only **materialized** tiles (raster bytes +present) as input — both v1 and v2 materialized tiles are accepted; the result is always the v2 +tile struct. Passing a **virtual** tile to a heavyweight function raises a clear error directing +you to materialize it first. For most raster work the lightweight tier covers the full function +set and is the recommended default. If you specifically need a heavyweight function, materialize +the tile first (`materialize=True`, or write via a writer, which always materializes) before +passing it. See [Virtual tiles and the light→heavy bridge](./execution-tiers#light-heavy-bridge) +for details. +::: diff --git a/docs/docs/advanced/library-integration.mdx b/docs/docs/advanced/library-integration.mdx index b8217319c..f4c7ec27d 100644 --- a/docs/docs/advanced/library-integration.mdx +++ b/docs/docs/advanced/library-integration.mdx @@ -53,7 +53,7 @@ Rasterio is typically available in Databricks ML runtimes, or install via: Convert GeoBrix raster tiles to rasterio datasets for processing: :::tip Understanding Tiles -GeoBrix tiles are structured types with three fields: `cellid`, `raster`, and `metadata`. The `raster` field contains either a file path (String) or binary content (Binary). See [Tile Structure](../api/tile-structure) for details. +GeoBrix tiles are structured types with eight fields (the v2 tile): `cellid`, `raster`, `path`, `window`, `clip_polygon`, `clip_crs`, `crs`, and `metadata`. `raster` holds the binary raster bytes (nullable — `null` on a virtual tile, where `path` + `window` locate the pixels to read lazily instead). See [Tile Structure](../api/tile-structure) for details. ::: **Environment:** Databricks · DBR 17.3 LTS · 1,000 × 1024² GeoTIFF tiles · whole-job > wall-clock median, one measured iteration (I/O, not JIT-sensitive — no warm-up). @@ -191,6 +212,45 @@ it is the trade for running where the heavyweight tier can't (Serverless, ARM, n init script). At scale the read still parallelizes across the cluster (~23 ms/tile here, spread over the workers). +### Reader plan-time listing + +For a comparison of the full read paths — Delta-scan FILE-column-table (~2 s, ~16–17× faster than directory on Serverless), directory enumeration (~10 s/10k files), and DataSource per-tile directory scan (~20 s classic / ~30 s Serverless per 1k tiles) — see the [capability tiers & read-options matrix](../readers-writers#capability-tiers) and [Serverless & Memory](../serverless-and-memory). The fast Serverless path is reading from a FILE-column table (Delta scan). + +Before any tile is read, the reader **enumerates the source directory** to determine tile windows. The default listing is fast — a stat-free walk that plans ~10k files in ~1.5 s on a UC Volume — and it runs once at plan time even when `virtualTiles=true` (only the per-file `rasterio.open` is skipped by virtual-tile planning; the directory walk still happens). + +Measured on a **10,002-file corpus** on a dogfood workspace (driver-side microbenchmark, +warm run): + +| Method | Median (s) | vs. baseline | +|---|---|---| +| Default directory listing (stat-free) | ~1.5 | — | +| Pre-computed manifest / tile table | **0.037** | **~37× faster** | + +The manifest path skips the directory walk and header opens entirely — planning reduces to a single file or table read regardless of tile count. Because the default listing is already fast, supplying a manifest or tile table is an **optional** optimization rather than a critical remedy; it is most useful when you already maintain a pre-computed tile index, or when the listing cost accumulates across a very large number of repeated job runs. + +:::tip Pre-computed tile inputs (`manifest` / tile table) +Supply a pre-computed tile list to skip the directory walk entirely: + +```python +# Option 1 — manifest (JSON or Parquet: path + window per tile) +df = spark.read.format("raster_gbx") \ + .option("manifest", "/Volumes/catalog/schema/vol/tiles.json") \ + .option("virtualTiles", "true") \ + .load("/Volumes/catalog/schema/vol/rasters") + +# Option 2 — tile INDEX in a Delta table: path (+ optional window) per tile — +# a saved file listing, NOT tile pixels; the reader still reads the files +df = spark.read.format("raster_gbx") \ + .option("tilesTable", "geospatial.myschema.tile_index") \ + .option("virtualTiles", "true") \ + .load("/") +``` + +Both options reduce planning to a single file or table read (**~37× faster** than the +directory walk at 10k files). The benefit is purely at plan time and is independent of +which functions run on the tiles. +::: + ### NetCDF readers The NetCDF reader benchmark has **two legs**, matching the two data shapes each reader supports: @@ -403,13 +463,117 @@ far higher: +### Virtual vs materialized input tiles (lightweight, 512²/4326) {#virtual-vs-materialized} + +Virtual tiles save I/O when operations are deferred — metadata reads and edits (set-CRS, set-SRID, nodata, band-select) complete from the header without pixels. Chain deferred edits before a materializing step to pay pixel I/O once, at the end. + +> **Environment:** Databricks · DBR 18.x · 20 workers · 512² 4-band `float32` tiles · 1,000 tiles per iteration. + +The result splits cleanly along the `Disposition`: + +- **`deferred` — virtual is ~1.5× faster on average.** The pending-instruction edits that carry no pixel work at all — `rst_setsrid`, `rst_setcrs`, `rst_initnodata`, `rst_band` — run **4.7–12.5× faster** on a virtual tile: it carries no pixel bytes to move across the cluster and never opens the raster, so the edit is almost free. +- **`materialized` — virtual is ~2× slower.** When every pixel is needed, the virtual tile is opened and read in full, so the `path` + `window` indirection is pure overhead on top of the same compute. +- **`na` — read the marker as "not a single-tile op," not "unaffected."** The grouped aggregators (multi-tile and geometry reducers) do sit at parity — the input form barely moves them. But the two byte constructors, `rst_fromcontent` and `rst_tryopen`, are also `na` (the marker only classifies a single input tile) *and yet consume the tile bytes*, so their timing still swings — `rst_fromcontent` 2.77× faster, `rst_tryopen` ~4× slower — driven by how the bytes reach them, not by a deferred/materialized decision. + +| Function | Light materialized/tile (ms) | Light virtual/tile (ms) | Disposition | Virtual speedup | +|---|---|---|---|---| +| `rst_initnodata` | 12.49 | 1.0 | deferred | 12.53× | +| `rst_setsrid` | 11.25 | 0.95 | deferred | 11.83× | +| `rst_setcrs` | 11.51 | 1.04 | deferred | 11.1× | +| `rst_band` | 4.77 | 1.0 | deferred | 4.74× | +| `rst_fromcontent` | 3.08 | 1.11 | na | 2.77× | +| `rst_scalex` | 3.24 | 1.75 | deferred | 1.85× | +| `rst_skewy` | 3.07 | 1.72 | deferred | 1.79× | +| `rst_rastertoworldcoordx` | 2.91 | 1.74 | deferred | 1.67× | +| `rst_crs` | 3.42 | 2.05 | deferred | 1.67× | +| `rst_upperlefty` | 3.18 | 1.91 | deferred | 1.66× | +| `rst_rastertoworldcoordy` | 3.64 | 2.23 | deferred | 1.63× | +| `rst_upperleftx` | 3.15 | 1.94 | deferred | 1.62× | +| `rst_rotation` | 2.9 | 1.88 | deferred | 1.54× | +| `rst_pixelheight` | 2.9 | 1.95 | deferred | 1.49× | +| `rst_pixelwidth` | 2.96 | 2.03 | deferred | 1.46× | +| `rst_scaley` | 2.99 | 2.07 | deferred | 1.44× | +| `rst_format` | 3.01 | 2.09 | deferred | 1.44× | +| `rst_numbands` | 3.51 | 2.52 | deferred | 1.39× | +| `rst_srid` | 2.92 | 2.22 | deferred | 1.32× | +| `rst_getnodata` | 2.85 | 2.25 | deferred | 1.27× | +| `rst_skewx` | 2.83 | 2.29 | deferred | 1.23× | +| `rst_height` | 3.44 | 3.09 | deferred | 1.11× | +| `rst_gridfrompoints_agg` | 10.24 | 10.33 | na | 0.99× | +| `rst_merge` | 34.5 | 35.24 | na | 0.98× | +| `rst_combineavg_agg` | 26.97 | 27.59 | na | 0.98× | +| `rst_frombands_agg` | 22.54 | 23.11 | na | 0.98× | +| `rst_width` | 3.92 | 4.06 | deferred | 0.97× | +| `rst_derivedband_agg` | 11.37 | 11.77 | na | 0.97× | +| `rst_combineavg` | 23.19 | 24.26 | na | 0.96× | +| `rst_merge_agg` | 41.42 | 42.97 | na | 0.96× | +| `rst_frombands` | 19.46 | 20.45 | na | 0.95× | +| `rst_dtmfromgeoms_agg` | 4.13 | 4.33 | na | 0.95× | +| `rst_contour` | 47.68 | 54.62 | materialized | 0.87× | +| `rst_tooverlappingtiles` | 28.06 | 37.0 | materialized | 0.76× | +| `rst_polygonize` | 15.77 | 23.08 | materialized | 0.68× | +| `rst_to_webmercator` | 15.61 | 24.11 | materialized | 0.65× | +| `rst_resample` | 15.96 | 24.56 | materialized | 0.65× | +| `rst_filter` | 16.39 | 26.41 | materialized | 0.62× | +| `rst_buildoverviews` | 13.01 | 21.13 | materialized | 0.62× | +| `rst_retile` | 15.8 | 25.75 | materialized | 0.61× | +| `rst_transformcrs` | 11.32 | 19.14 | materialized | 0.59× | +| `rst_transform` | 12.13 | 20.88 | materialized | 0.58× | +| `rst_separatebands` | 10.72 | 19.41 | materialized | 0.55× | +| `rst_threshold` | 8.6 | 16.83 | materialized | 0.51× | +| `rst_maketiles` | 16.24 | 31.8 | materialized | 0.51× | +| `rst_updatetype` | 7.03 | 15.56 | materialized | 0.45× | +| `rst_derivedband` | 5.49 | 13.74 | materialized | 0.4× | +| `rst_cog_convert` | 5.29 | 13.65 | materialized | 0.39× | +| `rst_resample_to_size` | 4.17 | 12.07 | materialized | 0.35× | +| `rst_proximity` | 4.67 | 13.59 | materialized | 0.34× | +| `rst_avg` | 4.1 | 12.37 | materialized | 0.33× | +| `rst_min` | 3.5 | 11.37 | materialized | 0.31× | +| `rst_max` | 3.24 | 10.59 | materialized | 0.31× | +| `rst_rastertoworldcoord` | 3.11 | 10.14 | deferred | 0.31× | +| `rst_slope` | 3.83 | 12.96 | materialized | 0.3× | +| `rst_median` | 3.5 | 11.52 | materialized | 0.3× | +| `rst_pixelcount` | 3.38 | 11.16 | materialized | 0.3× | +| `rst_hillshade` | 3.77 | 12.82 | materialized | 0.29× | +| `rst_roughness` | 3.43 | 11.7 | materialized | 0.29× | +| `rst_fillnodata` | 3.57 | 12.44 | materialized | 0.29× | +| `rst_evi` | 3.38 | 11.66 | materialized | 0.29× | +| `rst_index` | 3.29 | 11.3 | materialized | 0.29× | +| `rst_ndvi` | 3.57 | 12.75 | materialized | 0.28× | +| `rst_aspect` | 3.79 | 13.51 | materialized | 0.28× | +| `rst_ndwi` | 3.41 | 12.2 | materialized | 0.28× | +| `rst_nbr` | 3.45 | 12.23 | materialized | 0.28× | +| `rst_savi` | 3.13 | 11.34 | materialized | 0.28× | +| `rst_tri` | 3.55 | 12.92 | materialized | 0.27× | +| `rst_tpi` | 3.29 | 12.33 | materialized | 0.27× | +| `rst_isempty` | 3.19 | 12.04 | materialized | 0.27× | +| `rst_asformat` | 3.12 | 11.73 | materialized | 0.27× | +| `rst_convolve` | 4.48 | 17.24 | materialized | 0.26× | +| `rst_tryopen` | 2.96 | 11.43 | na | 0.26× | +| `rst_mapalgebra` | 3.44 | 19.98 | materialized | 0.17× | + +#### FILE vs FUSE: large-COG multi-window results {#file-capability-cog-multiwindow} + +In the per-tile-open (non-amortized) regime, FILE is not a throughput win — FUSE's flat open cost and block-level I/O beat per-tile HTTP stream setup. With amortization (grouped executor, `ORDER BY path`), byte-range stream reads win 10–290× — see [Virtual tile read performance](../api/performance#virtual-tile-read-performance). + +| Leg | Setup | FILE-on | FUSE (FILE-off) | FUSE advantage | +|---|---|---|---|---| +| High reuse | 1 × 648 MB COG, 1 000 windows | 27.4 s | 13.1 s | **2.1×** | +| High reuse, misaligned window | 1 × 648 MB COG, ~840 × 384 px windows (straddle 256px blocks) | 26.7 s | 18.9 s | **1.42×** | +| High reuse, 2.5 GB COG | 1 × 2.58 GB COG, 1 000 × 256 px windows | 22.8 s | 16.7 s | **1.37×** | +| Low reuse | 25 × 648 MB COG copies, 1 window each | 16.2 s | 1.4 s | **11.4×** | +| Real COG — NAIP (DEFLATE, 512px blocks) | 324 MB, 1 000 × 256 px windows | 23.4 s | 16.2 s | **1.44×** | +| Real COG — Sentinel-2 (DEFLATE, 1024px blocks) | 22 MB, 1 000 × 256 px windows | 19.4 s | 17.7 s | **0.91× (wash)** | + +--- + :::note Mode coverage (of the 107 RasterX functions) Not every function appears in both timing models — the gaps are deliberate measurement choices, not functional gaps (the lightweight tier implements all 107): - **Pure-core: 100 / 107.** The 7 absent are the grouped aggregators (`rst_*_agg`) — a UDAF has no single-tile, single-row form to time in isolation, so they appear only in spark-path. - **Spark-path: 83 / 107.** The 24 absent are pure-core-only because their call shape doesn't fit the spark-path tile-DataFrame model: geometry-input constructors (`rst_rasterize`, `rst_gridfrompoints`, `rst_dtmfromgeoms` — the tile DataFrame carries no geometry column), the path reader (`rst_fromfile`), functions needing an in-extent coordinate/geometry literal valid across the multi-CRS corpus (`rst_clip`, `rst_sample`, `rst_viewshed`, `rst_worldtorastercoord*`, `rst_resample_to_res`), render-engine-divergent tiles (`rst_tilexyz`, `rst_xyzpyramid`, `rst_color_relief`), and metadata/scalar accessors (`rst_metadata`, `rst_type`, `rst_summary`, …) whose spark-path cell would be an uncomparable timing-only result. ::: -### Pure-core (local, 1024²) +### Pure-core (local, 1024²) {#pure-core} The pure-core table is the **algorithm in isolation**: open one tile, call the function, measure — no Spark, no serialization. It is the fairest view of the raw implementation. It is one snapshot — absolute timings are environment-dependent — but the **relative** picture and the consistency outcome are stable. Per-tile timing is shown in **milliseconds**; cells whose output cannot be fingerprinted for a direct comparison (readers, metadata accessors) are timed and labelled `timing-only`. @@ -537,7 +701,7 @@ The pure-core table is the **algorithm in isolation**: open one tile, call the f - **Terrain (`rst_slope`/`aspect`/`hillshade`/`tri`/`tpi`/`roughness`)** is a steadier ~2–3.6× lightweight win, all within tolerance — including on geographic rasters, where both tiers auto-derive the horizontal scale from the CRS. - **Discrete-grid aggregation (`rst_h3_*`/`rst_quadbin_*`)** is a ~1.7–6× lightweight win and matches within tolerance. - > **British National Grid + quadbin raster-grid functions — cluster-measured.** The raster→grid family gained five British National Grid reducers — `rst_bng_rastertogridavg`, `rst_bng_rastertogridcount`, `rst_bng_rastertogridmax`, `rst_bng_rastertogridmin`, `rst_bng_rastertogridmedian` — alongside two more tessellation generators (`rst_quadbin_tessellate`, `rst_bng_tessellate`) and two grid rasterize aggregators (`rst_quadbin_rasterize_agg`, `rst_bng_rasterize_agg`). All nine were benchmarked on the 20-worker cluster with the same light-vs-heavy fingerprint parity check as their `rst_h3_*` / `rst_quadbin_*` siblings (a fixed deterministic cell set for the two aggregators). Because the BNG functions reproject the tile to EPSG:27700 internally and drop pixels outside Great Britain, they are measured on a **British-National-Grid tile** (EPSG:27700 over central London) so they bin **real cells**. On that tile the five BNG reducers agree **within tolerance** across tiers (identical cell sets; per-cell aggregate values agree to rel ≤ 0.001), and all three tessellation generators (`rst_h3_tessellate`, `rst_quadbin_tessellate`, `rst_bng_tessellate`) plus both `*_rasterize_agg` aggregators agree **exactly** (byte-identical cell sets and rasterized output). The tessellators use a **positive-area covering keep-test** on both tiers: a cell is emitted iff its geometry has greater-than-zero area overlap with the raster, so a cell that merely touches the tile along its exact edge (a shared boundary line, zero pixel overlap) is excluded on both tiers, while a within-extent cell whose pixels are all NoData is still emitted (NoData renders in place, not as a gap). On the measured British-National-Grid tile this yields identical `rst_bng_tessellate` cell sets across tiers — the earlier heavy-only edge-touch fringe (previously 48 vs 36 cells) is gone. + > **British National Grid + quadbin raster-grid functions — cluster-measured.** The raster→grid family gained five British National Grid reducers — `rst_bng_rastertogridavg`, `rst_bng_rastertogridcount`, `rst_bng_rastertogridmax`, `rst_bng_rastertogridmin`, `rst_bng_rastertogridmedian` — alongside two more tessellation generators (`rst_quadbin_tessellate`, `rst_bng_tessellate`) and two grid rasterize aggregators (`rst_quadbin_rasterize_agg`, `rst_bng_rasterize_agg`). All nine were benchmarked on the 20-worker cluster with the same light-vs-heavy fingerprint parity check as their `rst_h3_*` / `rst_quadbin_*` siblings (a fixed deterministic cell set for the two aggregators). Because the BNG functions reproject the tile to EPSG:27700 internally and drop pixels outside Great Britain, they are measured on a **British-National-Grid tile** (EPSG:27700 over central London) so they bin **real cells**. On that tile the five BNG reducers agree **within tolerance** across tiers (identical cell sets; per-cell aggregate values agree to rel ≤ 0.001), and all three tessellation generators (`rst_h3_tessellate`, `rst_quadbin_tessellate`, `rst_bng_tessellate`) plus both `*_rasterize_agg` aggregators agree **exactly** (decoded-pixel parity: cell sets + rasterized pixels identical across tiers). The tessellators use a **positive-area covering keep-test** on both tiers: a cell is emitted iff its geometry has greater-than-zero area overlap with the raster, so a cell that merely touches the tile along its exact edge (a shared boundary line, zero pixel overlap) is excluded on both tiers, while a within-extent cell whose pixels are all NoData is still emitted (NoData renders in place, not as a gap). On the measured British-National-Grid tile this yields identical `rst_bng_tessellate` cell sets across tiers — the earlier heavy-only edge-touch fringe (previously 48 vs 36 cells) is gone. > > Representative pure-core timings (512² · 4-band · `float32`; the BNG functions on an EPSG:27700 British-National-Grid tile, quadbin on EPSG:4326): `rst_bng_tessellate` is a **14.6× lightweight win** (heavy 1.56 s vs light 0.11 s, exact) and `rst_quadbin_tessellate` a **12.9× win** (1.23 s vs 0.10 s, exact). The BNG reducers, now binning real cells, are also **lightweight-favored — roughly 3–7×** (heavy ~1.1–1.6 s vs light ~0.22–0.37 s). This followed a rewrite of the lightweight BNG encoder from a pure-Python per-pixel loop into a single vectorized NumPy kernel (bit-identical cell IDs): before that change the same reducers were heavyweight-favored (~0.12–0.15×), with the per-pixel Python encode dominating; once vectorized, the encode is array math and the EPSG:27700 reprojection each BNG tile still carries is a minor residual term. (The **BNG reducers and `rst_bng_tessellate` reproject the tile to EPSG:27700 internally**, unlike the 4326-native H3 and quadbin functions.) The reducers and tessellators are registered on the lightweight tier as table functions (invoked with SQL `LATERAL`); their at-scale spark-path timings are in the [raster→grid family subsection below](#raster-grid-family-at-scale). - **Trivial metadata accessors (`rst_width`/`height`/`numbands` and the world↔raster coordinate helpers)** are microseconds on both tiers; here the JVM-native heavyweight path edges out the Python-worker call, so speedup dips below 1×. At this scale the absolute difference is a fraction of a millisecond. @@ -648,44 +812,48 @@ This repartitioning is **benchmark-harness tuning** — it lives in the bench co The discrete-global-grid functions — the `rst_{h3,quadbin,bng}_rastertogrid*` reducers, the three tessellation generators, and the three grid rasterize aggregators — are measured in their own table because they don't share the general table's shape. On the **lightweight tier the reducers and tessellators are registered as Python table functions**, so their spark-path form is a SQL `LATERAL` table-function join over the tile DataFrame (`SELECT t.* FROM tiles, LATERAL gbx_rst_h3_rastertogridavg(tile, resolution) t`), not a scalar column — the realistic distributed per-tile cost of the row fan-out. The heavyweight tier's equivalent expressions return an array from a scalar column; both tiers are timed on the **same 1,000-tile job** so the per-tile wall-clock is comparable. The three grid rasterize aggregators run as grouped `groupBy().agg(...)` on both tiers and are the only members with a comparable fingerprint (all **exact**); the reducers and tessellators are `timing-only` in spark-path (a `LATERAL` row stream emits no single-value fingerprint — cross-tier parity for these is enforced in the pure-core comparison and by the dedicated parity tests instead). -> **Environment:** Databricks · DBR 18.x · **20 workers** · 512² 4-band `float32` tiles · **1,000 tiles per iteration** (every tile processed each iteration). H3 and quadbin read the tile as EPSG:4326 lon/lat; the **BNG reducers and `rst_bng_tessellate` reproject each tile to EPSG:27700 internally**, so their timing includes that per-pixel warp — a like-for-like reading against the 4326-native H3/quadbin functions should account for it. `Heavy/tile` and `Light/tile` are the whole-job wall-clock amortized over the 1,000 tiles. +> **Environment:** Databricks · DBR 18.x · **20 workers** · 512² 4-band `float32` tiles · **1,000 tiles per iteration** (every tile processed each iteration). H3 and quadbin read the tile as EPSG:4326 lon/lat; the **BNG reducers and `rst_bng_tessellate` reproject each tile to EPSG:27700 internally**, so their timing includes that per-pixel warp — a like-for-like reading against the 4326-native H3/quadbin functions should account for it. `Heavy/tile` and `Light/tile` are the whole-job wall-clock amortized over the 1,000 tiles. `Light virtual/tile` repeats the lightweight timing when the function is fed a **virtual** input tile (a path + window, carrying no pixel bytes); `Disposition` marks whether the function read pixels (`materialized`) or stayed lazy (`deferred`) — `na` for the grouped aggregators, which take no single input tile. > > Ordered by **Speedup = `Heavy / Light`** (highest first). The three `*_rasterize_agg` aggregators burn a fixed deterministic cell set onto one small canvas per group (a different, smaller workload than a 512² tile), so read their **cross-tier ratio and `exact` parity** as the comparable result, not the absolute ms against the reducer rows. -| Function | Heavy/tile (ms) | Light/tile (ms) | Speedup | Consistency | -|---|---|---|---|---| -| `rst_quadbin_tessellate` | 58.55 | 6.16 | 9.50× | timing-only | -| `rst_h3_tessellate` | 27.47 | 4.53 | 6.07× | timing-only | -| `rst_bng_rastertogridsum` | 39.51 | 8.23 | 4.80× | timing-only | -| `rst_bng_rastertogridvariance` | 39.19 | 8.57 | 4.57× | timing-only | -| `rst_bng_rastertogridcount` | 39.24 | 8.62 | 4.55× | timing-only | -| `rst_bng_rastertogridmin` | 39.53 | 9.04 | 4.37× | timing-only | -| `rst_bng_rastertogridmax` | 39.52 | 9.23 | 4.28× | timing-only | -| `rst_bng_rastertogridmedian` | 38.10 | 8.94 | 4.26× | timing-only | -| `rst_bng_rastertogridstddev` | 38.29 | 9.00 | 4.25× | timing-only | -| `rst_bng_rastertogridavg` | 37.00 | 10.12 | 3.66× | timing-only | -| `rst_quadbin_rastertogridmedian` | 10.78 | 5.91 | 1.82× | timing-only | -| `rst_bng_tessellate` | 8.53 | 4.71 | 1.81× | timing-only | -| `rst_quadbin_rastertogridavg` | 6.79 | 5.37 | 1.27× | timing-only | -| `rst_quadbin_rastertogridvariance` | 7.48 | 5.96 | 1.26× | timing-only | -| `rst_quadbin_rastertogridstddev` | 7.87 | 6.32 | 1.24× | timing-only | -| `rst_quadbin_rastertogridmax` | 6.81 | 5.56 | 1.23× | timing-only | -| `rst_quadbin_rastertogridsum` | 6.84 | 5.61 | 1.22× | timing-only | -| `rst_quadbin_rastertogridmin` | 6.58 | 5.66 | 1.16× | timing-only | -| `rst_quadbin_rastertogridcount` | 6.43 | 5.62 | 1.14× | timing-only | -| `rst_h3_rastertogridmedian` | 31.87 | 39.39 | 0.81× | timing-only | -| `rst_h3_rastertogridvariance` | 28.60 | 39.56 | 0.72× | timing-only | -| `rst_quadbin_rasterize_agg` | 1.58 | 2.24 | 0.70× | exact | -| `rst_h3_rastertogridmax` | 26.28 | 38.64 | 0.68× | timing-only | -| `rst_h3_rastertogridmin` | 26.26 | 39.61 | 0.66× | timing-only | -| `rst_bng_rasterize_agg` | 1.54 | 2.34 | 0.66× | exact | -| `rst_h3_rastertogridstddev` | 27.35 | 42.79 | 0.64× | timing-only | -| `rst_h3_rastertogridavg` | 26.33 | 41.88 | 0.63× | timing-only | -| `rst_h3_rastertogridcount` | 25.05 | 39.91 | 0.63× | timing-only | -| `rst_h3_rasterize_agg` | 1.58 | 2.51 | 0.63× | exact | -| `rst_h3_rastertogridsum` | 25.86 | 42.92 | 0.60× | timing-only | - -The tessellators are the standout lightweight wins (`rst_quadbin_tessellate` 9.5×, `rst_h3_tessellate` 6.1×): the covering-cell walk is cheap in Python and the `LATERAL` fan-out amortizes the JVM↔Python boundary once per input tile. The **BNG reducers are now strongly lightweight-favored (~3.7–4.8×)** — the second-biggest wins in the table. The lightweight BNG encoder was rewritten from a pure-Python per-pixel loop into a single vectorized NumPy kernel (bit-identical cell IDs), so binning a full 512² tile is now array math rather than one interpreted `point_to_cell_id` call per pixel; the EPSG:27700 reprojection each BNG tile still carries turns out to be a minor term next to the encode it used to hide. (Before this change the same reducers were the most heavyweight-favored rows in the table, ~0.15–0.19×.) They bin **real Great Britain cells** on an EPSG:27700 tile over central London, not an empty grid. The quadbin reducers sit at slight lightweight advantage to parity (~1.1–1.8×), while the H3 reducers are modestly heavyweight-favored (~0.6–0.8×) — H3's finer resolution 7 bins more cells per tile, and H3's C-backed encoder is already fast, so the per-cell Python grouping work outweighs the compute saving at scale. The three grid rasterize aggregators are heavyweight-favored on their small burned canvas (~0.63–0.70×) and match **exactly** across tiers. +| Function | Heavy/tile (ms) | Light/tile (ms) | Light virtual/tile (ms) | Disposition | Speedup | Consistency | +|---|---|---|---|---|---|---| +| `rst_quadbin_tessellate` | 58.55 | 6.16 | 34.66 | materialized | 9.50× | timing-only | +| `rst_h3_tessellate` | 27.47 | 4.53 | 26.87 | materialized | 6.07× | timing-only | +| `rst_bng_rastertogridmin` | 39.53 | 7.85 | 16.49 | materialized | 5.04× | timing-only | +| `rst_bng_rastertogridstddev` | 38.29 | 7.68 | 16.93 | materialized | 4.99× | timing-only | +| `rst_bng_rastertogridsum` | 39.51 | 7.98 | 17.44 | materialized | 4.95× | timing-only | +| `rst_bng_rastertogridvariance` | 39.19 | 8.01 | 16.15 | materialized | 4.89× | timing-only | +| `rst_bng_rastertogridmax` | 39.52 | 8.14 | 16.13 | materialized | 4.86× | timing-only | +| `rst_bng_rastertogridcount` | 39.24 | 8.17 | 16.44 | materialized | 4.80× | timing-only | +| `rst_bng_rastertogridmedian` | 38.10 | 8.14 | 17.89 | materialized | 4.68× | timing-only | +| `rst_bng_rastertogridavg` | 37.00 | 8.43 | 17.05 | materialized | 4.39× | timing-only | +| `rst_quadbin_rastertogridmedian` | 10.78 | 5.91 | 14.14 | materialized | 1.82× | timing-only | +| `rst_quadbin_rastertogridavg` | 6.79 | 5.37 | 13.03 | materialized | 1.27× | timing-only | +| `rst_quadbin_rastertogridvariance` | 7.48 | 5.96 | 13.07 | materialized | 1.26× | timing-only | +| `rst_quadbin_rastertogridstddev` | 7.87 | 6.32 | 13.96 | materialized | 1.24× | timing-only | +| `rst_quadbin_rastertogridmax` | 6.81 | 5.56 | 12.93 | materialized | 1.23× | timing-only | +| `rst_quadbin_rastertogridsum` | 6.84 | 5.61 | 13.97 | materialized | 1.22× | timing-only | +| `rst_quadbin_rastertogridmin` | 6.58 | 5.66 | 14.52 | materialized | 1.16× | timing-only | +| `rst_quadbin_rastertogridcount` | 6.43 | 5.62 | 13.90 | materialized | 1.14× | timing-only | +| `rst_h3_rastertogridmedian` | 31.87 | 39.39 | 47.44 | materialized | 0.81× | timing-only | +| `rst_h3_rastertogridvariance` | 28.60 | 39.56 | 49.37 | materialized | 0.72× | timing-only | +| `rst_quadbin_rasterize_agg` | 1.58 | 2.24 | 2.65 | na | 0.70× | exact | +| `rst_h3_rastertogridmax` | 26.28 | 38.64 | 48.77 | materialized | 0.68× | timing-only | +| `rst_h3_rastertogridmin` | 26.26 | 39.61 | 45.62 | materialized | 0.66× | timing-only | +| `rst_bng_rasterize_agg` | 1.54 | 2.34 | 2.79 | na | 0.66× | exact | +| `rst_h3_rastertogridstddev` | 27.35 | 42.79 | 52.01 | materialized | 0.64× | timing-only | +| `rst_h3_rastertogridavg` | 26.33 | 41.88 | 48.78 | materialized | 0.63× | timing-only | +| `rst_h3_rastertogridcount` | 25.05 | 39.91 | 48.08 | materialized | 0.63× | timing-only | +| `rst_h3_rasterize_agg` | 1.58 | 2.51 | 2.62 | na | 0.63× | exact | +| `rst_h3_rastertogridsum` | 25.86 | 42.92 | 48.06 | materialized | 0.60× | timing-only | +| `rst_bng_tessellate` | 8.53 | 26.76 | 35.60 | materialized | 0.32× | timing-only | + +The tessellators split by projection. `rst_quadbin_tessellate` (9.5×) and `rst_h3_tessellate` (6.1×) are strong lightweight wins — they read the tile as native EPSG:4326, so the covering-cell walk is cheap in Python and the `LATERAL` fan-out amortizes the JVM↔Python boundary once per input tile. `rst_bng_tessellate` is the opposite (0.32×, heavyweight-favored): it reprojects each tile to EPSG:27700 before walking cells, and that per-pixel warp plus the cell walk runs slower in Python than in the JVM. + +The **BNG reducers are the table's largest lightweight wins (~4.4–5.0×)**. The lightweight BNG encoder is a single vectorized NumPy kernel (bit-identical cell IDs), so binning a full 512² tile is array math rather than one interpreted `point_to_cell_id` call per pixel, and the EPSG:27700 reprojection each BNG tile carries is a minor term next to the encode. Every BNG row bins **real Great Britain cells** on a 27700-native tile over central London; the lightweight spark-path had been measuring these against the general EPSG:4326 corpus tile, which falls outside Great Britain and yields an empty grid — that is now fixed, so the BNG rows (including `rst_bng_tessellate`) reflect real cells. The quadbin reducers sit at slight lightweight advantage to parity (~1.1–1.8×), while the H3 reducers are modestly heavyweight-favored (~0.6–0.8×) — resolution 7 bins more cells per tile and H3's C-backed encoder is already fast, so the per-cell Python grouping outweighs the compute saving at scale. The three grid rasterize aggregators are heavyweight-favored on their small burned canvas (~0.63–0.70×) and match **exactly** across tiers. + +The **Light virtual/tile** column times the lightweight tier fed a *virtual* input tile — a path plus a window, carrying no pixel bytes — and **Disposition** records whether the function ended up reading pixels. Every raster→grid function here is `materialized`: it must read every pixel to bin cells, so the virtual input is opened and read in full and the column runs **1.1–5.9× slower** than the materialized `Light/tile` — with nothing to defer, the virtual tile is pure open overhead. Virtual tiles pay off for header- and metadata-only functions, which stay `deferred` and never touch pixels, not for the pixel-reading work in this table. The rasterize aggregators show `na` because they reduce a group of cell IDs, not a single input tile, so the marker does not apply. ### Fan-out generators (streaming UDTFs) @@ -914,6 +1082,151 @@ speedup = heavy ÷ light, ≥1.0 → light faster): +## How the benchmark works & how to run it + +### What it measures + +Each function is timed under two independent models: + +- **Pure-core** — the raster operation in isolation: open **a single tile**, call the function, measure. It is always one tile per measurement (repeated once for each tile *shape* in the corpus — each tile-size / band-count / dtype / SRID combination), and it ignores `--row-counts` entirely. This is the fairest apples-to-apples view of the algorithm itself, with no Spark or serialization in the path. +- **Spark-path** — the registered function (`rst_*`) applied to a Spark DataFrame of *N* rows (the only model that uses `--row-counts`). This includes the realistic per-row overhead (UDF dispatch, serialization, Python-worker round-trips for the lightweight tier), and is swept across a **row ladder** (e.g. 10 → 100 → 1,000 → 10,000 rows). + +Both models run the function `--warmup` times **untimed** (to absorb cold caches, JIT, and worker spin-up) and then `--measured` times **timed**; the reported figure is the **median over the measured passes**. Locally the defaults are `2` warmup / `5` measured; on a cluster they are `1` / `3` for pure-core and `1` / `1` for spark-path (one full *N*-tile iteration is already substantial). + +Alongside timing, every pure-core result carries an **output fingerprint** (per-band statistics) so the two tiers can be checked for **consistency**: + +- **exact** — every statistic is bitwise-equal across tiers. +- **within_tol** — every statistic agrees to a relative tolerance of `1e-3` **or** an absolute tolerance of `1e-3` (the absolute floor handles near-zero values where a relative comparison is meaningless). +- **divergent** — neither tolerance is met. + +The goal of the [one-line tier swap](./execution-tiers#the-one-line-swap) is that results stay consistent; the benchmark is how that guarantee is verified. + +:::note +The benchmark is run **from the GeoBrix source repository** (it uses the repo's `gbx:bench:*` commands and a job notebook). A wheel-only install does not include these tools — the cluster benchmark *submits a job to your own provisioned cluster* from a checkout of the repo. +::: + +### Running on a cluster + +Running on a provisioned cluster is the **true comparison**: both tiers execute on the same hardware, against the same corpus, and the full row ladder and larger tiles are within reach. Results append to a `bench_results` Delta table and a `comparison.csv` / `summary.md` land on the configured Volume. + +#### Prerequisites + +Provision a cluster and stage the artifacts per the [installation guide](../installation): + +| Tier | Cluster | Artifacts | +|---|---|---| +| Heavyweight (rasterx) | x86 · DBR 17.3 LTS | Init script + bundle + GeoBrix wheel + the bench tests JAR (`geobrix-*-tests.jar`) | +| Lightweight (pyrx) | x86 **or ARM** | The `[light]` wheel only | + +Then fill in the cluster configuration file (`notebooks/tests/databricks_cluster_config.env`) with your cluster ID and Volume paths. + +#### Run + +```bash +# Both tiers, same cluster, full row ladder, all functions +bash scripts/commands/gbx-bench-cluster.sh \ + --cluster-id \ + --run-id cluster-2026-06 \ + --modes both \ + --row-counts 10,100,1000,10000 +``` + +Scale the run with the options below. `--cluster-id` and `--run-id` identify the run; the rest are optional. `--row-counts` and `--functions` take a **comma-separated list**; `--modes` and `--set` take a **single value**. + +| Option | Purpose | +|---|---| +| `--modes pure-core \| spark-path \| both` | Timing model — pick only one mode (default `both`, which runs both models). | +| `--row-counts 10,100,1000,10000` | Spark-path row ladder — each value is the number of **distinct tiles processed in one timed iteration** (the main scale dimension); one iteration is measured per rung. The largest value must be **≤ the corpus row-pool size** (the bench refuses to under-fill). Default `10,100,1000,10000`. | +| `--set core \| full` | Which function set to benchmark — the representative `core` set or every benchmarked function (`full`). Default `core`. | +| `--functions rst_slope,rst_ndvi` | Comma-separated list restricting the run to specific functions; overrides `--set`. Default: unset (benchmarks the `--set`). | +| `--warmup` / `--measured` | Warmup and measured iteration counts. Defaults: pure-core `1` / `3`, spark-path `1` / `1`. | +| `--lightweight-only` / `--heavyweight-only` | Run a single tier (mutually exclusive). `--lightweight-only` is **required on ARM clusters** (heavyweight is x86-only); `--heavyweight-only` skips the lightweight leg. Default: both tiers run. | +| `--no-wait` | Submit the job without blocking on completion. Default: waits for the run to finish. | + +```bash +# ARM cluster: lightweight only +bash scripts/commands/gbx-bench-cluster.sh --cluster-id --lightweight-only +``` + +Tile sizes, band counts, data types, and projections are set when the corpus is generated (see the local section's scale knobs); the same corpus is reused for both tiers so the comparison is fair. + +:::note +A spark-path iteration processes `max(--row-counts)` **distinct** tiles drawn from the corpus row pool — it does not recycle a small pool to reach the row count. The largest `--row-counts` value must therefore be **≤ the corpus row-pool size**; if it isn't, the bench **refuses to run** rather than silently under-fill (which would report a row count it never actually processed). Generate a larger pool or lower the row ladder. +::: + +#### Output + +- **`bench_results` Delta table** — every measured row (tagged with environment + run ID), so runs accumulate and can be queried/visualized over time. +- **`summary.md` / `comparison.csv`** on the Volume — the human-readable speedup + consistency report (see [Reading the output](#reading-the-output)), also rendered inline in the run notebook. + +### Running locally + +The local pipeline runs the heavyweight tier in the `geobrix-dev` Docker container and the lightweight tier in an isolated Python virtual environment, then compares them. Local runs are intentionally **single-tile** for pure-core (the algorithm cost in isolation); the full row ladder and at-scale spark-path numbers belong on a cluster, but can still be exercised in the local Docker environment at a modest scale. + +```bash +# Full local pipeline: generate corpus -> heavyweight -> lightweight -> compare +bash scripts/commands/gbx-bench-all.sh --run-id local-1 --modes pure-core +``` + +The heavyweight and lightweight legs run **one after the other** (never concurrently) so they don't contend for CPU and skew each other's timings. Outputs land in `test-logs/bench//`. + +#### Set (core vs full) + +`--set` selects **how many functions** the run benchmarks: + +- `--set core` (the default) runs a small, representative set covering each function family — accessors, terrain, band math, warps. It's fast and is the right choice for a routine check. +- `--set full` runs every benchmarked function. It takes longer but gives the complete coverage and parity picture. + +:::note +The **`core`** here is the *function set* (`--set`) — how many functions run. Don't confuse it with **pure-core**, the *timing model* (`--modes`) that times one tile in isolation. They are independent: you can run `--set core --modes pure-core`, `--set full --modes spark-path`, or any other combination. +::: + +An explicit `--functions` list overrides `--set`. + +```bash +# Routine check (default): the representative core set +bash scripts/commands/gbx-bench-all.sh --modes pure-core + +# Complete coverage: every benchmarked function +bash scripts/commands/gbx-bench-all.sh --set full --modes pure-core +``` + +Scale and shape the corpus with the options below. Each takes a **comma-separated list to sweep several values** (e.g. `--tile-px 256,512,1024`) **or a single value** (e.g. `--tile-px 1024`); the corpus is the combination across the options you set. + +| Option | Purpose | +|---|---| +| `--tile-px 256,512,1024,2048` | Tile sizes (pixels per side) — any size; larger tiles (1024², 2048², …) make the per-tile algorithm cost dominate the fixed overhead. Default `256,512`. | +| `--bands 1,4` | Band counts — any positive integer count. Default `2`. | +| `--dtypes uint8,int16,float32` | Pixel data types — these three are the full supported set (closed; other dtypes are not generated). Default `float32`. | +| `--srids 4326,3857,32618,27700` | Projections — the full supported set (closed): `4326` (WGS84 geographic), `3857` (WebMercator), `32618` (UTM 18N), `27700` (British National Grid); other SRIDs are not generated. Default `4326,32618`. Pick at least one geographic (lat/long) **and** one projected (metre) CRS to exercise both. | +| `--nodata-frac 0.0,0.25` | Fraction of pixels set to NoData — **any value in `0.0`–`1.0`** (0% to 100%; continuous, not a fixed set). Default `0.0`. Pass a comma-separated list to sweep several fractions. | +| `--row-counts 10,100,1000,10000` | Spark-path row ladder — each value is the number of **distinct tiles processed in one timed iteration** (the scale dimension); the bench measures one iteration per rung. The largest value must be **≤ the corpus row-pool size** (the bench refuses to under-fill). Default `2,4` (laptop-modest); run the full ladder on a cluster. | +| `--modes pure-core \| spark-path \| both` | Timing model — pick only one mode. Default `both` (runs both models). | +| `--warmup` / `--measured` | Untimed warmup passes / timed passes per measurement (median of the timed passes is reported). Defaults `2` / `5` locally. | + +The individual stages are also available as standalone commands — `gbx:bench:gen-data`, `gbx:bench:heavyweight`, `gbx:bench:lightweight`, and `gbx:bench:compare` — if you want to regenerate just one part of the pipeline. + +### Reading the output + +The comparison `summary.md` opens with **insights** (biggest wins, the consistency tally) followed by a per-function table. For a spark-path run it looks like: + +```text +Consistency (7 compared cells): exact 0 - within-tol 7 - divergent 0 + +Tile scale: 1000 tiles/iteration (spark-path) — every tile processed each timed iteration. + +| fn | hw_iter_s | lw_iter_s | hw_per_tile_s | lw_per_tile_s | speedup | consistency | +| rst_dtmfromgeoms_agg | 175.10 | 9.94 | 0.17510 | 0.00994 | 17.61 | within_tol | +| rst_slope | 7.31 | 7.57 | 0.00731 | 0.00757 | 0.97 | timing-only | +| rst_proximity | 0.52 | 8.12 | 0.00052 | 0.00812 | 0.06 | timing-only | +``` + +- **`hw_iter_s` / `lw_iter_s`** are the median wall-clock of **one full iteration over all *N* tiles** (the whole distributed job). **`hw_per_tile_s` / `lw_per_tile_s`** are that ÷ *N* — the amortized per-tile cost. (A pure-core summary uses `hw_ms` / `lw_ms` and `hw_mpix/s` / `lw_mpix/s` for the single-tile algorithm cost instead.) +- **`speedup`** is `heavy / light` — greater than 1 means the lightweight tier is faster. +- **`consistency`** is the `exact` / `within_tol` / `divergent` label defined above (`timing-only` where the output can't be fingerprinted for a direct comparison — readers, metadata accessors, and most non-aggregator spark-path cells). Per-cell deltas are in `comparison.csv` (the `max_rel_delta` column). + +A per-engine summary is also written for each tier (`heavyweight.summary.md`, `lightweight.summary.md`) with that tier's own timing and throughput in isolation. + ## Caveats - Absolute timings depend on the machine, the corpus, and the tier's install; treat the **relative** speedup and the **consistency** outcome as the durable signal, not the millisecond values. diff --git a/docs/docs/api/coordinate-reference-systems.mdx b/docs/docs/api/coordinate-reference-systems.mdx new file mode 100644 index 000000000..61ab50646 --- /dev/null +++ b/docs/docs/api/coordinate-reference-systems.mdx @@ -0,0 +1,286 @@ +--- +sidebar_position: 5 +title: Coordinate Reference Systems +--- + +# Coordinate Reference Systems + +A **coordinate reference system (CRS)** ties a raster's or geometry's coordinates to a place on Earth. GeoBrix supports CRS handling **across all three packages** — RasterX, GridX, and VectorX — and across **both raster execution tiers** (lightweight `pyrx` and heavyweight `rasterx`). This page explains the two ways GeoBrix names a CRS — the integer **SRID** and the **CRS string** — when to use each, and how a non-EPSG CRS (an ESRI code, WKT, or PROJ4 definition) survives a full read → operate → write round trip. + +## SRID vs CRS string + +GeoBrix names a CRS two ways, and both are first-class: + +| | **SRID (integer)** | **CRS string** | +|---|---|---| +| Type | `INT` | `STRING` | +| Example | `4326`, `54008` | `"EPSG:4326"`, `"ESRI:54008"`, WKT, PROJ4 | +| Can represent | an EPSG **or** ESRI code | **any** CRS — EPSG, ESRI, WKT, PROJ4 | +| No-code value | `NULL` (lightweight) / `0` (heavyweight) | always a value | +| Use for | the native ST bridge, authority-code workflows | authority-less CRSes, lossless round trips | + +The integer SRID is compact and maps directly onto Databricks' native ST functions (`ST_GeomFromWKB(wkb, srid)`), but it can **only** name a CRS that carries an authority code — EPSG or ESRI (see [the resolution rule](#how-an-integer-srid-becomes-a-crs) below). A great many real datasets carry an ESRI code (MODIS products use ESRI:54008, World Sinusoidal), and some imagery carries only an embedded WKT with no authority code at all. For an authority-less CRS the SRID is `NULL`/`0` and the CRS string is the only lossless representation. + +:::tip Rule of thumb +Reach for the **SRID** when you need the integer for the native ST bridge or an authority-code pipeline. Reach for the **CRS string** whenever an authority-less CRS (raw WKT/PROJ4) might be in play — it never loses one. +::: + +### The four CRS-string forms + +Every GeoBrix function that takes a CRS *string* accepts any of these four forms interchangeably — an authority code, an int-castable string, **WKT**, or **PROJ4**: + +```python +from databricks.labs.gbx.pyrx import functions as rx # or ...rasterx — same names + +# 1) Authority code (EPSG or ESRI) +rx.rst_setcrs("tile", "EPSG:4326") +rx.rst_setcrs("tile", "ESRI:54008") # World Sinusoidal (no EPSG code) + +# 2) Int-castable string -> treated as an EPSG/ESRI SRID (the int-cast rule) +rx.rst_setcrs("tile", "32633") # == rst_setsrid("tile", 32633) + +# 3) WKT — a full CRS definition with NO authority code (a custom projection that +# no EPSG/ESRI code names). WKT is the lossless form for such a CRS; paste an +# embedded .prj / GeoTIFF CRS verbatim. rst_crs echoes this WKT back unchanged. +rx.rst_setcrs("tile", ( + 'PROJCS["Custom_TM",' + 'GEOGCS["WGS 84",DATUM["WGS_1984",' + 'SPHEROID["WGS 84",6378137,298.257223563]],' + 'PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]],' + 'PROJECTION["Transverse_Mercator"],' + 'PARAMETER["central_meridian",13.7],' + 'PARAMETER["scale_factor",0.9996],' + 'PARAMETER["false_easting",500000],UNIT["metre",1]]' +)) + +# 4) PROJ4 string — e.g. an Albers Equal Area with custom standard parallels +# that no authority code names exactly: +rx.rst_transformcrs("tile", ( + "+proj=aea +lat_1=29.5 +lat_2=45.5 +lat_0=23 +lon_0=-96 " + "+x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs" +)) +``` + +WKT and PROJ4 are how a CRS **without** an authority code (so no SRID) is named losslessly — the reason the CRS string exists alongside the integer SRID. + +### The int-cast rule + +Everywhere GeoBrix accepts a CRS *string*, one rule decides how it is interpreted, and it lives in a single shared helper per tier (`pyrx.core.crs.resolve_crs` in the lightweight tier, `SpatialRefOps.resolveCrs` in the heavyweight tier): + +- A string that **casts cleanly to an integer** — `"4326"`, `" 32633 "` — is treated as an **EPSG SRID**. So `rst_setcrs(tile, '4326')` behaves exactly like `rst_setsrid(tile, 4326)`. +- **Otherwise** the string is parsed as a universal CRS definition — an authority code (`EPSG:4326`, `ESRI:54008`), WKT, or PROJ4 (`+proj=longlat +datum=WGS84 +no_defs`). + +This is how ESRI codes, WKT, and PROJ4 definitions all flow through the same string-taking API. + +### How an integer SRID becomes a CRS + +An integer SRID is **stored freely** — set and retrieved as any value `>= 0` (a negative SRID is rejected), whether or not it is currently a known code. Storing or reading a SRID never fails; matching the Databricks product, the interpretation and any error happen only when the SRID is **applied** to build a CRS (a reprojection, a stamp that writes CRS bytes). At that apply moment the integer is classified against the **authoritative PROJ code registries**: + +1. if the code is in the **EPSG** registry → `EPSG:`; +2. else if it is in the **ESRI** registry → `ESRI:` (e.g. `54008` = World Sinusoidal); +3. else it is **invalid** and applying it raises a clear error. + +The registries come from PROJ's `proj.db` (the SQLite database, since PROJ 6, that ships with the runtime) — they are authoritative and disjoint, so a code is classified correctly regardless of the numeric range. (This matters because the raw CRS constructors are lenient: `CRS.from_epsg(54008)` *succeeds* and would mislabel an ESRI code as EPSG — GeoBrix classifies by registry membership instead.) + +:::note EPSG / ESRI numeric collisions +A few codes exist in **both** registries. GeoBrix resolves such a code as **EPSG first**. When your data is genuinely in the ESRI CRS of a colliding code, pass it explicitly as a CRS string (`"ESRI:"`) via a `crs` argument or the reader's `geom_0_srid_proj`, rather than the bare integer. +::: + +### Canonical form + +When GeoBrix emits a CRS string — from `rst_crs`, in the tile struct's `crs` field, or in a NetCDF `crs_wkt` attribute — it uses a canonical form: the **authority string** (`AUTHORITY:CODE`, e.g. `EPSG:4326` or `ESRI:54008`) when the CRS carries one, otherwise the full **WKT**. Authority-else-WKT is more readable than PROJ4 and round-trips cleanly through both GDAL (heavyweight) and rasterio/pyproj (lightweight). + +:::note Compare CRS meaning, not the exact string +For a CRS that has an authority code (e.g. `ESRI:54008`), both tiers emit the identical authority string. For an **authority-less** CRS — one carrying only embedded WKT with no code — the two tiers can emit **different but equivalent** WKT serializations (GDAL's WKT flavor on the heavyweight tier, pyproj's on the lightweight tier). They describe the **same** CRS and compare equal as CRS objects; only the text differs. So compare CRS **meaning** (reproject/round-trip, or `SpatialReference.IsSame` / a pyproj CRS-equality check), not raw string equality. To pin a stable, identical string across tiers, stamp the CRS explicitly with `rst_setcrs` (e.g. `rst_setcrs(tile, 'ESRI:54008')`). +::: + +## Source CRS vs output CRS + +A geometry passed to a function has **two independent CRS roles**, and the parameter name tells you which: + +| Role | Parameter names | Meaning | +|---|---|---| +| **Source** | `srid` / `crs` / `clip_crs` | "my input geometry is already in this CRS" | +| **Output / target** | `out_srid` / `out_crs` | "project the output into this CRS" | + +So a bare `crs` (or `clip_crs`) declares what the input *is*; an `out_`-prefixed parameter controls how the output is *projected*. Functions that operate on an existing raster (`rst_clip`, `rst_sample`, `rst_viewshed`) take only a **source** parameter — the target is the raster's own CRS. Functions that *produce a new raster* (`rst_rasterize`, `rst_gridfrompoints`, `rst_dtmfromgeoms`, the grid `rasterize_agg` family) take an **`out_*`** parameter for the output CRS. + +Everywhere, `srid`/`out_srid` (integer) and `crs`/`out_crs` (string) are two spellings of the *same* parameter: the string form wins, and setting **both** raises. See the [master table](#crs-function-reference) for every function's parameter and role. + +### How a source CRS is resolved (per geometry) + +For any geometry input, its **source** CRS is resolved per-geometry: + +1. an **EWKB/EWKT** geometry's embedded SRID **always wins**; +2. else a plain **WKB/WKT** geometry uses the explicit `srid`/`crs`/`clip_crs` parameter; +3. else the geometry is **CRS-less** (treated as already in the target CRS). + +The explicit parameter is a per-geometry *fallback for plain WKB/WKT only*. This makes **mixed columns** first-class: a Column that mixes EWKB rows (embedded SRID) and plain-WKB rows can share one scalar `crs`/`srid` parameter — the EWKB rows keep their embedded SRID, the plain rows use the parameter, and **no row errors**. + +:::note The never-error invariant +Absent or CRS-less input **never throws** — it degrades to a sensible assumption (the geometry is treated as already in the target CRS; a CRS-less raster is assumed grid-native / basemap-less). The **only** conditions that raise are: (1) both `srid` and `crs` set, (2) both `out_srid` and `out_crs` set, or (3) an explicitly-supplied CRS string that is unresolvable. Everything else proceeds. +::: + +### Reprojection before a produce-new-raster burn + +`rst_rasterize`, `rst_gridfrompoints`, and `rst_dtmfromgeoms` reproject the input geometry from its **source** CRS into the **output** CRS (`out_crs`/`out_srid`) before burning — so a geometry in one CRS burned into an extent declared in another is correct, not garbage. When no `out_*` is given, the output carries the geometry's own source CRS (not a forced default); a CRS-less geometry is assumed already in the output CRS. + +### Grid aggregation auto-reprojects to grid-native + +`rst_h3_rastertogrid*` and `rst_quadbin_rastertogrid*` interpret pixels as EPSG:4326, and `rst_bng_rastertogrid*` as EPSG:27700. A raster carrying a **different** CRS is **auto-reprojected** to the grid-native CRS (nearest-neighbour, so pixel statistics are never interpolated) before the pixel→cell mapping — closing a prior footgun where a non-4326 raster's easting/northing were silently read as lon/lat. A CRS-less raster is assumed grid-native; supply the optional `crs` argument to declare the CRS of a CRS-less-but-known raster. + +### Performance + +CRS resolution and reprojection are cached internally: the resolved CRS objects and a thread-local, bounded pool of coordinate transformers are reused across rows (keyed by canonical CRS pair). Callers do **not** need to pre-warp rasters or batch inputs by CRS for performance — the engine reuses one transformer per CRS pair per worker thread. + +## Datum grids & grid shift + +Some coordinate transformations cannot be done with a formula alone — the offset between two datums varies from place to place and is captured in a **grid file** (an [NTv2](https://proj.org/en/stable/operations/transformations/hgridshift.html) `.gsb`, a NADCON grid, or a PROJ [`.tif` geoid/deformation grid](https://proj.org/en/stable/operations/transformations/gridshift.html)). Classic cases: **NAD27 ↔ NAD83** (US), **OSGB36 ↔ ETRS89** via `OSTN15` (Great Britain), and vertical/geoid shifts. When you reproject with `rst_transform` / `rst_transformcrs` (or a differently-CRS'd cutline / observer point), GeoBrix delegates the coordinate transform to **PROJ**, so any WKT or PROJ4 CRS that references a grid — via `+nadgrids=…`, a `+proj=hgridshift +grids=…` pipeline, or a WKT `BOUNDCRS` — is honored **as long as PROJ can find the grid file**. + +**How PROJ finds a grid.** PROJ searches its data directories (the `PROJ_DATA` / `PROJ_LIB` path — a colon-separated list) and, if enabled, the online **PROJ CDN**. GeoBrix sets `PROJ_DATA` to the bundled PROJ data directory when it is not already set, so the grids shipped with your PROJ install resolve out of the box. To enable on-demand download of missing grids from the CDN, set `PROJ_NETWORK=ON` in the executor environment. + +:::warning A missing grid degrades silently — it does not error +If a transformation's grid file is not found, PROJ falls back to a **lower-accuracy** transform (often a datum-shift approximation or a null shift) and continues — it does **not** raise. The reprojection "works" but can be off by metres. When your workflow depends on a datum grid (NADCON, OSTN15, a national grid), confirm the grid is present: PROJ emits a `proj_create: … grid … not found` warning, and `projinfo -s -t ` lists whether the chosen pipeline needs a grid you don't have. +::: + +:::note Serverless / no-egress +`PROJ_NETWORK=ON` needs outbound network access to the PROJ CDN, which is typically unavailable on Serverless. There, stage the grid files you need to a location PROJ searches instead of relying on the CDN. A GeoBrix-standardized way to reference **customer-supplied** grid files (staging to a Volume and extending the PROJ search path across both tiers) is a planned follow-up; today you can prepend your own directory to `PROJ_DATA` in the cluster/executor environment (e.g. `PROJ_DATA=/Volumes///proj-grids:/databricks/native/proj-data`). +::: + +## RasterX + +RasterX exposes both names through parallel functions. See the [RasterX Function Reference](./raster-functions) for full signatures and examples. + +**Read the CRS:** + +- [`rst_srid`](./raster-functions#rst_srid) — the stored SRID integer (an EPSG or ESRI code); `NULL`/`0` when the raster carries no authority code. +- [`rst_crs`](./raster-functions#rst_crs) — the CRS string; **always** returns a value, including for non-EPSG rasters. + +**Relabel the CRS header (no reprojection — the pixels don't move, only the label changes):** + +- [`rst_setsrid`](./raster-functions#rst_setsrid) — re-stamp from an integer SRID (an EPSG or ESRI code; `0` clears the CRS). +- [`rst_setcrs`](./raster-functions#rst_setcrs) — re-stamp from a CRS string (accepts ESRI/WKT/PROJ4; an int-castable string behaves like `rst_setsrid`). + +**Reproject (warp pixels into a new CRS):** + +- [`rst_transform`](./raster-functions#rst_transform) — reproject to an integer SRID (an EPSG or ESRI code). +- [`rst_transformcrs`](./raster-functions#rst_transformcrs) — reproject to a CRS string target (accepts non-EPSG ESRI/WKT/PROJ4). + +```python +from databricks.labs.gbx.pyrx import functions as rx # or ...rasterx — same names + +df.select( + rx.rst_srid("tile").alias("srid"), # 4326, or None for ESRI:54008 + rx.rst_crs("tile").alias("crs"), # "EPSG:4326" / "ESRI:54008" +) + +# Relabel (header only) vs reproject (moves pixels): +df.select(rx.rst_setcrs("tile", "ESRI:54008").alias("tagged")) +df.select(rx.rst_transformcrs("tile", "EPSG:3857").alias("webmercator")) +``` + +`rst_setcrs`/`rst_transformcrs` are **distinct operations, not aliases** of their integer counterparts: `rst_setcrs` relabels, `rst_transformcrs` reprojects, and each takes a string so it can name any CRS. + +### The tile struct's `crs` field + +Every raster [tile struct](./tile-structure) carries a `crs` field (and a `clip_crs` field for the clip polygon). On a **materialized** tile these are provenance — a record of the CRS already baked into the bytes. On a **[virtual tile](./virtual-tiles)** they are **instructions** — a pending target CRS applied when the tile is read, so a CRS relabel or reprojection can be carried by reference without materializing pixels early. Readers populate `crs` with the canonical CRS string, so a non-EPSG source's CRS is preserved from the moment it is read. + +### Non-EPSG round trips + +A non-EPSG CRS survives the full pipeline — **read → operate → write** — in both tiers: + +- **Read:** the reader stores the canonical CRS string (e.g. `ESRI:54008`) in `tile.crs`; `rst_crs` reads it back. +- **Operate:** identity and branch decisions compare CRS *objects* (rasterio `CRS.__eq__` / GDAL `IsSame`), not EPSG codes, so a CRS with no EPSG code takes the correct reproject-or-skip path instead of being mis-handled as "unknown". +- **Write:** the GeoTIFF and NetCDF writers persist the CRS via WKT (NetCDF stores it in the CF `crs_wkt` grid-mapping attribute) so it reads back identical. + +### Cross-tier CRS parity + +The lightweight and heavyweight tiers describe the **same** CRS for the same raster. For a CRS with an EPSG code both tiers emit the identical authority string (e.g. `EPSG:4326`). For a raw file whose embedded WKT carries **no** authority node, the two underlying libraries can produce *different but equivalent* canonical strings — rasterio/pyproj may identify the ESRI authority and emit `ESRI:54008`, while GDAL emits the equivalent WKT verbatim. These name the **same CRS** (they compare equal as CRS objects), and the decoded pixels and georeference are identical across tiers. Explicitly tagging a raster (`rst_setcrs(tile, 'ESRI:54008')`) yields the `ESRI:54008` authority string on both tiers. + +## GridX + +Each discrete global grid has a fixed native CRS, so GridX functions don't take a CRS argument — they assume the grid's CRS and reproject for you where needed: + +- **H3** and **quadbin** operate in **EPSG:4326** (WGS84 lon/lat). Raster tessellation functions interpret the raster as EPSG:4326 — reproject upstream with `rst_transform`/`rst_transformcrs` if your source differs. +- **BNG** (British National Grid) is **EPSG:27700**. BNG geometry outputs are plain WKB in EPSG:27700 with no SRID; assign the CRS when you read them back, e.g. `ST_GeomFromWKB(bng_geom, 27700)`. + +Visualization helpers such as [`plot_static`](./vizx-vector#static-maps) reproject each grid's native CRS onto the basemap automatically. + +## VectorX + +VectorX augments the product's built-in ST functions and follows the native-ST CRS conventions: + +- **Geometry input encodings** — every `gbx_st_*` geometry input accepts **WKB, EWKB, WKT, and EWKT** interchangeably. WKB/WKT carry no SRID; **EWKB/EWKT carry one** — pass whichever encoding your upstream produces. +- **SRID is applied at ingestion.** Functions that return plain WKB/WKT (e.g. `gbx_st_legacyaswkb`) carry no SRID; assign the CRS when you read the geometry back, e.g. with `ST_GeomFromWKB(wkb, srid)` or `ST_GeomFromWKT(wkt, srid)`. + +### The geometry CRS-string family + +Three functions mirror the raster family above, for geometries. They take a **CRS string** where the built-in `ST_SRID` / `ST_SetSRID` / `ST_Transform` take an integer — use the built-ins when an EPSG code is all you need, and these when the CRS can only be named as a string (an ESRI code, a WKT definition, a PROJ4 string). Both tiers register the same names. + +**Read the CRS:** + +- [`st_crs`](./vectorx-functions#st_crs) — the canonical CRS string for the geometry's embedded SRID; `NULL` for a plain WKB/WKT geometry, or for an SRID in no known registry. + +**Relabel (no reprojection — the coordinates don't move, only the label changes):** + +- [`st_setcrs`](./vectorx-functions#st_setcrs) — stamp an SRID from a CRS string. + +**Reproject:** + +- [`st_transformcrs`](./vectorx-functions#st_transformcrs) — reproject to a CRS-string target, with an optional third argument naming the source CRS for a plain SRID-less geometry. + +```sql +-- Both tiers, same names. In SQL these always return BINARY, so read the CRS +-- back with gbx_st_crs (or hand the bytes to ST_GeomFromWKB). +SELECT gbx_st_crs(geom) AS crs, + gbx_st_crs(gbx_st_setcrs(geom, 'ESRI:54008')) AS relabelled, + gbx_st_crs(gbx_st_transformcrs(geom, 'EPSG:3857')) AS webmercator +FROM geometries; +``` + +Three behaviors are worth knowing before you use them: + +- **SQL output is always `BINARY`.** `gbx_st_setcrs` and `gbx_st_transformcrs` return WKB/EWKB even when the geometry argument was a WKT or EWKT string, so one function has one declared return type and the result can be used in a view or any fixed schema. `gbx_st_crs` returns `STRING`. +- **A geometry can only carry an integer SRID.** So a target CRS with no integer authority code — a raw `PROJCS[...]` WKT, a PROJ4 string like `+proj=utm +zone=33 +datum=WGS84`, or a non-numeric code such as `OGC:CRS84` — behaves differently in the two operations: `st_transformcrs` reprojects the coordinates and **clears** the now-stale SRID (leaving it would label the geometry with a CRS it is no longer in), while `st_setcrs` **raises**, because there is no integer for it to stamp. Notably, a PROJ4 string is treated as authority-less on **both** tiers even though PROJ's fuzzy matcher could pair it with a nearby EPSG code: a geometry SRID is an exact identity claim, and a partial-confidence guess is never silently written into one. +- **Z coordinates.** A geometry whose vertices all carry a finite Z keeps its Z; a 2D geometry stays 2D. Where only *some* vertices carry a Z, the current behavior is that `st_transformcrs` reprojects the geometry as 2D — reprojecting a missing Z would propagate it into X and Y and destroy the horizontal position — while `st_setcrs` keeps the partial Z, because stamping an SRID never moves coordinates. A missing Z is never filled in with a substitute value. Note that this makes a `st_setcrs` → `st_transformcrs` chain on a partial-Z geometry return a 2D result, in every encoding and on both tiers. See [Known limitations](./vectorx-functions#crs) for that and the other edge cases (reprojection is not bit-exact; a mislabelled CRS or an out-of-domain coordinate yields `Infinity` on the lightweight tier and a projection error on the heavyweight tier; M values are dropped). + +Beyond the CRS family, `st_crs` also reads the SRID that a reader or a `gbx_st_*` generator embedded, so the same accessor works on geometries you produced elsewhere in GeoBrix. + +See the [VectorX Function Reference](./vectorx-functions#crs) for full signatures and the complete degrade table. + +## CRS function reference + +Every CRS-touching function across GeoBrix, with its CRS parameter(s) and role. **Role**: *source* = the CRS the input is in; *output* = the CRS to project the result into; *accessor* = reads/returns a CRS. Functions with an intrinsic CRS (grid-native, or the tile's own) take no CRS parameter. + +| Package | Function | Tiers | CRS param(s) | Role | Behavior | +|---|---|---|---|---|---| +| RasterX | [`rst_srid`](./raster-functions#rst_srid) | both | — | accessor | returns the stored SRID int (EPSG/ESRI) or NULL | +| RasterX | [`rst_crs`](./raster-functions#rst_crs) | both | — | accessor | returns the canonical CRS string (always) | +| RasterX | [`rst_setsrid`](./raster-functions#rst_setsrid) | both | `srid` | source | relabel from an int SRID (`0` clears; `>=0`) | +| RasterX | [`rst_setcrs`](./raster-functions#rst_setcrs) | both | `crs` | source | relabel from a CRS string | +| RasterX | [`rst_transform`](./raster-functions#rst_transform) | both | `srid` | output | reproject to an int SRID | +| RasterX | [`rst_transformcrs`](./raster-functions#rst_transformcrs) | both | `crs` | output | reproject to a CRS string | +| RasterX | [`rst_clip`](./raster-functions#rst_clip) | both | `clip_crs` | source | cutline CRS (reprojected to the tile CRS) | +| RasterX | [`rst_sample`](./raster-functions#rst_sample) | both | `crs` | source | sample-point CRS (reprojected to the tile CRS) | +| RasterX | [`rst_viewshed`](./raster-functions#rst_viewshed) | both | `crs` | source | observer-point CRS (reprojected to the tile CRS) | +| RasterX | [`rst_rasterize`](./raster-functions#rst_rasterize) (+`_agg`) | both | `out_srid` / `out_crs` | output | geom reprojected source→output before burn | +| RasterX | [`rst_gridfrompoints`](./raster-functions#rst_gridfrompoints) (+`_agg`) | both | `out_srid` / `out_crs` | output | output raster CRS (points assumed in it) | +| RasterX | [`rst_dtmfromgeoms`](./raster-functions#rst_dtmfromgeoms) (+`_agg`) | both | `out_srid` / `out_crs` | output | output raster CRS (points assumed in it) | +| RasterX | `rst_{h3,quadbin,bng}_rasterize_agg` | both | `out_srid` / `out_crs` | output | output raster CRS (bng always 27700) | +| RasterX | `rst_{h3,quadbin}_rastertogrid*` | both | `crs` | source | raster auto-reprojected to grid-native 4326 | +| RasterX | `rst_bng_rastertogrid*` | both | `crs` | source | raster auto-reprojected to grid-native 27700 | +| RasterX | `rst_h3_gridspec` | light | `out_srid` / `out_crs` | output | grid-spec output CRS (DataFrame helper) | +| GridX | `gbx_h3_cell_bbox` | both | `out_srid` / `out_crs` | output | cell bbox in the output CRS | +| VectorX | [`st_crs`](./vectorx-functions#st_crs) | both | — | accessor | returns the geometry's CRS string, or NULL when it carries no SRID | +| VectorX | [`st_setcrs`](./vectorx-functions#st_setcrs) | both | `crs` | source | relabel from a CRS string; raises when the CRS has no integer authority code | +| VectorX | [`st_transformcrs`](./vectorx-functions#st_transformcrs) | both | `target_crs`, `source_crs` | output (+ source) | reproject to a CRS string; `source_crs` names the input CRS for a plain SRID-less geometry | +| RasterX | GDAL/GTiff reader `clipCrs` option | both | `clipCrs` | source | stamps the v2 tile `clip_crs` field | +| VizX | [`plot_tile`](./vizx-raster) / [`plot_cog`](./vizx-raster) | light | `crs` | source | basemap CRS; override for a CRS-less raster | + +:::note Geometry SRIDs are integers +The three VectorX rows take a CRS **string**, but a geometry can only *carry* an integer SRID. So a target CRS with no integer authority code (raw WKT, PROJ4, or a non-numeric code such as `OGC:CRS84`) makes `st_transformcrs` clear the stale SRID while `st_setcrs` raises. Their SQL forms always return `BINARY`. See [the geometry CRS-string family](#vectorx) for the full rules. +::: + +:::note GridX CRS surface +The broader **GridX** CRS surface (custom-CRS input reprojection for `polyfill` / `tessellate` / `pointascell`, grid CRS accessors) is a separate follow-on; its rows will be added here when those functions ship. Each grid's fixed native CRS is documented in [GridX](#gridx) above. +::: diff --git a/docs/docs/api/error-handling.mdx b/docs/docs/api/error-handling.mdx new file mode 100644 index 000000000..0a8cceb74 --- /dev/null +++ b/docs/docs/api/error-handling.mdx @@ -0,0 +1,177 @@ +--- +sidebar_position: 6 +title: Error Handling +--- + +# Error Handling + +GeoBrix distinguishes a **bad parameter** from **bad data**. + +- A **bad or non-executable parameter** — an invalid CRS code, a malformed argument, a string that cannot be resolved — raises a clear error. It is a fix-your-code problem and fails fast so you catch it during development. +- **Bad data flowing through a column** — a corrupt geometry, a coordinate outside the target CRS's valid area — degrades to `NULL` (or an empty result) rather than failing the whole job. One bad row never kills the stage. + +This separation keeps large-scale spatial pipelines reliable: parameter mistakes surface immediately, while data quality issues in individual records are isolated and observable. + +## RasterX + +RasterX has four result shapes, and each expresses the bad-data-degrades rule in a way that fits its shape. + +### Scalar accessors + +Functions that read a single property from a raster tile (band count, CRS, pixel type, and similar) return `NULL` when the tile is corrupt or unreadable. The rest of the column is unaffected. + +```sql +SELECT path, gbx_rst_bandcount(tile) AS bands +FROM my_rasters +-- rows with a corrupt tile produce NULL for `bands`; the query does not fail +``` + +### Tile operations + +Functions that return a raster tile (transforms, resampling, band math) return an **empty tile** when they encounter bad data. An empty tile is a valid struct — it carries no pixel data, but its `metadata` map contains an `error_message` key that describes what went wrong. + +```sql +SELECT path, + gbx_rst_retile(tile, 256, 256) AS retiled, + gbx_rst_retile(tile, 256, 256).metadata['error_message'] AS err +FROM my_rasters +-- rows that could not be retiled produce an empty tile; err is non-NULL for those rows +``` + +To isolate problem rows: + +```sql +SELECT path, err +FROM ( + SELECT path, + gbx_rst_retile(tile, 256, 256).metadata['error_message'] AS err + FROM my_rasters +) +WHERE err IS NOT NULL +``` + +### Aggregators + +Aggregation functions (mosaic, union, merge) skip any corrupt member tile. The aggregate continues over the remaining valid members. A stage-level error is not raised. + +### Generators + +Functions that expand a single tile into multiple rows (tessellation, tiling, pyramid generation) emit one error row when a tile fails. The error row's tile column holds an empty tile with `error_message` set; it can be filtered or audited like any other row. + +### Flipping data errors to hard failures for debugging + +The Spark configuration key `spark.databricks.labs.gbx.expressions.crash.on.error` makes RasterX data-level errors raise hard failures instead of degrading. Use it when you want stack traces during development or when diagnosing unexpected `NULL`s or empty tiles in a pipeline. + +```python +spark.conf.set( + "spark.databricks.labs.gbx.expressions.crash.on.error", + "true" +) +``` + +Set it back to `false` (the default) before running production workloads. + +This switch applies **only to the heavyweight (JVM) RasterX expressions**. It is read by the Scala tile expressions and has no effect on the lightweight Python tier (`pyrx`), which does not consult Spark configuration, nor on VectorX (see below), which does not have an equivalent switch. + +## VectorX + +VectorX functions work on geometry columns (WKB, EWKB, WKT, or EWKT). They have no metadata carrier, so `NULL` is the single degrade signal for bad-data conditions. VectorX has no crash-on-error switch; the RasterX configuration key above does not affect it. + +### Bad geometry data + +If the input geometry is corrupt, unparseable, or produces a non-finite result, the function returns `NULL`. Other rows are unaffected. + +```sql +SELECT id, gbx_st_transformcrs(geom, 'EPSG:3857') AS geom_mercator +FROM my_table +-- rows with an unparseable geometry produce NULL; the query continues +``` + +### Bad CRS argument + +If the CRS argument cannot be resolved — an unrecognised authority code, an empty string, a value that is not a valid CRS — the function raises an error. This is a parameter problem: the CRS string is a constant in your query, not data flowing through the column. + +```sql +-- This raises an error: "FAKE:9999" is not a valid CRS. +SELECT gbx_st_transformcrs(geom, 'FAKE:9999') FROM my_table +``` + +Fix the CRS string; do not expect this to degrade quietly. + +### Reprojection domain check + +`gbx_st_transformcrs` checks whether each geometry falls within the target CRS's valid area. A geometry that lies outside that area — for example, a point at longitude 0° being reprojected into a CRS whose valid area covers only the eastern United States — returns `NULL` rather than producing a silently nonsensical coordinate. A geometry that straddles the boundary also returns `NULL`, erring on the side of correctness. + +Reprojections that fell outside the target CRS's valid area are returned as `NULL`. To find them: + +```sql +SELECT id +FROM ( + SELECT id, gbx_st_transformcrs(geom, 'EPSG:27700') AS projected + FROM my_table +) +WHERE projected IS NULL +``` + +When the target CRS carries no declared area of use, the domain check is skipped and the reprojection proceeds without a spatial guard. + +## GridX + +GridX functions (BNG, Quadbin, Custom) return cell ids, geometries, arrays, or structs — there is no metadata carrier — so `NULL` is the single degrade signal for bad-data conditions, the same pattern as VectorX. GridX has no crash-on-error switch. + +### Bad cell-id or geometry data + +A malformed cell id (an unrecognised BNG grid-square letter pair, a cell string that cannot be decoded) or an unparseable geometry returns `NULL`. Other rows are unaffected; one bad cell id never fails the stage. + +```sql +SELECT id, gbx_bng_aswkb(cellid) AS geom +FROM my_cells +-- rows with a malformed cellid produce NULL for `geom`; the query continues +``` + +**Aggregators** (`gbx_bng_cellunion_agg`, `gbx_bng_cellintersection_agg`) skip a corrupt member and continue over the remaining valid cell ids. + +**Generators** (`gbx_bng_kringexplode`, `gbx_bng_tessellateexplode`) emit zero rows for a bad input cell rather than a single NULL row. An inner join against a generator silently drops those inputs; if you need to surface which inputs produced nothing, join the generator's output back against your source table using an anti-join or left join on the original id. + +### Bad resolution or grid argument + +An out-of-range resolution, an unrecognised resolution string, or an invalid custom-grid specification raises an error rather than returning `NULL`. This is a parameter problem: the resolution is a constant in your query, not per-row data, so the error surfaces immediately during development rather than silently degrading at scale. + +```sql +-- This raises an error: 99 is not a valid BNG resolution index. +SELECT gbx_bng_pointascell(geom, 99) FROM my_table +``` + +Fix the resolution argument; do not expect this to degrade quietly. + +### Quadbin latitude clamp + +`gbx_quadbin_pointascell` follows the web-mercator convention: a latitude beyond ±85.05112878° is clamped to that limit, and a longitude beyond ±180° is clamped to ±180°. The function returns a real cell rather than `NULL`. A point at latitude 89° yields the same cell as one at 85.05112878°. + +This behaviour is intentional and differs from BNG and Custom, which return `NULL` for a coordinate outside their valid extent. + +## Catching degraded rows + +**VectorX / scalar accessors:** filter on `IS NOT NULL`: + +```sql +SELECT * +FROM results +WHERE geom IS NOT NULL +``` + +**RasterX tile operations:** filter on the `error_message` metadata key: + +```sql +SELECT * +FROM results +WHERE tile.metadata['error_message'] IS NULL +``` + +Or audit the errors: + +```sql +SELECT path, tile.metadata['error_message'] AS err +FROM results +WHERE tile.metadata['error_message'] IS NOT NULL +``` diff --git a/docs/docs/api/execution-tiers.mdx b/docs/docs/api/execution-tiers.mdx index 84aa10c87..bc286a041 100644 --- a/docs/docs/api/execution-tiers.mdx +++ b/docs/docs/api/execution-tiers.mdx @@ -4,9 +4,19 @@ sidebar_position: 1 # Choosing an Execution Tier -GeoBrix raster functions come in two interchangeable **Execution Tiers** — **Lightweight (pyrx)** and **Heavyweight (rasterx)**. Both tiers use the same `rst_*` Python function names and `gbx_rst_*` SQL names, so switching between them is a one-line import change. +**One API, two engines.** Every GeoBrix raster function exists in two interchangeable +**Execution Tiers**, behind the *same* `rst_*` Python names and `gbx_rst_*` SQL names: -Choose the tier that fits your environment and requirements; the rest of your code stays the same. +- **Lightweight (`pyrx`)** — pure Python/rasterio, no JAR, no init script. Runs **everywhere**: + Serverless, standard/shared clusters, ARM, Lakeflow declarative pipelines. Works with bytes-free + **[virtual tiles](./virtual-tiles)** so huge rasters ingest without out-of-memory. **The + recommended default.** +- **Heavyweight (`rasterx`)** — JVM-native (Scala + GDAL JNI) on a classic x86 cluster. Operates on + **materialized** (binary) tiles. + +Both tiers cover the **full raster function set** and agree within tolerance across the benchmark +suite — so switching is a **one-line import change** and the rest of your code is identical. Pick the +tier that fits your environment; the sections below cover what each provides and how to choose. ## The one-line swap @@ -24,6 +34,15 @@ After an explicit `rx.register(spark)`, the SQL names are identical too (`gbx_rs The one-line *import* swap is symmetric, but the *install* is not. The **lightweight** tier is just the `[light]` wheel (`%pip`, no JAR, no init script). The **heavyweight** tier additionally requires the **GeoBrix JAR as a cluster library and the GDAL init script** on a **classic x86 cluster** — the wheel alone will not resolve the import or the JVM expressions. See [Installation](../installation) for the heavyweight setup. ::: +## Function availability + +Both tiers implement the **full raster set** — every `rst_*` function, including the BNG and quadbin +raster-grid functions — so the tier choice is about *environment*, not *capability*. The remaining +heavyweight-only surfaces are narrow: the vector **OGR readers** (`shapefile_ogr`, `geojson_ogr`, …), +the `conforming` triangulation mode, and the heavy `pmtiles` DataSource writer. For the per-function +breakdown see the [Raster Functions](./raster-functions) availability section; for the tier tile +model (virtual vs. materialized), see [Virtual Tiles](./virtual-tiles). + ## Registering a subset (`only=`) `register()` installs every `gbx_*` SQL name for the tier. To register just the functions a session uses, pass `only=` (lightweight tiers — `pyrx`, `pygx`, `pyvx`): @@ -57,7 +76,7 @@ heavy.register(spark) # all heavy gbx_rst_* light.register(spark, only=["rst_slope"]) # gbx_rst_slope now lightweight ``` -The reverse — re-registering a few **heavy** functions over a lightweight session — is not yet available; `only=` is currently a lightweight-tier feature (heavy registers its full set). Mixing works because both tiers use the same tile struct and GTiff payload, so a tile produced by one tier flows into a function from the other. +The reverse — re-registering a few **heavy** functions over a lightweight session — is not yet available; `only=` is currently a lightweight-tier feature (heavy registers its full set). Mixing works for **materialized** tiles (raster bytes present) — both tiers share the same GTiff payload, so a bytes-carrying tile produced by one tier flows into a function from the other. A lightweight **virtual** tile (bytes-free path+window) must be materialized before a heavyweight function can use it; see [Virtual tiles and the light→heavy bridge](#light-heavy-bridge). ## Tradeoffs @@ -70,6 +89,7 @@ The lightweight raster tier is, in effect, [distributed rasterio](./rasterio-dis | ARM support | x86 only | x86 and ARM | | Serverless / shared clusters / Lakeflow SDP | Not supported | Supported | | Execution model | JVM-native (Scala + GDAL JNI) | Python-worker UDFs (rasterio + NumPy) | +| Tile model | Materialized (binary) tiles only | Materialized **and** bytes-free **[virtual tiles](./virtual-tiles)** (lazy windowed reads; no ingest OOM) | | Driver coverage | Full custom GDAL build | rasterio's bundled build (narrower) | | SQL default arguments | Supported | Pass all arguments explicitly | | Function coverage | Full raster set | Full raster set — every `rst_*` function, including the BNG/quadbin raster-grid functions | @@ -104,6 +124,49 @@ On per-operation timing the lightweight tier is **competitive-to-faster for the Across the benchmark suite the two tiers agree within tolerance on **115 of 116** functions. Only one differs, at raster edges: **`rst_convolve`** differs slightly — the heavyweight tier applies a GDAL block-halo convolution that no single lightweight boundary mode reproduces exactly; interior values match. All three tessellation generators (`rst_h3_tessellate`, `rst_quadbin_tessellate`, `rst_bng_tessellate`) use a **positive-area covering keep-test** on both tiers: a cell is emitted iff its geometry has greater-than-zero area overlap with the raster — a cell that merely touches the raster along a boundary edge or corner (zero pixel overlap) is excluded, while a within-extent cell whose pixels are all NoData is still emitted (NoData renders in place, it does not punch a gap into the mosaic). With that shared semantic the three tessellators agree **exactly** across tiers (identical cell sets), including on grid-aligned tiles where the raster edges land on cell boundaries. The BNG and quadbin raster-grid **reducers** (`rst_bng_rastertogrid{avg,count,max,min,median,sum,variance,stddev}`) were measured on a British-National-Grid tile with real cells and agree **within tolerance** across tiers; the two `*_rasterize_agg` aggregators agree **exactly**. See **[Benchmarking](./benchmarking)** for the full per-function heavy-vs-light results and how to run the benchmark on a cluster or locally. -## Function availability +## Virtual tiles and the light→heavy bridge {#light-heavy-bridge} + +**The lightweight tier is for light (virtual) raster tiles; the heavyweight tier is for heavy +(binary) raster tiles.** Both tiers share the same v2 tile struct — the difference is in the +`raster` field: a virtual tile has `raster = null` (bytes-free, path + window backed); a +materialized tile carries raster bytes (`raster` is not null). + +The **heavyweight tier** accepts both v1 and v2 **materialized** tiles as input and always emits +the v2 tile struct. Passing a **virtual** tile (bytes-free, path-backed) to a heavyweight function +raises a clear error telling you to materialize it in the lightweight tier first — either with +`materialize=True`, or by writing via any raster writer and reading the output back. The JVM cannot +lazily read from a Unity Catalog Volume FUSE path the way a Python worker can, so the materialization +step is required before crossing the tier boundary. -See the [Raster Functions](./raster-functions) availability section for what each tier provides. +**If a heavy function does not produce the expected result from a light-tier tile**, the tile may be +virtual (no bytes). You have two options: + +- **Stay in the lightweight tier.** For most raster work, `pyrx` covers the full function set and runs + everywhere (serverless, standard clusters, ARM). If you don't specifically need the JVM execution + model, there is no reason to cross to heavy. +- **Materialize before crossing.** If you do need a heavyweight function: call a tile-returning + lightweight function with `materialize=True` to produce a bytes-carrying tile, or write via any + raster writer (`raster_gbx`, `gtiff_gbx`, `cog_gbx`) and read the output back — every writer is a + materialization boundary. + +The lightweight-tier `rst_*` functions accept three optional force-output params (`virtualize_dir`, +`virtualize_prefix`, `materialize`) to control this; the heavyweight tier has none of them. See +[Virtual-tile force-output params](./raster-functions#virtual-tile-overrides) for the full param +reference. + +### Virtual↔materialized advice {#virtual-materialized-advice} + +When working with virtual tiles in the lightweight tier, the key question is: **which operations need +pixels, and which can stay lazy?** + +| Operation type | Behavior under auto | Notes | +|---|---|---| +| **Metadata accessors** (`rst_width`, `rst_height`, `rst_srid`, `rst_boundingbox`, `rst_format`, `rst_georeference`, `rst_metadata`, `rst_rotation`, `rst_scalex`, `rst_scaley`, `rst_skewx`, `rst_skewy`, `rst_numbands`, `rst_type`, `rst_getnodata`, `rst_upperleftx`, `rst_upperlefty`) | Free on virtual tiles — no pixels read | The header is opened lazily; `.read()` is never called | +| **Pixel accessors / stats** (`rst_avg`, `rst_min`, `rst_max`, `rst_median`, `rst_pixelcount`, `rst_summary`, `rst_histogram`, `rst_sample`, `rst_isempty`) | Read the window (transient materialize), return scalars/arrays | Pixels are materialized for the computation only; no bytes in the output row | +| **Reference / passthrough tile ops** (`rst_clip`, `rst_setsrid`, `rst_initnodata`, `rst_band`, **identity `rst_transform`** where target CRS == source CRS) | Reference/passthrough class — record instructions or clip references; no new pixels read | `rst_initnodata`, `rst_setsrid`, and `rst_band` record a pending instruction on the virtual tile and stay bytes-free; the instruction is applied at the next read. `rst_clip` and identity `rst_transform` record a region reference. None of these ops produce new pixels on a virtual tile. `virtualize_dir` has no meaningful effect on them. | +| **Pixel-producing tile ops** (slope, aspect, hillshade, terrain, focal, mapalgebra, spectral indices, rasterize, resample, **non-identity `rst_transform`**, **`rst_merge` / `rst_combineavg` / `rst_frombands`**) | Materialize and return bytes | Pass `virtualize_dir` to write the computed result to a durable path and get a light virtual row — the **only** way these return a virtual tile | +| **Writers** (`raster_gbx`, `gtiff_gbx`, `cog_gbx`) | Always a materialization boundary | A virtual DataFrame is directly writable — writers auto-materialize; `cog_gbx` converts whole-file virtual tiles path-direct (no bytes round-trip) | +| **Crossing to heavy** | Materialize first | Heavy consumes only materialized tiles — stay in light (recommended), or materialize first (`materialize=True`, or write + read back) before handing off | + +**Short rule:** chain deferrable ops as long as you like; when you need pixels (or need to cross to +heavy), materialize at that point — not before. diff --git a/docs/docs/api/gridx-functions.mdx b/docs/docs/api/gridx-functions.mdx index e48f2a1c6..3b7dc5cde 100644 --- a/docs/docs/api/gridx-functions.mdx +++ b/docs/docs/api/gridx-functions.mdx @@ -3,11 +3,14 @@ sidebar_position: 6 title: GridX Function Reference --- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; import CodeFromTest from '@site/src/components/CodeFromTest'; +import FunctionExamples from '@site/src/components/FunctionExamples'; import gridxFunctionsExamples from '!!raw-loader!../../tests/python/api/gridx_functions.py'; import gridxFunctionsSqlExamples from '!!raw-loader!../../tests/python/api/gridx_functions_sql.py'; -import packagesExamples from '!!raw-loader!../../tests/python/packages/examples.py'; -import gridxScalaCode from '!!raw-loader!../../tests/scala/packages/GridxPackageExamples.scala'; +import gridxLightCode from '!!raw-loader!../../tests/python/api/gridx_functions_python_light.py'; +import scalaApiExamplesCode from '!!raw-loader!../../tests/scala/api/ScalaApiExamples.scala'; import Tier from '@site/src/components/Tier'; import { Impl } from '@site/src/components/Tier'; import GridXIcon from '../../../resources/images/brand/GridX.png'; @@ -42,50 +45,54 @@ Use `RegisterBatch` with `functions=gridx.quadbin` or `functions=gridx.bng` to r - **K-Ring / K-Loop Neighbourhoods**: Filled rings and hollow rings for both grid systems - **Polyfill and Tessellation**: Cover geometries with cells; tessellation returns per-cell clipped chip geometries -## Usage Examples +## Setup {#setup} -### Python/PySpark +With GeoBrix already [installed](../installation), register GridX in your session before running any example. - + + -### Scala +```python +from databricks.labs.gbx.pygx import functions as gx +gx.register(spark) # registers all gbx_bng_*, gbx_quadbin_*, gbx_custom_* SQL functions +``` - + + -### SQL +```python +# BNG +from databricks.labs.gbx.gridx.bng import functions as bx +bx.register(spark) - +# Quadbin +from databricks.labs.gbx.gridx.quadbin import functions as qx +qx.register(spark) -:::note SQL function prefixes -In SQL, GridX functions are prefixed with **`gbx_`** (e.g. `gbx_bng_aswkb`, `gbx_quadbin_pointascell`). For SQL, Python, and Scala usage patterns, see [Language Bindings](./language-bindings). -::: +# Custom grid +from databricks.labs.gbx.gridx.custom import functions as cx +cx.register(spark) +``` + + + -## Common setup +The examples on this page read from **ten canonical DataFrames**, one per fixture group. Each is available as a temp view for SQL examples. Point the placeholders at your own data to reproduce the examples: -Run this once before the examples below. It registers GridX (BNG) so you can use `bng_*` in Python and `gbx_bng_*` in SQL. +| View / DataFrame | Schema | Backed by | Backs | +|---|---|---|---| +| `bng_cells` | `cellid STRING` | Inline literal `'TQ3080'` (1km cell, central London, EPSG:27700) | BNG scalar ops: `bng_aswkb`, `bng_aswkt`, `bng_cellarea`, `bng_centroid`, `bng_kring`, `bng_kloop` | +| `bng_cell_pairs` | `cellid1 STRING`, `cellid2 STRING` | Inline literals `'TQ3080'`, `'TQ3081'` (adjacent cells, distance = 1) | `bng_distance`, `bng_euclideandistance` | +| `bng_points` | `easting INT`, `northing INT`, `geom STRING` | Easting 530000, northing 180000, WKT point (EPSG:27700) | `bng_pointascell`, `bng_eastnorthasbng` | +| `bng_polygons` | `geom STRING` | 3km × 3km BNG polygon in **EPSG:27700** (London area) | `bng_geomkring`, `bng_geomkloop`, `bng_polyfill`, `bng_tessellate`, and the 5 `*explode` generators | +| `bng_chips` | `chip STRUCT` | 9 tessellation chips from the BNG polygon at resolution 3 | `bng_cellintersection`, `bng_cellunion`, `bng_cellintersection_agg`, `bng_cellunion_agg` | +| `quadbin_cells` | `cell LONG` | Inline literal `5233961839712272383` (San Francisco, zoom 10) | `quadbin_aswkb`, `quadbin_centroid`, `quadbin_resolution`, `quadbin_kring`, `quadbin_cellunion` | +| `quadbin_cell_pairs` | `cell1 LONG`, `cell2 LONG` | Two cells at `(0.0, 0.0, z10)` and `(0.0, 0.1, z10)` — distance = 1 | `quadbin_distance` | +| `quadbin_polygons` | `geom STRING` | WGS84 polygon `(-1,-1) → (1,1)` | `quadbin_polyfill`, `quadbin_tessellate` | +| `quadbin_kring_cells` | `cell LONG` | 9 cells from `kring(SF-z10, k=1)` | `quadbin_cellunion_agg` | +| `custom_grids` | `grid STRUCT`, `cell LONG`, `point STRING` | BNG-like custom grid; cell at `POINT(530000 180000)`, resolution 5 | All 7 `custom_*` functions | - +All ten views are built from inline literals in the doc-test fixture helpers (no external files or `/Volumes` dependency, so the examples run anywhere). :::tip Map any GridX cell set with `plot_static` [`vizx.plot_static`](./vizx-vector#static-maps) renders a column of cell ids straight from a Spark DataFrame as a **static, GitHub-renderable map over a basemap** — and it's the one helper that covers **every** GridX grid: pass `grid_system="quadbin"`, `"bng"`, or `"custom"` (as well as `"h3"`). Each grid's native CRS is handled for you (e.g. BNG's EPSG:27700 is reprojected for the basemap), and `"custom"` takes the same grid spec via `grid_conf=`. It's the quickest way to eyeball the cells the functions below produce. See [`plot_static`](./vizx-vector#static-maps) for the full parameter list. @@ -93,6 +100,78 @@ Run this once before the examples below. It registers GridX (BNG) so you can use --- +## Examples — Conventions {#conventions} + +### How to read the four tabs + +Every function on this page shows **one example**, expressed identically across four tabs: + +| Tab | Tier | Badge | +|---|---|---| +| **SQL** | Both (default) | — | +| **Python (light)** | `pygx` lightweight tier | — | +| **Python (heavy)** | heavyweight tier | Blue | +| **Scala** | heavyweight tier | Blue | + +All four tabs operate on the **same input fixture** with the **same arguments**. Where a genuine tier difference exists — a diverging output schema or a behavior available only in one tier — the affected tab carries a labeled `:::note`. A difference without a label is a documentation error. + +In each Python example, `df = spark.table("")` or an equivalent inline `spark.sql(...)` call loads the canonical fixture. Each SQL example reads `FROM ` or uses an inline literal — no separate `CREATE TEMP VIEW` step is shown. + +### Output representation + +The output cells in every function table follow a uniform convention so readers can compare tabs at a glance. + +**Cell-id strings** — BNG cell-ids (e.g. `'TQ3080'`) and quadbin cells (e.g. `5233961839712272383`) are shown in full when short. + +**Binary geometry ([E]WKB)** — geometry returned as `BINARY` is elided with one token and a format annotation. Use `(WKB binary)` when no SRID is embedded; `(EWKB binary)` when an SRID is embedded (quadbin functions embed SRID 4326): + +``` +... (WKB binary) +... (EWKB binary) +``` + +**WKT strings** — short WKT strings are shown in full; longer strings are truncated with a type annotation: + +``` +POLYGON ((530000 180000, ...)) +POLYGON (... (WKT) +``` + +**Cell-id arrays** — arrays of cell-ids show the first few entries followed by `...` when the array is long: + +``` +['TQ2979', 'TQ2980', 'TQ2981', ..., 'TQ3181'] (9 cells) +``` + +**Doubles / numeric scalars** — shown as the real value (e.g. `1.0` for `bng_cellarea`). + +**Cell width** — output cells are capped at approximately 60 characters. A longer value is truncated with `...` and annotated with its type. + +**Identical-across-tier values** — when all four tabs produce the same result (e.g. `'TQ3080'` from `bng_pointascell`), each tab shows that value identically with the same annotation. Genuine tier differences are called out in a labeled `:::note`. + +### BNG domain rules + +The following rules apply to **all BNG functions** (`gbx_bng_*`): + +- **Coordinate system** — BNG geometry inputs (points and polygons) must be in **EPSG:27700** eastings/northings. WGS84 lon/lat coordinates (e.g. `POINT(-0.1278 51.5074)`) yield **empty arrays or null results** — this is the most common source of confusion. +- **Resolution** — accepts **integer indices ±1..±6** (1 = 100km, 2 = 10km, 3 = 1km, 4 = 100m, 5 = 10m, 6 = 1m; negative values are quadrants) **or** string keys from the resolution map (e.g. `'1km'`, `'100m'`, `'10m'`). **Never** pass metres-as-integer (e.g. `1000`) — that interpretation is not supported by `BNG.getResolution` and raises an error. +- **`bng_cellarea` returns square kilometres** — not square metres. A 1km cell returns `1.0`. +- **Cell-id format** — standard BNG grid references (e.g. `'TQ3080'`). At resolution 3 (1km), the format is two letters + four digits; the resolution level is encoded in the string length. + +### BNG `*explode` generators — SQL LATERAL only + +The five BNG explode generators (`bng_kringexplode`, `bng_kloopexplode`, `bng_geomkringexplode`, `bng_geomkloopexplode`, `bng_tessellateexplode`) are Python UDTFs in the lightweight tier — they have **no Python DataFrame Column form**. Calling them as Column expressions raises `NotImplementedError`. Use SQL `LATERAL` for both tiers: + +```sql +SELECT t.* +FROM (SELECT 'TQ3080' AS cellid, 1 AS k) src, +LATERAL gbx_bng_kringexplode(src.cellid, src.k) t +``` + +The Python (light) tab for these functions shows the same `spark.sql("... LATERAL ...")` invocation. + +--- + ## Quadbin (CARTO v0) Quadbin runs in both tiers. The lightweight tier is the `pygx` quadbin package — powered by the **quadbin** library + shapely; `quadbin_distance` / `quadbin_polyfill` cell math mirrors the heavyweight `Quadbin.scala`, and geometry outputs are EWKB SRID 4326 — identical `gbx_quadbin_*` SQL names, so it is a drop-in swap. @@ -119,14 +198,19 @@ Convert a lon/lat coordinate (EPSG:4326) to the quadbin cell containing it at th Powered by the **quadbin** package (numpy-vectorized encoder). Encodes WGS84 lon/lat to a quadbin cell at the given zoom; bit-identical to the heavyweight `Quadbin.scala` (incl. the antimeridian/pole tile clamp). ::: -**Signature:** `quadbin_pointascell(lon: Column, lat: Column, zoom: Column): Column` +**Signature:** `quadbin_pointascell(longitude: Column, latitude: Column, zoom: Column): Column` **Returns:** - `BIGINT` quadbin cell ID -**SQL:** - - + --- @@ -145,9 +229,14 @@ Powered by the **quadbin** package + **shapely**. Cell boundary polygon as EWKB **Returns:** - Binary EWKB polygon (SRID-tagged 4326) -**SQL:** - - + --- @@ -166,9 +255,14 @@ Powered by the **quadbin** package + **shapely**. Cell centroid (bbox-corner mea **Returns:** - Binary EWKB point (SRID-tagged 4326) -**SQL:** - - + --- @@ -187,9 +281,14 @@ Powered by the **quadbin** package (numpy-vectorized). Extracts the zoom level f **Returns:** - `INT` zoom level (0..26) -**SQL:** - - + --- @@ -208,9 +307,14 @@ Powered by the **quadbin** package + **shapely**. Enumerates the cells covering **Returns:** - `ARRAY` of cell IDs covering the bbox -**SQL:** - - + --- @@ -229,9 +333,14 @@ Powered by the **quadbin** package. All cells within Chebyshev distance `k` (inc **Returns:** - `ARRAY` of cell IDs (length `(2k+1)^2`) -**SQL:** - - + --- @@ -250,9 +359,14 @@ Powered by the **quadbin** package + **shapely**. Bbox polyfill then per-cell in **Returns:** - `ARRAY>` -**SQL:** - - + --- @@ -271,9 +385,14 @@ Powered by **shapely** (`union_all` of the cell polygons). Dissolves an `ARRAY + --- @@ -292,9 +411,14 @@ Powered by **shapely**. Grouped aggregate — `groupBy(...).agg(gx.quadbin_cellu **Returns:** - `BINARY` EWKB multipolygon (SRID-tagged 4326) representing the dissolved coverage -**SQL:** - - + --- @@ -313,9 +437,14 @@ Powered by the **quadbin** package. Chebyshev distance on tile coordinates; mirr **Returns:** - `INT` cell-step distance -**SQL:** - - + --- @@ -383,9 +512,14 @@ Powered by a **pure-Python port of `BNG.scala`** + **shapely**. The cell footpri **Returns:** - Binary WKB geometry representation -**SQL:** - - + --- @@ -407,9 +541,14 @@ Powered by a **pure-Python port of `BNG.scala`** + **shapely**. The cell footpri **Returns:** - String WKT geometry representation -**SQL:** - - + --- @@ -435,9 +574,14 @@ Powered by the **pure-Python port of `BNG.scala`**. Returns the cell area in **s **Returns:** - Double representing the cell area in square kilometres -**SQL:** - - + --- @@ -459,9 +603,14 @@ Powered by the **pure-Python port of `BNG.scala`** + **shapely**. Cell-center PO **Returns:** - Point geometry at cell center -**SQL:** - - + --- @@ -469,24 +618,29 @@ Powered by the **pure-Python port of `BNG.scala`** + **shapely**. Cell-center PO -Calculate distance between two BNG cells. +Return the grid-step distance between two BNG cells. :::note Lightweight tier (pygx) -Powered by the **pure-Python port of `BNG.scala`** — grid-step distance between two cells, exact-parity with the heavyweight tier. +Powered by the **pure-Python port of `BNG.scala`** — Chebyshev grid-step count between two cells, exact-parity with the heavyweight tier. Returns **LONG** (grid steps, not metres). ::: -**Signature:** `bng_distance(cell1: Column, cell2: Column): Column` +**Signature:** `bng_distance(cellid1: Column, cellid2: Column): Column` **Parameters:** -- `cell1` - First BNG cell reference -- `cell2` - Second BNG cell reference +- `cellid1` - First BNG cell reference (STRING) +- `cellid2` - Second BNG cell reference (STRING) **Returns:** -- Double representing distance in meters - -**SQL:** - - +- LONG — Chebyshev grid-step distance. Adjacent cells (sharing an edge or corner) return 1; the cell itself returns 0. + + --- @@ -494,53 +648,67 @@ Powered by the **pure-Python port of `BNG.scala`** — grid-step distance betwee -Calculate Euclidean distance between two BNG cells. +Return the Chebyshev grid-unit distance between two BNG cells. :::note Lightweight tier (pygx) -Powered by the **pure-Python port of `BNG.scala`** — straight-line distance between two cell centroids, exact-parity with the heavyweight tier. +Powered by the **pure-Python port of `BNG.scala`** — Chebyshev distance in grid units, exact-parity with the heavyweight tier. Returns **LONG** (grid units, not metres). ::: -**Signature:** `bng_euclideandistance(cell1: Column, cell2: Column): Column` +**Signature:** `bng_euclideandistance(cellid1: Column, cellid2: Column): Column` **Parameters:** -- `cell1` - First BNG cell reference -- `cell2` - Second BNG cell reference +- `cellid1` - First BNG cell reference (STRING) +- `cellid2` - Second BNG cell reference (STRING) **Returns:** -- Double representing Euclidean distance in meters - -**SQL:** - - +- LONG — Chebyshev distance in grid units. Adjacent cells return 1; diagonal cells also return 1. + + --- ## Cell Operations -Operations combining multiple cells. +Operations combining multiple BNG chip structs. + +:::warning Chip-struct inputs required +`bng_cellintersection` and `bng_cellunion` take **`STRUCT`** chip inputs — the same struct produced by `bng_tessellate` / `bng_tessellateexplode`. Passing plain `STRING` cell IDs throws `ClassCastException`. Use `bng_tessellate(geom, resolution)` to produce chip inputs. +::: ### bng_cellintersection -Get the intersection of two BNG cells. +Intersect two BNG chip structs and return the dissolved intersection chip. :::note Lightweight tier (pygx) -Powered by the **pure-Python port of `BNG.scala`** + **shapely**. Intersects two cell chips, returning the dissolved chip. +Powered by the **pure-Python port of `BNG.scala`** + **shapely**. Returns the same `STRUCT` as the heavyweight tier (AGREE). ::: -**Signature:** `bng_cellintersection(cell1: Column, cell2: Column): Column` +**Signature:** `bng_cellintersection(chip1: Column, chip2: Column): Column` **Parameters:** -- `cell1` - First BNG cell reference -- `cell2` - Second BNG cell reference +- `chip1` - First chip struct (`STRUCT`) +- `chip2` - Second chip struct (same schema) **Returns:** -- BNG cell ID representing the intersection - -**SQL:** - - +- `STRUCT` — dissolved intersection chip. A fully interior chip (core=true) intersected with itself returns `{cellid, true, null}`. + + --- @@ -548,24 +716,29 @@ Powered by the **pure-Python port of `BNG.scala`** + **shapely**. Intersects two -Get the union of two BNG cells. +Union two BNG chip structs and return the dissolved union chip. :::note Lightweight tier (pygx) -Powered by the **pure-Python port of `BNG.scala`** + **shapely**. Unions two cell chips, returning the dissolved chip. +Powered by the **pure-Python port of `BNG.scala`** + **shapely**. Returns the same `STRUCT` as the heavyweight tier (AGREE). ::: -**Signature:** `bng_cellunion(cell1: Column, cell2: Column): Column` +**Signature:** `bng_cellunion(chip1: Column, chip2: Column): Column` **Parameters:** -- `cell1` - First BNG cell reference -- `cell2` - Second BNG cell reference +- `chip1` - First chip struct (`STRUCT`) +- `chip2` - Second chip struct (same schema) **Returns:** -- BNG cell ID representing the union - -**SQL:** - - +- `STRUCT` — dissolved union chip. A fully interior chip (core=true) unioned with itself returns `{cellid, true, null}`. + + --- @@ -593,9 +766,14 @@ Powered by the **pure-Python port of `BNG.scala`** (numpy-vectorized encoder). E **Returns:** - String BNG cell reference -**SQL:** - - + --- @@ -612,21 +790,20 @@ Powered by the **pure-Python port of `BNG.scala`** + **shapely**. The point coor **Signature:** `bng_pointascell(point: Column, resolution: Column): Column` **Parameters:** -- `point` - Point geometry as **WKT** (string) or **WKB** (binary). For example, WKT: `'POINT(-0.1278 51.5074)'`. Do not use `st_point()` or other DBR native geometry functions—they return a type GeoBrix does not accept. +- `point` - Point geometry as **WKT** (string) or **WKB** (binary). Must be in **EPSG:27700** eastings/northings (e.g. `'POINT(530000 180000)'` for London). Do not use `st_point()` or other DBR native geometry functions — they return a type GeoBrix does not accept. - `resolution` - BNG resolution: integer index (e.g. 3 for 1 km) or string (e.g. '1km', '100m') **Returns:** - String BNG cell reference -**Python (point as WKT column):** - - - -**Scala:** Pass a WKT or WKB column and BNG resolution (e.g. `bx.bng_pointascell(lit("POINT(530000 180000)"), lit("1km"))` for BNG coords, or `lit(3)` for 1 km index). Do not use `st_point()`. - -**SQL:** - - + --- @@ -653,9 +830,14 @@ Powered by the **pure-Python port of `BNG.scala`**. All cells within ring distan **Returns:** - Array of BNG cell references in the k-ring -**SQL:** - - + --- @@ -678,9 +860,14 @@ Powered by the **pure-Python port of `BNG.scala`**. The cells at exactly ring di **Returns:** - Array of BNG cell references at exactly distance k -**SQL:** - - + --- @@ -704,9 +891,14 @@ Powered by the **pure-Python port of `BNG.scala`** + **shapely**. Polyfills the **Returns:** - Array of BNG cell references -**SQL:** - - + --- @@ -730,9 +922,14 @@ Powered by the **pure-Python port of `BNG.scala`** + **shapely**. Polyfills the **Returns:** - Array of BNG cell references at exactly distance k -**SQL:** - - + --- @@ -759,9 +956,14 @@ Powered by the **pure-Python port of `BNG.scala`** + **shapely**. Enumerates the **Returns:** - Array of BNG cell IDs covering the geometry -**SQL:** - - + --- @@ -784,9 +986,14 @@ Powered by the **pure-Python port of `BNG.scala`** + **shapely**. Polyfill plus **Returns:** - Array of structs containing cell ID and geometry -**SQL:** - - + --- @@ -825,9 +1032,14 @@ GROUP BY group_key **Returns:** - BNG cell ID representing common intersection -**SQL:** - - + --- @@ -862,9 +1074,14 @@ GROUP BY group_key **Returns:** - BNG cell ID representing bounding union -**SQL:** - - + --- @@ -891,9 +1108,12 @@ Powered by the **pure-Python port of `BNG.scala`** as a streaming UDTF — one o **Returns:** - Exploded rows, one per cell in k-ring -**SQL:** - - + --- @@ -916,9 +1136,12 @@ Powered by the **pure-Python port of `BNG.scala`** as a streaming UDTF — one o **Returns:** - Exploded rows, one per cell in k-loop -**SQL:** - - + --- @@ -942,9 +1165,12 @@ Powered by the **pure-Python port of `BNG.scala`** + **shapely** as a streaming **Returns:** - Exploded rows, one per cell -**SQL:** - - + --- @@ -968,9 +1194,12 @@ Powered by the **pure-Python port of `BNG.scala`** + **shapely** as a streaming **Returns:** - Exploded rows, one per cell at distance k -**SQL:** - - + --- @@ -993,9 +1222,12 @@ Powered by the **pure-Python port of `BNG.scala`** + **shapely** as a streaming **Returns:** - Exploded rows with cell ID and geometry for each cell -**SQL:** - - + --- @@ -1053,9 +1285,14 @@ Define a user-specified regular grid from an origin, extent, cell size, split fa **Returns:** - `STRUCT` — a grid descriptor struct passed to all other `gbx_custom_*` functions. -**SQL:** - - + --- @@ -1075,9 +1312,14 @@ Index a point geometry into a custom grid cell ID at the specified resolution le **Returns:** - `BIGINT` cell ID encoding the grid position at the given resolution -**SQL:** - - + --- @@ -1094,11 +1336,16 @@ Return the WKB footprint polygon of a custom grid cell. - `grid` — custom grid descriptor returned by `gbx_custom_grid` **Returns:** -- `BINARY` WKB polygon representing the cell boundary - -**SQL:** - - +- `BINARY` WKB polygon representing the cell boundary (geometry `... (WKB binary)`) + + --- @@ -1117,9 +1364,14 @@ Return the WKT footprint polygon of a custom grid cell. **Returns:** - `STRING` WKT polygon representing the cell boundary -**SQL:** - - + --- @@ -1136,11 +1388,16 @@ Return the centroid of a custom grid cell as a WKB point. - `grid` — custom grid descriptor returned by `gbx_custom_grid` **Returns:** -- `BINARY` WKB point at the cell center (in the grid's CRS) - -**SQL:** - - +- `BINARY` WKB point at the cell center (geometry `... (WKB binary)`) + + --- @@ -1160,9 +1417,14 @@ Fill a geometry with all custom grid cell IDs at the specified resolution. **Returns:** - `ARRAY` of cell IDs whose footprints intersect the geometry -**SQL:** - - + --- @@ -1182,9 +1444,14 @@ Return all custom grid cells within `k` steps of a center cell (filled neighborh **Returns:** - `ARRAY` of cell IDs within distance `k` (up to `(2k+1)^2` cells) -**SQL:** - - + --- diff --git a/docs/docs/api/h3-raster-tessellation.mdx b/docs/docs/api/h3-raster-tessellation.mdx index d7d7464ae..33ebb7981 100644 --- a/docs/docs/api/h3-raster-tessellation.mdx +++ b/docs/docs/api/h3-raster-tessellation.mdx @@ -88,7 +88,7 @@ FROM rasters, LATERAL gbx_rst_h3_tessellate(tile, 7) t; ``` -Each row in the result is a tile struct with `cellid` (the H3 cell integer ID), `raster` (clipped raster bytes), and `metadata`. +Each row in the result is a v2 tile struct — `cellid` (the H3 cell integer ID), `raster` (clipped raster bytes), `path`, `window`, `clip_polygon`, `clip_crs`, `crs`, and `metadata`. --- diff --git a/docs/docs/api/large-rasters.mdx b/docs/docs/api/large-rasters.mdx new file mode 100644 index 000000000..fa0686754 --- /dev/null +++ b/docs/docs/api/large-rasters.mdx @@ -0,0 +1,224 @@ +--- +sidebar_position: 4 +sidebar_label: Large Rasters +title: Large Rasters +--- + +# Large Rasters + +Preparing and working with rasters that are too big to load whole — multi-gigabyte +scenes, striped GeoTIFFs, satellite mosaics. This page covers the formats +involved, why Cloud-Optimized GeoTIFF (COG) is the target, and how GeoBrix +prepares COGs at scale on Databricks. For the reading model that builds on these +prepared COGs — bytes-free windowed reads and the virtual/materialized tile +lifecycle — see **[Virtual Tiles](./virtual-tiles)**. + +--- + +## Raster formats: striped, tiled, and COG + +A GeoTIFF stores its pixels in one of two internal layouts, and the layout +decides how efficiently you can read part of the image. + +- **Striped GeoTIFF** — pixels are stored in horizontal strips that span the + full width of the image (often a single row, or a small band of rows, per + strip). To read any region you must walk strips from the top. There is no way + to jump to an arbitrary sub-window cheaply, and reading "just a corner" can + force a read of much of the file. Many source rasters (including a lot of + satellite output) arrive striped. Striped layout is the worst case for + large-raster access: it resists partial reads and, when a tool tries to load a + large striped image, it tends to pull far more into memory than the region + actually needed. + +- **Tiled GeoTIFF** — pixels are stored in a regular grid of rectangular blocks + (for example 256×256 or 512×512). Any sub-window maps to a small set of blocks + that can be read directly, without scanning the rest of the file. Tiling is + what makes efficient windowed reads possible. + +- **Cloud-Optimized GeoTIFF (COG)** — a tiled GeoTIFF with two additional + guarantees: (1) it carries **overviews** — pre-computed, progressively + coarser downsampled copies of the image, so a zoomed-out view reads a small + overview instead of decoding full resolution; and (2) its internal directory + is laid out so a reader can fetch the header once and then request only the + byte ranges it needs. A COG is readable as an ordinary GeoTIFF everywhere, but + a COG-aware reader can serve any window or zoom level with a handful of ranged + reads. This is the format GeoBrix prepares and reads for large-raster work. + +:::note BigTIFF: the >4 GiB boundary +The classic TIFF container addresses data with 32-bit offsets, which caps a file +at roughly **4 GiB**. Any COG whose output would exceed that must use the +**BigTIFF** extension (64-bit offsets). BigTIFF is read transparently by modern +GDAL/rasterio-based tooling (which is what reads these files on Databricks). +GeoBrix writes BigTIFF **by default** so that preparation never fails at the +4 GiB boundary and every prepared COG has one predictable structure; you can +override this per the [options](#options) below when maximum compatibility with +very old, non-GDAL TIFF readers is required. +::: + +### Windows and virtual tiles + +A **window** is a rectangular region of a raster — a pixel offset plus a width +and height. Because a COG is tiled and carries overviews, a reader can +materialize a single window (at full resolution or from an overview) with a +small ranged read, instead of loading the whole image. + +Windows are the foundation for **virtual tiles** — bytes-free references (a path +plus a window) that flow through a DataFrame and materialize pixels only when an +operation needs them, keeping memory bounded when fanning a large raster into +many pieces. The full reading model, the tile struct, and the +virtual↔materialized lifecycle are covered on the **[Virtual Tiles](./virtual-tiles)** +page. The rest of *this* page focuses on **preparing** COGs — what makes those +efficient windowed reads possible in the first place. + +--- + +## Writing COGs, and why + +The single most useful thing you can do with a large or striped raster is +**master it into a COG once**, then read windows from it many times. Preparation +is where the cost of tiling and overview generation is paid — deliberately, up +front — so that every downstream read is cheap. Compression is applied per block +during preparation; see **[Materialized Compression](./materialized-compression)** +for codec choices, size-adaptive defaults, and when to override them. + +GeoBrix offers two ways to prepare COGs, sharing one conversion core. Both write +a COG that passes `rio_cogeo`'s COG validation. + +### `prepare_cogs` — driver-orchestrated preparation + +`prepare_cogs` prepares one master COG per source, running on the **driver**. It +accepts a directory, a single file, or a list freely mixing directories and +files, resolves them to a flat de-duplicated set, and converts each one, +printing per-file progress and returning a summary. + +```python +from databricks.labs.gbx.pyrx.core.preparer import prepare_cogs + +# sources may be a dir, a single file, or a list mixing both +summary = prepare_cogs( + "/Volumes////scenes", # input dir (or file, or list) + "/Volumes////cogs", # output dir + blocksize=512, + verbose=True, +) +# summary -> {"total": N, "ok": ..., "skipped": ..., "error": ..., +# "peak_rss_mib": ..., "elapsed_s": ..., "results": [...]} +``` + +Each source is converted to `.cog`. Preparation is **idempotent**: +by default a source whose output already exists is skipped, so re-running after +an interruption only fills the gaps. A failure on one file is isolated — it is +recorded in the summary and the batch continues — so one bad input never aborts +a large run. + +### `cog_gbx` writer with `driverMode` + +The [`cog_gbx` writer](../writers/cog) integrates preparation into a DataFrame +write. Its default mode converts per partition (suitable for moderate files); +its opt-in [`driverMode`](../writers/cog#drivermode-preparing-large-files) routes +conversion to the driver via `prepare_cogs`, which is the mode for large files: + +```python +(spark.read.format("file_gbx").load(input_dir) + .write.format("cog_gbx") + .option("driverMode", "true") + .option("cogSkipIfExists", "true") + .mode("overwrite") + .save(output_dir)) +``` + +:::warning Long driverMode writes: use `prepare_cogs` directly +In `driverMode`, conversion runs inside the `.save()` call. A write that blocks +for many minutes — a large batch, or very large files (rough throughput is on +the order of **1 GB/min**) — can have its connection cancelled, surfacing as a +failed run with a `CancelledKeyException`, even though the conversion itself is +fine. If you hit this, prepare the files by calling `prepare_cogs` **directly** +in your notebook instead of through the writer. It is plain Python on the driver +with no such connection to cancel, and it is idempotent, so re-running after a +cancellation resumes cleanly. +::: + +### Options + +These apply to both `cog_gbx` (as writer options) and `prepare_cogs`/`prepare_cog` +(as keyword arguments): + +| `cog_gbx` option | `prepare_cogs` arg | Default | Meaning | +|---|---|---|---| +| `cogBlockSize` | `blocksize` | `512` | Internal tile size in pixels. | +| `cogOverviewResampling` | `resampling` | `AVERAGE` | Resampling used to build overviews. | +| `compress` | `compress` | `auto` | Per-block compression: `auto` (size-adaptive ZSTD+predictor — recommended), `zstd`, `deflate`, `lzw`, `none`. See [Materialized Compression](./materialized-compression). | +| `compressLevel` | `compressLevel` | (codec-dependent) | Compression level for `zstd`/`deflate`. Ignored when `compress="auto"`. | +| `predictor` | `predictor` | (dtype-matched) | TIFF predictor tag (1–3). Ignored when `compress="auto"`. | +| `cogCompression` | `compression` | (deprecated) | **Deprecated alias** for `compress`; use `compress` instead. | +| `cogSubdataset` | `subdataset` | none | Subdataset to select from a multi-subdataset source (e.g. NetCDF). | +| `cogSkipIfExists` | `skip_if_exists` | `true` | Skip a source whose `.cog` output already exists (idempotent resume). | +| `cogBigTiff` | `bigtiff` | `YES` | BigTIFF policy: `YES` (always), `IF_SAFER`/`IF_NEEDED` (size-adaptive), `NO` (force classic TIFF — fails past ~4 GiB). | +| `driverMode` | — | `false` | Route conversion to the driver (large-file mode). | +| `driverModeVerbose` | `verbose` | `true` | Print per-file progress from the driver. | + +### Memory footprint, and staying on standard Serverless + +COG generation streams block-by-block within a bounded cache and processes one +file at a time. As a result, **peak memory is essentially flat regardless of +source size or batch count** — dominated by the transient cost of building the +overview pyramid, not by the size of the raster. In testing, preparing a +1.5 GiB source, ten 1.5 GiB sources, and a single 10 GiB source all peaked at +roughly the same **~2 GiB**. + +That comfortably fits a **standard Serverless driver**, so no special compute is +required to prepare large COGs. A **high-memory driver** adds headroom for +pushing to much larger single files, but is not required for the sizes above. + +The reason preparation runs on the **driver** — rather than distributed across +workers — is a hard platform limit: a Serverless worker task has a fixed +per-task memory ceiling (on the order of 1 GB) that no instance size raises. +Converting a multi-gigabyte source needs more than that transiently, so a +distributed (per-worker) conversion of a large file runs out of memory +regardless of the cluster's size. The driver is not under that per-task ceiling, +so driver-orchestrated preparation succeeds where a distributed conversion of +the same file cannot. This is why `prepare_cogs` and the `cog_gbx` `driverMode` +exist, and why increasing worker memory does not help large-file preparation. + +For the full connect-aware memory model that keeps raster **reads and writes** +within this per-task ceiling — the stream cap, the materialize-vs-virtual +decision, and the FILE Delta-table fast path — see +**[Serverless & Memory](../serverless-and-memory)**. + +--- + +## Reading prepared COGs + +Once a source is mastered as a COG, reads become cheap: a COG-aware reader can +fetch any window, at any zoom level, with a small number of ranged reads instead +of loading the whole image. GeoBrix's `cog_gbx` reader turns that into +**virtual tiles** — bytes-free rows that materialize pixels only on demand — so a +large raster fans into many pieces without any executor holding the whole image. + +The full reading model — the tile struct, virtual vs. materialized state, the +reader selection options, and how `rst_*` functions choose their output shape — +is covered on the **[Virtual Tiles](./virtual-tiles)** page. Prepare your COGs +here; read them there. + +:::tip Virtual tiles + large COGs: byte-range reads +When virtual tiles reference **windows** of a large Cloud-Optimized GeoTIFF, +Databricks FILE fetches only the bytes covering each window's COG tile blocks — +rather than opening and seeking the full file. The read advantage is realized with +**per-partition open-amortization**: opening the source once per partition and reading +all its windows from the same cached dataset handle, rather than opening a new connection +per tile. With amortization, byte-range stream reads are 10–290× faster than FUSE for +windowed COG reads. + +See [Virtual tile read performance](../api/performance#virtual-tile-read-performance) +for amortized numbers and guidance on the grouped executor. For per-tile-open (non-amortized) +benchmarks, see [FILE vs FUSE: large-COG results](../api/benchmarking#file-capability-cog-multiwindow) +— in that regime FUSE wins; the two pages cover different access patterns. +::: + +## See also + +- [Virtual Tiles](./virtual-tiles) — the reading model and virtual↔materialized lifecycle over prepared COGs +- [VRT & Mosaics](./vrt-mosaic) — decompose a large source into bounded mini-COGs + a portable VRT index (a 308 MB raster becomes ~130 mini-COGs of ~4 MB each, Serverless-safe) +- [COG writer (`cog_gbx`)](../writers/cog) — the DataFrame-write entry point for preparation, including `driverMode` +- [COG reader (`cog_gbx`)](../readers/cog) — windowed, bbox-clipped reads from prepared COGs +- [File lister (`file_gbx`)](../readers/file) — enumerate source files (paths only) to feed preparation diff --git a/docs/docs/api/materialized-compression.mdx b/docs/docs/api/materialized-compression.mdx new file mode 100644 index 000000000..77930da04 --- /dev/null +++ b/docs/docs/api/materialized-compression.mdx @@ -0,0 +1,144 @@ +--- +sidebar_position: 3 +sidebar_label: Materialized Compression +title: Materialized Compression +--- + +# Materialized Compression + +When a virtual tile becomes bytes on disk or in a Spark row — during a materialize operation, a write, or a tile format conversion — compression is applied. This page explains what GeoBrix compresses by default, how it chooses compression settings for different tile sizes, and when you might want to override the defaults. + +## What gets compressed + +A **materialized tile** carries its raster payload as encoded bytes. Those bytes flow through Spark DataFrames (each row's `tile.raster` field holds up to a few hundred megabytes) and are written to files when you persist results. At each step, GDAL's GeoTIFF codec is applied, choosing a balance of compression ratio, memory use, and encoding time. + +For the structure of a tile and the full virtual↔materialized lifecycle, see **[Tile Structure](./tile-structure)** and **[Virtual Tiles](./virtual-tiles)**. + +## GeoBrix's approach: ZSTD + dtype-matched predictor + +The default compression in GeoBrix is **ZSTD** paired with a **dtype-matched predictor** that reorders bytes to improve compression ratio: + +- **float32 / float64**: predictor 3 (horizontal differencing for floating-point data) +- **int16 / uint16 / int32 / uint32**: predictor 2 (delta encoding for integer data) +- **uint8 / int8**: predictor 1 (byte data; predictor adds negligible benefit) + +The `auto` (default) compression mode selects a **size-adaptive ZSTD level** so small tiles in Spark rows squeeze harder (higher level, negligible cost at small size) while large tiles stay lighter to protect worker memory and write time: + +| Decoded tile size | ZSTD level | Why | +|---|---|---| +| ≤ 4 MiB | 16 | Small encode cost (36–127 ms), high ratio, safe RSS | +| 4–128 MiB | 12 | Safe across the range; write ≤ 3× baseline, RSS ≤ 1.2× | +| 128 MiB – 1 GiB | 9 | Balances ratio and memory; large payloads need headroom | +| > 1 GiB | 6 | OOM guard; write time is flat, memory is the constraint | + +This ladder is grounded by a benchmark of real encoding across codec families, sizes, and data types (see **[Evidence](#evidence-table)** below). + +## When `auto` is the right choice + +`auto` is the default and works for **nearly all cases**: +- Tiles are compressed once at materialize or write time. +- Readers (both in-cluster and off-cluster) decompress just-in-time, and decompression time is flat across all ZSTD levels (no cost for picking a high level). +- The size-adaptive level protects Serverless workers from memory spikes at large payloads. + +Use `auto` unless you have a specific reason not to. + +## When to reach for other codecs + +**`compress="deflate"`** — For maximum portability to tools that don't support ZSTD: +- Older GDAL versions (pre-3.1) lack ZSTD support off-cluster. +- If you hand off files to external tools, DEFLATE is more widely supported. +- Trade-off: DEFLATE is 2–3× slower to write than ZSTD at the same ratio, and doesn't adapt to tile size. + +**`compress="zstd"` with explicit `compressLevel`** — For write-once, read-many catalogs: +- If you're building a reference dataset that will be read many times, a high fixed level (e.g. `compressLevel=19` for sizes up to 128 MiB) pays off over the lifetime of many reads. +- Warning: levels ≥ 19 get exponentially slower to write (100–300ms per small tile) and use significant memory (100+ MiB per encode); use only on the driver or small batches. + +**`compress="lzw"` or `compress="none"`** — Rarely needed: +- LZW **expands** float32 and continuous data; use only for categorical/integer classification data. +- No compression is occasionally useful for debugging, but costs both space and downstream reading time. + +## Control surface: `compress` / `compressLevel` / `predictor` + +When writing tiles, you can override the codec and settings: + +```python +# Write with the default size-adaptive ZSTD (recommended) +df.write.format("gtiff_gbx").option("compress", "auto").save(...) + +# Write with DEFLATE, level 9 (maximum portability) +df.write.format("gtiff_gbx") \ + .option("compress", "deflate") \ + .option("compressLevel", "9") \ + .save(...) + +# Explicit ZSTD at a fixed high level (write-once catalog) +df.write.format("gtiff_gbx") \ + .option("compress", "zstd") \ + .option("compressLevel", "16") \ + .save(...) + +# No compression (debugging only) +df.write.format("gtiff_gbx").option("compress", "none").save(...) +``` + +For `prepare_cogs` (driver-based COG preparation), pass the same options as kwargs: + +```python +from databricks.labs.gbx.pyrx.core.preparer import prepare_cogs + +prepare_cogs( + input_dir, + output_dir, + compress="deflate", + compressLevel=9, + verbose=True +) +``` + +The deprecated option name `cogCompression` is accepted as an alias for `compress` in the writer. + +## Portability: ZSTD off-cluster + +Cloud-Optimized GeoTIFFs prepared with ZSTD can be read on Databricks (in-cluster GDAL has ZSTD support), but **off-cluster tools may not**: + +- **GDAL 3.1+** (and rasterio 1.2+) supports ZSTD natively. +- **GDAL 3.0 and earlier** will fail to open ZSTD-compressed GeoTIFFs. + +If you expect downstream processing by tools without ZSTD support, use `compress="deflate"` instead. The compression ratio is similar (1–2% difference), and DEFLATE is universally available. + +## Evidence table + +The following table shows popular codec combinations (none, LZW, DEFLATE, and ZSTD at levels 9–16+predictor) across realistic tile sizes and data types. The benchmark ran on macOS arm64 with GDAL 3.12 and rasterio 1.5. Sizes 256 MiB–1 GiB are extrapolated and pending Serverless validation. + +| Size | Codec | Dtype | Ratio | Write (ms) | Read (ms) | Peak RSS (MiB) | +|---|---|---|---|---|---|---| +| 1 MiB | none | float32 | 0.99× | 8 | 0.5 | 14 | +| 1 MiB | DEFLATE z6 + p3 | float32 | 1.17× | 25 | 5.0 | 14 | +| 1 MiB | ZSTD L9 + p3 | float32 | 1.17× | 18 | 1.8 | 24 | +| 1 MiB | ZSTD L16 + p3 | float32 | 1.18× | 44 | 1.9 | 46 | +| 8 MiB | none | float32 | 1.00× | 12 | 2.0 | 45 | +| 8 MiB | DEFLATE z6 + p3 | float32 | 1.18× | 168 | 39 | 35 | +| 8 MiB | ZSTD L9 + p3 | float32 | 1.17× | 76 | 11 | 45 | +| 8 MiB | ZSTD L12 + p3 | float32 | 1.17× | 110 | 11 | 75 | +| 32 MiB | none | float32 | 1.00× | 23 | 8 | 143 | +| 32 MiB | DEFLATE z6 + p3 | float32 | 1.19× | 593 | 143 | 99 | +| 32 MiB | ZSTD L9 + p3 | float32 | 1.19× | 234 | 40 | 109 | +| 32 MiB | ZSTD L12 + p3 | float32 | 1.19× | 347 | 41 | 112 | +| 128 MiB | none | float32 | 1.00× | 63 | 36 | 532 | +| 128 MiB | DEFLATE z6 + p3 | float32 | 1.20× | 2827 | 535 | 355 | +| 128 MiB | ZSTD L9 + p3 | float32 | 1.20× | 887 | 167 | 375 | +| 128 MiB | ZSTD L12 + p3 | float32 | 1.20× | 1291 | 166 | 395 | + +**Key observations:** +- ZSTD L9 + predictor beats or matches DEFLATE, while writing **2–3× faster** across all sizes. +- ZSTD's compression ratio plateaus at level 9; higher levels (12, 16) add write time and memory with minimal ratio gain. +- Read decompression time is flat across levels — no penalty for high compression. +- LZW expands float32 data (not shown); use only for categorical integer rasters. + +--- + +## Next steps + +- [Large Rasters](./large-rasters) — preparing COGs and understanding memory constraints +- [Virtual Tiles](./virtual-tiles) — the virtual↔materialized lifecycle and when tiles materialize +- [Tile Structure](./tile-structure) — the internal schema of a raster tile diff --git a/docs/docs/api/performance.mdx b/docs/docs/api/performance.mdx index 20419d36b..9bd6bd59d 100644 --- a/docs/docs/api/performance.mdx +++ b/docs/docs/api/performance.mdx @@ -17,6 +17,101 @@ For measured timing and output-consistency numbers, see **[Benchmarking](./bench --- +## Virtual tile read performance {#virtual-tile-read-performance} + +:::note Key takeaways +- **Deferred edits run 1.5–12× faster on virtual tiles** — metadata operations (set-CRS/SRID/nodata/band-select) complete without pixel I/O. Chain several deferred edits before a terminal step to materialize once and pay pixel cost only at the end. +- **Open amortization is the lever** — grouping same-source windows (via `ORDER BY path` or repartitioning) cuts per-tile open cost from ~multiple-seconds to ~2 ms per window. Without amortization (per-tile-open), stream reads grow expensive with file size; FUSE becomes viable. +- **FILE-on ≈ FILE-off for reads on dedicated clusters** — FILE is a governance/access mechanism, not a per-tile read optimization. Amortization (grouped executor) is the performance lever. Serverless FILE support is coming soon. +- **COG + amortization beats FUSE** — byte-range stream reads are 10–290× faster than FUSE for windowed COGs when opens are amortized. Choose COG format and group by source path for best throughput. +::: + +Virtual tiles are read on demand. Reading cost is dominated by **opening** the source raster, not by the pixel read itself. A pipeline that reads many tiles from the same source raster should open that source once and amortize the cost across all its windows — not open it once per tile. + +### Two access modes: stream and FUSE + +GeoBrix's virtual tile reader supports two access modes: + +| Mode | Per-open cost | Typical per-window cost | Memory per open source | Best for | +|---|---|---|---|---| +| **Byte-range stream** (`.open()`) | Scales with file size and layout | ~2 ms (COG, with amortization) | **RAM ≈ source file size** | Grouped reads on small-to-moderate files | +| **FUSE** (`.as_local_file()`) | Flat ~7 ms at any size | ~124 ms (cold read baseline) | Low — reads only requested bytes lazily | Large files, per-tile-open, RAM-constrained workers | + +When the open cost is amortized across multiple windows from the same source, byte-range stream reads are **10–290× faster** than FUSE for windowed COG reads. Without amortization (one open per tile), FUSE's flat open cost wins: stream's per-open header parse grows with file size and becomes prohibitive at scale. + +**You don't choose stream vs FUSE — GeoBrix does, automatically, per source.** The reader inspects each source's size and layout and picks the access mode that's correct for it. This table is what GeoBrix does for you, not a decision you make: + +| Source (per file) | GeoBrix auto-selects | Why | +|---|---|---| +| Tiled / COG, ≤ 256 MiB | **byte-range stream** | random-window reads are efficient on a tiled layout; the source buffers into executor RAM | +| Large (over 256 MiB) **or** striped | **FUSE (lazy)** | fetches only the bytes each window needs — no whole-file RAM, safe at any size | +| stream **or** FUSE open fails | **staging fallback** (temp file) | correctness over speed | + +The 256 MiB stream cutoff (`GBX_STREAM_MAX_BYTES`) is an optional environment override — the default handles the common cases, so you normally set nothing. + +**Executor RAM is auto-managed by a byte-budgeted LRU cache — you don't size or clear it.** Open sources are held in a per-partition LRU keyed by source path, capped at a byte budget (`GBX_LRU_MAX_BYTES`, default 4 GiB). Stream handles count their buffered size against the budget; FUSE handles count only a small nominal (they don't buffer the file). When the budget would be exceeded, the least-recently-used source is evicted and closed. So a partition **never blows executor RAM no matter how many sources it touches**, and repeated windows from an already-open source are ~2 ms cache hits (the amortization in the next section). + +:::note Why the auto-rules look the way they do — the striped-file trap GeoBrix sidesteps +Opening a large source via byte-range buffers it into executor RAM, and for a **large striped GeoTIFF** parsing the strip-offset table over byte-range can reach **47–54 seconds per open at 10 GB, consuming 10 GB of executor RAM**. That is exactly why GeoBrix routes large or striped sources to FUSE (lazy, low-RAM) automatically rather than streaming them. For best windowed-read throughput, **prefer COGs** — a tiled COG under the threshold takes the fast stream path and amortizes across its windows. +::: + +### Per-partition open-amortization (the grouped executor) + +GeoBrix's grouped executor holds a small per-partition LRU cache of open stream handles, keyed by source path. The first access for a source in a partition opens it (cache miss); all subsequent windows from that source in the same partition are cache hits (~2 ms each). When tiles are sorted by source path within each partition, even a partition holding several sources fully amortizes at LRU size ~2. + +Measured on a 647 MB COG corpus (15 sources × 12 windows = 180 tiles): + +| Write layout | Read approach | Opens | Total time | +|---|---|---|---| +| Scattered (no alignment) | Dumb (as-written) | 135 | 173 s | +| `ORDER BY path` | Dumb (as-written) | **15** | 37 s | +| `ORDER BY path` | Smart (re-partition) | **15** | **11 s** | + +To enable amortization after a join or reshape that has scattered tiles across partitions, re-align before processing: + +```python +df = df.repartition(n, "tile.path").sortWithinPartitions("tile.path") +``` + +Size `n` for parallelism, not for file count: +- **Classic clusters (fixed workers):** `n ≈ 3–5× total worker cores` — e.g. 20 workers × 4 cores → `n ≈ 100` +- **Serverless (elastic workers):** `n` is the parallelism signal; smaller values pull in fewer workers, larger pull in more + +Do not set `n = number_of_files` for large corpora — over-partitioning produces tiny tasks and scheduling overhead. + +### MANAGED vs EXTERNAL: identical read performance + +A tile column declared `FILE MANAGED` and one declared `FILE EXTERNAL` perform identically for reads. Both resolve to the same underlying paths; the open-cost and windowed-read cost are indistinguishable. The choice between MANAGED and EXTERNAL is about **governance and lifecycle**, not read speed. MANAGED tiles carry governed lifecycle (tied to the table) and `content_type` detection; EXTERNAL tiles reference files managed independently in a Volume. Choose based on governance requirements, not on any expected read-latency difference. + +### Write-side layout for efficient reads + +The GeoBrix writer optionally lays tiles out source-aligned at write time, so the reader can amortize for free on the first read — no re-partition step required after a normal read-write round trip. + +| Write strategy | Effect on reads | +|---|---| +| `INSERT … ORDER BY tile.path` | Immediate physical grouping — dumb reads amortize without any re-partition | +| `CLUSTER BY (tile.path)` on the table + periodic `OPTIMIZE` | Durable layout after Delta applies liquid clustering | +| `CLUSTER BY (tile.path)` on the initial insert only (no `OPTIMIZE`) | May not be grouped yet — combine with `ORDER BY` in the `INSERT` for insert-time grouping | +| No alignment | Scattered — requires explicit `repartition + sortWithinPartitions` on every read | + +:::caution Avoid `partitionBy("tile.path")` +`tile.path` is high-cardinality — one value per source file. Using it as a Delta partition column creates thousands of partition directories, causing metadata explosion and slow query planning. Use `ORDER BY` and `CLUSTER BY` instead. +::: + +### FILE type and `path_mode` + +When FILE is available (Databricks Runtime 19 dedicated clusters; coming soon to Serverless), the tile struct's [`path_mode` field](./tile-structure) records how the virtual tile was stored: + +| `path_mode` | Meaning | +|---|---| +| `null` | Materialized tile (raster bytes present), or a plain FUSE-path virtual tile | +| `"external"` | FILE EXTERNAL virtual tile — opened via byte-range stream | +| `"managed"` | FILE MANAGED virtual tile — opened via byte-range stream; lifecycle governed by the table | + +On runtimes without FILE support (Serverless today, DBR 17/18), `path_mode` is always `null` and virtual tiles use the FUSE path — correct and unchanged. + +--- + ## Execution shapes ### Streaming UDTFs @@ -43,6 +138,10 @@ This is the right shape for **per-row (1→1) tile and geometry transforms**: op Functions that return `MapType` columns use a plain `@f.udf` for SQL registration (Arrow does not support `MapType` in all `pandas_udf` builds); the Python Column API routes through the `pandas_udf` path for all tile-returning operations. ::: +### Chaining functions + +Compose light-tier tile functions in **one expression** — `rx.rst_slope(rx.rst_setcrs(col("tile"), "EPSG:32633"))` — rather than materializing intermediates across separate `.select()` / `.withColumn()` steps. Spark **fuses chained Column-API calls into a single Python worker**, so the intermediate tile passes from the inner function to the outer one as a live in-worker object: the JVM↔Python boundary is crossed **once for the whole chain instead of once per function**. Splitting the same pipeline into separate steps sends each intermediate tile back to the JVM and re-serializes it at the next step — paying the per-batch boundary cost at every stage instead of once. Chaining is strictly the lighter path, and it composes with virtual-tile deferral (see [Choosing an Execution Tier](./execution-tiers)): chained deferrable ops both defer materialization *and* stay in one worker. + ### Vectorized cores The compute underneath each UDF shape comes from best-in-class Python libraries rather than partial reimplementations: diff --git a/docs/docs/api/pmtiles-functions.mdx b/docs/docs/api/pmtiles-functions.mdx index 9ae5e76ba..05c287fca 100644 --- a/docs/docs/api/pmtiles-functions.mdx +++ b/docs/docs/api/pmtiles-functions.mdx @@ -191,7 +191,7 @@ Tile type is detected automatically from the content of the first non-null paylo ### Typical pipelines - **Raster pyramid:** `gbx_rst_xyzpyramid(tile, minZoom, maxZoom)` produces per-tile rows of PNG bytes — pipe straight into `gbx_pmtiles_agg`. -- **Vector pyramid:** `gbx_st_asmvt_pyramid(geom_wkb, attrs, minZoom, maxZoom, layer)` produces per-tile MVT bytes — pipe straight into `gbx_pmtiles_agg`. Because `gbx_st_asmvt_pyramid` emits one row per feature, the aggregate merges all features that share a tile coordinate into a single multi-feature tile. +- **Vector pyramid:** `gbx_st_asmvt_pyramid(geom, attrs, minZoom, maxZoom, layer)` produces per-tile MVT bytes — pipe straight into `gbx_pmtiles_agg`. Because `gbx_st_asmvt_pyramid` emits one row per feature, the aggregate merges all features that share a tile coordinate into a single multi-feature tile. For pyramids that exceed the Spark cell ceiling, use the [PMTiles Writer](../writers/pmtiles) instead. @@ -227,7 +227,7 @@ PMTiles is designed to be served as a single static file via HTTP `Range` reques ## Limits in v0.4.0 - **No leaf directories.** If the global root directory would exceed 16,257 bytes (spec § 4), the encoder errors out and asks you to split your input. In practice this only happens with very large pyramids (tens of millions of tiles); the limit will be relaxed in a future release. -- **No read path.** `spark.read.format("pmtiles")` raises a friendly "Reading PMTiles archives is not supported in GeoBrix 0.4.0" error — use one of the JS / Python pmtiles client libraries for read access. +- **No heavyweight read path.** `spark.read.format("pmtiles")` raises a friendly error. Read within GeoBrix via the lightweight [`pmtiles_gbx` reader](../readers/pmtiles); the JS / Python pmtiles client libraries also work for external read access. - **No cross-task dedup in the DataSource.** Identical tiles across partitions are stored multiple times in the final file. The UDAF path does per-blob SHA-256 dedup, so for known-redundant pyramids prefer the UDAF if your data fits. ## References diff --git a/docs/docs/api/raster-functions.mdx b/docs/docs/api/raster-functions.mdx index a428a0730..de707d080 100644 --- a/docs/docs/api/raster-functions.mdx +++ b/docs/docs/api/raster-functions.mdx @@ -12,8 +12,18 @@ import RasterXIcon from '../../../resources/images/brand/RasterX.png'; import rasterxCode from '!!raw-loader!../../tests/python/api/rasterx_functions.py'; import pyrxCode from '!!raw-loader!../../tests/python/api/pyrx_functions.py'; import rasterxSqlCode from '!!raw-loader!../../tests/python/api/rasterx_functions_sql.py'; -import packagesExamples from '!!raw-loader!../../tests/python/packages/examples.py'; -import rasterxScalaCode from '!!raw-loader!../../tests/scala/packages/RasterxPackageExamples.scala'; +import scalaApiExamplesCode from '!!raw-loader!../../tests/scala/api/ScalaApiExamples.scala'; +import rasterxLightCode from '!!raw-loader!../../tests/python/api/rasterx_functions_python_light.py'; +import rasterxAccessorsLightCode from '!!raw-loader!../../tests/python/api/rasterx_accessors_python_light.py'; +import rasterxTileopsLightCode from '!!raw-loader!../../tests/python/api/rasterx_tileops_python_light.py'; +import rasterxAggregatorsLightCode from '!!raw-loader!../../tests/python/api/rasterx_aggregators_python_light.py'; +import rasterxBandmathLightCode from '!!raw-loader!../../tests/python/api/rasterx_bandmath_python_light.py'; +import rasterxTerrainLightCode from '!!raw-loader!../../tests/python/api/rasterx_terrain_python_light.py'; +import rasterxTransformsLightCode from '!!raw-loader!../../tests/python/api/rasterx_transforms_python_light.py'; +import rasterxGeneratorsLightCode from '!!raw-loader!../../tests/python/api/rasterx_generators_python_light.py'; +import rasterxGridaggLightCode from '!!raw-loader!../../tests/python/api/rasterx_gridagg_python_light.py'; +import FunctionExamples from '@site/src/components/FunctionExamples'; +import VirtualTileOverrides from '../_partials/_virtual-tile-overrides.mdx'; # RasterX Function Reference @@ -64,13 +74,34 @@ RasterX exposes 87+ SQL functions (registered as `gbx_rst_*`; available in Pytho ## Tile payload -Every RasterX function returns a tile whose `raster` field is a **self-contained, in-memory raster** (GTiff by default) — safe to serialize between Spark stages and executors, persist to Delta, hand off to `rasterio` / `gdal`, or write back out via the `gdal` writer. The bytes are never an XML reference to a per-executor `/vsimem/` tempfile or to a path that only exists on the producing node. +Every RasterX (heavyweight) function returns a **materialized** tile whose `raster` field is a **self-contained, in-memory raster** (GTiff by default) — safe to serialize between Spark stages and executors, persist to Delta, hand off to `rasterio` / `gdal`, or write back out via the `gdal` writer. The bytes are never an XML reference to a per-executor `/vsimem/` tempfile or to a path that only exists on the producing node. -Functions that internally build via an intermediate VRT — `gbx_rst_merge`, `gbx_rst_merge_agg`, `gbx_rst_frombands`, `gbx_rst_combineavg`, `gbx_rst_combineavg_agg`, `gbx_rst_derivedband`, `gbx_rst_derivedband_agg` — materialize the result to GTiff before returning, so downstream stages on different executors see real raster bytes. Inspect a tile's payload format from `tile.metadata.driver`; for any of the functions above, it will read `GTiff` (not `VRT`). See [Beta Release Notes](../beta-release-notes#whats-new-in-v030) for the v0.3.0 correctness fix that introduced this invariant. See [Tile structure](./tile-structure) for the full tile-struct schema. +See **[Tile structure](./tile-structure)** for the full tile-struct schema (the shared `cellid` / `raster` / `path` / `window` / … struct), and **[Virtual Tiles](./virtual-tiles)** for the bytes-free lightweight-tier variant and how tiles move between materialized and virtual. + +Functions that internally build via an intermediate VRT — `gbx_rst_merge`, `gbx_rst_merge_agg`, `gbx_rst_frombands`, `gbx_rst_combineavg`, `gbx_rst_combineavg_agg`, `gbx_rst_derivedband`, `gbx_rst_derivedband_agg` — materialize the result to GTiff before returning, so downstream stages on different executors see real raster bytes. Inspect a tile's payload format from `tile.metadata.driver`; for any of the functions above, it will read `GTiff` (not `VRT`). See [Release Notes](../release-notes#whats-new-in-v030) for the v0.3.0 correctness fix that introduced this invariant. See [Tile structure](./tile-structure) for the full tile-struct schema. + +## Virtual-tile force-output params {#virtual-tile-overrides} + + ## Setup -Pick your execution tier and run this once. Both tiers alias the module as `rx`, so every example below is identical regardless of tier — only this setup differs. (See [Choosing an Execution Tier](./execution-tiers) for the comparison.) +With the geobrix library already installed ([Installation](../installation)), pick your execution tier and run this once. Both tiers alias the module as `rx`, so every example below is identical regardless of tier — only this setup differs. (See [Choosing an Execution Tier](./execution-tiers) for the comparison.) + +The examples on this page read from **four temp views**, each a raster loaded as a `tile` column. Point the four path placeholders at your own rasters: + +| Placeholder | Sample file | Reader (light / heavy) — Temp view | Backs | +|---|---|---|---| +| `GTIFF_SAMPLE_DIR` | single-band GeoTIFF | `gtiff_gbx` / `gtiff_gdal` — `rasters` | **Default** — most accessor, tile-ops, transform, and generator examples | +| `GTIFF_MULTI_DIR` | multi-band GeoTIFF (red/NIR/green) | `gtiff_gbx` / `gtiff_gdal` — `multiband_rasters` | band-math and spectral-index examples, `rst_numbands`, `rst_bandmetadata` | +| `DTM_DIR` | digital elevation model | `gtiff_gbx` / `gtiff_gdal` — `dem_rasters` | terrain examples (`rst_slope`, `rst_aspect`, …) | +| `NETCDF_DIR` | NetCDF with subdatasets | `netcdf_gbx` / `netcdf_gdal` — `netcdf_rasters` | `rst_subdatasets`, `rst_getsubdataset` | + +Each format has a named reader in **both** tiers (light `*_gbx` / heavy `*_gdal`); the setup for each tier below uses its own. The generic readers (`raster_gbx` light, `gdal` heavy) also read any raster their tier's engine supports. + +The four `*_DIR` placeholders in the code below are wired to small sample rasters **committed in the GeoBrix repo** under [`src/test/resources/binary/`](https://github.com/databrickslabs/geobrix/tree/main/src/test/resources/binary) (resolved by the `_SAMPLE_PATHS[...]` / `*_path()` helpers you'll see in the snippet). To run the examples yourself, stage those sample files — or your own rasters — in a Unity Catalog Volume directory and set each placeholder to its Volume path. + +The setup below loads `GTIFF_SAMPLE_DIR` into `rasters` and then loads the other three the same way — swap the path and view name (and, for NetCDF, the reader). Every example on the page assumes these four views exist. @@ -98,7 +129,9 @@ Pick your execution tier and run this once. Both tiers alias the module as `rx`, code={rasterxCode} /> -After registering RasterX, create the SQL view so the SQL examples below can use `FROM rasters`: +**Scala** uses this same heavyweight (`rasterx`) setup: create the four views with `spark.read.format("gtiff_gdal").load(...).createOrReplaceTempView("rasters")` (and `netcdf_gdal` for the NetCDF view), then each Scala example reads `val df = spark.table("multiband_rasters")`. Registration is only needed for the SQL functions — the Scala `rx.*` Column API works without it. + +Prefer pure SQL? Register RasterX (Python: `rx.register(spark)`), then create the same four views directly so the SQL examples can use `FROM rasters`, `FROM multiband_rasters`, and so on: -## Usage Examples +## Tier availability -### Python/PySpark +As of v0.4.0, **all RasterX functions run in both execution tiers** — the lightweight `pyrx` (pure-Python) and heavyweight `rasterx` tiers share the same `rst_*` / `gbx_rst_*` names, and each function below carries a `:::note Lightweight tier (pyrx)` admonition with its backing library and any behavioral differences. For the heavyweight VRT Python pixel-function configuration (used by `gbx_rst_combineavg` / `gbx_rst_derivedband`), see **[VRT Python pixel functions](#vrt-python-pixel-functions)** at the end of this page. -These examples assume your tier is set up as in [Setup](#setup) above — imported as `rx`, registered (for the SQL examples), and a raster `DataFrame` (`raster_df`, with a `tile` column) loaded for your tier. The calls are identical in both tiers: +## Examples — Conventions -```python -# Read raster properties off the `tile` column: -metadata_df = raster_df.select( - rx.rst_width("tile").alias("width"), - rx.rst_height("tile").alias("height"), - rx.rst_numbands("tile").alias("bands"), - rx.rst_srid("tile").alias("srid"), -) -metadata_df.show() -``` +Every function on this page shows one example, expressed identically across four tabs. This section defines the shared setup so each function tab contains only its invocation — nothing more. -### Scala +### Canonical sample files - +Each function uses one of four canonical files, chosen by what the function demonstrates: -### SQL +| File | What it demonstrates | Used by | +|---|---|---| +| `nyc_sentinel2_red.tif` | Single-band GeoTIFF (Sentinel-2 red band, NYC area) | **Default** — most accessor, tile-ops, transform, and generator functions | +| `rgb_nir_small.tif` | 3-band GeoTIFF with per-band metadata (red, NIR, green — 8×8 px) | Band-math and spectral-index functions (`rst_ndvi`, `rst_evi`, …), `rst_numbands`, `rst_bandmetadata` | +| `srtm_n40w073.tif` | Digital Elevation Model (SRTM, NYC area) | Terrain functions (`rst_slope`, `rst_aspect`, `rst_hillshade`, …) | +| CMIP5 NetCDF (`prAdjust_day_…nc`) | Multi-variable NetCDF with two subdatasets | `rst_subdatasets`, `rst_getsubdataset` only | - +### The `tile`-column convention ---- +In every example on this page: -:::note SQL examples -Examples on this page use **SQL**, where RasterX functions are prefixed with **`gbx_`** (e.g. `gbx_rst_boundingbox`, `gbx_rst_width`). For Python and Scala usage and more tips, see [Language Bindings](./language-bindings). In the lightweight tier, the registered `gbx_rst_*` SQL functions require every argument to be passed explicitly — optional defaults are honored only through the Python `prx.*` API. -::: +- **SQL:** `rasters` is a temporary view whose `tile` column holds the canonical sample loaded via the reader. `FROM rasters` means "the sample as tiles." +- **Python (light and heavy):** `df` is a DataFrame with a `tile` column loaded from the canonical file. `df.select(...)` means "apply this function to the sample tiles." +- **Scala:** `rasters` is a DataFrame with the same `tile` column. `rasters.select(...)` is identical in intent to the Python form. -## Tier availability +Each function's example therefore shows only the invocation — the load is the convention, not the code. -As of v0.4.0, **all RasterX functions run in both execution tiers** — the lightweight `pyrx` (pure-Python) and heavyweight `rasterx` tiers share the same `rst_*` / `gbx_rst_*` names, and each function below carries a `:::note Lightweight tier (pyrx)` admonition with its backing library and any behavioral differences. For the heavyweight VRT Python pixel-function configuration (used by `gbx_rst_combineavg` / `gbx_rst_derivedband`), see **[VRT Python pixel functions](#vrt-python-pixel-functions)** at the end of this page. +### How to read the four tabs + +| Tab | Tier | Color | +|---|---|---| +| **SQL** | Both (default) | — | +| **Python (light)** | `pyrx` lightweight tier | — | +| **Python (heavy)** | `rasterx` heavyweight tier | Blue badge | +| **Scala** | `rasterx` heavyweight tier | Blue badge | + +All four tabs show the **same operation** on the **same file**. Where a tier's output genuinely differs in form, a short note explains why — for example: + +- Geometry-returning functions (`rst_boundingbox`, `rst_georeference`): SQL returns WKT; Python and Scala return a binary column. The note reads `...` `(WKB binary)`. +- Subdataset maps (`rst_subdatasets`): SQL returns `map`; rendered as `{SUBDATASET_1_NAME -> ..., SUBDATASET_1_DESC -> ...}`. A note clarifies the key pattern. +- Band-metadata maps (`rst_bandmetadata`): similarly noted where the rendering differs between tiers. + +Scalar values are **identical across all tabs** — same fixture, same function, same result. + +Tile-returning functions (`rst_clip`, `rst_resample`, `rst_transform`, the aggregators, the constructors, …) return a **[v2 Tile](./tile-structure)**. Their output is shown as a representative struct — `{0, , , {driver -> GTiff, ...}}` — rather than a literal byte dump. Tiles loaded via `rst_fromcontent` / `rst_fromfile` (as in [Setup](#setup)) are **materialized**: the `raster` field holds the encoded bytes and `path` is null. [**Virtual tiles**](./virtual-tiles) — a populated `path` with lazily-read bytes — arise from the force-output parameters, not from the default load. + +**SQL naming and arguments:** in the SQL tab, RasterX functions carry the **`gbx_`** prefix (e.g. `gbx_rst_boundingbox`, `gbx_rst_width`); Python and Scala use the bare `rst_*` name via `rx`. In the lightweight tier the registered `gbx_rst_*` SQL functions require **every argument to be passed explicitly** — optional defaults are honored only through the Python `rx.*` API. See [Language Bindings](./language-bindings) for more. + +### Per-function notes + +Functions that use a non-default fixture (multiband GeoTIFF, DEM, or NetCDF) carry a one-line note immediately before their code example flagging the file. Functions that produce a tile rather than consuming one (constructors: `rst_fromfile`, `rst_fromcontent`, `rst_frombands`, …) show a fuller load/build example with a note — the bare-invocation model does not apply when there is no input `tile` yet. ## Accessor Functions @@ -178,9 +216,16 @@ Powered by **rasterio** + **NumPy**. Per-band mean over valid (non-NoData) pixel Returns `NULL` for a band with zero valid pixels (all NoData) on both tiers. -**SQL:** +_Examples use the multiband fixture (`rgb_nir_small.tif`, 3 bands: red, NIR, green); the canonical single-band sentinel2 tile is all-NoData for this function._ - + ### rst_bandmetadata @@ -192,9 +237,16 @@ Powered by **rasterio**. **Signature:** `rst_bandmetadata(tile: Column, band: Column): Column` — Band metadata map. -**SQL:** +_Examples use the multiband fixture (`rgb_nir_small.tif`, 3 bands with per-band GDAL metadata tags); plain single-band GeoTIFFs without tags return an empty map `{}`._ - + ### rst_boundingbox @@ -206,9 +258,41 @@ Powered by **rasterio**. **Signature:** `rst_boundingbox(tile: Column): Column` — Bounding box geometry. -**SQL:** + + +### rst_crs + + + +:::note Lightweight tier (pyrx) +Powered by **rasterio** + **pyproj**. +::: + +**Signature:** `rst_crs(tile: Column): Column` — the tile's CRS as a string (authority string like `EPSG:4326` / `ESRI:54008`, else WKT); **always** returns a value, including for non-EPSG rasters where `rst_srid` is NULL. See [Coordinate Reference Systems](./coordinate-reference-systems). + +Returns the authority string (`AUTHORITY:CODE`, e.g. `EPSG:4326` or `ESRI:54008`) when the CRS has one, otherwise its WKT. Unlike `rst_srid` (which returns the integer EPSG code, or `NULL`/`0` for a CRS with no EPSG code), `rst_crs` always returns a value — including for non-EPSG rasters defined by an ESRI code, WKT, or PROJ4 string. - +:::tip CRS: SRID int vs CRS string +- **`rst_srid`** returns the integer EPSG code (e.g. `4326`); `NULL` (lightweight) or `0` (heavyweight) when the CRS has no EPSG code. Use it when you need the numeric SRID for the native ST bridge or an EPSG-only workflow. +- **`rst_crs`** returns the CRS string and never loses a non-EPSG CRS. +- **`rst_setcrs`** / **`rst_transformcrs`** take a CRS *string*. An int-castable string is treated as an EPSG SRID (`'4326'` behaves like SRID `4326`); otherwise the string is parsed as an authority code (`EPSG:`/`ESRI:`), WKT, or PROJ4. This is how ESRI codes, WKT, and PROJ4 definitions survive a round trip. +::: + + ### rst_format @@ -220,9 +304,14 @@ Powered by **rasterio**. **Signature:** `rst_format(tile: Column): Column` — GDAL format name. -**SQL:** - - + ### rst_georeference @@ -247,9 +336,14 @@ The result is a **MapType** with the following keys, corresponding to [GDAL's 6- See the [GDAL geotransform tutorial](https://gdal.org/en/stable/tutorials/geotransforms_tut.html) and [raster data model](https://gdal.org/en/stable/user/raster_data_model.html) for details. -**SQL:** - - + ### rst_getnodata @@ -261,9 +355,14 @@ Powered by **rasterio**. Returns the dataset NoData value repeated once per band **Signature:** `rst_getnodata(tile: Column): Column` — NoData values per band. -**SQL:** - - + ### rst_getsubdataset @@ -275,9 +374,16 @@ Powered by **rasterio**. Subdataset availability depends on rasterio's bundled G **Signature:** `rst_getsubdataset(tile: Column, subsetName: Column): Column` — Extract subdataset. -**SQL:** +_Examples use the CMIP5 NetCDF fixture (`prAdjust_day_HadGEM2-CC_*.nc`) which has two subdatasets: `time_bnds` and `prAdjust`. Subdatasets require a multi-layer format such as NetCDF; plain GeoTIFFs return no subdatasets._ - + ### rst_height @@ -289,9 +395,14 @@ Powered by **rasterio**. **Signature:** `rst_height(tile: Column): Column` — Height in pixels. -**SQL:** - - + ### rst_max @@ -305,9 +416,16 @@ Powered by **rasterio** + **NumPy**. Per-band maximum over valid (non-NoData) pi Returns `NULL` for a band with zero valid pixels (all NoData) on both tiers. -**SQL:** +_Examples use the multiband fixture (`rgb_nir_small.tif`, 3 bands); the canonical single-band sentinel2 tile is all-NoData for this function._ - + ### rst_median @@ -321,9 +439,16 @@ Powered by **rasterio** + **NumPy**. Per-band median over valid (non-NoData) pix Returns `NULL` for a band with zero valid pixels (all NoData) on both tiers. -**SQL:** +_Examples use the multiband fixture (`rgb_nir_small.tif`, 3 bands); the canonical single-band sentinel2 tile is all-NoData for this function._ - + ### rst_memsize @@ -335,9 +460,14 @@ Powered by **rasterio**. Returns the serialized raster size in bytes. **Signature:** `rst_memsize(tile: Column): Column` — In-memory size in bytes. -**SQL:** - - + ### rst_metadata @@ -349,9 +479,14 @@ Powered by **rasterio**. **Signature:** `rst_metadata(tile: Column): Column` — Metadata map. -**SQL:** - - + ### rst_min @@ -365,9 +500,16 @@ Powered by **rasterio** + **NumPy**. Per-band minimum over valid (non-NoData) pi Returns `NULL` for a band with zero valid pixels (all NoData) on both tiers. -**SQL:** +_Examples use the multiband fixture (`rgb_nir_small.tif`, 3 bands); the canonical single-band sentinel2 tile is all-NoData for this function._ - + ### rst_numbands @@ -379,9 +521,16 @@ Powered by **rasterio**. **Signature:** `rst_numbands(tile: Column): Column` — Number of bands. -**SQL:** +_Examples use the multiband fixture (`rgb_nir_small.tif`, 3 bands: red, NIR, green)._ - + ### rst_pixelcount @@ -393,9 +542,16 @@ Powered by **rasterio** + **NumPy**. Per-band count of valid (non-NoData) pixels **Signature:** `rst_pixelcount(tile: Column): Column` — Total pixel count. -**SQL:** +_Examples use the multiband fixture (`rgb_nir_small.tif`, 8×8, 3 bands, no NoData set), yielding 64 valid pixels per band; the canonical single-band sentinel2 tile has NoData=0 with all pixels equal zero, returning `[0]`._ - + ### rst_pixelheight @@ -407,9 +563,14 @@ Powered by **rasterio**. **Signature:** `rst_pixelheight(tile: Column): Column` — Pixel height in ground units. -**SQL:** - - + ### rst_pixelwidth @@ -421,9 +582,14 @@ Powered by **rasterio**. **Signature:** `rst_pixelwidth(tile: Column): Column` — Pixel width in ground units. -**SQL:** - - + ### rst_rotation @@ -435,9 +601,14 @@ Powered by **rasterio**. **Signature:** `rst_rotation(tile: Column): Column` — Rotation in radians. -**SQL:** - - + ### rst_scalex / rst_scaley @@ -449,9 +620,23 @@ Powered by **rasterio**. **Signature:** `rst_scalex(tile: Column): Column`, `rst_scaley(tile: Column): Column` — Scale (pixel size) in X/Y. -**SQL:** + - + ### rst_skewx / rst_skewy @@ -463,9 +648,23 @@ Powered by **rasterio**. **Signature:** `rst_skewx(tile: Column): Column`, `rst_skewy(tile: Column): Column` — Skew in X/Y. -**SQL:** + - + ### rst_srid @@ -475,11 +674,16 @@ Powered by **rasterio**. Powered by **rasterio**. ::: -**Signature:** `rst_srid(tile: Column): Column` — Spatial reference ID (e.g. EPSG). +**Signature:** `rst_srid(tile: Column): Column` — the stored spatial reference ID integer (an EPSG or ESRI code), or NULL when the tile has none. Returns the code as stored; it is classified only when applied. See [Coordinate Reference Systems](./coordinate-reference-systems) for the SRID-vs-CRS-string model and the epsg→esri resolution rule. -**SQL:** - - + ### rst_subdatasets @@ -491,9 +695,16 @@ Powered by **rasterio**. Empty for single-dataset rasters (e.g. a plain GeoTIFF) **Signature:** `rst_subdatasets(tile: Column): Column` — List of subdataset names. -**SQL:** +_Examples use the CMIP5 NetCDF fixture (`prAdjust_day_HadGEM2-CC_*.nc`) which has two subdatasets: `time_bnds` and `prAdjust`. Subdatasets require a multi-layer format; plain GeoTIFFs return an empty map._ - + ### rst_summary @@ -505,9 +716,16 @@ Powered by **rasterio** + **NumPy**. Returns a JSON summary (driver, size, CRS, **Signature:** `rst_summary(tile: Column): Column` — Statistical summary of values. -**SQL:** +_Examples use the multiband fixture (`rgb_nir_small.tif`, 3 bands) which has real pixel data._ - + ### rst_type @@ -519,9 +737,16 @@ Powered by **rasterio**. **Signature:** `rst_type(tile: Column): Column` — Data type per band. -**SQL:** +_Examples use the multiband fixture (`rgb_nir_small.tif`, 3 bands, `UInt16`)._ - + ### rst_upperleftx / rst_upperlefty @@ -533,9 +758,23 @@ Powered by **rasterio**. **Signature:** `rst_upperleftx(tile: Column): Column`, `rst_upperlefty(tile: Column): Column` — Upper-left corner coordinates. -**SQL:** + - + ### rst_width @@ -547,9 +786,14 @@ Powered by **rasterio**. **Signature:** `rst_width(tile: Column): Column` — Width in pixels. -**SQL:** - - + --- @@ -557,6 +801,54 @@ Powered by **rasterio**. Combine or merge rasters in group-by (7 total). +### rst_bng_rasterize_agg + + + +:::note Lightweight tier (pyrx) +The lightweight implementation is backed by `pygx._bng` cell math and `rasterio`. BNG cell IDs (**STRING**) are parsed via `pygx._bng` to EPSG:27700-native cell geometries and burned into the output band — the output canvas is EPSG:27700 throughout with no warp step (this aggregator takes cellid+value rows, not a raster tile). +::: + +:::warning NoData sentinel is -9999.0 +Pixels not covered by any geometry in the group are set to **`-9999.0`** (band-registered NoData). Filter or mask downstream via `gbx_rst_getnodata` or `IS NULL` on the extracted band value. +::: + +:::warning Lightweight SQL returns BINARY (not the tile struct) +Heavyweight `gbx_rst_bng_rasterize_agg` returns a tile **`STRUCT`** (the v2 8-field tile); the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_bng_rasterize_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: + +```sql +-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry +SELECT + cell_id AS cellid, + gbx_rst_fromcontent(gbx_rst_bng_rasterize_agg(cellid, burn_value), 'GTiff') AS tile +FROM bng_cell_values +GROUP BY cell_id +``` +::: + +Streaming aggregator that burns BNG cell geometry/value pairs (one row per cell) into a single rasterized tile per group. The input raster is automatically reprojected to **EPSG:27700** (British National Grid) before rasterization. BNG cell IDs are **STRING**. The inverse of `rst_bng_rastertogrid*`: where those functions reduce raster pixels to per-cell statistics, this one synthesizes a raster from per-cell values. + +:::note Lightweight-only `out_crs` parameter (no-op for BNG) +The lightweight **Python** binding accepts an optional trailing `out_crs` (string CRS) argument that the heavyweight/Scala tier does not — but for BNG it is a **no-op**: the output is always EPSG:27700. Both `out_srid` and `out_crs` are ignored here (they exist only for signature parity with the H3/quadbin aggregators). +::: + +**Signature:** `rst_bng_rasterize_agg(cellid: Column, value: Column, out_srid: Column, pixel_size: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, mode: Column, kring_pad: Column): Column` + +**Parameters:** `cellid` — BNG cell ID (STRING); `value` — numeric burn value; `out_srid` — EPSG code for the output CRS (typically `27700`); `pixel_size` — output raster cell size in metres (usually `1.0` for 1km BNG cells); `xmin/ymin/xmax/ymax` — output extent in EPSG:27700 metres; `width/height` — output raster dimensions in pixels; `mode` — aggregation mode for overlapping values (typically `"last"`); `kring_pad` — cell neighbourhood expansion (typically `0`) + +_Multi-row fixture: 3 BNG 1km STRING cell rows near central London (EPSG:27700) with burn values 1.0/2.0/3.0. All tabs use the same grouped-agg rasterize invocation._ + + + +--- + ### rst_combineavg_agg @@ -566,7 +858,7 @@ Powered by **rasterio** + **NumPy**. Aggregate — `groupBy(...).agg(rx.rst_comb ::: :::warning Lightweight SQL returns BINARY (not the tile struct) -Heavyweight `gbx_rst_combineavg_agg` returns a tile **`STRUCT`**; the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_combineavg_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: +Heavyweight `gbx_rst_combineavg_agg` returns a tile **`STRUCT`** (the v2 8-field tile); the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_combineavg_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: ```sql -- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry @@ -580,9 +872,16 @@ GROUP BY group_key **Signature:** `rst_combineavg_agg(tile: Column): Column` — Average tiles per group. -**SQL:** +_Multi-tile fixture: 3 per-band rows from `rgb_nir_small.tif` (same grid). All tabs use the same grouped-agg invocation on multiple `tile` rows._ - + ### rst_derivedband_agg @@ -593,7 +892,7 @@ Powered by **rasterio** with GDAL VRT Python pixel functions. Aggregate — `gro ::: :::warning Lightweight SQL returns BINARY (not the tile struct) -Heavyweight `gbx_rst_derivedband_agg` returns a tile **`STRUCT`**; the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_derivedband_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: +Heavyweight `gbx_rst_derivedband_agg` returns a tile **`STRUCT`** (the v2 8-field tile); the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_derivedband_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: ```sql -- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry @@ -607,20 +906,27 @@ GROUP BY group_key **Signature:** `rst_derivedband_agg(tile: Column, pyfunc: String, funcName: String): Column` — Apply Python UDF to tiles per group. -**SQL:** +_Multi-tile fixture: 3 per-band rows from `rgb_nir_small.tif`. All tabs use the same grouped-agg invocation; pixel function selects band 0 (identity)._ - + ### rst_dtmfromgeoms_agg :::note Lightweight tier (pyrx) -Powered by **rasterio** + **SciPy** (`scipy.spatial.Delaunay`). Aggregate — builds one TIN DTM tile per group from the group's Z-valued points via barycentric interpolation over an **unconstrained** Delaunay triangulation; `breaklines`, `merge_tolerance`, and `snap_tolerance` are accepted but not enforced (the heavyweight tier builds a constrained TIN). +Powered by **rasterio** + **SciPy** (`scipy.spatial.Delaunay`). Aggregate — builds one TIN DTM tile per group from the group's Z-valued points via barycentric interpolation over an **unconstrained** Delaunay triangulation; `breaklines`, `merge_tolerance`, and `snap_tolerance` are accepted but not enforced (the heavyweight tier builds a constrained TIN). The lightweight Python binding also accepts an optional trailing `out_crs` (string CRS) argument that the SQL/heavyweight tiers do not — the heavyweight builder takes the int `out_srid` only. `out_crs` wins over `out_srid` when both are given. This is a lightweight superset, not a heavyweight regression. ::: :::warning Lightweight SQL returns BINARY (not the tile struct) -Heavyweight `gbx_rst_dtmfromgeoms_agg` returns a tile **`STRUCT`**; the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_dtmfromgeoms_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: +Heavyweight `gbx_rst_dtmfromgeoms_agg` returns a tile **`STRUCT`** (the v2 8-field tile); the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_dtmfromgeoms_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: ```sql -- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry @@ -637,13 +943,20 @@ GROUP BY group_key Streaming aggregator that accepts one Z-valued point WKB per row and produces a TIN/Delaunay DTM raster tile per group; breaklines are supplied as a per-group constant array to enforce hard terrain edges. -**Signature:** `rst_dtmfromgeoms_agg(point: Column, breaklines: Column, mergeTolerance: Column, snapTolerance: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, srid: Column): Column` +**Signature:** `rst_dtmfromgeoms_agg(point: Column, breaklines: Column, mergeTolerance: Column, snapTolerance: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, out_srid: Column): Column` **Parameters:** `point` — WKB point geometry with Z coordinate (one per row); `breaklines` — constant WKB array of breakline geometries per group (pass `null` or empty array if unused); remaining parameters match `rst_dtmfromgeoms` -**SQL:** +_Multi-row fixture: 4 Z-valued WKB POINT rows (elevation 100–250 m) over a `[0,0,1,1]` EPSG:4326 extent. All tabs use the same grouped-agg invocation._ - + ### rst_frombands_agg @@ -654,7 +967,7 @@ Powered by **rasterio**. Aggregate — `groupBy(...).agg(rx.rst_frombands_agg("t ::: :::warning Lightweight SQL returns BINARY (not the tile struct) -Heavyweight `gbx_rst_frombands_agg` returns a tile **`STRUCT`**; the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_frombands_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: +Heavyweight `gbx_rst_frombands_agg` returns a tile **`STRUCT`** (the v2 8-field tile); the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_frombands_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: ```sql -- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry @@ -672,81 +985,27 @@ Streaming aggregator that collects ordered per-band tiles (one row per band) int **Parameters:** `tile` — Single-band raster tile; `bandIndex` — 1-based band position within the output raster -**SQL:** - - - -### rst_merge_agg - - - -:::note Lightweight tier (pyrx) -Powered by **rasterio** (`rasterio.merge`). Aggregate — `groupBy(...).agg(rx.rst_merge_agg("tile"))` merges the group's tiles into one mosaic tile (output spans the union extent). -::: - -:::warning Lightweight SQL returns BINARY (not the tile struct) -Heavyweight `gbx_rst_merge_agg` returns a tile **`STRUCT`**; the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_merge_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: - -```sql --- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry -SELECT - group_key AS cellid, - gbx_rst_fromcontent(gbx_rst_merge_agg(tile), 'GTiff') AS tile -FROM tiles -GROUP BY group_key -``` -::: - -**Signature:** `rst_merge_agg(tile: Column): Column` — Merge tiles per group. - -**SQL:** - - - -### rst_rasterize_agg - - - -:::note Lightweight tier (pyrx) -Powered by **rasterio** (`rasterio.features`). Aggregate — burns the group's `(geom, value)` rows into one tile over the given extent/size/SRID (last-wins on overlap). -::: - -:::warning Lightweight SQL returns BINARY (not the tile struct) -Heavyweight `gbx_rst_rasterize_agg` returns a tile **`STRUCT`**; the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_rasterize_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: - -```sql --- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry -SELECT - group_key AS cellid, - gbx_rst_fromcontent( - gbx_rst_rasterize_agg(geom, value, 0,0,10,10, 8,8, 32633), - 'GTiff' - ) AS tile -FROM features -GROUP BY group_key -``` -::: - -Streaming aggregator that burns geometry/value pairs (one row per feature) into a single rasterized tile per group; use when features arrive as individual rows rather than as a pre-built collection. - -**Signature:** `rst_rasterize_agg(geom: Column, value: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, srid: Column): Column` - -**Parameters:** `geom` — WKB geometry to burn; `value` — numeric burn value; `xmin/ymin/xmax/ymax` — output extent (in the target CRS); `width/height` — output raster dimensions in pixels; `srid` — EPSG code for the output CRS - -**SQL:** +_Multi-tile fixture: 3 per-band rows from `rgb_nir_small.tif` with `band_index` 1/2/3. All tabs stack via the same grouped-agg invocation._ - + ### rst_gridfrompoints_agg :::note Lightweight tier (pyrx) -Powered by **rasterio** + **SciPy** (`cKDTree` IDW). Aggregate — `groupBy(...).agg(rx.rst_gridfrompoints_agg(...))` inverse-distance-interpolates the group's points into one Float64 grid tile (NoData −9999). +Powered by **rasterio** + **SciPy** (`cKDTree` IDW). Aggregate — `groupBy(...).agg(rx.rst_gridfrompoints_agg(...))` inverse-distance-interpolates the group's points into one Float64 grid tile (NoData −9999). The lightweight Python binding also accepts an optional trailing `out_crs` (string CRS) argument that the SQL/heavyweight tiers do not — the heavyweight builder takes the int `out_srid` only. `out_crs` wins over `out_srid` when both are given. This is a lightweight superset, not a heavyweight regression. ::: :::warning Lightweight SQL returns BINARY (not the tile struct) -Heavyweight `gbx_rst_gridfrompoints_agg` returns a tile **`STRUCT`**; the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_gridfrompoints_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: +Heavyweight `gbx_rst_gridfrompoints_agg` returns a tile **`STRUCT`** (the v2 8-field tile); the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_gridfrompoints_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: ```sql -- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry @@ -763,13 +1022,20 @@ GROUP BY group_key Streaming IDW-interpolation aggregator that accepts one point geometry and one scalar value per row and produces a Float64 GeoTIFF tile per group; use when observations arrive one per row rather than as pre-built arrays. -**Signature:** `rst_gridfrompoints_agg(point: Column, value: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, widthPx: Column, heightPx: Column, srid: Column, power: Column, maxPts: Column): Column` +**Signature:** `rst_gridfrompoints_agg(point: Column, value: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, widthPx: Column, heightPx: Column, out_srid: Column, power: Column, maxPts: Column): Column` **Parameters:** `point` — WKB point geometry (one per row); `value` — scalar observation for the point; `xmin/ymin/xmax/ymax` — output extent in CRS units (constant per group); `widthPx/heightPx` — output dimensions in pixels; `srid` — EPSG code; `power` — IDW distance-decay exponent (2.0 is standard); `maxPts` — maximum nearest neighbours considered per output pixel -**SQL:** +_Multi-row fixture: 4 WKB POINT rows with observations 10–40 over a `[0,0,1,1]` EPSG:4326 extent. All tabs use the same grouped-agg IDW invocation._ - + ### rst_h3_rasterize_agg @@ -780,7 +1046,7 @@ Powered by **rasterio** + **h3**. Aggregate — `groupBy(...).agg(rx.rst_h3_rast ::: :::warning Lightweight SQL returns BINARY (not the tile struct) -Heavyweight `gbx_rst_h3_rasterize_agg` returns a tile **`STRUCT`**; the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_h3_rasterize_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: +Heavyweight `gbx_rst_h3_rasterize_agg` returns a tile **`STRUCT`** (the v2 8-field tile); the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_h3_rasterize_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: ```sql -- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry @@ -797,11 +1063,15 @@ GROUP BY region_id Streaming aggregator that burns H3 cell centroid pixels (or spatial-envelope pixels) into one raster tile per group. This is the **inverse** of [`rst_h3_rastertogrid*`](#rst_h3_rastertogridavg): where those functions reduce raster pixels to per-cell statistics, `rst_h3_rasterize_agg` reconstructs a raster from per-cell values. Use [`rst_frombands_agg`](#rst_frombands_agg) to stack per-threshold rasters (each produced by one `rst_h3_rasterize_agg` call) into a single multi-band output. +:::note Lightweight-only `out_crs` parameter +The lightweight **Python** binding accepts an optional trailing `out_crs` (string CRS) argument that the heavyweight/Scala tier does not. When supplied it overrides the integer `out_srid` for the output CRS; the heavyweight tier accepts only the integer `out_srid` (its `builder()` is strictly 12-argument). This is a lightweight superset, not a heavyweight regression. +::: + :::tip Worked example The [H3 Rasterize notebook](../notebooks/h3-rasterize) walks this through end to end on a San Francisco Bay Area DEM: elevation isobands → H3 polyfill → a shared canvas from [`rst_h3_gridspec`](#h3-grid) → per-band `rst_h3_rasterize_agg` → multi-band stack via [`rst_frombands_agg`](#rst_frombands_agg), visualized with the [`gbx.vizx`](./vizx) helpers. The same pattern maps directly to a telco multi-threshold signal-coverage stack. ::: -**Signature:** `rst_h3_rasterize_agg(cellid: Column, value: Column, srid: Column, pixel_size: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, mode: Column, kring_pad: Column): Column` +**Signature:** `rst_h3_rasterize_agg(cellid: Column, value: Column, out_srid: Column, pixel_size: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, mode: Column, kring_pad: Column): Column` **Parameters:** - `cellid` — H3 cell ID (BIGINT or STRING) to burn (one per row) @@ -813,107 +1083,198 @@ The [H3 Rasterize notebook](../notebooks/h3-rasterize) walks this through end to - `mode` — `'centroids'` (default, burns the cell-centroid pixel only) or `'spatial_envelope'` (burns all pixels inside the hexagon envelope) - `kring_pad` — ring count by which to expand the auto-computed canvas (default `1`) -**SQL:** +_Multi-row fixture: 3 H3 resolution-9 cell rows with burn values 1.0/2.0/3.0. All tabs use the same grouped-agg rasterize invocation._ - + -### rst_quadbin_rasterize_agg +### rst_merge_agg :::note Lightweight tier (pyrx) -The lightweight implementation is backed by `pygx._quadbin` cell math and `rasterio` — it burns each quadbin cell's geometry/value into the output band via `rasterio.features.rasterize`, matching the heavyweight cell set and burn values exactly. +Powered by **rasterio** (`rasterio.merge`). Aggregate — `groupBy(...).agg(rx.rst_merge_agg("tile"))` merges the group's tiles into one mosaic tile (output spans the union extent). ::: -:::warning NoData sentinel is -9999.0 -Pixels not covered by any geometry in the group are set to **`-9999.0`** (band-registered NoData). Filter or mask downstream via `gbx_rst_getnodata` or `IS NULL` on the extracted band value. +:::warning Lightweight SQL returns BINARY (not the tile struct) +Heavyweight `gbx_rst_merge_agg` returns a tile **`STRUCT`** (the v2 8-field tile); the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_merge_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: + +```sql +-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry +SELECT + group_key AS cellid, + gbx_rst_fromcontent(gbx_rst_merge_agg(tile), 'GTiff') AS tile +FROM tiles +GROUP BY group_key +``` ::: -Streaming aggregator that burns quadbin cell geometry/value pairs (one row per cell) into a single rasterized tile per group. The input raster is interpreted as EPSG:4326 (lon/lat); resolution is the quadbin zoom level (0..26). The inverse of `rst_quadbin_rastertogrid*`: where those functions reduce raster pixels to per-cell statistics, this one synthesizes a raster from per-cell values. - -`gbx_rst_quadbin_rasterize_agg` returns a tile **`STRUCT`**. - -**Signature:** `rst_quadbin_rasterize_agg(geom: Column, value: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, srid: Column): Column` - -**Parameters:** `geom` — WKB geometry to burn (one per row); `value` — numeric burn value; `xmin/ymin/xmax/ymax` — output extent (in the target CRS); `width/height` — output raster dimensions in pixels; `srid` — EPSG code for the output CRS +**Signature:** `rst_merge_agg(tile: Column): Column` — Merge tiles per group. -**SQL:** +_Multi-tile fixture: 3 per-band rows from `rgb_nir_small.tif`. All tabs use the same grouped-agg mosaic invocation._ - + -### rst_bng_rasterize_agg +### rst_quadbin_rasterize_agg :::note Lightweight tier (pyrx) -The lightweight implementation is backed by `pygx._bng` cell math and `rasterio`. BNG cell IDs (**STRING**) are parsed via `pygx._bng` to EPSG:27700-native cell geometries and burned into the output band — the output canvas is EPSG:27700 throughout with no warp step (this aggregator takes cellid+value rows, not a raster tile). +The lightweight implementation is backed by `pygx._quadbin` cell math and `rasterio` — it burns each quadbin cell's geometry/value into the output band via `rasterio.features.rasterize`, matching the heavyweight cell set and burn values exactly. ::: :::warning NoData sentinel is -9999.0 Pixels not covered by any geometry in the group are set to **`-9999.0`** (band-registered NoData). Filter or mask downstream via `gbx_rst_getnodata` or `IS NULL` on the extracted band value. ::: -Streaming aggregator that burns BNG cell geometry/value pairs (one row per cell) into a single rasterized tile per group. The input raster is automatically reprojected to **EPSG:27700** (British National Grid) before rasterization. BNG cell IDs are **STRING**. The inverse of `rst_bng_rastertogrid*`: where those functions reduce raster pixels to per-cell statistics, this one synthesizes a raster from per-cell values. +:::warning Lightweight SQL returns BINARY (not the tile struct) +Heavyweight `gbx_rst_quadbin_rasterize_agg` returns a tile **`STRUCT`** (the v2 8-field tile); the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_quadbin_rasterize_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: -`gbx_rst_bng_rasterize_agg` returns a tile **`STRUCT`**. +```sql +-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry +SELECT + cell_id AS cellid, + gbx_rst_fromcontent(gbx_rst_quadbin_rasterize_agg(cellid, burn_value), 'GTiff') AS tile +FROM quadbin_cell_values +GROUP BY cell_id +``` +::: -**Signature:** `rst_bng_rasterize_agg(geom: Column, value: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, srid: Column): Column` +Streaming aggregator that burns quadbin cell geometry/value pairs (one row per cell) into a single rasterized tile per group. The input raster is interpreted as EPSG:4326 (lon/lat); resolution is the quadbin zoom level (0..26). The inverse of `rst_quadbin_rastertogrid*`: where those functions reduce raster pixels to per-cell statistics, this one synthesizes a raster from per-cell values. -**Parameters:** `geom` — WKB geometry to burn (one per row); `value` — numeric burn value; `xmin/ymin/xmax/ymax` — output extent in EPSG:27700 metres; `width/height` — output raster dimensions in pixels; `srid` — EPSG code for the output CRS (typically `27700`) +:::note Lightweight-only `out_crs` parameter +The lightweight **Python** binding accepts an optional trailing `out_crs` (string CRS) argument that the heavyweight/Scala tier does not. When supplied it overrides the integer `out_srid` for the output CRS; the heavyweight tier accepts only the integer `out_srid` (its `builder()` is strictly 12-argument). This is a lightweight superset, not a heavyweight regression. +::: -**SQL:** +**Signature:** `rst_quadbin_rasterize_agg(cellid: Column, value: Column, out_srid: Column, pixel_size: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, mode: Column, kring_pad: Column): Column` - +**Parameters:** `cellid` — quadbin cell ID (BIGINT); `value` — numeric burn value; `out_srid` — EPSG code for the output CRS; `pixel_size` — output raster cell size in the target CRS; `xmin/ymin/xmax/ymax` — output extent in the target CRS; `width/height` — output raster dimensions in pixels; `mode` — aggregation mode for overlapping values (typically `"last"`); `kring_pad` — cell neighbourhood expansion (typically `0`) ---- +_Multi-row fixture: 3 quadbin zoom-12 cell rows near central London with burn values 1.0/2.0/3.0. All tabs use the same grouped-agg rasterize invocation._ + + + +### rst_rasterize_agg + + + +:::note Lightweight tier (pyrx) +Powered by **rasterio** (`rasterio.features`). Aggregate — burns the group's `(geom, value)` rows into one tile over the given extent/size/SRID (last-wins on overlap). +::: + +:::warning Lightweight SQL returns BINARY (not the tile struct) +Heavyweight `gbx_rst_rasterize_agg` returns a tile **`STRUCT`** (the v2 8-field tile); the lightweight **SQL** function returns **`BINARY`** (the raster bytes). A PySpark grouped-aggregate `pandas_udf` cannot return a `StructType`, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight **Python** wrapper `rx.rst_rasterize_agg(...)` returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as `cellid` and wrap the BINARY with `gbx_rst_fromcontent`: + +```sql +-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry +SELECT + group_key AS cellid, + gbx_rst_fromcontent( + gbx_rst_rasterize_agg(geom, value, 0,0,10,10, 8,8, 32633), + 'GTiff' + ) AS tile +FROM features +GROUP BY group_key +``` +::: + +Streaming aggregator that burns geometry/value pairs (one row per feature) into a single rasterized tile per group; use when features arrive as individual rows rather than as a pre-built collection. + +**Signature:** `rst_rasterize_agg(geom: Column, value: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, out_srid: Column): Column` + +**Parameters:** `geom` — WKB geometry to burn; `value` — numeric burn value; `xmin/ymin/xmax/ymax` — output extent (in the target CRS); `width/height` — output raster dimensions in pixels; `srid` — EPSG code for the output CRS + +_Multi-row fixture: 3 polygon rows burned at values 1.0/2.0/3.0 over a `[0,0,4,4]` EPSG:4326 extent. All tabs use the same grouped-agg burn invocation._ + + ## Constructor Functions Create or load rasters from path, binary content, or bands (4 total). -### rst_fromfile +### rst_dtmfromgeoms - + -:::tip Loading rasters at scale? Use the [Raster Reader](../readers/raster). -`rst_fromfile` is a **convenience** for pulling columnar raster paths into a tile column inline. To ingest rasters as a normal Spark job — partitioned parallel reads, optional tiling (`sizeInMB`), and FUSE-safe Volume staging — use the **[Raster Reader](../readers/raster)** (`raster_gbx` / `gtiff_gbx`, or heavyweight `gdal` / `gtiff_gdal`): `spark.read.format("raster_gbx").load(path)`. +:::note Lightweight tier (pyrx) +Powered by **rasterio** + **SciPy** (`scipy.spatial.Delaunay`). Barycentric interpolation over an **unconstrained** Delaunay TIN; cells outside the convex hull are NoData. `breaklines`, `merge_tolerance`, and `snap_tolerance` are accepted for signature parity but not enforced — the heavyweight tier builds a constrained TIN that honors them. ::: -:::note Lightweight tier only (pyrx) — callable from Python and SQL -Powered by **rasterio**. Opens the raster at `path` and re-encodes it as a GeoTIFF tile; the `driver` arg is a format hint (rasterio auto-detects on open). A missing/unreadable path returns null. Requires `geobrix[light]`. +Create a DTM raster tile via TIN/Delaunay interpolation from an array of Z-valued point WKB geometries, with an optional array of breakline WKB geometries to preserve sharp terrain transitions. -`gbx_rst_fromfile` has **no heavyweight (JVM) implementation**. On Databricks the executor JVM cannot read a Unity Catalog Volume (`/Volumes/...`) FUSE path — the UC credential is held only by Spark's managed Python worker — so the function is registered as a Python UDF even when you call it from SQL. With `geobrix[light]` installed it is available in SQL (`SELECT gbx_rst_fromfile(...)`) and in Python (`rx.rst_fromfile(...)`); without `[light]` it is not registered, and the Python binding raises with guidance. +:::note Lightweight-only `out_crs` parameter +The lightweight **Python** binding accepts an optional trailing `out_crs` (string CRS) argument that the SQL and heavyweight/Scala tiers do not — the heavyweight `RST_DTMFromGeoms` builder is strictly ≤12-argument (`…, out_srid, [no_data]`) and rejects a 13th argument. In lightweight Python, `out_crs` (string) wins over the int `out_srid`. This is a lightweight superset, not a heavyweight regression. ::: -Load a raster from a file path. +**Signature:** `rst_dtmfromgeoms(points_array: Column, breaklines_array: Column, merge_tolerance: Column, snap_tolerance: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width_px: Column, height_px: Column, out_srid: Column, [no_data: Column = null]): Column` — output CRS via `out_srid` (int); input points assumed already in the output CRS. See [Coordinate Reference Systems](./coordinate-reference-systems#source-crs-vs-output-crs). -**Signature:** `rst_fromfile(path: Column, driver: Column): Column` +**Parameters:** `points_array` — Array of WKB point geometries with Z coordinates; `breaklines_array` — Array of WKB line/polygon geometries enforcing hard edges (pass `null` or empty array if unused); `merge_tolerance`/`snap_tolerance` — Delaunay triangulation tolerances (vertex-merge distance and snapping distance; small values such as `0.0` and `0.01` are typical); `xmin`/`ymin`/`xmax`/`ymax` — output extent in CRS units; `width_px`/`height_px` — output raster dimensions in pixels (for N-metre cells set `width_px = round((xmax-xmin)/N)`); `out_srid` — EPSG code for the output CRS. An optional trailing `no_data` argument overrides the default fill for cells outside the triangulated hull. -**Parameters:** `path` — File path; `driver` — GDAL driver name (e.g. `GTiff`) + -**Returns:** Binary raster tile data +--- -**SQL:** +### rst_frombands - + -:::tip Portable alternative — `binaryFile` + `rst_fromcontent` -If `geobrix[light]` is not installed, or you want a tier-agnostic path that works on any compute, read the bytes with Spark's built-in `binaryFile` reader and build the tile from content. This reads `/Volumes` reliably (the reader runs in Spark, which holds the credential) and works in both tiers: +:::note Lightweight tier (pyrx) +Powered by **rasterio**. Stacks an `ARRAY` of single-band tiles into one multi-band tile in array order (element 0 → band 1), preserving georeference/CRS/dtype/NoData from the first. +::: -```python -df = ( - spark.read.format("binaryFile") - .load("/Volumes/main/geobrix_samples/geobrix-examples/nyc/*.tif") - .selectExpr("path", "gbx_rst_fromcontent(content, 'GTiff') AS tile") -) -``` +Create a raster from an array of band tiles. -```sql -SELECT path, gbx_rst_fromcontent(content, 'GTiff') AS tile -FROM read_files('/Volumes/main/geobrix_samples/geobrix-examples/nyc/', format => 'binaryFile') -``` +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). ::: +**Signature:** `rst_frombands(bands: Column): Column` + +_Constructor — stacks per-band tiles into a multi-band tile. Example splits the multiband fixture into per-band tiles via `rst_band`, then re-stacks them into a 3-band tile._ + + + --- ### rst_fromcontent @@ -932,47 +1293,67 @@ Create a raster from binary content. **Returns:** Binary raster tile data -**SQL:** - - - ---- - -### rst_dtmfromgeoms +_Constructor — builds a tile from binary content. Example uses Spark's `binaryFile` reader to load bytes, then `rst_fromcontent` to decode them — this is the canonical tier-agnostic pattern (works on any compute including Serverless)._ - - -:::note Lightweight tier (pyrx) -Powered by **rasterio** + **SciPy** (`scipy.spatial.Delaunay`). Barycentric interpolation over an **unconstrained** Delaunay TIN; cells outside the convex hull are NoData. `breaklines`, `merge_tolerance`, and `snap_tolerance` are accepted for signature parity but not enforced — the heavyweight tier builds a constrained TIN that honors them. +:::caution Serverless memory — you own the bytes +`rst_fromcontent` materializes bytes that are **already in a column**. Those bytes are in executor memory before this call; GeoBrix does not guard their size. On Serverless / Spark Connect the connect-aware cap is **64 MiB** per tile — keep individual `content` values at or under that limit. For larger sources, use the virtual-tile raster readers (the default) and let the runtime page bytes lazily. See [Serverless & Memory](../serverless-and-memory#rst_fromcontent-you-own-the-bytes). ::: -Create a DTM raster tile via TIN/Delaunay interpolation from an array of Z-valued point WKB geometries, with an optional array of breakline WKB geometries to preserve sharp terrain transitions. + -**Signature:** `rst_dtmfromgeoms(points: Column, breaklines: Column, mergeTolerance: Column, snapTolerance: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, srid: Column): Column` +--- -**Parameters:** `points` — Array of WKB point geometries with Z coordinates; `breaklines` — Array of WKB line/polygon geometries enforcing hard edges (pass `null` or empty array if unused); `mergeTolerance/snapTolerance` — Delaunay triangulation tolerances (vertex-merge distance and snapping distance; small values such as `0.0` and `0.01` are typical); `xmin/ymin/xmax/ymax` — output extent in CRS units; `width/height` — output raster dimensions in pixels (for N-metre cells set `width = round((xmax-xmin)/N)`); `srid` — EPSG code for the output CRS. An optional trailing `noData` argument overrides the default fill for cells outside the triangulated hull. +### rst_fromfile -**SQL:** + - +Reference a raster on disk as a tile, by path. ---- +**The lightweight tier returns a _virtual_ tile by default** — bytes-free, pointing at `path` over its whole-file window, with `width`/`height`/CRS read from the header. No pixels are read until a downstream op needs them, so this is the lazy way to load rasters by path. There is no `virtualize_dir`/`virtualize_prefix` argument here: the source file already **is** the durable backing store, so nothing needs writing. A path that cannot be opened returns null. -### rst_frombands +**Signature:** `rst_fromfile(path: Column, driver: Column = 'GTiff'[, materialize: bool = False])` — `materialize` is a lightweight-**Python** argument; from Python, `materialize=True` reads the pixels now and returns a materialized tile. - +**Parameters:** `path` — raster file path; `driver` — GDAL driver-name hint carried into metadata (rasterio auto-detects the real format on open); `materialize` (light Python only) — `True` reads pixels now and returns a materialized tile, default `False` returns a virtual tile. -:::note Lightweight tier (pyrx) -Powered by **rasterio**. Stacks an `ARRAY` of single-band tiles into one multi-band tile in array order (element 0 → band 1), preserving georeference/CRS/dtype/NoData from the first. +:::note Tier differences — virtual vs materialized +`gbx_rst_fromfile` is a Python UDF (requires `geobrix[light]`; **no Scala/JVM form** — the executor JVM cannot read a UC Volume `/Volumes/...` FUSE path, whose credential is held only by Spark's managed Python worker). The **SQL call is the same 2-argument form in both tiers**, but the tier that registered decides the result: the **lightweight** registration (`pyrx`) returns a **virtual** tile, the **heavyweight** registration (`rasterx`) returns a **materialized** tile (JVM/heavy callers cannot use a virtual path-only tile). Whichever `register()` ran last wins. From **Python**, pass `materialize=True` to force bytes regardless of tier. Without `[light]` the function is not registered and the Python binding raises with guidance. ::: -Create a raster from an array of band tiles. +:::caution `materialize=True` errors over the Serverless cap +Calling `rst_fromfile(..., materialize=True)` on a file larger than the connect-aware stream cap (**64 MiB on Serverless / Spark Connect**, 256 MiB on classic) raises a `ValueError`. The error is intentional — silently materializing an oversized tile would OOM the executor. The virtual default (`materialize=False`, which is the lightweight default) is Serverless-safe: pixels are read lazily, size-gated, at each downstream operation. See [Serverless & Memory](../serverless-and-memory#rst_fromfile-materialize-true). +::: -**Signature:** `rst_frombands(bands: Column): Column` + -**SQL:** +:::tip Loading rasters at scale? Use the [Raster Reader](../readers/raster) +`rst_fromfile` pulls columnar raster paths into a tile column inline. To ingest rasters as a normal Spark job — partitioned parallel reads, optional tiling (`sizeInMB`), FUSE-safe Volume staging — use the **[Raster Reader](../readers/raster)** (`raster_gbx` / `gtiff_gbx`, or heavyweight `gdal` / `gtiff_gdal`): `spark.read.format("raster_gbx").load(path)`. It, too, emits virtual tiles by default. +::: - +:::note Have the bytes already, or need a materialized tile? Use `rst_fromcontent` +If you already hold raster bytes in a column (e.g. from Spark's built-in `binaryFile` reader), build the tile with [`gbx_rst_fromcontent(content, driver)`](#rst_fromcontent). Note this is **not** a lazy equivalent of `rst_fromfile` in the lightweight tier: `rst_fromcontent` takes bytes you have already read, so it always produces a **materialized** tile. (In the heavyweight tier all tiles are materialized, so there the two are interchangeable.) It also works without `geobrix[light]` and on any compute, since the `binaryFile` reader runs in Spark, which holds the Volume credential: + +```python +df = ( + spark.read.format("binaryFile") + .load("/Volumes/main/geobrix_samples/geobrix-examples/nyc/*.tif") + .selectExpr("path", "gbx_rst_fromcontent(content, 'GTiff') AS tile") +) +``` +::: --- @@ -986,13 +1367,22 @@ Powered by **rasterio** + **SciPy** (`cKDTree` IDW). Inverse-distance interpolat IDW-interpolate an array of Z-valued point geometries to a Float64 GeoTIFF tile covering an explicit bounding box and pixel grid. Supply the points and their scalar values as arrays in a single row; use `rst_gridfrompoints_agg` when points arrive one per row. -**Signature:** `rst_gridfrompoints(points: Column, values: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, widthPx: Column, heightPx: Column, srid: Column, power: Column, maxPts: Column): Column` +:::note Lightweight-only `out_crs` parameter +The lightweight **Python** binding accepts an optional trailing `out_crs` (string CRS) argument that the SQL and heavyweight/Scala tiers do not — the heavyweight `RST_GridFromPoints` builder is strictly ≤11-argument (`…, out_srid, [power, [max_pts]]`) and rejects a 12th argument. In lightweight Python, `out_crs` (string) wins over the int `out_srid`. This is a lightweight superset, not a heavyweight regression. +::: -**Parameters:** `points` — `ARRAY` of WKB point geometries; `values` — `ARRAY` of scalar observations, one per point; `xmin/ymin/xmax/ymax` — output extent in CRS units; `widthPx/heightPx` — output dimensions in pixels; `srid` — EPSG code; `power` — IDW distance-decay exponent (2.0 is the standard); `maxPts` — maximum nearest neighbours considered per output pixel +**Signature:** `rst_gridfrompoints(points_array: Column, values_array: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width_px: Column, height_px: Column, out_srid: Column, [power: Column, [max_pts: Column]]): Column` — output CRS via `out_srid` (int); input points are assumed already in the output CRS. See [Coordinate Reference Systems](./coordinate-reference-systems#source-crs-vs-output-crs). -**SQL:** +**Parameters:** `points_array` — `ARRAY` of WKB point geometries; `values_array` — `ARRAY` of scalar observations, one per point; `xmin`/`ymin`/`xmax`/`ymax` — output extent in CRS units; `width_px`/`height_px` — output dimensions in pixels; `out_srid` — EPSG code; `power` — IDW distance-decay exponent (2.0 is the standard); `max_pts` — maximum nearest neighbours considered per output pixel - + --- @@ -1000,69 +1390,95 @@ IDW-interpolate an array of Z-valued point geometries to a Float64 GeoTIFF tile Produce multiple tiles or bands (7 total). -### rst_h3_tessellate +### rst_bng_tessellate :::note Lightweight tier (pyrx) -Powered by **rasterio** + **h3**. Clips the raster to each overlapping H3 cell at `resolution`, returning one row per cell via streaming UDTF (matches the heavyweight generator behavior). +The lightweight implementation is backed by `pygx._bng` cell math and `rasterio`. It auto-warps the input to **EPSG:27700** via `rasterio.warp`, enumerates the overlapping BNG cells, and clips one tile per cell — rendering **STRING** BNG cell IDs at the row boundary and matching the heavyweight cell set and clip windows. ::: -**Signature:** `rst_h3_tessellate(tile: Column, resolution: Column, mode: Column = "covering"): Column` — Tessellate raster to H3 cells. `mode` is `"covering"` (default — every overlapping cell, clipped) or `"centroid"` (pixel-centroid single-assignment partition). See [H3 Raster Tessellation](./h3-raster-tessellation) for the full mode guide. +Tessellate a raster to British National Grid (BNG) cells — one row per overlapping cell, each clipped to that cell's extent. The raster is automatically reprojected to **EPSG:27700** (British National Grid) before tessellation, so any projected or geographic input is handled transparently. Resolution accepts integer indices **±1..±6** (1 = 100 km, 2 = 10 km, 3 = 1 km, 4 = 100 m, 5 = 10 m, 6 = 1 m; negative indices address quadrant sub-cells) or string keys from `BNG.resolutionMap` (e.g. `"1km"`, `"100m"`). Cell IDs are **STRING** (e.g. `"TQ28"`, `"SU3412"`). -Covering mode uses a **positive-area overlap** rule (shared by all three grids — `rst_h3_tessellate`, `rst_quadbin_tessellate`, `rst_bng_tessellate`): a cell is emitted **iff its geometry has greater-than-zero area overlap** with the raster. A cell that merely touches the raster along a boundary edge or corner (zero pixel overlap — common on grid-aligned tiles where the raster edges land on cell boundaries) is **excluded**. A within-extent cell whose pixels are all NoData **is** emitted — NoData renders in place, it does not punch a gap into the mosaic. This rule is identical on the lightweight and heavyweight tiers, so covering-mode cell sets match exactly across tiers. +Natural fit for CV image-tiling workflows that require Ordnance Survey–scale cells aligned to the national grid — for example tiling aerial or satellite imagery into 1 km BNG cells for object-detection model inference. See issue [#49](https://github.com/databrickslabs/geobrix/issues/49) for the background. + +**Signature:** `rst_bng_tessellate(tile: Column, resolution: Column): Column` **SQL:** - + -### rst_quadbin_tessellate +### rst_h3_tessellate :::note Lightweight tier (pyrx) -The lightweight implementation is backed by `pygx._quadbin` cell math and `rasterio`. It enumerates the overlapping quadbin cells for the raster bbox and clips one tile per cell, matching the heavyweight cell set and clip windows. +Powered by **rasterio** + **h3**. Clips the raster to each overlapping H3 cell at `resolution`, returning one row per cell via streaming UDTF (matches the heavyweight generator behavior). ::: -Tessellate a raster to CARTO quadbin v0 cells — one row per overlapping cell, each clipped to that cell's extent. The input raster must be in EPSG:4326 (lon/lat); reproject upstream with `rst_transform` if your source CRS differs. Resolution is the quadbin zoom level (0..26). Each output row carries the quadbin cell ID (`BIGINT`) and the clipped tile struct. - -**Signature:** `rst_quadbin_tessellate(tile: Column, resolution: Column): Column` +**Signature:** `rst_h3_tessellate(tile: Column, resolution: Column, mode: Column = "covering"): Column` — Tessellate raster to H3 cells. `mode` is `"covering"` (default — every overlapping cell, clipped) or `"centroid"` (pixel-centroid single-assignment partition). See [H3 Raster Tessellation](./h3-raster-tessellation) for the full mode guide. -**SQL:** +Covering mode uses a **positive-area overlap** rule (shared by all three grids — `rst_h3_tessellate`, `rst_quadbin_tessellate`, `rst_bng_tessellate`): a cell is emitted **iff its geometry has greater-than-zero area overlap** with the raster. A cell that merely touches the raster along a boundary edge or corner (zero pixel overlap — common on grid-aligned tiles where the raster edges land on cell boundaries) is **excluded**. A within-extent cell whose pixels are all NoData **is** emitted — NoData renders in place, it does not punch a gap into the mosaic. This rule is identical on the lightweight and heavyweight tiers, so covering-mode cell sets match exactly across tiers. - + -### rst_bng_tessellate +### rst_maketiles :::note Lightweight tier (pyrx) -The lightweight implementation is backed by `pygx._bng` cell math and `rasterio`. It auto-warps the input to **EPSG:27700** via `rasterio.warp`, enumerates the overlapping BNG cells, and clips one tile per cell — rendering **STRING** BNG cell IDs at the row boundary and matching the heavyweight cell set and clip windows. +Powered by **rasterio**. Streams one tile row per subdivided region via streaming UDTF. It derives a square tile size from the MB budget and always partitions; it does not honor the heavyweight `size_in_mb = -1` (single tile) or `0` (64 MB) sentinels or the power-of-four split, so tile counts and dimensions differ. ::: -Tessellate a raster to British National Grid (BNG) cells — one row per overlapping cell, each clipped to that cell's extent. The raster is automatically reprojected to **EPSG:27700** (British National Grid) before tessellation, so any projected or geographic input is handled transparently. Resolution accepts integer indices **±1..±6** (1 = 100 km, 2 = 10 km, 3 = 1 km, 4 = 100 m, 5 = 10 m, 6 = 1 m; negative indices address quadrant sub-cells) or string keys from `BNG.resolutionMap` (e.g. `"1km"`, `"100m"`). Cell IDs are **STRING** (e.g. `"TQ28"`, `"SU3412"`). - -Natural fit for CV image-tiling workflows that require Ordnance Survey–scale cells aligned to the national grid — for example tiling aerial or satellite imagery into 1 km BNG cells for object-detection model inference. See issue [#49](https://github.com/databrickslabs/geobrix/issues/49) for the background. - -**Signature:** `rst_bng_tessellate(tile: Column, resolution: Column): Column` +**Signature:** `rst_maketiles(tile: Column, sizeInMB: Column): Column` — Subdivide into smaller tiles by approximate size in MB. **SQL:** - + -### rst_maketiles +### rst_quadbin_tessellate :::note Lightweight tier (pyrx) -Powered by **rasterio**. Streams one tile row per subdivided region via streaming UDTF. It derives a square tile size from the MB budget and always partitions; it does not honor the heavyweight `size_in_mb = -1` (single tile) or `0` (64 MB) sentinels or the power-of-four split, so tile counts and dimensions differ. +The lightweight implementation is backed by `pygx._quadbin` cell math and `rasterio`. It enumerates the overlapping quadbin cells for the raster bbox and clips one tile per cell, matching the heavyweight cell set and clip windows. ::: -**Signature:** `rst_maketiles(tile: Column, tileWidth: Column, tileHeight: Column): Column` — Subdivide into smaller tiles. +Tessellate a raster to CARTO quadbin v0 cells — one row per overlapping cell, each clipped to that cell's extent. The input raster must be in EPSG:4326 (lon/lat); reproject upstream with `rst_transform` if your source CRS differs. Resolution is the quadbin zoom level (0..26). Each output row carries the quadbin cell ID (`BIGINT`) and the clipped tile struct. + +**Signature:** `rst_quadbin_tessellate(tile: Column, resolution: Column): Column` **SQL:** - + ### rst_retile @@ -1076,7 +1492,14 @@ Powered by **rasterio**. Streams one tile row per retiled region via streaming U **SQL:** - + ### rst_separatebands @@ -1090,7 +1513,14 @@ Powered by **rasterio**. Streams one band-tile row per band via streaming UDTF **SQL:** - + ### rst_tooverlappingtiles @@ -1104,7 +1534,14 @@ Powered by **rasterio**. Streams one tile row per overlapping region via streami **SQL:** - + --- @@ -1112,35 +1549,83 @@ Powered by **rasterio**. Streams one tile row per overlapping region via streami Aggregate raster values to H3 grid cells, and utility functions for H3-based canvas setup (9 total). -### rst_h3_rastertogridavg +### gbx_h3_cell_bbox - + :::note Lightweight tier (pyrx) -Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. +Powered by **h3**. Returns a `STRUCT` bounding box for the given H3 cell in the requested `srid`. In `'centroids'` mode the box tightly wraps the centroid point; in `'spatial_envelope'` mode it wraps the full hexagon outline. When `kring_pad > 0` the k-ring of that radius is computed first and the bounding box covers all cells in the ring. The lightweight SQL function requires all four arguments explicitly; the Python API (`rx.h3_cell_bbox(cellid, srid, mode, kring_pad)`) honors the same defaults as the Scala implementation. ::: -**Signature:** `rst_h3_rastertogridavg(tile: Column, resolution: Column): Column` +Scalar function — returns the bounding box `STRUCT` for one H3 cell in the given CRS. Use this to drive the `xmin/ymin/xmax/ymax` and grid-size parameters of [`rst_h3_rasterize_agg`](#rst_h3_rasterize_agg) when you need a consistent per-cell canvas, or to clip and inspect cell extents in downstream queries. + +**Signature:** `h3_cell_bbox(cellid: Column, srid: Column, mode: Column, kring_pad: Column): Column` + +**Parameters:** +- `cellid` — H3 cell ID (BIGINT or STRING) +- `srid` — EPSG code; `4326` for WGS 84 lon/lat +- `mode` — `'centroids'` (centroid point envelope) or `'spatial_envelope'` (hexagon boundary envelope) +- `kring_pad` — expand by this many k-rings before computing the bounding box; `0` = no expansion + +**Returns:** `STRUCT` **SQL:** - + -### rst_h3_rastertogridcount +### rst_h3_gridspec (Python / DataFrame helper) - +:::note Lightweight (pyrx) Python helper only +`rst_h3_gridspec` is **not registered as a SQL function** and is **not available in the heavyweight tier**. It is a pure-Python / PySpark DataFrame helper in the `pyrx` package. -:::note Lightweight tier (pyrx) -Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the per-cell pixel count (integer). The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. +For the heavyweight tier, compose the equivalent shared canvas using the registered scalar `gbx_h3_cell_bbox(cellid, srid, mode, kring_pad)` with native Spark `min`/`max` aggregates and the same floor/ceil snap arithmetic: +```sql +SELECT min(cell_bbox.xmin), min(cell_bbox.ymin), + max(cell_bbox.xmax), max(cell_bbox.ymax) +FROM (SELECT gbx_h3_cell_bbox(cellid, 4326, 'centroids', 1) AS cell_bbox FROM cells) +``` ::: -**Signature:** `rst_h3_rastertogridcount(tile: Column, resolution: Column): Column` — Pixel count per H3 cell. +`rst_h3_gridspec` computes the shared, snapped canvas (extent + pixel size) that a group of H3 cells should use so that per-threshold rasters align on a common grid and can be stacked via `rst_frombands_agg` or mosaicked via `rst_merge_agg`. -**SQL:** +**Signature:** +```python +rx.rst_h3_gridspec(df, cell_col="cellid", *group_cols, + srid=4326, pixel_size=None, + mode="centroids", kring_pad=1) +``` - +**Typical multi-threshold workflow:** -### rst_h3_rastertogridmax +1. Call `rx.rst_h3_gridspec(df, cell_col="cellid", srid=4326, mode='centroids', kring_pad=1)` once on the distinct cell set. It returns the grouped DataFrame with a `grid` struct column — one row per group containing the shared canvas. +2. For each threshold band, run `rst_h3_rasterize_agg` with those fixed bounds — all output tiles share the same origin and pixel grid. +3. Stack aligned bands with `rst_frombands_agg` (ordered by `band_index`), or mosaic per-cell tiles with `rst_merge_agg`. + +**Parameters:** +- `df` — input Spark DataFrame containing H3 cell IDs +- `cell_col` — column name holding H3 cell IDs (integer or string); default `"cellid"` +- `*group_cols` — additional grouping columns (e.g. a transmitter ID, year, month); one grid spec row is produced per group +- `srid` — EPSG code for the output CRS; `4326` for WGS 84 (default) +- `pixel_size` — ground resolution in CRS units; `None` = auto-derived from the H3 resolution via an edge-length heuristic (default) +- `mode` — `'centroids'` (default) or `'spatial_envelope'` +- `kring_pad` — k-ring expansion applied per cell before computing its bounding box (default `1`) + +**Returns:** the grouped DataFrame with a `grid` column of type: +``` +STRUCT +``` + +--- + +### rst_h3_rastertogridavg @@ -1148,27 +1633,49 @@ Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of ` Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: -**Signature:** `rst_h3_rastertogridmax(tile: Column, resolution: Column): Column` — Max value per H3 cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_h3_rastertogridavg` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_h3_rastertogridavg(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridavg(tile, 4) t`. Both forms appear in the SQL tab below. +::: + +**Signature:** `rst_h3_rastertogridavg(tile: Column, resolution: Column): Column` **SQL:** - + -### rst_h3_rastertogridmin +### rst_h3_rastertogridcount :::note Lightweight tier (pyrx) -Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. +Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the per-cell pixel count (integer). The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: -**Signature:** `rst_h3_rastertogridmin(tile: Column, resolution: Column): Column` — Min value per H3 cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_h3_rastertogridcount` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_h3_rastertogridcount(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridcount(tile, 4) t`. Both forms appear in the SQL tab below. +::: + +**Signature:** `rst_h3_rastertogridcount(tile: Column, resolution: Column): Column` — Pixel count per H3 cell. **SQL:** - + -### rst_h3_rastertogridmedian +### rst_h3_rastertogridmax @@ -1176,39 +1683,72 @@ Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of ` Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: -**Signature:** `rst_h3_rastertogridmedian(tile: Column, resolution: Column): Column` — Median value per H3 cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_h3_rastertogridmax` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_h3_rastertogridmax(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridmax(tile, 4) t`. Both forms appear in the SQL tab below. +::: + +**Signature:** `rst_h3_rastertogridmax(tile: Column, resolution: Column): Column` — Max value per H3 cell. **SQL:** - + -### rst_h3_rastertogridsum +### rst_h3_rastertogridmedian :::note Lightweight tier (pyrx) -Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the total of the valid pixel values in each cell. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. +Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: -**Signature:** `rst_h3_rastertogridsum(tile: Column, resolution: Column): Column` — Sum of pixel values per H3 cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_h3_rastertogridmedian` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_h3_rastertogridmedian(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridmedian(tile, 4) t`. Both forms appear in the SQL tab below. +::: + +**Signature:** `rst_h3_rastertogridmedian(tile: Column, resolution: Column): Column` — Median value per H3 cell. **SQL:** - + -### rst_h3_rastertogridvariance +### rst_h3_rastertogridmin :::note Lightweight tier (pyrx) -Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the population variance (`÷ n`, two-pass) of the valid pixel values in each cell — a single-pixel cell yields `0.0`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. +Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: -**Signature:** `rst_h3_rastertogridvariance(tile: Column, resolution: Column): Column` — Population variance of pixel values per H3 cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_h3_rastertogridmin` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_h3_rastertogridmin(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridmin(tile, 4) t`. Both forms appear in the SQL tab below. +::: + +**Signature:** `rst_h3_rastertogridmin(tile: Column, resolution: Column): Column` — Min value per H3 cell. **SQL:** - + ### rst_h3_rastertogridstddev @@ -1218,80 +1758,72 @@ Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of ` Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the population standard deviation (`sqrt` of the population variance) of the valid pixel values in each cell — a single-pixel cell yields `0.0`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_h3_rastertogridstddev` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_h3_rastertogridstddev(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridstddev(tile, 4) t`. Both forms appear in the SQL tab below. +::: + **Signature:** `rst_h3_rastertogridstddev(tile: Column, resolution: Column): Column` — Population standard deviation of pixel values per H3 cell. **SQL:** - + -### gbx_h3_cell_bbox +### rst_h3_rastertogridsum - + :::note Lightweight tier (pyrx) -Powered by **h3**. Returns a `STRUCT` bounding box for the given H3 cell in the requested `srid`. In `'centroids'` mode the box tightly wraps the centroid point; in `'spatial_envelope'` mode it wraps the full hexagon outline. When `kring_pad > 0` the k-ring of that radius is computed first and the bounding box covers all cells in the ring. The lightweight SQL function requires all four arguments explicitly; the Python API (`rx.h3_cell_bbox(cellid, srid, mode, kring_pad)`) honors the same defaults as the Scala implementation. +Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the total of the valid pixel values in each cell. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: -Scalar function — returns the bounding box `STRUCT` for one H3 cell in the given CRS. Use this to drive the `xmin/ymin/xmax/ymax` and grid-size parameters of [`rst_h3_rasterize_agg`](#rst_h3_rasterize_agg) when you need a consistent per-cell canvas, or to clip and inspect cell extents in downstream queries. - -**Signature:** `h3_cell_bbox(cellid: Column, srid: Column, mode: Column, kring_pad: Column): Column` - -**Parameters:** -- `cellid` — H3 cell ID (BIGINT or STRING) -- `srid` — EPSG code; `4326` for WGS 84 lon/lat -- `mode` — `'centroids'` (centroid point envelope) or `'spatial_envelope'` (hexagon boundary envelope) -- `kring_pad` — expand by this many k-rings before computing the bounding box; `0` = no expansion +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_h3_rastertogridsum` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_h3_rastertogridsum(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridsum(tile, 4) t`. Both forms appear in the SQL tab below. +::: -**Returns:** `STRUCT` +**Signature:** `rst_h3_rastertogridsum(tile: Column, resolution: Column): Column` — Sum of pixel values per H3 cell. **SQL:** - - -### rst_h3_gridspec (Python / DataFrame helper) - -:::note Lightweight (pyrx) Python helper only -`rst_h3_gridspec` is **not registered as a SQL function** and is **not available in the heavyweight tier**. It is a pure-Python / PySpark DataFrame helper in the `pyrx` package. - -For the heavyweight tier, compose the equivalent shared canvas using the registered scalar `gbx_h3_cell_bbox(cellid, srid, mode, kring_pad)` with native Spark `min`/`max` aggregates and the same floor/ceil snap arithmetic: -```sql -SELECT min(cell_bbox.xmin), min(cell_bbox.ymin), - max(cell_bbox.xmax), max(cell_bbox.ymax) -FROM (SELECT gbx_h3_cell_bbox(cellid, 4326, 'centroids', 1) AS cell_bbox FROM cells) -``` -::: - -`rst_h3_gridspec` computes the shared, snapped canvas (extent + pixel size) that a group of H3 cells should use so that per-threshold rasters align on a common grid and can be stacked via `rst_frombands_agg` or mosaicked via `rst_merge_agg`. - -**Signature:** -```python -rx.rst_h3_gridspec(df, cell_col="cellid", *group_cols, - srid=4326, pixel_size=None, - mode="centroids", kring_pad=1) -``` + -**Typical multi-threshold workflow:** +### rst_h3_rastertogridvariance -1. Call `rx.rst_h3_gridspec(df, cell_col="cellid", srid=4326, mode='centroids', kring_pad=1)` once on the distinct cell set. It returns the grouped DataFrame with a `grid` struct column — one row per group containing the shared canvas. -2. For each threshold band, run `rst_h3_rasterize_agg` with those fixed bounds — all output tiles share the same origin and pixel grid. -3. Stack aligned bands with `rst_frombands_agg` (ordered by `band_index`), or mosaic per-cell tiles with `rst_merge_agg`. + -**Parameters:** -- `df` — input Spark DataFrame containing H3 cell IDs -- `cell_col` — column name holding H3 cell IDs (integer or string); default `"cellid"` -- `*group_cols` — additional grouping columns (e.g. a transmitter ID, year, month); one grid spec row is produced per group -- `srid` — EPSG code for the output CRS; `4326` for WGS 84 (default) -- `pixel_size` — ground resolution in CRS units; `None` = auto-derived from the H3 resolution via an edge-length heuristic (default) -- `mode` — `'centroids'` (default) or `'spatial_envelope'` -- `kring_pad` — k-ring expansion applied per cell before computing its bounding box (default `1`) +:::note Lightweight tier (pyrx) +Powered by **rasterio** + **h3**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the population variance (`÷ n`, two-pass) of the valid pixel values in each cell — a single-pixel cell yields `0.0`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. +::: -**Returns:** the grouped DataFrame with a `grid` column of type: -``` -STRUCT -``` +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_h3_rastertogridvariance` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_h3_rastertogridvariance(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridvariance(tile, 4) t`. Both forms appear in the SQL tab below. +::: ---- +**Signature:** `rst_h3_rastertogridvariance(tile: Column, resolution: Column): Column` — Population variance of pixel values per H3 cell. + +**SQL:** + + ## Grid Functions (quadbin) @@ -1305,11 +1837,22 @@ Aggregate raster values to CARTO quadbin v0 grid cells (8 reducers). Each reduce Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) of `ARRAY`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_quadbin_rastertogridavg` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_quadbin_rastertogridavg(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridavg(tile, 4) t`. Both forms appear in the SQL tab below. +::: + **Signature:** `rst_quadbin_rastertogridavg(tile: Column, resolution: Column): Column` — Mean pixel value per quadbin cell. **SQL:** - + ### rst_quadbin_rastertogridcount @@ -1319,11 +1862,22 @@ Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the per-cell pixel count (integer). The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_quadbin_rastertogridcount` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_quadbin_rastertogridcount(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridcount(tile, 4) t`. Both forms appear in the SQL tab below. +::: + **Signature:** `rst_quadbin_rastertogridcount(tile: Column, resolution: Column): Column` — Pixel count per quadbin cell. **SQL:** - + ### rst_quadbin_rastertogridmax @@ -1333,13 +1887,24 @@ Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) of `ARRAY`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_quadbin_rastertogridmax` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_quadbin_rastertogridmax(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridmax(tile, 4) t`. Both forms appear in the SQL tab below. +::: + **Signature:** `rst_quadbin_rastertogridmax(tile: Column, resolution: Column): Column` — Max pixel value per quadbin cell. **SQL:** - + -### rst_quadbin_rastertogridmin +### rst_quadbin_rastertogridmedian @@ -1347,13 +1912,24 @@ Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) of `ARRAY`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: -**Signature:** `rst_quadbin_rastertogridmin(tile: Column, resolution: Column): Column` — Min pixel value per quadbin cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_quadbin_rastertogridmedian` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_quadbin_rastertogridmedian(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridmedian(tile, 4) t`. Both forms appear in the SQL tab below. +::: + +**Signature:** `rst_quadbin_rastertogridmedian(tile: Column, resolution: Column): Column` — Median pixel value per quadbin cell. **SQL:** - + -### rst_quadbin_rastertogridmedian +### rst_quadbin_rastertogridmin @@ -1361,55 +1937,99 @@ Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) of `ARRAY`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: -**Signature:** `rst_quadbin_rastertogridmedian(tile: Column, resolution: Column): Column` — Median pixel value per quadbin cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_quadbin_rastertogridmin` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_quadbin_rastertogridmin(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridmin(tile, 4) t`. Both forms appear in the SQL tab below. +::: + +**Signature:** `rst_quadbin_rastertogridmin(tile: Column, resolution: Column): Column` — Min pixel value per quadbin cell. **SQL:** - + -### rst_quadbin_rastertogridsum +### rst_quadbin_rastertogridstddev :::note Lightweight tier (pyrx) -Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the total of the valid pixel values in each cell. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. +Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the population standard deviation (`sqrt` of the population variance) of the valid pixel values in each cell — a single-pixel cell yields `0.0`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: -**Signature:** `rst_quadbin_rastertogridsum(tile: Column, resolution: Column): Column` — Sum of pixel values per quadbin cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_quadbin_rastertogridstddev` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_quadbin_rastertogridstddev(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridstddev(tile, 4) t`. Both forms appear in the SQL tab below. +::: + +**Signature:** `rst_quadbin_rastertogridstddev(tile: Column, resolution: Column): Column` — Population standard deviation of pixel values per quadbin cell. **SQL:** - + -### rst_quadbin_rastertogridvariance +--- + +### rst_quadbin_rastertogridsum :::note Lightweight tier (pyrx) -Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the population variance (`÷ n`, two-pass) of the valid pixel values in each cell — a single-pixel cell yields `0.0`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. +Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the total of the valid pixel values in each cell. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: -**Signature:** `rst_quadbin_rastertogridvariance(tile: Column, resolution: Column): Column` — Population variance of pixel values per quadbin cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_quadbin_rastertogridsum` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_quadbin_rastertogridsum(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridsum(tile, 4) t`. Both forms appear in the SQL tab below. +::: + +**Signature:** `rst_quadbin_rastertogridsum(tile: Column, resolution: Column): Column` — Sum of pixel values per quadbin cell. **SQL:** - + -### rst_quadbin_rastertogridstddev +### rst_quadbin_rastertogridvariance :::note Lightweight tier (pyrx) -Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the population standard deviation (`sqrt` of the population variance) of the valid pixel values in each cell — a single-pixel cell yields `0.0`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. +Powered by **rasterio** + **quadbin**. Returns an `ARRAY` (one element per band) of `ARRAY`, where `measure` is the population variance (`÷ n`, two-pass) of the valid pixel values in each cell — a single-pixel cell yields `0.0`. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with `rst_transform` if your source CRS differs. ::: -**Signature:** `rst_quadbin_rastertogridstddev(tile: Column, resolution: Column): Column` — Population standard deviation of pixel values per quadbin cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_quadbin_rastertogridvariance` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_quadbin_rastertogridvariance(tile, 4) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridvariance(tile, 4) t`. Both forms appear in the SQL tab below. +::: -**SQL:** +**Signature:** `rst_quadbin_rastertogridvariance(tile: Column, resolution: Column): Column` — Population variance of pixel values per quadbin cell. - +**SQL:** ---- + ## Grid Functions (BNG) {#bng-grid} @@ -1429,11 +2049,22 @@ A cell that covers only NoData pixels returns **`NULL`** for the measure (not `0 Mean pixel value per BNG cell. The raster is reprojected to EPSG:27700 internally before sampling. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_bng_rastertogridavg` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_bng_rastertogridavg(tile, 3) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridavg(tile, 3) t`. Both forms appear in the SQL tab below. +::: + **Signature:** `rst_bng_rastertogridavg(tile: Column, resolution: Column): Column` — Mean pixel value per BNG cell. **SQL:** - + ### rst_bng_rastertogridcount @@ -1441,11 +2072,22 @@ Mean pixel value per BNG cell. The raster is reprojected to EPSG:27700 internall Valid (non-NoData) pixel count per BNG cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_bng_rastertogridcount` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_bng_rastertogridcount(tile, 3) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridcount(tile, 3) t`. Both forms appear in the SQL tab below. +::: + **Signature:** `rst_bng_rastertogridcount(tile: Column, resolution: Column): Column` — Pixel count per BNG cell. **SQL:** - + ### rst_bng_rastertogridmax @@ -1453,11 +2095,45 @@ Valid (non-NoData) pixel count per BNG cell. Maximum pixel value per BNG cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_bng_rastertogridmax` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_bng_rastertogridmax(tile, 3) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridmax(tile, 3) t`. Both forms appear in the SQL tab below. +::: + **Signature:** `rst_bng_rastertogridmax(tile: Column, resolution: Column): Column` — Max pixel value per BNG cell. **SQL:** - + + +### rst_bng_rastertogridmedian + + + +Median pixel value per BNG cell. + +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_bng_rastertogridmedian` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_bng_rastertogridmedian(tile, 3) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridmedian(tile, 3) t`. Both forms appear in the SQL tab below. +::: + +**Signature:** `rst_bng_rastertogridmedian(tile: Column, resolution: Column): Column` — Median pixel value per BNG cell. + +**SQL:** + + ### rst_bng_rastertogridmin @@ -1465,23 +2141,47 @@ Maximum pixel value per BNG cell. Minimum pixel value per BNG cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_bng_rastertogridmin` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_bng_rastertogridmin(tile, 3) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridmin(tile, 3) t`. Both forms appear in the SQL tab below. +::: + **Signature:** `rst_bng_rastertogridmin(tile: Column, resolution: Column): Column` — Min pixel value per BNG cell. **SQL:** - + -### rst_bng_rastertogridmedian +### rst_bng_rastertogridstddev -Median pixel value per BNG cell. +Population standard deviation (`sqrt` of the population variance) of pixel values per BNG cell — a single-pixel cell yields `0.0`. -**Signature:** `rst_bng_rastertogridmedian(tile: Column, resolution: Column): Column` — Median pixel value per BNG cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_bng_rastertogridstddev` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_bng_rastertogridstddev(tile, 3) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridstddev(tile, 3) t`. Both forms appear in the SQL tab below. +::: + +**Signature:** `rst_bng_rastertogridstddev(tile: Column, resolution: Column): Column` — Population standard deviation of pixel values per BNG cell. **SQL:** - + + +--- ### rst_bng_rastertogridsum @@ -1489,11 +2189,22 @@ Median pixel value per BNG cell. Sum of pixel values per BNG cell. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_bng_rastertogridsum` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_bng_rastertogridsum(tile, 3) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridsum(tile, 3) t`. Both forms appear in the SQL tab below. +::: + **Signature:** `rst_bng_rastertogridsum(tile: Column, resolution: Column): Column` — Sum of pixel values per BNG cell. **SQL:** - + ### rst_bng_rastertogridvariance @@ -1501,25 +2212,22 @@ Sum of pixel values per BNG cell. Population variance (`÷ n`, two-pass) of pixel values per BNG cell — a single-pixel cell yields `0.0`. -**Signature:** `rst_bng_rastertogridvariance(tile: Column, resolution: Column): Column` — Population variance of pixel values per BNG cell. - -**SQL:** - - - -### rst_bng_rastertogridstddev - - - -Population standard deviation (`sqrt` of the population variance) of pixel values per BNG cell — a single-pixel cell yields `0.0`. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_bng_rastertogridvariance` as a scalar (ARRAY-returning) function — call it directly and `explode(...)` to flatten to rows: `SELECT gbx_rst_bng_rastertogridvariance(tile, 3) AS grid FROM multiband_rasters`. The lightweight (pyrx) tier registers it as a streaming **table function**, so lightweight SQL must use `LATERAL`: `SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridvariance(tile, 3) t`. Both forms appear in the SQL tab below. +::: -**Signature:** `rst_bng_rastertogridstddev(tile: Column, resolution: Column): Column` — Population standard deviation of pixel values per BNG cell. +**Signature:** `rst_bng_rastertogridvariance(tile: Column, resolution: Column): Column` — Population variance of pixel values per BNG cell. **SQL:** - - ---- + ## Operations @@ -1533,11 +2241,20 @@ Transform and analyze rasters (20 total). Powered by **rasterio**. Output formats are limited to rasterio's bundled-GDAL writable driver set; the tile is re-encoded in the requested format. ::: -**Signature:** `rst_asformat(tile: Column, newFormat: Column): Column` — Convert to another format. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_asformat(tile: Column, newFormat: Column): Column` — Convert to another format. - + ### rst_clip @@ -1547,15 +2264,26 @@ Powered by **rasterio**. Output formats are limited to rasterio's bundled-GDAL w Powered by **rasterio** (`rasterio.mask`). The clip geometry is assumed to be in the raster's CRS; the heavyweight tier has additional SRID-inheritance fallbacks. ::: -**Signature:** `rst_clip(tile: Column, clip: Column, cutlineAllTouched: Column): Column` — Clip by geometry. The `clip` argument must be **WKT** (string), **EWKT** (SRID-prefixed string), **WKB** (binary), or **EWKB** (SRID-embedded binary); do not use `st_geomfromtext()` or other DBR native geometry. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: + +**Signature:** `rst_clip(tile: Column, clip: Column, cutlineAllTouched: Column, [clip_crs: Column = null]): Column` — Clip by geometry. The `clip` argument must be **WKT** (string), **EWKT** (SRID-prefixed string), **WKB** (binary), or **EWKB** (SRID-embedded binary); do not use `st_geomfromtext()` or other DBR native geometry. Optional `clip_crs` (string, source role) declares the CRS of a plain WKB/WKT cutline — an EWKB/EWKT embedded SRID wins, absent → assumed already in the raster CRS. See [Coordinate Reference Systems](./coordinate-reference-systems#source-crs-vs-output-crs). **CRS handling:** - **EWKT** (`SRID=4326;POLYGON(...)`) or **EWKB** (SRID encoded in the byte header) — the SRID is read, and if it differs from the raster's CRS the cutline is reprojected before clipping. Use this form whenever the geometry and raster may be in different CRSs. - **Plain WKT / WKB** (no SRID) — the geometry is assumed to already be in the raster's CRS. If that assumption is wrong (for example, lon/lat polygons against a UTM raster), the cutline will land outside the raster and you'll get an empty or blank output. Either switch to EWKT/EWKB, or reproject the geometry to the raster's CRS first. -**SQL:** +_Example clip geometry: a WKT polygon in the raster's native CRS (EPSG:32618, no SRID prefix). To clip with a WGS84 geometry use EWKT: `SRID=4326;POLYGON(...)` — the embedded SRID triggers auto-reprojection to the raster's CRS._ - + ### rst_combineavg @@ -1565,11 +2293,20 @@ Powered by **rasterio** (`rasterio.mask`). The clip geometry is assumed to be in Powered by **rasterio** + **NumPy**. Takes an `ARRAY` and returns the NoData-aware per-pixel mean; input tiles must share the same grid (shape/extent/CRS). cellid is preserved when all inputs share one, else −1. ::: -**Signature:** `rst_combineavg(tiles: Column): Column` — Average multiple tiles (e.g. temporal composite). +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_combineavg(tiles: Column): Column` — Average multiple tiles (e.g. temporal composite). - + ### rst_convolve @@ -1579,11 +2316,22 @@ Powered by **rasterio** + **NumPy**. Takes an `ARRAY` and returns the NoDa Powered by **SciPy** (`scipy.ndimage`). Output is Float64 and border pixels are filled by edge-replication; the heavyweight tier preserves the input dtype and leaves border pixels unchanged. ::: +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: + **Signature:** `rst_convolve(tile: Column, kernel: Column): Column` — Apply convolution kernel. -**SQL:** +_Example kernel: 3×3 identity (`[[0,0,0],[0,1,0],[0,0,0]]`), which passes pixels through unchanged._ - + ### rst_derivedband @@ -1593,11 +2341,20 @@ Powered by **SciPy** (`scipy.ndimage`). Output is Float64 and border pixels are Powered by **rasterio** with GDAL VRT Python pixel functions. A pixel function authored for one tier runs unchanged in the other. ::: -**Signature:** `rst_derivedband(tile: Column, pyfunc: String, funcName: String): Column` — Apply Python UDF to derive band. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_derivedband(tile: Column, pyfunc: String, funcName: String): Column` — Apply Python UDF to derive band. - + ### rst_filter @@ -1607,11 +2364,20 @@ Powered by **rasterio** with GDAL VRT Python pixel functions. A pixel function a Powered by **SciPy** (`scipy.ndimage`). The averaging filter is named `'mean'` (not `'avg'`) and `'mode'` is unavailable; the averaging output is Float32 and near-edge values may differ slightly from the heavyweight tier. ::: -**Signature:** `rst_filter(tile: Column, kernelSize: Column, operation: Column): Column` — Spatial filter (e.g. median, avg). +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_filter(tile: Column, kernelSize: Column, operation: Column): Column` — Spatial filter (e.g. median, avg). - + ### rst_initnodata @@ -1621,11 +2387,20 @@ Powered by **SciPy** (`scipy.ndimage`). The averaging filter is named `'mean'` ( Powered by **rasterio**. When no NoData is set it assigns -9999.0; the heavyweight tier assigns a data-type-appropriate sentinel per band, so the NoData value can differ for integer or byte rasters. ::: -**Signature:** `rst_initnodata(tile: Column): Column` — Initialize NoData values. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_initnodata(tile: Column): Column` — Initialize NoData values. - + ### rst_isempty @@ -1639,23 +2414,61 @@ Powered by **rasterio**. Returns `true` when the raster has no size **or** every band is entirely NoData. -**SQL:** +_Examples use the multiband fixture (`rgb_nir_small.tif`, 3 bands with real pixel data); the canonical single-band sentinel2 tile has NoData=0 with all pixels equal zero, returning `true`._ - + ### rst_mapalgebra :::note Lightweight tier (pyrx) -Powered by **NumExpr**. Bands map to `A`, `B`, `C`, … and the expression is evaluated with NumExpr (no `gdal_calc` NumPy builtins); single-band Float32 output. +Powered by **NumExpr**. Bands map to `A`, `B`, `C`, … and the `calc` expression is evaluated with NumExpr (no `gdal_calc` NumPy builtins); single-band Float32 output. ::: -**Signature:** `rst_mapalgebra(tiles: Column, expression: Column): Column` — Map algebra expression (e.g. A-B). +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_mapalgebra(tiles: Column, expression: Column): Column` — Map algebra expression (e.g. `A - B`). + +**How the inputs bind.** `tiles` is an `ARRAY`. By default, band 1 of each tile, **in array order**, binds to the variables `A`, `B`, `C`, … (`A` = first tile, `B` = second, …). The result is a single-band Float32 tile on the first input's georeference. + +**Spec format (same on both tiers).** The expression is accepted in **either** of two forms — identical across SQL, Scala, and both Python tiers: + +- **JSON envelope** (recommended; the `gdal_calc` spec shape): a JSON object with a `calc` expression — `'{"calc": "A * 2"}'`. +- **Bare expression string**: the calc expression on its own — `"A * 2"`. + + +**Selecting a specific band or raster per variable (`A_index` / `A_band`).** You do **not** need to decompose a multiband raster to do band math. The JSON envelope's per-variable keys — `A_index` / `A_band` (and `B_`, `C_`, …) — map each variable to a chosen **raster** (0-based, into the `tiles` array) and a chosen **1-based band**, mirroring `gdal_calc`. For example, **NDVI from bands 4 (NIR) and 3 (Red) of a single raster**: + +```json +{"calc": "(A - B) / (A + B)", "A_index": 0, "B_index": 0, "A_band": 4, "B_band": 3} +``` + +Both `A` and `B` read raster `0` (the one tile in the array), with `A` = band 4 and `B` = band 3 — the direct equivalent of `gdal_calc -A in.tif --A_band=4 -B in.tif --B_band=3 --calc="(A - B) / (A + B)"`. A variable with no `*_index`/`*_band` keeps the default (its ordinal raster, band 1). **This works on both tiers.** + +The only `gdal_calc` envelope key the lightweight tier does **not** support is `extra_options` (raw CLI flags with no NumExpr equivalent); it raises a clear error rather than dropping it silently. The heavy tier honors `extra_options` as well. + +:::caution Expression language differs by engine +The **envelope** is portable, but the **calc expression language is not fully portable** because each tier uses a different evaluator: the heavy tier shells out to GDAL's `gdal_calc` (NumPy expression syntax); the lightweight tier evaluates with [NumExpr](https://numexpr.readthedocs.io/). They agree on ordinary arithmetic and comparisons (`A * 2`, `(A - B) / (A + B)`, `A > 0`), so those expressions run **unchanged on both tiers**. They diverge on function spellings — e.g. `gdal_calc` accepts NumPy calls like `numpy.where(A > 0, A, 0)`, whereas NumExpr wants `where(A > 0, A, 0)`. Keep expressions to basic arithmetic for cross-tier portability. For the heavy tier's full syntax and options, see the GDAL raster calculator reference: [gdal_raster_calc](https://gdal.org/en/stable/programs/gdal_raster_calc.html#gdal-raster-calc). +::: - + ### rst_merge @@ -1665,11 +2478,20 @@ Powered by **NumExpr**. Bands map to `A`, `B`, `C`, … and the expression is ev Powered by **rasterio** (`rasterio.merge`). Takes an `ARRAY` (in one row) and mosaics them into a single tile spanning the union extent (first-tile-wins on overlap, in array order). ::: -**Signature:** `rst_merge(tiles: Column): Column` — Merge tiles into mosaic. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_merge(tiles: Column): Column` — Merge tiles into mosaic. - + ### rst_ndvi @@ -1679,11 +2501,20 @@ Powered by **rasterio** (`rasterio.merge`). Takes an `ARRAY` (in one row) Powered by **rasterio** + **NumPy**. Valid pixels match the heavyweight tier; pixels with a zero denominator are set to NoData (-9999) rather than left as non-finite values. ::: -**Signature:** `rst_ndvi(tile: Column, redBand: Column, nirBand: Column): Column` — NDVI from band indices. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_ndvi(tile: Column, redBand: Column, nirBand: Column): Column` — NDVI from band indices. - + ### rst_rastertoworldcoord @@ -1695,9 +2526,14 @@ Powered by **rasterio**. **Signature:** `rst_rastertoworldcoord(tile: Column, pixelX: Column, pixelY: Column): Column` — Pixel to world coordinates as a struct with `.x` and `.y` fields. -**SQL:** - - + ### rst_rastertoworldcoordx / rst_rastertoworldcoordy @@ -1709,105 +2545,205 @@ Powered by **rasterio**. **Signature:** `rst_rastertoworldcoordx(tile: Column, pixelX: Column, pixelY: Column): Column`, `rst_rastertoworldcoordy(tile: Column, pixelX: Column, pixelY: Column): Column` — World X / Y coordinate of a pixel. + + +### rst_resample + + + +:::note Lightweight tier (pyrx) +Powered by **rasterio** (`rasterio.warp`). +::: + +Resample a raster tile by a multiplicative factor via `gdal.Warp -r`, scaling pixel dimensions up or down relative to the source. + +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: + +**Signature:** `rst_resample(tile: Column, factor: Column, algorithm: Column): Column` + +**Parameters:** `factor` — multiplicative scale factor applied to both width and height (e.g. `2.0` doubles the pixel grid); `algorithm` — gdalwarp resampling method name (e.g. `bilinear`, `near`, `cubic`, `cubicspline`, `lanczos`, `average`) + +:::tip Esri Resample / changing cell size +The `rst_resample*` family is GeoBrix's equivalent of the ArcGIS **Resample** tool — producing a *new* raster at a different cell size with a chosen resampling method (`algorithm`: `near`, `bilinear`, `cubic`, `cubicspline`, `lanczos`, `average`). Pick the variant by how you specify the target: + +- **`rst_resample`** — by a multiplicative **factor** (e.g. `2.0` doubles the grid). +- **[`rst_resample_to_res`](#rst_resample_to_res)** — by target **ground resolution** in CRS units (e.g. metres per pixel). +- **[`rst_resample_to_size`](#rst_resample_to_size)** — by target **pixel dimensions** (width × height). + +Unlike [`rst_sample`](#rst_sample) — which reads a value at a point and is nearest-pixel only — resampling rewrites the whole grid. So to *sample* with **bilinear or cubic interpolation**, resample first with the desired `algorithm`, then call `rst_sample`. + +See [Raster Sampling](./raster-sampling) for the full workflow guide. +::: + **SQL:** - + -### rst_transform +### rst_resample_to_res :::note Lightweight tier (pyrx) -Powered by **rasterio** (`rasterio.warp`). Reprojection uses the GDAL build bundled with rasterio, whose projection database and driver set may be narrower than the heavyweight tier. +Powered by **rasterio** (`rasterio.warp`). +::: + +Resample a raster tile to an explicit ground resolution in CRS units via `gdal.Warp -tr`. + +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). ::: -**Signature:** `rst_transform(tile: Column, targetSrid: Column): Column` — Reproject to target CRS. `targetSrid` must be a positive EPSG code; `0` or an unknown code is rejected with a clear error. +**Signature:** `rst_resample_to_res(tile: Column, xRes: Column, yRes: Column, algorithm: Column): Column` + +**Parameters:** `xRes` — target pixel width in CRS units (e.g. metres for a metric projection); `yRes` — target pixel height in CRS units; `algorithm` — gdalwarp resampling method name (e.g. `average`, `bilinear`, `near`) + +Part of the `rst_resample*` family — see the [Esri Resample note](#rst_resample) for how it relates to the ArcGIS Resample tool and `rst_sample`. **SQL:** - + -### rst_tryopen +### rst_resample_to_size :::note Lightweight tier (pyrx) -Powered by **rasterio**. +Powered by **rasterio** (`rasterio.warp`). ::: -**Signature:** `rst_tryopen(tile: Column): Column` — Validate raster can be opened. +Resample a raster tile to an explicit pixel grid size via `gdal.Warp -ts`. + +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: + +**Signature:** `rst_resample_to_size(tile: Column, widthPx: Column, heightPx: Column, algorithm: Column): Column` + +**Parameters:** `widthPx` — target output width in pixels; `heightPx` — target output height in pixels; `algorithm` — gdalwarp resampling method name (e.g. `near` for categorical rasters, `bilinear` for continuous) + +Part of the `rst_resample*` family — see the [Esri Resample note](#rst_resample) for how it relates to the ArcGIS Resample tool and `rst_sample`. **SQL:** - + -### rst_updatetype +### rst_transform :::note Lightweight tier (pyrx) -Powered by **rasterio**. Output is re-encoded as GeoTIFF; a NoData value that is not representable in the target type is dropped. +Powered by **rasterio** (`rasterio.warp`). Reprojection uses the GDAL build bundled with rasterio, whose projection database and driver set may be narrower than the heavyweight tier. ::: -**Signature:** `rst_updatetype(tile: Column, newType: Column): Column` — Convert raster data type. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_transform(tile: Column, targetSrid: Column): Column` — Reproject to a target SRID (a positive EPSG or ESRI code, classified at apply time); `0` or a code in neither registry is rejected with a clear error. Use `rst_transformcrs` to reproject to a target given as a CRS string. See [Coordinate Reference Systems](./coordinate-reference-systems). - + -### rst_resample +### rst_transformcrs :::note Lightweight tier (pyrx) -Powered by **rasterio** (`rasterio.warp`). +Powered by **rasterio** (`rasterio.warp`) + **pyproj**. Reprojection uses the GDAL build bundled with rasterio, whose projection database and driver set may be narrower than the heavyweight tier. ::: -Resample a raster tile by a multiplicative factor via `gdal.Warp -r`, scaling pixel dimensions up or down relative to the source. - -**Signature:** `rst_resample(tile: Column, factor: Column, algorithm: Column): Column` - -**Parameters:** `factor` — multiplicative scale factor applied to both width and height (e.g. `2.0` doubles the pixel grid); `algorithm` — gdalwarp resampling method name (e.g. `bilinear`, `near`, `cubic`, `cubicspline`, `lanczos`, `average`) +**Signature:** `rst_transformcrs(tile: Column, targetCrs: Column): Column` — reproject to a target CRS given as a *string* (`EPSG:x` / `ESRI:x` / WKT / PROJ4; an int-castable string is treated as a SRID). See [Coordinate Reference Systems](./coordinate-reference-systems). -**SQL:** +Distinct from `rst_transform` (integer EPSG only): `rst_transformcrs` accepts any CRS string — an authority code (`EPSG:3857`, `ESRI:54008`), WKT, or PROJ4 — so you can reproject to a non-EPSG target. An int-castable string (`'3857'`) is treated as an EPSG SRID. See the [CRS: SRID int vs CRS string](#rst_crs) note above. - + -### rst_resample_to_res +### rst_tryopen :::note Lightweight tier (pyrx) -Powered by **rasterio** (`rasterio.warp`). +Powered by **rasterio**. ::: -Resample a raster tile to an explicit ground resolution in CRS units via `gdal.Warp -tr`. - -**Signature:** `rst_resample_to_res(tile: Column, xRes: Column, yRes: Column, algorithm: Column): Column` - -**Parameters:** `xRes` — target pixel width in CRS units (e.g. metres for a metric projection); `yRes` — target pixel height in CRS units; `algorithm` — gdalwarp resampling method name (e.g. `average`, `bilinear`, `near`) +**Signature:** `rst_tryopen(tile: Column): Column` — Validate raster can be opened. -**SQL:** +_Examples use the multiband fixture (`rgb_nir_small.tif`, committed to the repo, always openable)._ - + -### rst_resample_to_size +### rst_updatetype :::note Lightweight tier (pyrx) -Powered by **rasterio** (`rasterio.warp`). +Powered by **rasterio**. Output is re-encoded as GeoTIFF; a NoData value that is not representable in the target type is dropped. ::: -Resample a raster tile to an explicit pixel grid size via `gdal.Warp -ts`. - -**Signature:** `rst_resample_to_size(tile: Column, widthPx: Column, heightPx: Column, algorithm: Column): Column` - -**Parameters:** `widthPx` — target output width in pixels; `heightPx` — target output height in pixels; `algorithm` — gdalwarp resampling method name (e.g. `near` for categorical rasters, `bilinear` for continuous) +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_updatetype(tile: Column, newType: Column): Column` — Convert raster data type. - + ### rst_worldtorastercoord @@ -1819,13 +2755,14 @@ Powered by **rasterio**. **Signature:** `rst_worldtorastercoord(tile: Column, worldX: Column, worldY: Column): Column` — World to pixel coordinates as a struct with `.x` and `.y` fields. -**rst_worldtorastercoord** — full struct (pixel with `.x` and `.y`): - - - -**rst_worldtorastercoord** — multiple points (e.g. from a locations table): - - + ### rst_worldtorastercoordx / rst_worldtorastercoordy @@ -1837,13 +2774,14 @@ Powered by **rasterio**. **Signature:** `rst_worldtorastercoordx(tile: Column, worldX: Column, worldY: Column): Column`, `rst_worldtorastercoordy(tile: Column, worldX: Column, worldY: Column): Column` — Pixel column / row for a world coordinate. -**rst_worldtorastercoordx** — pixel column only: - - - -**rst_worldtorastercoordy** — pixel row only: - - + --- @@ -1851,37 +2789,51 @@ Powered by **rasterio**. Reproject rasters to EPSG:3857 (Web Mercator) and emit slippy-map XYZ tiles. Pair with [`gbx_pmtiles_agg`](./pmtiles-functions#pmtiles_agg) or the [PMTiles writer](../writers/pmtiles) to publish a raster pyramid as a single `.pmtiles` archive. See the [Helios notebooks](../notebooks/helios) for a worked example: NAIP aerial scenes are reprojected and pyramided into a PMTiles archive in NB02. -### rst_to_webmercator +### rst_tilexyz :::note Lightweight tier (pyrx) -Powered by **rasterio** (`rasterio.warp`). Uses rasterio's bundled GDAL build, whose projection and driver coverage may be narrower than the heavyweight tier. +Powered by **rio-tiler** + **morecantile**. Out-of-extent tiles return a transparent PNG (never null); available output formats depend on rasterio's bundled GDAL build. ::: -**Signature:** `rst_to_webmercator(tile: Column): Column` — Reproject a raster to EPSG:3857 (Web Mercator) using bilinear resampling by default. The returned tile carries `srid = 3857`. +**Signature:** `rst_tilexyz(tile: Column, z: Column, x: Column, y: Column, format: Column, tileSize: Column, resampling: Column): Column` — Render a single web-mercator XYZ tile from a raster as encoded image bytes (e.g. PNG, JPEG, WebP) at the given tile coordinates and pixel size. -**SQL:** +:::note Display RGB(A) output (both tiers) +The output is a display web-map tile, not the source's raw bands. **PNG and WebP** are **RGBA** (4-band); **JPEG** is **RGB** (3-band, no alpha). The alpha channel is a binary transparency mask derived from the source's valid-data footprint, so a pixel that is NoData — whether outside the raster or an internal hole — renders **transparent**, and both the lightweight and heavyweight tiers agree on which pixels are transparent. Band mapping matches the lightweight tier: a single-band source becomes greyscale RGB (`R=G=B`), a two-band source uses band 1 as greyscale RGB and band 2 as the alpha channel, three bands map to R, G, B, an existing fourth band is treated as alpha, and five-or-more-band sources use the first three bands as RGB. Non-8-bit sources are contrast-rescaled to 8-bit per the `rescale` argument (default `"auto"`; see the `rescale` note above). WebP alpha requires GDAL WebP-alpha support in the runtime; where absent, WebP falls back to RGB. +::: - + -### rst_tilexyz +### rst_to_webmercator :::note Lightweight tier (pyrx) -Powered by **rio-tiler** + **morecantile**. Out-of-extent tiles return a transparent PNG (never null); available output formats depend on rasterio's bundled GDAL build. +Powered by **rasterio** (`rasterio.warp`). Uses rasterio's bundled GDAL build, whose projection and driver coverage may be narrower than the heavyweight tier. ::: -**Signature:** `rst_tilexyz(tile: Column, z: Column, x: Column, y: Column, format: Column, tileSize: Column, resampling: Column): Column` — Render a single web-mercator XYZ tile from a raster as encoded image bytes (e.g. PNG, JPEG, WebP) at the given tile coordinates and pixel size. - -:::note Display RGB(A) output (both tiers) -The output is a display web-map tile, not the source's raw bands. **PNG and WebP** are **RGBA** (4-band); **JPEG** is **RGB** (3-band, no alpha). The alpha channel is a binary transparency mask derived from the source's valid-data footprint, so a pixel that is NoData — whether outside the raster or an internal hole — renders **transparent**, and both the lightweight and heavyweight tiers agree on which pixels are transparent. Band mapping matches the lightweight tier: a single-band source becomes greyscale RGB (`R=G=B`), a two-band source uses band 1 as greyscale RGB and band 2 as the alpha channel, three bands map to R, G, B, an existing fourth band is treated as alpha, and five-or-more-band sources use the first three bands as RGB. Non-8-bit sources are contrast-rescaled to 8-bit per the `rescale` argument (default `"auto"`; see the `rescale` note above). WebP alpha requires GDAL WebP-alpha support in the runtime; where absent, WebP falls back to RGB. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). ::: -**SQL:** +**Signature:** `rst_to_webmercator(tile: Column): Column` — Reproject a raster to EPSG:3857 (Web Mercator) using bilinear resampling by default. The returned tile carries `srid = 3857`. - + ### rst_xyzpyramid @@ -1893,9 +2845,14 @@ Powered by **rio-tiler** + **morecantile**. Streams one XYZ tile row per interse **Signature:** `rst_xyzpyramid(tile: Column, minZoom: Column, maxZoom: Column): Column` — Generator: explode a raster into one row per intersecting `(z, x, y)` tile across a zoom range, producing PNG bytes per tile. Use `LATERAL VIEW` to materialize the rows; the output struct exposes `z`, `x`, `y`, and `bytes`. Each tile is rendered by `rst_tilexyz`, so the per-tile PNG is display **RGBA** with the same band mapping and binary NoData alpha described above. -**SQL:** - - + --- @@ -1903,74 +2860,115 @@ Powered by **rio-tiler** + **morecantile**. Streams one XYZ tile row per interse Move data between the raster (`tile`) and vector (`geom`) worlds. -### rst_rasterize +### rst_polygonize - + :::note Lightweight tier (pyrx) Powered by **rasterio** (`rasterio.features`). ::: -**Signature:** `rst_rasterize(geom: Column, burnValue: Column, xMin: Column, yMin: Column, xMax: Column, yMax: Column, width: Column, height: Column, srid: Column): Column` — Burn a polygon (WKB) into a fresh GeoTIFF tile at the given extent and pixel dimensions. Pixels inside the polygon carry `burnValue`; pixels outside are NoData. +:::note Tier differences — SQL invocation +Heavyweight registers `gbx_rst_polygonize` as a scalar (ARRAY-returning) function — call +it directly: `SELECT gbx_rst_polygonize(tile, band, connectedness) AS features FROM `, +where each array element is a `struct(geom_wkb, value)`. The lightweight (pyrx) tier +registers it as a streaming Python **table function**, so lightweight SQL must use `LATERAL` +— which also streams polygon rows without buffering (avoids OOM on rasters with unbounded +polygon fan-out): `SELECT t.geom_wkb, t.value FROM
, LATERAL gbx_rst_polygonize(tile, band, connectedness) t`. +Both forms appear in the SQL tab below. +::: + +**Signature (heavyweight):** `rst_polygonize(tile: Column, band: Column, connectedness: Column): Column` — Trace contiguous-value regions of a tile into an array of features. Each feature carries the source pixel value as the `value` field. **SQL:** - + -### rst_polygonize +--- - +### rst_rasterize + + :::note Lightweight tier (pyrx) -Powered by **rasterio** (`rasterio.features`). In pyrx, `gbx_rst_polygonize` is a -streaming Python UDTF — invoke it as a SQL LATERAL table function to stream polygon -rows without buffering (avoids OOM on rasters with unbounded polygon fan-out): +Powered by **rasterio** (`rasterio.features`). +::: -```sql -SELECT t.geom_wkb, t.value -FROM
, LATERAL gbx_rst_polygonize(tile, band, connectedness) t -``` +:::note Lightweight-only `out_crs` parameter +The lightweight **Python** binding accepts an optional trailing `out_crs` (string CRS) argument that the SQL and heavyweight/Scala tiers do not — the heavyweight `RST_Rasterize` builder is strictly 9-argument (`out_srid` only) and rejects a 10th argument. In lightweight Python, `out_crs` (string) wins over the int `out_srid`; both → error; neither → the geometry's carried source CRS. This is a lightweight superset, not a heavyweight regression. ::: -**Signature (heavyweight):** `rst_polygonize(tile: Column, band: Column, connectedness: Column): Column` — Trace contiguous-value regions of a tile into an array of features. Each feature carries the source pixel value as the `value` field. +**Signature:** `rst_rasterize(geom: Column, burnValue: Column, xMin: Column, yMin: Column, xMax: Column, yMax: Column, width: Column, height: Column, [out_srid: Column = null]): Column` — Burn a polygon (WKB) into a fresh GeoTIFF tile at the given extent and pixel dimensions. Pixels inside the polygon carry `burnValue`; pixels outside are NoData. The output CRS is `out_srid` (integer EPSG code); the geometry is reprojected from its source CRS into the output CRS before burning. When `out_srid` is omitted the geometry's carried source CRS is used. See [Coordinate Reference Systems](./coordinate-reference-systems#source-crs-vs-output-crs). **SQL:** - - ---- + ## Terrain Analysis {#terrain} Thin wrappers around `gdal.DEMProcessing` for digital elevation model (DEM) derivatives. Each function takes a single-band DEM tile and returns a derived tile of the same footprint. See the [Helios notebooks](../notebooks/helios) for a worked example: `gbx_rst_slope`, `gbx_rst_aspect`, and `gbx_rst_hillshade` are applied to 3DEP DEMs in NB03 to produce terrain layers and a per-H3-cell solar score. -### rst_slope +### rst_aspect :::note Lightweight tier (pyrx) -Powered by **NumPy** — a reimplementation of GDAL `gdaldem`. Results are close but not bit-identical: edge pixels are filled by replicating the border (gdaldem leaves them NoData) and NoData cells are not excluded from the 3×3 window. +Powered by **NumPy** — a reimplementation of GDAL `gdaldem`; results are close but not bit-identical to the heavyweight tier (edge pixels are filled and NoData is not excluded from the 3×3 window). ::: -**Signature:** `rst_slope(tile: Column, unit: Column, scale: Column): Column` — Compute slope per pixel. `unit` is `'degrees'` or `'percent'`; `scale` is the elevation/horizontal unit ratio (use `111120` for unprojected lon/lat in degrees). +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_aspect(tile: Column, trigonometric: Column, zeroForFlat: Column): Column` — Compass direction of steepest descent in degrees (0=N, 90=E, 180=S, 270=W). Flat areas return `-9999` unless `zeroForFlat = true`. Set `trigonometric = true` for mathematical convention (0=E, counter-clockwise). - + -### rst_aspect +### rst_color_relief -:::note Lightweight tier (pyrx) -Powered by **NumPy** — a reimplementation of GDAL `gdaldem`; results are close but not bit-identical to the heavyweight tier (edge pixels are filled and NoData is not excluded from the 3×3 window). +:::note Cross-tier output differs (both tiers, by design) +The two tiers use **different color interpolation engines**, so their pixel values are close but **not identical** and are not byte-comparable: the heavyweight tier calls GDAL `gdal.DEMProcessing` color-relief (GDAL's native C interpolation), while the lightweight (`pyrx`) tier is a **NumPy** reimplementation using per-channel `np.interp`. Two known differences on the lightweight tier: the `gdaldem` `default` color keyword is **not supported** (it is skipped; out-of-range elevations are clamped to the nearest color stop by `np.interp` instead), and boundary/edge interpolation may differ slightly. Both tiers emit an RGB or RGBA Byte tile (RGBA when any color-table entry carries an alpha column). Because the outputs diverge, this function is not cross-tier fingerprint-compared in benchmarks (measured `timing-only`); pick the tier by your pipeline (heavyweight for `gdaldem`-exact output, lightweight for the pure-Python/Serverless path). ::: -**Signature:** `rst_aspect(tile: Column, trigonometric: Column, zeroForFlat: Column): Column` — Compass direction of steepest descent in degrees (0=N, 90=E, 180=S, 270=W). Flat areas return `-9999` unless `zeroForFlat = true`. Set `trigonometric = true` for mathematical convention (0=E, counter-clockwise). +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_color_relief(tile: Column, colorTablePath: Column): Column` — Apply a `gdaldem` color table (`elevation R G B [A]` per line) to produce an RGB(A) visualization tile. Special values `nv` (NoData color), `0%`, and `100%` (percentages of the band value range) are honored on both tiers; the `default` keyword is honored only on the heavyweight tier (see the tier note above). + + - +--- ### rst_hillshade @@ -1980,13 +2978,26 @@ Powered by **NumPy** — a reimplementation of GDAL `gdaldem`; results are close Powered by **NumPy** — a reimplementation of GDAL `gdaldem`; results are close but not bit-identical to the heavyweight tier (edge pixels are filled and NoData is not excluded from the 3×3 window). ::: -**Signature:** `rst_hillshade(tile: Column, azimuth: Column, altitude: Column, zFactor: Column): Column` — 8-bit (0..255) shaded relief image. Common values: NW sun azimuth `315.0`, altitude `45.0`, `zFactor = 1.0`. +:::note Lightweight-only `xscale` / `yscale` parameters +The lightweight **Python** binding accepts two optional horizontal-scale overrides — `xscale` and `yscale` — that the SQL and heavyweight/Scala tiers do not (the heavyweight `RST_Hillshade` builder is strictly 4-argument: `tile, azimuth, altitude, z_factor`). By default the horizontal scale is auto-derived from the CRS; pass **both** `xscale` and `yscale` to override it. This is a lightweight superset, not a heavyweight regression. +::: -**SQL:** +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: - +**Signature:** `rst_hillshade(tile: Column, azimuth: Column, altitude: Column, zFactor: Column): Column` — 8-bit (0..255) shaded relief image. Common values: NW sun azimuth `315.0`, altitude `45.0`, `zFactor = 1.0`. -### rst_tri + + +### rst_roughness @@ -1994,27 +3005,45 @@ Powered by **NumPy** — a reimplementation of GDAL `gdaldem`; results are close Powered by **NumPy** — a reimplementation of GDAL `gdaldem`; results are close but not bit-identical to the heavyweight tier (edge pixels are filled and NoData is not excluded from the 3×3 window). ::: -**Signature:** `rst_tri(tile: Column): Column` — Terrain Ruggedness Index — mean absolute difference between a pixel and its 8 neighbours. Useful for landscape-ecology habitat scoring. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_roughness(tile: Column): Column` — Largest absolute difference between a pixel and any of its 8 neighbours in a 3×3 window. - + -### rst_tpi +### rst_slope :::note Lightweight tier (pyrx) -Powered by **NumPy** — a reimplementation of GDAL `gdaldem`; results are close but not bit-identical to the heavyweight tier (edge pixels are filled and NoData is not excluded from the 3×3 window). +Powered by **NumPy** — a reimplementation of GDAL `gdaldem`. Results are close but not bit-identical: edge pixels are filled by replicating the border (gdaldem leaves them NoData) and NoData cells are not excluded from the 3×3 window. ::: -**Signature:** `rst_tpi(tile: Column): Column` — Topographic Position Index — pixel value minus the mean of its 8 neighbours. Positive values are ridges, negative values are valleys. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_slope(tile: Column, unit: Column, xscale: Column, yscale: Column): Column` — Compute slope per pixel. `unit` is `'degrees'` or `'percent'`; `xscale` and `yscale` are the elevation/horizontal unit ratio per axis (supply both or neither; when omitted, GDAL 3.11+ auto-derives from the raster CRS). For isotropic scaling pass the same value to both (e.g. `1.0, 1.0` for a projected CRS in metres). - + -### rst_roughness +### rst_tpi @@ -2022,27 +3051,43 @@ Powered by **NumPy** — a reimplementation of GDAL `gdaldem`; results are close Powered by **NumPy** — a reimplementation of GDAL `gdaldem`; results are close but not bit-identical to the heavyweight tier (edge pixels are filled and NoData is not excluded from the 3×3 window). ::: -**Signature:** `rst_roughness(tile: Column): Column` — Largest absolute difference between a pixel and any of its 8 neighbours in a 3×3 window. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_tpi(tile: Column): Column` — Topographic Position Index — pixel value minus the mean of its 8 neighbours. Positive values are ridges, negative values are valleys. - + -### rst_color_relief +### rst_tri -:::note Cross-tier output differs (both tiers, by design) -The two tiers use **different color interpolation engines**, so their pixel values are close but **not identical** and are not byte-comparable: the heavyweight tier calls GDAL `gdal.DEMProcessing` color-relief (GDAL's native C interpolation), while the lightweight (`pyrx`) tier is a **NumPy** reimplementation using per-channel `np.interp`. Two known differences on the lightweight tier: the `gdaldem` `default` color keyword is **not supported** (it is skipped; out-of-range elevations are clamped to the nearest color stop by `np.interp` instead), and boundary/edge interpolation may differ slightly. Both tiers emit an RGB or RGBA Byte tile (RGBA when any color-table entry carries an alpha column). Because the outputs diverge, this function is not cross-tier fingerprint-compared in benchmarks (measured `timing-only`); pick the tier by your pipeline (heavyweight for `gdaldem`-exact output, lightweight for the pure-Python/Serverless path). +:::note Lightweight tier (pyrx) +Powered by **NumPy** — a reimplementation of GDAL `gdaldem`; results are close but not bit-identical to the heavyweight tier (edge pixels are filled and NoData is not excluded from the 3×3 window). ::: -**Signature:** `rst_color_relief(tile: Column, colorTablePath: Column): Column` — Apply a `gdaldem` color table (`elevation R G B [A]` per line) to produce an RGB(A) visualization tile. Special values `nv` (NoData color), `0%`, and `100%` (percentages of the band value range) are honored on both tiers; the `default` keyword is honored only on the heavyweight tier (see the tier note above). - -**SQL:** +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: - +**Signature:** `rst_tri(tile: Column): Column` — Terrain Ruggedness Index — mean absolute difference between a pixel and its 8 neighbours. Useful for landscape-ecology habitat scoring. ---- + ## Spectral Indices @@ -2056,27 +3101,47 @@ Multi-band satellite math built on `gbx_rst_mapalgebra`. Band arguments are 1-ba Powered by **rasterio** + **NumPy**. Valid pixels match the heavyweight tier; pixels with a zero denominator are set to NoData (-9999) rather than left as non-finite values. ::: -**Signature:** `rst_evi(tile: Column, redBand: Column, nirBand: Column, blueBand: Column): Column` — Enhanced Vegetation Index. Formula: `G * (NIR - Red) / (NIR + C1*Red - C2*Blue + L)` with MODIS canonical coefficients `G=2.5, L=1.0, C1=6.0, C2=7.5`. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_evi(tile: Column, redBand: Column, nirBand: Column, blueBand: Column): Column` — Enhanced Vegetation Index. Formula: `G * (NIR - Red) / (NIR + C1*Red - C2*Blue + L)` with MODIS canonical coefficients `G=2.5, L=1.0, C1=6.0, C2=7.5`. - + -### rst_savi +### rst_index :::note Lightweight tier (pyrx) -Powered by **rasterio** + **NumPy**. Valid pixels match the heavyweight tier; pixels with a zero denominator are set to NoData (-9999) rather than left as non-finite values. +Powered by **rasterio** + **NumExpr**. Generic named-index dispatcher over a `band_map`; single-band Float32. Zero-denominator pixels are set to NoData (−9999). ::: -**Signature:** `rst_savi(tile: Column, redBand: Column, nirBand: Column, l: Column): Column` — Soil-Adjusted Vegetation Index. Formula: `(NIR - Red) / (NIR + Red + L) * (1 + L)`. `L = 0.5` (the canonical default) is a balanced soil/vegetation tradeoff; `L = 0` reduces SAVI to NDVI. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_index(tile: Column, indexName: Column, bandMap: Column): Column` — Generic dispatcher that picks a built-in formula by name and wires bands via a `MAP` (e.g. `map('red', 1, 'nir', 2)`). Built-in names: `ndvi`, `gndvi`, `msavi`, `ndvi_re`, `ndmi`, `ndsi`. - + -### rst_ndwi +--- + +### rst_nbr @@ -2084,13 +3149,22 @@ Powered by **rasterio** + **NumPy**. Valid pixels match the heavyweight tier; pi Powered by **rasterio** + **NumPy**. Valid pixels match the heavyweight tier; pixels with a zero denominator are set to NoData (-9999) rather than left as non-finite values. ::: -**Signature:** `rst_ndwi(tile: Column, greenBand: Column, nirBand: Column): Column` — Normalized Difference Water Index (McFeeters 1996). Formula: `(Green - NIR) / (Green + NIR)`. Positive values typically indicate open water. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_nbr(tile: Column, nirBand: Column, swirBand: Column): Column` — Normalized Burn Ratio. Formula: `(NIR - SWIR) / (NIR + SWIR)`. The pre-/post-fire difference (`dNBR`) is the canonical burn-severity index. - + -### rst_nbr +### rst_ndwi @@ -2098,27 +3172,43 @@ Powered by **rasterio** + **NumPy**. Valid pixels match the heavyweight tier; pi Powered by **rasterio** + **NumPy**. Valid pixels match the heavyweight tier; pixels with a zero denominator are set to NoData (-9999) rather than left as non-finite values. ::: -**Signature:** `rst_nbr(tile: Column, nirBand: Column, swirBand: Column): Column` — Normalized Burn Ratio. Formula: `(NIR - SWIR) / (NIR + SWIR)`. The pre-/post-fire difference (`dNBR`) is the canonical burn-severity index. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: -**SQL:** +**Signature:** `rst_ndwi(tile: Column, greenBand: Column, nirBand: Column): Column` — Normalized Difference Water Index (McFeeters 1996). Formula: `(Green - NIR) / (Green + NIR)`. Positive values typically indicate open water. - + -### rst_index +### rst_savi :::note Lightweight tier (pyrx) -Powered by **rasterio** + **NumExpr**. Generic named-index dispatcher over a `band_map`; single-band Float32. Zero-denominator pixels are set to NoData (−9999). +Powered by **rasterio** + **NumPy**. Valid pixels match the heavyweight tier; pixels with a zero denominator are set to NoData (-9999) rather than left as non-finite values. ::: -**Signature:** `rst_index(tile: Column, indexName: Column, bandMap: Column): Column` — Generic dispatcher that picks a built-in formula by name and wires bands via a `MAP` (e.g. `map('red', 1, 'nir', 2)`). Built-in names: `ndvi`, `gndvi`, `msavi`, `ndvi_re`, `ndmi`, `ndsi`. - -**SQL:** +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: - +**Signature:** `rst_savi(tile: Column, redBand: Column, nirBand: Column, l: Column): Column` — Soil-Adjusted Vegetation Index. Formula: `(NIR - Red) / (NIR + Red + L) * (1 + L)`. `L = 0.5` (the canonical default) is a balanced soil/vegetation tradeoff; `L = 0` reduces SAVI to NDVI. ---- + ## Pixel ops + extraction @@ -2132,9 +3222,22 @@ Per-pixel transformations and band-level extraction. Powered by **rasterio**. ::: +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: + **Signature:** `rst_band(tile: Column, bandIndex: Column): Column` — Extract a single band from a multi-band raster as a new single-band tile (`gdal.Translate -b N`). 1-based band index. - +_Example uses the multiband fixture (`rgb_nir_small.tif`, 3 bands) to demonstrate extraction; result has `num_bands = 1`._ + + ### rst_buildoverviews @@ -2144,9 +3247,20 @@ Powered by **rasterio**. Powered by **rasterio**. Builds internal GeoTIFF overviews. ::: +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: + **Signature:** `rst_buildoverviews(tile: Column, levels: Column, [resampling: Column = lit("average")]): Column` — Add pyramid overview levels to a tile via `ds.BuildOverviews`. `levels` is an `ARRAY` (e.g. `array(2, 4, 8, 16)`); `resampling` is one of `nearest`, `average`, `gauss`, `cubic`, `cubicspline`, `lanczos`, `bilinear`, `mode`. - + ### rst_fillnodata @@ -2156,9 +3270,20 @@ Powered by **rasterio**. Builds internal GeoTIFF overviews. Powered by **rasterio** (`rasterio.fill`). ::: +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: + **Signature:** `rst_fillnodata(tile: Column, [maxSearchDist: Column = lit(100), smoothingIter: Column = lit(0)]): Column` — Fill NoData pixels via `gdal.FillNodata` using inverse-distance interpolation from neighbors within `maxSearchDist` pixels. `smoothingIter` applies an optional post-fill 3×3 smoothing pass. - + ### rst_histogram @@ -2170,7 +3295,16 @@ Powered by **rasterio** + **NumPy**. Per-band bucket counts via `numpy.histogram **Signature:** `rst_histogram(tile: Column, [bands: Column = null, nBuckets: Column = lit(256), min: Column = null, max: Column = null, includeNodata: Column = lit(false)]): Column` — Compute per-band histograms via `band.GetHistogram`. Returns `MAP>` keyed by `"band_"` with bucket counts. If `bands` is null, all bands are processed; if `min` / `max` are null, GDAL auto-detects the range. - +_Examples use the multiband fixture (`rgb_nir_small.tif`, 3 bands) so the histogram has entries for each band._ + + ### rst_sample @@ -2180,9 +3314,46 @@ Powered by **rasterio** + **NumPy**. Per-band bucket counts via `numpy.histogram Powered by **rasterio**. Point geometries only; the point is assumed to be in the raster's CRS. ::: -**Signature:** `rst_sample(tile: Column, geom: Column): Column` — Sample the raster at the geometry's location(s). For a `POINT`, returns `ARRAY` of one value per band at the nearest pixel. Geometry is interpreted in EPSG:4326 lon/lat unless its EWKB carries a different SRID. +**Signature:** `rst_sample(tile: Column, geom: Column, [crs: Column = null]): Column` — Sample the raster at the geometry's location(s). For a `POINT`, returns `ARRAY` of one value per band at the nearest pixel. The point is reprojected from its source CRS to the raster CRS: an EWKB/EWKT embedded SRID wins, else the optional `crs` (string, source role), else assumed already aligned. See [Coordinate Reference Systems](./coordinate-reference-systems#source-crs-vs-output-crs). + +:::tip Esri RS_VALUE / raster sampling +`rst_sample` is GeoBrix's equivalent of the ArcGIS **Sample** tool's `RS_VALUE` — reading a raster's cell value at a point. It returns `ARRAY` (one value per band, in band order), or `null` where the point falls outside the raster (the NoData/out-of-extent case). Sampling is nearest-pixel; for bilinear or cubic, run [`rst_resample`](#rst_resample) first, then sample. + +- **Multiple rasters** (Esri's `RS_VALUE1`, `RS_VALUE2`, …): call `rst_sample` once per raster and alias each result column. +- **A points table against a raster tile set** (the Sample-tool workflow at scale): join the points to the tiles that contain them — with `st_intersects`, or a raster grid tessellation such as `rst_h3_tessellate` for an index join — then call `rst_sample(tile, point)` per row. + +See [Raster Sampling](./raster-sampling) for the full workflow guide. +::: + + + +### rst_setcrs + + + +:::note Lightweight tier (pyrx) +Powered by **rasterio** + **pyproj**. Stamps the CRS without reprojecting. +::: + +**Signature:** `rst_setcrs(tile: Column, crs: Column): Column` — Stamp a CRS onto a raster that lacks (or has a wrong) spatial reference, from a CRS *string* (`EPSG:x` / `ESRI:x` / WKT / PROJ4; an int-castable string is treated as a SRID). Does NOT reproject — only rewrites the CRS metadata. Use `rst_transformcrs` when you need an actual reprojection. See [Coordinate Reference Systems](./coordinate-reference-systems). - +Distinct from `rst_setsrid` (integer EPSG only): `rst_setcrs` accepts any CRS string — an authority code (`EPSG:3857`, `ESRI:54008`), WKT, or PROJ4 — so a non-EPSG CRS can be applied. An int-castable string (`'4326'`) behaves like `rst_setsrid(tile, 4326)`. See the [CRS: SRID int vs CRS string](#rst_crs) note above. + + ### rst_setsrid @@ -2192,9 +3363,20 @@ Powered by **rasterio**. Point geometries only; the point is assumed to be in th Powered by **rasterio**. Stamps the CRS without reprojecting. ::: -**Signature:** `rst_setsrid(tile: Column, srid: Column): Column` — Stamp an EPSG code onto a raster that lacks (or has a wrong) spatial reference. Does NOT reproject — only sets `ds.SetProjection(...)`. Use `rst_transform` when you need an actual reprojection. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: + +**Signature:** `rst_setsrid(tile: Column, srid: Column): Column` — Stamp a SRID (`>= 0`; an EPSG or ESRI code) onto a raster that lacks (or has a wrong) spatial reference; `0` clears the CRS. Does NOT reproject — only rewrites the CRS metadata. A negative SRID is rejected, and a positive code that is neither EPSG nor ESRI raises when stamped. Use `rst_transform` when you need an actual reprojection, or `rst_setcrs` to stamp from a CRS string. See [Coordinate Reference Systems](./coordinate-reference-systems). - + ### rst_threshold @@ -2204,9 +3386,20 @@ Powered by **rasterio**. Stamps the CRS without reprojecting. Powered by **NumPy**. This tier keeps each passing pixel's original value and sets failing pixels to NoData; the heavyweight tier instead returns a 0/1 binary mask (Float32). Choose the tier that matches the output you need. ::: +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: + **Signature:** `rst_threshold(tile: Column, op: Column, value: Column): Column` — Binarize the raster: pixels matching `op value` get `1`, others get `0`. `op` is one of `>`, `>=`, `<`, `<=`, `==`, `!=`. Output is a `Byte` raster (0/1) sized to the input extent. Implemented as a `gbx_rst_mapalgebra` template. - + ## Analysis @@ -2217,36 +3410,65 @@ Higher-level analytical transforms wrapping single GDAL primitives — COG layou :::note Lightweight tier (pyrx) -Powered by **rio-cogeo**. Re-encodes the tile as a Cloud Optimized GeoTIFF (validated with `cog_validate`); `compression` maps to a rio-cogeo profile (`deflate`, `lzw`, `zstd`, `lerc`, `jpeg`, `webp`, `none`). The tile's `metadata.driver` is `GTiff` (a COG is a valid GeoTIFF). +Re-encodes the tile as a Cloud Optimized GeoTIFF (validated with `cog_validate`). `compression` defaults to `"auto"` — a **size-adaptive ZSTD** level with a dtype-matched predictor (the GeoBrix materialize baseline; see [Materialized Compression](./materialized-compression)) — or an explicit codec name (`zstd`, `deflate`, `lzw`, `lerc`, `jpeg`, `webp`, `none`). The tile's `metadata.driver` is `GTiff` (a COG is a valid GeoTIFF). ::: -**Signature:** `rst_cog_convert(tile: Column, [compression: Column = lit("DEFLATE"), blocksize: Column = lit(512), overviewResampling: Column = lit("AVERAGE")]): Column` — Re-layout a raster tile as a Cloud Optimized GeoTIFF via `gdal.Translate -of COG`. `compression` is one of `NONE`, `DEFLATE`, `LZW`, `ZSTD`, `LERC`, `JPEG`, `WEBP`. `blocksize` is the internal tile size in pixels (square). `overviewResampling` is the algorithm for the auto-generated overview pyramid. Output is a GTiff-on-disk variant suitable for HTTP range serving. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: - +**Signature:** `rst_cog_convert(tile: Column, [compression: Column, blocksize: Column = lit(512), overviewResampling: Column = lit("AVERAGE")]): Column` — Re-layout a raster tile as a Cloud Optimized GeoTIFF via `gdal.Translate -of COG`. Both tiers use a **ZSTD + dtype-predictor** baseline. `compression` accepts an explicit codec on either tier — `ZSTD`, `DEFLATE`, `LZW`, `NONE`, `LERC`, `JPEG`, `WEBP` — and its default is `ZSTD` on the heavyweight tier. The **lightweight tier additionally accepts `AUTO`** (its default): a size-adaptive ZSTD level with the dtype predictor. `blocksize` is the internal tile size in pixels (square). `overviewResampling` is the algorithm for the auto-generated overview pyramid. Output is a GTiff-on-disk variant suitable for HTTP range serving. See [Materialized Compression](./materialized-compression) for the codec/level details and when to override. See [Materialized Compression](./materialized-compression) for the codec/level details and when to override. -### rst_proximity + + +### rst_contour :::note Lightweight tier (pyrx) -Powered by **SciPy** (`scipy.ndimage.distance_transform_edt`). Distance to the nearest source pixel (`target_values`, or any non-zero pixel by default), in `GEO` (CRS units) or `PIXEL` units; pixels beyond `max_distance` → NoData −1.0. Single-band Float32. +Powered by **scikit-image** (`measure.find_contours`). Returns `ARRAY` of contour LineStrings (in the raster CRS) at each fixed `level`, or at `base + k*interval` across the data range when `levels` is empty; NoData is masked before tracing. The marching-squares line geometry differs slightly from the heavyweight GDAL contours. ::: -**Signature:** `rst_proximity(tile: Column, [targetValues: Column = null, distUnits: Column = lit("GEO"), maxDistance: Column = null]): Column` — Compute a Float32 raster where each pixel holds the distance to the nearest source pixel via `gdal.ComputeProximity`. `targetValues` is a comma-separated list of source-pixel values (e.g. `"1,2,3"`); `null` means any non-NoData pixel is a target. `distUnits` is `"GEO"` (CRS ground units, default) or `"PIXEL"`. `maxDistance` caps the output; pixels beyond it get the NoData sentinel `-1.0`. +**Signature:** `rst_contour(tile: Column, levels: Column, [interval: Column = lit(0.0), base: Column = lit(0.0), attrField: Column = lit("elev")]): Column` — Generate contour LineString features via `gdal.ContourGenerateEx`. Pass a non-empty `levels` `ARRAY` for fixed contour values, or pass `array()` and set `interval` (>0) for equal-step contours at `base + n*interval`. Returns `ARRAY` — one entry per contour line in the source raster's CRS. - + -### rst_contour +### rst_proximity :::note Lightweight tier (pyrx) -Powered by **scikit-image** (`measure.find_contours`). Returns `ARRAY` of contour LineStrings (in the raster CRS) at each fixed `level`, or at `base + k*interval` across the data range when `levels` is empty; NoData is masked before tracing. The marching-squares line geometry differs slightly from the heavyweight GDAL contours. +Powered by **SciPy** (`scipy.ndimage.distance_transform_edt`). Distance to the nearest source pixel (`target_values`, or any non-zero pixel by default), in `GEO` (CRS units) or `PIXEL` units; pixels beyond `max_distance` → NoData −1.0. Single-band Float32. ::: -**Signature:** `rst_contour(tile: Column, levels: Column, [interval: Column = lit(0.0), base: Column = lit(0.0), attrField: Column = lit("elev")]): Column` — Generate contour LineString features via `gdal.ContourGenerateEx`. Pass a non-empty `levels` `ARRAY` for fixed contour values, or pass `array()` and set `interval` (>0) for equal-step contours at `base + n*interval`. Returns `ARRAY` — one entry per contour line in the source raster's CRS. +:::note Lightweight Python — virtual-tile force-output +The lightweight **Python** binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — `virtualize_dir`, `virtualize_prefix`, and `materialize` — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See [Virtual-tile force-output params](#virtual-tile-overrides). +::: + +**Signature:** `rst_proximity(tile: Column, [targetValues: Column = null, distUnits: Column = lit("GEO"), maxDistance: Column = null]): Column` — Compute a Float32 raster where each pixel holds the distance to the nearest source pixel via `gdal.ComputeProximity`. `targetValues` is a comma-separated list of source-pixel values (e.g. `"1,2,3"`); `null` means any non-NoData pixel is a target. `distUnits` is `"GEO"` (CRS ground units, default) or `"PIXEL"`. `maxDistance` caps the output; pixels beyond it get the NoData sentinel `-1.0`. - + ### rst_viewshed @@ -2256,9 +3478,20 @@ Powered by **scikit-image** (`measure.find_contours`). Returns `ARRAY + --- diff --git a/docs/docs/api/raster-sampling.mdx b/docs/docs/api/raster-sampling.mdx new file mode 100644 index 000000000..4ed5c7d52 --- /dev/null +++ b/docs/docs/api/raster-sampling.mdx @@ -0,0 +1,338 @@ +--- +sidebar_position: 6 +sidebar_label: Raster Sampling +title: Raster Sampling +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Tier from '@site/src/components/Tier'; + +# Raster Sampling + +**Raster sampling** is the operation of reading raster values at one or more point locations — the equivalent of Esri's ArcGIS **Sample** tool and its `RS_VALUE` output columns. You have a raster (a DEM, a satellite scene, a modeled surface) and a set of coordinates; you want the cell value at each coordinate. + +GeoBrix covers this with [`rst_sample`](./raster-functions#rst_sample), available in both execution tiers. + +## Value at a point — `rst_sample` + + + +```sql +gbx_rst_sample(tile, geom [, crs]) → ARRAY +``` + +**`rst_sample`** reads the raster at a single `POINT` geometry and returns `ARRAY` — one value per band, in band-index order. A single-band DEM returns `[302.0]`; a four-band multispectral tile returns `[b1, b2, b3, b4]`. Index into the array to select a specific band: `result[0]` for band 1. + +### CRS handling + +The point's CRS and the raster's CRS do not need to match — `rst_sample` reprojects automatically using the **source-CRS rule**: + +1. **Embedded SRID wins** — an EWKB or EWKT geometry with an embedded SRID (e.g. `SRID=4326;POINT(...)`) is reprojected from that SRID to the raster's CRS. +2. **Explicit `crs` argument** — if the geometry has no embedded SRID, the optional third argument names the point's source CRS (`'EPSG:4326'`, `'32618'`, WKT, PROJ4). +3. **Assumed aligned** — if neither an SRID nor a `crs` argument is supplied, the point is assumed to be in the raster's CRS already. + +See [Coordinate Reference Systems](./coordinate-reference-systems#source-crs-vs-output-crs) for the full rule. + +### NoData and out-of-extent + +If the point falls outside the raster's extent, or if the raster's geotransform is degenerate (zero determinant), `rst_sample` returns `null` (SQL `NULL`). NoData pixels at the sampled location are also returned as `null`. Use `COALESCE` or a `CASE` expression downstream if you need a sentinel value instead. + +### Sampling method + +`rst_sample` uses **nearest-pixel** sampling. For bilinear, cubic, or other interpolation methods, see [Interpolation](#interpolation-bilinear-and-cubic) below. + +For `rst_sample` examples (doc-tested), see the [`rst_sample` function reference](./raster-functions#rst_sample). + +--- + +## Interpolation — bilinear and cubic {#interpolation-bilinear-and-cubic} + +`rst_sample` reads the nearest pixel. To sample with a different interpolation method, **resample the raster first**, then sample: + +1. **Choose a target resolution** using one of the `rst_resample*` variants: + - [`rst_resample(tile, factor, algorithm)`](./raster-functions#rst_resample) — by a multiplicative factor. + - [`rst_resample_to_res(tile, xRes, yRes, algorithm)`](./raster-functions#rst_resample_to_res) — by target ground resolution in CRS units. + - [`rst_resample_to_size(tile, widthPx, heightPx, algorithm)`](./raster-functions#rst_resample_to_size) — by target pixel dimensions. + +2. **Pass the resampled tile to `rst_sample`**. + +The `algorithm` parameter accepts the standard GDAL resampling names: `near`, `bilinear`, `cubic`, `cubicspline`, `lanczos`, `average`. + +:::note +Resampling rewrites the whole pixel grid; `rst_sample` then reads the nearest pixel of that rewritten grid. The combination gives you bilinear- or cubic-quality values at query points, at the cost of one extra warp pass per tile. For nearest-pixel sampling there is no benefit in resampling first. +::: + +For `rst_resample*` examples (doc-tested), see the [`rst_resample` function reference](./raster-functions#rst_resample). + +--- + +## Workflows + +The patterns below are **illustrative** SQL, Python, and Scala sketches. A runnable end-to-end sampling notebook is a doc-test follow-up. + +### Workflow 1 — single point value + +Extract a value (e.g. elevation) at one location across a set of raster tiles. + + + + +```sql +-- Nearest-pixel sample from a DEM at a POINT already in the raster's CRS +SELECT + tile_id, + gbx_rst_sample(tile, 'SRID=32618;POINT(500320 4500320)') AS elevation_array, + gbx_rst_sample(tile, 'SRID=32618;POINT(500320 4500320)')[0] AS elevation_m +FROM dem_rasters; + +-- Bilinear: resample first, then sample +SELECT + tile_id, + gbx_rst_sample( + gbx_rst_resample(tile, 1.0, 'bilinear'), + 'SRID=32618;POINT(500320 4500320)' + )[0] AS elevation_bilinear +FROM dem_rasters; +``` + + + + +```python +import pyspark.sql.functions as F +from databricks.labs.gbx.pyrx import functions as rx # lightweight +# from databricks.labs.gbx.rasterx import functions as rx # heavyweight — same call + +point = F.lit("SRID=32618;POINT(500320 4500320)") + +# Nearest-pixel +df = spark.table("dem_rasters") +df_sampled = df.select( + "tile_id", + rx.rst_sample("tile", point).alias("elevation_array"), + rx.rst_sample("tile", point)[0].alias("elevation_m"), +) + +# Bilinear: resample first, then sample +df_bilinear = df.select( + "tile_id", + rx.rst_sample( + rx.rst_resample("tile", F.lit(1.0), F.lit("bilinear")), + point + )[0].alias("elevation_bilinear"), +) +``` + + + + +```scala +import com.databricks.labs.gbx.rasterx.functions as rasterx +import org.apache.spark.sql.functions.lit + +val point = lit("SRID=32618;POINT(500320 4500320)") + +val df = spark.table("dem_rasters") + +// Nearest-pixel +val dfSampled = df.select( + col("tile_id"), + rasterx.rst_sample(col("tile"), point).alias("elevation_array"), + rasterx.rst_sample(col("tile"), point).getItem(0).alias("elevation_m") +) + +// Bilinear: resample first, then sample +val dfBilinear = df.select( + col("tile_id"), + rasterx.rst_sample( + rasterx.rst_resample(col("tile"), lit(1.0), lit("bilinear")), + point + ).getItem(0).alias("elevation_bilinear") +) +``` + + + + +--- + +### Workflow 2 — multiple rasters (RS_VALUE1 / RS_VALUE2 / …) + +Esri's Sample tool, when given multiple rasters, produces columns named `RS_VALUE1`, `RS_VALUE2`, and so on. In GeoBrix, compose one `rst_sample` call per raster and alias each result: + + + + +```sql +-- Sample elevation, slope, and aspect at one point across aligned tile sets. +-- Replace the tile joins with the pattern that matches your tile schema. +WITH point AS ( + SELECT 'SRID=32618;POINT(500320 4500320)' AS geom +) +SELECT + d.tile_id, + gbx_rst_sample(d.tile, p.geom)[0] AS RS_VALUE1, -- elevation (band 1 of DEM) + gbx_rst_sample(s.tile, p.geom)[0] AS RS_VALUE2, -- slope + gbx_rst_sample(a.tile, p.geom)[0] AS RS_VALUE3 -- aspect +FROM + dem_rasters d + JOIN slope_rasters s ON d.tile_id = s.tile_id + JOIN aspect_rasters a ON d.tile_id = a.tile_id + CROSS JOIN point p; +``` + + + + +```python +from databricks.labs.gbx.pyrx import functions as rx # or ...rasterx + +point = F.lit("SRID=32618;POINT(500320 4500320)") + +# Join three tile tables on a shared tile_id key, then sample each +df = ( + spark.table("dem_rasters").alias("d") + .join(spark.table("slope_rasters").alias("s"), "tile_id") + .join(spark.table("aspect_rasters").alias("a"), "tile_id") + .select( + "tile_id", + rx.rst_sample(F.col("d.tile"), point)[0].alias("RS_VALUE1"), + rx.rst_sample(F.col("s.tile"), point)[0].alias("RS_VALUE2"), + rx.rst_sample(F.col("a.tile"), point)[0].alias("RS_VALUE3"), + ) +) +``` + + + + +For a multi-band raster, index with `[0]`, `[1]`, `[2]`, … to extract individual bands into separate columns. + +--- + +### Workflow 3 — points table × raster tiles at scale + +The real-world sampling scenario: a table of many points (field survey, GPS tracks, sensor readings) against a tiled raster dataset. The key is a **spatial join** that pairs each point with the tile(s) that cover it, then calls `rst_sample` per matched row. + +Two spatial-join strategies: + +#### Option A — geometry intersection + +Use `st_intersects` on the raster tile's bounding geometry and the point. This is the general-purpose approach and works for any tiled raster. + + + + +```sql +-- Assume dem_rasters has a precomputed tile_bounds column (WKB geometry of the tile extent) +SELECT + p.id, + p.geom AS point_geom, + gbx_rst_sample(d.tile, p.geom)[0] AS elevation_m +FROM + points_table p + JOIN dem_rasters d + ON st_intersects(d.tile_bounds, p.geom) +WHERE d.tile_bounds IS NOT NULL; +``` + + + + +```python +from databricks.labs.gbx.pyrx import functions as rx +from pyspark.sql import functions as F + +points = spark.table("points_table") +tiles = spark.table("dem_rasters") + +result = ( + points.join( + tiles, + # st_intersects is a Databricks built-in spatial predicate (not a GeoBrix function) + F.expr("st_intersects(tile_bounds, geom)"), + "inner", + ) + .select( + points["id"], + points["geom"], + rx.rst_sample(tiles["tile"], points["geom"])[0].alias("elevation_m"), + ) +) +``` + + + + +#### Option B — grid index join (H3 or BNG) + +When the raster is already indexed by a discrete grid (H3, BNG, quadbin), a grid-key equi-join scales better than a geometry predicate at millions of rows. + + + + +```sql +-- 1. Assign each point an H3 index at the raster's native resolution +-- 2. Join to the H3-indexed raster table +-- 3. Sample +SELECT + p.id, + gbx_rst_sample(d.tile, p.geom)[0] AS elevation_m +FROM + (SELECT id, geom, h3_pointash3(geom, 8) AS h3_index FROM points_table) p + JOIN dem_rasters_h3 d ON p.h3_index = d.h3_index; +``` + + + + +```python +from databricks.labs.gbx.pyrx import functions as rx +# h3_pointash3 is a Databricks built-in H3 function +from pyspark.sql.functions import expr + +points = spark.table("points_table").withColumn( + "h3_index", expr("h3_pointash3(geom, 8)") +) +tiles = spark.table("dem_rasters_h3") # pre-indexed by h3_index + +result = ( + points.join(tiles, "h3_index", "inner") + .select( + points["id"], + rx.rst_sample(tiles["tile"], points["geom"])[0].alias("elevation_m"), + ) +) +``` + + + + +:::tip Scale note +For millions of points × thousands of tiles, the **grid-index equi-join** is significantly faster than a geometry intersection — it avoids a cross-product and uses Spark's hash or sort-merge join. Index the raster tiles at ingest time with `rst_h3_tessellate` (see [H3 Raster Tessellation](./h3-raster-tessellation)) and assign points the same H3 resolution at query time. +::: + +--- + +## Related functions + +| Function | What it does | Relation to sampling | +|---|---|---| +| [`rst_gridfrompoints`](./raster-functions#rst_gridfrompoints) / [`rst_gridfrompoints_agg`](./raster-functions#rst_gridfrompoints_agg) | **Inverse**: interpolate a raster *from* a set of Z-valued points (IDW) | Produces a raster from points; sampling reads a raster at points | +| [`rst_h3_rastertogrid*`](./raster-functions#rst_h3_rastertogridavg) | **Zonal aggregation**: reduce pixels within each H3 cell to a statistic (avg, sum, max, …) | Aggregates pixels to grid cells; sampling reads one pixel at a coordinate | +| [`rst_bng_rastertogrid*`](./raster-functions#rst_bng_rastertogridavg) | Same as above for British National Grid | Same distinction — aggregation vs point lookup | +| [`rst_quadbin_rastertogrid*`](./raster-functions#rst_quadbin_rastertogridavg) | Same for CARTO quadbin | Same distinction | +| [`rst_h3_tessellate`](./raster-functions#rst_h3_tessellate) | Tile indexing for H3-grid joins | Supports the scale-out join in Workflow 3 | + +The `rst_*_rastertogrid*` family and `rst_gridfrompoints*` answer **area questions** (what is the average elevation in this H3 cell?). `rst_sample` answers **point questions** (what is the elevation exactly here?). Choose based on the spatial resolution your downstream analysis requires. + +--- + +## Function reference cross-links + +- [`rst_sample`](./raster-functions#rst_sample) — full signature, examples, tier notes +- [`rst_resample`](./raster-functions#rst_resample) — resample by factor +- [`rst_resample_to_res`](./raster-functions#rst_resample_to_res) — resample by ground resolution +- [`rst_resample_to_size`](./raster-functions#rst_resample_to_size) — resample by pixel dimensions +- [Coordinate Reference Systems](./coordinate-reference-systems) — CRS handling rules used by `rst_sample` diff --git a/docs/docs/api/stac.mdx b/docs/docs/api/stac.mdx index 02ba8c601..1e430910f 100644 --- a/docs/docs/api/stac.mdx +++ b/docs/docs/api/stac.mdx @@ -34,7 +34,7 @@ pip install "geobrix[light,stac]" From a Databricks notebook (Serverless or classic): ```python -%pip install --quiet "geobrix[light,stac] @ file:///Volumes////geobrix-0.4.3-py3-none-any.whl" +%pip install --quiet "geobrix[light,stac] @ file:///Volumes////geobrix-0.5.0-py3-none-any.whl" ``` ## Import diff --git a/docs/docs/api/tile-structure.mdx b/docs/docs/api/tile-structure.mdx index 71b607641..d2c01428f 100644 --- a/docs/docs/api/tile-structure.mdx +++ b/docs/docs/api/tile-structure.mdx @@ -12,95 +12,47 @@ Understanding the internal structure of GeoBrix tiles is essential for advanced ## Overview -In GeoBrix, a **tile** is not a simple binary column—it's a **structured type** (struct) containing three fields that together represent a raster dataset along with its metadata and optional grid cell information. +In GeoBrix, a **tile** is not a simple binary column—it's a **structured type** (struct) that represents a raster dataset along with its metadata, grid-cell information, and — when the tile is a bytes-free reference — the source path, window, and clip/CRS provenance. The same struct is shared by both execution tiers. ## Tile Schema A tile has the following structure: -![GeoBrix tile schema — cellid (bigint, nullable), raster (binary, required), metadata (map of string to string), with non-tessellated vs tessellated examples](../../../resources/images/diagrams/rasterx/rasterx-tile-structure.png) +![GeoBrix tile schema — cellid (bigint, nullable), raster (binary, nullable), path (string, nullable), path_mode (string, storage mode: null / 'external' / 'managed'), window/clip_polygon/clip_crs/crs provenance fields, metadata (map of string to string, last field)](../../../resources/images/diagrams/rasterx/rasterx-tile-structure.png) ``` struct< - cellid: bigint, -- Grid cell ID (nullable) - raster: binary, -- Raster bytes - metadata: map -- Driver, extension, size, etc. + cellid: bigint, -- Grid cell ID (nullable) + raster: binary, -- Raster bytes (NULL when virtual) + path: string, -- Source path (set when virtual) + path_mode: string, -- Storage mode: null, 'external', or 'managed' + window: struct, -- Pixel window + clip_polygon: binary, -- Optional clip geometry (WKB) + clip_crs: string, -- CRS for clip_polygon + crs: string, -- Working/target CRS + metadata: map -- Driver, extension, size, etc. > ``` +A tile is **materialized** when `raster` carries the encoded bytes (the common case, and the only form the heavyweight tier consumes). A tile is **virtual** when `raster` is `NULL` and `path` + `window` are set — a bytes-free reference that materializes pixels on demand (a lightweight-tier capability). For the full virtual/materialized model, see **[Virtual Tiles](./virtual-tiles)**. + ### Field Descriptions | Field | Type | Nullable | Description | |-------|------|----------|-------------| | `cellid` | `bigint` (Long) | Yes | Grid cell identifier for tessellated rasters. `null` for non-tessellated rasters. | -| `raster` | `binary` | No | Binary raster content (bytes). | +| `raster` | `binary` | Yes | Encoded raster bytes when materialized; `null` when the tile is virtual (bytes-free). | +| `path` | `string` | Yes | Source path for a virtual tile; `null` for a materialized tile. | +| `path_mode` | `string` | Yes | Storage mode for a virtual tile. `null` = materialized (raster bytes present) or plain FUSE-path virtual. `"external"` = FILE EXTERNAL virtual tile. `"managed"` = FILE MANAGED virtual tile (governed lifecycle). | +| `window` | `struct` | Yes | The pixel window (offset + size). On a virtual tile it is the window to read; on a materialized tile it is provenance of the window already extracted. | +| `clip_polygon` | `binary` | Yes | Optional clip geometry (WKB). Instruction on a virtual tile; applied-provenance on a materialized tile. | +| `clip_crs` | `string` | Yes | CRS for `clip_polygon` (e.g. `"EPSG:4326"`). | +| `crs` | `string` | Yes | Working/target CRS. | | `metadata` | `map` | Yes | Key-value map containing driver name, file extension, size, and other metadata. | ---- - -## Field Details - -### 1. cellid - -The `cellid` field identifies which grid cell a tile belongs to when using tessellation (e.g., `rst_h3_tessellate`). - -**Properties:** -- **Type**: `bigint` (64-bit integer) -- **Nullable**: Yes -- **Purpose**: Enables spatial indexing and joining of tessellated rasters - -**Values:** -- `null` - For non-tessellated rasters (e.g., from `rst_fromfile`) -- `> 0` - For tessellated rasters (H3 cell ID) - -**Example:** - - - - - -### 2. raster - -The `raster` field contains the actual raster bytes — the full file content in memory. - -**All tile constructors and readers produce binary content:** -- `rst_fromfile(path, driver)` → reads the file at `path` into **binary** bytes -- `rst_fromcontent(content, driver)` → embeds the given **binary** bytes -- GDAL reader → **binary** (raster bytes) - -**Properties:** -- **Type**: `binary` -- **Nullable**: No -- **Purpose**: Self-contained raster payload carried through the plan; downstream operators - (`rst_clip`, `rst_transform`, ...) read and produce bytes, so there is no orphan-path risk. - -**Binary Format:** -- Complete raster file (e.g. GeoTIFF) in memory -- Can be deserialized with GDAL/rasterio -- Typically compressed (LZW, DEFLATE, etc.) - -**Example:** - - - -### 3. metadata - -The `metadata` field contains key-value pairs describing the raster format and properties. - -**Properties:** -- **Type**: `map` -- **Nullable**: Yes -- **Purpose**: Provides format information needed for GDAL operations - -**Common Keys:** -- `driver` - GDAL driver name (e.g., "GTiff", "NetCDF", "HDF4") -- `extension` - File extension (e.g., ".tif", ".nc") -- `size` - Size in bytes (as string) matching the length of the raster payload -- Other format-specific metadata - -**Example:** - - +:::note Reference vs. instruction +On a **materialized** tile, `window` / `clip_polygon` / `clip_crs` / `crs` are **provenance** — a record of what was already applied to produce the bytes. On a **virtual** tile they are **instructions** — pending operations applied when the tile is read. See [Virtual Tiles](./virtual-tiles#reference-vs-instruction). +::: --- @@ -135,22 +87,38 @@ Filter tiles based on driver or other metadata: +Access individual metadata keys from a tile: + + + ### Using Tiles in Custom UDFs Access tile components for custom processing: -### Processing Binary Raster Data +### Materialized vs Virtual: path and raster + +A virtual tile has `tile.raster = null` and `tile.path` set to the source file path. A materialized tile is the opposite — `tile.raster` carries the encoded bytes and `tile.path` is null. Check both fields to classify a tile at runtime: + + + +### Tile Storage Mode + +`tile.path_mode` records the storage model used by a virtual tile's backing data. It is `null` for both materialized tiles and plain FUSE-path virtual tiles. Use `tile.raster is null` (or check `tile.path`) to distinguish those two cases. The field carries `"external"` for FILE EXTERNAL tiles and `"managed"` for FILE MANAGED tiles (governed lifecycle via Delta): -When the `raster` field contains binary data, use it with rasterio or GDAL: + + +When `tile.raster` contains binary data, use it with rasterio or GDAL: ### Comparing `rst_fromfile` vs `rst_fromcontent` -Both produce tiles whose `raster` field is binary. Use `rst_fromfile` when you have a path, -and `rst_fromcontent` when you already have bytes (e.g. from `spark.read.format("binaryFile")`). +`rst_fromfile` takes a path and returns a **virtual** tile in the light tier (bytes-free by default; pass `materialize=True` to force bytes). `rst_fromcontent` always returns a **materialized** tile — it takes bytes you already have (e.g. from `spark.read.format("binaryFile")`) and embeds them directly. + +To enumerate paths from a Volume directory and decode each one in a single pipeline, compose +`gbx_file_read` with `rst_fromfile` — see [GBX Common Functions](../common-functions). @@ -176,15 +144,16 @@ Created by `rst_h3_tessellate`: **Characteristics:** -- `cellid` contains H3 cell ID -- Raster clipped to cell bounds -- Enables spatial joins and grid-based processing -- Metadata includes `RASTERX_CELL_ID` key +- `cellid` contains the grid cell id (H3, quadbin, or BNG depending on the tessellate function used) +- `tile.metadata["gridSystem"]` names the DGGS: `"h3"`, `"quadbin"`, or `"bng"` +- Raster is clipped to the cell's bounds +- Enables equi-joins with any DGGS-indexed table via `cellid` + `gridSystem` for unambiguous matching --- ## Next Steps +- [Virtual Tiles](./virtual-tiles) - The virtual↔materialized model over this struct - [Raster Functions](./raster-functions) - Functions that work with tiles - [Custom UDFs](../advanced/custom-udfs) - Build custom tile processing - [Library Integration](../advanced/library-integration) - Use tiles with rasterio/xarray diff --git a/docs/docs/api/vectorx-functions.mdx b/docs/docs/api/vectorx-functions.mdx index 29c3f9acb..1a7c19584 100644 --- a/docs/docs/api/vectorx-functions.mdx +++ b/docs/docs/api/vectorx-functions.mdx @@ -5,10 +5,11 @@ title: VectorX Function Reference import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -import CodeFromTest from '@site/src/components/CodeFromTest'; +import FunctionExamples from '@site/src/components/FunctionExamples'; import vectorxFunctionsExamples from '!!raw-loader!../../tests/python/api/vectorx_functions.py'; import vectorxSqlCode from '!!raw-loader!../../tests/python/api/vectorx_functions_sql.py'; -import quickstartCode from '!!raw-loader!../../tests/python/quickstart/examples.py'; +import vectorxLightCode from '!!raw-loader!../../tests/python/api/vectorx_functions_python_light.py'; +import scalaApiExamplesCode from '!!raw-loader!../../tests/scala/api/ScalaApiExamples.scala'; import Tier from '@site/src/components/Tier'; import { Impl } from '@site/src/components/Tier'; import VectorXIcon from '../../../resources/images/brand/VectorX.png'; @@ -19,6 +20,7 @@ VectorX augments the product's native `ST_*` functions with vector-tile encoding - **Vector tile encoding** — `gbx_st_asmvt` aggregator + `gbx_st_asmvt_pyramid` generator for publishing Mapbox Vector Tile (MVT) layers, available in **both** the lightweight (`pyvx`) and heavyweight (`vectorx`) tiers - **TIN surface modeling** — `gbx_st_triangulate`, `gbx_st_interpolateelevationbbox`, and `gbx_st_interpolateelevationgeom` for Delaunay triangulation and grid elevation interpolation from Z-valued points, in **both** tiers (with a `constrained`/`conforming` mode selector) +- **CRS strings** — `gbx_st_crs`, `gbx_st_setcrs`, and `gbx_st_transformcrs` for reading, stamping, and reprojecting a geometry's coordinate reference system by **CRS string** (ESRI codes, WKT, PROJ4 — not just an EPSG integer), in **both** tiers - **OGR-based vector readers** — Shapefile, GeoJSON, GeoPackage, FileGDB (heavyweight only) - **Legacy Mosaic conversion** — `gbx_st_legacyaswkb` for migrating geometries written by DBLabs Mosaic, in **both** tiers @@ -37,10 +39,125 @@ VectorX augments the product's native `ST_*` functions with vector-tile encoding | `st_triangulate` | Supported (`constrained`) | Supported (`constrained` + `conforming`) | | `st_interpolateelevationbbox` | Supported (`constrained`) | Supported (`constrained` + `conforming`) | | `st_interpolateelevationgeom` | Supported (`constrained`) | Supported (`constrained` + `conforming`) | +| `st_crs` | Supported | Supported | +| `st_setcrs` | Supported | Supported | +| `st_transformcrs` | Supported | Supported | | `st_legacyaswkb` | Supported | Supported | --- +## Setup {#setup} + +With GeoBrix already [installed](../installation), register VectorX in your session before running any example. Both tiers alias the module as `vx`, so every example below is identical regardless of tier — only this import line differs. + + + + +```python +from databricks.labs.gbx.pyvx import functions as vx +vx.register(spark) +``` + + + + +```python +from databricks.labs.gbx.vectorx import functions as vx +vx.register(spark) +``` + +For the legacy conversion function (`st_legacyaswkb`), use: + +```python +from databricks.labs.gbx.vectorx.jts.legacy import functions as vx +vx.register(spark) +``` + + + + +The examples on this page read from **four canonical DataFrames**, one per function family. Each is available as a temp view for SQL examples. Point the placeholders at your own data to reproduce the examples: + +| View / DataFrame | Schema | Backed by | Backs | +|---|---|---|---| +| `tin_survey` | `pts ARRAY`, `bl ARRAY` | 4 WKB POINT Z forming a 10×10 m square (elevations 0, 0, 10, 5 m) | TIN examples: `st_triangulate`, `st_interpolateelevationbbox`, `st_interpolateelevationgeom` | +| `mvt_features` | `z INT`, `x INT`, `y INT`, `geom_wkb BINARY`, `attrs STRUCT` | 2 tile-local WKB POINTs in tile (z=0, x=0, y=0) | Vector-tile examples: `st_asmvt`, `st_asmvt_pyramid` | +| `vector_geoms` | `geom STRING` | EWKT literal `'SRID=4326;POINT (13 42)'` | CRS examples: `st_crs`, `st_setcrs`, `st_transformcrs` | +| `legacy_geoms` | `geom_legacy STRUCT` | Legacy Mosaic struct for POINT(13, 42) | Migration: `st_legacyaswkb` | + +All four views are built from inline literals in the doc-test fixture helpers (no external files or `/Volumes` dependency, so the examples run anywhere): WKB `POINT Z` mass points for `tin_survey`, tile-local WKB points for `mvt_features`, an EWKT string for `vector_geoms`, and a legacy Mosaic struct for `legacy_geoms`. + +--- + +## Examples — Conventions {#conventions} + +### How to read the four tabs + +Every function on this page shows **one example**, expressed identically across four tabs: + +| Tab | Tier | Badge | +|---|---|---| +| **SQL** | Both (default) | — | +| **Python (light)** | `pyvx` lightweight tier | — | +| **Python (heavy)** | `vectorx` heavyweight tier | Blue | +| **Scala** | `vectorx` heavyweight tier | Blue | + +All four tabs operate on the **same input fixture** with the **same arguments**. Where a genuine tier difference exists — a diverging output schema or a mode available only in one tier — the affected tab carries a labeled `:::note`. A difference without a label is a documentation error. + +In each Python example, `df = spark.table("")` or an equivalent inline `spark.sql(...)` call loads the canonical fixture. Each SQL example reads `FROM ` or uses an inline CTE — no separate `CREATE TEMP VIEW` step is shown. + +### Output representation + +The output cells in every function table follow a uniform convention so readers can compare tabs at a glance without decoding byte strings. + +**Binary geometry ([E]WKB)** — geometry returned as `BINARY` is elided with one token and a format annotation. Use `(WKB binary)` when the output carries no embedded SRID; `(EWKB binary)` when an SRID is embedded: + +``` +... (WKB binary) +... (EWKB binary) +``` + +The WKB bytes are always the canonical output. Decode with `ST_GeomFromWKB`, `ST_GeomFromEWKB`, or any ISO WKB reader. + +**MVT bytes** — `st_asmvt` and `st_asmvt_pyramid` return Mapbox Vector Tile protobufs (BINARY). Output is shown as: + +``` +... (MVT binary) +``` + +**WKT / EWKT strings** — short strings are shown in full; longer strings are truncated with a type annotation: + +``` +POINT (13 42) +MULTIPOLYGON (((... (WKT) +``` + +**CRS strings** — short authority strings (`EPSG:4326`, `ESRI:54008`) are shown in full; long WKT CRS definitions are truncated: + +``` +PROJCS["British National Grid", ...] (CRS) +``` + +**Cell width** — output cells are capped at approximately 60 characters. A longer value is truncated with `...` and annotated with its type. + +**Identical-across-tier values** — when all four tabs produce the same result (e.g. `EPSG:4326` from `st_crs`), each tab shows that value identically with the same annotation. Genuine tier differences are called out in a labeled `:::note`. + +### CRS-family functions return BINARY + +`gbx_st_setcrs` and `gbx_st_transformcrs` return `BINARY` (WKB or EWKB) in **all input encodings and both tiers** — a STRING (WKT/EWKT) input still yields BINARY output. Only `gbx_st_crs` returns `STRING`. The clickable pointer to the CRS contract: [Coordinate Reference Systems](./coordinate-reference-systems). + +### Light tab for light-only UDTF generators + +`gbx_st_asmvt_pyramid`, `gbx_st_triangulate`, `gbx_st_interpolateelevationbbox`, and `gbx_st_interpolateelevationgeom` are Python UDTFs in the lightweight tier — they have **no Python DataFrame Column form**. The Python (light) tab for these functions shows: + +```python +spark.sql("SELECT t.* FROM , LATERAL gbx_(...) t") +``` + +This is the same invocation as the SQL tab, driven via Python. SQL `LATERAL` works for **both** tiers; the Python DataFrame Column form (`.select(vx.fn(...))`) is heavyweight-only for these generators. + +--- + ## Vector tile output {#vector-tile-output} Encode features into [Mapbox Vector Tile (MVT)](https://github.com/mapbox/vector-tile-spec) protobufs. Pair the per-tile MVT bytes with [`gbx_pmtiles_agg`](./pmtiles-functions#pmtiles_agg) or the [PMTiles writer](../writers/pmtiles) to publish a vector pyramid as a single `.pmtiles` archive targeting MapLibre, deck.gl, Mapbox GL JS, or Felt. @@ -92,21 +209,19 @@ This means downstream clients (MapLibre GL JS, Mapbox GL JS, deck.gl) receive nu ### st_asmvt - - - - + Aggregator that encodes a group of features into a single MVT protobuf blob for one -`(z, x, y)` tile. Each call to `groupBy(z, x, y).agg(vx.st_asmvt(...))` produces the -MVT bytes for exactly one tile. +`(z, x, y)` tile. Each `groupBy(z, x, y).agg(vx.st_asmvt(...))` call produces the +MVT bytes for exactly one tile. Both tiers expose the same grouped-aggregate API and +produce identical output bytes — a one-line swap between `pyvx` and `vectorx`. -**Signature:** `st_asmvt(geom_wkb, attrs, layer_name) → BINARY` +**Signature:** `st_asmvt(geom, attrs, layer_name) → BINARY` **Parameters:** -- `geom_wkb` (`BINARY`) — Feature geometry as WKB in **tile-local coordinates** (pixel - space, 0..extent). Clip and project each feature to the tile coordinate system upstream +- `geom` (`BINARY`) — Feature geometry (WKB, EWKB, WKT, or EWKT) in **tile-local coordinates** (pixel + space, `0..extent`). Clip and project each feature to the tile coordinate system upstream before calling this aggregator. - `attrs` (`STRUCT<...>`) — Per-feature attribute struct. Integer, float, boolean, and string fields are encoded with native MVT protobuf value types. @@ -115,132 +230,56 @@ MVT bytes for exactly one tile. **Returns:** `BINARY` — the MVT protobuf for one tile layer. Feed directly into `gbx_pmtiles_agg` or the [PMTiles Writer](../writers/pmtiles). -**PySpark:** - -```python -from databricks.labs.gbx.pyvx import functions as vx -from pyspark.sql import functions as F - -vx.register(spark) - -# features_df: (z, x, y, geom_wkb BINARY in tile-local coords, name STRING, id LONG) -tiles_df = ( - features_df - .groupBy("z", "x", "y") - .agg( - vx.st_asmvt( - F.col("geom_wkb"), - F.struct(F.col("name"), F.col("id")), - "roads", # layer name — plain string becomes a literal Column - ).alias("mvt_bytes") - ) -) -# tiles_df: (z INT, x INT, y INT, mvt_bytes BINARY) -``` - -**SQL** (after `vx.register(spark)` in the same session): - -```sql -SELECT - z, x, y, - gbx_st_asmvt(geom_wkb, struct(name, id), 'roads') AS mvt_bytes -FROM features_with_tile_coords -GROUP BY z, x, y -``` - - - - - - -Aggregator that encodes a group of features into a single MVT protobuf blob for one `(z, x, y)` tile. - -**Signature:** `st_asmvt(geomWkb: Column, attrs: Column, layerName: Column): Column` - -**Parameters:** -- `geomWkb` (`BINARY`) — Feature geometry in **tile-local coordinates** as WKB. Compose any `ST_Intersection` against the tile envelope and coordinate translation upstream. -- `attrs` (`STRUCT<...>`) — Per-feature attributes. Integer, long, double, and boolean fields are encoded as native MVT value types; other types are encoded as strings. -- `layerName` (`STRING`) — MVT layer name. - -**Returns:** `BINARY` — the MVT protobuf bytes for one layer of the tile. - -Typical use: `GROUP BY (z, x, y)` after composing tile-local coordinates upstream, so each group becomes one tile. +**Composability:** The `BINARY` output is the natural input to `gbx_pmtiles_agg` for packaging +multiple `(z, x, y)` tiles into a single PMTiles file. -**SQL:** - - - -**PySpark:** - -```python -from databricks.labs.gbx.vectorx import functions as vx -from pyspark.sql.functions import col, struct - -df.groupBy("z", "x", "y").agg( - vx.st_asmvt(col("geom_wkb"), struct(col("name"), col("id")), "roads").alias("mvt") -) -``` - -**Composability:** The `BINARY` output is the natural input to `gbx_pmtiles_agg` for packaging multiple `(z, x, y)` tiles into a single PMTiles file. - -**Limitations in 0.4.0:** -- Caller composes any `ST_Simplify` upstream. -- Caller composes tile-coordinate transform upstream. - - - + --- ### st_asmvt_pyramid - - - - + -Generator (Python UDTF) that explodes one input feature into one row per intersecting -`(z, x, y)` tile across a zoom range, with MVT bytes already encoded in each row. The -per-tile clip and coordinate transform happen inside the function — no upstream -`ST_Intersection` is needed. +Generator that explodes one input feature into one row per intersecting `(z, x, y)` tile +across a zoom range, with MVT bytes already encoded in each row. The per-tile clip and +coordinate transform happen inside the function — no upstream `ST_Intersection` is needed. -The output schema `(z, x, y, mvt_bytes)` is identical to the heavyweight generator, -so SQL pipelines built against either tier are interchangeable. In the lightweight -tier this generator is invoked only via SQL `LATERAL`; it has no Python DataFrame -Column form (that form is heavy-only). +Input geometry must be in **EPSG:4326 lon/lat**. The function handles per-tile projection +internally. `max_z ≤ 20`; total tile count across the zoom range capped at 10⁶. -**Signature (SQL):** `gbx_st_asmvt_pyramid(geom_wkb, attrs, min_z, max_z, layer_name, extent)` +**Signature:** `gbx_st_asmvt_pyramid(geom, attrs, min_z, max_z, [layer_name], [extent])` **Parameters:** -- `geom_wkb` (`BINARY`) — Feature geometry as WKB in **EPSG:4326 lon/lat**. The UDTF - clips each feature to every intersecting tile and transforms to tile-local coordinates - internally. +- `geom` (`BINARY`) — Feature geometry (WKB, EWKB, WKT, or EWKT) in **EPSG:4326 lon/lat**. - `attrs` (`STRUCT<...>`) — Per-feature attributes. Same native-typed encoding as `st_asmvt`. - `min_z`, `max_z` (`INT`) — Inclusive zoom range (`0..20`). - `layer_name` (`STRING`, optional) — MVT layer name; defaults to `"layer"`. - `extent` (`INT`, optional) — MVT tile extent in pixels; defaults to `4096`. -**Returns:** One row per intersecting tile — schema `(z INT, x INT, y INT, mvt_bytes BINARY)`. - -**Caps:** `max_z ≤ 20`; total tile count across the zoom range capped at 10⁶. - -**SQL — LATERAL table function:** - +:::note SQL invocation form differs between tiers +The **lightweight** tier is a Python UDTF registered via `spark.udtf.register` — invoke it with `LATERAL` (SQL standard table-function syntax): ```sql --- After vx.register(spark): -SELECT t.* -FROM features, - LATERAL gbx_st_asmvt_pyramid( - geom_wkb, struct(name, id), 0, 12, 'roads', 4096 - ) t +FROM features, LATERAL gbx_st_asmvt_pyramid(geom_wkb, attrs, 0, 12, 'layer', 4096) t +-- Output columns: t.z, t.x, t.y, t.mvt_bytes (direct) ``` +The **heavyweight** tier is a JVM generator expression — invoke it with `LATERAL VIEW` (Hive-style): +```sql +FROM features LATERAL VIEW gbx_st_asmvt_pyramid(geom_wkb, attrs, 0, 12, 'layer', 4096) t AS tile +-- Output column: t.tile.z, t.tile.x, t.tile.y, t.tile.mvt_bytes (struct wrapper) +``` +The Python DataFrame Column form (`vx.st_asmvt_pyramid(col(...), ...)`) is **heavyweight-only**; the lightweight UDTF has no Column API. +::: -`LATERAL` materializes one `(z, x, y, mvt_bytes)` row per tile the feature intersects. -Each executor processes its partition's features in parallel — tile fan-out is distributed -across the cluster, not serialized on the driver. - -**Full pipeline — vector pyramid to PMTiles:** +**Full pipeline — vector pyramid to PMTiles (lightweight):** ```python from databricks.labs.gbx.pyvx import functions as vx @@ -275,71 +314,100 @@ with open("/tmp/roads.pmtiles", "wb") as fh: For larger pyramids that exceed the Spark cell limit, use the [PMTiles Writer](../writers/pmtiles) (`pmtiles_gbx` for the lightweight tier) instead of `pmtiles_agg`. - - + - +--- -Generator that explodes one feature into one row per intersecting `(z, x, y)` tile across a zoom range, with the MVT bytes already encoded per tile. Pairs with `gbx_rst_xyzpyramid` (the raster sibling) and feeds directly into `gbx_pmtiles_agg`. +## Triangulation and elevation -**Signature:** `st_asmvt_pyramid(geomWkb: Column, attrs: Column, minZoom: Column, maxZoom: Column, layerName: Column): Column` +These generators build a Delaunay triangulated irregular network (TIN) from Z-valued mass points and optional breaklines, then either expose the triangles directly or sample the surface on a regular grid to produce elevation points. Useful for surface modeling, DTM/DEM derivation, and elevation sampling from survey point clouds. All three are available in **both** tiers; breaklines are honored in both. -**Parameters:** -- `geomWkb` (`BINARY`) — Feature geometry in **EPSG:4326 lon/lat** as WKB. The function performs the per-tile clip and tile-local coordinate transform; no upstream `ST_Intersection` required. -- `attrs` (`STRUCT<...>`) — Per-feature attributes. Integer, long, double, and boolean fields are encoded as native MVT value types; other types are encoded as strings. -- `minZoom`, `maxZoom` (`INT`) — Inclusive zoom-level range (`0..20`). -- `layerName` (`STRING`) — MVT layer name (constant per call). -- `extent` (`INT`, optional) — MVT tile extent in pixels; default `4096` (MVT v2 standard). +### Triangulation modes: `constrained` vs `conforming` -**Returns:** One row per intersecting tile; each row's `tile` struct exposes `(z INT, x INT, y INT, mvt_bytes BINARY)`. Use `LATERAL VIEW` to materialize rows and pipe `mvt_bytes` into `gbx_pmtiles_agg`. +Each TIN function takes a trailing `mode` argument (default `'constrained'`): -**SQL:** +| Mode | Tiers | Behavior | +|---|---|---| +| `constrained` *(default)* | Lightweight **and** heavyweight | Constrained Delaunay triangulation. Breaklines are honored as forced edges with **no Steiner points** — the output vertex set is exactly your input mass points plus breakline vertices. Identical algorithm in both tiers, so the result is a seamless cross-tier swap. | +| `conforming` | Heavyweight only | JTS conforming-Delaunay triangulation: the mesh may insert additional **Steiner points** along breakline segments to satisfy the Delaunay property near constraints. Produces a smoother mesh around dense breaklines at the cost of extra vertices. | - +The lightweight tier raises `NotImplementedError` on `mode='conforming'` — it has no Steiner-point refinement. This is a **documented, intentional divergence**, analogous to the [H3 covering note](./gridx-functions) elsewhere in these docs: the two tiers agree exactly in the default (`constrained`) mode, and the heavyweight tier offers `conforming` as a deliberate opt-in superset. If you need a result that is byte-identical across tiers, stay on `constrained` (the default). -**PySpark:** +:::note Invocation surface +In the **lightweight** tier these TIN generators are PySpark UDTFs with **no Python DataFrame Column form** — invoke them via SQL `LATERAL` (e.g. `... , LATERAL gbx_st_triangulate(...) t`), the same pattern as `gbx_st_asmvt_pyramid`. In the **heavyweight** tier they are exposed as generator Columns (usable in `select(...)`) and via SQL `LATERAL VIEW`. SQL `LATERAL` works for **both** tiers; the Python DataFrame Column form is heavyweight-only for these generators. +::: -```python -from databricks.labs.gbx.vectorx import functions as vx -from pyspark.sql.functions import col, struct +### st_interpolateelevationbbox -df.select( - vx.st_asmvt_pyramid( - col("geom_wkb"), struct(col("name"), col("id")), 0, 8, "roads" - ).alias("t") -).select("t.tile.z", "t.tile.x", "t.tile.y", "t.tile.mvt_bytes") -``` + -**Composability:** Output rows compose directly with `gbx_pmtiles_agg` — group by `(z, x, y)`, aggregate `mvt_bytes` to produce a PMTiles blob with `tile_type = mvt`. For multi-feature tiles, pre-explode tile assignments and then `groupBy(z, x, y).agg(st_asmvt(...))` using the aggregator. +Builds a TIN from mass points and breaklines, then samples elevation on a regular pixel grid covering an **explicit bounding box**. Use this when you already know the output extent in absolute coordinates — for example, when snapping to a fixed tile extent or aligning with a raster grid. -**Limitations in 0.4.0:** -- Single-feature input per row. Multi-feature aggregation per tile requires the aggregator pattern above. -- `max_z <= 20`; total tile count across the zoom range capped at 10^6 (mirrors `gbx_rst_xyzpyramid`). -- Inputs must be in EPSG:4326; reproject upstream for other CRS. +**Signature:** `gbx_st_interpolateelevationbbox(points_array, breaklines_array, merge_tolerance, snap_tolerance, split_point_finder, xmin, ymin, xmax, ymax, width_px, height_px, srid, [mode])` - - +**Parameters:** +- `points_array` — Mass-point geometries with Z values. Accepts WKB/EWKB/WKT/EWKT. +- `breaklines_array` — Breakline geometries (or an empty array). +- `merge_tolerance` (`DOUBLE`) — Merge distance for coincident points. +- `snap_tolerance` (`DOUBLE`) — Snap distance to breakline vertices. +- `split_point_finder` (`STRING`) — Conforming-mesh strategy (e.g. `'NONENCROACHING'`). +- `xmin`, `ymin`, `xmax`, `ymax` (`DOUBLE`) — Bounding box corners in the coordinate reference system given by `srid`. +- `width_px`, `height_px` (`INT`) — Number of grid columns and rows. Together with the bbox dimensions these determine the cell size. +- `srid` (`INT`) — EPSG code of the bounding box coordinates (e.g. `27700` for British National Grid). +- `mode` (`STRING`, optional) — `'constrained'` (default, both tiers) or `'conforming'` (heavyweight only). + +**Generator:** Emits one row per in-hull grid cell (cells whose centers fall outside the TIN convex hull are dropped). The output schema column is `elevation_point` (`BINARY` WKB POINT Z). Use with SQL `LATERAL` to materialize the grid. + + --- -## Triangulation and elevation +### st_interpolateelevationgeom -These generators build a Delaunay triangulated irregular network (TIN) from Z-valued mass points and optional breaklines, then either expose the triangles directly or sample the surface on a regular grid to produce elevation points. Useful for surface modeling, DTM/DEM derivation, and elevation sampling from survey point clouds. All three are available in **both** tiers; breaklines are honored in both. + -### Triangulation modes: `constrained` vs `conforming` +Builds a TIN from mass points and breaklines, then samples elevation on a regular grid **anchored to a geometry origin** with explicit cell sizes. Use this when the grid must be defined relative to a known point — for example, when the grid origin comes from data (a survey control point) or when different rows need different grid placements. -Each TIN function takes a trailing `mode` argument (default `'constrained'`): +**Signature:** `gbx_st_interpolateelevationgeom(points_array, breaklines_array, merge_tolerance, snap_tolerance, split_point_finder, grid_origin, grid_cols, grid_rows, cell_size_x, cell_size_y, [mode])` -| Mode | Tiers | Behavior | -|---|---|---| -| `constrained` *(default)* | Lightweight **and** heavyweight | Constrained Delaunay triangulation. Breaklines are honored as forced edges with **no Steiner points** — the output vertex set is exactly your input mass points plus breakline vertices. Identical algorithm in both tiers, so the result is a seamless cross-tier swap. | -| `conforming` | Heavyweight only | JTS conforming-Delaunay triangulation: the mesh may insert additional **Steiner points** along breakline segments to satisfy the Delaunay property near constraints. Produces a smoother mesh around dense breaklines at the cost of extra vertices. | +**Parameters:** +- `points_array` — Mass-point geometries with Z values. Accepts WKB/EWKB/WKT/EWKT. +- `breaklines_array` — Breakline geometries (or an empty array). +- `merge_tolerance` (`DOUBLE`) — Merge distance for coincident points. +- `snap_tolerance` (`DOUBLE`) — Snap distance to breakline vertices. +- `split_point_finder` (`STRING`) — Conforming-mesh strategy (e.g. `'NONENCROACHING'`). +- `grid_origin` — POINT geometry anchoring the top-left corner of the output grid. The output SRID is inherited from this geometry (encode as EWKB/EWKT to carry a non-zero SRID) — no separate `srid` argument. +- `grid_cols`, `grid_rows` (`INT`) — Number of grid columns and rows. +- `cell_size_x` (`DOUBLE`) — Horizontal cell size in the geometry's units (positive steps right). +- `cell_size_y` (`DOUBLE`) — Vertical cell size in the geometry's units. Pass a **negative** value to step downward (standard raster convention, e.g. `-10.0` for 10-unit cells stepping south). +- `mode` (`STRING`, optional) — `'constrained'` (default, both tiers) or `'conforming'` (heavyweight only). -The lightweight tier raises `NotImplementedError` on `mode='conforming'` — it has no Steiner-point refinement. This is a **documented, intentional divergence**, analogous to the [H3 covering note](./gridx-functions) elsewhere in these docs: the two tiers agree exactly in the default (`constrained`) mode, and the heavyweight tier offers `conforming` as a deliberate opt-in superset. If you need a result that is byte-identical across tiers, stay on `constrained` (the default). +**Generator:** Emits one row per in-hull grid cell. The output schema column is `elevation_point` (`BINARY` WKB POINT Z). Use with SQL `LATERAL` to materialize the grid. -:::note Invocation surface -In the **lightweight** tier these TIN generators are PySpark UDTFs with **no Python DataFrame Column form** — invoke them via SQL `LATERAL` (e.g. `... , LATERAL gbx_st_triangulate(...) t`), the same pattern as `gbx_st_asmvt_pyramid`. In the **heavyweight** tier they are exposed as generator Columns (usable in `select(...)`) and via SQL `LATERAL VIEW`. SQL `LATERAL` works for **both** tiers; the Python DataFrame Column form is heavyweight-only for these generators. -::: + + +--- ### st_triangulate @@ -347,91 +415,166 @@ In the **lightweight** tier these TIN generators are PySpark UDTFs with **no Pyt Builds a Delaunay TIN from mass-point geometries (with Z values) and optional breakline geometries, emitting one triangle polygon per row. Use this when you need the raw triangulation — e.g., to inspect mesh quality, clip triangles to an area of interest, or feed a custom sampler. -**Signature:** `gbx_st_triangulate(points, breaklines, mergeTolerance, snapTolerance, splitPointFinder, mode)` +**Signature:** `gbx_st_triangulate(points_array, breaklines_array, merge_tolerance, snap_tolerance, split_point_finder, [mode])` **Parameters:** -- `points` — Array column of point geometries with Z values (the mass points that define the surface). Accepts WKB/EWKB/WKT/EWKT. -- `breaklines` — Array column of linestring geometries that the mesh must honor as edges (e.g., ridge lines, drainage channels). Pass an empty array if no breaklines are needed. -- `mergeTolerance` (`DOUBLE`) — Distance below which coincident points are merged before triangulation. -- `snapTolerance` (`DOUBLE`) — Distance within which points are snapped to breakline vertices. -- `splitPointFinder` (`STRING`) — Conforming-mesh refinement strategy. Use `'NONENCROACHING'` for a mesh that avoids encroaching on breakline segments; `'MIDPOINT'` is also valid (heavyweight `conforming` mode). +- `points_array` — Array column of point geometries with Z values (the mass points that define the surface). Accepts WKB/EWKB/WKT/EWKT. +- `breaklines_array` — Array column of linestring geometries that the mesh must honor as edges (e.g., ridge lines, drainage channels). Pass an empty array if no breaklines are needed. +- `merge_tolerance` (`DOUBLE`) — Distance below which coincident points are merged before triangulation. +- `snap_tolerance` (`DOUBLE`) — Distance within which points are snapped to breakline vertices. +- `split_point_finder` (`STRING`) — Conforming-mesh refinement strategy. Use `'NONENCROACHING'` for a mesh that avoids encroaching on breakline segments; `'MIDPOINT'` is also valid (heavyweight `conforming` mode). - `mode` (`STRING`, optional) — `'constrained'` (default, both tiers) or `'conforming'` (heavyweight only). See [Triangulation modes](#triangulation-modes-constrained-vs-conforming) above. **Generator:** Emits one row per output triangle. Use with SQL `LATERAL` to materialize the triangles; the output schema column is `triangle` (`BINARY` WKB polygon). -**SQL** (works in both tiers after `vx.register(spark)`): + - +:::note `mode='conforming'` is heavyweight-only +The `'constrained'` mode (default) is available in **both** tiers. The `'conforming'` mode — which inserts Steiner points along breakline segments for a smoother mesh — is **heavyweight-only**: the lightweight `pyvx` tier raises `NotImplementedError` if you pass `mode='conforming'`. The examples on all tabs use the default `'constrained'` mode. +::: -**PySpark** (heavyweight DataFrame Column form): +--- -```python -from databricks.labs.gbx.vectorx import functions as vx -from pyspark.sql import functions as F +## Coordinate reference systems {#crs} -df.select( - vx.st_triangulate( - F.col("masspoints"), F.col("breaklines"), 0.01, 0.01, "NONENCROACHING" - ).alias("t") -).select("t.triangle") -``` +Read, stamp, and reproject a geometry's CRS by **CRS string** — an ESRI code, a WKT definition, or a PROJ4 string, not only an EPSG integer. These complement the product's built-in `ST_SRID` / `ST_SetSRID` / `ST_Transform`, which take an integer SRID: use the built-ins when an EPSG code is all you need, and these when the CRS can only be named as a string. See [Coordinate Reference Systems](./coordinate-reference-systems) for the SRID-vs-CRS-string model shared with RasterX. + +All three are available in **both** tiers under the same names, so a CRS query is a one-line tier swap. + +### Shared contracts + +**Geometry input** — every geometry argument accepts **WKB, EWKB, WKT, and EWKT**. WKB/WKT carry no SRID; EWKB/EWKT carry one. + +**SQL output is always BINARY.** `gbx_st_setcrs` and `gbx_st_transformcrs` return `BINARY` (WKB/EWKB) whichever encoding the geometry argument arrived in — a STRING geometry input still yields BINARY. One function has one declared return type: an input-dependent return type cannot be used in a view or any fixed schema, and WKB is how the rest of `gbx_st_*` and the built-in `ST_*` functions exchange geometries. To read a CRS back as text, wrap the result in `gbx_st_crs`; to hand it to a built-in, `ST_GeomFromWKB` accepts it directly. `gbx_st_crs` returns `STRING`. + +**Errors are the exception, not the rule.** These functions degrade rather than fail a whole column: + +| Situation | Result | +|---|---| +| NULL geometry | NULL | +| NULL `target_crs` | NULL | +| Geometry has no resolvable source CRS (plain WKB/WKT, no `source_crs`) | input returned **unchanged** | +| Embedded SRID is in no registry (e.g. `999999`) | input returned **unchanged** | +| `source_crs` cannot be parsed | input returned **unchanged** | +| `target_crs` cannot be parsed | **raises** | +| `gbx_st_setcrs` given a CRS with no integer authority code | **raises** | + +**Z coordinates.** A geometry whose vertices *all* carry a finite Z keeps its Z. A genuinely 2D geometry stays 2D — no Z ordinate is invented. For a geometry where only *some* vertices carry a Z, the current behavior differs between the two operations, because only one of them touches coordinates: + +- `gbx_st_transformcrs` reprojects it as **2D**. Reprojecting a missing Z propagates it into X and Y and destroys the horizontal position, so dropping the Z is what keeps every X/Y correct. +- `gbx_st_setcrs` keeps the partial Z as-is, since stamping an SRID never moves coordinates. -In the lightweight tier, call the registered UDTF via SQL `LATERAL` (shown above) — there is no `vx.st_triangulate(...)` Column form. +A missing Z is never filled in with a substitute value such as `0`, which would be indistinguishable from a surveyed elevation downstream. Both tiers behave identically here, in every input encoding. + +:::caution Known limitations +These apply in **every** input encoding and on **both** tiers — they are properties of the operations, not of how you pass the geometry. + +- **Chaining `setcrs` → `transformcrs` on a partial-Z geometry yields a 2D result**, for the reason described just above: the reproject drops a partial Z. So a geometry that entered the chain with some elevations leaves it with none. +- **Reprojecting out and back is not bit-exact.** A round trip such as `EPSG:4326` → `EPSG:32633` → `EPSG:4326` returns `11.000000000000002` where it started at `11` — a floating-point artifact of the projection math, in the last decimal place or two. It does not compound meaningfully: further round trips stay in that same last-place range rather than drifting away. Compare reprojected coordinates with a tolerance, never for exact equality. +- **`st_setcrs` relabels without reprojecting**, so stamping a CRS whose units do not match the coordinates leaves the geometry mislabelled. A later `st_transformcrs` then transforms from the wrong CRS: projected metres tagged `EPSG:4326` come back as `Infinity` coordinates on the lightweight tier, and raise a projection error on the heavyweight tier. Use `st_transformcrs` when you want the coordinates moved. +- **Coordinates outside the target CRS's valid domain** behave the same way — for example a latitude of `100`, which does not exist. The lightweight tier returns `Infinity` coordinates; the heavyweight tier raises a projection error. Filter to the target CRS's area of use before reprojecting if your input may contain out-of-range coordinates. +- **M (measure) values are dropped.** Both tiers carry X, Y and Z only, so a `ZM` geometry comes back as `Z` with the measure gone, and a geometry carrying M but no Z comes back plain 2D — no `Z` is invented to fill the slot. This is worth noting because the product's own geometry type does persist M. +::: --- -### st_interpolateelevationbbox +### st_crs - + -Builds a TIN from mass points and breaklines, then samples elevation on a regular pixel grid covering an **explicit bounding box**. Use this when you already know the output extent in absolute coordinates — for example, when snapping to a fixed tile extent or aligning with a raster grid. +Returns the canonical CRS string for the geometry's embedded SRID, or `NULL`. -**Signature:** `gbx_st_interpolateelevationbbox(points, breaklines, mergeTolerance, snapTolerance, splitPointFinder, xmin, ymin, xmax, ymax, widthPx, heightPx, srid, mode)` +**Signature:** `gbx_st_crs(geom)` **Parameters:** -- `points` — Mass-point geometries with Z values. Accepts WKB/EWKB/WKT/EWKT. -- `breaklines` — Breakline geometries (or an empty array). -- `mergeTolerance` (`DOUBLE`) — Merge distance for coincident points. -- `snapTolerance` (`DOUBLE`) — Snap distance to breakline vertices. -- `splitPointFinder` (`STRING`) — Conforming-mesh strategy (e.g. `'NONENCROACHING'`). -- `xmin`, `ymin`, `xmax`, `ymax` (`DOUBLE`) — Bounding box corners in the coordinate reference system given by `srid`. -- `widthPx`, `heightPx` (`INT`) — Number of grid columns and rows. Together with the bbox dimensions these determine the cell size. -- `srid` (`INT`) — EPSG code of the bounding box coordinates (e.g. `27700` for British National Grid). -- `mode` (`STRING`, optional) — `'constrained'` (default, both tiers) or `'conforming'` (heavyweight only). +- `geom` — Geometry. Accepts WKB/EWKB/WKT/EWKT. -**Generator:** Emits one row per in-hull grid cell (cells whose centers fall outside the TIN convex hull are dropped). The output schema column is `elevation_point` (`BINARY` WKB POINT Z). Use with SQL `LATERAL` to materialize the grid. +**Returns:** `STRING` — the authority string (`'EPSG:4326'`, `'ESRI:54008'`, …), or `NULL` for a plain WKB/WKT geometry with no embedded SRID, a NULL input, or an SRID in no known registry. -**SQL** (works in both tiers): +An SRID is classified against the authoritative PROJ registries, so an ESRI-range code comes back as `ESRI:` rather than being mislabelled `EPSG:`. - + --- -### st_interpolateelevationgeom +### st_setcrs - + -Builds a TIN from mass points and breaklines, then samples elevation on a regular grid **anchored to a geometry origin** with explicit cell sizes. Use this when the grid must be defined relative to a known point — for example, when the grid origin comes from data (a survey control point) or when different rows need different grid placements. +Stamps a CRS on a geometry **without reprojecting** — it relabels, it does not move coordinates. The counterpart to the built-in `ST_SetSRID`, taking a CRS string instead of an integer. -**Signature:** `gbx_st_interpolateelevationgeom(points, breaklines, mergeTolerance, snapTolerance, splitPointFinder, gridOrigin, gridCols, gridRows, cellSizeX, cellSizeY, mode)` +**Signature:** `gbx_st_setcrs(geom, crs)` **Parameters:** -- `points` — Mass-point geometries with Z values. Accepts WKB/EWKB/WKT/EWKT. -- `breaklines` — Breakline geometries (or an empty array). -- `mergeTolerance` (`DOUBLE`) — Merge distance for coincident points. -- `snapTolerance` (`DOUBLE`) — Snap distance to breakline vertices. -- `splitPointFinder` (`STRING`) — Conforming-mesh strategy (e.g. `'NONENCROACHING'`). -- `gridOrigin` — POINT geometry anchoring the top-left corner of the output grid. The output SRID is inherited from this geometry (encode as EWKB/EWKT to carry a non-zero SRID) — no separate `srid` argument. -- `gridCols`, `gridRows` (`INT`) — Number of grid columns and rows. -- `cellSizeX` (`DOUBLE`) — Horizontal cell size in the geometry's units (positive steps right). -- `cellSizeY` (`DOUBLE`) — Vertical cell size in the geometry's units. Pass a **negative** value to step downward (standard raster convention, e.g. `-10.0` for 10-unit cells stepping south). -- `mode` (`STRING`, optional) — `'constrained'` (default, both tiers) or `'conforming'` (heavyweight only). +- `geom` — Geometry. Accepts WKB/EWKB/WKT/EWKT. +- `crs` (`STRING`) — Target CRS. An authority string (`'EPSG:4326'`, `'ESRI:54008'`) or an int-castable string / integer (`32633`, `'32633'`), which behaves like `ST_SetSRID(geom, 32633)`. -**Generator:** Emits one row per in-hull grid cell. The output schema column is `elevation_point` (`BINARY` WKB POINT Z). Use with SQL `LATERAL` to materialize the grid. +**Returns:** `BINARY` — EWKB with the new SRID embedded. Coordinate **values** are preserved exactly, to the last decimal place — no reprojection and no rounding. (The output bytes are not identical to the input's: embedding the SRID is what makes it EWKB.) -**SQL** (works in both tiers): +**Raises** when `crs` has no integer authority code, because a geometry can store only an integer SRID. That covers: - +- **authority-less definitions** — a raw `PROJCS[...]` WKT or a PROJ4 string such as `'+proj=utm +zone=33 +datum=WGS84'`. PROJ's fuzzy matcher *would* pair that PROJ4 string with `EPSG:32633` at partial confidence, but a geometry SRID is an exact identity claim — a guess is never silently written into one. Use `gbx_st_transformcrs` if you want the coordinates in that CRS. + + This distinction is not academic: a PROJ4 string that resembles a registry CRS is not necessarily equivalent to it. One that omits a datum shift (`+towgs84=0,0,0,0,0,0,0`) can place coordinates **hundreds of metres** from the EPSG code it superficially matches. GeoBrix treats such a definition as its own CRS throughout — including when selecting the transformation used to reproject — rather than silently substituting the near-match. If you want the registry CRS, name it explicitly (`'EPSG:28992'`). +- **non-numeric authority codes** — `'OGC:CRS84'`, `'IGNF:LAMB93'`: real, resolvable CRSes whose code simply is not an integer. + + + +--- + +### st_transformcrs + + + +Reprojects a geometry's coordinates into `target_crs`. The counterpart to the built-in `ST_Transform`, taking a CRS string instead of an integer — so an ESRI code, a WKT definition, or a PROJ4 string can be a target. + +**Signature:** `gbx_st_transformcrs(geom, target_crs [, source_crs])` + +**Parameters:** +- `geom` — Geometry. Accepts WKB/EWKB/WKT/EWKT. +- `target_crs` (`STRING`) — CRS to reproject into: authority string, int-castable string/integer, WKT, or PROJ4. +- `source_crs` (`STRING`, optional) — CRS the input is in. Used **only** for a plain (SRID-less) geometry; a geometry that carries an embedded SRID ignores this argument, so a mixed column is safe. + +**Returns:** `BINARY` — the reprojected geometry. + +**Which SRID comes out** follows the **target**: + +| `target_crs` | Example | Output | +|---|---|---| +| Has an integer authority code | `'EPSG:32633'`, `'ESRI:54008'`, `32633` | EWKB with that SRID stamped — a plain input is upgraded to carry one | +| Has no integer authority code | raw `PROJCS[...]` WKT, `'+proj=utm +zone=33 …'`, `'OGC:CRS84'` | plain WKB: coordinates reprojected, and the now-**stale** source SRID **cleared** — leaving it would label the geometry with a CRS it is no longer in | + +Source CRS resolution order: the geometry's embedded SRID first, then `source_crs`, and if neither resolves the geometry is returned unchanged. + + --- @@ -455,28 +598,13 @@ A scalar function in **both** tiers (same registered name, same output bytes), s - **SRID is applied separately at ingestion.** The output is plain WKB and carries no SRID; assign the CRS when you read it back, e.g. `ST_GeomFromWKB(gbx_st_legacyaswkb(geom_legacy), 27700)`. - **M (measure) values are out of scope** for this conversion. -## Common setup - -Run this once before the examples below. It registers VectorX so you can use `st_legacyaswkb` in Python and `gbx_st_legacyaswkb` in SQL. - - - -**Python:** - - - -**SQL:** - - - -**Quick Start example** (point geometry round-trip): - - --- diff --git a/docs/docs/api/virtual-tiles.mdx b/docs/docs/api/virtual-tiles.mdx new file mode 100644 index 000000000..4b7ce5cd7 --- /dev/null +++ b/docs/docs/api/virtual-tiles.mdx @@ -0,0 +1,204 @@ +--- +sidebar_position: 5 +sidebar_label: Virtual Tiles +title: Virtual Tiles +--- + +# Virtual Tiles + +Reading a multi-gigabyte raster the naïve way pulls the whole image into memory — +and fanning it into pieces multiplies that cost until an executor runs out of +memory. GeoBrix avoids this with **virtual tiles**: each row of the DataFrame +carries a *reference* to a window of a raster — a path plus a pixel window — +instead of the pixels themselves. Pixels are read lazily, one window at a time, +only when an operation actually needs them. + +This page explains the virtual-tile model, why it dissolves the +ingest-memory problem, and how tiles move between **virtual** (bytes-free) and +**materialized** (bytes-in-hand) states across the read → operate → write +lifecycle. + +![GeoBrix virtual & materialized tiles lifecycle — sources (striped/tiled GeoTIFF, COG, NetCDF, tables) optionally prepared to COGs, read via distributed readers into virtual or materialized tiles, operated on by rst_* functions that can stay virtual or materialize, and written back to files or Databricks SQL tables](../../../resources/images/diagrams/rasterx/virtual-tiles-lifecycle.png) + +## What a virtual tile is + +Every GeoBrix raster tile is one typed struct — the **v2 tile struct** — shared +by both execution tiers: + +``` +struct< + cellid: bigint, -- grid cell id (nullable) + raster: binary, -- the raster payload (NULL when virtual) + path: string, -- source path (set when virtual) + window: struct, -- the pixel window + clip_polygon: binary, -- optional clip geometry (WKB) + clip_crs: string, -- CRS for clip_polygon + crs: string, -- working/target CRS + metadata: map, -- driver, extension, format keys + path_mode: string -- storage mode: null, 'external', or 'managed' +> +``` + +A tile is **virtual** when `raster` is `NULL` and `path` + `window` are set — it +is a bytes-free reference to a window of a raster on durable storage. A tile is +**materialized** when `raster` carries the encoded bytes. The struct is identical +either way, so a DataFrame can hold a mix, and a tile can move between the two +states without changing shape. See **[Tile Structure](./tile-structure)** for a +field-by-field reference to this struct. + +### Reference vs. instruction + +The `window` / `clip_polygon` / `clip_crs` / `crs` fields mean different things +depending on the tile's state, and this is the key to reasoning about them: + +- On a **materialized** tile they are **provenance** — a record of what has + *already been applied* to produce the bytes in `raster`. +- On a **virtual** tile they are **instructions** — pending operations that are + applied when the tile is read (staged from `path`, the `window` extracted, the + clip and CRS applied), producing the pixels on demand. + +## Why it matters + +Carrying references instead of pixels is what makes large-raster ingest scale: + +- **Bytes-free rows.** A virtual tile row is roughly **100 bytes** (a path and a + four-integer window). The materialized bytes it stands in for are **148–527 KB** + per tile — so virtual rows are on the order of **1,400–5,000× smaller**. At + ingest you hold N tiny descriptor rows instead of hundreds of MiB of encoded + tiles, and the accumulation that causes Serverless out-of-memory failures + simply does not happen. +- **Windowed, parallel reads.** Readers fan a source out across the cluster — one + window per tile, one tile per task — and each window is a small ranged read + against a [Cloud-Optimized GeoTIFF](./large-rasters#raster-formats-striped-tiled-and-cog). + No executor ever holds the whole image, and there is no driver-side `collect`. +- **Lazy composition.** Deferrable operations chain on virtual tiles without ever + touching pixels — the DataFrame keeps carrying references until an operation + genuinely needs the data. + +This is the cloud-native raster model — don't move pixels; read windows on +demand — expressed as a Spark DataFrame. + +## The lifecycle + +The diagram above traces the four stages: + +1. **Source.** Striped GeoTIFFs, tiled GeoTIFFs, COGs, NetCDFs, and tabular + tile-struct columns are all usable **as-is**. File formats can *optionally* be + standardized into COGs first with + [`prepare_cogs`](./large-rasters#prepare_cogs--driver-orchestrated-preparation), + which makes windowed reads cheap (a striped source can inflate a single window + ~570× versus a clean tiled block). Optimization is a choice, not a gate. +2. **Distributed load.** A reader (`cog_gbx`, `gdal`, `netcdf_gbx`, …) partitions + the source across executors and emits tile rows — the lightweight raster readers + (`raster_gbx`, `gtiff_gbx`, `cog_gbx`) emit **virtual tiles by default**; + pass `.option("virtualTiles", "false")` to get materialized bytes instead. Selection + options (`tileSize`, `overlapPercent`, `clipPolygons`, …) shape the windowing. +3. **Operate.** Any `rst_*` function accepts either a virtual or a materialized + tile. The output shape is your choice — see below. +4. **Write.** Persist to files (COG, GeoTIFF, NetCDF, …) with a writer, or save a + tile DataFrame to a **Databricks SQL table**. + +## Virtual Tiles + FILE + +FILE is a [Databricks data type](https://docs.databricks.com/aws/en/pyspark/reference/file-type) that provides **governed access to files on compute without a FUSE mount**. GeoBrix detects FILE availability and uses it when present. If FILE is not available, virtual tiles work via the FUSE path unchanged. + +When FILE is available (Databricks Runtime 19 dedicated clusters), GeoBrix: + +- Stamps each virtual tile row with a `path_mode` field: `"external"` or `"managed"` (see [Tile Structure](./tile-structure)). +- Opens the file via `fref.open()` (byte-range stream) rather than through the FUSE mount. +- Falls back gracefully to the FUSE path if the FILE feature-detect fails. + +**Key point:** A `FileRef` is minted and consumed within each tile operation, then discarded. It is never stored as a DataFrame column and does not affect the tile struct's public shape. + +### Read performance with FILE + +The decisive factor for virtual-tile read performance is **open amortization** — opening a source raster once and reading many windows from that handle, rather than once per tile. Under the grouped executor pattern (per-partition LRU of open stream handles), byte-range stream reads are **10–290× faster** than FUSE for windowed COG reads. Without amortization (per-tile-open), the per-open cost of a stream scales with file size and can become prohibitive. See [Virtual tile read performance](./performance#virtual-tile-read-performance) for the full benchmark data, handling rules, and write-layout guidance. + +### Supported environments + +- **Databricks Runtime 19 dedicated clusters** (single-user, with `fileReferenceCreationMode=MANAGED` in cluster config): FILE engages where the runtime feature-detect succeeds. Where it does not, tiles use FUSE automatically. +- **Databricks Runtime 19 on Serverless:** coming soon. +- **Local development, CI, Serverless Compute (today), DBR 17/18:** virtual tiles use the traditional FUSE + rasterio path — correct and unchanged. + +## Reader selection surface + +The lightweight readers expose the windowing surface as read options: + +- **`virtualTiles`** — emit bytes-free virtual tiles (`true`, the default) or + materialized tiles (`false`). +- **`tileSize`** — regular tiling grid; **`overlapPercent`** — overlap between + adjacent tiles so per-tile operations don't clip features at the edges. +- **`clipPolygons`** / **`clipCrs`** — emit only the tile(s) intersecting each + polygon; **`windows`** — explicit pixel windows. + +See the [`cog_gbx` reader](../readers/cog) and the +[Readers overview](../readers/overview) for the full option reference — this page +does not duplicate them. + +:::tip Many-file directories +When a directory contains thousands of small tiles, loading it with `.load(dir)` +incurs planning overhead even with virtual tiles: the reader still walks the directory +and opens each header to compute `window` dimensions. Measured on a 10,002-file corpus, +a pre-computed manifest drops plan time from **1.357 s to 0.037 s — about 37× faster**. +Use the `manifest` or `tilesTable` reader option to supply pre-computed tile paths and +windows, reducing planning to a single file or table read regardless of tile count. +See [Raster Reader performance](../readers/raster#loading-many-small-files) and +[Benchmarking → Reader plan-time listing](./benchmarking#reader-plan-time-listing). +::: + +## Operating on tiles: your choice of output + +Every lightweight tile-returning `rst_*` function takes three optional +force-output parameters: + +- **`virtualize_dir`** — write the produced tile to a durable path and hand back a + **virtual** row (bytes-free), so a chain stays light after a pixel-producing op. +- **`virtualize_prefix`** — an optional filename prefix to deconflict outputs. +- **`materialize`** — force raster bytes into the row. + +The default is automatic: reference/passthrough operations stay virtual, and +pixel-producing operations materialize — but you can always ask for the other. +For the full rule (which operations are free on virtual tiles and which +materialize), see the +[Virtual↔materialized advice](./execution-tiers#virtual-materialized-advice) on +the Execution Tiers page. + +### Instructions that stay virtual + +A few cheap, common operations record an **instruction** on a virtual tile instead +of reading pixels — the tile stays bytes-free and the instruction is applied on the +next read (alongside the window and any clip/reproject): + +- `rst_initnodata` — set the NoData value +- `rst_setsrid` — relabel the CRS (assign an EPSG code; not a reproject) +- `rst_band` — select a band + +They accumulate: chain them on a virtual tile and all apply together when the tile is +finally read (e.g. at tessellation). Pass `materialize=True` (or `virtualize_dir`) to +apply them immediately and produce bytes. + +## Tiers: light tiles vs. heavy tiles + +**The lightweight tier is for light (virtual) raster tiles; the heavyweight tier +is for heavy (binary) raster tiles.** + +- The **lightweight tier** (`pyrx`) generates and operates on virtual tiles, and + materializes them on demand. Virtual tiles are a lightweight-tier capability. +- The **heavyweight tier** (`rasterx`) accepts both v1 and v2 **materialized** + tiles as input and always emits the v2 tile struct. It operates only on + materialized tiles: a virtual tile passed to a heavyweight function raises a + clear error telling you to materialize it first (call the lightweight function + with `materialize=True`, or write it out and read it back). Writing is itself a + materialization boundary. + +## See also + +- [Large Rasters](./large-rasters) — preparing COGs at scale (`prepare_cogs`, + `cog_gbx` `driverMode`) and the format/memory details behind windowed reads. +- [VRT & Mosaics](./vrt-mosaic) — a `.vrt` index over mini-COG tiles expands into + one virtual tile row per member; `rst_*` functions run per-tile unchanged. +- [Execution Tiers](./execution-tiers) — the full virtual↔materialized taxonomy + and the light→heavy bridge. +- [COG reader (`cog_gbx`)](../readers/cog) and [Readers overview](../readers/overview) + — the windowed-read option reference. +- [Writers overview](../writers/overview) — persisting tiles to files. diff --git a/docs/docs/api/vizx-raster.mdx b/docs/docs/api/vizx-raster.mdx index 29e5d17d9..f2638a079 100644 --- a/docs/docs/api/vizx-raster.mdx +++ b/docs/docs/api/vizx-raster.mdx @@ -55,6 +55,44 @@ plot_file("/Volumes/main/geobrix_samples/geobrix-examples/nyc/dem.tif") plot_file("/Volumes/main/my_schema/my_vol/sentinel_stack.tif", composite="depth") ``` +### `plot_mosaic` + +```python +plot_mosaic(vrt, *, bbox=None, bbox_crs=None, max_pixels=2000, resampling="bilinear", + show_cells=False, fig_w=10, fig_h=10, composite="auto", bands=None, + stretch="perband", fill=None, emphasis="blend", debug_mode=1) -> None +``` + +Render a VRT mini-COG mosaic (produced by the `cog_gbx` writer with `vrtMosaic=true`) as a single georeferenced image. `vrt` may be a path to a `.vrt` file or a directory containing exactly one `.vrt` — the directory form is the natural result of a `cog_gbx` write. All remaining parameters (`composite`, `bands`, `stretch`, `fill`, `emphasis`) behave identically to [`plot_raster`](#plot_raster) and [`plot_file`](#plot_file). Returns `None`. + +:::note `max_pixels` is a read ceiling, not an upsampling floor +GDAL selects each member's internal overview tier (or block-streams a single decimated read of the base level) so that the decoded pixel count never exceeds `max_pixels` on either axis. This bounds **peak RAM to `max_pixels²` regardless of mosaic size** — a 10,000-cell mosaic costs no more memory than a 4-cell one. It never upsamples: zooming into a viewport backed by an 800-pixel COG yields at most 800 real pixels even with `max_pixels=2000`. An overview-free COG (e.g. written by the DGGS writer without `--overviews`) still renders via on-the-fly decimation — you always get *something*, never a blank plot. +::: + +**`bbox` and `bbox_crs`** — pass an optional `(minx, miny, maxx, maxy)` viewport to zoom into a sub-region. Coordinates are interpreted in `bbox_crs` if supplied, or in the mosaic's own CRS otherwise. `bbox_crs` accepts any CRS form recognised by GeoBrix: an integer SRID (`4326`), an authority string (`"EPSG:4326"`, `"ESRI:54008"`), a WKT string, or a PROJ4 string. A `ValueError` is raised if the bbox does not intersect the mosaic. + +**`show_cells`** — h3 mosaics only: set `show_cells=True` to overlay the hex-cell boundary of each member tile as a white outline. Raises `ValueError` for non-h3 mosaics (`gridSystem != "h3"`). + +```python +from databricks.labs.gbx.vizx import plot_mosaic + +# Render a full h3 mosaic from its output directory. +# The directory contains mosaic.vrt and the per-cell mini-COGs. +plot_mosaic( + "/Volumes/main/geobrix_samples/my_schema/h3_mosaic/", + show_cells=True, + max_pixels=1024, +) + +# Viewport zoom: show only the Manhattan area using WGS84 bbox. +plot_mosaic( + "/Volumes/main/geobrix_samples/my_schema/h3_mosaic/mosaic.vrt", + bbox=(-74.02, 40.70, -73.93, 40.78), + bbox_crs="EPSG:4326", + max_pixels=2000, +) +``` + ### `plot_mask_layers` ```python @@ -93,6 +131,53 @@ plot_mask_layers( See the [H3 rasterize notebook](../notebooks/h3-rasterize) for a full worked example of building shared-canvas presence-mask layers. +## Rendering from a Spark DataFrame + +### `plot_tiles` + +```python +plot_tiles(df, tile_col="tile", *, mode="facet", limit=None, + fig_w=10, fig_h=10, max_pixels=2000, composite="auto", emphasis="blend") +``` + +Render raster tiles directly from a (filtered) Spark DataFrame — no manual materialize step required. Virtual and materialized tiles (v1 and v2 schema) are both handled automatically. + +**Parameters:** + +| Parameter | Type | Default | Description | +|---|---|---|---| +| `df` | `DataFrame` | — | A Spark DataFrame with a tile column. Filter it before calling — only the first `limit` rows are pulled to the driver. | +| `tile_col` | `str` | `"tile"` | Name of the tile struct column. | +| `mode` | `str` | `"facet"` | Rendering mode: `"facet"` (grid of thumbnails), `"first"` (single tile), or `"mosaic"` (stitch same-CRS tiles into one georeferenced image). | +| `limit` | `int` or `None` | mode default | Maximum rows to pull to the driver. Defaults to 25 (facet), 1 (first), or 64 (mosaic). For `facet` and `mosaic`, a `UserWarning` is issued when the DataFrame has more rows than `limit`. `first` silently renders the first tile with no warning — pass a filtered DataFrame to choose which tile. The entire DataFrame is never collected. | +| `fig_w` | `float` | `10` | Figure width in inches. | +| `fig_h` | `float` | `10` | Figure height in inches. | +| `max_pixels` | `int` | `2000` | Decimate above this longest-edge pixel count before rendering. | +| `composite` | `str` | `"auto"` | Band-composite mode (see [Raster rendering](#raster-rendering) above). | + +**Mode notes:** + +- `"facet"` — renders each tile as a separate panel in a grid. Works with any mix of CRS values. +- `"first"` — renders the first tile only. Works with any CRS. +- `"mosaic"` — merges tiles into a single georeferenced image via `rasterio.merge`. All tiles must share the same CRS and the same dtype/band count; a `ValueError` is raised on mismatch. The merged array is decimated to `max_pixels` before rendering. + +**Returns:** a matplotlib `Axes` for `"first"` and `"mosaic"`, a `Figure` for `"facet"`. + +```python +from databricks.labs.gbx.vizx import plot_tiles + +# Show the first 4 tiles from a GeoTIFF reader result as a 2×2 grid: +# The reader schema is (source, tile) — path and cellid are nested inside tile. +df = spark.read.format("gtiff_gbx").load("/Volumes/main/samples/sentinel/") +plot_tiles(df.filter("tile.cellid is not null"), mode="facet", limit=4) + +# Inspect a single tile quickly: +plot_tiles(df.filter("tile.path = '/Volumes/main/samples/sentinel/B04.tif'"), mode="first") + +# Stitch adjacent tiles from a single scene into one image (all must share CRS): +plot_tiles(scene_df, mode="mosaic", limit=16) +``` + ## Escape hatches When you need raster math that isn't in the `rst_*` surface, drop down to NumPy / rasterio per tile with the [pyrx escape hatches](./raster-functions#escape-hatches) (`tile_to_numpy`, `rst_apply`). diff --git a/docs/docs/api/vizx.mdx b/docs/docs/api/vizx.mdx index a378927d6..0fdc945af 100644 --- a/docs/docs/api/vizx.mdx +++ b/docs/docs/api/vizx.mdx @@ -24,7 +24,7 @@ pip install "geobrix[vizx]" From a Databricks notebook (commonly alongside the lightweight tier): ```python -%pip install --quiet "geobrix[light,vizx] @ file:///Volumes////geobrix-0.4.3-py3-none-any.whl" +%pip install --quiet "geobrix[light,vizx] @ file:///Volumes////geobrix-0.5.0-py3-none-any.whl" ``` ## Import diff --git a/docs/docs/api/vrt-mosaic.mdx b/docs/docs/api/vrt-mosaic.mdx new file mode 100644 index 000000000..25c1f166c --- /dev/null +++ b/docs/docs/api/vrt-mosaic.mdx @@ -0,0 +1,318 @@ +--- +sidebar_position: 4.5 +sidebar_label: VRT & Mosaics +title: VRT & Mosaics +--- + +import CodeFromTest from '@site/src/components/CodeFromTest'; +import vrtExamples from '!!raw-loader!../../tests/python/readers/vrt_mosaic_examples.py'; + +# VRT & Mosaics + +A **VRT mosaic** is a directory of COGs, or bounded mini-COG tiles, plus a lightweight +GDAL VRT index (`mosaic.vrt`) that describes how they fit together. The index is +a portable, spatially-aware XML file — a lens over the tile set that can be +opened by any GDAL-based tool without knowing where the tiles came from. On the +reading side, pointing a GeoBrix reader at `mosaic.vrt` expands it back into one +virtual tile per member, so all `rst_*` functions work per-tile unchanged. + +![GeoBrix VRT mosaic flow — a single large raster too big to hold as one COG or in per-task memory is split by the cog_gbx writer in mosaic mode into a grid of bounded mini-COGs, each read window-by-window so the source is never fully materialised in RAM; the tiles are described by a lightweight, portable mosaic.vrt index — a GDAL lens that the writer can persist with relative paths or mint_vrt can build on demand with absolute paths, and that opens in any GDAL tool such as QGIS, gdalinfo, or rio-tiler with no GeoBrix needed; a GeoBrix reader pointed at the VRT then expands it back into one bytes-free virtual tile per member, so rst_* functions run per-tile distributed across the cluster and a windowed read touches only the tiles intersecting the viewport](../../../resources/images/diagrams/rasterx/vrt-mosaic.png) + +This complements the [File Column](../readers/file) table and +[Volume](../api/large-rasters) modes as a storage form for distributed raster +processing: + +| Form | What lives on disk | Best for | +|---|---|---| +| Single master COG | One large `.tif` per source | Windowed reads of one file at a time | +| VRT mosaic | Directory of mini-COGs + `mosaic.vrt` | Splitting sources that exceed per-task memory, portability across GDAL tools | +| FILE-column Delta table | FILE references in a Delta table | Governed, lifecycle-managed, long-term storage | + +--- + +## Prepare: write a VRT mosaic + +Pass `vrtMosaic=true` to `cog_gbx` to enter mosaic mode. The writer reads each +source file window-by-window (one tile at a time — the source is never fully +materialised in RAM), writes one mini-COG per tile, and then builds `mosaic.vrt` +in the output directory. The tile grid defaults to `gridSystem="none"` (native +pixel tiling), so you normally set only `vrtMosaic` and, optionally, `tileSize`. + + + +After the write, the output directory contains: + +``` +mosaic/ + tile__0_0.tif # mini-COG, row 0 col 0 + tile__0_1.tif # mini-COG, row 0 col 1 + ... + mosaic.vrt # VRT index referencing all tiles +``` + +The `` is a stable, source-namespaced token +(up to 16 alphanum characters of the source basename + 8 hex characters of a +SHA-1 hash of the source path). Two sources in the same write never collide on +tile names, and a re-run produces the same names for the same sources +(`cogSkipIfExists=true` by default, so idempotent resumption is free). + +### Options + +| Option | Default | Description | +|---|---|---| +| `vrtMosaic` | — | Set `"true"` to activate VRT-mosaic mode. (Supplying `gridSystem` also activates it; one of the two must be present.) | +| `gridSystem` | `"none"` | Tile grid. `"none"` (native pixel tiling, default), `"quadbin"` (cell-aligned to the quadbin grid, reprojected to EPSG:3857), or `"h3"` (cell-aligned to the h3 grid, reprojected to EPSG:4326, hex-clipped). BNG is planned. | +| `gridResolution` | — | Grid resolution. Required when `gridSystem="quadbin"` or `gridSystem="h3"`; invalid with `gridSystem="none"`. Quadbin: range `0`–`20`. H3: range `0`–`15`. | +| `tileSize` | `"1024"` | Tile edge length in pixels. Tiles are square. The last row and column are clipped to the source boundary. | +| `overlapPercent` | `"0"` | Tile-edge halo as a percentage of `tileSize`. A value of `5` expands each tile's read window by `ceil(tileSize * 5 / 100)` pixels on every side, clamped to the source bounds. Default 0 — non-overlapping grid. | +| `mergeStrategy` | `"none"` | How overlapping tiles are blended: `none` (last-write wins), `min`, `max`, `avg`, `first`, or `last`. Relevant only when tiles from different sources overlap spatially. | +| `pruneEmpty` | `"true"` | Skip tiles whose pixels are entirely NoData. Saves storage and reader work for sparse sources. | +| `writeVrt` | `"true"` | Emit `mosaic.vrt` alongside the tiles. Set `"false"` if you only need the tiles and will build the index separately with [`mint_vrt`](#mint-on-demand-vrt). | +| `vrtPaths` | `"relative"` | How tile paths are written inside the VRT. `"relative"` (default) — bare filenames, VRT is portable as long as tiles stay in the same directory. `"absolute"` — full paths, VRT works from any location. | + +**Mutually exclusive constraints:** + +- `driverMode=true` and mosaic mode cannot be combined — mosaic mode writes + per-tile mini-COGs on executors; `driverMode` produces a single driver-side COG. + Use one or the other. +- `tileSize` and `overlapPercent` are only valid with `gridSystem="none"`. + DGGS-aligned options (`gridMinResolution`, `gridMaxResolution`, + `gridStepResolution`) are reserved for future grid systems and are rejected for + `gridSystem="none"`. + +Tile-encoding options (`cogBlockSize`, `cogOverviewResampling`, `compress`, +`compressLevel`, `predictor`, `cogBigTiff`, `cogSkipIfExists`) are passed +through to each mini-COG. See the [COG Writer](../writers/cog#cog-options) for the +full encoding option reference. + +--- + +### Quadbin (map-render) mosaic + +When `gridSystem="quadbin"`, the writer reprojects each source to EPSG:3857 and +writes one mini-COG per overlapping quadbin cell at the requested resolution. +Tiles are aligned to the quadbin grid, making the output directly compatible +with XYZ/quadkey map rendering pipelines and spatial join workflows that operate +on quadbin cell identifiers. + + + +Cell tiles are named `cell__.tif` and the VRT index is +written as `mosaic.vrt` alongside them. The `` is the same stable +source-namespaced token used in native mode, so two sources in the same write +never collide. + +When `raster_gbx` expands the VRT, each row's `tile.metadata` carries: + +- `tile.metadata["cellid"]` — the quadbin cell id (string) +- `tile.metadata["gridSystem"]` — `"quadbin"` + +These fields are absent on native (`gridSystem="none"`) tiles. + +:::note +`tileSize` and `overlapPercent` are not valid with `gridSystem="quadbin"`. +Set only `gridSystem` and `gridResolution`. +::: + +--- + +### H3 (unification) mosaic + +When `gridSystem="h3"`, the writer reprojects each source to EPSG:4326 and +writes one mini-COG per overlapping h3 cell at the requested resolution. Each +cell's pixels are clipped to its hexagon boundary — pixels outside the hexagonal +footprint become NoData. This aligns the raster output to the h3 grid and makes +the `cellid` field a plain equi-join key for unifying with any other h3-indexed +dataset (gridded analytics, coverage tables, weather measurements, and similar). + + + +Cell tiles are named `cell__.tif` and the VRT index is +written as `mosaic.vrt` alongside them. The `` is the same stable +source-namespaced token used in native and quadbin modes, so two sources in the +same write never collide. + +When `raster_gbx` expands the VRT, each row's `tile.metadata` carries: + +- `tile.metadata["cellid"]` — the h3 cell id (h3index string) +- `tile.metadata["gridSystem"]` — `"h3"` + +The `cellid` value is the canonical h3index string — compatible with +`h3.str_to_int` / `h3.int_to_str` and with h3-indexed tables written by any +other workflow. Join the expanded mosaic rows to a tabular h3-indexed DataFrame +on `cellid` to unify raster pixel statistics with non-raster data sharing the +same grid. + +:::note +`tileSize` and `overlapPercent` are not valid with `gridSystem="h3"`. +Set only `gridSystem` and `gridResolution`. +::: + +--- + +## Mint: on-demand VRT + +`mint_vrt` builds a transient VRT over an arbitrary list of tile paths without +writing an index file alongside the tiles. The VRT is placed in a temporary +directory with absolute `SourceFilename` paths so rasterio can resolve each +member regardless of working directory. + + + +Pass an explicit `out` path to write the VRT to a fixed location instead of a +temp directory. Choose the location by **who needs to open the `.vrt` file, and +for how long** — its member tiles are always referenced by absolute path, so +they must be reachable from wherever the VRT is opened: + +```python +# Driver-local, transient — a one-shot windowed read in this same process. +# /tmp is visible only to the driver and is gone when the driver restarts. +vrt_path = mint_vrt(tile_paths, out="/tmp/query_mosaic.vrt") + +# Shared and durable — reopenable later, and readable by workers, another +# notebook, or an external GDAL tool (QGIS, gdalinfo, rio-tiler). The tiles it +# references must also live on the Volume so the absolute paths resolve there. +vrt_path = mint_vrt(tile_paths, out="/Volumes/catalog/schema/vol/mosaic/query.vrt") +``` + +:::note Where a minted VRT is reachable +A minted VRT is usable at a given place only if **both** the `.vrt` file and its +absolute member paths resolve there. + +- **No `out` (temp dir) or a `/tmp` path** — driver-local and ephemeral. This is + the primary use of `mint_vrt`: build a VRT over a dynamic tile subset, open it + with rasterio **in the same driver/notebook process** for a windowed read, then + discard it. It is invisible to workers and to later sessions, and you are + responsible for deleting the temp file when you are done with it. +- **A `/Volumes/...` path (FUSE)** — shared across the driver, workers, and + external clients, and it persists. Use it when the index must outlive the + process or be read from somewhere else. Because `mint_vrt` bakes **absolute** + member paths, the tiles themselves must also be on the Volume. + +If you instead want a **portable, movable** index co-located with its tiles — one +where you can copy or move the whole directory — prefer the writer's persisted +`mosaic.vrt` (`writeVrt=true`), which uses **relative** paths by default. A +minted VRT is the tool for a *dynamic subset* (pinned or ephemeral); the persisted +mosaic is the tool for the *whole tile set*. +::: + +`mint_vrt` is Connect-safe — pure Python + rasterio, no Spark session, no +`_jvm`, no `osgeo`. It can be called from a notebook driver cell or from +any Python process. + +**Parameters:** + +| Parameter | Required | Description | +|---|---|---| +| `tile_paths` | Yes | Non-empty list of absolute paths to COG tiles. All tiles must share the same CRS, pixel size, band count, and dtype. | +| `out` | No | Optional destination path for the VRT. When omitted, the VRT goes to a private temp directory (driver-local, transient). Put it on a Volume (`/Volumes/...`) if it must be reopened later or read from workers or other clients — see the reachability note above. | + +**Returns:** absolute path (`str`) to the written VRT file. + +--- + +## Read: point a reader at the VRT + +Point `raster_gbx` (or `cog_gbx`) at `mosaic.vrt` and the reader parses the +VRT XML, enumerates member paths, and emits **one virtual tile row per member**: + + + +Each row is a whole-file virtual tile (`raster=NULL`, `path` set to the member +file path, `window=NULL`). All downstream `rst_*` functions operate on the tile +bytes read lazily from the member path on demand — no pixel data is loaded before +the operation actually needs it. + +### Directory vs. VRT load + +When the load path points **directly at `mosaic.vrt`** the reader expands the +VRT into one row per member. When the load path points at the **containing +directory** the reader walks the directory for raster files and excludes +`.vrt` files — they are indexes, not raster members, and including them would +double-count the mosaic. Both patterns produce one row per tile, but the VRT +path is the canonical way to load a mosaic. + +### Spatial filtering + +Pass `clipPolygons` to restrict the expansion to only the members that intersect +a given area of interest. Only the intersecting mini-COGs contribute rows; the +rest are skipped without being opened: + +```python +# Polygon in the CRS of the source raster +aoi_wkt = "POLYGON ((400000 4999000, 401000 4999000, 401000 5000000, 400000 5000000, 400000 4999000))" + +df_clipped = ( + spark.read.format("raster_gbx") + .option("clipPolygons", aoi_wkt) + .option("clipCrs", "EPSG:32632") + .load("/Volumes/catalog/schema/vol/mosaic/mosaic.vrt") +) +``` + +--- + +## Serverless-safe by construction + +Mosaic mode reads each tile window independently — the source is opened once +per executor task, and each tile is read as a bounded pixel array +(`tileSize × tileSize × bands × itemsize`). The source is never pulled fully +into memory. At the default `tileSize=1024`, a single-band `uint16` tile is +approximately 2 MB — well within the Serverless per-task budget regardless of +how large the source raster is. + +`mint_vrt` and `_parse_vrt_members` (the VRT parser inside the reader) are pure +Python — no GDAL Python bindings (`osgeo`), no native extension — so they work +identically on Serverless and on classic compute. + +See [Serverless & Memory](../serverless-and-memory) for the full per-task memory +model and guidance on routing large files. + +--- + +## What a VRT mosaic is good for + +- **Distributed processing of large rasters** — a single source file too large + to process whole is split into bounded mini-COGs; `rst_*` functions then run + per-tile across the cluster. +- **Windowed locality** — a windowed read via `mint_vrt` or `rasterio.open(vrt)` + touches only the mini-COGs whose spatial extent intersects the requested + viewport. GDAL resolves the intersection from the VRT XML header without + opening any tile that falls outside the window. +- **Portable GDAL artifact** — `mosaic.vrt` is a standard GDAL VRT file readable + by QGIS, gdalinfo, rio-tiler, and any GDAL 2+ tool. With `vrtPaths="relative"` + (the default), the whole directory (tiles + VRT) can be copied or moved and + the index stays valid. +- **Expandable to rows for distributed work** — pointing any GeoBrix reader at + the VRT turns a static directory of tiles into a partitioned Spark DataFrame in + one line, feeding `rst_*` pipelines without any manual enumeration. + +--- + +## Upcoming + +BNG grid-aligned mosaics are planned — tiles aligned to BNG cells and tagged +with a cell identifier for spatial joins and data unification. In-notebook +windowed rendering of a VRT mosaic is also on the roadmap. Pyramid options +(`gridMinResolution`, `gridMaxResolution`, `gridStepResolution`) are accepted +by the parser but raise a clear error until their grid-system support lands. + +--- + +## See also + +- [Large Rasters](./large-rasters) — COG preparation, the memory model, and when to split a large source into a mosaic +- [Virtual Tiles](./virtual-tiles) — how VRT expansion fits into the virtual/materialized tile lifecycle +- [COG Writer (`cog_gbx`)](../writers/cog) — full option reference for the COG preparation writer +- [COG Reader (`cog_gbx`)](../readers/cog) — windowed, bbox-clipped reads from prepared COGs +- [Serverless & Memory](../serverless-and-memory) — Serverless per-task memory model and safe write patterns diff --git a/docs/docs/common-functions.mdx b/docs/docs/common-functions.mdx new file mode 100644 index 000000000..a40e6e800 --- /dev/null +++ b/docs/docs/common-functions.mdx @@ -0,0 +1,113 @@ +--- +sidebar_position: 2 +title: GBX Common Functions +--- + +import CodeFromTest from '@site/src/components/CodeFromTest'; +import commonExamples from '!!raw-loader!../tests/python/readers/common_functions_examples.py'; + +# GBX Common Functions + +Every lightweight reader and writer is built on **one shared file-access base** +(`databricks.labs.gbx.ds.file_gbx`). This page catalogs the common functions that +base exposes, and draws the two boundaries that decide which one you reach for: +**FUSE vs FILE** and **generic vs format-specific**. + +![GBX common functions — three tiers: a session-ful generic function layer with gbx_file_read and gbx_file_write; a format-specific decoder/encoder layer with rst_fromfile and vector_file_read; and a session-free core floor with list_local_files, enumerate_files, and to_local_path; with FUSE-vs-FILE tier bands](../../resources/images/diagrams/rasterx/gbx-common-functions.png) + +## Generic functions (session-ful, function layer) + +These functions require a `SparkSession` and operate at the function layer (driver code). +They are not callable from inside a session-less DataSource reader or writer on Spark Connect. + +| Function | Signature | Returns | Use when | +|---|---|---|---| +| `gbx_file_read` | `gbx_file_read(spark, source, *, source_type="auto", access="auto", recursive=True, include_hidden=False, extensions=None, path_glob_filter=None)` | DataFrame `[path, size, file]` — path is the `/Volumes/...` string; file is a FILE reference when the runtime supports FILE, else null | You want FILE/FUSE references from a Volume path or a FILE-column table, format-agnostic. | +| `gbx_file_write` | `gbx_file_write(df, target, *, file_mode="auto", filespace=None, layout="order", overwrite=False, file_col="tile_file", spark=None)` | `None` (writes a Delta table) | You want to land a DataFrame into a Delta table with an optional FILE column (MANAGED / EXTERNAL / FUSE). | + +`gbx_file_read` returns **path and FILE references — never bytes**. To decode the rasters at +each path into tile structs, compose with `rst_fromfile` (the canonical raster pattern): + + + +### access / source_type options + +| Parameter | Values | Behavior | +|---|---|---| +| `access` | `"auto"` (default) | Silently uses the best available tier: FILE refs when capable, null `file` on FUSE. Never raises. | +| `access` | `"external"` | Requires a FILE-capable runtime (`read_files` / `list_files` tier). Raises `ValueError` on FUSE-only runtimes. | +| `access` | `"managed"` | Valid **only** for a FILE-column table source. Raises `ValueError` for any Volume path/directory (MANAGED refs are minted on write, not enumerated on read). | +| `source_type` | `"auto"` (default) | Classifies the source automatically: a path/URI → `"location"`, otherwise → `"table"`. | + +### file_mode / layout options for gbx_file_write + +| Parameter | Values | Behavior | +|---|---|---| +| `file_mode` | `"auto"` (default) | FILE-capable runtime + `filespace` → `"managed"`; FILE-capable + no filespace → `"external"`; FUSE-only → `"fuse"`. | +| `file_mode` | `"managed"` | Explicit FILE MANAGED via `create_file`. Requires `filespace`. Raises on FUSE-only runtimes. | +| `file_mode` | `"external"` | Explicit FILE EXTERNAL via `try_to_file`. Raises on FUSE-only runtimes. | +| `file_mode` | `"fuse"` | Plain Delta write, no FILE column regardless of runtime. | +| `layout` | `"order"` (default) | `ORDER BY path` at write time — scan-friendly. | +| `layout` | `"cluster"` | `CLUSTER BY path` in the DDL (FILE-mode tables only); run `OPTIMIZE
` afterward for durable clustering. | +| `layout` | `"plain"` | No ordering — fastest write, scan order determined by the cluster. | + +## Session-free core (safe inside DataSource readers/writers) + +These functions require no `SparkSession` to function — their minimum viable mode is +FUSE (`os.walk` / `stat`), which is always available. `list_local_files` is FUSE-only. +`enumerate_files` accepts an optional `spark=` and issues Spark SQL (`read_files` / +`list_files`) when a FILE-capable session is present, degrading to a FUSE list when not — +it is **not** session-free in the strict sense but degrades gracefully. All are +**Connect-safe**: no `sparkContext`, `.rdd`, `_jvm`, or `conf.set`. + +`list_local_files` is the single shared enumeration routine every DataSource reader +consumes internally. `enumerate_files` and the path helpers are used at the function +layer, not inside DataSource readers. + +| Function | Signature | Returns | Role | +|---|---|---|---| +| `list_local_files` | `list_local_files(path, *, recursive=True, include_hidden=False, extensions=None, path_glob_filter=None)` | `list[str]` (sorted paths) | Directory enumeration — the single shared routine every DataSource reader uses. FUSE only; no session required. | +| `enumerate_files` | `enumerate_files(path, *, recursive=True, include_hidden=False, extensions=None, path_glob_filter=None, spark=None)` | DataFrame `[path, size, file]` or `list[dict]` | Issues Spark SQL when a FILE-capable session is present; falls back to a FUSE list of dicts otherwise. Used at the function layer, not inside DataSource readers. | +| `to_local_path` | `to_local_path(path) -> str` | FUSE path string | Normalizes a URI-scheme path (`dbfs:/Volumes/...`) to a bare FUSE path (`/Volumes/...`). | +| `to_spark_uri` | `to_spark_uri(path) -> str` | URI string | Inverse: bare FUSE path to `dbfs:/Volumes/...` for Spark SQL contexts. | + +By default, files whose names start with `_` or `.` (Spark/Hadoop metadata files such as +`_SUCCESS`, `_committed_*`, `.crc`) are **skipped**. Pass `include_hidden=True` to include them. + +A positive selection filter — either `extensions` (a tuple of suffixes, e.g. `(".tif", ".nc")`) +or `path_glob_filter` (an fnmatch-style glob, e.g. `"*.tif"`) — narrows which files are returned. +The two are mutually exclusive. + + + +## The two boundaries + +**FUSE vs FILE.** `df.write.format(...)` / `spark.read.format(...)` DataSources are +**FUSE-only** — on Spark Connect they run session-less, so no FILE-tier SQL is available. +FILE-tier read and write live in the **function layer** (`gbx_file_read` / `gbx_file_write` +and the per-format entries), where a `SparkSession` is present. + +**Generic vs format-specific.** `gbx_file_read` / `gbx_file_write` move path references and +FILE handles. The format-specific decoders/encoders (`rst_fromfile`, `vector_file_read`, +`write_file_table`) sit on top and understand the payload. + +## No-gating rule + +`access` / `file_mode` = `"auto"` silently uses the best available tier. An explicit +`"managed"` / `"external"` on a runtime without FILE raises a clear, actionable error that +names the DBR version requirement and the auto-fallback option — never a silent downgrade. + +See [Shared file-access base](./readers-writers#file-gbx) for tier detection details and the +access-flow diagram. diff --git a/docs/docs/installation.mdx b/docs/docs/installation.mdx index 3a4b381d3..742f2ac97 100644 --- a/docs/docs/installation.mdx +++ b/docs/docs/installation.mdx @@ -20,18 +20,42 @@ GeoBrix supports both current Databricks Runtime LTS releases: |---|---|---|---|---|---|---|---| | **17.3 LTS** | 24.04 | 4.0.0 | 3.12.3 | 2.13.16 | 17 | **5+** (Py 3.12) | ✅ Supported | | **18 LTS** | 24.04 | 4.1.0 | 3.12.3 | 2.13.16 | 21 | **5+** (Py 3.12) | ✅ Supported | +| **19 LTS** | 26.04 | 4.2.0 | 3.12.3 | 2.13.16 | 21 | (n/a yet) | ✅ Supported (light tier) | -A **single wheel + single JAR** runs on both: Scala 2.13.16 matches both runtimes, the JAR is compiled to Java-17 bytecode so it loads on both JVMs, and Spark is a `provided` dependency. +A **single wheel + single JAR** runs on both 17.3 and 18 LTS: Scala 2.13.16 matches both runtimes, the JAR is compiled to Java-17 bytecode so it loads on both JVMs, and Spark is a `provided` dependency. The **Serverless env** column is the minimum [Serverless environment version](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/five) for the lightweight tier: **version 5+** provides Python 3.12, which the `[light]` dependencies require (Python ≥ 3.11). Older environment versions (Python 3.10) can't install `geobrix[light]`. Release notes for env v5: [AWS](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/five) · [Azure](https://learn.microsoft.com/azure/databricks/release-notes/serverless/environment-version/five) · [GCP](https://docs.databricks.com/gcp/en/release-notes/serverless/environment-version/five). -:::note DBR 19 LTS is coming soon -**DBR 19 LTS is coming soon**, built on **Ubuntu 26.04**. The **lightweight** tier (pure-Python, rasterio's bundled GDAL) will be unaffected; the **heavyweight** tier's native GDAL/OGR libraries are compiled against the cluster OS, so they will need to be rebuilt for the new base image. +:::note DBR 19 LTS — heavy tier coming soon +**DBR 19 LTS** (Ubuntu 26.04) is now supported by the **lightweight** tier. The **heavyweight** tier's native GDAL/OGR libraries are compiled against the cluster OS, so heavyweight on DBR 19 requires a GDAL rebuild for the new base image — not yet available. ::: +:::note GeoBrix Light and Databricks runtimes +**GeoBrix Light defaults to Databricks Serverless environments (currently environment v5).** +On Serverless, `%pip install geobrix[light]` installs cleanly — no further configuration needed. + +**On a classic DBR cluster, additional dependencies must be accounted for** — each classic DBR +generation ships a different immutable base (`protobuf`/`grpcio-status`, `idna`, +`typing_extensions`, …), so the default `[light]` extra can conflict there. On DBR 19 in +particular, installing `geobrix[light]` triggers a protobuf conflict that causes hangs and +kernel crashes. On classic compute, use the extra that matches your runtime from the table below. + +| Runtime | Install command | Full feature set | +|---|---|---| +| Serverless (env v5+) | `%pip install "geobrix[light] @ file:///Volumes/…"` | `geobrix[light_all]` | +| Classic DBR 17.3–18.x | `%pip install "geobrix[light] @ file:///Volumes/…"` | `geobrix[light_all]` | +| Classic DBR 19+ | `%pip install "geobrix[light_dbr19] @ file:///Volumes/…"` | `geobrix[light_dbr19_all]` | + +*(Full feature set = `[light/_dbr19]` + `[stac]` + `[vizx]` + `[overture]` in one install. `[databricks]` is added separately if needed.)* + +The `file:///Volumes/…` path is a placeholder for the wheel on your Unity Catalog Volume — see +the install steps below for the exact pattern including the PEP 508 `package @ file://` quoting +required on Serverless. +::: + The lightweight tier is a single Python wheel installed with the `light` extra — **no init script, no JAR, no native GDAL bundle** (rasterio's bundled GDAL does the work). It runs on serverless compute, standard (shared) clusters, Lakeflow declarative pipelines, and ARM. One wheel covers the whole lightweight tier: RasterX (the full `rst_*` set, via `databricks.labs.gbx.pyrx`), VectorX (via `databricks.labs.gbx.pyvx`), and GridX quadbin (via `databricks.labs.gbx.pygx`). The wheel ships as a **GitHub release artifact** for GeoBrix 0.4.0+ — it is **not** published to PyPI. Install it from a Unity Catalog Volume so the same path works on both Serverless and Classic compute: diff --git a/docs/docs/intro.mdx b/docs/docs/intro.mdx index 9fe916eaa..9988963d0 100644 --- a/docs/docs/intro.mdx +++ b/docs/docs/intro.mdx @@ -83,15 +83,18 @@ GridX's 40 functions break down as **BNG (23)**, **Quadbin (10)**, and **custom ## Supported Databricks Runtimes -GeoBrix supports both current Databricks Runtime LTS releases — a **single wheel + single JAR** runs on both (Scala 2.13.16 matches both, the JAR is Java-17 bytecode that loads on both JVMs, and Spark is a `provided` dependency): +GeoBrix supports the current Databricks Runtime LTS releases — a **single wheel + single JAR** runs on **17.3 and 18 LTS** (Scala 2.13.16 matches both runtimes, the JAR is Java-17 bytecode that loads on both JVMs, and Spark is a `provided` dependency); the **lightweight tier** additionally runs on **DBR 19 LTS**: -| DBR LTS | Ubuntu | Spark | Python | Scala | Java | GeoBrix | -|---|---|---|---|---|---|---| -| **17.3 LTS** | 24.04 | 4.0.0 | 3.12.3 | 2.13.16 | 17 | ✅ Supported | -| **18 LTS** | 24.04 | 4.1.0 | 3.12.3 | 2.13.16 | 21 | ✅ Supported | +| DBR LTS | Ubuntu | Spark | Python | Scala | Java | Serverless env | GeoBrix | +|---|---|---|---|---|---|---|---| +| **17.3 LTS** | 24.04 | 4.0.0 | 3.12.3 | 2.13.16 | 17 | **5+** (Py 3.12) | ✅ Supported | +| **18 LTS** | 24.04 | 4.1.0 | 3.12.3 | 2.13.16 | 21 | **5+** (Py 3.12) | ✅ Supported | +| **19 LTS** | 26.04 | 4.2.0 | 3.12.3 | 2.13.16 | 21 | (n/a yet) | ✅ Supported (light tier) | -:::note DBR 19 LTS is coming soon -**DBR 19 LTS is coming soon**, built on **Ubuntu 26.04**. The **lightweight** tier (pure-Python, rasterio's bundled GDAL) will be unaffected; the **heavyweight** tier's native GDAL/OGR libraries are compiled against the cluster OS, so they will need to be rebuilt for the new base image. +The **Serverless env** column is the minimum Serverless environment version for the lightweight tier: **version 5+** provides Python 3.12, which the `[light]` dependencies require (Python ≥ 3.11). Older environment versions (Python 3.10) can't install `geobrix[light]`. Env v5 release notes: [AWS](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/five) · [Azure](https://learn.microsoft.com/azure/databricks/release-notes/serverless/environment-version/five) · [GCP](https://docs.databricks.com/gcp/en/release-notes/serverless/environment-version/five). + +:::note DBR 19 LTS — heavy tier coming soon +**DBR 19 LTS** (Ubuntu 26.04) is now supported by the **lightweight** tier. The **heavyweight** tier's native GDAL/OGR libraries are compiled against the cluster OS, so heavyweight on DBR 19 requires a GDAL rebuild for the new base image — not yet available. ::: ## Background diff --git a/docs/docs/limitations.mdx b/docs/docs/limitations.mdx index 250289f6f..2273e4d3f 100644 --- a/docs/docs/limitations.mdx +++ b/docs/docs/limitations.mdx @@ -7,13 +7,13 @@ import limitationsCode from '!!raw-loader!../tests-dbr/python/limitations/exampl # Known Limitations -GeoBrix Beta has some known limitations that will be addressed in future releases. +GeoBrix has some known limitations that will be addressed in future releases. ## Databricks Spatial Types ### Current State -The Beta does not yet support Databricks Spatial Types directly but is standardized to WKB or WKT where geometries are involved. +GeoBrix does not yet support Databricks Spatial Types directly but is standardized to WKB or WKT where geometries are involved. ### Workaround @@ -87,6 +87,17 @@ Databricks Runtime: - **Minimum**: DBR 17.1 (recommended/tested: DBR 17.3 LTS or 18 LTS) - GeoBrix is designed to work with Databricks product spatial functions (available DBR 17.1+) +### Serverless Spark configuration + +On Serverless compute most Spark properties are locked to platform defaults and cannot be +overridden (for example, adaptive-query-execution toggles are ignored). Serverless does, however, +accept a **limited set** of Spark configs — including **`spark.sql.files.maxPartitionBytes`**, which +you can tune down (e.g. `"32m"`) to reduce the rows/bytes per partition on file and Delta scans. This +is a useful lever for controlling memory per task in decode-heavy pipelines (each partition holds +fewer files/rows, so each task decodes less at once). See the Databricks docs on +[configuring Spark properties for Serverless](https://docs.databricks.com/aws/en/spark/conf) for the +full allowlist. + ## Format Support - [OGR](./readers/vector) - focus is on named vector readers in GDAL's OGR package. diff --git a/docs/docs/notebooks/eo-series.mdx b/docs/docs/notebooks/eo-series.mdx index 08d7000aa..1bb481112 100644 --- a/docs/docs/notebooks/eo-series.mdx +++ b/docs/docs/notebooks/eo-series.mdx @@ -45,8 +45,8 @@ Downloads are throttled on the Planetary Computer free tier. Notebooks use [`Sta ![Notebook 03 — Sentinel-2 scene → typed tile struct → H3 res-7 tessellation → per-cell timeseries](../../../resources/images/diagrams/eo-series/eo-series-03.png) -- **One-step raster ingestion** — the `gtiff` reader (and the `binaryFile` → `rst_fromcontent` pattern) materializes a typed `tile` column with bytes, bbox, SRID, and standardized nodata in a single pass. -- **Spatial-indexed raster tables** — `rst_h3_tessellate` shreds each Sentinel-2 scene into H3 resolution-7 cells, producing `band_b0X_h3` Delta tables that join cleanly across bands and dates. +- **Virtual-tile ingestion — bytes-free.** The `gtiff_gbx` reader loads each Sentinel-2 scene as [virtual tiles](../api/virtual-tiles): with `.option("tileSize", "2048,2048")` a 5490×5490 scene fans into ~9 windows, and each tile row carries a source `path` + pixel `window` **instead of raster bytes** — so `band_b0X_tile` is a table of lightweight references (`tile.raster` is `null`; `tile.path`/`tile.window` set), no pixels in Delta. `rst_initnodata` records a pending nodata instruction and `rst_srid`/`rst_boundingbox` answer from the header, all without materializing. +- **Spatial-indexed raster tables (the materialization cut-over)** — `rst_h3_tessellate` shreds each scene into H3 resolution-7 cells, producing `band_b0X_h3` Delta tables that join cleanly across bands and dates. This is where the virtual tiles **materialize**: each window's pixels are read once, at tessellation. - **Raster analytics from SQL/PySpark** — `rst_summary` for per-tile stats, `h3_kring` + `rst_merge_agg` for spatial neighborhoods, and the `rst_apply` escape-hatch for raster-to-timeseries projection — no driver-side rasterio loops. ### 04 — Band Stacking + Clipping @@ -66,7 +66,7 @@ Downloads are throttled on the Planetary Computer free tier. Notebooks use [`Sta | `config_nb.ipynb` | Shared setup (`%run ./config_nb` from every main notebook). Installs the `geobrix[light,stac,vizx]` wheel + EO deps, selects the tier (option-1 `pyrx` default / option-2 `rasterx`), registers functions + light readers/writers, imports the visualization helpers from `databricks.labs.gbx.vizx` (`plot_raster`, `plot_file`, `as_gdf`, `cells_as_gdf`) and the pyrx escape-hatches (`rst_apply`, `tile_to_numpy`), sets Unity Catalog `catalog_name` / `schema_name`, creates the `/Volumes///data/alaska` ETL tree, exposes the `FORCE_REBUILD` toggle and the Serverless-safe `set_conf_safe()` helper, instantiates `stac_client = StacClient()`, and defines tiling helpers (`finalize_tiled_band_tbl`, `gen_tessellate_tiled_band`). | | `01. Search STACs.ipynb` | Loads the TIGER US Counties shapefile via the `shapefile_gbx` reader, filters to Ketchikan, tessellates into H3 resolution-2 cells, converts each cell to GeoJSON, and calls `stac_client.search(df_cells, geojson_col="geojson", collections=["sentinel-2-l2a"], ...)` to fan out per-cell queries to Planetary Computer. Writes the resulting STAC asset metadata — one row per `(cell, item, asset)` — to a timestamped Delta directory (`cell_assets_.delta`). | | `02. Download STACs.ipynb` | Reads `cell_assets_*.delta` and calls `stac_client.download(band_rows, out_dir, asset_names=[band], ...)` for each band. Creates one `band_` Delta table per band with `item_id`, `band_name`, `date`, `out_file_path`, `out_file_sz`, and `is_out_file_valid` columns. Calls `stac_client.repair("band_")` to re-download and merge any files that failed read-validation. | -| `03. Gridded EO Data.ipynb` | For each band, joins the Delta band table with the `gtiff` reader, materializes `band__tile` (adds `size`, `bbox`, `srid`, and standardized nodata), then tessellates each tile to H3 resolution 7 into `band__h3`. Demonstrates `rst_summary`, bounding-box reprojection, `h3_kring` with `rst_merge_agg`, and raster → timeseries projection via the `rst_apply` escape-hatch. | +| `03. Gridded EO Data.ipynb` | For each band, reads via the `gtiff_gbx` reader in **virtual-tile** mode (`tileSize`) into a bytes-free `band__tile` (`path`+`window` references, plus `size`, `bbox`, `srid`, and a pending nodata instruction — no pixels in Delta), then tessellates each tile to H3 resolution 7 into `band__h3`, **materializing pixels at the tessellation cut-over**. Demonstrates `rst_summary`, bounding-box reprojection, `h3_kring` with `rst_merge_agg`, and raster → timeseries projection via the `rst_apply` escape-hatch. | | `04. Band Stacking + Clipping.ipynb` | Joins the four `band__h3` tables on `(cellid, date)`, stacks bands in (R, G, B, NIR) order with `rst_frombands` into the `band_stack` table, writes multi-band TIFs back out via the `gtiff` writer (`nameCol`-driven filenames), and demonstrates per-tile clipping with `rst_clip` using a centroid-envelope buffer built from Databricks built-in ST functions. | --- @@ -74,7 +74,7 @@ Downloads are throttled on the Planetary Computer free tier. Notebooks use [`Sta ## Prerequisites - **Databricks Runtime 17.3 LTS / 18 LTS, or Serverless** (Scala 2.13 / Spark 4 / Python 3.12). The lightweight default runs on Serverless; the heavyweight tweak needs a classic x86 cluster. -- **GeoBrix** (version 0.4.0). `config_nb.ipynb` `%pip`-installs the `geobrix[light,stac,vizx]` wheel — pure-Python bindings + rasterio + the STAC client dependencies (`pystac-client`, `planetary-computer`, `tenacity`, `requests`) + the visualization extras (`matplotlib`, `geopandas`, `mapclassify`) — nothing is assumed pre-staged. For the heavyweight tweak, flip option-2 (`rasterx`) in `config_nb.ipynb` and attach the GeoBrix JAR + GDAL init script to the cluster. +- **GeoBrix** (version 0.5.0). `config_nb.ipynb` `%pip`-installs the `geobrix[light,stac,vizx]` wheel — pure-Python bindings + rasterio + the STAC client dependencies (`pystac-client`, `planetary-computer`, `tenacity`, `requests`) + the visualization extras (`matplotlib`, `geopandas`, `mapclassify`) — nothing is assumed pre-staged. For the heavyweight tweak, flip option-2 (`rasterx`) in `config_nb.ipynb` and attach the GeoBrix JAR + GDAL init script to the cluster. - **Unity Catalog**: edit `config_nb.ipynb` to set `catalog_name` and `schema_name` to your own locations. A Volume named `data` must already exist under `/`. The notebooks create a schema if missing but will not create the Volume for you. - **Compute sizing**: the lightweight default runs on Serverless. On classic clusters, the captured heavyweight runs used AWS `m5d.xlarge` (2–16 workers) for search/download and `r6id.2xlarge` (20 workers) for raster processing; an `x86` instance is required for the GDAL natives. For a single county a much smaller cluster is sufficient. diff --git a/docs/docs/notebooks/h3-rasterize.mdx b/docs/docs/notebooks/h3-rasterize.mdx index 1326820e5..20ef25b15 100644 --- a/docs/docs/notebooks/h3-rasterize.mdx +++ b/docs/docs/notebooks/h3-rasterize.mdx @@ -35,7 +35,7 @@ The notebook uses the **lightweight tier** — pure Python/PySpark bindings (`da - **Databricks Runtime 17.3 LTS / 18.1+ or Serverless** (Spark 4 / Python 3.12). Lightweight default runs on Serverless. Session temp tables (Step 4) require Serverless or DBR 18.1+; they are not available on dedicated / single-user clusters. - **GeoBrix** (version 0.4.0). The `%pip install` cell installs the `geobrix[light,vizx]` wheel, which pulls in `rasterio` (used for the driver-side DEM read and isoband extraction), `h3` (polyfill), `matplotlib`, `geopandas`, and `mapclassify` (visualization). No JAR or GDAL init script is required. - **Unity Catalog Volume**: the DEM is staged to `/Volumes/geospatial_docs/geobrix/sample-data/geobrix-examples/sf/elevation/srtm_n37w123.tif`. Update `DEM_PATH` if your Volume layout differs. The Volume root must already exist; the staging cell creates sub-directories but not the root. -- **Wheel path**: update the `%pip install` cell to point at your staged `geobrix-0.4.3-py3-none-any.whl` if its Volume path differs from the default. +- **Wheel path**: update the `%pip install` cell to point at your staged `geobrix-0.5.0-py3-none-any.whl` if its Volume path differs from the default. --- diff --git a/docs/docs/quick-start.mdx b/docs/docs/quick-start.mdx index 73dc646a4..23efd019b 100644 --- a/docs/docs/quick-start.mdx +++ b/docs/docs/quick-start.mdx @@ -175,6 +175,13 @@ Register the lightweight readers once per session, then load by format string (` outputConstant="READ_GEOTIFF_LIGHT_output" /> +:::note Virtual tiles by default +The lightweight raster reader returns **[virtual tiles](./api/virtual-tiles)** by default — each row +carries a bytes-free `(path, window)` reference instead of the raster pixels, so a large file fans +into tiles without accumulating memory. Pass `.option("virtualTiles", "false")` (or use +`materialize=True` on a downstream `rst_*` call) when you need the bytes in-row. +::: + #### Read Shapefile 'file')` — native FILE refs, file metadata, recursive listing | +| `list_files` | DBR 18 LTS + | `list_files(…)` — metadata + FILE refs, directory listing only | +| `fuse` | All runtimes (OSS / DBR < 13) | `os.walk` + FUSE mount — always available; no FILE column | + +### Read-options matrix + +Measured on 1,000 raster tiles or 100k vector features; times are whole-job wall-clock. + +| Source | Read mechanism | Runtime | Serverless | Classic | Default / when | +|---|---|---|---|---|---| +| **FILE-column table** *(recommended on Serverless)* | Delta scan of `path` (EXTERNAL) | any | **~2 s** | ~2 s | default for a table source (`vector_file_read(table)`, raster `tilesTable`) | +| **FILE-column table** | Delta scan of FILE `.uri` (MANAGED) | DBR 13.3+ | **~2 s** | ~2 s | auto when the table is MANAGED | +| **Directory** | `read_files(format=>'file')` — enumerate + FILE refs + size | DBR 13.3+ | ~10 s / 10k *(enumerate)* | ~same | auto dir default on 13.3+ | +| **Directory** | `list_files` — enumerate (metadata only) | DBR 18+ | ~10 s / 10k *(enumerate)* | ~same | used when `read_files` is unavailable | +| **Directory** | FUSE walk + **per-tile open** (`raster_gbx` DataSource) | all | **~30 s / 1k** ⚠️ | ~20 s / 1k | fallback tile read (no FILE tier); slower than the Delta-table path on all runtimes | + +The `read_files`/`list_files` rows time the **listing** step (per 10k files, comparable, metadata-bound). The Delta-scan and DataSource rows time the actual tile **read/open** (per 1k tiles). The performance gap is the per-tile open/decode cost, not listing — the Delta-scan FILE-column-table path (~2 s) beats the DataSource directory scan by ~10× on classic and **~16–17× on Serverless** (~2 s vs. ~30 s per 1k tiles). **On Serverless, prefer reading from a FILE-column table (Delta scan).** See [Serverless & Memory](./serverless-and-memory) for the connect-aware stream caps (64 MiB on Serverless/Connect, 256 MiB on classic) and a full guide to memory-safe pipelines. + +The **no-gating rule**: the auto default (`access="auto"` on readers, `file_mode="auto"` on +writers) silently downgrades to the best available tier and never errors. Explicitly requesting +`"managed"` or `"external"` on a FUSE-only runtime raises a clear, actionable error +describing the upgrade path. Your pipelines run on any runtime without code changes; you +only need to opt in to FILE when you want the governance and lifecycle benefits. + +### Read resolver + +When a lightweight reader opens a source file it calls `open_for_read(source, access="auto")`. +The resolver checks the tier and picks a strategy: + +- **FILE-capable runtime (DBR 13.3+):** routes to the FILE API. Files at or under + `GBX_STREAM_MAX_BYTES` (default **64 MiB on Serverless/Connect**, 256 MiB on classic) + open a byte-range **stream** — one round-trip, no FUSE, no local copy, and the fastest + path for typical tile sizes. Files over the cap fall through to `as_local_file()` — a + lazy local-file reference backed by the FILE handle — which pages bytes from the Volume + on demand and is a **memory-safety mechanism, not a performance optimization**. A blanket + FUSE-direct path for small files is slower, not faster; the streaming path is preferred + whenever the file fits in the cap. Open dataset handles are held in a per-partition LRU + cache keyed by source path, amortising the open cost across all windows of the same source. +- **FUSE fallback (OSS / DBR < 13):** resolves to the bare `/Volumes/…` path and uses a + probe-then-stage strategy — tries random-access directly; falls back to a sequential + copy only if the probe fails. + +### Write committer + +Writers call `open_for_write(spark, df, target, file_mode="auto", filespace=…, layout=…)`. +`file_mode="auto"` resolves as follows: + +| Condition | Effective mode | Mechanism | +|---|---|---| +| FILE available + `filespace` given | `"managed"` | `create_file` — MANAGED FILE column | +| FILE available, no `filespace` | `"external"` | `try_to_file` — EXTERNAL FILE column | +| No FILE (fuse tier) | `"fuse"` | Plain Delta write, path STRING / raster BINARY | + +Explicit modes (`"managed"`, `"external"`, `"fuse"`) override auto-selection. Requesting +`"managed"` or `"external"` on a fuse-only runtime raises a clear error. + +### Layout options + +The `layout` parameter controls row ordering in the output Delta table: + +| Value | Behaviour | +|---|---| +| `"order"` | `ORDER BY path` at write time (default — scan-friendly) | +| `"cluster"` | `CLUSTER BY path` in the DDL (FILE-mode tables only); run `OPTIMIZE
` afterward for durable clustering | +| `"plain"` | No ordering — fastest write, scan order determined by the cluster | + +`partitionBy` is not supported; use `layout="order"` unless you have a specific clustering need. + +### File enumeration + +`enumerate_files(path, *, recursive=True, include_hidden=False, extensions=None, path_glob_filter=None)` +lists files in a directory and returns a Spark DataFrame (FILE-capable tiers) or a Python list (FUSE tier), +with columns `path`, `size`, and `file` (a FILE reference when available, `None` on FUSE). + +By default, files whose names start with `_` or `.` (Spark/Hadoop metadata files such as +`_SUCCESS`, `_committed_*`, `.crc`) are **skipped**. Pass `include_hidden=True` to include them. + +A positive selection filter — either `extensions` (a tuple of suffixes, e.g. `(".tif", ".nc")`) +or `path_glob_filter` (an fnmatch-style glob, e.g. `"*.tif"`) — narrows which files are returned. +The two filters are **mutually exclusive** and are ANDed with `include_hidden`: for example, +`include_hidden=True` + `path_glob_filter="[!.]*"` returns underscore-named files such as +`_data.tif` but still excludes dot-named files such as `.crc`. + +### Light-tier FILE flow + +![file_gbx access flow — the READ lane resolves a source (path, directory, or FILE ref) through open_for_read, a once-per-session capability-tier probe (read_files on DBR 13.3+, list_files on DBR 18+, FUSE floor), and size-adaptive routing (FILE byte-range stream, FUSE-of-FILE for large or striped files, or a probe-then-stage copy) into the rasterio / pyogrio reader, where a per-partition open-LRU amortizes the open cost across a source’s windows; the WRITE lane resolves writer output through open_for_write into MANAGED (create_file), EXTERNAL (try_to_file), or FUSE (plain Delta) modes and their targets — a FILE-column Delta table or a Volume path — with ORDER BY path by default and opt-in CLUSTER BY plus OPTIMIZE; access="auto" downgrades to FUSE gracefully while an explicit FILE mode on a FUSE-only runtime raises a clear error](../../resources/images/diagrams/rasterx/file-gbx-flow.png) + +See [`file_gbx` Reader](./readers/file) for enumeration and read-mode details, and +[`file_gbx` Writer](./writers/file) for write-mode and ingest options. + ## Next Steps - [Readers Overview](./readers/overview) — every reader, tier differences, output schemas, options. - [Writers Overview](./writers/overview) — the column contract, single-file vs sharded writers, per-format details. +- [VRT & Mosaics](./api/vrt-mosaic) — tile a large source into bounded mini-COGs + a portable VRT index; read back with `raster_gbx` or `cog_gbx`. +- [Serverless & Memory](./serverless-and-memory) — connect-aware caps, materialize-vs-virtual, memory-safe write patterns, and caveats. - [Benchmarking](./api/benchmarking) — tier-vs-tier timing and parity methodology. diff --git a/docs/docs/readers/cog.mdx b/docs/docs/readers/cog.mdx new file mode 100644 index 000000000..c7a8596c8 --- /dev/null +++ b/docs/docs/readers/cog.mdx @@ -0,0 +1,170 @@ +--- +sidebar_position: 3 +sidebar_label: COG +--- + +import CodeFromTest from '@site/src/components/CodeFromTest'; +import cogExamples from '!!raw-loader!../../tests/python/readers/cog_gbx_examples.py'; + +# COG Reader + +`cog_gbx` reads Cloud-Optimized GeoTIFFs (COGs) into the shared `(source, tile)` +schema. Its defining feature is **area-of-interest clipping**: pass one or more +clip polygons and the reader issues HTTP range-requests (or FUSE range-reads) +that fetch only the bytes that intersect each AOI, skipping the rest of the file +entirely. + +This makes `cog_gbx` the natural read path after COG preparation with the +`cog_gbx` writer — the prepare-then-read pipeline: + +1. **List** source files with `file_gbx`. +2. **Prepare** — convert each to a master COG with the `cog_gbx` writer + (internal tiling + overview levels baked in). +3. **Read** — clip to any AOI with `cog_gbx` + `clipPolygons`, pulling only the + required tiles and overview levels. + +`cog_gbx` also reads a **VRT mosaic**: point it at a `mosaic.vrt` index (written +by the [`cog_gbx` writer's mosaic mode](../writers/cog#mosaic-mode)) and the +reader expands the index into one virtual tile per member mini-COG, so `rst_*` +functions process the whole mosaic per-tile. See +[Reading a VRT mosaic](#reading-a-vrt-mosaic) below and the +[VRT & Mosaics](../api/vrt-mosaic) reference. + +:::note Lightweight only +`cog_gbx` is a pure-Python lightweight DataSource (no JAR). Register it with +`register(spark)` before use. +::: + +## Options + +| Option | Default | Description | +|--------|---------|-------------| +| `virtualTiles` | `"true"` | **Default.** Emit bytes-free **virtual tiles** — each row carries the source `path` + pixel `window` instead of raster bytes; pixels are read lazily when an operation needs them (the ingest-OOM-dissolving default for the light tier). Set `"false"` to materialize raster bytes into each row. See [Virtual Tiles](../api/virtual-tiles). | +| `clipPolygons` | _none_ | Area(s) of interest: one WKT/EWKT string, or a JSON-array string for a list, e.g. `'["POLYGON((...))","POLYGON((...))"]'`. One tile per polygon whose envelope intersects the COG; only the intersecting blocks/overviews are fetched. Materialized tiles are pre-clipped (NoData outside the polygon); virtual tiles defer the clip. Mutually exclusive with `windows`. | +| `windows` | _none_ | Pixel window(s): JSON 4-int array `"[col,row,w,h]"`, or a JSON array of them. Mutually exclusive with `clipPolygons`/`tileSize`. | +| `clipCrs` | _none_ | CRS for `clipPolygons` lacking an embedded SRID (e.g. `"EPSG:27700"`). Precedence: embedded EWKB/EWKT SRID → `clipCrs` → the COG's CRS. The reader reprojects the polygon internally. | +| `splitStrategy` | `"none"` | Split large COGs: `none` (default — one tile per file), `serverless`, `classic`, or `auto`. Splitting respects the COG's internal tile grid. | +| `sizeInMB` | `"-1"` | Power-user override: set a positive value to pin the per-tile budget in MiB. `-1` defers to `splitStrategy`. | +| `tileSize` | _none_ | Regular fixed-size grid over the COG: `"w,h"` or a single `"n"` (square). One tile per cell; mutually exclusive with `clipPolygons`/`windows`. Materialized cells guarded to ~2 GB; virtual unguarded. See [Raster Options](./raster#options). | +| `overlapPercent` | `0` | Overlap % between `tileSize` cells (`tileSize`-only). See [Raster Options](./raster#options). | +| `filterRegex` | `".*"` | When loading a directory, keep files whose full path matches this regex. | + +:::note Virtual tiles are the default +Light raster readers now emit **virtual tiles** by default (`virtualTiles=true`) — bytes-free +`(path, window)` references that read pixels lazily. Previously the reader materialized raster +bytes into every row. To restore materialized reads, pass `.option("virtualTiles", "false")`. +A virtual tile passed to a heavyweight function must be materialized first — see +[Virtual Tiles](../api/virtual-tiles). +::: + +:::tip Default is no-split +Both `raster_gbx` and `cog_gbx` default to `splitStrategy=none` — one tile per +file. Splitting is opt-in. COG files prepared by the `cog_gbx` writer already +carry internal tiling and overviews, so the reader can serve any AOI without +splitting the source into partitions. +::: + +## Register + +```python +from databricks.labs.gbx.ds.register import register +register(spark) +``` + +## Read a COG directory + +```python +from databricks.labs.gbx.ds.register import register +register(spark) + +df = spark.read.format("cog_gbx").load("/Volumes/main/.../cog-prepared/nyc-sentinel2") +df.show() +``` + +## Read with AOI clip + + + +## Output schema + +`cog_gbx` emits the standard raster schema: + +``` +root + |-- source: string — path to the source COG file + |-- tile: struct + | |-- cellid: bigint (nullable) + | |-- raster: binary (nullable) — clipped tile bytes (GeoTIFF); null for virtual tiles + | |-- path: string (nullable) — source path (provenance / virtual read target) + | |-- window: struct (nullable) — pixel window read + | |-- clip_polygon: binary (nullable) — the AOI applied (materialized) or to apply (virtual) + | |-- clip_crs: string (nullable) — CRS of clip_polygon + | |-- crs: string (nullable) — working/target CRS + | |-- metadata: map — driver, CRS, extent, … +``` + +For a materialized tile the `raster` bytes carry the CRS and geotransform of the +clipped window (not the full source file), and `window`/`clip_polygon`/`clip_crs` +are provenance of what was applied. A virtual tile (from `.option("virtualTiles", +"true")`) leaves `raster` null and carries those fields as a deferred instruction. +Downstream `rst_*` functions consume either identically. + +## Full prepare-then-read pipeline + + + + + +## Reading a VRT mosaic + +When the load path points **directly at a `mosaic.vrt`** (can be written by the +[`cog_gbx` writer's mosaic mode](../writers/cog#mosaic-mode)), the reader +parses the VRT XML, enumerates its member paths, and emits **one virtual tile row +per member mini-COG** — each row a whole-file reference (`raster=NULL`, `path` +set to the member, `window=NULL`). All downstream `rst_*` functions then run +per-tile, exactly as for a directory of flat COGs: + +```python +from databricks.labs.gbx.ds.register import register +register(spark) + +# One virtual tile per member mini-COG +df = spark.read.format("cog_gbx").load("/Volumes/.../mosaic/mosaic.vrt") +``` + +**Options that apply to a VRT load:** + +| Option | Effect on a VRT load | +|--------|----------------------| +| _`.vrt` path recognition_ | Pointing the load at a `mosaic.vrt` triggers member expansion (one row per tile). Pointing at the **containing directory** instead walks the directory for raster files and skips `.vrt` indexes, so the tiles are read directly. Both yield one row per tile; the VRT path is the canonical way to load a mosaic. | +| `clipPolygons` + `clipCrs` | Restrict the expansion to only the members whose extent intersects the area of interest — non-intersecting mini-COGs are skipped without being opened. `clipCrs` supplies the CRS for a polygon lacking an embedded SRID. | +| `virtualTiles` | Applies as elsewhere (default `"true"`). Members are emitted as bytes-free virtual tiles; set `"false"` to materialize member bytes into each row. | + +Full detail — including `mint_vrt` for on-demand transient VRTs — is on the +[VRT & Mosaics](../api/vrt-mosaic) page. + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this reader and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Used | All directory reads — `recursive`, `include_hidden`, `extensions`, `path_glob_filter` options are routed through this shared predicate. | +| `enumerate_files` (FILE-tier enumeration) | Not in the DataSource | The DataSource is session-less on Connect and does not call `enumerate_files`; FILE-tier enumeration is only available through `gbx_file_read` at the function layer. | +| `gbx_file_read` / `gbx_file_write` (FILE tier) | Not in the DataSource | The DataSource is FUSE-only (session-less on Connect); FILE reads go through `gbx_file_read` → `rst_fromfile` at the function layer. | + +:::note Shared file-access layer +Lightweight readers use the shared [`file_gbx` file-access base](../readers-writers#file-gbx) for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there. +::: + +## Next steps + +- [File Lister](./file) — list source files before preparation +- [COG Writer](../writers/cog) — prepare master COGs from source rasters +- [VRT & Mosaics](../api/vrt-mosaic) — load a `mosaic.vrt` to expand a tile directory into virtual tile rows +- [Raster Functions](../api/raster-functions) — process the clipped tiles diff --git a/docs/docs/readers/file.mdx b/docs/docs/readers/file.mdx new file mode 100644 index 000000000..a9800a1da --- /dev/null +++ b/docs/docs/readers/file.mdx @@ -0,0 +1,209 @@ +--- +sidebar_position: 2 +sidebar_label: File Lister (file_gbx) +--- + +import CodeFromTest from '@site/src/components/CodeFromTest'; +import cogExamples from '!!raw-loader!../../tests/python/readers/cog_gbx_examples.py'; + +# File Access — `file_gbx` + +`file_gbx` has two roles: + +1. **File lister DataSource** — emits one row per file (path, name, extension, size, + modification time). Use it as the first step in a preparation pipeline: list the + files you want to process, then pipe those references into a writer such as `cog_gbx`. +2. **Shared file-access base** — the Python module that every lightweight reader and + writer uses internally for FILE / FUSE routing, directory enumeration, and write-mode + selection. See the [Readers & Writers — Shared file-access base](../readers-writers#file-gbx) + section for the capability tier diagram and the no-gating rule. + +`file_gbx` is format-agnostic: it lists any file type and never decodes file content. + +:::note Lightweight only +`file_gbx` is a pure-Python lightweight DataSource. Register it with `register(spark)` +before use. There is no heavyweight GDAL counterpart. +::: + +## Output schema (DataSource) + +``` +root + |-- path: string — full absolute path to the file + |-- name: string — filename including extension + |-- extension: string — lowercase, no leading dot; NULL for files with no extension + |-- size: long — file size in bytes + |-- modificationTime: timestamp — last-modified time +``` + +No raster or vector content is loaded. The `path` values are the references you pass to a +subsequent writer. + +## Register + +```python +from databricks.labs.gbx.ds.register import register +register(spark) +``` + +## DataSource options + +| Option | Default | Description | +|--------|---------|-------------| +| `filterRegex` | `".*"` | Keep only files whose full path matches this regex. | + +:::note Always recursive +The `file_gbx` DataSource always walks subdirectories. Use `filterRegex` to scope which +files are included. +::: + +## List files + + + +## Filter by extension + + + +## Typical use: COG preparation + +The primary use of `file_gbx` is to feed the `cog_gbx` writer: list the source files, +then write master COGs — one per source file — for efficient windowed reading later. + + + +After preparation, read clipped windows with the `cog_gbx` reader. See +[COG Reader](./cog) for the read path. + +--- + +## Python API + +The `file_gbx` module also exposes a Python API for direct use in UDFs, scripts, and +pipeline code. + +### `enumerate_files` — directory listing + +```python +from databricks.labs.gbx.ds.file_gbx import enumerate_files + +# Basic: list all non-hidden files recursively (default) +files = enumerate_files("/Volumes/main/geo/rasters", spark=spark) + +# Filter to GeoTIFFs only +tifs = enumerate_files( + "/Volumes/main/geo/rasters", + extensions=(".tif", ".tiff"), + spark=spark, +) + +# Include Hadoop metadata files (_SUCCESS, _committed_*, .crc, …) +all_files = enumerate_files( + "/Volumes/main/geo/rasters", + include_hidden=True, + spark=spark, +) +``` + +On FILE-capable runtimes (DBR 13.3+), `enumerate_files` returns a Spark DataFrame with +columns `path`, `size`, and `file` (a FILE reference). On FUSE-only runtimes it returns a +Python list of `{path, size, file}` dicts where `file` is `None`. + +#### Hidden-file filtering + +By default, files whose names start with `_` or `.` are **skipped** — this matches +Spark/Hadoop conventions where `_SUCCESS`, `_committed_*`, `_delta_log`, and `.crc` are +metadata artefacts, not data files. Pass `include_hidden=True` to re-admit them. + +#### Positive selection filters + +Use `extensions` or `path_glob_filter` to narrow which files are returned. The two +parameters are **mutually exclusive** — providing both raises `ValueError`. + +- **`extensions`**: a tuple of case-insensitive suffixes, e.g. `(".tif", ".nc")`. Sugar + for `path_glob_filter` — compiled internally to `["*.tif", "*.nc"]`. +- **`path_glob_filter`**: an fnmatch-style glob applied to each file's basename, e.g. + `"*.tif"` or `"[!.]*"`. + +The filter is **ANDed with `include_hidden`**: setting `include_hidden=True` + +`path_glob_filter="[!.]*"` includes underscore-named files (`_data.tif`) but still +excludes dot-named files (`.crc`, `.DS_Store`). + +```python +# Include _data.tif but exclude .crc / .DS_Store +filtered = enumerate_files( + "/Volumes/main/geo/rasters", + include_hidden=True, + path_glob_filter="[!.]*", # starts with any char that is NOT '.' + spark=spark, +) +``` + +### Capability tiers and the no-gating rule + +`file_access_tier(spark)` returns the best tier available at runtime: + +```python +from databricks.labs.gbx.ds.file_gbx import file_access_tier + +tier = file_access_tier(spark) +# Returns: "read_files" (DBR 13.3+), "list_files" (DBR 18+), or "fuse" (always) +``` + +`open_for_read(source, access="auto")` is the read resolver every lightweight reader calls: +it validates the access mode and enforces the no-gating rule (the size-adaptive routing — +FILE byte-range stream, FUSE-of-FILE, or staging — runs in the reader layer on top of it). +With `access="auto"` it accepts the best available tier and never errors. Passing +`access="managed"` or `access="external"` on a FUSE-only runtime raises a clear error: + +```python +from databricks.labs.gbx.ds.file_gbx import open_for_read + +# Auto — never errors regardless of runtime: +path = open_for_read("/Volumes/main/geo/scene.tif", spark=spark) + +# Explicit FILE — raises ValueError on FUSE-only runtimes: +path = open_for_read( + "/Volumes/main/geo/scene.tif", + access="managed", + spark=spark, +) +``` + +### Ingest existing files into a managed FILE-column table + +`ingest_files` reads files from an external Volume path via `read_files(format=>'file')` +and inserts them as FILE MANAGED references into a Delta table, without copying the bytes: + +```python +from databricks.labs.gbx.ds.file_gbx import ingest_files + +ingest_files( + spark, + src="/Volumes/main/geo/archive/rasters", + target="main.geo.raster_registry", + filespace="/Volumes/main/geo/managed_store", + file_col="tile_file", # name of the FILE-typed column + layout="order", # ORDER BY path (default) + recursive=True, + overwrite=False, # CREATE TABLE IF NOT EXISTS (idempotent) +) +``` + +`ingest_files` requires a FILE-capable runtime (DBR 13.3+). On FUSE-only runtimes it +raises `ValueError` with an upgrade message — use +[`open_for_write(file_mode="fuse")`](../writers/file) for a plain Delta write instead. + +## Next steps + +- [COG Reader](./cog) — windowed read from prepared COGs +- [COG Writer](../writers/cog) — convert source files to master COGs +- [Raster Reader](./raster) — decode rasters into the `(source, tile)` schema +- [`file_gbx` Writer](../writers/file) — write-mode and ingest options +- [Shared file-access base](../readers-writers#file-gbx) — capability tiers and the no-gating rule diff --git a/docs/docs/readers/filegdb.mdx b/docs/docs/readers/filegdb.mdx index 195f4624e..3855fc7fc 100644 --- a/docs/docs/readers/filegdb.mdx +++ b/docs/docs/readers/filegdb.mdx @@ -171,3 +171,18 @@ The OpenFileGDB driver provides read-only access. You cannot create, modify, or - [Check Examples](../examples/overview) - [Other Readers](./overview) - [Learn about VectorX](../api/vectorx-functions) + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this reader and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Used | Directory reads — `recursive`, `include_hidden`, `extensions`, and `path_glob_filter` are routed through this shared predicate for all format-specific readers. | +| `gbx_file_read` / FILE-tier read | Not in the DataSource | The DataSource is session-less on Connect; FILE-tier reads go through `vector_file_read` (function layer) which injects `_file_ref` on the driver before `mapInPandas`. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | FILE-tier writes go through `vector_file_write` (function layer). The DataSource writer commits via FUSE. | + +:::note Shared file-access layer +Lightweight readers use the shared [`file_gbx` file-access base](../readers-writers#file-gbx) for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there. +::: diff --git a/docs/docs/readers/geojson.mdx b/docs/docs/readers/geojson.mdx index 0d4089856..a6459e6c7 100644 --- a/docs/docs/readers/geojson.mdx +++ b/docs/docs/readers/geojson.mdx @@ -187,3 +187,18 @@ root - [Check Examples](../examples/overview) - [Other Readers](./overview) - [Learn about VectorX](../api/vectorx-functions) + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this reader and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Used | Directory reads — `recursive`, `include_hidden`, `extensions`, and `path_glob_filter` are routed through this shared predicate for all format-specific readers. | +| `gbx_file_read` / FILE-tier read | Not in the DataSource | The DataSource is session-less on Connect; FILE-tier reads go through `vector_file_read` (function layer) which injects `_file_ref` on the driver before `mapInPandas`. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | FILE-tier writes go through `vector_file_write` (function layer). The DataSource writer commits via FUSE. | + +:::note Shared file-access layer +Lightweight readers use the shared [`file_gbx` file-access base](../readers-writers#file-gbx) for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there. +::: diff --git a/docs/docs/readers/geopackage.mdx b/docs/docs/readers/geopackage.mdx index 93eb48e24..2869c476f 100644 --- a/docs/docs/readers/geopackage.mdx +++ b/docs/docs/readers/geopackage.mdx @@ -146,3 +146,18 @@ root - [Check Examples](../examples/overview) - [Other Readers](./overview) - [Learn about VectorX](../api/vectorx-functions) + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this reader and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Used | Directory reads — `recursive`, `include_hidden`, `extensions`, and `path_glob_filter` are routed through this shared predicate for all format-specific readers. | +| `gbx_file_read` / FILE-tier read | Not in the DataSource | The DataSource is session-less on Connect; FILE-tier reads go through `vector_file_read` (function layer) which injects `_file_ref` on the driver before `mapInPandas`. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | FILE-tier writes go through `vector_file_write` (function layer). The DataSource writer commits via FUSE. | + +:::note Shared file-access layer +Lightweight readers use the shared [`file_gbx` file-access base](../readers-writers#file-gbx) for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there. +::: diff --git a/docs/docs/readers/geotiff.mdx b/docs/docs/readers/geotiff.mdx index fa859e15f..5bc9caf24 100644 --- a/docs/docs/readers/geotiff.mdx +++ b/docs/docs/readers/geotiff.mdx @@ -32,13 +32,35 @@ Both named readers preset the GeoTIFF driver and inherit their respective generi ### Lightweight (`gtiff_gbx`) -Inherits the [lightweight `raster_gbx` options](./raster#options): +Inherits the [lightweight `raster_gbx` options](./raster#options). Key options: | Option | Default | Description | |--------|---------|-------------| -| `sizeInMB` | `"-1"` | Default (`<= 0`) = no split: one whole-image tile per file. Set a positive MB value to tile large rasters into multiple tiles. | +| `virtualTiles` | `"true"` | **Default.** Emit bytes-free **virtual tiles** — each row carries the source `path` + pixel `window` instead of raster bytes; pixels are read lazily when an operation needs them (the ingest-OOM-dissolving default for the light tier). Set `"false"` to materialize raster bytes into each row. See [Virtual Tiles](../api/virtual-tiles). | +| `splitStrategy` | `"none"` | When to split large rasters: `none` (default — one tile per file), `auto`, `serverless`, `classic`. See [Raster Options](./raster#options). | +| `sizeInMB` | `"-1"` | Power-user budget override in MiB (positive value). `-1` = use `splitStrategy`. | +| `clipPolygons` | _none_ | Area(s) of interest: one WKT/EWKT string, or a JSON-array string for a list. One tile per intersecting polygon; mutually exclusive with `windows`/`tileSize`. See [Raster Options](./raster#options). | +| `windows` | _none_ | Pixel window(s): JSON 4-int array `"[col,row,w,h]"`, or a JSON array of them. Mutually exclusive with `clipPolygons`/`tileSize`. | +| `clipCrs` | _none_ | CRS for `clipPolygons` lacking an embedded SRID (embedded SRID → `clipCrs` → raster CRS). | +| `tileSize` | _none_ | Regular fixed-size grid: `"w,h"` or a single `"n"` (square). One tile per cell; mutually exclusive with `clipPolygons`/`windows`. Materialized cells guarded to ~2 GB; virtual unguarded. See [Raster Options](./raster#options). | +| `overlapPercent` | `0` | Overlap % between `tileSize` cells (`tileSize`-only). See [Raster Options](./raster#options). | | `filterRegex` | `".*"` | When loading a directory, keep files whose full path matches this regex. | +:::note Virtual tiles are the default +Light raster readers now emit **virtual tiles** by default (`virtualTiles=true`) — bytes-free +`(path, window)` references that read pixels lazily. Previously the reader materialized raster +bytes into every row. To restore materialized reads, pass `.option("virtualTiles", "false")`. +A virtual tile passed to a heavyweight function must be materialized first — see +[Virtual Tiles](../api/virtual-tiles). +::: + +:::note COG output is a writer concern +The `tileFormat`, `cogBlockSize`, and `cogOverviewResampling` reader options are +removed. COG creation belongs to the `cog_gbx` writer — see +[COG Writer](../writers/cog). The `gtiff_gbx` writer still accepts `cog=true` to +re-encode a tile DataFrame as COG. +::: + ### Heavyweight (`gtiff_gdal`) Inherits all [heavyweight `gdal` reader options](./raster#options). Common options include: @@ -229,3 +251,18 @@ GeoBrix reads compressed GeoTIFFs transparently. The compression format is autom - [GDAL Reader](./raster) - Generic raster reader for all GDAL formats - [Raster Functions](../api/raster-functions) - Raster processing operations - [Quick Start](../quick-start) - Get started with GeoBrix + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this reader and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Used | All directory reads — `recursive`, `include_hidden`, `extensions`, `path_glob_filter` options are routed through this shared predicate. | +| `enumerate_files` (FILE-tier enumeration) | Not in the DataSource | The DataSource is session-less on Connect and does not call `enumerate_files`; FILE-tier enumeration is only available through `gbx_file_read` at the function layer. | +| `gbx_file_read` / `gbx_file_write` (FILE tier) | Not in the DataSource | The DataSource is FUSE-only (session-less on Connect); FILE reads go through `gbx_file_read` → `rst_fromfile` at the function layer. | + +:::note Shared file-access layer +Lightweight readers use the shared [`file_gbx` file-access base](../readers-writers#file-gbx) for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there. +::: diff --git a/docs/docs/readers/netcdf.mdx b/docs/docs/readers/netcdf.mdx index be2d081f0..24588d998 100644 --- a/docs/docs/readers/netcdf.mdx +++ b/docs/docs/readers/netcdf.mdx @@ -68,6 +68,12 @@ Raw sensor-geometry products (no per-pixel lon/lat) are rejected with an actionable error rather than silently guessed at. ::: +:::note NetCDF readers always materialize tiles +NetCDF raster reads are multidimensional per-variable reads that materialize tile +bytes directly into each row. Virtual-tile support (`virtualTiles`) for the NetCDF +readers is not part of this release. +::: + ### Options | Option | Default | Description | @@ -192,3 +198,18 @@ guarantees. - [GeoTIFF Reader](./geotiff) — for COG/GeoTIFF rasters (e.g. Sentinel-2, EMIT CH4). - [Raster Functions](../api/raster-functions) — tessellation, band math, tiling. - [Execution Tiers](../api/execution-tiers) — lightweight vs. heavyweight. + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this reader and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Used | All directory reads — `recursive`, `include_hidden`, `extensions`, `path_glob_filter` options are routed through this shared predicate. | +| `enumerate_files` (FILE-tier enumeration) | Not in the DataSource | The DataSource is session-less on Connect and does not call `enumerate_files`; FILE-tier enumeration is only available through `gbx_file_read` at the function layer. | +| `gbx_file_read` / `gbx_file_write` (FILE tier) | Not in the DataSource | The DataSource is FUSE-only (session-less on Connect); FILE reads go through `gbx_file_read` → `rst_fromfile` at the function layer. | + +:::note Shared file-access layer +Lightweight readers use the shared [`file_gbx` file-access base](../readers-writers#file-gbx) for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there. +::: diff --git a/docs/docs/readers/overview.mdx b/docs/docs/readers/overview.mdx index 717a84800..a2acbd386 100644 --- a/docs/docs/readers/overview.mdx +++ b/docs/docs/readers/overview.mdx @@ -14,13 +14,13 @@ GeoBrix provides Spark readers for geospatial file formats. The lightweight tier ships native Python DataSource V2 readers — no JAR, no init script. -:::tip Why this scales beyond a single node -These are Spark DataSource V2 readers, not single-node `rasterio`/`pyogrio` wrappers. The -work is **partitioned and read in parallel across the cluster** — vector readers slice -features by `chunkSize`, raster readers can split large files by `sizeInMB` — and the result is -a distributed DataFrame ready for joins and aggregations with no driver-side `collect`. A -single-node `pyogrio.read_*` or `rasterio.open` reads one file sequentially on one machine; -these readers fan the same work across executors and scale past a single machine's memory. +:::tip Distributed readers, virtual tiles by default +These are **Spark DataSource V2** readers (not single-node `rasterio`/`pyogrio` wrappers): work is +partitioned and read in parallel across the cluster — vector readers slice features by `chunkSize`, +raster readers split large files by `sizeInMB` — returning a distributed DataFrame with no driver-side +`collect`. Raster readers load **virtual tiles** by default — bytes-free `(path, window)` refs that read +pixels lazily, so a multi-gigabyte raster fans into tiles without OOM (`.option("virtualTiles", "false")` +for materialized reads). See **[Virtual Tiles](../api/virtual-tiles)**. ::: :::note Register first @@ -42,22 +42,83 @@ register(spark, only=["raster_gbx", "gtiff_gbx"]) An unrecognized format raises `ValueError`. ::: +:::tip On Serverless, read from a FILE-column table +The **Delta-scan FILE-column-table read path** is the recommended read path on Serverless: a `tilesTable` +(raster) or `vector_file_read(table)` (vector) scans the Delta table and resolves FILE references without +per-tile opens — measured at **~1.8–2 s** for 1,000 raster tiles or 100k vector features on both +Serverless and classic clusters. + +The `raster_gbx` **directory DataSource path** incurs a per-tile open and is **~16–17× slower on Serverless** +(~30 s vs. ~2 s per 1,000 tiles). When running on Serverless, write your rasters or vectors to a FILE-column +table first, then read from that table rather than the directory directly. + +All tile reads are size-gated by a **connect-aware cap** (64 MiB on Serverless / Spark Connect, 256 MiB on +classic; override with `GBX_STREAM_MAX_BYTES`). Files under the cap stream in full; files over the cap open +lazily so reads never load an unbounded tile into executor memory. See [Serverless & Memory](../serverless-and-memory) +for the full guide including write-path safety and caveats. + +See the full [capability tiers & read-options matrix](../readers-writers#capability-tiers). +::: + ### Available Readers | Reader | Format Name | Description | |--------|-------------|-------------| | [Raster Reader](./raster) | `raster_gbx` | Pure-Python catch-all raster reader (no JAR; DataSource V2) | | [GeoTIFF Reader](./geotiff) | `gtiff_gbx` | Pure-Python GeoTIFF reader (preset `driver="GTiff"`) | +| [COG Reader](./cog) | `cog_gbx` | Pure-Python cloud-optimized GeoTIFF reader — windowed/overview-aware reads | +| [NetCDF Reader](./netcdf) | `netcdf_gbx` | Pure-Python CF NetCDF reader — CF grids (raster) or CF-DSG points (vector) | +| [File Reader](./file) | `file_gbx` | Pure-Python file lister — enumerates matching paths as rows (read-only) | | [PMTiles Reader](./pmtiles) | `pmtiles_gbx` | Pure-Python PMTiles reader — mosaic pyramid from rasters (`source="raster"`) or tiles from an archive (`source="archive"`) | | [Vector Reader](./vector) | `vector_gbx` | Pure-Python catch-all vector reader (pyogrio; same OGR schema) | | [Shapefile Reader](./shapefile) | `shapefile_gbx` | Pure-Python Shapefile reader (preset OGR driver) | | [GeoJSON Reader](./geojson) | `geojson_gbx` | Pure-Python GeoJSON reader (preset OGR driver) | +| [GeoJSONL Reader](../writers/geojsonl) | `geojsonl_gbx` | Pure-Python newline-delimited GeoJSONL reader — reads GeoJSONSeq shard directories | | [GeoPackage Reader](./geopackage) | `gpkg_gbx` | Pure-Python GeoPackage reader (preset OGR driver) | | [GeoDatabase Reader](./filegdb) | `file_gdb_gbx` | Pure-Python File Geodatabase reader (preset OGR driver) | See the [Raster Reader](./raster) page for full raster usage/options, and the vector reader pages for the OGR `(geom_0, geom_0_srid, geom_0_srid_proj, …attributes)` schema. +:::note `.limit()` and filters are not pushed into the reader — use the function layer for a bounded scan +The lightweight DataSource readers **enumerate the entire source at planning time**, regardless of a +downstream `.limit(N)`, `WHERE`, or column projection. Spark 4.0.0's Python DataSource API exposes no +limit/filter **pushdown** hook, so `spark.read.format("raster_gbx").load(dir).limit(10)` still lists — +and plans a partition for — *every* file under `dir`, then discards all but 10 rows afterward. The +listing is a fast, stat-free directory walk, so this is cheap for typical directories; on a very large +directory, every file is still planned. + +When you want the limit (or a filter) honored **during planning**, use the function layer instead: + +- **`gbx_file_read(spark, dir).limit(N)`** — on a FILE-capable runtime this reads through the native + `read_files` table function, which pushes `.limit()` and predicate/column filters, so only the needed + files are enumerated. +- **`rst_fromfile("path")`** — a per-row function returning virtual tiles, so + `gbx_file_read(dir).limit(N).select(..., rst_fromfile("path"))` decodes only the N selected rows and + reads no pixels until a downstream op forces it. + +See **[GBX Common Functions](../common-functions)**. +::: + +### FILE-column-table reads {#file-column-table-reads} + +Both the raster and vector light-tier readers can read directly from a FILE-column Delta table +(a table whose rows carry FILE references to geospatial files) in addition to Volume paths. +Both auto-order by the resolved source path by default. The ordering effect differs by format: + +- **Raster (`tilesTable`)** — a single GeoTIFF can produce many tile rows; auto-ordering + groups those tiles together so each executor opens the source file once and reads all its + windows, amortizing the open cost. See [Raster Reader → tilesTable auto-order](./raster#tiles-table-auto-order). +- **Vector (`vector_file_read`)** — one vector file maps to exactly one table row (one FILE + ref). Every source opens exactly once regardless of row order, so auto-ordering is a + **deterministic, cross-format-consistent default**, not a throughput lever. See + [Vector Reader → Reading from a FILE-column table](./vector#vector-file-table-read). + +Both support an opt-out: `skipOrdering="true"` (raster DataSource option) and +`skip_ordering=True` (`vector_file_read` kwarg). Use either when the table is already +physically ordered (written with `layout="order"` or `layout="cluster"`) to avoid the +redundant sort step. + ## Benchmarks Each `*_gbx` lightweight reader corresponds to a `*_ogr` (or `gdal`/`gtiff_gdal`) heavyweight @@ -85,6 +146,7 @@ Heavyweight readers are implemented as Spark DataSource V2 connectors backed by | Reader | Format Name | Description | |--------|-------------|-------------| | [Raster Reader](./raster) | `gdal` / `gtiff_gdal` | GDAL-backed raster readers (generic and GeoTIFF named) | +| [NetCDF Reader](./netcdf) | `netcdf_gdal` | GDAL-backed named reader for NetCDF grids (raster) | ### Vector Readers (OGR-based) @@ -95,6 +157,7 @@ Heavyweight readers are implemented as Spark DataSource V2 connectors backed by | [GeoJSON Reader](./geojson) | `geojson_ogr` | Named reader for GeoJSON/GeoJSONSeq | | [GeoPackage Reader](./geopackage) | `gpkg_ogr` | Named reader for GeoPackage | | [FileGDB Reader](./filegdb) | `file_gdb_ogr` | Named reader for ESRI File Geodatabase | +| [NetCDF Reader](./netcdf) | `netcdf_ogr` | OGR-backed named reader for NetCDF as vector features (CF-DSG points) | ## Basic Usage diff --git a/docs/docs/readers/pmtiles.mdx b/docs/docs/readers/pmtiles.mdx index d30ff8053..0332ce63c 100644 --- a/docs/docs/readers/pmtiles.mdx +++ b/docs/docs/readers/pmtiles.mdx @@ -25,7 +25,7 @@ straight into a write with no reshaping. :::note Lightweight-only read path `pmtiles_gbx` is a pure-Python DataSource V2 reader: **no JAR, no init script**, and it runs on Serverless, standard (shared), and ARM clusters. The -`beta-release-notes` note that PMTiles "read is not yet supported" refers to the +Release Notes' note that heavyweight PMTiles read "is not supported" refers to the **heavyweight** `spark.read.format("pmtiles")` path — the lightweight `pmtiles_gbx` reader documented here **is** a supported read path. ::: @@ -104,3 +104,18 @@ stores — PNG/JPEG/WebP/MVT). - [PMTiles Writer](../writers/pmtiles) — package `(z, x, y, bytes)` rows into archives. - [VizX — PMTiles viewer](../api/vizx-pmtiles) — inspect a `.pmtiles` archive in a notebook. - [Helios notebooks](../notebooks/helios) — end-to-end tiling to PMTiles over a San Francisco AOI. + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this reader and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Used | All directory reads — `recursive`, `include_hidden`, `extensions`, `path_glob_filter` options are routed through this shared predicate. | +| `enumerate_files` (FILE-tier enumeration) | Not in the DataSource | The DataSource is session-less on Connect and does not call `enumerate_files`; FILE-tier enumeration is only available through `gbx_file_read` at the function layer. | +| `gbx_file_read` / `gbx_file_write` (FILE tier) | Not in the DataSource | The DataSource is FUSE-only (session-less on Connect); FILE reads go through `gbx_file_read` → `rst_fromfile` at the function layer. | + +:::note Shared file-access layer +Lightweight readers use the shared [`file_gbx` file-access base](../readers-writers#file-gbx) for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there. +::: diff --git a/docs/docs/readers/raster.mdx b/docs/docs/readers/raster.mdx index 3ab6be633..5e67ad383 100644 --- a/docs/docs/readers/raster.mdx +++ b/docs/docs/readers/raster.mdx @@ -7,6 +7,7 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeFromTest from '@site/src/components/CodeFromTest'; import rasterGbxExamples from '!!raw-loader!../../tests/python/readers/raster_gbx_read_examples.py'; +import largeRasterExamples from '!!raw-loader!../../tests/python/readers/large_raster_examples.py'; import gdalExamples from '!!raw-loader!../../tests/python/readers/gdal_examples.py'; import gdalScala from '!!raw-loader!../../tests/scala/readers/GDALExamples.scala'; @@ -35,16 +36,128 @@ the [Benchmarking](../api/benchmarking) page for light-vs-heavy timings and meth ### Lightweight (`raster_gbx`) +The lightweight reader defaults to **no splitting** (`splitStrategy=none`): +one whole-image tile per file. Splitting is opt-in; COG creation is a +separate concern handled by the `cog_gbx` writer. See +[COG Reader](./cog) and [COG Writer](../writers/cog) for the +prepare-then-read pipeline. + | Option | Default | Description | |--------|---------|-------------| -| `sizeInMB` | `"-1"` | Default (`<= 0`) = no split: one whole-image tile per file. Set a positive MB value to tile large rasters into multiple tiles. | +| `virtualTiles` | `"true"` | **Default.** Emit bytes-free **virtual tiles** — each row carries the source `path` + pixel `window` instead of raster bytes; pixels are read lazily when an operation needs them (the ingest-OOM-dissolving default for the light tier). Set `"false"` to materialize raster bytes into each row. See [Virtual Tiles](../api/virtual-tiles). | +| `splitStrategy` | `"none"` | When to split large rasters: `none` (default — one whole-image tile per file), `auto` (resolve to `serverless` or `classic` by environment), `serverless` (opt-in, 64 MiB decoded budget per tile), `classic` (opt-in, 1 536 MiB decoded budget per tile). | +| `sizeInMB` | `"-1"` | Power-user override: set a positive value to pin the per-tile decoded-memory budget in MiB. A positive value implies opt-in split regardless of `splitStrategy`. `-1` = defer to `splitStrategy`. | | `filterRegex` | `".*"` | When loading a directory, keep files whose full path matches this regex. | +| `clipPolygons` | _none_ | Area(s) of interest to clip to. One WKT/EWKT string for a single polygon, or a JSON-array string for a list, e.g. `'["POLYGON((...))","POLYGON((...))"]'` (raw WKB/EWKB bytes are accepted only from programmatic callers). One tile is emitted per polygon whose envelope intersects the raster; a polygon that misses the raster emits no tile. Materialized tiles are pre-clipped (pixels outside the polygon are NoData); virtual tiles carry the clip as a deferred instruction. Mutually exclusive with `windows` and `tileSize`. Pairs naturally with the [COG Reader](./cog). | +| `windows` | _none_ | Pixel window(s) to read. A JSON 4-int array `"[col,row,w,h]"` for one, or a JSON array of them `"[[..],[..]]"` for a list. One tile per window; partial windows clip to the raster extent, fully-outside windows are skipped. Mutually exclusive with `clipPolygons` and `tileSize`. | +| `clipCrs` | _none_ | CRS for `clipPolygons` that lack an embedded SRID (e.g. `"EPSG:27700"`). Precedence: an EWKB/EWKT polygon's embedded SRID → `clipCrs` → the raster's CRS. | +| `tileSize` | _none_ | Cut the whole raster into a regular fixed-size pixel grid: `"w,h"` (e.g. `"512,512"`) or a single `"512"` for square tiles. One tile per grid cell; edge cells are clamped to the extent. Mutually exclusive with `clipPolygons` and `windows`. Materialized tiles are guarded to the ~2 GB Spark-cell limit (a too-large `tileSize` raises a clear error); virtual tiles (`virtualTiles=true`) carry no bytes and are unguarded. | +| `overlapPercent` | `0` | Overlap between adjacent `tileSize` cells, as a percentage of tile size (`overlap_px = ceil(tile_dim · pct/100)`, `step = tile_dim − overlap_px`). Applies to **`tileSize` only** (an error otherwise). Default `0` = non-overlapping grid. | +| `tilesTable` | _none_ | Delta table to use as a tile index (see [pre-computed tile inputs](#advanced-pre-computed-tile-inputs) below). | +| `skipOrdering` | `"false"` | Pass `"true"` to suppress the auto-sort-by-source-path that `tilesTable` reads apply by default. See [tilesTable auto-order](#tiles-table-auto-order) below. | + +:::tip Why overlap? +A hard tiling grid can slice straight through a spatial feature (a field, a building, an airport) +that straddles a seam, so no single tile sees it whole. Set `overlapPercent` so adjacent `tileSize` +cells overlap — a feature near a boundary then appears complete in at least one tile, which per-tile +analysis needs. Overlap only makes sense for a reader-chosen grid (`tileSize`); `windows` and +`clipPolygons` are extents you specify explicitly, so they take no overlap. +::: + +:::note Virtual tiles are the default +Light raster readers now emit **virtual tiles** by default (`virtualTiles=true`) — bytes-free +`(path, window)` references that read pixels lazily. Previously the reader materialized raster +bytes into every row. To restore materialized reads, pass `.option("virtualTiles", "false")`. +A virtual tile passed to a heavyweight function must be materialized first — see +[Virtual Tiles](../api/virtual-tiles). +::: + +:::note Default changed from `auto` to `none` +Prior to this release the reader defaulted to `splitStrategy=auto`, which +auto-split large rasters on a decoded-memory budget. The default is now +`none` — one tile per file. To opt back into splitting, pass +`.option("splitStrategy", "serverless")` or `.option("splitStrategy", "classic")`. + +When splitting is enabled, split tiles are emitted as plain GeoTIFF. +COG output is a `cog_gbx` writer concern, not a reader concern: the +`tileFormat`, `cogBlockSize`, and `cogOverviewResampling` reader options +are removed. For COG-encoded output, use the +[COG Writer](../writers/cog) to prepare master COGs, then read them with +the [COG Reader](./cog). + +**Striped GeoTIFFs** (no internal tiling) split into full-width row-bands. +**Internally-tiled sources** split on a block-snapped grid. +::: + +#### Directory reads {#loading-many-small-files} + +Reading a directory plans every file (fast, stat-free listing — ~1.5 s at 10k files); for pre-computed windows or very large tile counts see [Advanced: pre-computed tile inputs](#advanced-pre-computed-tile-inputs) below. + +:::tip Reading GeoBrix output with a stock Spark reader +Stock Spark file readers (`spark.read.format("binaryFile")`, `text`, `parquet`, …) +silently skip any file whose name starts with `_` or `.` — Spark's hidden-file filter, +applied during directory listing. Some GeoBrix raster tiles are written with a leading +`_` in their (hashed) name, so a stock reader can miss a large fraction of a directory. +Note that **`.option("pathGlobFilter", "*.tif")` does not override this filter** — the +filter runs before the glob, so `_`-prefixed files stay excluded (an explicit `_*` glob, +explicit `_`-file paths, and `recursiveFileLookup` do not help either). + +To read every tile, enumerate the paths on the driver (which does not apply the filter) +and build the DataFrame explicitly — ideal when you only need the file list, e.g. to feed +a VRT builder: + +```python +from pathlib import Path +paths = [str(p) for p in Path(output_dir).rglob("*.tif")] +files_df = spark.createDataFrame([(p,) for p in paths], ["path"]) +``` + +GeoBrix's own raster readers (`raster_gbx` / `gtiff_gbx` / `cog_gbx`) list files directly +and are **not** affected — this applies only to stock Spark file readers. +::: + +**Opt-in split (serverless budget):** + + + +:::tip Classic `READ_WITH_OPTIONS` example +The `sizeInMB` option remains available as a power-user override: +::: `gtiff_gbx` is `raster_gbx` with the GeoTIFF driver preset. +### COG lane for large rasters + +For large rasters, the recommended path is to **prepare master COGs** with the +`cog_gbx` writer and then **clip windows** with the `cog_gbx` reader. This +keeps the reader simple (no split, no re-encode) and lets GDAL's range-read +mechanism fetch only the bytes that intersect the AOI. + +**Who benefits from pre-built COG overviews:** + +- **`rst_tilexyz` and `rst_xyzpyramid`** — the XYZ tile-serving pipeline uses + [rio-tiler](https://cogeotiff.github.io/rio-tiler/), which automatically + selects the appropriate COG overview level for the requested zoom. +- **`rst_resample*`** — the resample family triggers overview-level selection + automatically when a COG source has pre-built overviews. + +**COG preparation + windowed read:** + +See [COG Writer](../writers/cog) for the preparation step and +[COG Reader](./cog) for the windowed read. + +**Force-writing COG output from a tile DataFrame:** the `gtiff_gbx` writer +still accepts a `cog=true` option to re-encode any `(source, tile)` DataFrame +as COG when you already have tiles in memory: + + + ### Heavyweight (`gdal`) | Option | Default | Description | @@ -299,8 +412,102 @@ The `hrrr-weather` dataset is included in the **complete** sample-data bundle, n - Use **GRIB2** for weather models - Use **Zarr** for cloud-native analysis at scale +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this reader and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Used | All directory reads — `recursive`, `include_hidden`, `extensions`, `path_glob_filter` options are routed through this shared predicate. | +| `enumerate_files` (FILE-tier enumeration) | Not in the DataSource | The DataSource is session-less on Connect and does not call `enumerate_files`; FILE-tier enumeration is only available through `gbx_file_read` at the function layer. | +| `gbx_file_read` / `gbx_file_write` (FILE tier) | Not in the DataSource | The DataSource is FUSE-only (session-less on Connect); FILE reads go through `gbx_file_read` → `rst_fromfile` at the function layer. | + +## Advanced: pre-computed tile inputs (`manifest` / tile index) {#advanced-pre-computed-tile-inputs} + +The default directory read is fast — a stat-free listing plans ~10k files in ~1.5 s — so most reads need nothing special. The `manifest` and `tilesTable` options are an optional optimization for two specific cases: + +- **Pre-computed windows you already maintain** — a manifest or tile index lets the reader use your saved file listing directly, skipping the directory walk entirely. +- **Very large tile counts** where even a fast listing accumulates across many repeated job runs. + +Both options reduce plan time from the directory walk (~1.5 s at 10k files) to a single file or table read. See [Benchmarking → Reader plan-time listing](../api/benchmarking#reader-plan-time-listing) for the measured numbers. + +- **`manifest` option** — supply a JSON or Parquet file listing `path` + `window` for each tile; the reader skips the directory walk and header opens entirely. + ```python + spark.read.format("raster_gbx") \ + .option("manifest", "/Volumes/catalog/schema/vol/tiles.json") \ + .option("virtualTiles", "true") \ + .load("/Volumes/catalog/schema/vol/rasters") + ``` +- **`tilesTable` option** — point to a Delta table that serves as a **tile index**: one row per tile with a `path` column (+ optional `window` / dimension columns). It is a catalog of file *references*, **not** the tile pixels — the reader still opens each referenced file and extracts its window; it only skips discovering the files by walking the directory. + ```python + spark.read.format("raster_gbx") \ + .option("tilesTable", "geospatial.myschema.tile_index") \ + .option("virtualTiles", "true") \ + .load("/") + ``` +- **Fewer, larger COGs** — consolidate small files into Cloud-Optimized GeoTIFFs using the [COG Writer](../writers/cog); the reader then opens a small number of large files whose headers are comparatively cheap. + +**A tile index is not a materialized-tile table.** `manifest` / `tilesTable` catalog *where the rasters are* (paths, plus optional windows) — essentially a saved file listing. The raster bytes stay in files and are read lazily (especially with `virtualTiles=true`); the index only lets the reader skip re-walking the directory. That is different from an ingest that has already **decoded** rasters into a Delta table with a tile-struct/bytes column — if you have that, read it directly with `spark.table(...)`; you don't need a reader at all. + +### tilesTable auto-order and `skipOrdering` {#tiles-table-auto-order} + +When `tilesTable` is set, the reader **auto-orders tiles by source path** before planning +partitions. This groups all tiles from the same source GeoTIFF into adjacent partitions, so a +worker that opens a large raster once can decode all its windows without reopening it. For a +table with many tiles per source file (a common windowed-GeoTIFF ingest), this amortization +meaningfully reduces open overhead — each file is opened once per partition instead of once per +tile. + +Pass `skipOrdering="true"` when the table is already physically ordered (e.g. written with +`layout="order"` or `layout="cluster"`) or when you apply ordering downstream: + +```python +spark.read.format("raster_gbx") \ + .option("tilesTable", "geospatial.myschema.tile_index") \ + .option("skipOrdering", "true") \ + .option("virtualTiles", "true") \ + .load("/") +``` + +#### Durable co-location: `CLUSTER BY` + `OPTIMIZE` + +`layout="order"` (the default write layout) sorts rows by path at insert time, which groups +tiles per source for the lifetime of that write. New inserts and compaction can scatter them. +For a stable layout that survives compaction, use `layout="cluster"` at write time and run +`OPTIMIZE` afterward: + +```python +from databricks.labs.gbx.ds.file_gbx import gbx_file_write + +gbx_file_write( + tiles_df, + target="geospatial.myschema.tile_index", + layout="cluster", # CLUSTER BY path in the DDL + file_mode="auto", +) +``` + +```sql +-- After initial write (or after bulk inserts), materialize clustering: +OPTIMIZE geospatial.myschema.tile_index; +``` + +With `layout="cluster"`, `OPTIMIZE` physically co-locates tiles from the same source into the +same data files. Subsequent reads with `tilesTable` have source-adjacent rows from the +start — `skipOrdering="true"` is safe once the table has been optimized, and it avoids +re-sorting an already-ordered scan. + +`layout="order"` (insert-time `ORDER BY path`) is the simpler default: no DDL change, no +`OPTIMIZE` required. It works well for tables that are built once. Use `layout="cluster"` when +the table is updated incrementally and you run periodic `OPTIMIZE` as part of maintenance. + ## Next Steps - [GeoTIFF Reader](./geotiff) - Named reader for GeoTIFF format - [Raster Functions](../api/raster-functions) - Raster processing operations - [Quick Start](../quick-start) - Get started with GeoBrix + +:::note Shared file-access layer +Lightweight readers use the shared [`file_gbx` file-access base](../readers-writers#file-gbx) for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there. +::: diff --git a/docs/docs/readers/shapefile.mdx b/docs/docs/readers/shapefile.mdx index f3ba05557..f672bc132 100644 --- a/docs/docs/readers/shapefile.mdx +++ b/docs/docs/readers/shapefile.mdx @@ -150,3 +150,18 @@ Read ESRI Shapefile format using the `shapefile` reader. - [GeoPackage Reader](./geopackage) - [FileGDB Reader](./filegdb) - [API Reference](../api/overview) + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this reader and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Used | Directory reads — `recursive`, `include_hidden`, `extensions`, and `path_glob_filter` are routed through this shared predicate for all format-specific readers. | +| `gbx_file_read` / FILE-tier read | Not in the DataSource | The DataSource is session-less on Connect; FILE-tier reads go through `vector_file_read` (function layer) which injects `_file_ref` on the driver before `mapInPandas`. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | FILE-tier writes go through `vector_file_write` (function layer). The DataSource writer commits via FUSE. | + +:::note Shared file-access layer +Lightweight readers use the shared [`file_gbx` file-access base](../readers-writers#file-gbx) for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there. +::: diff --git a/docs/docs/readers/vector.mdx b/docs/docs/readers/vector.mdx index 528ff1686..f7cb7fdf4 100644 --- a/docs/docs/readers/vector.mdx +++ b/docs/docs/readers/vector.mdx @@ -82,6 +82,76 @@ df.write.mode("overwrite").saveAsTable("main.geo.features") # Delta table on Da Reading a folder fans the files across the cluster (one partition per file), so ingest scales with the data — unlike a single-node `pyogrio.read_*` that parses one file on one machine. See [Benchmarking](../api/benchmarking) for light-vs-heavy ingest figures. +### Reading from a FILE-column table {#vector-file-table-read} + +`vector_file_read` (light tier) accepts either a Volume path/directory **or** a FILE-column +Delta table name. A FILE-column table stores one vector file per row — each FILE reference +points to a complete vector file (e.g. a `.gpkg`). Reading decodes all features from each +referenced file via pyogrio. + +```python +from databricks.labs.gbx.pyvx import vector_file_read + +# Read all vector files referenced in a FILE-column table +features = vector_file_read( + spark, + "main.geo.vector_files", # fully-qualified table name + driver="GPKG", + as_wkb=True, +) +features.show() +``` + +`source_type="auto"` (the default) distinguishes a table from a path: a string starting +with `/`, matching a URI scheme, or having a known vector file extension is treated as a path; +a dotted name, an extension-less string, or a non-existent path is treated as a table. Pass +`source_type="table"` or `source_type="path"` to bypass the heuristic. + +**Auto-order default.** `vector_file_read` auto-orders by the resolved source path before +decoding. For a vector FILE table, one file maps to one row — every source opens exactly once +regardless of row order. Auto-ordering produces a **deterministic, cross-format-consistent +default** row sequence; it is not a throughput lever here (there are no multiple tiles per file +to amortize). Pass `skip_ordering=True` when the table is already physically ordered or when +you control ordering downstream: + +```python +features = vector_file_read( + spark, + "main.geo.vector_files", + skip_ordering=True, # preserve table/scan order; no additional sort step +) +``` + +#### Durable co-location: `CLUSTER BY` + `OPTIMIZE` + +Use `layout="cluster"` in `vector_file_write` (or `gbx_file_write`) to declare `CLUSTER BY path` +in the table DDL. After a bulk insert, run `OPTIMIZE` to materialize the clustering — rows for +the same source land in the same data files, which makes repeated reads and compaction-stable: + +```python +from databricks.labs.gbx.pyvx import vector_file_write + +vector_file_write( + spark, + local_out="/tmp/roads.gpkg", + target="main.geo.vector_files", + layout="cluster", # CLUSTER BY path in the DDL + file_mode="auto", +) +``` + +```sql +-- After initial write or after bulk inserts, materialize clustering: +OPTIMIZE main.geo.vector_files; +``` + +Once the table has been optimized, pass `skip_ordering=True` when reading — the rows are already +co-located, and skipping the sort avoids a redundant step. + +`layout="order"` (the default) applies `ORDER BY path` at insert time and works well for tables +built in a single write. Use `layout="cluster"` for tables updated incrementally and maintained +with periodic `OPTIMIZE`. + @@ -189,9 +259,24 @@ For common formats, GeoBrix provides named readers for convenience (sample-data +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this reader and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Used | Directory reads — `recursive`, `include_hidden`, `extensions`, and `path_glob_filter` are routed through this shared predicate for all format-specific readers. | +| `gbx_file_read` / FILE-tier read | Not in the DataSource | The DataSource is session-less on Connect; FILE-tier reads go through `vector_file_read` (function layer) which injects `_file_ref` on the driver before `mapInPandas`. `vector_file_read` also accepts a FILE-column Delta table name — see [Reading from a FILE-column table](#vector-file-table-read). | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | FILE-tier writes go through `vector_file_write` (function layer). The DataSource writer commits via FUSE. | + ## Next Steps - [Shapefile Reader](./shapefile) - [GeoJSON Reader](./geojson) - [GeoPackage Reader](./geopackage) - [File GeoDatabase Reader](./filegdb) + +:::note Shared file-access layer +Lightweight readers use the shared [`file_gbx` file-access base](../readers-writers#file-gbx) for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there. +::: diff --git a/docs/docs/beta-release-notes.mdx b/docs/docs/release-notes.mdx similarity index 83% rename from docs/docs/beta-release-notes.mdx rename to docs/docs/release-notes.mdx index 4843a6ceb..2430b3601 100644 --- a/docs/docs/beta-release-notes.mdx +++ b/docs/docs/release-notes.mdx @@ -1,21 +1,40 @@ --- sidebar_position: 5 -title: Beta Release Notes +title: Release Notes --- -# Beta Release Notes +# Release Notes -:::info Current version: 0.4.3 +:::info Current version: 0.5.0 The changes on this page are relative to 0.1.0 (and earlier). ::: -This page tracks **API and naming changes** since the GeoBrix project started. After the project is approved, formal release notes will take over; until then, use this as the single place to look up what changed and why. +This page tracks **API and naming changes** across GeoBrix releases — the single place to look up what changed and why. + +--- + +## What's new in v0.5.0 + +Introduces **virtual tiles** — a bytes-free way to read and process huge rasters without out-of-memory — unifies both execution tiers on one tile struct, and adds the COG preparation lane (`file_gbx` + `cog_gbx`) with **VRT mosaics** for large-raster workflows, tapping the new Databricks **[FILE type](https://docs.databricks.com/aws/en/sql/language-manual/data-types/file-type)** for governed, memory-safe file access where the runtime provides it, on top of v0.4.3. + +- **Virtual tiles — bytes-free windowed reads for large rasters.** The lightweight `cog_gbx` / `raster_gbx` / `gtiff_gbx` readers can emit **virtual tiles**: each row carries a source `path` + pixel `window` instead of the raster bytes, and pixels are read lazily, one window at a time, only when an operation needs them. A virtual-tile row is ~100 bytes versus 148–527 KB of materialized bytes (~1,400–5,000× smaller), so fanning a multi-gigabyte raster into many tiles no longer accumulates into a Serverless out-of-memory failure. Opt in with `.option("virtualTiles", "true")`. See the new **[Virtual Tiles](./api/virtual-tiles)** page. +- **Light raster readers now default to virtual tiles (breaking behavior change).** The lightweight raster readers (`raster_gbx`, `gtiff_gbx`, `cog_gbx`) previously materialized tile bytes by default; they now emit **[virtual tiles](./api/virtual-tiles)** by default. Code that reads a raster and immediately operates on `tile.raster` bytes — or passes the tile to a heavyweight function — will see `null` bytes until the tile is materialized. Pass `.option("virtualTiles", "false")` to restore the old behavior, or call a downstream `rst_*` function with `materialize=True`. +- **The [Tile Structure](./api/tile-structure) output shape widens from the 3-field struct to the v2 8-field struct.** The tile struct gains five fields — `path`, `window`, `clip_polygon`, `clip_crs`, `crs` — alongside `cellid`, `raster`, and `metadata`, and `raster` is now **nullable** (`null` on a virtual tile). The same 8-field struct is produced and consumed by both tiers, so consumers that assumed the 3-field shape need updating. On a materialized tile the new fields are provenance (what was applied to the bytes); on a virtual tile they are instructions (applied on read). +- **Every lightweight `rst_*` function is virtual-tile-aware.** Functions consume virtual or materialized tiles through one shared open path: metadata accessors answer from the header without reading pixels, reference/passthrough ops (`rst_clip`, `rst_setsrid`, identity `rst_transform`) stay virtual, and pixel-producing ops materialize only the window they need. Tile-returning functions gain three optional force-output params — **`virtualize_dir`**, **`virtualize_prefix`**, **`materialize`** — so you can flip a result back to a bytes-free virtual row (or force bytes) at any step. See [Virtual↔materialized advice](./api/execution-tiers#virtual-materialized-advice). +- **Both tiers accept v1 and v2 tiles and always emit v2.** The heavyweight (`rasterx`) and lightweight (`pyrx`) tiers both read the legacy 3-field tile and the new 8-field tile, and every function emits the v2 struct — so output composes directly across tiers. Heavyweight operates only on **materialized** (binary) tiles as of now, so a virtual tile passed to a heavyweight function raises a clear materialize-first error (`materialize=True`, or write it out and read it back). The lightweight tier is for both **light (virtual)** and **materialized** tiles. See [Virtual tiles and the light→heavy bridge](./api/execution-tiers#light-heavy-bridge). +- **New `file_gbx` reader — path lister.** Lists files in a directory as path-reference rows `(path, name, extension, size, modificationTime)`, without loading any raster content. `extension` is lowercase, no leading dot, and `NULL` for files without an extension. Options: `filterRegex`, `recursiveFileLookup`. This is the entry point for the COG preparation pipeline. See [File Lister](./readers/file). +- **New `cog_gbx` writer — master-COG preparation.** Takes path-reference rows from `file_gbx` and converts each source file to a spec-valid Cloud-Optimized GeoTIFF using GDAL's `driver="COG"` creation path. Each output COG carries internal tiling and pre-built overview levels. Options: `cogBlockSize` (default `512`), `cogOverviewResampling` (default `AVERAGE`), `cogCompression` (default `DEFLATE`), plus `driverMode` for driver-orchestrated large-file preparation. Output files pass `rio_cogeo.cogeo.cog_validate`. See [COG Writer](./writers/cog) and [Large Rasters](./api/large-rasters). +- **New `cog_gbx` reader — COG-aware windowed read.** Reads Cloud-Optimized GeoTIFFs into the shared tile schema, issuing range-reads that fetch only the bytes a window needs. The reader windowing surface is **`tileSize`** (regular grid) + **`overlapPercent`**, arbitrary **`clipPolygons`** (emit only the tiles intersecting each polygon, with **`clipCrs`**), or explicit pixel **`windows`** — lists are passed as JSON strings over `.option()`. See [COG Reader](./readers/cog) and [Readers overview](./readers/overview). +- **Raster reader `splitStrategy` defaults to `none`.** The `raster_gbx`, `gtiff_gbx`, and `cog_gbx` readers default to one whole-image tile per file; splitting is opt-in via `.option("splitStrategy", "serverless")` / `"classic"`, and the `sizeInMB` power-user override remains available. For large rasters the recommended pattern is prepare-then-read: list with `file_gbx`, prepare master COGs once with the `cog_gbx` writer, then read any AOI window cheaply with virtual tiles. See [Large Rasters](./api/large-rasters). +- **VRT mosaics — tile a large source into bounded mini-COGs + a portable index.** The `cog_gbx` writer's `vrtMosaic` mode splits each source into bounded mini-COGs — read window-by-window, so the source is never fully held in RAM — and writes a lightweight, portable `mosaic.vrt` index over them; each mini-COG is small by construction, so a source too big to hold as one COG (or in per-task memory) is processed safely on Serverless. Point `raster_gbx` (or `cog_gbx`) at a `mosaic.vrt` and the reader expands it into one virtual tile per member, so every `rst_*` function processes the mosaic per-tile; `clipPolygons` restricts the expansion to only the members intersecting an area of interest. `mint_vrt` builds a transient index over an ad-hoc tile list for on-demand windowed reads, and `mosaic.vrt` opens in any GDAL tool (QGIS, gdalinfo, rio-tiler) with no GeoBrix needed. Native pixel tiling (`gridSystem="none"`) with `tileSize` / `overlapPercent`; quadbin cell-aligned tiling (`gridSystem="quadbin"` + `gridResolution`) reprojects to EPSG:3857 and tags each cell with `GBX_CELLID`, surfacing in `tile.metadata["cellid"]` after expansion; VRT is optional (`writeVrt="false"` writes tiles only). H3 grid-aligned mode (`gridSystem="h3"`) reprojects to EPSG:4326, clips each cell to its hexagon, and tags every member with `GBX_CELLID` (h3index), surfacing in `tile.metadata["cellid"]` for equi-join unification with h3-indexed tabular data. BNG is planned. See the **[VRT & Mosaics](./api/vrt-mosaic)** page. +- **VectorX CRS functions (`gbx_st_crs`, `gbx_st_setcrs`, `gbx_st_transformcrs`) — both tiers.** Three new functions complement the Databricks built-ins `st_srid` / `st_setsrid` / `st_transform` with authority-string CRS handling: `gbx_st_crs(geom)` returns the geometry's CRS as a canonical authority string (e.g. `EPSG:4326`, `ESRI:54008`), or `NULL` for a geometry with no SRID. `gbx_st_setcrs(geom, crs)` stamps a CRS onto a geometry as a label-only change — coordinates are not moved; it raises on an authority-less CRS (raw WKT or PROJ4) because a geometry can only carry an integer SRID. `gbx_st_transformcrs(geom, target_crs[, source_crs])` reprojects coordinates; an authority-coded target (`EPSG:n` / `ESRI:n`) stamps `n` on the result, while an authority-less target reprojects and clears the stale SRID; an optional third argument supplies a source CRS for a geometry that carries no SRID. All three accept WKB/EWKB/WKT/EWKT input; the SQL surface returns `BINARY` (`gbx_st_crs` returns `STRING`). Available on both the heavyweight (`vectorx`) and lightweight (`pyvx`) tiers. See [VectorX Functions](./api/vectorx-functions#crs). +- **RasterX CRS functions (`gbx_rst_crs`, `gbx_rst_setcrs`, `gbx_rst_transformcrs`) — both tiers.** The raster counterparts of the same idea: they take and return CRS *strings*, so a non-EPSG coordinate system survives a round trip that the integer-SRID functions (`gbx_rst_srid` / `gbx_rst_setsrid` / `gbx_rst_transform`) cannot represent. `gbx_rst_crs(tile)` returns the tile's CRS as an authority string (`EPSG:4326`, `EPSG:32618`) when it carries one, and its full WKT when it does not — so it always returns a value where `gbx_rst_srid` gives `0` for a CRS with no EPSG code. `gbx_rst_setcrs(tile, crs)` relabels the tile's spatial reference **without warping pixels** — the georeference (`upperLeftX`, `scaleX`, …) and the pixel dimensions are unchanged, which is what you want when a file arrived with missing or wrong CRS metadata. `gbx_rst_transformcrs(tile, target_crs)` genuinely reprojects: it resamples the pixel grid, so both the georeference and the raster dimensions change (a 64×64 `EPSG:4326` tile becomes 55×72 in `EPSG:3857`). Both writers accept an authority code (`EPSG:3857`, `ESRI:54008`), WKT, or PROJ4, and an int-castable string (`'4326'`) is treated as an EPSG code. Available on both the heavyweight (`rasterx`) and lightweight (`pyrx`) tiers. See [Raster Functions](./api/raster-functions#rst_crs). --- ## What's new in v0.4.3 -In-flight beta release. Completes both-tier parity for the raster-grid family, adds heavyweight NetCDF readers and a lightweight NetCDF writer, and improves the documentation — on top of v0.4.2. +Completes both-tier parity for the raster-grid family, adds heavyweight NetCDF readers and a lightweight NetCDF writer, and improves the documentation — on top of v0.4.2. - **Raster BNG and quadbin grid functions are now both-tier.** The nine raster-grid functions added for quadbin and BNG in v0.4.0 — five BNG reducers (`gbx_rst_bng_rastertogrid{avg,count,max,min,median}`), two tessellate generators (`gbx_rst_quadbin_tessellate`, `gbx_rst_bng_tessellate`), and two rasterize aggregators (`gbx_rst_quadbin_rasterize_agg`, `gbx_rst_bng_rasterize_agg`) — now run on the lightweight `pyrx` tier too, matching the heavyweight tier on cell set, clip windows, and measures. **Every RasterX function is now available in both tiers.** The lightweight BNG reducers use a vectorized NumPy encoder that runs ~3.7–4.8× the heavyweight per-tile speed. See [Raster Functions](./api/raster-functions) and [Choosing an Execution Tier](./api/execution-tiers). - **New raster-grid reducers: `sum`, `variance`, and `stddev` (all three grids, both tiers).** `gbx_rst_{h3,quadbin,bng}_rastertogrid{sum,variance,stddev}` extend the `rastertogrid` reducer family (`avg`/`count`/`max`/`min`/`median`). `variance` and `stddev` are **population** statistics (`÷ n`) from a numerically-stable two-pass algorithm; cross-tier measures match within tolerance. See [Raster Functions](./api/raster-functions). @@ -29,7 +48,7 @@ In-flight beta release. Completes both-tier parity for the raster-grid family, a ## What's new in v0.4.2 -In-flight beta release. Adds the Genie Map example app, plus correctness and consistency fixes, on top of v0.4.1. +Adds the Genie Map example app, plus correctness and consistency fixes, on top of v0.4.1. - **Genie Map — interactive methane-map Databricks App.** A new example app that turns the GeoBrix-processed Permian methane gold data (the [Vapor-Eyes](./notebooks/vapor-eyes) `geospatial_docs.vapor_eyes_lf` schema) into an interactive map with two ways to explore the same data side by side: **move the map** (every pan/zoom runs parameterized viewport SQL and redraws H3 hotspot / well-density hexagons, well points, and EMIT plume points), and **ask a question** (a natural-language prompt routes to a curated Genie Space — a geometry answer lands as a new map layer, and a feature-level answer becomes a chart whose selection cross-filters the map, and vice-versa). Client is React + [kepler.gl](https://kepler.gl); server is Databricks [AppKit](https://github.com/databricks-solutions/databricks-appkit). Grew out of the Data + AI Summit 2026 session *[Scaling Geospatial Analytics at S&P Global Energy: From Billions of Points to AI-Powered Map Agents with Databricks](https://www.databricks.com/dataaisummit/session/scaling-geospatial-analytics-sp-global-energy-billions-points-ai-powered)* by [Hubert Boguski](https://www.linkedin.com/in/hubertboguski/) (S&P Global) and [Michael Johns](https://www.linkedin.com/in/michaeljohns2/) (Databricks). See [Genie Map](./examples/genie-map). - **Raster value reducers return `NULL` for all-nodata bands (behavior change).** `gbx_rst_max`, `gbx_rst_min`, `gbx_rst_avg`, and `gbx_rst_median` now return SQL `NULL` for a band with zero valid pixels — for example an H3 covering-tessellation cell that clips only NoData — on **both** the lightweight and heavyweight tiers. Previously the lightweight tier returned `NaN` and the heavyweight tier returned `0.0`; neither was catchable or aggregation-safe. `NaN` silently passed `WHERE measure IS NOT NULL` and could overwrite a real value in `MAX()` during a `GROUP BY` seam reconciliation (NaN sorts greater than everything); `0.0` was indistinguishable from a genuine zero. The new `NULL` is catchable via `WHERE measure IS NULL` and ignored by aggregates like `MAX`/`MIN`/`AVG`. `gbx_rst_pixelcount` is unchanged — an all-nodata band still returns `0` (a count of zero is meaningful). Relatedly, the lightweight `gbx_rst_isempty` is now all-nodata-aware: a dimensionally-valid raster whose every band is entirely NoData now returns `true`, matching the heavyweight tier. See [Raster Functions](./api/raster-functions). @@ -38,7 +57,7 @@ In-flight beta release. Adds the Genie Map example app, plus correctness and con ## What's new in v0.4.1 -In-flight beta release. Adds the satellite-data readers and clients for atmospheric/methane workflows and a full worked example, on top of the v0.4.0 lightweight tier. +Adds the satellite-data readers and clients for atmospheric/methane workflows and a full worked example, on top of the v0.4.0 lightweight tier. - **`netcdf_gbx` reader.** A net-new lightweight vector DataSource that transcodes a netCDF-4 swath (e.g. a Sentinel-5P TROPOMI L2 granule) directly to one point per ground pixel — no regridding, with science/quality variables (e.g. `qa_value`) passed through untouched — so H3 binning or quality filtering runs on the native per-pixel measurements. Registered via `databricks.labs.gbx.ds`. See [netCDF Reader](./readers/netcdf). - **Earthdata client (`databricks.labs.gbx.earthdata`).** A NASA Earthdata Login token client for authenticated access to LP DAAC products (used by the EMIT downloader). See the [EMIT Downloader](./sample-data/emit-downloader). @@ -49,7 +68,7 @@ In-flight beta release. Adds the satellite-data readers and clients for atmosphe ## What's new in v0.4.0 -In-flight beta release. Per-version highlights; full migration tables are in the per-component sections below. +Per-version highlights; full migration tables are in the per-component sections below. - **Lightweight execution tier (pyrx, pygx, pyvx).** A pure-Python implementation of the GeoBrix API that needs no JAR and no init script, and runs on serverless compute, standard (shared) clusters, Lakeflow declarative pipelines, and ARM. It keeps the same function names and the same `gbx_*` SQL after `register`, so switching tiers is a one-line import change. **RasterX** (`pyrx`, on [rasterio](https://rasterio.readthedocs.io/)) implements every `rst_*` function; **GridX** (`pygx`) covers quadbin, BNG, and custom grids; **VectorX** (`pyvx`) covers MVT, TIN surface modeling, and legacy-geometry migration. With this release GridX and VectorX are fully both-tier — the lightweight tier reaches 1:1 parity with the heavyweight one across all three packages. See [Choosing an Execution Tier](./api/execution-tiers). - **Serverless support is verified and documented.** `geobrix[light]` installs and runs on Databricks Serverless (environment v5), standard (shared) clusters, and ARM. Install with the quoted PEP 508 named form — `%pip install "geobrix[light] @ file:///Volumes/.../geobrix-0.4.0-py3-none-any.whl"` — not the path-with-extra form (`'…whl[light]'`), which fails on Serverless because `%pip` writes the surrounding quotes into the requirement and pip reads `[light]` as part of the filename. `mapbox-vector-tile` is pinned to 2.1.x so its `protobuf` dependency stays `<6` (Spark Connect compatibility on Serverless), and `idna` is pinned `<3.8` to avoid a core-package-change notice. See [Installation](./installation?tier=lightweight). @@ -66,7 +85,7 @@ In-flight beta release. Per-version highlights; full migration tables are in the - **Full raster-grid surface for quadbin and BNG (9 functions).** The raster-grid API that H3 has had since v0.3.0 is now complete for the other two discrete-global-grid families on the heavy tier. **Quadbin** adds two new operations: `gbx_rst_quadbin_tessellate` (one clipped chip per overlapping quadbin cell — a streaming generator) and `gbx_rst_quadbin_rasterize_agg` (burn per-cell values back into a raster — the inverse of `rastertogrid`). **BNG** adds the matching full surface: five reducers (`gbx_rst_bng_rastertogrid{avg,count,max,min,median}`), a tessellate generator (`gbx_rst_bng_tessellate`), and a rasterize aggregator (`gbx_rst_bng_rasterize_agg`). BNG functions automatically reproject the input raster to EPSG:27700 — no upstream `rst_transform` needed. BNG resolution accepts integer indices ±1..±6 or string keys (`"1km"`, `"100m"`); cell IDs are STRING. `gbx_rst_bng_tessellate` in particular enables BNG-scale raster tiling for computer-vision model inference — aligning aerial or satellite imagery to Ordnance Survey 1 km or 100 m grid cells. Closes [#49](https://github.com/databrickslabs/geobrix/issues/49). The rasterize aggregators use `−9999.0` as the band-registered NoData sentinel (same as H3). These nine functions shipped heavy-tier first; their lightweight `pyrx` implementations follow in v0.4.2 (see the v0.4.2 note above), bringing every RasterX function to both tiers. See [Raster Functions](./api/raster-functions). - **Raster→quadbin aggregators (5 functions).** `gbx_rst_quadbin_rastertogrid{avg,count,max,min,median}` extend the H3 aggregation pattern to CARTO quadbin v0 cells. Natural fit for raster heatmaps that render in slippy-map viewers — cells align with the same XYZ pyramid that PMTiles / MVT readers consume. Resolution capped at z=20. See [Raster Functions](./api/raster-functions). - **Web-mercator XYZ tile output (3 functions).** `gbx_rst_to_webmercator` reprojects a raster to EPSG:3857 (default `bilinear`); `gbx_rst_tilexyz(tile, z, x, y, [format, size, resampling])` renders a single XYZ tile to PNG / JPEG / WEBP bytes (returns `BinaryType`; out-of-extent tiles get a transparent PNG, not null); `gbx_rst_xyzpyramid(tile, min_z, max_z, ...)` is a generator that explodes one raster into one row per intersecting `(z, x, y)` tile across a zoom range. `max_z` capped at 20; total tile-count across zoom range capped at 10^6. Foundation for the PMTiles publishing pipeline. See [Raster Functions](./api/raster-functions). -- **Vector↔raster bridge (`gbx_rst_rasterize`, `gbx_rst_polygonize`).** Two reciprocal RasterX functions that span GeoBrix's vector and raster worlds. `gbx_rst_rasterize(geom_wkb, value, xmin, ymin, xmax, ymax, width_px, height_px, srid)` burns a vector geometry into a fresh GTiff-backed raster tile at the given extent / resolution (pixels inside the geometry carry `value`, pixels outside are NoData = `-9999.0`). `gbx_rst_polygonize(tile, [band, [connectedness]])` extracts `ARRAY` from `tile` — one feature per contiguous value region, NoData pixels excluded. The pair composes: `polygonize(rasterize(geom, v, ...))` returns at least one feature with value `v` covering approximately the same area as the input `geom`, with edges quantized to the pixel grid. See [Raster Functions § Vector bridge](./api/raster-functions#vector-bridge). +- **Vector↔raster bridge (`gbx_rst_rasterize`, `gbx_rst_polygonize`).** Two reciprocal RasterX functions that span GeoBrix's vector and raster worlds. `gbx_rst_rasterize(geom, value, xmin, ymin, xmax, ymax, width_px, height_px, srid)` burns a vector geometry into a fresh GTiff-backed raster tile at the given extent / resolution (pixels inside the geometry carry `value`, pixels outside are NoData = `-9999.0`). `gbx_rst_polygonize(tile, [band, [connectedness]])` extracts `ARRAY` from `tile` — one feature per contiguous value region, NoData pixels excluded. The pair composes: `polygonize(rasterize(geom, v, ...))` returns at least one feature with value `v` covering approximately the same area as the input `geom`, with edges quantized to the pixel grid. See [Raster Functions § Vector bridge](./api/raster-functions#vector-bridge). - **Terrain analysis (7 functions).** `gbx_rst_slope`, `gbx_rst_aspect`, `gbx_rst_hillshade`, `gbx_rst_tri`, `gbx_rst_tpi`, `gbx_rst_roughness`, `gbx_rst_color_relief` — all thin wrappers over `gdal.DEMProcessing`. Each takes a single-band DEM tile and returns a derived tile (Float32 for slope/aspect/TRI/TPI/roughness, Byte for hillshade, RGB(A) Byte for color_relief). Defaults mirror the gdaldem CLI (hillshade NW sun at 315° azimuth, 45° altitude; slope in degrees). Foundation for terrain-derived workflows — solar exposure, viewshed pre-processing, watershed and runoff analysis, road grading. See [Raster Functions § Terrain](./api/raster-functions#terrain). - **Slope and hillshade auto-scale from the raster CRS (breaking default on geographic rasters).** `gbx_rst_slope` and `gbx_rst_hillshade` (and the lightweight `prx.rst_slope` / `prx.rst_hillshade`) now derive the horizontal scale from the raster's coordinate reference system by default, matching GDAL `gdaldem`. On geographic (lat/long, e.g. EPSG:4326) rasters the scale is computed from latitude (degree→metre), so a global or geographic DEM produces correct, non-saturated slope and shading without any extra argument; on projected (metre) rasters output is unchanged. Previously these two ran unscaled on geographic input, which over-steepened and saturated the result. This changes the default output for geographic rasters to the GDAL-consistent value. To pin a specific scale, pass it explicitly — `gbx_rst_slope(tile, 'degrees', 111120)` for a degree grid, or `prx.rst_slope(tile, xscale=..., yscale=...)` / `prx.rst_hillshade(tile, xscale=..., yscale=...)`. `gbx_rst_aspect` is a direction and is unaffected. See [Raster Functions § Terrain](./api/raster-functions#terrain). - **Spectral indices (5 functions).** `gbx_rst_evi`, `gbx_rst_savi`, `gbx_rst_ndwi`, `gbx_rst_nbr`, plus a generic `gbx_rst_index(tile, formula_name, band_map)` — all compositions over `gbx_rst_mapalgebra`. Each takes user-supplied 1-based band indices, builds a per-pixel formula string, and dispatches to gdal_calc; output is a single-band Float32 GTiff sized to the input extent. The generic dispatcher ships built-in NDVI, GNDVI, MSAVI, red-edge NDVI, NDMI, and NDSI formulae and is the entry point users should reach for first for any named multi-band index; the four specialized expressions surface EVI / SAVI / NDWI / NBR with their canonical coefficient defaults (EVI: `L=1.0, C1=6.0, C2=7.5, G=2.5` per MODIS; SAVI: `L=0.5`) so vegetation, water and burn-severity workflows compose without a hand-written formula string. See [Raster Functions § Spectral indices](./api/raster-functions#spectral-indices). @@ -90,6 +109,7 @@ In-flight beta release. Per-version highlights; full migration tables are in the - **Example notebooks default to the lightweight tier.** The EO Series and xView walkthroughs now run on the lightweight API (`pyrx` / `pygx` / `pyvx` plus the `gbx_*` DataSource readers and writers) by default, so they execute on Databricks Serverless (environment v5) with no JAR and no init script; each notebook calls out the one-line import to switch back to the heavyweight tier. The EO Series uses the new `StacClient` for its Planetary Computer search, download, and repair steps. See [EO Series](./notebooks/eo-series) and [xView](./notebooks/xview). - **H3 cell rasterize example notebook.** A complete polygon → H3 polyfill → per-band rasterize → multi-band stack walkthrough on a San Francisco Bay Area DEM, treating elevation isobands as a stand-in for signal-strength coverage tiers (a telco coverage-analysis pattern). Exercises `rst_h3_gridspec`, `rst_h3_rasterize_agg`, and `rst_frombands_agg`, materializes the per-band tiles into a session-scoped temp table, and uses the `gbx.vizx` helpers (`plot_mask_layers`, `plot_raster(composite="depth")`) to inspect the result. See [H3 Rasterize](./notebooks/h3-rasterize). - **Helios distributed-tiling notebook series.** A four-notebook solar site-selection walkthrough over one San Francisco AOI: building footprints → vector PMTiles (NB01), a NAIP aerial basemap → raster PMTiles (NB02), 3DEP terrain → COG catalog + hillshade PMTiles + a per-H3-cell solar score (NB03), and a distributed **sharded** PMTiles mosaic with a `mosaic.json` manifest for client-side assembly (NB04). Runs on the lightweight tier / Serverless with no JAR, dogfooding `gbx_st_asmvt_pyramid`, `gbx_rst_xyzpyramid`, `gbx_pmtiles_agg`, the sample downloaders, and the `gbx.vizx` PMTiles viewers. See [Helios](./notebooks/helios). +- **Grid explode functions now return `cellid` (lowercase) in the output struct (breaking schema change).** The generator functions `gbx_bng_kringexplode`, `gbx_bng_kloopexplode`, `gbx_bng_geomkringexplode`, `gbx_bng_geomkloopexplode`, and `gbx_bng_tessellateexplode` previously returned a struct column named `cellId` (camelCase). The column is now `cellid` (all-lowercase), matching the tile-struct field name, the chip-struct field name, and the Databricks product convention. SQL queries that reference the result by field name (`result.cellId`) need to be updated to `result.cellid`. --- @@ -220,7 +240,7 @@ All specific function renames from that standardization are listed in the compon | (none) | `gtiff_gdal` | **New** reader: named GDAL reader for GeoTIFF; use instead of `gdal` with `option("driver", "GTiff")`. | :::info -Reader renames above are planned for 0.2.0. Beta (0.1.x) may still expose the baseline names in some contexts. +Reader renames above landed in 0.2.0; earlier 0.1.x releases may still expose the baseline names in some contexts. ::: --- @@ -273,7 +293,7 @@ SELECT gbx_bng_pointascell(pt, '1km') FROM ...; - **Migrating code:** Search for the **baseline** name in your code or config; replace with **Current** and apply any behavior notes. - **Docs or tests:** After a change, add one row here so future readers know what changed and why. -- **After approval:** Move content into formal release notes (e.g. per-version sections) and keep this page for historical beta-only changes, or retire it. +- **Housekeeping:** Keep per-version sections here as the canonical change log; prune superseded interim notes as versions settle. --- diff --git a/docs/docs/security.mdx b/docs/docs/security.mdx index 639d0260c..d595786f4 100644 --- a/docs/docs/security.mdx +++ b/docs/docs/security.mdx @@ -57,7 +57,7 @@ serving a same-version-but-different-bytes wheel fails closed. | Path | Lockfile | |---|---| | CI (Scala + Python build) | [`python/geobrix/requirements-ci.txt`](https://github.com/databrickslabs/geobrix/blob/main/python/geobrix/requirements-ci.txt) | -| CI (lightweight pyrx build) | [`python/geobrix/requirements-pyrx-ci.txt`](https://github.com/databrickslabs/geobrix/blob/main/python/geobrix/requirements-pyrx-ci.txt) | +| CI (lightweight build) | [`python/geobrix/requirements-light-ci.txt`](https://github.com/databrickslabs/geobrix/blob/main/python/geobrix/requirements-light-ci.txt) | | Dev container | [`python/geobrix/requirements-dev-container.txt`](https://github.com/databrickslabs/geobrix/blob/main/python/geobrix/requirements-dev-container.txt) | | Notebook test harness | [`notebooks/tests/requirements.txt`](https://github.com/databrickslabs/geobrix/blob/main/notebooks/tests/requirements.txt) | @@ -66,7 +66,7 @@ wheel must match the GDAL native version installed on the host, so it is installed separately against the detected version. The native side is pinned via the init script (see below). The lightweight `pyrx` build path has no such exception — it uses rasterio's bundled-GDAL binary wheel, which is hash-pinned -in `requirements-pyrx-ci.txt` like every other dependency (no native GDAL, no +in `requirements-light-ci.txt` like every other dependency (no native GDAL, no JAR). ### Pinned GDAL native + multi-layer trust chain @@ -189,10 +189,10 @@ with the bundle by GeoBrix version. ### 3. Pin the GeoBrix version in your cluster libraries -GeoBrix is **Beta** — APIs may break to stabilize, and there are no function -aliases. Pin the exact wheel and JAR version in your cluster configuration -and only bump deliberately. See the -[Beta Release Notes](./beta-release-notes) for the change list per version. +GeoBrix APIs may change to stabilize, and there are no function aliases — +one canonical name per function. Pin the exact wheel and JAR version in your +cluster configuration and only bump deliberately. See the +[Release Notes](./release-notes) for the change list per version. ### 4. Restrict GDAL drivers for untrusted inputs diff --git a/docs/docs/serverless-and-memory.mdx b/docs/docs/serverless-and-memory.mdx new file mode 100644 index 000000000..966900159 --- /dev/null +++ b/docs/docs/serverless-and-memory.mdx @@ -0,0 +1,156 @@ +--- +sidebar_position: 6 +title: Serverless & Memory +--- + +# Serverless & Memory + +GeoBrix reads, writes, and processes rasters and vectors with defined per-runtime memory limits so that pipelines run on Databricks Serverless without OOM risk. Raster and vector have separate memory stories: rasters are governed by a connect-aware tile cap; vectors are bounded by pushdown filters and the FILE-column table path. This page covers both stories, the FILE Delta-table fast path that benefits both data types, and the edge cases to keep in mind. + +## Connect-aware memory cap (raster tiles) + +When GeoBrix opens a raster tile it decides whether to **stream the whole file into memory** or **open it lazily via a local-file reference**. The decision is governed by a per-runtime threshold: + +| Runtime | Default stream cap | Override | +|---|---|---| +| Serverless / Spark Connect | **64 MiB** | `GBX_STREAM_MAX_BYTES=` env var | +| Classic (non-Connect) cluster | **256 MiB** | `GBX_STREAM_MAX_BYTES=` env var | + +- **At or under the cap** — the tile is read in full via a FILE byte-range stream in one network round-trip. No FUSE mount, no local copy. +- **Over the cap** — the tile is opened lazily via a local-file reference backed by the FILE handle. Bytes are paged from the Volume as the rasterio reader requests them; the tile never fully lands in executor memory. + +The lazy path is a **memory-safety mechanism**, not a performance optimization. For most tile sizes (well under 64 MiB), the streaming path is faster. `GBX_STREAM_MAX_BYTES` is a per-executor environment variable — set it in your cluster's environment variables section or in the job task environment to adjust the threshold for your tile sizes. + +This cap governs **raster-tile materialization only**. Vector reads have a different memory profile — see the next section. + +## Vector memory profile + +Vector reads are not governed by the raster connect-aware cap. The `vector_gbx` reader assigns **one Spark task per source file** and reads the entire file in a single pyogrio pass. For large files this can approach the ~1 GB Serverless per-task memory limit. Three mitigations address this: + +**1. Spatial and attribute pushdown.** The `bbox` and `where` options pass filters to pyogrio so only matching features are parsed and materialized: + +```python +df = (spark.read.format("vector_gbx") + .option("bbox", "-0.5,51.3,0.1,51.7") # xmin,ymin,xmax,ymax in the layer CRS + .option("where", "population > 100000") + .load("/Volumes/…/regions.gpkg")) +``` + +Use pushdown whenever you only need a geographic or attribute subset of a large file — it keeps per-task memory well within the Serverless limit. + +**2. Staging for random-access formats.** GeoPackage (`.gpkg`) and FileGDB (`.gdb`) rely on seeked I/O. The reader stages these formats to worker-local temp via a sequential copy before opening them, which keeps FUSE reads sequential and lets GDAL seek freely on local disk. The staged copy is cached per (worker process, source path), so multiple partitions of the same file share one copy. + +**3. FILE-column vector table.** For a directory of vector files that is read repeatedly, store them as a FILE-column Delta table via `vector_file_write` or the `vector_gbx` writer with `file_mode="external"`. Reading from the Delta table resolves FILE references without per-file FUSE opens and enables the same open-amortization as the raster fast path: + +```python +from databricks.labs.gbx.pyvx.file_read import vector_file_read + +# Read from a FILE-column vector table (table mode) +df = vector_file_read(spark, "catalog.schema.roads_table", source_type="table") +``` + +**Vector writes.** The commit phase runs on the driver and assembles the output file by streaming one Arrow-IPC fragment at a time — driver RAM is bounded at one batch, not the full dataset — for `geojson_gbx`, `shapefile_gbx`, and `gpkg_gbx`. `geojsonl_gbx` goes further: each Spark partition writes an independent shard directly to the Volume with **no driver merge at all**, so write throughput scales with parallelism. For OpenFileGDB, the driver streams fragments one at a time with OGR transaction batching (100,000 rows per commit), but this path requires the native GDAL Python bindings (`osgeo`) and is therefore only available on classic clusters. + +For very large vector output, prefer `geojsonl_gbx` (parallel shards, no driver bottleneck) or a FILE-column vector table. Assembling a single very large GeoJSON or GPKG file still routes all data through the driver's local disk — bounded in RAM but not in time or disk space. + +## FILE Delta-table read: the fast path + +The fastest way to read rasters or vectors on Serverless — and on classic clusters — is to read from a **FILE-column Delta table**: a `tilesTable` for rasters, or a table produced by `vector_file_write` (or `vector_gbx` with `file_mode="external"`) for vectors. The Delta scan resolves FILE references without per-file opens. + +Measured on 1,000 raster tiles: + +| Read path | Serverless | Classic | +|---|---|---| +| **FILE-column Delta table** (recommended) | **~1.8–2 s** | **~2 s** | +| Directory scan (`raster_gbx` DataSource) | ~30 s | ~20 s | + +**~16–17× faster on Serverless.** The gap is the per-tile open cost: the Delta-scan path amortizes it across the whole job via Arrow transport; the directory DataSource opens each tile individually on the worker. + +The same open-amortization benefit applies to vector: `vector_file_read(..., source_type="table")` avoids per-file FUSE opens for a FILE-column vector table. Measured vector-table throughput numbers are not yet available; the mechanism is the same as the raster case. + +### Ingest workflow (raster) + +Ingest once, then read from the Delta table for all downstream jobs: + +```python +from databricks.labs.gbx.ds.register import register +register(spark) + +# Step 1 — read as virtual tiles (no bytes loaded yet) +df = spark.read.format("raster_gbx").load("/Volumes/…/rasters/") + +# Step 2 — write to a FILE-column Delta table +(df.write.format("raster_gbx") + .option("file_mode", "external") + .save("/Volumes/…/tiles_table/")) + +# Step 3 — all downstream jobs read from the Delta table (~2 s / 1k tiles) +tiles = spark.read.format("delta").load("/Volumes/…/tiles_table/") +``` + +See [Readers & Writers](./readers-writers#file-gbx) for the full file-access base, FILE modes, and layout options. + +## What is Serverless-safe + +### Reads + +| Read path | Serverless-safe? | Notes | +|---|---|---| +| **— Raster —** | | | +| FILE-column raster table | Yes | Fastest path; ~1.8–2 s / 1k tiles | +| `raster_gbx` directory read | Yes | Size-gated; ~30 s / 1k tiles — prefer Delta-table path | +| Virtual tiles (default) | Yes | No bytes materialized at the reader stage | +| Materialized tiles (`virtualTiles=false`) | Yes | Bytes size-gated by the connect-aware cap | +| `rst_fromfile` (default, virtual) | Yes | No bytes loaded at read time | +| `rst_fromfile(materialize=True)` | **Caution** — see [below](#rst_fromfile-materialize-true) | Errors if file exceeds cap | +| `rst_fromcontent` | Yes, with size caveat | Bytes are in a column — [you own their size](#rst_fromcontent-you-own-the-bytes) | +| **— Vector —** | | | +| FILE-column vector table (`vector_file_read`, table mode) | Yes | Open-amortized; avoids per-file FUSE opens | +| `vector_gbx` with `bbox` / `where` pushdown | Yes | Only matching features parsed; recommended for large files | +| `vector_gbx` single-file read (small–medium) | Yes | One task per file; memory scales with file size | +| `vector_gbx` single-file read (very large file) | **Caution** — see [below](#large-vector-file-reads) | Can approach the ~1 GB per-task Serverless limit | + +### Writes and ingest + +| Writer path | Serverless-safe? | Notes | +|---|---|---| +| **— Raster —** | | | +| `RasterGbxWriter` (raster formats) | Yes | Block-streams large tiles through the write path | +| COG writer (`cog_gbx`) | Yes | Auto-routes large sources to a driver-side encoder | +| Large tile with pending warp or clip | **Caution** — see [below](#large-tile-pending-warp-or-clip) | Full materialization before write | +| **— Vector —** | | | +| `geojsonl_gbx` (partitioned shards) | Yes | No driver merge; each partition writes independently | +| `geojson_gbx` / `shapefile_gbx` / `gpkg_gbx` | Yes | Driver commit streams one fragment batch at a time | +| Very large single-file output (`geojson_gbx`, `gpkg_gbx`) | **Caution** — see [below](#large-vector-file-reads) | All data flows through driver local disk; prefer `geojsonl_gbx` or FILE table | +| `file_gdb_gbx` (OpenFileGDB write) | No | Requires native GDAL (`osgeo`); classic clusters only | + +## Caveats + +### `rst_fromfile(materialize=True)` {#rst_fromfile-materialize-true} + +`rst_fromfile` with `materialize=True` forces the raster bytes into the tile's `raster` field immediately. If the file is larger than the connect-aware cap (**64 MiB on Serverless / Spark Connect**, 256 MiB on classic), this raises a `ValueError`. The error is intentional — silently materializing an oversized tile would OOM the executor. + +**Fix:** Use the virtual default (`materialize=False`, or omit the argument). Virtual tiles are Serverless-safe by construction — pixels are read lazily, size-gated, at each downstream operation. + +### `rst_fromcontent` — you own the bytes {#rst_fromcontent-you-own-the-bytes} + +`rst_fromcontent(content, driver)` builds a materialized tile from bytes already in a column (e.g. from Spark's `binaryFile` reader or a prior operation). Those bytes are already in executor memory before this call; GeoBrix does not guard their size. On Serverless / Spark Connect, keep individual `content` values at or under **64 MiB**. For larger sources, use the virtual-tile raster readers (the default) and let the connect-aware cap gate pixel reads lazily. + +### Large tile with pending warp or clip {#large-tile-pending-warp-or-clip} + +If a tile reaching the writer has a pending warp or clip operation that has not yet been resolved, the writer must fully materialize the tile to apply the transform before writing. A tile over the connect-aware cap in this state will fully materialize, not lazily page. Split such tiles or resolve the warp/clip in an intermediate step before reaching the writer. + +### Large vector file reads and writes {#large-vector-file-reads} + +The `vector_gbx` reader assigns one Spark task per source file and reads it whole. A single large shapefile, GPKG, or GeoJSON file approaching hundreds of megabytes can push close to the ~1 GB Serverless per-task memory limit. + +**For reads:** pass `bbox` and/or `where` options to restrict what is parsed. For GPKG and FileGDB, the reader stages the file to worker-local temp; memory is bounded by the features that match after pushdown. For files you read repeatedly, ingest them into a FILE-column vector table and read from there instead. + +**For writes:** `geojsonl_gbx` is the safest choice for large outputs — each Spark partition writes an independent `.geojsonl` shard to the Volume with no driver-side assembly. `geojson_gbx`, `shapefile_gbx`, and `gpkg_gbx` all stream driver-side (bounded RAM), but a single very large output file still requires the full dataset to flow through the driver's local disk. For outputs that will be read back repeatedly, write to a FILE-column vector table via `vector_file_write`. + +## See also + +- [Readers & Writers](./readers-writers) — capability tiers, read-options matrix, file-access base +- [Virtual Tiles](./api/virtual-tiles) — how virtual tiles defer pixel I/O +- [Performance](./api/performance) — open amortization, grouping patterns +- [Benchmarking](./api/benchmarking) — measured tier-vs-tier and FILE-capability results diff --git a/docs/docs/support.mdx b/docs/docs/support.mdx index 1502c7562..16f03a1c5 100644 --- a/docs/docs/support.mdx +++ b/docs/docs/support.mdx @@ -33,7 +33,7 @@ They will be reviewed as time permits, but there are **no formal SLAs for suppor When filing an issue, please include: #### Environment Information -- Databricks Runtime version (e.g., DBR 17.3 LTS or 18 LTS) +- Databricks Runtime version (e.g., DBR 17.3 LTS, 18 LTS, or 19 LTS) - Cluster configuration (node types, number of workers) - GeoBrix version - Python/Scala version @@ -53,7 +53,7 @@ When filing an issue, please include: {`**Environment:** -- DBR: 17.3 LTS (or 18 LTS) +- DBR: 17.3 LTS (or 18 LTS or 19 LTS) - Cluster: 2 workers, Standard_DS3_v2 - GeoBrix: 0.4.0 - Python: 3.12 @@ -83,6 +83,20 @@ OOM error on executors.`} (The `gdal` reader shown above is part of the heavyweight tier — see [Choosing an Execution Tier](./api/execution-tiers).) +### Reproduce on Databricks Runtime 19 + +On a Databricks Runtime 19 cluster (light tier): + + +{`# Adapt the path below to your staged wheel location (see installation guide). +%pip install "geobrix[light] @ file:///Volumes////geobrix/geobrix--py3-none-any.whl" + +from databricks.labs.gbx import pyrx + +# Define sample code here to reproduce the issue. +# Include the output and full error traceback.`} + + ## Community Resources ### GitHub Discussions diff --git a/docs/docs/writers/cog.mdx b/docs/docs/writers/cog.mdx new file mode 100644 index 000000000..bf5a71b6d --- /dev/null +++ b/docs/docs/writers/cog.mdx @@ -0,0 +1,257 @@ +--- +sidebar_position: 3 +sidebar_label: COG +--- + +import CodeFromTest from '@site/src/components/CodeFromTest'; +import cogExamples from '!!raw-loader!../../tests/python/readers/cog_gbx_examples.py'; + +# COG Writer + +`cog_gbx` is the **COG preparation writer**: it takes path-reference rows from +`file_gbx`, opens each source file, and converts it to a spec-valid +Cloud-Optimized GeoTIFF — one master COG per source file — written to the +output directory. + +A master COG carries: +- **Internal tiling** — the file is divided into a regular grid of blocks so + range-reads can fetch any window without scanning the whole file. +- **Overview levels** — pre-built down-sampled versions at progressively + coarser resolution, so the `cog_gbx` reader (and rio-tiler / GDAL) can + serve a low-zoom clip without decoding full-resolution data. +- **DEFLATE (or chosen) compression** — applied per block. + +The output is a directory of `.tif` files that can be read back with the +`cog_gbx` reader for windowed (bbox-clipped) access. + +It converts per partition by default (for moderate files) and offers an opt-in +[`driverMode`](#drivermode-preparing-large-files) that prepares large files on +the driver. For the concepts behind COG preparation — striped vs. tiled layout, +overviews, BigTIFF, and the memory model — see +[Large Rasters](../api/large-rasters). + +`cog_gbx` also writes **mosaics**: instead of one master COG per source, it +can tile each source into bounded mini-COGs and optionally emit a portable +`mosaic.vrt` index over them — so a source too large to hold in a single COG +(or in per-task memory) is split into pieces every `rst_*` function can process +independently. The tile grid can be native pixel-based (`gridSystem="none"`, +default) or quadbin cell-aligned (`gridSystem="quadbin"`). The VRT index is +optional (`writeVrt="false"` writes tiles only). +See [Mosaic mode](#mosaic-mode) below for the writer options, and the +[VRT & Mosaics](../api/vrt-mosaic) reference for the full pattern. + +:::note Lightweight only +`cog_gbx` is a pure-Python lightweight DataSource (no JAR). It uses GDAL's +`driver="COG"` creation path internally, which produces spec-compliant COGs +without the memory overhead of re-encode paths. Register with `register(spark)` +before use. +::: + +## Input schema + +`cog_gbx` takes **path-reference rows** from `file_gbx` — not a raster tile +DataFrame. The required columns are the ones `file_gbx` emits: + +``` +root + |-- path: string — absolute path to the source raster file + |-- name: string — (used to derive the output filename when nameCol is unset) + |-- ... other file_gbx columns (ignored by the writer) +``` + +Alternatively, supply any DataFrame with at minimum a `path` string column and +set `nameCol="path"` to use the basename of the path as the output filename. + +## COG Options + +These options control per-file COG encoding. For VRT-mosaic mode +(`vrtMosaic` / `gridSystem`), see [Mosaic-mode options](#mosaic-mode-options) +below — mosaic mode has its own tile-grid and index options in addition to +these encoding options. + +| Option | Default | Description | +|--------|---------|-------------| +| `cogBlockSize` | `"512"` | Internal tile size in pixels for the COG grid. | +| `cogOverviewResampling` | `"AVERAGE"` | Resampling algorithm for overview levels. Any GDAL-supported value: `AVERAGE`, `NEAREST`, `BILINEAR`, `CUBICSPLINE`, `LANCZOS`, … | +| `compress` | `"auto"` | Compression codec: `auto` (size-adaptive ZSTD+predictor — recommended), `zstd`, `deflate`, `lzw`, `none`. See [Materialized Compression](../api/materialized-compression). | +| `compressLevel` | (codec-dependent) | Compression level: for `zstd` and `deflate` only. Ignored when `compress="auto"`. | +| `predictor` | (dtype-matched) | TIFF predictor tag (1–3) for byte reordering. Ignored when `compress="auto"`. | +| `cogCompression` | (deprecated) | **Deprecated alias** for `compress`; use `compress` instead. | +| `cogSubdataset` | none | Subdataset to select from a multi-subdataset source (e.g. a NetCDF variable). | +| `cogSkipIfExists` | `"true"` | Skip a source whose output already exists — idempotent resume after an interrupted run. | +| `cogBigTiff` | `"YES"` | BigTIFF policy: `YES` (always — required for outputs over ~4 GiB), `IF_SAFER`/`IF_NEEDED` (size-adaptive), `NO` (force classic TIFF, fails past ~4 GiB). See the [BigTIFF note](../api/large-rasters#raster-formats-striped-tiled-and-cog). | +| `driverMode` | `"false"` | Route conversion to the **driver** instead of per-partition workers — the mode for large single files. See [driverMode](#drivermode-preparing-large-files) below. | +| `driverModeVerbose` | `"true"` | In `driverMode`, print per-file progress from the driver. | +| `nameCol` | `"name"` | Column whose value becomes the output filename (without extension). Defaults to the `name` column emitted by `file_gbx`. | + +## Register + +```python +from databricks.labs.gbx.ds.register import register +register(spark) +``` + +## Prepare master COGs + + + +## Output + +One `.tif` file per input row, written under the target directory. Output filenames +are derived from the `name` column (or `nameCol` override) with a `.tif` extension. + +Each output file passes `rio_cogeo.cogeo.cog_validate` — the `cog_gbx` writer +uses GDAL's `driver="COG"` creation path, which produces a genuine, spec-valid COG +in a single encode pass. + +## Prepare then read + +The `cog_gbx` writer and reader are designed to work together as a two-step +preparation + windowed-read pattern: + +``` +file_gbx → cog_gbx writer (prepare: one master COG per source file) + ↓ + cog_gbx reader + bbox (read: fetch only the AOI window) +``` + +After preparation, read with bbox clip: + + + +## `driverMode`: preparing large files + +By default `cog_gbx` converts each source **inside a distributed write task** +(per partition). On Databricks Serverless a write task runs under a fixed +per-task memory ceiling (on the order of 1 GB) that no instance size raises, and +GDAL's overview build for a **very large single source** (roughly 1 GiB or more, +especially a striped GeoTIFF) can exceed it. The default per-partition mode is +therefore intended for **moderate files**. + +Set **`driverMode="true"`** to prepare large files. In this mode the write step +on the workers only gathers the source paths (no conversion), and the actual +conversion runs on the **driver**, which is not under the per-task worker +ceiling: + +```python +(spark.read.format("file_gbx").load(input_dir) + .write.format("cog_gbx") + .option("driverMode", "true") + .option("cogSkipIfExists", "true") + .mode("overwrite") + .save(output_dir)) +``` + +Driver-orchestrated preparation streams block-by-block and processes one file at +a time, so peak memory stays flat (~2 GiB in testing) regardless of source size +or batch count — a **standard Serverless driver handles multi-gigabyte sources**; +no classic cluster is required. See +[Large Rasters → memory footprint](../api/large-rasters#memory-footprint-and-staying-on-standard-serverless) +for the full explanation. + +:::warning Long driverMode writes: call `prepare_cogs` directly +In `driverMode` the conversion runs inside the `.save()` call. A write that +blocks for many minutes — a large batch, or very large files (roughly 1 GB/min) +— can have its connection cancelled and fail even though the conversion is fine. +If you hit this, prepare the files by calling `prepare_cogs` **directly** in your +notebook instead of through the writer — it is plain Python on the driver with +no such connection, and it is idempotent so re-running resumes cleanly. See +[Large Rasters → Writing COGs](../api/large-rasters#writing-cogs-and-why). +::: + +## Mosaic mode + +For sources too large to process as a single master COG — or when you want a +spatially indexed, portable tile set — use **mosaic mode**. Mosaic mode has two +independent choices: + +- **Tile grid (`gridSystem`)** — how tiles are aligned on disk and (for DGGS + grids) tagged with a cell identifier. `"none"` (native pixel tiling, the + default), `"quadbin"` (cell-aligned to the quadbin grid, reprojected to + EPSG:3857), or `"h3"` (cell-aligned to the h3 grid, reprojected to EPSG:4326, + hex-clipped). BNG is planned. +- **VRT index (`writeVrt`)** — whether to emit a `mosaic.vrt` index alongside + the tiles. Defaults to `"true"`. Set `"false"` to write tiles only (build the + index later with `mint_vrt`, or enumerate tiles directly). + +Activate mosaic mode by passing `vrtMosaic="true"` or by supplying `gridSystem`. +For native tiling, `vrtMosaic="true"` alone is enough: + +```python +( + sources + .write.format("cog_gbx") + .option("vrtMosaic", "true") + .option("tileSize", "1024") # optional; defaults to 1024 + .mode("overwrite") + .save(output_dir) +) +``` + +### Mosaic-mode options + +These options apply when `vrtMosaic="true"`. All the COG encoding options above +(`cogBlockSize`, `cogOverviewResampling`, `compress`, `compressLevel`, +`predictor`, `cogBigTiff`, `cogSkipIfExists`) pass through to each mini-COG. + +| Option | Default | Description | +|--------|---------|-------------| +| `vrtMosaic` | — | Set `"true"` to activate mosaic mode. (Supplying `gridSystem` also activates it; one of the two must be present.) | +| `gridSystem` | `"none"` | Tile grid. `"none"` (native pixel tiling, default), `"quadbin"` (cell-aligned to the quadbin grid, reprojected to EPSG:3857), or `"h3"` (cell-aligned to the h3 grid, reprojected to EPSG:4326, hex-clipped). BNG is planned. | +| `gridResolution` | — | Grid resolution. Required when `gridSystem="quadbin"` or `gridSystem="h3"`; invalid with `gridSystem="none"`. Quadbin: range `0`–`20`. H3: range `0`–`15`. | +| `tileSize` | `"1024"` | Tile edge length in pixels (square tiles). The last row and column are clipped to the source boundary. Valid only with `gridSystem="none"`. | +| `overlapPercent` | `"0"` | Tile-edge halo as a percent of `tileSize` (e.g. `5` adds `ceil(tileSize·5/100)` px on each side, clamped to source bounds). Valid only with `gridSystem="none"`. | +| `mergeStrategy` | `"none"` | How spatially overlapping tiles from different sources blend: `none` (last-write wins), `min`, `max`, `avg`, `first`, `last`. | +| `pruneEmpty` | `"true"` | Skip tiles whose pixels are entirely NoData — saves storage and reader work on sparse sources. | +| `writeVrt` | `"true"` | Emit `mosaic.vrt` alongside the tiles. `"false"` writes tiles only — build the index later with `mint_vrt` or enumerate tiles directly. | +| `vrtPaths` | `"relative"` | Tile paths inside the VRT: `"relative"` (bare filenames — directory is portable) or `"absolute"` (works from any location). | + +:::note Disallowed combinations raise a clear error +`driverMode=true` cannot combine with mosaic mode (mosaic writes per-tile +mini-COGs on executors; `driverMode` produces a single driver-side COG). The +DGGS-only options (`gridMinResolution`, `gridMaxResolution`, +`gridStepResolution`) are rejected for `gridSystem="none"`, and `tileSize` / +`overlapPercent` are rejected for any grid-aligned system. +::: + +See [VRT & Mosaics](../api/vrt-mosaic) for the full pattern — quadbin mosaic +with `tile.metadata` cell ids, `mint_vrt` for on-demand transient VRTs, reading +a mosaic back, and spatial filtering of VRT loads. + +## When to use `cog_gbx` writer vs `gtiff_gbx` writer with `cog=true` + +| | `cog_gbx` writer | `gtiff_gbx` with `cog=true` | +|---|---|---| +| **Input** | path-reference rows (`file_gbx` output) | raster tile DataFrame (`(source, tile)`) | +| **Purpose** | prepare master COGs from source files | re-encode already-loaded tiles as COG | +| **Use when** | you have files on disk to convert | you have tiles in memory to write | + +For a tile DataFrame produced by a raster reader, use +`df.write.format("gtiff_gbx").option("cog", "true").save(...)`. For source +files on disk that need preparation before distributed reading, use `cog_gbx`. + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this writer and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Not used by the writer | Writers do not enumerate source files; they write a DataFrame's output partitions. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | The DataSource writer (`df.write.format(...)`) is session-less on Connect and commits via FUSE. FILE-tier writes use `gbx_file_write` at the function layer instead. | +| `gbx_file_read` (FILE-tier read) | Not in the DataSource | FILE-tier reads for raster data go through `gbx_file_read` at the function layer. | + +:::note Shared file-access layer +Lightweight writers commit via the shared [`file_gbx` file-access base](../readers-writers#file-gbx) — `file_mode`, `layout`, and the no-gating rule are described there. See also [`file_gbx` Writer](./file) for the write API. +::: + +## Next steps + +- [Large Rasters](../api/large-rasters) — formats, BigTIFF, the memory model, and `prepare_cogs` +- [VRT & Mosaics](../api/vrt-mosaic) — tile a large source into bounded mini-COGs; native or quadbin grid; optional VRT index +- [File Lister](../readers/file) — list source files before preparation +- [COG Reader](../readers/cog) — windowed read from prepared COGs +- [Raster Reader](../readers/raster) — decode rasters into the `(source, tile)` schema diff --git a/docs/docs/writers/file.mdx b/docs/docs/writers/file.mdx new file mode 100644 index 000000000..fc1f8f050 --- /dev/null +++ b/docs/docs/writers/file.mdx @@ -0,0 +1,130 @@ +--- +sidebar_label: File Writer (file_gbx) +--- + +# File Writer — `file_gbx` + +`file_gbx` is the shared write committer for all lightweight writers. You do not call it +directly through the DataSource V2 `write.format(...)` API — instead, every `*_gbx` writer +that produces a Delta table with a FILE column routes through `open_for_write` internally. +This page documents the write-side API for use in scripts and pipeline code. + +For the read side and a diagram of the full FILE flow, see the +[`file_gbx` Reader](../readers/file) page and the +[Shared file-access base](../readers-writers#file-gbx) section. + +:::note Lightweight only +`file_gbx` write support is light-tier only. The heavyweight (JAR-backed) writers use +their own GDAL/OGR output paths and are unaffected. +::: + +## `open_for_write` — write committer + +```python +from databricks.labs.gbx.ds.file_gbx import open_for_write + +open_for_write( + spark, + df, # DataFrame with a 'tile' struct or pre-flattened columns + "main.geo.raster_tiles", # fully-qualified Delta table name + file_mode="auto", # "auto" | "managed" | "external" | "fuse" + filespace="/Volumes/main/geo/store", # required for managed mode + layout="order", # "order" | "cluster" | "plain" + overwrite=False, + file_col="tile_file", # name of the FILE-typed column +) +``` + +### `file_mode` options + +| Value | Condition | What happens | +|---|---|---| +| `"auto"` | FILE available + `filespace` given | → `"managed"`: `create_file` writes a MANAGED FILE column | +| `"auto"` | FILE available, no `filespace` | → `"external"`: `try_to_file` writes an EXTERNAL FILE column | +| `"auto"` | No FILE (fuse tier, DBR < 13) | → `"fuse"`: plain Delta, path STRING / raster BINARY | +| `"managed"` | explicit, `filespace` required | `create_file` — FILE lifecycle managed by the table | +| `"external"` | explicit | `try_to_file` — FILE reference pointing to an existing Volume path | +| `"fuse"` | explicit | Plain Delta write regardless of FILE capability | + +Requesting `"managed"` without a `filespace` raises `ValueError` immediately. Requesting +`"managed"` or `"external"` on a fuse-only runtime raises a clear error with upgrade steps. + +### `layout` options + +| Value | Behaviour | +|---|---| +| `"order"` | `ORDER BY path` at write time (default — scan-friendly) | +| `"cluster"` | `CLUSTER BY path` in the DDL (FILE-mode tables only); durable clustering requires a subsequent `OPTIMIZE
` run | +| `"plain"` | No ordering — fastest write, scan order determined by the cluster | + +:::caution CLUSTER BY needs OPTIMIZE +Writing with `layout="cluster"` declares the clustering column in the table DDL but does +**not** immediately reorganize existing data. Run `OPTIMIZE
` afterward to apply +durable clustering. On FUSE-mode tables `layout="cluster"` falls back to `ORDER BY` with +a warning, because `CLUSTER BY` requires a FILE-column table. +::: + +`partitionBy` is not supported. Passing an invalid layout value raises `ValueError`. + +## Vector writer FILE options + +The lightweight vector writers (`vector_gbx`, `shapefile_gbx`, `geojson_gbx`, `gpkg_gbx`, +`file_gdb_gbx`) also accept the FILE write options as DataSource V2 options: + +```python +( + df.write + .format("shapefile_gbx") + .option("fileMode", "managed") # "fuse" (default) | "managed" | "external" + .option("filespace", "/Volumes/main/geo/store") # required for managed + .option("layout", "order") # "order" | "cluster" | "plain" + .mode("overwrite") + .save("main.geo.road_shapefiles") # Delta table name (when fileMode != fuse) + # or /Volumes/… path (when fileMode = fuse) +) +``` + +| Option | Default | Description | +|---|---|---| +| `fileMode` | `"fuse"` | Write mode: `"fuse"` (plain file on Volume), `"managed"` (FILE MANAGED), `"external"` (FILE EXTERNAL) | +| `filespace` | — | Required when `fileMode="managed"`: the `/Volumes/…` path for the managed filespace | +| `layout` | `"order"` | Row ordering: `"order"`, `"cluster"`, or `"plain"` (see above) | + +When `fileMode="fuse"`, `.save(path)` receives a Volume path and the writer assembles +a single file on FUSE as usual. When `fileMode="managed"` or `"external"`, `.save(table)` +receives a fully-qualified Delta table name and the writer routes through `open_for_write`. + +## `ingest_files` — register existing files as FILE MANAGED + +`ingest_files` reads files from an external Volume path via `read_files(format=>'file')` +(DBR 13.3+) and inserts them as FILE MANAGED references into a Delta table, without copying +the byte content: + +```python +from databricks.labs.gbx.ds.file_gbx import ingest_files + +ingest_files( + spark, + src="/Volumes/main/geo/archive/rasters", + target="main.geo.raster_registry", + filespace="/Volumes/main/geo/managed_store", + file_col="tile_file", + layout="order", + recursive=True, + overwrite=False, # CREATE TABLE IF NOT EXISTS — idempotent +) +``` + +`ingest_files` requires FILE support (DBR 13.3+). On FUSE-only runtimes it raises +`ValueError` — use `open_for_write(file_mode="fuse")` for a plain Delta write instead. + +The managed table schema is `(path STRING, FILE MANAGED)`. On the first call +the table is created (with `CREATE TABLE IF NOT EXISTS` when `overwrite=False`), so +repeated calls are idempotent. + +## Next steps + +- [`file_gbx` Reader](../readers/file) — enumeration, `include_hidden`, filters, `open_for_read` +- [Shared file-access base](../readers-writers#file-gbx) — capability tiers, the no-gating rule, layout options +- [Raster Writer](./raster) — write raster tiles via the shared committer +- [Vector Writer](./vector) — vector FILE mode options (`fileMode`, `filespace`, `layout`) diff --git a/docs/docs/writers/filegdb.mdx b/docs/docs/writers/filegdb.mdx index f1d1c0e3e..abcf10b1b 100644 --- a/docs/docs/writers/filegdb.mdx +++ b/docs/docs/writers/filegdb.mdx @@ -133,3 +133,18 @@ df.write.format("file_gdb_gbx").mode("overwrite").save("/Volumes/main/geo/export ``` Each partition is written concurrently, then merged into one output file. See [Benchmarking](../api/benchmarking) for light-vs-heavy export figures. + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this writer and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Not used by the writer | Writers do not enumerate source files; they write a DataFrame's output partitions. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | The DataSource writer (`df.write.format("file_gdb_gbx")`) is session-less on Connect and commits via FUSE. FILE-tier writes use `vector_file_write` at the function layer instead. | +| `gbx_file_read` (FILE-tier read) | Not in the DataSource | FILE-tier reads for vector data go through `vector_file_read` at the function layer. | + +:::note Shared file-access layer +Lightweight writers commit via the shared [`file_gbx` file-access base](../readers-writers#file-gbx) — `file_mode`, `layout`, and the no-gating rule are described there. See also [`file_gbx` Writer](./file) for the write API. +::: diff --git a/docs/docs/writers/geojson.mdx b/docs/docs/writers/geojson.mdx index 0b5bfb11d..0361511f0 100644 --- a/docs/docs/writers/geojson.mdx +++ b/docs/docs/writers/geojson.mdx @@ -110,3 +110,18 @@ df.write.format("geojson_gbx").mode("overwrite").save("/Volumes/main/geo/exports ``` Each partition is written concurrently, then merged into one output file. See [Benchmarking](../api/benchmarking) for light-vs-heavy export figures. + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this writer and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Not used by the writer | Writers do not enumerate source files; they write a DataFrame's output partitions. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | The DataSource writer (`df.write.format("geojson_gbx")`) is session-less on Connect and commits via FUSE. FILE-tier writes use `vector_file_write` at the function layer instead. | +| `gbx_file_read` (FILE-tier read) | Not in the DataSource | FILE-tier reads for vector data go through `vector_file_read` at the function layer. | + +:::note Shared file-access layer +Lightweight writers commit via the shared [`file_gbx` file-access base](../readers-writers#file-gbx) — `file_mode`, `layout`, and the no-gating rule are described there. See also [`file_gbx` Writer](./file) for the write API. +::: diff --git a/docs/docs/writers/geojsonl.mdx b/docs/docs/writers/geojsonl.mdx index a66501449..50e1410c1 100644 --- a/docs/docs/writers/geojsonl.mdx +++ b/docs/docs/writers/geojsonl.mdx @@ -184,3 +184,18 @@ df.write.format("geojsonl_gbx").mode("overwrite").option( - [Vector Writer](./vector) — the generic OGR writer (any driver). - [Writers Overview](./overview) — all writers, register-first, and benchmarks. - [Readers Overview](../readers/overview) — the corresponding read paths. + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this writer and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Not used by the writer | Writers do not enumerate source files; they write a DataFrame's output partitions. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | The DataSource writer (`df.write.format("geojsonl_gbx")`) is session-less on Connect and commits via FUSE. FILE-tier writes use `vector_file_write` at the function layer instead. | +| `gbx_file_read` (FILE-tier read) | Not in the DataSource | FILE-tier reads for vector data go through `vector_file_read` at the function layer. | + +:::note Shared file-access layer +Lightweight writers commit via the shared [`file_gbx` file-access base](../readers-writers#file-gbx) — `file_mode`, `layout`, and the no-gating rule are described there. See also [`file_gbx` Writer](./file) for the write API. +::: diff --git a/docs/docs/writers/geopackage.mdx b/docs/docs/writers/geopackage.mdx index 87a8f42d2..e994e2326 100644 --- a/docs/docs/writers/geopackage.mdx +++ b/docs/docs/writers/geopackage.mdx @@ -112,3 +112,18 @@ df.write.format("gpkg_gbx").mode("overwrite").save("/Volumes/main/geo/exports/di ``` Each partition is written concurrently, then merged into one output file. See [Benchmarking](../api/benchmarking) for light-vs-heavy export figures. + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this writer and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Not used by the writer | Writers do not enumerate source files; they write a DataFrame's output partitions. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | The DataSource writer (`df.write.format("gpkg_gbx")`) is session-less on Connect and commits via FUSE. FILE-tier writes use `vector_file_write` at the function layer instead. | +| `gbx_file_read` (FILE-tier read) | Not in the DataSource | FILE-tier reads for vector data go through `vector_file_read` at the function layer. | + +:::note Shared file-access layer +Lightweight writers commit via the shared [`file_gbx` file-access base](../readers-writers#file-gbx) — `file_mode`, `layout`, and the no-gating rule are described there. See also [`file_gbx` Writer](./file) for the write API. +::: diff --git a/docs/docs/writers/geotiff.mdx b/docs/docs/writers/geotiff.mdx index af31b8d45..0d2094a55 100644 --- a/docs/docs/writers/geotiff.mdx +++ b/docs/docs/writers/geotiff.mdx @@ -86,3 +86,26 @@ GDAL-writer options (`path` / `nameCol` / `ext`, format & compression from + +:::tip Splitting source files into a tile grid? +This writer takes an already-loaded tile DataFrame (`(source, tile)` schema). +If you have **source files on disk** that you want to split into a grid of +mini-COGs — native pixel tiling or quadbin cell-aligned — use `cog_gbx` instead: +it reads raw files, tiles them, and optionally emits a `mosaic.vrt` index. +See the [COG Writer](./cog#mosaic-mode) and [VRT & Mosaics](../api/vrt-mosaic). +::: + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this writer and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Not used by the writer | Writers do not enumerate source files; they write a DataFrame's output partitions. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | The DataSource writer (`df.write.format(...)`) is session-less on Connect and commits via FUSE. FILE-tier writes use `gbx_file_write` at the function layer instead. | +| `gbx_file_read` (FILE-tier read) | Not in the DataSource | FILE-tier reads for raster data go through `gbx_file_read` at the function layer. | + +:::note Shared file-access layer +Lightweight writers commit via the shared [`file_gbx` file-access base](../readers-writers#file-gbx) — `file_mode`, `layout`, and the no-gating rule are described there. See also [`file_gbx` Writer](./file) for the write API. +::: diff --git a/docs/docs/writers/netcdf.mdx b/docs/docs/writers/netcdf.mdx index 1f1841368..042215659 100644 --- a/docs/docs/writers/netcdf.mdx +++ b/docs/docs/writers/netcdf.mdx @@ -186,3 +186,18 @@ would be mislabeled as degrees. - [GeoTIFF Writer](./geotiff) — for COG/GeoTIFF raster output. - [Writers Overview](./overview) — all writer formats and the tier split. - [Raster Functions](../api/raster-functions) — tessellation, band math, tiling, `gbx_rst_merge_agg`. + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this writer and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Not used by the writer | Writers do not enumerate source files; they write a DataFrame's output partitions. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | The DataSource writer (`df.write.format(...)`) is session-less on Connect and commits via FUSE. FILE-tier writes use `gbx_file_write` at the function layer instead. | +| `gbx_file_read` (FILE-tier read) | Not in the DataSource | FILE-tier reads for raster data go through `gbx_file_read` at the function layer. | + +:::note Shared file-access layer +Lightweight writers commit via the shared [`file_gbx` file-access base](../readers-writers#file-gbx) — `file_mode`, `layout`, and the no-gating rule are described there. See also [`file_gbx` Writer](./file) for the write API. +::: diff --git a/docs/docs/writers/overview.mdx b/docs/docs/writers/overview.mdx index 50af19058..c452f185e 100644 --- a/docs/docs/writers/overview.mdx +++ b/docs/docs/writers/overview.mdx @@ -9,6 +9,30 @@ import TabItem from '@theme/TabItem'; GeoBrix provides Spark writers for geospatial file formats. +## Available Writers + +Every named writer, by tier. Lightweight (`*_gbx`) writers are pure-Python DataSource V2 (no +JAR); heavyweight writers are GDAL/Scala-backed. Detailed options and the vector column +contract follow below. + +| Format | Lightweight | Heavyweight | +|---|---|---| +| [Raster (generic)](./raster) | `raster_gbx` | `gdal` | +| [GeoTIFF](./geotiff) | `gtiff_gbx` | `gtiff_gdal` | +| [COG](./cog) | `cog_gbx` | — (light-only) | +| [PMTiles](./pmtiles) | `pmtiles_gbx` | `pmtiles` | +| [NetCDF](./netcdf) | `netcdf_gbx` | — (light-only) | +| [Vector (generic)](./vector) | `vector_gbx` | — (light-only) | +| [Shapefile](./shapefile) | `shapefile_gbx` | — (light-only) | +| [GeoJSON](./geojson) | `geojson_gbx` | — (light-only) | +| [GeoJSONL](./geojsonl) | `geojsonl_gbx` | `geojsonl_ogr` | +| [GeoPackage](./geopackage) | `gpkg_gbx` | — (light-only) | +| [File Geodatabase](./filegdb) | `file_gdb_gbx` | — (hybrid; needs native GDAL) | + +The single-file vector formats (Vector, Shapefile, GeoJSON, GeoPackage, File Geodatabase) are +**lightweight-only** to write; the heavyweight tier writes raster (`gdal` / `gtiff_gdal`), +`pmtiles`, and the sharded `geojsonl_ogr`. + ## Named Vector Formats The vector readers and writers share a small column contract for geometry and @@ -174,12 +198,27 @@ register(spark, only=["raster_gbx", "geojson_gbx"]) An unrecognized format raises `ValueError`. ::: -### Available Writers +:::tip On Serverless, write to a FILE-column table for fast reads +On Serverless, prefer writing raster or vector output to a **FILE-column Delta table** (use +`file_mode="external"` or `file_mode="managed"` on the writer). Once written, reads via +`tilesTable` (raster) or `vector_file_read(table)` (vector) use the Delta-scan path — measured +at **~2 s** per 1,000 tiles on both Serverless and classic — instead of the per-tile FUSE opens +that make the directory DataSource path **~3.7× slower on Serverless**. + +Per-tile FILE writes themselves are not the fast path; the benefit is on the read side once the +table exists. For large-dataset write throughput on Serverless, `geojsonl_gbx` (sharded) and +`gpkg_gbx` scale horizontally without single-node assembly. + +See the full [capability tiers & read-options matrix](../readers-writers#capability-tiers). +::: + +### Lightweight writers in detail | Writer | Format Name | Description | |--------|-------------|-------------| | [Raster Writer](./raster) | `raster_gbx` | Pure-Python catch-all raster writer (no JAR; DataSource V2) | | [GeoTIFF Writer](./geotiff) | `gtiff_gbx` | Pure-Python GeoTIFF writer (driver forced to GTiff) | +| [COG Writer](./cog) | `cog_gbx` | Pure-Python cloud-optimized GeoTIFF writer — converts source files to master COGs (path-direct where possible). | | [PMTiles Writer](./pmtiles) | `pmtiles_gbx` | Package a tile pyramid into spatially-sharded PMTiles archives + a catalog. | | [Vector Writer](./vector) | `vector_gbx` | Pure-Python generic vector writer (pyogrio); any OGR-supported driver. | | [Shapefile Writer](./shapefile) | `shapefile_gbx` | Pure-Python Shapefile writer (OGR driver: `ESRI Shapefile`). | @@ -227,7 +266,7 @@ Heavyweight writers are implemented as Spark DataSource V2 connectors backed by **PMTiles writer:** - **Input schema:** exactly `(z: int, x: int, y: int, bytes: binary)`. - **Mode:** `.mode("overwrite")` is required; default `ErrorIfExists` is rejected upstream by Spark. -- **Output path:** the final `.pmtiles` file, not a directory. Read support is not implemented in 0.4.0. +- **Output path:** the final `.pmtiles` file, not a directory. The heavyweight DataSource is write-only; read PMTiles via the lightweight `pmtiles_gbx` reader. :::note Heavyweight vector writing The heavyweight tier writes raster (GDAL), PMTiles, and — its first vector writer — the diff --git a/docs/docs/writers/pmtiles.mdx b/docs/docs/writers/pmtiles.mdx index 849583bd7..23329706d 100644 --- a/docs/docs/writers/pmtiles.mdx +++ b/docs/docs/writers/pmtiles.mdx @@ -263,7 +263,7 @@ Override via `.option("tileType", "")` when auto-detection isn't appropria ## Reading PMTiles -Reading PMTiles is not supported in GeoBrix 0.4.0 — `spark.read.format("pmtiles")` raises a friendly "Reading PMTiles archives is not supported in GeoBrix 0.4.0" error. Use one of the client libraries instead: +Reading PMTiles through the **heavyweight** DataSource is not supported — `spark.read.format("pmtiles")` raises a friendly error. To read PMTiles **within GeoBrix**, use the lightweight [`pmtiles_gbx` reader](../readers/pmtiles), which reads tiles from an existing archive or builds a mosaic pyramid from rasters. For browser/inspection use, the client libraries also work: - [pmtiles JS library](https://github.com/protomaps/PMTiles) for MapLibre / browser rendering. - The Python [`pmtiles`](https://pypi.org/project/pmtiles/) package for tile inspection and extraction. @@ -292,3 +292,18 @@ PMTiles is designed to be served as a single static file with HTTP `Range` reque - [Raster Functions](../api/raster-functions#rst_xyzpyramid) — Generate per-tile PNG bytes with `gbx_rst_xyzpyramid`. - [Helios notebooks](../notebooks/helios) — worked end-to-end example: the PMTiles writer packages raster XYZ pyramids (NB02) and terrain hillshade (NB03) over a San Francisco AOI. - [VectorX Function Reference](../api/vectorx-functions#st_asmvt_pyramid) — Generate per-tile MVT bytes with `gbx_st_asmvt_pyramid`. + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this writer and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Not used by the writer | Writers do not enumerate source files; they write a DataFrame's output partitions. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | The DataSource writer (`df.write.format(...)`) is session-less on Connect and commits via FUSE. FILE-tier writes use `gbx_file_write` at the function layer instead. | +| `gbx_file_read` (FILE-tier read) | Not in the DataSource | FILE-tier reads for raster data go through `gbx_file_read` at the function layer. | + +:::note Shared file-access layer +Lightweight writers commit via the shared [`file_gbx` file-access base](../readers-writers#file-gbx) — `file_mode`, `layout`, and the no-gating rule are described there. See also [`file_gbx` Writer](./file) for the write API. +::: diff --git a/docs/docs/writers/raster.mdx b/docs/docs/writers/raster.mdx index cb7b979b7..1afcc6635 100644 --- a/docs/docs/writers/raster.mdx +++ b/docs/docs/writers/raster.mdx @@ -278,7 +278,31 @@ Written rasters are standard GDAL-compatible files: +:::tip Splitting source files into a tile grid? +This writer takes an already-loaded tile DataFrame (`(source, tile)` schema). +If you have **source files on disk** that you want to split into a grid of +mini-COGs — native pixel tiling or quadbin cell-aligned — use `cog_gbx` instead: +it reads raw files, tiles them, and optionally emits a `mosaic.vrt` index. +See the [COG Writer](./cog#mosaic-mode) and [VRT & Mosaics](../api/vrt-mosaic). +::: + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this writer and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Not used by the writer | Writers do not enumerate source files; they write a DataFrame's output partitions. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | The DataSource writer (`df.write.format(...)`) is session-less on Connect and commits via FUSE. FILE-tier writes use `gbx_file_write` at the function layer instead. | +| `gbx_file_read` (FILE-tier read) | Not in the DataSource | FILE-tier reads for raster data go through `gbx_file_read` at the function layer. | + +:::note Shared file-access layer +Lightweight writers commit via the shared [`file_gbx` file-access base](../readers-writers#file-gbx) — `file_mode`, `layout`, and the no-gating rule are described there. See also [`file_gbx` Writer](./file) for the write API. +::: + ## Next Steps - [GDAL Reader](../readers/raster) — The corresponding read path. - [Raster Functions](../api/raster-functions) — Transforms to run before the write. +- [COG Writer](./cog) — Prepare source files into COGs; mosaic mode for large sources, native or quadbin tile grid. diff --git a/docs/docs/writers/shapefile.mdx b/docs/docs/writers/shapefile.mdx index f670b45ad..788a9074c 100644 --- a/docs/docs/writers/shapefile.mdx +++ b/docs/docs/writers/shapefile.mdx @@ -135,3 +135,18 @@ df.write \ The resulting archive is readable by `shapefile_gbx` (and any tool that handles `.shp.zip` / `/vsizip/`), and can be uploaded directly to GIS portals that expect a single compressed Shapefile. + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this writer and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Not used by the writer | Writers do not enumerate source files; they write a DataFrame's output partitions. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | The DataSource writer (`df.write.format("shapefile_gbx")`) is session-less on Connect and commits via FUSE. FILE-tier writes use `vector_file_write` at the function layer instead. | +| `gbx_file_read` (FILE-tier read) | Not in the DataSource | FILE-tier reads for vector data go through `vector_file_read` at the function layer. | + +:::note Shared file-access layer +Lightweight writers commit via the shared [`file_gbx` file-access base](../readers-writers#file-gbx) — `file_mode`, `layout`, and the no-gating rule are described there. See also [`file_gbx` Writer](./file) for the write API. +::: diff --git a/docs/docs/writers/vector.mdx b/docs/docs/writers/vector.mdx index cc2769098..a948abc7f 100644 --- a/docs/docs/writers/vector.mdx +++ b/docs/docs/writers/vector.mdx @@ -153,3 +153,18 @@ df.write.format("vector_gbx").option("driverName", "GeoJSON").mode("overwrite"). ``` Each partition is written concurrently, then merged into one output file. See [Benchmarking](../api/benchmarking) for light-vs-heavy export figures. + +## Common functions: used vs excluded + +See [GBX Common Functions](../common-functions) for the full catalog of shared file-access +primitives. The table below shows which are active in this writer and which are not, and why. + +| Common capability | Used here? | How / why | +|---|---|---| +| `list_local_files` (session-free enumeration) | Not used by the writer | Writers do not enumerate source files; they write a DataFrame's output partitions. | +| `gbx_file_write` / FILE-tier write | Not in the DataSource | The DataSource writer (`df.write.format("vector_gbx")`) is session-less on Connect and commits via FUSE. FILE-tier writes use `vector_file_write` at the function layer instead. | +| `gbx_file_read` (FILE-tier read) | Not in the DataSource | FILE-tier reads for vector data go through `vector_file_read` at the function layer. | + +:::note Shared file-access layer +Lightweight writers commit via the shared [`file_gbx` file-access base](../readers-writers#file-gbx) — `file_mode`, `layout`, and the no-gating rule are described there. See also [`file_gbx` Writer](./file) for the write API. +::: diff --git a/docs/docusaurus.config.js b/docs/docusaurus.config.js index 9f98a6367..e0fbb8f78 100644 --- a/docs/docusaurus.config.js +++ b/docs/docusaurus.config.js @@ -66,6 +66,14 @@ const config = { }, // Convert absolute paths to relative in HTML/CSS so static zip works when opening index.html from any folder (file://) ...(process.env.DOCS_STATIC_ZIP === '1' ? ['@someok/docusaurus-plugin-relative-paths'] : []), + [ + '@docusaurus/plugin-client-redirects', + { + redirects: [ + { from: '/docs/beta-release-notes', to: '/docs/release-notes' }, + ], + }, + ], ], presets: [ @@ -123,7 +131,7 @@ const config = { position: 'left' }, { - to: '/docs/readers/overview', + to: '/docs/readers-writers', label: 'Readers & Writers', position: 'left' }, diff --git a/docs/package-lock.json b/docs/package-lock.json index 1c9b36310..080ce8d28 100644 --- a/docs/package-lock.json +++ b/docs/package-lock.json @@ -1,14 +1,15 @@ { "name": "geobrix-docs", - "version": "0.3.0", + "version": "0.5.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "geobrix-docs", - "version": "0.3.0", + "version": "0.5.0", "dependencies": { "@docusaurus/core": "^3.1.0", + "@docusaurus/plugin-client-redirects": "3.9.2", "@docusaurus/preset-classic": "^3.1.0", "@mdx-js/react": "^3.0.0", "clsx": "^2.0.0", @@ -3645,6 +3646,30 @@ "react-dom": "*" } }, + "node_modules/@docusaurus/plugin-client-redirects": { + "version": "3.9.2", + "resolved": "https://npm-proxy.cloud.databricks.com/@docusaurus/plugin-client-redirects/-/plugin-client-redirects-3.9.2.tgz", + "integrity": "sha512-lUgMArI9vyOYMzLRBUILcg9vcPTCyyI2aiuXq/4npcMVqOr6GfmwtmBYWSbNMlIUM0147smm4WhpXD0KFboffw==", + "license": "MIT", + "dependencies": { + "@docusaurus/core": "3.9.2", + "@docusaurus/logger": "3.9.2", + "@docusaurus/utils": "3.9.2", + "@docusaurus/utils-common": "3.9.2", + "@docusaurus/utils-validation": "3.9.2", + "eta": "^2.2.0", + "fs-extra": "^11.1.1", + "lodash": "^4.17.21", + "tslib": "^2.6.0" + }, + "engines": { + "node": ">=20.0" + }, + "peerDependencies": { + "react": "^18.0.0 || ^19.0.0", + "react-dom": "^18.0.0 || ^19.0.0" + } + }, "node_modules/@docusaurus/plugin-content-blog": { "version": "3.9.2", "resolved": "https://registry.npmjs.org/@docusaurus/plugin-content-blog/-/plugin-content-blog-3.9.2.tgz", @@ -12691,9 +12716,9 @@ "license": "ISC" }, "node_modules/minimatch": { - "version": "3.1.2", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.2.tgz", - "integrity": "sha512-J7p63hRiAjw1NDEww1W7i37+ByIrOWO5XQQAzZ3VOcL0PNybwpfmV/N05zFAzwQ9USyEcX6t3UO+K5aqBQOIHw==", + "version": "3.1.5", + "resolved": "https://npm-proxy.cloud.databricks.com/minimatch/-/minimatch-3.1.5.tgz", + "integrity": "sha512-VgjWUsnnT6n+NUk6eZq77zeFdpW2LWDzP6zFGrCbHXiYNul5Dzqk2HHQ5uFH2DNW5Xbp8+jVzaeNt94ssEEl4w==", "license": "ISC", "dependencies": { "brace-expansion": "^1.1.7" @@ -15301,9 +15326,9 @@ } }, "node_modules/react-loadable-ssr-addon-v5-slorber": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/react-loadable-ssr-addon-v5-slorber/-/react-loadable-ssr-addon-v5-slorber-1.0.1.tgz", - "integrity": "sha512-lq3Lyw1lGku8zUEJPDxsNm1AfYHBrO9Y1+olAYwpUJ2IGFBskM0DMKok97A6LWUpHm+o7IvQBOWu9MLenp9Z+A==", + "version": "1.0.3", + "resolved": "https://npm-proxy.cloud.databricks.com/react-loadable-ssr-addon-v5-slorber/-/react-loadable-ssr-addon-v5-slorber-1.0.3.tgz", + "integrity": "sha512-GXfh9VLwB5ERaCsU6RULh7tkemeX15aNh6wuMEBtfdyMa7fFG8TXrhXlx1SoEK2Ty/l6XIkzzYIQmyaWW3JgdQ==", "license": "MIT", "dependencies": { "@babel/runtime": "^7.10.3" @@ -16170,15 +16195,15 @@ } }, "node_modules/serve-handler": { - "version": "6.1.6", - "resolved": "https://registry.npmjs.org/serve-handler/-/serve-handler-6.1.6.tgz", - "integrity": "sha512-x5RL9Y2p5+Sh3D38Fh9i/iQ5ZK+e4xuXRd/pGbM4D13tgo/MGwbttUk8emytcr1YYzBYs+apnUngBDFYfpjPuQ==", + "version": "6.1.7", + "resolved": "https://npm-proxy.cloud.databricks.com/serve-handler/-/serve-handler-6.1.7.tgz", + "integrity": "sha512-CinAq1xWb0vR3twAv9evEU8cNWkXCb9kd5ePAHUKJBkOsUpR1wt/CvGdeca7vqumL1U5cSaeVQ6zZMxiJ3yWsg==", "license": "MIT", "dependencies": { "bytes": "3.0.0", "content-disposition": "0.5.2", "mime-types": "2.1.18", - "minimatch": "3.1.2", + "minimatch": "3.1.5", "path-is-inside": "1.0.2", "path-to-regexp": "3.3.0", "range-parser": "1.2.0" diff --git a/docs/package.json b/docs/package.json index 72e6785b7..3426fc3c0 100644 --- a/docs/package.json +++ b/docs/package.json @@ -1,12 +1,12 @@ { "name": "geobrix-docs", - "version": "0.4.3", + "version": "0.5.0", "private": true, "scripts": { "docusaurus": "docusaurus", "start": "docusaurus start", "build": "docusaurus build", - "build:static-zip": "DOCS_STATIC_ZIP=1 docusaurus build && cp -r build build-static-zip && node scripts/relativize-static-build.js build-static-zip && npm run build", + "build:static-zip": "DOCS_STATIC_ZIP=1 docusaurus build && rm -rf build-static-zip && cp -r build build-static-zip && node scripts/relativize-static-build.js build-static-zip && npm run build", "verify-static-docs": "node scripts/verify-static-docs.mjs", "swizzle": "docusaurus swizzle", "deploy": "docusaurus deploy", @@ -17,6 +17,7 @@ }, "dependencies": { "@docusaurus/core": "^3.1.0", + "@docusaurus/plugin-client-redirects": "3.9.2", "@docusaurus/preset-classic": "^3.1.0", "@mdx-js/react": "^3.0.0", "clsx": "^2.0.0", diff --git a/docs/scripts/check-docs-examples.py b/docs/scripts/check-docs-examples.py new file mode 100644 index 000000000..f82da4ed9 --- /dev/null +++ b/docs/scripts/check-docs-examples.py @@ -0,0 +1,322 @@ +#!/usr/bin/env python3 +"""Structural guards for the RasterX documentation example tabs & output tables. + +Two checks, both pure file parsing (host-only, no Docker, no Spark): + + 1. OUTPUT TABLES — every ``*_example_output`` string constant in the doc-example + modules renders width- and tick-consistent ASCII result tables. Catches orphan + separator rows and mismatched column widths (the ``rst_bng_tessellate`` / + ``rst_quadbin_tessellate`` output-table bug class). + + 2. TAB COMPLETENESS — every ```` on the raster-functions + page renders one tab per tier listed in ``function-info.json`` ``functions[gbx_X]. + bindings``; for each declared tier the referenced source module must define BOTH the + example symbol (``X``) and its output constant (``X_output``). Also + flags single-tab ```` used on a function whose bindings declare more + than one tier. Catches missing ``_output`` constants (``rst_rastertoworldcoordx``) + and SQL-only-tab regressions (``gbx_h3_cell_bbox``). + +Exit code: 0 when clean, 1 on any gap. Run via ``gbx:test:docs-examples`` (or directly). +Pure stdlib; runs on the host, no Docker. +""" + +from __future__ import annotations + +import ast +import json +import re +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[2] +MDX = REPO_ROOT / "docs" / "docs" / "api" / "raster-functions.mdx" +FUNCTION_INFO = ( + REPO_ROOT / "src/main/resources/com/databricks/labs/gbx/function-info.json" +) +API_DIR = REPO_ROOT / "docs" / "tests" / "python" / "api" +SCALA_EXAMPLES = ( + REPO_ROOT / "docs" / "tests" / "scala" / "api" / "ScalaApiExamples.scala" +) + +_BORDER = re.compile(r"^\+[-+]*\+$") # +----+----+ +_ROW = re.compile(r"^\|.*\|$") # |..|..| + +_SUFFIX = { + "sql": "_sql_example", + "python-light": "_python_light_example", + "python-heavy": "_python_heavy_example", + "scala": "_scala_example", +} +_PROP_TO_TIER = { + "sqlSource": "sql", + "pythonLightSource": "python-light", + "pythonHeavySource": "python-heavy", + "scalaSource": "scala", +} + + +# --------------------------------------------------------------------------- # +# Check 1: output-table well-formedness +# --------------------------------------------------------------------------- # +def _example_modules() -> list[Path]: + files: set[Path] = set() + files.update(API_DIR.glob("*_sql.py")) + files.update(API_DIR.glob("rasterx_*python_light.py")) + files.add(API_DIR / "rasterx_functions.py") + return sorted(f for f in files if f.exists() and not f.name.startswith("test_")) + + +def _output_constants(path: Path): + tree = ast.parse(path.read_text()) + for node in tree.body: + if isinstance(node, ast.Assign) and isinstance(node.value, ast.Constant): + if isinstance(node.value.value, str): + for target in node.targets: + if isinstance(target, ast.Name) and target.id.endswith("_output"): + yield target.id, node.value.value + + +def _table_blocks(text: str): + cur: list[str] = [] + for line in text.splitlines(): + s = line.rstrip() + if _BORDER.match(s) or _ROW.match(s): + cur.append(s) + elif cur: + yield cur + cur = [] + if cur: + yield cur + + +def _table_problems(block: list[str]) -> list[str]: + problems: list[str] = [] + widths = {len(l) for l in block} + if len(widths) > 1: + problems.append(f"inconsistent line widths {sorted(widths)}") + borders = [l for l in block if _BORDER.match(l)] + rows = [l for l in block if _ROW.match(l) and not _BORDER.match(l)] + if borders and rows: + ticks = {i for i, c in enumerate(borders[0]) if c == "+"} + for r in rows: + pipes = {i for i, c in enumerate(r) if c == "|"} + if pipes != ticks: + problems.append( + f"column ticks misaligned: border {sorted(ticks)} != row {sorted(pipes)}" + ) + break + return problems + + +def check_output_tables() -> list[str]: + failures: list[str] = [] + scanned = 0 + for path in _example_modules(): + for name, text in _output_constants(path): + scanned += 1 + probs = [] + for block in _table_blocks(text): + probs.extend(_table_problems(block)) + if probs: + rel = path.relative_to(REPO_ROOT) + failures.append(f"{rel}::{name}: " + "; ".join(probs)) + print(f" scanned {scanned} *_example_output constants") + return failures + + +# --------------------------------------------------------------------------- # +# Check 2: tab completeness +# --------------------------------------------------------------------------- # +_name_cache: dict[Path, set[str]] = {} + + +def _module_names(rel_path: str) -> set[str]: + path = REPO_ROOT / rel_path + if path in _name_cache: + return _name_cache[path] + names: set[str] = set() + if path.exists(): + if path.suffix == ".py": + for node in ast.walk(ast.parse(path.read_text())): + if isinstance(node, ast.FunctionDef): + names.add(node.name) + elif isinstance(node, ast.Assign): + for t in node.targets: + if isinstance(t, ast.Name): + names.add(t.id) + else: + names.update(re.findall(r"\bval\s+([A-Za-z0-9_]+)\s*:", path.read_text())) + _name_cache[path] = names + return names + + +def _bindings_for(fns: dict, name: str) -> set[str]: + entry = fns.get(name) or fns.get("gbx_" + name) or {} + return set(entry.get("bindings", [])) + + +def check_tab_completeness() -> list[str]: + failures: list[str] = [] + if not FUNCTION_INFO.exists(): + print(" function-info.json absent — skipping tab-completeness check") + return failures + fns = json.loads(FUNCTION_INFO.read_text())["functions"] + mdx = MDX.read_text() + + fe_checked = 0 + for block in re.findall(r"", mdx, re.S): + nm = re.search(r'name="([^"]+)"', block) + if not nm: + continue + name = nm.group(1) + tiers = _bindings_for(fns, name) + if not tiers: + failures.append( + f"{name}: no bindings in function-info.json (no tabs render)" + ) + continue + sources = dict(re.findall(r'(\w+Source)="([^"]+)"', block)) + for prop, tier in _PROP_TO_TIER.items(): + if tier not in tiers: + continue + src = sources.get(prop) + if not src: + failures.append(f"{name}: binding '{tier}' present but no {prop} attr") + continue + example = name + _SUFFIX[tier] + output = example + "_output" + defined = _module_names(src) + if example not in defined: + failures.append( + f"{name}: tab '{tier}' missing example `{example}` in {src}" + ) + if output not in defined: + failures.append( + f"{name}: tab '{tier}' missing output `{output}` in {src}" + ) + fe_checked += 1 + + # single-tab CodeFromTest on a multi-tier function + for block in re.findall(r"", mdx, re.S): + fn = re.search(r'functionName="([^"]+)"', block) + if not fn: + continue + base = re.sub( + r"_(sql|python_light|python_heavy|scala)_example$", "", fn.group(1) + ) + tiers = _bindings_for(fns, base) + if len(tiers) > 1: + failures.append( + f"{base}: single-tab but bindings={sorted(tiers)} " + f"(should be 4-tab )" + ) + print(f" checked {fe_checked} declared FunctionExamples tabs") + return failures + + +# --------------------------------------------------------------------------- # +# Check 3: trailing-annotation consistency across identical-output tabs +# --------------------------------------------------------------------------- # +# A doc-example output constant is an ASCII table optionally followed by ONE +# trailing prose annotation describing the output, e.g. +# "(aspect in compass degrees: 0=N, 90=E, ...)". Tabs of the SAME function whose +# output TABLE is byte-identical describe the same result, so they must carry the +# same annotation. Tabs whose tables legitimately differ (e.g. the rst_*_rastertogrid* +# family: flat rows vs ARRAY> vs Seq[Seq[Row]], or the SQL dual +# heavy/BINARY block) are never compared here — so this only flags true drift. +# +# A trailing "; light tier returns ..." (or heavyweight) clause is tier-specific and +# allowed to differ; it is stripped before comparison. + +_TIER_CLAUSE = re.compile(r"^(the )?(light|heavy)(weight)?( tier)?\b", re.I) + + +def _split_output(text: str): + lines = [ln.rstrip() for ln in text.strip().splitlines() if ln.strip()] + table = "\n".join(ln for ln in lines if ln[:1] in ("+", "|")) + ann = lines[-1] if lines and lines[-1][:1] not in ("+", "|", "#") else None + return table, ann + + +def _normalize_annotation(ann): + """Canonical form for comparison: drop surrounding parens, tier-specific + clauses, case, whitespace, trailing period.""" + if ann is None: + return None + s = ann.strip() + if s.startswith("(") and s.endswith(")"): + s = s[1:-1] + parts = [p.strip() for p in s.split(";")] + parts = [p for p in parts if p and not _TIER_CLAUSE.match(p)] + s = "; ".join(parts) + s = re.sub(r"\s+", " ", s).lower().rstrip(".") + return s or None + + +def check_annotation_consistency() -> list[str]: + from collections import defaultdict + + constants: dict[str, str] = {} + for path in _example_modules(): + for name, text in _output_constants(path): + constants[name] = text + if SCALA_EXAMPLES.exists(): + stext = SCALA_EXAMPLES.read_text() + for m in re.finditer( + r'val\s+(\w+_output)\s*:\s*String\s*=\s*\n?\s*"""(.*?)"""', stext, re.S + ): + constants[m.group(1)] = m.group(2) + + groups: dict[str, dict] = defaultdict(dict) + for name, text in constants.items(): + mo = re.search(r"_(sql|python_light|python_heavy|scala)_example_output$", name) + if mo: + groups[name[: mo.start()]][mo.group(1)] = _split_output(text) + + failures: list[str] = [] + compared = 0 + for base, tiers in sorted(groups.items()): + by_table: dict[str, list] = defaultdict(list) + for tier, (table, ann) in tiers.items(): + if table.strip(): + by_table[table].append((tier, ann)) + for table, members in by_table.items(): + if len(members) < 2: + continue + compared += 1 + cores = {_normalize_annotation(ann) for _, ann in members} + if len(cores) > 1: + detail = ", ".join(f"{t}={ann!r}" for t, ann in members) + failures.append( + f"{base}: tabs with identical output table disagree on the trailing " + f"annotation — {detail}" + ) + print(f" compared {compared} identical-output tab groups") + return failures + + +def main() -> int: + print("Docs example guards (host-only):") + print("• output tables") + table_failures = check_output_tables() + print("• tab completeness") + tab_failures = check_tab_completeness() + print("• annotation consistency") + annotation_failures = check_annotation_consistency() + + all_failures = table_failures + tab_failures + annotation_failures + print() + if all_failures: + print(f"❌ docs example guards FAILED — {len(all_failures)} issue(s):") + for f in all_failures: + print(f" - {f}") + return 1 + print( + "✅ docs example guards OK — output tables well-formed, all declared tabs " + "complete, annotations consistent." + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/scripts/check-param-names.py b/docs/scripts/check-param-names.py new file mode 100644 index 000000000..000a74902 --- /dev/null +++ b/docs/scripts/check-param-names.py @@ -0,0 +1,181 @@ +#!/usr/bin/env python3 +"""Invariant A (param-name correctness) + Invariant B (cross-tier arity parity). + +Invariant A compares live Python signatures against the frozen canonical fixture +(docs/tests-function-info/canonical_param_names.txt): + - Heavy tier: EXACT equality — the heavy param list must equal canonical exactly + (same names, same order). + - Light tier: PREFIX match — the canonical list must appear as an in-order prefix of + the light param list. Light MAY have extra trailing params (e.g. virtualize_dir, + xscale, yscale) that are not in the canonical surface; that is expected and allowed. + If any of the first len(canonical) names differ, or light is shorter than canonical, + that is a violation. + +Invariant B (arity parity): for functions present in BOTH tiers, require + len(light) >= len(heavy). +Light dropping a param that heavy carries is a violation; light may add extra trailing +params (allowed by the "light tier may exceed heavy" rule). + +Functions listed in param_name_waiver.txt are exempt while the rename migration is +underway. Pure stdlib; runs on the host. Exit 0 on pass, 1 on any non-waived violation. +""" +from __future__ import annotations +import argparse, re, sys +from pathlib import Path + +REPO = Path(__file__).resolve().parents[2] +FIXTURE = REPO / "docs/tests-function-info/canonical_param_names.txt" +WAIVER = REPO / "docs/tests-function-info/param_name_waiver.txt" +PY_SRC = REPO / "python/geobrix/src" + +# Heavy shim files (databricks.labs.gbx.) and light bindings (pyrx/pyvx/pygx). +# Exclude build/lib (build artifact). +HEAVY_GLOBS = ["databricks/labs/gbx/rasterx/functions.py", + "databricks/labs/gbx/vectorx/functions.py", + "databricks/labs/gbx/gridx/bng/functions.py", + "databricks/labs/gbx/gridx/grid/functions.py", + "databricks/labs/gbx/gridx/h3/functions.py"] +LIGHT_GLOBS = ["databricks/labs/gbx/pyrx/functions.py", + "databricks/labs/gbx/pyvx/functions.py", + "databricks/labs/gbx/pygx/functions.py"] + +def _brackets_stripped(tokens: list[str]) -> list[str]: + return [t.strip().strip("[]").strip() for t in tokens if t.strip()] + +def load_fixture() -> dict[str, list[str]]: + out = {} + for line in FIXTURE.read_text().splitlines(): + line = line.rstrip() + if not line or line.startswith("#"): + continue + name, _, args = line.partition("\t") + out[name.strip()] = _brackets_stripped(args.split(",")) + return out + +def load_waiver() -> set[str]: + if not WAIVER.exists(): + return set() + entries = set() + for l in WAIVER.read_text().splitlines(): + stripped = l.strip() + if not stripped or stripped.startswith("#"): + continue + # Strip inline comment (e.g. "gbx_foo # reason") + name = stripped.split("#")[0].strip() + if name: + entries.add(name) + return entries + +def _find_def(text: str, pyname: str) -> str | None: + # Match `def (` and capture the full parenthesized arg list across newlines. + m = re.search(rf"def\s+{re.escape(pyname)}\s*\((.*?)\)\s*(->|:)", text, re.S) + if m is None: + return None + # Strip inline `# ...` comments (e.g. `def f( # noqa: E741`) that would otherwise + # be tokenized as a bogus first parameter and drop the real first arg. + return re.sub(r"#[^\n]*", "", m.group(1)) + +_IDENT_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") + +def extract_py_params(path: Path, gbx_name: str) -> list[str] | None: + pyname = gbx_name[len("gbx_"):] + if not path.exists(): + return None + arglist = _find_def(path.read_text(), pyname) + if arglist is None: + return None + params = [] + for raw in arglist.split(","): + tok = raw.strip() + if not tok or tok.startswith("*"): + continue + pname = tok.split(":")[0].split("=")[0].strip() + # Skip malformed tokens from Union[T, None] comma-splits (e.g. "None]") + if pname and pname != "self" and _IDENT_RE.match(pname): + params.append(pname) + return params + +def _first_existing(globs: list[str], gbx_name: str) -> list[str] | None: + for g in globs: + params = extract_py_params(PY_SRC / g, gbx_name) + if params is not None: + return params + return None + +def check_invariant_a(report: bool = False, + b_names: set[str] | None = None) -> list[str]: + """Check param-name correctness vs canonical fixture. + + Heavy tier: exact match (canonical == heavy). + Light tier: prefix match (canonical is an in-order prefix of light; light + may carry extra trailing params beyond the canonical length). + + If ``b_names`` is provided (functions that already have a [B] arity-gap + violation), light-tier [A] checks are suppressed for those functions — the + naming issue is masked by the missing params and cannot be fixed without + first closing the arity gap. + """ + fixture, waiver = load_fixture(), load_waiver() + b_names = b_names or set() + violations = [] + for gbx_name, canon in fixture.items(): + heavy = _first_existing(HEAVY_GLOBS, gbx_name) + light = _first_existing(LIGHT_GLOBS, gbx_name) + # Heavy: exact equality. + if heavy is not None and heavy != canon: + msg = f"[A] {gbx_name}: heavy params {heavy} != canonical {canon}" + if report or gbx_name not in waiver: + violations.append(msg) + # Light: canonical must be an in-order prefix of light. + # Skip when a [B] arity gap already covers this function — the naming + # mismatch is a symptom of missing params, not an independent naming bug. + if light is not None and gbx_name not in b_names: + n = len(canon) + if len(light) < n or light[:n] != canon: + msg = (f"[A] {gbx_name}: light params {light} does not have " + f"canonical {canon} as prefix") + if report or gbx_name not in waiver: + violations.append(msg) + return violations + + +def check_invariant_b(report: bool = False) -> list[str]: + """Check cross-tier arity parity: len(light) >= len(heavy) for both-tier functions. + + Light dropping a param that heavy has is a violation. Light may have extra + trailing params beyond heavy's count (per "light tier may exceed heavy" rule). + """ + fixture, waiver = load_fixture(), load_waiver() + violations = [] + for gbx_name in fixture: + heavy = _first_existing(HEAVY_GLOBS, gbx_name) + light = _first_existing(LIGHT_GLOBS, gbx_name) + if heavy is None or light is None: + continue # parity only meaningful when both tiers exist + if len(light) < len(heavy): + msg = (f"[B] {gbx_name}: light arity {len(light)} < heavy arity {len(heavy)}" + f" (light dropped params: {heavy[len(light):]})") + if report or gbx_name not in waiver: + violations.append(msg) + return violations + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--report", action="store_true", + help="list ALL violations ignoring the waiver") + args = ap.parse_args() + # Compute [B] first so [A] can suppress light-tier checks that are masked + # by an arity gap (the naming issue cannot be fixed without closing the gap). + b_violations = check_invariant_b(args.report) + b_names = {v.split(":")[0].replace("[B] ", "") for v in b_violations} + violations = check_invariant_a(args.report, b_names=b_names) + b_violations + if violations: + print("\n".join(sorted(violations))) + print(f"\n{len(violations)} param-name violation(s).") + return 1 + print("check-param-names: OK") + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/scripts/check-release-notes-functions.py b/docs/scripts/check-release-notes-functions.py index d396d07b8..b49f0a0b8 100644 --- a/docs/scripts/check-release-notes-functions.py +++ b/docs/scripts/check-release-notes-functions.py @@ -11,7 +11,7 @@ Collect lines matching ``^+gbx_[a-z0-9_]+`` (added registered functions; the ``+++`` file-header line is excluded by the regex). 2. For each added name, check whether it appears (substring) anywhere in - ``docs/docs/beta-release-notes.mdx``. Also accept the bare name (strip + ``docs/docs/release-notes.mdx``. Also accept the bare name (strip leading ``gbx_``) as a match -- some bullets reference the bare form. A match on either counts. 3. Exit 1 listing every added function NOT mentioned in the release notes. @@ -32,7 +32,7 @@ REPO_ROOT = Path(__file__).resolve().parents[2] REGISTERED_TXT = REPO_ROOT / "docs/tests-function-info/registered_functions.txt" -RELEASE_NOTES = REPO_ROOT / "docs/docs/beta-release-notes.mdx" +RELEASE_NOTES = REPO_ROOT / "docs/docs/release-notes.mdx" # Matches a newly added registered-function line: ``+gbx_foo_bar`` # The ``+++`` diff file-header lines are excluded because they contain a path, diff --git a/docs/scripts/extend-function-metadata.py b/docs/scripts/extend-function-metadata.py new file mode 100755 index 000000000..948054f46 --- /dev/null +++ b/docs/scripts/extend-function-metadata.py @@ -0,0 +1,449 @@ +#!/usr/bin/env python3 +""" +Parser for Scala expression builders to extract parameter names and arities. + +Reads Scala source files and extracts: + - Case class field names (strip 'Expr' suffix, convert to snake_case) + - Builder arity patterns (e.g. 'case 5 =>' vs 'case 6 =>') + - Optional parameters (detected when builder injects Literal(...) defaults) + +Output: dict keyed by function name with values like: + { + "usage_args": "tile, band_idx, [resampling]", + "field_count": 3, + "optional_from": 3 + } + +Validation: Raises if a derived usage_args doesn't match real builder arity or contains unparseable syntax. +""" + +import re +import os +from pathlib import Path +from typing import Dict, List, Optional, Tuple + +# Diagnostics collected during a scan and printed by main(). Kept module-level so the +# generator can surface them rather than swallowing an ambiguous parse. +MULTI_COMPANION_NOTES: List[str] = [] + + +def camel_to_snake(name: str) -> str: + """Convert camelCase to snake_case.""" + s1 = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name) + return re.sub('([a-z0-9])([A-Z])', r'\1_\2', s1).lower() + + +def extract_case_class_fields(scala_content: str, class_name: str) -> Optional[List[str]]: + """ + Extract field names from a case class definition, filtering out internal aggregation state. + + Example: + case class ST_Triangulate( + pointsArray: Expression, + breaklinesArray: Expression, + mergeTolerance: Expression, + ... + modeExpr: Expression + ) extends ... + + Returns: ["pointsArray", "breaklinesArray", "mergeTolerance", ..., "modeExpr"] + Filters out: fields with type Int/Long = (default value) which are aggregation internals. + """ + # Match the case class declaration and fields (until we hit ) extends or {) + pattern = rf'case\s+(?:final\s+)?class\s+{re.escape(class_name)}\s*\(\s*(.*?)\s*\)\s*(?:extends|:|$)' + match = re.search(pattern, scala_content, re.DOTALL | re.MULTILINE) + if not match: + return None + + fields_str = match.group(1) + # Split by comma, but be careful about nested parens + fields = [] + current_field = "" + paren_depth = 0 + for char in fields_str: + if char == '(': + paren_depth += 1 + current_field += char + elif char == ')': + paren_depth -= 1 + current_field += char + elif char == ',' and paren_depth == 0: + if current_field.strip(): + # Extract field declaration: "name: Type" or "name: Type = default" + field_decl = current_field.strip() + # Skip if it's an Int/Long with a default value (aggregation buffer state) + if not re.search(r':\s*(?:Int|Long)\s*=', field_decl): + # Extract just the name (before the colon) + field_name = field_decl.split(':')[0].strip() + if field_name: + fields.append(field_name) + current_field = "" + else: + current_field += char + + if current_field.strip(): + field_decl = current_field.strip() + if not re.search(r':\s*(?:Int|Long)\s*=', field_decl): + field_name = field_decl.split(':')[0].strip() + if field_name: + fields.append(field_name) + + return fields if fields else None + + +def strip_expr_suffix(field_name: str) -> str: + """Strip 'Expr' suffix and convert to snake_case.""" + if field_name.endswith('Expr'): + field_name = field_name[:-4] # Remove 'Expr' + return camel_to_snake(field_name) + + +def should_exclude_field(field_name: str) -> bool: + """Check if a field should be excluded from user-facing parameter list. + + Excludes: + - exprConf / exprConfExpr: internal Spark config state + """ + return field_name.lower() in ('exprconf', 'exprconfexpr') + + +def extract_builder_arities(scala_content: str, function_name: str) -> Optional[Tuple[int, Optional[int]]]: + """ + Extract builder arity branches from a WithExpressionInfo companion object. + + Example: + override def builder(): FunctionBuilder = (c: Seq[Expression]) => c.length match { + case 5 => ST_Triangulate(c(0), c(1), c(2), c(3), c(4), Literal("constrained")) + case 6 => ST_Triangulate(c(0), c(1), c(2), c(3), c(4), c(5)) + case n => throw ... + } + + Returns: (min_arity, max_arity) or (exact_arity, None) if only one branch. + (5, 6) means 5 or 6 args, with 6th being optional. + + Brace style must NOT matter. Both of these are in use and mean the same thing:: + + = (c: Seq[Expression]) => c.length match { // ST_Triangulate + = (c: Seq[Expression]) => { // ST_AsMvtPyramid + c.length match { + + An earlier version anchored the regex directly on ``c.length`` after the arrow, silently + returned None for the second + form, and so dropped the `[extent]` bracket from gbx_st_asmvt_pyramid — publishing an + optional argument as required, the exact defect this parser exists to prevent. Locate + `builder()` and scan its body instead of matching one specific layout. + """ + # Locate builder() and take everything to the end of the enclosing companion. Scanning a + # generous window is safe: we only look for `case =>` and `c()` tokens, and a + # companion holds at most one builder. + start = re.search(r'def\s+builder\s*\(\s*\)\s*:\s*FunctionBuilder\s*=', scala_content) + if not start: + return None + builder_body = scala_content[start.end():] + + # Pattern 1: explicit arity branches, `case 5 =>`. + arities = sorted({int(a) for a in re.findall(r'case\s+(\d+)\s*=>', builder_body)}) + + # Pattern 2: `case Seq(a, b) =>` — arity is the number of top-level binders per branch. + if not arities: + seq_arities = set() + for grp in re.findall(r'case\s+Seq\(([^)]*)\)\s*=>', builder_body): + grp = grp.strip() + seq_arities.add(len([p for p in grp.split(',') if p.strip()]) if grp else 0) + arities = sorted(seq_arities) + + # Pattern 3: single fixed form, `=> new X(c(0), c(1))` / `=> X(c.head)`. Arity is the + # highest c(i) index + 1, or 1 for the c.head/c(0)-only shape. + if not arities: + idxs = [int(i) for i in re.findall(r'c\((\d+)\)', builder_body)] + if idxs: + arities = [max(idxs) + 1] + elif re.search(r'c\.head', builder_body): + arities = [1] + + if not arities: + return None + if len(arities) == 1: + return (arities[0], None) + # Contiguous run (N, N+1, ... M) => args N+1..M are optional. Non-contiguous runs still + # bracket everything past the shortest branch: over-marking as optional is honest here, + # since the shortest form IS callable. + return (arities[0], arities[-1]) + + +def build_usage_args( + fields: List[str], + min_arity: int, + max_arity: Optional[int] +) -> Tuple[str, Optional[int]]: + """ + Build a usage_args string from field names and arity. + + Filters out internal fields (exprConf) and applies snake_case conversion. + + Returns: (usage_args, optional_from_index) where optional_from_index is 0-based position + of the first optional arg, or None if no optional args. + """ + # Filter out internal fields + user_fields = [f for f in fields if not should_exclude_field(f)] + + if max_arity is None or max_arity == min_arity: + # No optional args + param_names = [strip_expr_suffix(f) for f in user_fields[:min_arity]] + return (", ".join(param_names), None) + + # There are optional args starting at position min_arity + mandatory = [strip_expr_suffix(f) for f in user_fields[:min_arity]] + optional = [strip_expr_suffix(f) for f in user_fields[min_arity:max_arity]] + + # Format optional params with brackets + all_parts = mandatory + [f"[{o}]" for o in optional] + return (", ".join(all_parts), min_arity) + + +def parse_expression_file(filepath: str) -> Optional[Dict]: + """ + Parse a single Scala expression file and extract metadata. + + Returns: + { + "function_name": "gbx_st_triangulate", + "class_name": "ST_Triangulate", + "usage_args": "points_geom, breaklines_geom, merge_tolerance, snap_tolerance, split_point_finder, [mode]", + "optional_from": 6, + "field_count": 6 + } + or None if parsing fails. + """ + try: + with open(filepath, 'r') as f: + content = f.read() + except Exception as e: + print(f"Error reading {filepath}: {e}") + return None + + # Extract class name from "case class ClassName" + class_match = re.search(r'case\s+class\s+(\w+)', content) + if not class_match: + return None + class_name = class_match.group(1) + + # Extract function name from companion object's override def name + name_match = re.search(r'override\s+def\s+name\s*:\s*String\s*=\s*["\']([^"\']+)["\']', content) + if not name_match: + return None + function_name = name_match.group(1) + + # A file may hold SEVERAL companions sharing ONE registered SQL name, each fronting a + # different-arity case class (ST_TransformCrs / ST_TransformCrs3 both register + # gbx_st_transformcrs with 2 and 3 fields). Taking the first `case class` then describes + # only the narrowest overload: that is how `[source_crs]` was dropped from + # gbx_st_transformcrs. When it happens, prefer the WIDEST case class so the optional + # trailing args are visible, and report it so the ambiguity stays auditable. + sql_names = set(re.findall(r'override\s+def\s+name\s*:\s*String\s*=\s*["\']([^"\']+)["\']', content)) + if len(sql_names) == 1: + candidates = re.findall(r'case\s+class\s+(\w+)', content) + if len(candidates) > 1: + widest, widest_n = class_name, len(extract_case_class_fields(content, class_name) or []) + for cand in candidates: + n = len(extract_case_class_fields(content, cand) or []) + if n > widest_n: + widest, widest_n = cand, n + if widest != class_name: + MULTI_COMPANION_NOTES.append( + f"{function_name}: {len(candidates)} case classes share one SQL name; " + f"described the widest ({widest}, {widest_n} fields) not the first ({class_name})" + ) + class_name = widest + + # Extract case class fields + fields = extract_case_class_fields(content, class_name) + if not fields: + return None + + # Extract builder arities + arities = extract_builder_arities(content, function_name) + if not arities: + # If we can't parse builder, use field count as exact arity + arities = (len(fields), None) + + min_arity, max_arity = arities + usage_args, optional_from = build_usage_args(fields, min_arity, max_arity or min_arity) + + return { + "function_name": function_name, + "class_name": class_name, + "usage_args": usage_args, + "optional_from": optional_from, + "field_count": len(fields), + "arities": arities + } + + +def scan_expressions_directory(expressions_dir: str) -> Dict[str, Dict]: + """ + Recursively scan a directory of Scala expression files and extract metadata. + + Returns: dict keyed by function_name with parsed metadata. + """ + result = {} + expressions_path = Path(expressions_dir) + + if not expressions_path.is_dir(): + return result + + for scala_file in expressions_path.rglob("*.scala"): + # Skip utility files, tests, etc. + if any(skip in scala_file.name for skip in ["Util", "Config", "Test", "Mock"]): + continue + + parsed = parse_expression_file(str(scala_file)) + if parsed: + result[parsed["function_name"]] = { + "usage_args": parsed["usage_args"], + "optional_from": parsed["optional_from"], + "class_name": parsed["class_name"], + "file": str(scala_file), + } + + return result + + +def validate_usage_args(usage_args: str) -> bool: + """Validate that usage_args is well-formed (params comma-separated, optional bracketed).""" + # Basic validation: no unmatched brackets, each bracketed item is a single word + if usage_args.count('[') != usage_args.count(']'): + return False + + # Check that [x] appears only around single identifiers + invalid_brackets = re.findall(r'\[\w+,|\],\[\w+\w+\]', usage_args) + if invalid_brackets: + return False + + return True + + +def collect_scala_overrides(geobrix_root: str) -> Dict[str, str]: + """Map SQL name -> hand-written `override def usageArgs` value, if any. + + These are the human-authored baseline. A derived value must never be WORSE than one of + them, so they are compared against, not ignored. + """ + out: Dict[str, str] = {} + root = Path(geobrix_root) / "src/main/scala" + for f in root.rglob("*.scala"): + txt = f.read_text() + for m in re.finditer( + r'object\s+(\w+)\s+extends\s+[\w\s.]*?WithExpressionInfo\s*\{(.*?)\n\}', txt, re.S + ): + body = m.group(2) + nm = re.search(r'def\s+name\s*:\s*String\s*=\s*"([^"]+)"', body) + ua = re.search( + r'override\s+def\s+usageArgs\s*:\s*String\s*=\s*((?:"[^"]*"\s*\+?\s*)+)', body, re.S + ) + if nm and ua: + val = "".join(re.findall(r'"([^"]*)"', ua.group(1))).strip() + # Several companions may share a SQL name; keep the most informative text. + if val and len(val) > len(out.get(nm.group(1), "")): + out[nm.group(1)] = val + return out + + +def check_no_regression(derived: Dict[str, Dict], overrides: Dict[str, str]) -> List[str]: + """Reject a derived usage_args that loses information vs a hand-written override. + + Two losses are hard failures because both publish a wrong SQL contract: + * dropping an optional-arg bracket that the override marked (renders optional as required) + * dropping a parameter the override listed (hides a callable form) + Returns a list of human-readable problems; empty means clean. + """ + problems: List[str] = [] + for name, ov in overrides.items(): + d = derived.get(name, {}).get("usage_args") + if not d: + continue + if "[" in ov and "[" not in d: + problems.append( + f"{name}: override marks an optional arg but derived does not\n" + f" override: {ov}\n derived : {d}" + ) + ov_n = len([p for p in ov.split(",") if p.strip()]) + d_n = len([p for p in d.split(",") if p.strip()]) + if d_n < ov_n: + problems.append( + f"{name}: derived drops {ov_n - d_n} parameter(s) the override lists\n" + f" override: {ov}\n derived : {d}" + ) + return problems + + +def main(geobrix_root: str = None) -> Dict[str, Dict]: + """ + Main entry point: scan all expression directories and return parsed metadata. + + Returns: dict keyed by function_name with {"usage_args": ..., "optional_from": ...} + """ + if geobrix_root is None: + geobrix_root = os.environ.get('GEOBRIX_ROOT', '/Users/mjohns/IdeaProjects/geobrix') + + result = {} + + # Scan RasterX, VectorX expression directories + for package in ['rasterx', 'vectorx']: + expressions_dir = os.path.join(geobrix_root, f'src/main/scala/com/databricks/labs/gbx/{package}/expressions') + print(f"Scanning {expressions_dir}...") + parsed = scan_expressions_directory(expressions_dir) + result.update(parsed) + print(f" Found {len(parsed)} functions") + + # GridX is structured differently: bng/, quadbin/, h3/, grid/ subdirectories + gridx_root = os.path.join(geobrix_root, 'src/main/scala/com/databricks/labs/gbx/gridx') + for subdir in ['bng', 'quadbin', 'h3', 'grid', 'custom']: + gridx_dir = os.path.join(gridx_root, subdir) + if os.path.isdir(gridx_dir): + print(f"Scanning {gridx_dir}...") + parsed = scan_expressions_directory(gridx_dir) + result.update(parsed) + print(f" Found {len(parsed)} functions") + + # Validate all parsed usage_args + invalid = [ + f"{n}: malformed usage_args {m['usage_args']!r}" + for n, m in result.items() + if not validate_usage_args(m["usage_args"]) + ] + + # Never publish a signature worse than the hand-written override it replaces. + regressions = check_no_regression(result, collect_scala_overrides(geobrix_root)) + + if MULTI_COMPANION_NOTES: + print("\nNOTE: files with multiple companions under one SQL name:") + for n in MULTI_COMPANION_NOTES: + print(f" - {n}") + + # FAIL LOUDLY. A silent warning is how an optional arg got published as required; the + # generator treats a non-zero exit as fatal rather than emitting degraded metadata. + if invalid or regressions: + print("\nERROR: refusing to emit metadata — derived signatures are not trustworthy:") + for p in invalid + regressions: + print(f" - {p}") + raise SystemExit( + f"{len(invalid)} malformed + {len(regressions)} regression(s) vs Scala overrides. " + "Fix the parser or the Scala source; do not hand-edit function-info.json." + ) + + return result + + +if __name__ == "__main__": + import sys + import json + + root = sys.argv[1] if len(sys.argv) > 1 else None + parsed = main(root) + + # Print a summary + print(f"\nTotal functions parsed: {len(parsed)}") + + # Output as JSON for consumption by the function-info generator + print(json.dumps(parsed, indent=2)) diff --git a/docs/scripts/generate-function-info.py b/docs/scripts/generate-function-info.py index c3bffe192..de99e1fb9 100644 --- a/docs/scripts/generate-function-info.py +++ b/docs/scripts/generate-function-info.py @@ -17,8 +17,10 @@ import json import os +import re import sys -from typing import List, Optional +from pathlib import Path +from typing import Dict, List, Optional, Set # Script lives in docs/scripts/; repo root is two levels up. REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")) @@ -46,6 +48,49 @@ ) +# Load parsed builder metadata (usage_args from Scala case classes) +def _load_parsed_builders() -> Dict[str, dict]: + """Load parsed builder metadata from extend_function_metadata.py.""" + try: + # Execute the extend_function_metadata script as a subprocess + import subprocess + import json + + script_path = os.path.join( + os.path.dirname(__file__), "extend-function-metadata.py" + ) + result = subprocess.run( + [sys.executable, script_path, REPO_ROOT], + capture_output=True, + text=True, + timeout=30, + ) + # Fail LOUDLY, never silently. Returning {} on error looks like "no functions have + # signatures" and would quietly wipe usage_args out of the JSON on the next run. + if result.returncode != 0: + raise SystemExit( + "generate-function-info: signature parser FAILED — refusing to write " + "metadata that would silently drop usage_args.\n" + f"{result.stdout[-2000:]}\n{result.stderr[-2000:]}" + ) + # Find the first '{' and parse from there (skip diagnostic output) + output = result.stdout + json_start = output.find("{") + if json_start < 0: + raise SystemExit( + "generate-function-info: signature parser produced no JSON.\n" + f"{output[-2000:]}" + ) + try: + return json.loads(output[json_start:]) + except json.JSONDecodeError as e: + raise SystemExit(f"generate-function-info: parser JSON is invalid: {e}") + except SystemExit: + raise + except Exception as e: + raise SystemExit(f"generate-function-info: could not run signature parser: {e}") + + def _split_sql_statements(sql: str) -> List[str]: """Split SQL on ';' outside single-quoted string literals and -- line comments.""" stmts: List[str] = [] @@ -123,7 +168,10 @@ def format_examples_block(sql_line: str) -> str: def _collect_from_module( - mod, local_prefix: str, spark_prefix: str, registered_for_package: Optional[List[str]] = None + mod, + local_prefix: str, + spark_prefix: str, + registered_for_package: Optional[List[str]] = None, ) -> dict: """ Collect examples from one doc module. @@ -149,7 +197,7 @@ def _collect_from_module( if not callable(getattr(mod, attr)): continue middle = attr[: -len("_sql_example")] - dedicated_targets.add(spark_prefix + middle[len(local_prefix):]) + dedicated_targets.add(spark_prefix + middle[len(local_prefix) :]) result = {} for attr in dir(mod): @@ -180,7 +228,7 @@ def _collect_from_module( # is no dedicated cellunion_sql_example), but a name that DOES have its own # dedicated example function never picks up another's example as substring. middle = attr[: -len("_sql_example")] - exact_target = spark_prefix + middle[len(local_prefix):] + exact_target = spark_prefix + middle[len(local_prefix) :] for name in registered_for_package: if name not in stmt or name in result: continue @@ -209,6 +257,8 @@ def discover_and_collect(registered: Optional[List[str]] = None) -> dict: sys.path.insert(0, DOCS_ROOT) # Examples in rasterx_functions_sql.py import `path_config` from docs/tests/python/ sys.path.insert(0, os.path.join(DOCS_ROOT, "tests", "python")) + # ...and `_fixtures` (e.g. multiband_path) from docs/tests/python/api/ + sys.path.insert(0, os.path.join(DOCS_ROOT, "tests", "python", "api")) result = {} try: for module_path, local_prefix, spark_prefix in MODULES: @@ -218,7 +268,9 @@ def discover_and_collect(registered: Optional[List[str]] = None) -> dict: if registered else None ) - collected = _collect_from_module(mod, local_prefix, spark_prefix, reg_for_pkg) + collected = _collect_from_module( + mod, local_prefix, spark_prefix, reg_for_pkg + ) # First example wins for each name for k, v in collected.items(): if k not in result: @@ -293,12 +345,128 @@ def _package_for(name: str) -> str: return "other" -def build_functions_object(registered: list, doc_examples: dict) -> dict: +# All MODULES entries (including optional ones) used for base derivation. +# Each tuple: (spark_prefix, local_prefix) +_ALL_MODULE_PREFIXES = [ + (spark_prefix, local_prefix) for (_mod, local_prefix, spark_prefix) in MODULES +] + [ + (VECTORX_MODULE[2], VECTORX_MODULE[1]), + (PMTILES_MODULE[2], PMTILES_MODULE[1]), +] + + +def _base_for_spark_name(spark_name: str) -> str: + """ + Derive the 'base' name for tier example symbol lookup. + + Mirrors the SQL convention: strip spark_prefix, prepend local_prefix. + E.g. gbx_rst_avg (spark_prefix='gbx_rst_', local_prefix='rst_') -> 'rst_avg'. + Falls back to stripping 'gbx_' if no prefix matches. + """ + for spark_prefix, local_prefix in _ALL_MODULE_PREFIXES: + if spark_name.startswith(spark_prefix): + suffix = spark_name[len(spark_prefix) :] + return local_prefix + suffix + # Fallback: strip leading 'gbx_' + if spark_name.startswith("gbx_"): + return spark_name[4:] + return spark_name + + +# tier -> (doc-test source path relative to docs/, symbol template, binding label) +# Missing files are tolerated; their tier is simply absent from bindings for all functions. +# Glob patterns (containing '*') expand to all matching files; their text is concatenated. +_TIER_SCANS = [ + ( + # Covers rasterx_*_python_light.py (RasterX family files) AND + # vectorx_functions_python_light.py (VectorX, created in T1). + "tests/python/api/*_python_light.py", + "def {base}_python_light_example", + "python-light", + ), + ( + "tests/python/api/rasterx_functions.py", + "def {base}_python_heavy_example", + "python-heavy", + ), + ( + # VectorX heavy examples live in vectorx_functions.py (T2-T5 add + # *_python_heavy_example functions here; scanned alongside rasterx). + "tests/python/api/vectorx_functions.py", + "def {base}_python_heavy_example", + "python-heavy", + ), + ( + # GridX heavy examples live in gridx_functions.py (T2-T7 add + # *_python_heavy_example functions here for BNG/quadbin/custom). + "tests/python/api/gridx_functions.py", + "def {base}_python_heavy_example", + "python-heavy", + ), + ( + "tests/scala/api/ScalaApiExamples.scala", + "val {base}_scala_example", + "scala", + ), +] + + +def _scan_tier_bindings(docs_root: str, spark_names: List[str]) -> Dict[str, Set[str]]: + """ + Return spark_name -> set of tier labels whose example symbol is present in source text. + + Detection is TEXT-SCAN only (no import/execution). Missing files are tolerated + (that tier is simply absent from bindings for all functions). A glob pattern + (containing '*') expands to all matching files; their text is concatenated before + scanning, and an empty glob match is treated as missing (empty text, tier absent). + """ + found: Dict[str, Set[str]] = {name: set() for name in spark_names} + docs_path = Path(docs_root) + for rel, template, label in _TIER_SCANS: + if "*" in rel: + matches = sorted(docs_path.glob(rel)) + text = "".join(p.read_text() for p in matches) + else: + path = docs_path / rel + text = path.read_text() if path.exists() else "" + for spark_name in spark_names: + base = _base_for_spark_name(spark_name) + symbol = template.format(base=base) + # Match symbol followed by '(' (Python def) or ':'/whitespace (Scala val). + if re.search(re.escape(symbol) + r"\s*[(:]", text): + found[spark_name].add(label) + return found + + +def build_functions_object( + registered: list, + doc_examples: dict, + parsed_builders: Optional[Dict] = None, + docs_root: Optional[str] = None, +) -> dict: """ Build the "functions" object: only functions with non-empty examples from docs. Ordered by package, then sorted by function name. Section markers _package_ - separate packages. Empty usage is not allowed; only doc-derived entries are included. + separate packages. Optionally merge in parsed builder metadata (usage_args). + Empty usage is not allowed; only doc-derived entries are included. + + Each function entry gains a 'bindings' list (subset of + ["sql","python-light","python-heavy","scala"]) recording which tiers have an example. + 'sql' is present whenever the entry has a non-empty examples value. + The other three are present when a text-scan of the tier's doc-test source file finds + the corresponding example symbol (e.g. def rst_avg_python_light_example). """ + if parsed_builders is None: + parsed_builders = {} + + # Pre-compute per-tier bindings via text scan (missing files -> empty set). + tier_bindings: Dict[str, Set[str]] = _scan_tier_bindings( + docs_root or DOCS_ROOT, list(registered) + ) + + # Fixed order for the 'bindings' list. + _BINDING_ORDER = ["sql", "python-light", "python-heavy", "scala"] + by_package = {} for name in registered: pkg = _package_for(name) @@ -316,7 +484,22 @@ def build_functions_object(registered: list, doc_examples: dict) -> dict: entry = doc_examples.get(name) or {} examples = (entry.get("examples") or "").strip() if examples: - out[name] = {"examples": examples} + func_entry = {"examples": examples} + # Add parsed builder metadata if available + if name in parsed_builders: + parsed = parsed_builders[name] + if "usage_args" in parsed and parsed["usage_args"]: + func_entry["usage_args"] = parsed["usage_args"] + # Build bindings list in fixed order + present = tier_bindings.get(name, set()) + bindings = [] + if examples: + bindings.append("sql") + for b in _BINDING_ORDER[1:]: + if b in present: + bindings.append(b) + func_entry["bindings"] = bindings + out[name] = func_entry other_names = by_package.get("other", []) if other_names: out["_package_other"] = "--- other ---" @@ -324,12 +507,28 @@ def build_functions_object(registered: list, doc_examples: dict) -> dict: entry = doc_examples.get(name) or {} examples = (entry.get("examples") or "").strip() if examples: - out[name] = {"examples": examples} + func_entry = {"examples": examples} + # Add parsed builder metadata if available + if name in parsed_builders: + parsed = parsed_builders[name] + if "usage_args" in parsed and parsed["usage_args"]: + func_entry["usage_args"] = parsed["usage_args"] + # Build bindings list in fixed order + present = tier_bindings.get(name, set()) + bindings = [] + if examples: + bindings.append("sql") + for b in _BINDING_ORDER[1:]: + if b in present: + bindings.append(b) + func_entry["bindings"] = bindings + out[name] = func_entry return out def main(): import argparse + parser = argparse.ArgumentParser( description="Generate function-info.json from doc SQL examples (no empty usage; fix missing upstream in docs)" ) @@ -338,10 +537,13 @@ def main(): os.makedirs(RESOURCE_DIR, exist_ok=True) registered = load_registered_functions_txt() doc_examples = discover_and_collect(registered) + parsed_builders = _load_parsed_builders() if not registered: # No registered list: output only doc-derived (legacy) - functions = build_functions_object(sorted(doc_examples.keys()), doc_examples) + functions = build_functions_object( + sorted(doc_examples.keys()), doc_examples, parsed_builders + ) with open(RESOURCE_FILE, "w") as f: json.dump({"functions": functions}, f, indent=2) count = len([k for k in functions if not k.startswith("_")]) @@ -349,12 +551,18 @@ def main(): return # Full overwrite from registered; only include entries with non-empty examples. - functions = build_functions_object(registered, doc_examples) + functions = build_functions_object(registered, doc_examples, parsed_builders) included = {k for k in functions if not k.startswith("_")} missing_or_empty = [n for n in registered if n not in included] if missing_or_empty: - print("ERROR: Empty or missing usage is not allowed. Fix upstream: add SQL examples in docs.", file=sys.stderr) - print("Functions missing a doc SQL example (add *_sql_example() in the API function ref):", file=sys.stderr) + print( + "ERROR: Empty or missing usage is not allowed. Fix upstream: add SQL examples in docs.", + file=sys.stderr, + ) + print( + "Functions missing a doc SQL example (add *_sql_example() in the API function ref):", + file=sys.stderr, + ) for name in sorted(missing_or_empty): pkg = _package_for(name) if pkg in ("rasterx", "rasterx_h3"): @@ -368,7 +576,10 @@ def main(): else: path = "docs/tests/python/api/*_functions_sql.py" print(f" {name} -> {path}", file=sys.stderr) - print(f"\nTotal: {len(missing_or_empty)} function(s) need a doc SQL example.", file=sys.stderr) + print( + f"\nTotal: {len(missing_or_empty)} function(s) need a doc SQL example.", + file=sys.stderr, + ) sys.exit(1) with open(RESOURCE_FILE, "w") as f: diff --git a/docs/sidebars.js b/docs/sidebars.js index f9624bfae..22560505c 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -19,7 +19,8 @@ const sidebars = { 'installation', 'quick-start', 'databricks-spatial', - 'beta-release-notes', + 'serverless-and-memory', + 'release-notes', { type: 'category', label: 'Examples', @@ -73,6 +74,7 @@ const sidebars = { collapsed: true, link: { type: 'doc', id: 'readers-writers' }, items: [ + 'common-functions', { type: 'category', label: 'Readers', @@ -80,7 +82,8 @@ const sidebars = { items: [ 'readers/overview', { type: 'category', label: 'General', collapsed: true, items: ['readers/raster', 'readers/vector'] }, - { type: 'category', label: 'Named', collapsed: true, items: ['readers/geotiff', 'readers/netcdf', 'readers/pmtiles', 'readers/shapefile', 'readers/geojson', 'readers/geopackage', 'readers/filegdb'] }, + { type: 'category', label: 'Named', collapsed: true, items: ['readers/geotiff', 'readers/cog', 'readers/netcdf', 'readers/pmtiles', 'readers/shapefile', 'readers/geojson', 'readers/geopackage', 'readers/filegdb'] }, + 'readers/file', ], }, { @@ -90,7 +93,8 @@ const sidebars = { items: [ 'writers/overview', { type: 'category', label: 'General', collapsed: true, items: ['writers/raster', 'writers/vector'] }, - { type: 'category', label: 'Named', collapsed: true, items: ['writers/geotiff', 'writers/netcdf', 'writers/pmtiles', 'writers/shapefile', 'writers/geojson', 'writers/geojsonl', 'writers/geopackage', 'writers/filegdb'] }, + { type: 'category', label: 'Named', collapsed: true, items: ['writers/geotiff', 'writers/cog', 'writers/netcdf', 'writers/pmtiles', 'writers/shapefile', 'writers/geojson', 'writers/geojsonl', 'writers/geopackage', 'writers/filegdb'] }, + 'writers/file', ], }, ], @@ -101,8 +105,9 @@ const sidebars = { collapsed: true, items: [ 'api/overview', - 'api/tile-structure', 'api/execution-tiers', + 'api/coordinate-reference-systems', + 'api/error-handling', 'api/language-bindings', { type: 'category', @@ -110,8 +115,14 @@ const sidebars = { collapsed: true, link: { type: 'doc', id: 'api/raster-functions' }, items: [ + 'api/tile-structure', 'api/rasterio-distributed', + 'api/large-rasters', + 'api/virtual-tiles', + 'api/vrt-mosaic', + 'api/materialized-compression', 'api/h3-raster-tessellation', + 'api/raster-sampling', ], }, { type: 'doc', id: 'api/vectorx-functions', label: 'VectorX' }, diff --git a/docs/src/components/CodeFromTest.js b/docs/src/components/CodeFromTest.js index 8004a9237..8ca40075d 100644 --- a/docs/src/components/CodeFromTest.js +++ b/docs/src/components/CodeFromTest.js @@ -114,11 +114,25 @@ function pythonFunctionToSnippet(fullFunction) { } const withoutReturns = bodyLines.slice(0, end); + // Strip trailing lint-directive comments (noqa / type: ignore / pragma) so + // they don't leak into the rendered example. These are required on the source + // line (e.g. `# noqa: PLC0415` for the function-local imports the single-source + // doc-test pattern uses) but are pure noise to a reader. Only the directive is + // removed; a preceding explanatory comment on the same line is preserved. + // A comment-only directive line (nothing but the directive) is dropped. + const lintDirective = /\s*#\s*(?:noqa\b[^#]*|type:\s*ignore\b[^#]*|pragma:\s*no\s*cover\b[^#]*)$/; + const deLinted = withoutReturns.map(l => { + if (!lintDirective.test(l)) return l; + const stripped = l.replace(lintDirective, ''); + // If only indentation remains (the line was just a directive comment), drop it. + return stripped.trim() === '' && l.trim().startsWith('#') ? null : stripped; + }).filter(l => l !== null); + // Dedent: use minimum non-zero indent - const nonEmpty = withoutReturns.filter(l => l.trim() !== ''); - if (nonEmpty.length === 0) return withoutReturns.join('\n').trim(); + const nonEmpty = deLinted.filter(l => l.trim() !== ''); + if (nonEmpty.length === 0) return deLinted.join('\n').trim(); const minIndent = Math.min(...nonEmpty.map(l => l.match(/^\s*/)[0].length)); - const dedented = withoutReturns.map(l => l.length >= minIndent ? l.slice(minIndent) : l); + const dedented = deLinted.map(l => l.length >= minIndent ? l.slice(minIndent) : l); return dedented.join('\n').trim(); } diff --git a/docs/src/components/FunctionExamples.js b/docs/src/components/FunctionExamples.js new file mode 100644 index 000000000..6a7fe9532 --- /dev/null +++ b/docs/src/components/FunctionExamples.js @@ -0,0 +1,58 @@ +// docs/src/components/FunctionExamples.js +import React from 'react'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeFromTest from '@site/src/components/CodeFromTest'; +import functionInfo from '@site/../src/main/resources/com/databricks/labs/gbx/function-info.json'; + +// Fixed tab order. label = tab text; key = binding label in function-info.json; +// lang = CodeFromTest language; suffix = example-function name suffix. +const TABS = [ + { key: 'sql', label: 'SQL', lang: 'sql', suffix: '_sql_example' }, + { key: 'python-light', label: 'Python (light)', lang: 'python', suffix: '_python_light_example' }, + { key: 'python-heavy', label: 'Python (heavy)', lang: 'python', suffix: '_python_heavy_example' }, + { key: 'scala', label: 'Scala', lang: 'scala', suffix: '_scala_example' }, +]; + +function bindingsFor(name) { + const fns = functionInfo.functions || functionInfo; + const entry = fns[name] || fns['gbx_' + name] || {}; + return new Set(entry.bindings || []); +} + +export default function FunctionExamples(props) { + const { name, testFile } = props; + const present = bindingsFor(name); + const codeByKey = { + 'sql': props.sql, + 'python-light': props.pythonLight, + 'python-heavy': props.pythonHeavy, + 'scala': props.scala, + }; + const sourceByKey = { + 'sql': props.sqlSource, + 'python-light': props.pythonLightSource, + 'python-heavy': props.pythonHeavySource, + 'scala': props.scalaSource, + }; + return ( + + {TABS.map((t) => ( + + {present.has(t.key) && codeByKey[t.key] ? ( + + ) : ( +

Not available in this tier.

+ )} + + ))} + + ); +} diff --git a/docs/src/css/custom.css b/docs/src/css/custom.css index eee8e7d2e..3219e000b 100644 --- a/docs/src/css/custom.css +++ b/docs/src/css/custom.css @@ -83,3 +83,46 @@ border-left-color: #6b8cff; background: rgba(107, 140, 255, 0.08); } + +/* -------------------------------------------------------------------------- + * Per-example language tabs (groupId="gbx-example-lang"): SQL(1) and + * Python-light(2) keep the default green accent. Python-heavy(3) and + * Scala-heavy(4) get the blue heavyweight tint — same tokens as + * .gbx-tier-tabs above. Scoped to .gbx-example-lang-tabs. + * ------------------------------------------------------------------------ */ + +/* Heavy tabs = 3rd + 4th .tabs__item; recolor active underline + label. */ +.gbx-example-lang-tabs.tabs .tabs__item:nth-child(3).tabs__item--active, +.gbx-example-lang-tabs.tabs .tabs__item:nth-child(4).tabs__item--active { + --ifm-tabs-color-active: #2545b3; + --ifm-tabs-color-active-border: #2545b3; +} + +/* Tint the visible heavy content panel while it is selected. */ +.tabs-container:has( + > .gbx-example-lang-tabs .tabs__item:nth-child(3).tabs__item--active + ) + div[role='tabpanel']:not([hidden]), +.tabs-container:has( + > .gbx-example-lang-tabs .tabs__item:nth-child(4).tabs__item--active + ) + div[role='tabpanel']:not([hidden]) { + border-left: 3px solid #2545b3; + background: #f5f8ff; + padding: 0.75rem 1rem; + border-radius: 0 4px 4px 0; +} + +[data-theme='dark'] + .tabs-container:has( + > .gbx-example-lang-tabs .tabs__item:nth-child(3).tabs__item--active + ) + div[role='tabpanel']:not([hidden]), +[data-theme='dark'] + .tabs-container:has( + > .gbx-example-lang-tabs .tabs__item:nth-child(4).tabs__item--active + ) + div[role='tabpanel']:not([hidden]) { + border-left-color: #6b8cff; + background: rgba(107, 140, 255, 0.08); +} diff --git a/docs/src/pages/index.js b/docs/src/pages/index.js index 666fc335c..ac1a77497 100644 --- a/docs/src/pages/index.js +++ b/docs/src/pages/index.js @@ -71,7 +71,7 @@ function HomepageFeatures() { c.charCodeAt(0)); -const protocol = new pmtiles.Protocol(); -maplibregl.addProtocol("pmtiles", protocol.tile); -const archive = new pmtiles.PMTiles(new pmtiles.FileSource( - new File([bin.buffer], "gbx.pmtiles"))); -protocol.add(archive); -// MapLibre now resolves pmtiles://gbx/z/x/y range-requests against -// the in-browser File object — zero HTTP calls. -``` - -The entire archive rides in the HTML blob. `pmtiles.FileSource` handles the spec-compliant -range-request logic in the browser JS engine. No tile server, no remote HTTP connection, -no CORS issue, no port binding. - -### 2. Pin CDN script versions for reproducibility - -```python -_MAPLIBRE_JS = "https://unpkg.com/maplibre-gl@4.7.1/dist/maplibre-gl.js" -_MAPLIBRE_CSS = "https://unpkg.com/maplibre-gl@4.7.1/dist/maplibre-gl.css" -_PMTILES_JS = "https://unpkg.com/pmtiles@3.2.1/dist/pmtiles.js" -``` - -Unpinned CDN references can pull a breaking major when the notebook re-renders months later. -Pin at the minor-version level; update deliberately. - -### 3. Size-guard + static fallback for large archives and GitHub-renderable output - -Base64 encoding inflates the original archive bytes by approximately 33% -(`embed_size = len(data) * 4 / 3`). For a 64 MB archive the HTML blob is ~85 MB, which -overloads `displayHTML`. A size guard caps the interactive path and degrades gracefully: - -```python -embed_mb = (len(data) * 4 / 3) / (1024 * 1024) -if embed_mb > max_embed_mb: # default 64 MB - if _is_raster_type(info["tile_type"]): - return _static_raster_fallback(data, info, **kw) # plot_raster - return _static_vector_fallback(data, info, **kw) # plot_static / contextily -``` - -`max_embed_mb=0` deliberately forces the static path — useful for notebook authors who want -the cell to produce a PNG/matplotlib figure that GitHub can render. `fallback=False` raises -instead of degrading, making the oversized case an explicit error for callers that want to -enforce the interactive path. - -### 4. Type detection from the PMTiles header - -```python -_RASTER_TYPES = frozenset({"png", "jpeg", "webp", "avif"}) - -def _is_raster_type(tile_type: str) -> bool: - return tile_type in _RASTER_TYPES -``` - -`pmtiles_info` reads the archive header's `tile_type` field. The viewer auto-selects -a MapLibre raster or vector layer, and the static fallback chooses between `plot_raster` -(raster tiles decoded via rasterio MemoryFile) and `plot_static` (MVT tiles decoded to -GeoDataFrame via mapbox_vector_tile + shapely, then laid over a contextily basemap). -No user-visible type flag is required. - ---- - -## Applicability matrix - -### (a) Other light-tier functions this applies to - -| Function / module | Applies? | Notes | -|---|---|---| -| Future `gbx.vizx` viewers (additional tile formats) | Yes — same pattern | Any new inline viewer should embed the data bytes directly and size-guard before interactive. | -| `gbx.vizx.plot_cog` | Partial | COG is rendered static-only (decimated rasterio read + contextily basemap); no in-browser FileSource needed because COG tiles are not bundled as a PMTiles archive. Decimated read avoids a full-resolution driver read — that is a correctness/resource pattern, not a compute gain. | -| `gbx.pmtiles.pmtiles_info` | Supporting role | Provides the header metadata (type, bounds, zoom range) that drives both the interactive MapLibre style auto-selection and the static fallback branch. Keeping the inspector as a standalone public function means future viewers do not need to duplicate header-parsing logic. | -| Overture / STAC download notebooks | No | Those are distributed Spark paths; the rendering pattern is orthogonal. | - -### (b) Heavy-tier functions (same + similar) - -| Function / Scala class | Applies? | Notes | -|---|---|---| -| Any Scala expression | N/A | These viewers are driver-side Python functions, not Spark expressions. There is no JVM equivalent of displayHTML / MapLibre HTML rendering. The heavy tier does not have an in-notebook map viewer path. | -| `gbx_pmtiles_agg` (heavy Scala aggregator) | N/A — produces the archive, does not render it | The heavy aggregator writes the PMTiles bytes to a path or returns them; `plot_pmtiles` is then called driver-side to view the result. No architectural change to the aggregator is needed. | - -**Verdict:** Driver-side, light-tier-only pattern. The heavy tier writes PMTiles; the light tier -renders them. The FileSource embedding technique is specific to the browser JS sandbox and has -no heavy-tier counterpart. - ---- - -## Evidence - -**Correctness / portability:** - -- `test_build_html_embeds_base64_and_registers_protocol` asserts the HTML blob contains the - base64 string, `new pmtiles.Protocol`, `addProtocol`, `pmtiles.FileSource`, and `pmtiles://`. - This proves no HTTP tile-server URL is emitted — every tile fetch resolves in-browser against - the embedded File object. -- `test_plot_pmtiles_interactive_routes_through_displayhtml` runs the full code path with a - real PMTiles archive (offline, no network): `displayHTML` is called, `maplibre-gl@4.7.1` and - `pmtiles@3.2.1` appear in the output. No HTTP connection is opened. -- `test_plot_pmtiles_size_guard_uses_raster_fallback` and `test_plot_pmtiles_size_guard_uses_vector_fallback` - confirm the `max_embed_mb` guard triggers and routes to the correct static plotter. -- `test_plot_pmtiles_oversized_without_fallback_raises` confirms `fallback=False` raises with - a clear message rather than silently truncating. - -**Base64 overhead math:** -Base64 encodes 3 bytes as 4 ASCII characters → overhead is exactly 4/3 ≈ 33.3%. -A 48 MB archive embeds as ~64 MB of ASCII; a 96 MB archive would embed as ~128 MB. -The default `max_embed_mb=64` guards against HTML blobs that overload `displayHTML` in -Databricks notebooks (observed limit ~100-150 MB in practice; 64 MB gives headroom). - -**No measured distributed compute speedup:** This is a rendering/portability pattern. -There is no distributed Spark path involved; the rendering is entirely driver-side. -Timing comparisons with a tile server are deliberately omitted — the gain is availability -(works offline, works in static export) not wall-clock speedup. - ---- - -## Canonical code references - -- `python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py` - - `_build_pmtiles_html` — constructs the self-contained HTML with base64 + FileSource - - `plot_pmtiles` — main entry; size-guard + interactive vs static routing - - `_static_raster_fallback`, `_static_vector_fallback` — tile decode + plot for static path - - `_is_raster_type` — header-driven type detection -- `python/geobrix/src/databricks/labs/gbx/vizx/_cog.py` - - `plot_cog` — decimated COG read + contextily basemap (static-only; see applicability note) -- `python/geobrix/src/databricks/labs/gbx/pmtiles/_inspect.py` - - `pmtiles_info` — header reader supplying tile_type, bounds, zoom range to the viewer -- Tests: `python/geobrix/test/vizx/test_pmtiles.py` (offline; all assertions listed above) -- Tests: `python/geobrix/test/pmtiles_light/test_inspect.py` (inspector offline) diff --git a/docs/superpowers/performance/pmtiles-vector-merge.md b/docs/superpowers/performance/pmtiles-vector-merge.md deleted file mode 100644 index 39e6553c9..000000000 --- a/docs/superpowers/performance/pmtiles-vector-merge.md +++ /dev/null @@ -1,157 +0,0 @@ -# PMTiles vector-merge: correctness fix + engineering lessons (2026-06-28) - -Two engineering lessons captured from the `gbx_pmtiles_agg` vector-merge fix. -One is a pattern (vector tile aggregation must merge per tile, not first-wins); -the other is a GDAL gotcha (`GetMemFileBuffer` vs OGR-written `/vsimem/` files). -Neither was a measured speedup — the fix is purely correctness. - ---- - -## Lesson 1 — Vector tile aggregation must MERGE per `(z,x,y)`, not first-wins - -### Problem - -`gbx_pmtiles_agg` silently dropped features in dense vector pipelines. When a -`st_asmvt_pyramid → groupBy → gbx_pmtiles_agg` run produced more than one MVT -blob for the same `(z, x, y)` (i.e. when multiple features land in the same tile -at the same zoom level), only the first blob was kept. All other features were -silently discarded. The resulting PMTiles archive was structurally valid but -incomplete — a correctness bug with no runtime error. - -A secondary heavy-tier bug: the accumulator (`PMTilesAcc`) wrote one directory -entry per `(z, x, y, payload)` tuple, so duplicate tile coordinates produced -two entries for the same tile ID, yielding a malformed PMTiles archive -(spec requires exactly one directory entry per tile ID). - -### Symptom / signature - -- Dense vector data (many features per tile at any zoom level) loads into a - PMTiles viewer with far fewer features than expected. -- No exception is raised. The archive is byte-valid and opens in viewers; only - the feature count reveals the drop. -- Pure raster pipelines (`PNG`/`JPEG`/`WebP` tiles) were unaffected: first-wins - is correct for raster (each tile carries one image). - -### Fix / pattern - -**Both tiers: group payloads by tile ID first, then resolve each group to one blob.** - -- Light tier (`pmtiles/_agg_light.py`): `_assemble_archive` now accumulates all - non-null payloads into `tileid_payloads: dict[int, list[bytes]]`. For vector - (`TileType.MVT`) groups with more than one blob, `_merge_mvt_blobs` decodes each - blob with `mapbox_vector_tile.decode`, unions `features` per layer name, and - re-encodes. For raster groups, first blob wins (unchanged). Single-blob groups - take a fast `return blobs[0]` path — zero decode/encode overhead for the common - sparse case. -- Heavy tier (`pmtiles/PMTiles_Agg.scala`): `eval()` builds a `LinkedHashMap[tileId → - ArrayBuffer[(z,x,y,bytes)]]`, then resolves each group: MVT multi-blob paths call - `mergeMvtPayloads` (decode via `MvtDecoder`, union features per layer, re-encode - by concatenating raw protobuf bytes per-layer — valid because MVT layer fields are - `repeated`, so protobuf concatenation merges them). Raster: first blob only. - -**Merge cost** (light tier, measured): for typical tile payloads (<50 KB, -<100 features), the decode+encode round-trip costs < 0.5 ms per tile. The -single-blob fast path costs ~0 (a list-length check). No `benchmarking.mdx` -update is warranted unless a cluster bench shows measurable regression on -dense-data real workloads. - -### Applicability matrix - -| Scope | Status | -|---|---| -| `gbx_pmtiles_agg` light tier (`_agg_light.py`) | FIXED — merge-per-tile, raster first-wins preserved | -| `gbx_pmtiles_agg` heavy tier (`PMTiles_Agg.scala`) | FIXED — group-by-tileid + MVT merge, directory dedup | -| All other GeoBrix `_agg` functions | AUDITED — no other `_agg` shares the drop-on-collision flaw; all others combine correctly by construction (spatial union, set union, etc.) | -| Raster pipelines (`PNG`/`JPEG`/`WebP`) | Unchanged — first-wins is correct; one raster image per tile | - -**Pattern scope:** any PMTiles aggregator or tile packer that accumulates -`(z, x, y, bytes)` rows must group by tile ID before writing the directory. -The flaw is latent in any "accumulate-then-flush" design that doesn't deduplicate -tile IDs before writing. - -### Evidence - -- 5 new light-tier unit tests (`test_agg_light_core.py`) cover: single MVT blob - pass-through, two-blob merge with feature union, three-blob merge with two - layers, raster first-wins unchanged, empty-group empty-archive. -- 3 new cross-tier parity tests (`test_parity_pmtiles_merge.py`) confirm light - and heavy produce matching feature counts and layer names for multi-feature tiles. -- 3 new Scala unit tests (`PMTiles_AggTest.scala`) cover: multi-feature vector - merge, duplicate-tileid dedup, mixed raster first-wins. -- 2 `MvtDecoderTest.scala` tests confirm the decode path used by `mergeMvtPayloads`. -- 278 vectorx + pmtiles Scala tests green (including `st_asmvt`, `st_asmvt_pyramid`). -- **No measured speedup** — this is a correctness fix. - -### Canonical code refs - -- Light: `python/geobrix/src/databricks/labs/gbx/pmtiles/_agg_light.py` — - `_merge_mvt_blobs`, `_assemble_archive` -- Heavy: `src/main/scala/com/databricks/labs/gbx/pmtiles/PMTiles_Agg.scala` — - `eval()`, `mergeMvtPayloads`; `MvtDecoder.scala`; `MvtWriter.scala` - ---- - -## Lesson 2 — GDAL gotcha: `GetMemFileBuffer` returns NULL for OGR-written `/vsimem/` files - -### Problem - -During development of the heavy MVT merge path, the first attempt used the OGR -MVT creation driver writing to a `/vsimem/` path, then called `gdal.GetMemFileBuffer` -to retrieve the bytes. This silently returned `null`, producing an empty byte array -for every tile — no exception, no GDAL error, just empty output. - -### Root cause - -`gdal.GetMemFileBuffer(path)` only returns the buffer for files that were -**explicitly created** via `gdal.FileFromMemBuffer(path, bytes)`. It does NOT work -for files that an OGR or GDAL driver created by writing to a `/vsimem/` path -(e.g. an OGR `CreateDataSource("/vsimem/foo")` output). The GDAL Java bindings -have no mechanism to retrieve driver-written vsimem bytes via `GetMemFileBuffer`; -the function returns null for these paths. - -This is a GDAL SWIG binding limitation, not a file system limitation. The -`/vsimem/` virtual file system itself is fine; the bytes are there. The Java -binding's `GetMemFileBuffer` simply doesn't enumerate or read from paths it -didn't register via `FileFromMemBuffer`. - -### Fix / pattern - -**Use a real temp directory for OGR driver output, then read with standard Java file I/O.** - -```scala -// Create a temp PARENT dir; let OGR create the directory structure inside it. -val tmpParent: Path = Files.createTempDirectory("gbx_mvt_par_") -val tmpRoot: Path = tmpParent.resolve("tile") -val ds = driver.CreateDataSource(tmpRoot.toAbsolutePath.toString, createOpts) -// ... write features, SyncToDisk, ds.delete() ... -// Read the emitted .pbf with standard Java I/O (not GetMemFileBuffer): -val pbfFile = Paths.get(rootPath, "0", "0", "0.pbf") -val bytes = if (Files.exists(pbfFile)) Files.readAllBytes(pbfFile) else Array.emptyByteArray -// Clean up with Files.walkFileTree. -``` - -For **reading** back a known byte array (the inverse direction — e.g. feeding bytes -into an OGR driver for decode), `FileFromMemBuffer` + `driver.Open("/vsimem/path")` -works correctly. The gotcha only applies to the **creation** direction (driver writes -to vsimem → caller tries to read back with `GetMemFileBuffer`). - -### Where this applies - -| Use case | Safe approach | -|---|---| -| Read MVT bytes into OGR (decode) | `FileFromMemBuffer` + `driver.Open(vsimemPath)` — works; see `MvtDecoder.scala` | -| Write MVT bytes from OGR (encode) | Real temp dir + `Files.readAllBytes(pbfFile)` — see `MvtWriter.scala` | -| Any OGR driver writing to `/vsimem/` | Same: temp dir, read with Java I/O, clean up with `walkFileTree` | -| GDAL raster drivers writing to `/vsimem/` | Same: temp dir or use GDAL's `/vsimem/` with `ReadDirRecursive` + per-file read | - -The `MvtWriter.scala` scaladoc describes this explicitly: -> "Intermediate state lives in a Java temp directory (not `/vsimem/`) because -> `gdal.GetMemFileBuffer` only works for `FileFromMemBuffer`-created files — -> it returns null for driver-written vsimem files, silently dropping the output." - -### Canonical code refs - -- `src/main/scala/com/databricks/labs/gbx/vectorx/mvt/MvtWriter.scala` — temp - dir pattern, `Files.readAllBytes`, `walkFileTree` cleanup -- `src/main/scala/com/databricks/labs/gbx/vectorx/mvt/MvtDecoder.scala` — - counter-example: `FileFromMemBuffer` + `driver.Open` for the read direction diff --git a/docs/superpowers/performance/serverless-aoi-ingestion-strategy.md b/docs/superpowers/performance/serverless-aoi-ingestion-strategy.md deleted file mode 100644 index c8a646dc6..000000000 --- a/docs/superpowers/performance/serverless-aoi-ingestion-strategy.md +++ /dev/null @@ -1,148 +0,0 @@ -# Pattern: Serverless-first AOI ingestion strategy - -**Status:** Pattern/correctness confirmed; cluster-scale speedup NOT yet measured (deferred to -optional cluster smoke; see Evidence section). - ---- - -## Problem - -A downloader/reader that ingests geographic data for an area of interest (AOI) has two naive -choices: - -1. **Whole-file download on the driver** — pull entire continental/regional files, then filter - client-side. Bottleneck: all bytes move through the driver; no parallelism; blocks on - network I/O; scales linearly with source file size, not with AOI size. -2. **Number-only repartition fan-out** — `df.repartition(N)` per asset, then write. On - Serverless, AQE coalesces a round-robin repartition toward 1 (serial), defeating the intent. - -Both patterns fail at scale on Databricks Serverless: the first is serial by design; the second -silently degrades to serial at runtime. - ---- - -## Symptom / signature - -- A handful of large cloud parquet files cover the AOI; only a small fraction of rows are - actually within the bbox. -- A whole-file download of those files is gigabytes; the AOI subset is megabytes. -- `repartition(N)` (number-only) appears in the plan but write-task count stays at 1 in the - Spark UI. -- Worker network activity is absent (all I/O on the driver) or uniformly flat (serial tasks). - ---- - -## The fix / pattern - -**Serverless-first AOI ingestion** — three complementary moves: - -### 1. Distributed read-in-place with bbox-struct predicate pushdown - -When cloud parquet paths are available (object-store schemes or FUSE-mounted Volumes): - -```python -df = spark.read.parquet(cloud_href) -if "bbox" in df.columns: # Overture schema: nested bbox struct - df = df.filter( - (F.col("bbox.xmin") <= F.lit(maxx)) - & (F.col("bbox.xmax") >= F.lit(minx)) - & (F.col("bbox.ymin") <= F.lit(maxy)) - & (F.col("bbox.ymax") >= F.lit(miny)) - ) -# AOI rows only move to executors; full files never land on the driver. -``` - -The Spark parquet reader + predicate pushdown means only row-groups whose bbox overlaps the -AOI are read. Bytes move on workers in parallel; the driver handles only metadata. - -### 2. Column-hash repartition (NOT number-only) - -```python -key = "id" if "id" in df.columns else df.columns[0] -df.repartition(partitions, F.col(key)).write.mode("overwrite").parquet(target) -``` - -On Serverless, `repartition(N)` (round-robin) is AQE-coalesced toward 1. Hashing by a -real column forces the shuffle to respect N partitions; AQE cannot coalesce a hash -repartition. See `[[serverless-fanout-repartition-by-column]]` for the full rule. - -### 3. Cloud-scheme routing with HTTP-href fallback - -Route to the distributed path when hrefs start with a cloud object-store scheme -(`s3://`, `abfs://`, `gs://`, etc.) or `/` (FUSE Volume). Fall back to whole-file HTTP -download (fanned out per-href with column-hash repartition) when only `https://` is -available: - -```python -is_cloud = all( - h.startswith(cloud_scheme) or h.startswith("/") for h in hrefs -) -if is_cloud: - _download_distributed(assets_df, ...) # read-in-place -else: - _download_fallback(assets_df, ...) # whole-file HTTP, still column-hash fanned -``` - -The fallback still uses `repartition(N, F.col("href"))` — serial driver download is NOT -the fallback; parallel per-href download is. - ---- - -## Applicability matrix - -### (a) Other light-tier functions this applies to - -| Function / module | Applies? | Notes | -|---|---|---| -| `gbx.stac.StacClient.download` | Yes — same pattern | Uses per-href fan-out with column-hash repartition; add distributed read-in-place for cloud hrefs in a future pass | -| `gbx.sample._bundle` / `GbxBundle` | Partial | Per-file download; add column-hash repartition if not already present | -| Future `sample/` sources (NAIP, 3DEP, etc.) | Yes | Adopt the same cloud-scheme router: distributed-read when cloud path available, HTTP fallback otherwise | -| `pyrx` DataSource V2 raster readers (`raster_gbx`, `gtiff_gbx`) | No — different shape | Reads are already distributed per-tile by the DataSource V2 scan; AOI is applied via partition pruning, not a bbox struct filter | -| Light vector readers (`ogr_pyvx`, etc.) | Partial | pyogrio reads are per-file on executors; add bbox pushdown where the format supports spatial filter (e.g. GeoPackage rtree) | - -### (b) Heavy-tier functions (same + similar) - -| Function / Scala class | Applies? | Notes | -|---|---|---| -| Overture path (heavy) | N/A — no heavy Overture path | `OvertureClient` is light-only; no Scala expression reads Overture GeoParquet | -| Heavy OGR/GDAL readers | N/A — different arch | Heavy reads go through JVM OGR/GDAL DataSourceV2; spatial filtering is OGR `SetSpatialFilter` at the driver or executor level, not Spark predicate pushdown | -| Heavy `rst_fromfile` / raster readers | N/A | GDAL reads are per-tile on the JVM; predicate pushdown is not the bottleneck | - -**Verdict:** This is a light-tier downloader pattern. The heavy tier reads via OGR/GDAL or -Spark DataSource V2 with JVM-level spatial filtering; the Spark bbox-struct predicate pushdown -technique is specific to GeoParquet sources with a nested `bbox` struct column (Overture schema). - ---- - -## Evidence - -**Pattern/correctness:** The distributed-read path and bbox-struct filter are structurally -correct — the Spark predicate pushdown on a nested struct column (`bbox.xmin`, etc.) is -standard Parquet predicate pushdown, confirmed in the Spark 4.0 Parquet reader. Column-hash -repartition correctness was empirically confirmed 2026-06-22 (see -`[[serverless-fanout-repartition-by-column]]`). - -**Cluster-scale speedup: NOT YET MEASURED.** SP1 has only offline unit tests (injected catalog -+ fetcher; no real network). The distributed-read advantage (AOI bytes vs full-file bytes) is -a cluster-scale property. The open item from the design is: confirm Serverless can read -`s3://overturemaps-us-west-2/...` / `abfs://...` directly (requester-pays / credential config); -if blocked, the HTTP-href fallback becomes the primary path. A cluster smoke is deferred but -will produce concrete row-count and timing numbers for this entry. - -**When the cluster smoke runs**, add numbers here: -- Full-file size (per theme/type asset) vs AOI-subset bytes written -- Wall time: distributed-read path vs hypothetical whole-file driver download -- Task count in Spark UI: confirm N > 1 with column-hash repartition - ---- - -## Canonical code references - -- `python/geobrix/src/databricks/labs/gbx/sample/overture.py` - - `OvertureClient._download_distributed` — distributed read + bbox filter + column-hash write - - `OvertureClient._download_fallback` — HTTP fan-out with column-hash repartition by `href` - - `OvertureClient.download` — cloud-scheme router (distributed vs fallback) - - `OvertureClient.read` — re-reads downloaded parquet with optional bbox re-filter -- `python/geobrix/src/databricks/labs/gbx/sample/_overture_discover.py` - - `normalize_bbox`, `bbox_intersects` — shared bbox helpers used in pushdown -- Tests: `python/geobrix/test/sample/test_overture.py` (offline, injected seams) diff --git a/docs/superpowers/plans/2026-05-28-rst-dtmfromgeoms-wireup.md b/docs/superpowers/plans/2026-05-28-rst-dtmfromgeoms-wireup.md deleted file mode 100644 index ac13994eb..000000000 --- a/docs/superpowers/plans/2026-05-28-rst-dtmfromgeoms-wireup.md +++ /dev/null @@ -1,1280 +0,0 @@ -# gbx_rst_dtmfromgeoms (+ _agg) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Wire up, fix, and test the ported `gbx_rst_dtmfromgeoms` (Delaunay-TIN DTM from Z-valued points + breaklines) and ship its streaming aggregator `gbx_rst_dtmfromgeoms_agg`. - -**Architecture:** A pure `RST_DTMFromGeoms.execute(points, breaklines, …)` compute path (triangulate → interpolate Z at bbox grid cell-centers → direct-fill Float64 GTiff) is shared by the non-agg expression and the `TypedImperativeAggregate` aggregator. The non-agg parses array inputs; the aggregator streams point geometries into a serializable buffer and reads breaklines/extent as per-group constants. Mirrors the existing `RST_GridFromPoints` / `RST_GridFromPointsAgg` pairing exactly. - -**Tech Stack:** Scala 2.13 / Spark 4.0 Catalyst expressions, JTS (`ConformingDelaunayTriangulationBuilder`), GDAL Java bindings, PySpark `call_function` bindings. All build/test runs happen inside the `geobrix-dev` Docker container via `gbx:*` commands. - -**Spec:** `docs/superpowers/specs/2026-05-28-rst-dtmfromgeoms-wireup-design.md` - -**Conventions reminder:** -- Run Scala/Python/doc tests via `gbx:*` commands inside Docker (never `mvn`/`pytest` on the host). -- Long-running suites (`gbx:test:scala`, builds) should be dispatched as background work. -- After any change to Scala source, the assembly JAR is stale — `gbx:test:python` will warn; rebuild with `gbx:docker:exec "mvn clean package -PskipScoverage -DskipTests"` before the Python/doc tests. -- `gh auth switch --user mjohns-databricks` before any push. - ---- - -## File Structure - -| File | Responsibility | -|---|---| -| `src/main/scala/com/databricks/labs/gbx/rasterx/operations/InterpolateElevation.scala` | TIN math: triangulation, Z-interpolation, **bbox-based** grid generation. NaN/out-of-hull cells skipped (not thrown). | -| `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeoms.scala` | Non-agg expression: modern bbox+pixels signature, Int+Long eval, correct `safeEval`, builder; **owns the shared pure `execute`** + `tileRow`. | -| `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeomsAgg.scala` | **New.** `TypedImperativeAggregate` streaming points; breaklines/extent are per-group constants; delegates to `RST_DTMFromGeoms.execute`. | -| `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/DTMFromGeomsAcc.scala` | **New.** Serializable point-WKB accumulation buffer for the aggregator. | -| `src/main/scala/com/databricks/labs/gbx/rasterx/functions.scala` | Register both functions. | -| `pom.xml` | Remove the two scoverage `excludedFiles` entries. | -| `docs/tests-function-info/registered_functions.txt` | Add both canonical names. | -| `docs/tests/python/api/rasterx_functions_sql.py` | A `*_sql_example()` + `_output` for each. | -| `src/main/resources/com/databricks/labs/gbx/function-info.json` | Regenerated. | -| `python/geobrix/src/databricks/labs/gbx/rasterx/functions.py` | `rst_dtmfromgeoms` + `rst_dtmfromgeoms_agg` wrappers. | -| `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeomsTest.scala` | **New.** Known-plane, breakline, out-of-hull, validation, agg≡non-agg, buffer roundtrip. | -| `python/geobrix/test/rasterx/test_dtmfromgeoms.py` | **New.** Python binding smoke tests for both. | -| `docs/tests/python/api/` SQL doc test wiring | New SQL doc examples execute under Docker. | - ---- - -## Task 1: bbox grid + non-throwing interpolation in `InterpolateElevation` - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/operations/InterpolateElevation.scala` -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/operations/InterpolateElevationTest.scala` (create) - -- [ ] **Step 1: Write the failing test** - -Create `src/test/scala/com/databricks/labs/gbx/rasterx/operations/InterpolateElevationTest.scala`: - -```scala -package com.databricks.labs.gbx.rasterx.operations - -import com.databricks.labs.gbx.vectorx.jts.JTS -import org.locationtech.jts.geom.{Coordinate, GeometryFactory, LineString} -import org.scalatest.funsuite.AnyFunSuite -import org.scalatest.matchers.should.Matchers._ - -class InterpolateElevationTest extends AnyFunSuite { - - private val gf = new GeometryFactory() - - /** z = 2*x + 3*y + 5 sampled at the 4 corners of a 100x100 extent. */ - private def planePoints() = Seq( - JTS.point(new Coordinate(0.0, 0.0, 2 * 0.0 + 3 * 0.0 + 5)), - JTS.point(new Coordinate(100.0, 0.0, 2 * 100.0 + 3 * 0.0 + 5)), - JTS.point(new Coordinate(0.0, 100.0, 2 * 0.0 + 3 * 100.0 + 5)), - JTS.point(new Coordinate(100.0, 100.0, 2 * 100.0 + 3 * 100.0 + 5)) - ) - - test("pointGridBBox emits widthPx*heightPx cell centers inside the extent") { - val grid = InterpolateElevation.pointGridBBox(0.0, 0.0, 100.0, 100.0, 10, 10, 32633) - grid.getNumGeometries shouldBe 100 - // first cell center is at (xmin + xRes/2, ymin + yRes/2) = (5, 5) - val p0 = grid.getGeometryN(0) - p0.getCoordinate.x shouldBe 5.0 +- 1e-9 - p0.getCoordinate.y shouldBe 5.0 +- 1e-9 - } - - test("interpolate reproduces a planar surface exactly (linear TIN)") { - val mp = JTS.multiPoint(planePoints().toArray) - val grid = InterpolateElevation.pointGridBBox(0.0, 0.0, 100.0, 100.0, 10, 10, 32633) - val out = InterpolateElevation.interpolate(mp, Seq.empty[LineString], grid, 0.0, 0.0) - out should not be empty - out.foreach { p => - val expected = 2 * p.getX + 3 * p.getY + 5 - p.getCoordinate.getZ shouldBe expected +- 1e-6 - } - } - - test("interpolate skips (does not throw on) points outside the convex hull") { - val mp = JTS.multiPoint(planePoints().toArray) - // Grid extends well beyond the 100x100 point hull; outer cells have no triangle. - val grid = InterpolateElevation.pointGridBBox(-50.0, -50.0, 150.0, 150.0, 20, 20, 32633) - noException should be thrownBy { - InterpolateElevation.interpolate(mp, Seq.empty[LineString], grid, 0.0, 0.0) - } - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Dispatch (background, Docker): -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.operations.InterpolateElevationTest' --log dtm-interp.log -``` -Expected: FAIL — `pointGridBBox` does not exist (compile error) / current `interpolate` throws on NaN. - -- [ ] **Step 3: Add `pointGridBBox` and make `interpolate` skip NaN** - -In `InterpolateElevation.scala`, add the bbox grid method (keep the existing `pointGrid` for now or remove it — it is only used by the old `eval`, which Task 3 rewrites; remove it in Task 3): - -```scala - /** Regular grid of cell-center points over a bbox, row-major by column then row. - * Cell size is derived: xRes = (xmax-xmin)/widthPx, yRes = (ymax-ymin)/heightPx. - * Centers: x = xmin + (i + 0.5)*xRes, y = ymin + (j + 0.5)*yRes. - */ - def pointGridBBox( - xmin: Double, ymin: Double, xmax: Double, ymax: Double, - widthPx: Int, heightPx: Int, srid: Int - ): MultiPoint = { - val xRes = (xmax - xmin) / widthPx - val yRes = (ymax - ymin) / heightPx - val pts = for (i <- 0 until widthPx; j <- 0 until heightPx) yield { - val x = xmin + (i + 0.5) * xRes - val y = ymin + (j + 0.5) * yRes - val p = JTS.point(new Coordinate(x, y)) - p.setSRID(srid) - p - } - JTS.multiPoint(pts.toArray) - } -``` - -Change the tail of `interpolate` from a throwing `.map` to a skipping `.flatMap`: - -```scala - .flatMap({ case (point: Point, poly: Polygon) => - val polyCoords = poly.getCoordinates - val tri = new Triangle(polyCoords(0), polyCoords(1), polyCoords(2)) - val z = tri.interpolateZ(point.getCoordinate) - if (z.isNaN) { - None // cell with degenerate triangle -> caller treats as no_data - } else { - val ip = JTS.point(new Coordinate(point.getX, point.getY, z)) - ip.setSRID(multipoint.getSRID) - Some(ip) - } - }) - .toSeq -``` - -(Replace the existing `.map({ case (point, poly) => … }).toSeq` block; the `if (z.isNaN) { throw … }` line is removed.) - -- [ ] **Step 4: Run test to verify it passes** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.operations.InterpolateElevationTest' --log dtm-interp.log -``` -Expected: PASS (3 tests). - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/operations/InterpolateElevation.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/operations/InterpolateElevationTest.scala -git commit -m "feat(rasterx): bbox grid + non-throwing interpolation in InterpolateElevation" -``` - ---- - -## Task 2: Shared `RST_DTMFromGeoms.execute` (direct-fill rasterize) - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeoms.scala` -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeomsTest.scala` (create) - -This task adds the pure `execute` + `tileRow` to the companion object. The expression-class rework (signature, eval entry points) is Task 3 — keep this task focused on the compute path so it can be tested in isolation by direct call. - -- [ ] **Step 1: Write the failing test** - -Create `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeomsTest.scala`: - -```scala -package com.databricks.labs.gbx.rasterx.expressions - -import com.databricks.labs.gbx.rasterx.gdal.GDALManager -import com.databricks.labs.gbx.vectorx.jts.JTS -import org.apache.spark.sql.catalyst.InternalRow -import org.gdal.gdal.gdal -import org.locationtech.jts.geom.{Coordinate, Geometry, LineString} -import org.scalatest.BeforeAndAfterAll -import org.scalatest.funsuite.AnyFunSuite -import org.scalatest.matchers.should.Matchers._ - -import java.nio.file.Files - -class RST_DTMFromGeomsTest extends AnyFunSuite with BeforeAndAfterAll { - - override def beforeAll(): Unit = { - GDALManager.loadSharedObjects(Iterable.empty[String]) - GDALManager.configureGDAL("/tmp", "/tmp", logCPL = true, CPL_DEBUG = "OFF") - gdal.AllRegister() - import com.databricks.labs.gbx.util.NodeFilePathUtil - Files.createDirectories(NodeFilePathUtil.rootPath) - } - - /** z = 2*x + 3*y + 5 sampled at the 4 corners of a 100x100 extent (EPSG:32633). */ - private def planePoints(): Seq[Geometry] = Seq( - JTS.point(new Coordinate(0.0, 0.0, 5.0)), - JTS.point(new Coordinate(100.0, 0.0, 205.0)), - JTS.point(new Coordinate(0.0, 100.0, 305.0)), - JTS.point(new Coordinate(100.0, 100.0, 505.0)) - ) - - /** Read a single pixel value (col,row) from the GTiff bytes in a tile row. */ - private def pixel(row: InternalRow, col: Int, r: Int): Double = { - val bytes = row.getBinary(1) - bytes should not be null - val tmp = s"/vsimem/dtm_readback_${java.util.UUID.randomUUID().toString.replace("-", "")}.tif" - gdal.FileFromMemBuffer(tmp, bytes) - val ds = gdal.Open(tmp) - try { - val buf = new Array[Double](1) - ds.GetRasterBand(1).ReadRaster(col, r, 1, 1, buf) - buf(0) - } finally { ds.delete(); gdal.Unlink(tmp) } - } - - test("execute reproduces the planar surface at cell centers") { - val row = RST_DTMFromGeoms.execute( - planePoints(), Seq.empty[LineString], - mergeTolerance = 0.0, snapTolerance = 0.0, - xmin = 0.0, ymin = 0.0, xmax = 100.0, ymax = 100.0, - widthPx = 10, heightPx = 10, srid = 32633, noData = -9999.0 - ) - row should not be null - // Pixel (col=0,row=0) is the top-left cell. Its center is x=5, y=95 (row 0 = max y). - // Expected z = 2*5 + 3*95 + 5 = 300. - pixel(row, 0, 0) shouldBe 300.0 +- 1e-3 - // Pixel (col=9,row=9): center x=95, y=5 -> z = 2*95 + 3*5 + 5 = 210. - pixel(row, 9, 9) shouldBe 210.0 +- 1e-3 - } - - test("execute writes no_data for cells outside the point hull") { - val row = RST_DTMFromGeoms.execute( - planePoints(), Seq.empty[LineString], - 0.0, 0.0, - xmin = -100.0, ymin = -100.0, xmax = 200.0, ymax = 200.0, - widthPx = 30, heightPx = 30, srid = 32633, noData = -9999.0 - ) - // top-left corner cell center (~ -95, 195) is far outside the 0..100 hull. - pixel(row, 0, 0) shouldBe -9999.0 +- 1e-6 - } - - test("execute honors a breakline without throwing") { - val bl = JTS.fromWKT("LINESTRING (0 50, 100 50)").asInstanceOf[LineString] - noException should be thrownBy { - RST_DTMFromGeoms.execute( - planePoints(), Seq(bl), 0.0, 0.01, - 0.0, 0.0, 100.0, 100.0, 10, 10, 32633, -9999.0) - } - } - - test("execute rejects degenerate extents and non-positive dims") { - an[IllegalArgumentException] should be thrownBy { - RST_DTMFromGeoms.execute(planePoints(), Seq.empty, 0.0, 0.0, 0.0, 0.0, 0.0, 100.0, 10, 10, 32633, -9999.0) - } - an[IllegalArgumentException] should be thrownBy { - RST_DTMFromGeoms.execute(planePoints(), Seq.empty, 0.0, 0.0, 0.0, 0.0, 100.0, 100.0, 0, 10, 32633, -9999.0) - } - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_DTMFromGeomsTest' --log dtm-exec.log -``` -Expected: FAIL — `RST_DTMFromGeoms.execute` does not exist (compile error). - -- [ ] **Step 3: Add `execute` + `tileRow` to the companion** - -In `RST_DTMFromGeoms.scala`, add these imports if missing: - -```scala -import com.databricks.labs.gbx.rasterx.util.VectorRasterBridge -import com.databricks.labs.gbx.util.SerializationUtil -import org.locationtech.jts.geom.Geometry -``` - -Add to `object RST_DTMFromGeoms`: - -```scala - /** Pure compute path shared by the non-agg expression and the aggregator. - * Builds a constrained-Delaunay TIN from `points` (+ optional `breaklines`), - * interpolates Z at the bbox cell centers, and writes a single-band Float64 - * GTiff tile. Cells outside the triangulated hull are `noData`. - */ - def execute( - points: Seq[Geometry], - breaklines: Seq[LineString], - mergeTolerance: Double, - snapTolerance: Double, - xmin: Double, ymin: Double, xmax: Double, ymax: Double, - widthPx: Int, heightPx: Int, srid: Int, - noData: Double - ): InternalRow = { - require(widthPx > 0, s"rst_dtmfromgeoms: width_px must be positive; got $widthPx") - require(heightPx > 0, s"rst_dtmfromgeoms: height_px must be positive; got $heightPx") - require(xmax > xmin, s"rst_dtmfromgeoms: xmax ($xmax) must be > xmin ($xmin)") - require(ymax > ymin, s"rst_dtmfromgeoms: ymax ($ymax) must be > ymin ($ymin)") - require(points.nonEmpty, "rst_dtmfromgeoms: at least one point is required") - - val mp = JTS.multiPoint(points.toArray) - mp.setSRID(srid) - val grid = InterpolateElevation.pointGridBBox(xmin, ymin, xmax, ymax, widthPx, heightPx, srid) - val interpolated = InterpolateElevation.interpolate(mp, breaklines, grid, mergeTolerance, snapTolerance) - - val ds = VectorRasterBridge.buildEmptyRaster(xmin, ymin, xmax, ymax, widthPx, heightPx, srid, noData) - try { - val xRes = (xmax - xmin) / widthPx - val yRes = (ymax - ymin) / heightPx - val arr = Array.fill[Double](widthPx * heightPx)(noData) - interpolated.foreach { p => - val col = math.floor((p.getX - xmin) / xRes).toInt - val r = math.floor((ymax - p.getY) / yRes).toInt - if (col >= 0 && col < widthPx && r >= 0 && r < heightPx) { - arr(r * widthPx + col) = p.getCoordinate.getZ - } - } - ds.GetRasterBand(1).WriteRaster(0, 0, widthPx, heightPx, arr) - ds.FlushCache() - tileRow(VectorRasterBridge.toGTiffBytes(ds)) - } finally { - ds.delete() - } - } - - /** Build the (index_id, raster, metadata) tile row downstream serializers expect. */ - def tileRow(bytes: Array[Byte]): InternalRow = { - val mtd = Map( - "driver" -> "GTiff", - "extension" -> "tif", - "size" -> bytes.length.toString, - "parentPath" -> "", - "all_parents" -> "", - "last_command" -> "gbx_rst_dtmfromgeoms" - ) - InternalRow.fromSeq(Seq(0L, bytes, SerializationUtil.toMapData[String, String](mtd))) - } -``` - -- [ ] **Step 4: Run test to verify it passes** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_DTMFromGeomsTest' --log dtm-exec.log -``` -Expected: PASS (4 tests). - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeoms.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeomsTest.scala -git commit -m "feat(rasterx): shared RST_DTMFromGeoms.execute with direct-fill rasterize" -``` - ---- - -## Task 3: Rework the `RST_DTMFromGeoms` expression (signature, eval, builder) - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeoms.scala` -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/operations/InterpolateElevation.scala` (remove now-dead old `pointGrid`) -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeomsTest.scala` (extend) - -- [ ] **Step 1: Write the failing test** (append to `RST_DTMFromGeomsTest.scala`) - -```scala - test("builder accepts 11 args (no_data defaulted) and 12 args") { - val lit = (v: Any) => org.apache.spark.sql.catalyst.expressions.Literal(v) - val base = Seq[org.apache.spark.sql.catalyst.expressions.Expression]( - lit(null), lit(null), lit(0.0), lit(0.0), - lit(0.0), lit(0.0), lit(100.0), lit(100.0), - lit(10), lit(10), lit(32633) - ) - // 11 args -> no_data defaulted, builds without error. - RST_DTMFromGeoms.builder()(base) shouldBe a[RST_DTMFromGeoms] - // 12 args -> explicit no_data. - RST_DTMFromGeoms.builder()(base :+ lit(-1.0)) shouldBe a[RST_DTMFromGeoms] - // wrong arity -> error. - an[IllegalArgumentException] should be thrownBy { RST_DTMFromGeoms.builder()(base.take(5)) } - } -``` - -- [ ] **Step 2: Run test to verify it fails** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_DTMFromGeomsTest' --log dtm-exec.log -``` -Expected: FAIL — current builder takes the old 11-positional shape and there is no `no_data` default. - -- [ ] **Step 3: Replace the case class and companion `eval`/`builder`** - -Replace the whole `case class RST_DTMFromGeoms(...)` and the `eval`/`builder`/`name` parts of the companion with the modern form (keep the `execute`/`tileRow` from Task 2). Use `RST_GridFromPoints` as the structural template. - -Case class: - -```scala -case class RST_DTMFromGeoms( - pointsArray: Expression, - breaklinesArray: Expression, - mergeTolerance: Expression, - snapTolerance: Expression, - xminExpr: Expression, - yminExpr: Expression, - xmaxExpr: Expression, - ymaxExpr: Expression, - widthPxExpr: Expression, - heightPxExpr: Expression, - sridExpr: Expression, - noDataExpr: Expression -) extends InvokedExpression { - - override def children: Seq[Expression] = Seq( - pointsArray, breaklinesArray, mergeTolerance, snapTolerance, - xminExpr, yminExpr, xmaxExpr, ymaxExpr, - widthPxExpr, heightPxExpr, sridExpr, noDataExpr, - ExpressionConfigExpr() - ) - override def dataType: DataType = RST_ExpressionUtil.tileDataType(BinaryType) - override def nullable: Boolean = true - override def prettyName: String = RST_DTMFromGeoms.name - override def replacement: Expression = invoke(RST_DTMFromGeoms) - override protected def withNewChildrenInternal(nc: IndexedSeq[Expression]): Expression = - copy(nc(0), nc(1), nc(2), nc(3), nc(4), nc(5), nc(6), nc(7), nc(8), nc(9), nc(10), nc(11)) -} -``` - -Companion `eval` (two arity-on-int entry points) + `doInvoke` + `builder` + `name`: - -```scala - import org.apache.spark.sql.catalyst.expressions.Literal - - /** Default no-data sentinel (matches RST_GridFromPoints). */ - val DefaultNoData: Double = -9999.0 - - // Int-args entry (Catalyst / SQL literals). - def eval( - pointsArray: ArrayData, breaklinesArray: ArrayData, - mergeTolerance: Double, snapTolerance: Double, - xmin: Double, ymin: Double, xmax: Double, ymax: Double, - widthPx: Int, heightPx: Int, srid: Int, noData: Double, - conf: UTF8String - ): InternalRow = doInvoke( - pointsArray, breaklinesArray, mergeTolerance, snapTolerance, - xmin, ymin, xmax, ymax, widthPx, heightPx, srid, noData, conf) - - // Long-args entry (PySpark passes Python ints as Long). - def eval( - pointsArray: ArrayData, breaklinesArray: ArrayData, - mergeTolerance: Double, snapTolerance: Double, - xmin: Double, ymin: Double, xmax: Double, ymax: Double, - widthPx: Long, heightPx: Long, srid: Long, noData: Double, - conf: UTF8String - ): InternalRow = doInvoke( - pointsArray, breaklinesArray, mergeTolerance, snapTolerance, - xmin, ymin, xmax, ymax, widthPx.toInt, heightPx.toInt, srid.toInt, noData, conf) - - private def doInvoke( - pointsArray: ArrayData, breaklinesArray: ArrayData, - mergeTolerance: Double, snapTolerance: Double, - xmin: Double, ymin: Double, xmax: Double, ymax: Double, - widthPx: Int, heightPx: Int, srid: Int, noData: Double, - conf: UTF8String - ): InternalRow = - Option( - RST_ErrorHandler.safeEval( - () => { - val exprConf = ExpressionConfig.fromB64(conf.toString) - RST_ExpressionUtil.init(exprConf) - if (pointsArray == null) return null - val pts = JTS.fromArrayData(pointsArray, pointsArray.getClass; ???) - null // replaced below - }, - null, BinaryType, conf - ) - ).map(_.asInstanceOf[InternalRow]).orNull -``` - -> NOTE: decoding `ArrayData` needs the element `DataType`, which the expression knows from -> `pointsArray.dataType` / `breaklinesArray.dataType` (the case-class fields), not the companion. -> So decode in the **case class** (where the field types are available) and pass decoded -> sequences down, OR pass the element types into `doInvoke`. Use the latter to keep `execute` -> reuse clean. Concretely, change the case class to compute element types and the companion -> `eval` to receive them is awkward; instead decode in `doInvoke` using `JTS.fromArrayData` -> with the element type carried via the array's own struct. The original code used -> `JTS.fromArrayData(pointsArray, pdt)` where `pdt` came from the expression. Mirror that by -> having the **expression** override `eval`-routing through `invoke` with the element types -> appended — but simpler: decode using the WKB/WKT element inspection helper below, which needs -> no external DataType. - -Replace the `doInvoke` body's decoding with a self-describing decoder (no external DataType needed), mirroring `RST_GridFromPoints.geomsFromArrayData`: - -```scala - private def doInvoke( - pointsArray: ArrayData, breaklinesArray: ArrayData, - mergeTolerance: Double, snapTolerance: Double, - xmin: Double, ymin: Double, xmax: Double, ymax: Double, - widthPx: Int, heightPx: Int, srid: Int, noData: Double, - conf: UTF8String - ): InternalRow = - Option( - RST_ErrorHandler.safeEval( - () => { - val exprConf = ExpressionConfig.fromB64(conf.toString) - RST_ExpressionUtil.init(exprConf) - if (pointsArray == null) return null - val pts = geomsFromArrayData(pointsArray).toSeq - val lines = (if (breaklinesArray == null) Seq.empty[Geometry] - else geomsFromArrayData(breaklinesArray).toSeq) - .map(_.asInstanceOf[LineString]) - execute(pts, lines, mergeTolerance, snapTolerance, - xmin, ymin, xmax, ymax, widthPx, heightPx, srid, noData) - }, - null, BinaryType, conf - ) - ).map(_.asInstanceOf[InternalRow]).orNull - - /** Decode an ARRAY of point/line geometries; element may be BINARY (WKB) or STRING (WKT). */ - private def geomsFromArrayData(data: ArrayData): Array[Geometry] = { - val n = data.numElements() - val out = new Array[Geometry](n) - var i = 0 - while (i < n) { - if (!data.isNullAt(i)) { - out(i) = data.get(i, null) match { - case b: Array[Byte] => JTS.fromWKB(b) - case s: UTF8String => JTS.fromWKT(s.toString) - case other => throw new IllegalArgumentException( - "rst_dtmfromgeoms: geometry array element must be BINARY (WKB) or STRING (WKT); " + - s"got ${if (other == null) "null" else other.getClass.getName}") - } - } - i += 1 - } - out.filter(_ != null) - } - - override def name: String = "gbx_rst_dtmfromgeoms" - - override def builder(): FunctionBuilder = (c: Seq[Expression]) => c.length match { - case 11 => RST_DTMFromGeoms(c(0), c(1), c(2), c(3), c(4), c(5), c(6), c(7), c(8), c(9), c(10), - Literal(DefaultNoData)) - case 12 => RST_DTMFromGeoms(c(0), c(1), c(2), c(3), c(4), c(5), c(6), c(7), c(8), c(9), c(10), c(11)) - case n => throw new IllegalArgumentException( - s"gbx_rst_dtmfromgeoms takes 11 or 12 arguments (points, breaklines, merge_tolerance, " + - s"snap_tolerance, xmin, ymin, xmax, ymax, width_px, height_px, srid, [no_data]); got $n") - } -``` - -Remove the old single packed-tuple `eval`, the `firstElementType`/`secondElementType` helpers, the `splitPointFinder`/`gridOrigin`/`gridWidth*`/`gridSize*` fields, and the unused imports (`ArrayData` stays; remove `UTF8String`-only-for-origin usages as needed — keep what compiles). Update the header comment to describe the registered modern signature (drop "Not yet implemented for production"). - -In `InterpolateElevation.scala`, delete the now-unused old `def pointGrid(origin: Point, …)` (superseded by `pointGridBBox`). The `TriangulationSplitPointTypeEnum` object is also now unused — remove it. - -- [ ] **Step 4: Run test to verify it passes** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_DTMFromGeomsTest' --log dtm-exec.log -``` -Expected: PASS (5 tests incl. builder arity). - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeoms.scala \ - src/main/scala/com/databricks/labs/gbx/rasterx/operations/InterpolateElevation.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeomsTest.scala -git commit -m "feat(rasterx): modern bbox+pixels signature, Int/Long eval, safeEval fix for rst_dtmfromgeoms" -``` - ---- - -## Task 4: Aggregator `RST_DTMFromGeomsAgg` + `DTMFromGeomsAcc` - -**Files:** -- Create: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/DTMFromGeomsAcc.scala` -- Create: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeomsAgg.scala` -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeomsTest.scala` (extend) - -- [ ] **Step 1: Write the failing test** (append to `RST_DTMFromGeomsTest.scala`) - -```scala - test("DTMFromGeomsAcc serialize/deserialize roundtrips point WKBs") { - val buf = DTMFromGeomsAcc.empty - planePoints().foreach(p => buf.add(JTS.toWKB(p))) - val restored = DTMFromGeomsAcc.deserialize(buf.serialize) - restored.points.length shouldBe 4 - restored.points.zip(buf.points).foreach { case (a, b) => a shouldBe b } - } - - test("RST_DTMFromGeomsAgg produces the same raster as the non-agg execute") { - val lit = (v: Any) => org.apache.spark.sql.catalyst.expressions.Literal(v) - val buf = DTMFromGeomsAcc.empty - planePoints().foreach(p => buf.add(JTS.toWKB(p))) - val agg = RST_DTMFromGeomsAgg( - pointExpr = null, - breaklinesExpr = lit(null), - mergeToleranceExpr = lit(0.0), snapToleranceExpr = lit(0.0), - xminExpr = lit(0.0), yminExpr = lit(0.0), xmaxExpr = lit(100.0), ymaxExpr = lit(100.0), - widthPxExpr = lit(10), heightPxExpr = lit(10), sridExpr = lit(32633), - noDataExpr = lit(-9999.0) - ) - val aggRow = agg.eval(buf).asInstanceOf[InternalRow] - val nonAggRow = RST_DTMFromGeoms.execute( - planePoints(), Seq.empty[LineString], 0.0, 0.0, - 0.0, 0.0, 100.0, 100.0, 10, 10, 32633, -9999.0) - pixel(aggRow, 0, 0) shouldBe pixel(nonAggRow, 0, 0) +- 1e-9 - pixel(aggRow, 9, 9) shouldBe pixel(nonAggRow, 9, 9) +- 1e-9 - } -``` - -- [ ] **Step 2: Run test to verify it fails** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_DTMFromGeomsTest' --log dtm-agg.log -``` -Expected: FAIL — `DTMFromGeomsAcc` / `RST_DTMFromGeomsAgg` do not exist. - -- [ ] **Step 3a: Create `DTMFromGeomsAcc.scala`** - -```scala -package com.databricks.labs.gbx.rasterx.expressions - -import java.io.{ByteArrayInputStream, ByteArrayOutputStream, DataInputStream, DataOutputStream} -import scala.collection.mutable.ArrayBuffer - -/** Mutable aggregation buffer for [[RST_DTMFromGeomsAgg]]: accumulates point WKB byte - * arrays (Z carried in the geometry). Shipped between executors via serialize/deserialize. - */ -final class DTMFromGeomsAcc( - val points: ArrayBuffer[Array[Byte]] = ArrayBuffer.empty, - private var byteSize: Long = 0L -) extends Serializable { - - def add(wkb: Array[Byte]): DTMFromGeomsAcc = { - if (wkb != null && wkb.length > 0) { - points += wkb - byteSize += wkb.length.toLong - DTMFromGeomsAcc.guardSize(byteSize) - } - this - } - - def merge(other: DTMFromGeomsAcc): DTMFromGeomsAcc = { - points ++= other.points - byteSize += other.byteSize - DTMFromGeomsAcc.guardSize(byteSize) - this - } - - def serialize: Array[Byte] = { - val bos = new ByteArrayOutputStream() - val out = new DataOutputStream(bos) - out.writeInt(points.length) - for (wkb <- points) { out.writeInt(wkb.length); out.write(wkb) } - bos.toByteArray - } -} - -object DTMFromGeomsAcc { - - /** Hard cap on accumulated WKB bytes per buffer (guards memory blow-ups). */ - val MAX_BUFFER_BYTES: Long = 200L * 1024L * 1024L - - def empty: DTMFromGeomsAcc = new DTMFromGeomsAcc() - - def deserialize(bytes: Array[Byte]): DTMFromGeomsAcc = { - val in = new DataInputStream(new ByteArrayInputStream(bytes)) - val n = in.readInt() - val buf = ArrayBuffer.empty[Array[Byte]] - var total = 0L - var i = 0 - while (i < n) { - val len = in.readInt() - val wkb = new Array[Byte](len) - if (len > 0) in.readFully(wkb) - buf += wkb - total += len.toLong - i += 1 - } - new DTMFromGeomsAcc(buf, total) - } - - private[expressions] def guardSize(currentBytes: Long): Unit = { - if (currentBytes > MAX_BUFFER_BYTES) { - throw new IllegalStateException( - s"rst_dtmfromgeoms_agg buffer exceeded ${MAX_BUFFER_BYTES / (1024 * 1024)} MiB " + - s"(current = ${currentBytes / (1024 * 1024)} MiB). Tile the workload by extent.") - } - } -} -``` - -- [ ] **Step 3b: Create `RST_DTMFromGeomsAgg.scala`** (mirror `RST_GridFromPointsAgg`) - -```scala -package com.databricks.labs.gbx.rasterx.expressions - -import com.databricks.labs.gbx.expressions.WithExpressionInfo -import com.databricks.labs.gbx.vectorx.jts.JTS -import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.analysis.FunctionRegistry.FunctionBuilder -import org.apache.spark.sql.catalyst.expressions.aggregate.{ImperativeAggregate, TypedImperativeAggregate} -import org.apache.spark.sql.catalyst.expressions.{Expression, Literal} -import org.apache.spark.sql.catalyst.util.ArrayData -import org.apache.spark.sql.types._ -import org.apache.spark.unsafe.types.UTF8String -import org.locationtech.jts.geom.{Geometry, LineString} - -/** UDAF: `gbx_rst_dtmfromgeoms_agg(point, breaklines, merge_tolerance, snap_tolerance, - * xmin, ymin, xmax, ymax, width_px, height_px, srid, [no_data])`. - * - * Streams one Z-valued `point` per row into a buffer; every other argument is a - * per-group constant (read once in `eval`). Breaklines arrive as a constant ARRAY. - * Delegates to [[RST_DTMFromGeoms.execute]] so the result equals the non-agg form. - */ -final case class RST_DTMFromGeomsAgg( - pointExpr: Expression, - breaklinesExpr: Expression, - mergeToleranceExpr: Expression, - snapToleranceExpr: Expression, - xminExpr: Expression, yminExpr: Expression, xmaxExpr: Expression, ymaxExpr: Expression, - widthPxExpr: Expression, heightPxExpr: Expression, sridExpr: Expression, - noDataExpr: Expression, - mutableAggBufferOffset: Int = 0, - inputAggBufferOffset: Int = 0 -) extends TypedImperativeAggregate[DTMFromGeomsAcc] { - - import RST_DTMFromGeomsAgg.{evalDouble, evalInt, evalExpr, geomsFromArrayData} - - override lazy val deterministic: Boolean = true - override val nullable: Boolean = true - override val dataType: DataType = StructType(Seq( - StructField("index_id", LongType, nullable = true), - StructField("raster", BinaryType, nullable = true), - StructField("metadata", MapType(StringType, StringType), nullable = true) - )) - override def prettyName: String = RST_DTMFromGeomsAgg.name - - override def children: Seq[Expression] = Seq( - pointExpr, breaklinesExpr, mergeToleranceExpr, snapToleranceExpr, - xminExpr, yminExpr, xmaxExpr, ymaxExpr, - widthPxExpr, heightPxExpr, sridExpr, noDataExpr) - - override protected def withNewChildrenInternal(nc: IndexedSeq[Expression]): RST_DTMFromGeomsAgg = { - require(nc.length == 12, s"RST_DTMFromGeomsAgg expects 12 children; got ${nc.length}") - copy(nc(0), nc(1), nc(2), nc(3), nc(4), nc(5), nc(6), nc(7), nc(8), nc(9), nc(10), nc(11)) - } - - override def withNewMutableAggBufferOffset(n: Int): ImperativeAggregate = copy(mutableAggBufferOffset = n) - override def withNewInputAggBufferOffset(n: Int): ImperativeAggregate = copy(inputAggBufferOffset = n) - - override def createAggregationBuffer(): DTMFromGeomsAcc = DTMFromGeomsAcc.empty - - override def update(buffer: DTMFromGeomsAcc, input: InternalRow): DTMFromGeomsAcc = { - val pt = evalExpr(pointExpr, input) - if (pt == null) return buffer - val wkb = pt match { - case b: Array[Byte] => b - case s: UTF8String => JTS.toWKB(JTS.fromWKT(s.toString)) - case other => throw new IllegalArgumentException( - s"rst_dtmfromgeoms_agg: point column must be BINARY (WKB) or STRING (WKT); got ${other.getClass.getName}") - } - buffer.add(wkb) - } - - override def merge(a: DTMFromGeomsAcc, b: DTMFromGeomsAcc): DTMFromGeomsAcc = a.merge(b) - - override def eval(buffer: DTMFromGeomsAcc): Any = { - val empty = InternalRow.empty - val breaklines: Seq[LineString] = evalExpr(breaklinesExpr, empty) match { - case null => Seq.empty - case ad: ArrayData => geomsFromArrayData(ad).map(_.asInstanceOf[LineString]).toSeq - case other => throw new IllegalArgumentException( - s"rst_dtmfromgeoms_agg: breaklines must be an ARRAY of geometries; got ${other.getClass.getName}") - } - val points: Seq[Geometry] = buffer.points.toSeq.map(JTS.fromWKB) - RST_DTMFromGeoms.execute( - points, breaklines, - evalDouble(mergeToleranceExpr, empty, "merge_tolerance"), - evalDouble(snapToleranceExpr, empty, "snap_tolerance"), - evalDouble(xminExpr, empty, "xmin"), evalDouble(yminExpr, empty, "ymin"), - evalDouble(xmaxExpr, empty, "xmax"), evalDouble(ymaxExpr, empty, "ymax"), - evalInt(widthPxExpr, empty, "width_px"), evalInt(heightPxExpr, empty, "height_px"), - evalInt(sridExpr, empty, "srid"), - evalDouble(noDataExpr, empty, "no_data")) - } - - override def serialize(b: DTMFromGeomsAcc): Array[Byte] = b.serialize - override def deserialize(bytes: Array[Byte]): DTMFromGeomsAcc = DTMFromGeomsAcc.deserialize(bytes) -} - -object RST_DTMFromGeomsAgg extends WithExpressionInfo { - - override def name: String = "gbx_rst_dtmfromgeoms_agg" - - private[expressions] def evalExpr(e: Expression, row: InternalRow): Any = e.eval(row) - - private[expressions] def geomsFromArrayData(data: ArrayData): Array[Geometry] = { - val n = data.numElements() - val out = scala.collection.mutable.ArrayBuffer.empty[Geometry] - var i = 0 - while (i < n) { - if (!data.isNullAt(i)) { - out += (data.get(i, null) match { - case b: Array[Byte] => JTS.fromWKB(b) - case s: UTF8String => JTS.fromWKT(s.toString) - case other => throw new IllegalArgumentException( - s"rst_dtmfromgeoms_agg: breakline element must be BINARY/STRING; got ${other.getClass.getName}") - }) - } - i += 1 - } - out.toArray - } - - private[expressions] def evalDouble(e: Expression, row: InternalRow, label: String): Double = - evalExpr(e, row) match { - case null => throw new IllegalArgumentException(s"rst_dtmfromgeoms_agg: $label must not be null") - case d: Double => d - case f: Float => f.toDouble - case i: Int => i.toDouble - case l: Long => l.toDouble - case dec: org.apache.spark.sql.types.Decimal => dec.toDouble - case o => throw new IllegalArgumentException(s"rst_dtmfromgeoms_agg: $label must be numeric; got ${o.getClass.getName}") - } - - private[expressions] def evalInt(e: Expression, row: InternalRow, label: String): Int = - evalExpr(e, row) match { - case null => throw new IllegalArgumentException(s"rst_dtmfromgeoms_agg: $label must not be null") - case i: Int => i - case l: Long => l.toInt - case o => throw new IllegalArgumentException(s"rst_dtmfromgeoms_agg: $label must be INT or LONG; got ${o.getClass.getName}") - } - - override def builder(): FunctionBuilder = (c: Seq[Expression]) => c.length match { - case 11 => RST_DTMFromGeomsAgg(c(0), c(1), c(2), c(3), c(4), c(5), c(6), c(7), c(8), c(9), c(10), - Literal(RST_DTMFromGeoms.DefaultNoData)) - case 12 => RST_DTMFromGeomsAgg(c(0), c(1), c(2), c(3), c(4), c(5), c(6), c(7), c(8), c(9), c(10), c(11)) - case n => throw new IllegalArgumentException( - s"$name takes 11 or 12 arguments (point, breaklines, merge_tolerance, snap_tolerance, " + - s"xmin, ymin, xmax, ymax, width_px, height_px, srid, [no_data]); got $n") - } -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_DTMFromGeomsTest' --log dtm-agg.log -``` -Expected: PASS (7 tests incl. agg≡non-agg + buffer roundtrip). - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/DTMFromGeomsAcc.scala \ - src/main/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeomsAgg.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_DTMFromGeomsTest.scala -git commit -m "feat(rasterx): streaming RST_DTMFromGeomsAgg aggregator (agg == non-agg)" -``` - ---- - -## Task 5: Register both functions; remove scoverage exclusions - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/functions.scala` -- Modify: `pom.xml` - -- [ ] **Step 1: Uncomment + add registrations** - -In `functions.scala`, replace the line `// rd.register(RST_DTMFromGeoms)` with: - -```scala - rd.register(RST_DTMFromGeoms) -``` - -Add the aggregator registration alongside the other aggregators (near `RST_DerivedBandAgg` / the agg grouping): - -```scala - rd.register(RST_DTMFromGeomsAgg) -``` - -Both expressions are in package `...rasterx.expressions`; add imports if the file imports expressions individually (follow the existing import style in `functions.scala`). - -- [ ] **Step 2: Remove scoverage exclusions** - -In `pom.xml`, in **both** `` entries (lines ~466 and ~508), remove the -`.*RST_DTMFromGeoms\.scala;.*InterpolateElevation\.scala` portions. If they are the only -entries, set the element to empty (``); if combined with others -via `;`, remove just these two patterns and their separators. - -- [ ] **Step 3: Build to verify registration compiles and resolves** - -Rebuild the JAR (this also refreshes the stale JAR for later Python/doc tests): -``` -gbx:docker:exec "mvn clean package -PskipScoverage -DskipTests" -``` -Expected: BUILD SUCCESS. - -- [ ] **Step 4: Quick registration smoke test (optional but cheap)** - -``` -gbx:docker:exec "echo 'spark not needed'" # registration is exercised by function-info in Task 6 -``` -(There is no standalone registration unit test in this repo; Task 6's `gbx:test:function-info` is the registration gate.) - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/functions.scala pom.xml -git commit -m "feat(rasterx): register rst_dtmfromgeoms + _agg; drop scoverage exclusions" -``` - ---- - -## Task 6: registered_functions.txt + SQL doc examples + regenerate function-info - -**Files:** -- Modify: `docs/tests-function-info/registered_functions.txt` -- Modify: `docs/tests/python/api/rasterx_functions_sql.py` -- Regenerated: `src/main/resources/com/databricks/labs/gbx/function-info.json` - -- [ ] **Step 1: Add the two canonical names** - -Add to `docs/tests-function-info/registered_functions.txt` (place near the other `gbx_rst_*` -operations / aggregators; exact position is not significant — the parity check is set-based): - -``` -gbx_rst_dtmfromgeoms -gbx_rst_dtmfromgeoms_agg -``` - -- [ ] **Step 2: Add SQL doc examples** - -Append to `docs/tests/python/api/rasterx_functions_sql.py`: - -```python -def rst_dtmfromgeoms_sql_example(): - """DTM via Delaunay-TIN interpolation from Z-valued points (+ optional breaklines).""" - return """ --- TIN interpolation from arrays of Z-valued point WKB and breakline WKB. --- Output is a 100 x 100 Float64 GTiff over the extent. For N-metre cells set --- width_px = round((xmax-xmin)/N): here a 1000 m extent at 10 m cells -> 100 px. -SELECT gbx_rst_dtmfromgeoms( - points_wkb_array, breaklines_wkb_array, - 0.0, 0.01, - 0.0, 0.0, 1000.0, 1000.0, - 100, 100, 32633 -) AS dtm -FROM survey_points; -""" - - -rst_dtmfromgeoms_sql_example_output = """ -+---+ -|dtm| -+---+ -|...| -+---+ -""" - - -def rst_dtmfromgeoms_agg_sql_example(): - """DTM aggregator - one Z-valued point per row, grouped by extent key.""" - return """ --- Stream survey points per region into one TIN DTM tile. Breaklines are a --- per-group constant array; for 10 m cells over a 1000 m extent use 100 px. -SELECT region_id, - gbx_rst_dtmfromgeoms_agg( - point_wkb, breaklines_wkb_array, - 0.0, 0.01, - bbox_xmin, bbox_ymin, bbox_xmax, bbox_ymax, - 100, 100, 32633 - ) AS dtm -FROM survey_points -GROUP BY region_id; -""" - - -rst_dtmfromgeoms_agg_sql_example_output = """ -+---------+---+ -|region_id|dtm| -+---------+---+ -|... |...| -+---------+---+ -""" -``` - -- [ ] **Step 3: Regenerate function-info.json** - -``` -gbx:docs:function-info -``` -Expected: regenerates `function-info.json`; both `gbx_rst_dtmfromgeoms` and -`gbx_rst_dtmfromgeoms_agg` now appear as keys with non-empty usage. - -- [ ] **Step 4: Verify function-info coverage** - -``` -gbx:test:function-info --log dtm-fninfo.log -``` -Expected: PASS — every registered function (incl. the two new ones) has a non-empty example. - -- [ ] **Step 5: Commit** - -```bash -git add docs/tests-function-info/registered_functions.txt \ - docs/tests/python/api/rasterx_functions_sql.py \ - src/main/resources/com/databricks/labs/gbx/function-info.json -git commit -m "docs(rasterx): register rst_dtmfromgeoms(+_agg) in function-info + examples" -``` - ---- - -## Task 7: Python bindings + binding tests - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/rasterx/functions.py` -- Test: `python/geobrix/test/rasterx/test_dtmfromgeoms.py` (create) - -- [ ] **Step 1: Write the failing Python test** - -Create `python/geobrix/test/rasterx/test_dtmfromgeoms.py` (mirror the session/import pattern of the -existing rasterx tests — copy the `JAR`/`SparkSession`/`register` boilerplate header from -`python/geobrix/test/rasterx/test_vector_raster_bridge.py`, then): - -```python -def test_rst_dtmfromgeoms_returns_tile(spark): - from databricks.labs.gbx.rasterx import functions as F - from pyspark.sql import functions as f - - # Four Z-valued corner points of a 100x100 extent, as WKT (z = 2x+3y+5). - pts = [ - "POINT Z (0 0 5)", "POINT Z (100 0 205)", - "POINT Z (0 100 305)", "POINT Z (100 100 505)", - ] - df = spark.createDataFrame([(pts, [])], ["points", "breaklines"]) - out = df.select( - F.rst_dtmfromgeoms( - f.col("points"), f.col("breaklines"), - f.lit(0.0), f.lit(0.0), - f.lit(0.0), f.lit(0.0), f.lit(100.0), f.lit(100.0), - f.lit(10), f.lit(10), f.lit(32633), - ).alias("dtm") - ).collect() - assert out[0]["dtm"] is not None - assert out[0]["dtm"]["raster"] is not None - - -def test_rst_dtmfromgeoms_agg_returns_tile(spark): - from databricks.labs.gbx.rasterx import functions as F - from pyspark.sql import functions as f - - rows = [ - (1, "POINT Z (0 0 5)"), (1, "POINT Z (100 0 205)"), - (1, "POINT Z (0 100 305)"), (1, "POINT Z (100 100 505)"), - ] - df = spark.createDataFrame(rows, ["region", "pt"]) - out = ( - df.groupBy("region") - .agg( - F.rst_dtmfromgeoms_agg( - f.col("pt"), f.array().cast("array"), - f.lit(0.0), f.lit(0.0), - f.lit(0.0), f.lit(0.0), f.lit(100.0), f.lit(100.0), - f.lit(10), f.lit(10), f.lit(32633), - ).alias("dtm") - ) - .collect() - ) - assert out[0]["dtm"] is not None - assert out[0]["dtm"]["raster"] is not None -``` - -- [ ] **Step 2: Run test to verify it fails** - -(JAR was rebuilt in Task 5; if Scala changed since, rebuild first.) -``` -gbx:test:python --path python/geobrix/test/rasterx/test_dtmfromgeoms.py --log dtm-py.log -``` -Expected: FAIL — `functions` has no attribute `rst_dtmfromgeoms` / `rst_dtmfromgeoms_agg`. - -- [ ] **Step 3: Add the two wrappers** to `python/geobrix/src/databricks/labs/gbx/rasterx/functions.py` - -```python -def rst_dtmfromgeoms( - points: ColLike, - breaklines: ColLike, - merge_tolerance: ColLike, - snap_tolerance: ColLike, - xmin: ColLike, - ymin: ColLike, - xmax: ColLike, - ymax: ColLike, - width_px: ColLike, - height_px: ColLike, - srid: ColLike, - no_data: ColLike = None, -) -> Column: - """DTM from Z-valued points + optional breaklines via Delaunay-TIN interpolation. - - Output is a single-band Float64 GTiff of ``width_px x height_px`` over the bbox. - For N-unit cells set ``width_px = round((xmax-xmin)/N)``, - ``height_px = round((ymax-ymin)/N)`` (e.g. a 1000 m extent at 10 m cells -> 100 px). - - Args: - points: Array column of Z-valued point geometries (WKB binary or WKT string). - breaklines: Array column of breakline LineString geometries; pass an empty array for none. - merge_tolerance: Delaunay segment-merge tolerance. - snap_tolerance: Vertex-to-breakline snap tolerance. - xmin, ymin, xmax, ymax: Output raster extent. - width_px, height_px: Output raster size in pixels. - srid: EPSG SRID. - no_data: No-data sentinel (default -9999.0). - - Returns: - Raster tile column. - """ - nd = f.lit(-9999.0) if no_data is None else _col(no_data) - return f.call_function( - "gbx_rst_dtmfromgeoms", - _col(points), _col(breaklines), - _col(merge_tolerance), _col(snap_tolerance), - _col(xmin), _col(ymin), _col(xmax), _col(ymax), - _col(width_px), _col(height_px), _col(srid), nd, - ) - - -def rst_dtmfromgeoms_agg( - point: ColLike, - breaklines: ColLike, - merge_tolerance: ColLike, - snap_tolerance: ColLike, - xmin: ColLike, - ymin: ColLike, - xmax: ColLike, - ymax: ColLike, - width_px: ColLike, - height_px: ColLike, - srid: ColLike, - no_data: ColLike = None, -) -> Column: - """DTM aggregator - one Z-valued ``point`` per row, grouped by extent key. - - Aggregator counterpart of :func:`rst_dtmfromgeoms`. ``point`` is the only - aggregated (per-row) input; ``breaklines`` and all extent/tolerance args are - per-group constants. Produces the same DTM as the non-agg form over the same grid. - - Returns: - Raster tile column. - """ - nd = f.lit(-9999.0) if no_data is None else _col(no_data) - return f.call_function( - "gbx_rst_dtmfromgeoms_agg", - _col(point), _col(breaklines), - _col(merge_tolerance), _col(snap_tolerance), - _col(xmin), _col(ymin), _col(xmax), _col(ymax), - _col(width_px), _col(height_px), _col(srid), nd, - ) -``` - -- [ ] **Step 4: Run test to verify it passes** - -``` -gbx:test:python --path python/geobrix/test/rasterx/test_dtmfromgeoms.py --log dtm-py.log -``` -Expected: PASS (2 tests). - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/rasterx/functions.py \ - python/geobrix/test/rasterx/test_dtmfromgeoms.py -git commit -m "feat(python): rst_dtmfromgeoms + rst_dtmfromgeoms_agg bindings + tests" -``` - ---- - -## Task 8: SQL doc tests execute under Docker - -**Files:** -- Verify: the SQL examples added in Task 6 run as doc tests. - -- [ ] **Step 1: Run the SQL doc tests** - -``` -gbx:test:sql-docs --log dtm-sqldocs.log -``` -Expected: PASS — the new `gbx_rst_dtmfromgeoms` / `_agg` SQL examples execute against real data -without error. If the example references a non-existent table (`survey_points`), adjust the -example to construct inline points via `VALUES` + `ST_*`/WKT (deterministic; matches how other -examples build inputs) so it actually executes, then re-run. - -- [ ] **Step 2: Commit any example adjustments** - -```bash -git add docs/tests/python/api/rasterx_functions_sql.py \ - src/main/resources/com/databricks/labs/gbx/function-info.json -git commit -m "test(docs): executable SQL doc examples for rst_dtmfromgeoms(+_agg)" -``` - -(Re-run `gbx:docs:function-info` if the example text changed, so function-info stays in sync; include the regenerated JSON in the commit.) - ---- - -## Task 9: Full verification - -- [ ] **Step 1: Binding parity** - -``` -bash scripts/commands/gbx-test-bindings.sh --log dtm-parity.log -``` -Expected: PASS — both `gbx_rst_dtmfromgeoms` and `gbx_rst_dtmfromgeoms_agg` present in Scala -(name literals), Python (`functions.py`), and `function-info.json`; no missing-binding failures. - -- [ ] **Step 2: Full rasterx Scala suite** (background) - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.*' --log dtm-scala-all.log -``` -Expected: PASS, including `RST_DTMFromGeomsTest` and `InterpolateElevationTest`. - -- [ ] **Step 3: Python rasterx suite** - -``` -gbx:test:python --path python/geobrix/test/rasterx/ --log dtm-py-all.log -``` -Expected: PASS. - -- [ ] **Step 4: function-info coverage** - -``` -gbx:test:function-info --log dtm-fninfo.log -``` -Expected: PASS. - -- [ ] **Step 5: Push** (after `gh auth switch --user mjohns-databricks`) - -The QC judge runs on push, including the `binding-parity` check (which now also covers the two -new functions). Address any findings; do not blind-override. - -```bash -gh auth switch --user mjohns-databricks -git push origin beta/0.4.0 -``` - ---- - -## Self-review notes (author) - -- **Spec coverage:** signature modernization (Task 3) ✓; bbox+pixels Scheme A (Tasks 1-3) ✓; - safeEval fix (Task 3) ✓; pointGrid arg-order bug eliminated via `pointGridBBox` (Task 1) ✓; - out-of-hull/NaN → no_data (Tasks 1-2) ✓; splitPointFinder dropped (Task 3) ✓; shared `execute` - (Task 2) ✓; `_agg` with streamed points + constant-array breaklines (Task 4) ✓; register both + - remove scoverage exclusions (Task 5) ✓; registered_functions.txt + function-info via SQL examples - (Task 6) ✓; Python bindings (Task 7) ✓; Scala/Python/SQL doc tests + agg≡non-agg (Tasks 2,4,7,8) ✓; - binding-parity + verification (Task 9) ✓. -- **Type consistency:** `RST_DTMFromGeoms.execute(points: Seq[Geometry], breaklines: Seq[LineString], …)` - is called identically from `doInvoke` (Task 3) and the aggregator `eval` (Task 4); - `DTMFromGeomsAcc.points: ArrayBuffer[Array[Byte]]` with `add(wkb)` / `serialize` / `deserialize` - used consistently in Task 4 tests and impl; `DefaultNoData` defined once on `RST_DTMFromGeoms` - and reused by the agg builder. -- **Known follow-up flagged in Task 8:** the SQL example may need inline `VALUES`-built points to - be executable; resolved within the task rather than left as a placeholder. diff --git a/docs/superpowers/plans/2026-05-28-three-agg-variants.md b/docs/superpowers/plans/2026-05-28-three-agg-variants.md deleted file mode 100644 index 86626410e..000000000 --- a/docs/superpowers/plans/2026-05-28-three-agg-variants.md +++ /dev/null @@ -1,144 +0,0 @@ -# Three `_agg` Streaming Variants Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Steps use checkbox (`- [ ]`) syntax. - -**Goal:** Add three streaming aggregators — `gbx_quadbin_cellunion_agg`, `gbx_rst_rasterize_agg`, `gbx_rst_frombands_agg` — each a `TypedImperativeAggregate` that lets users `GROUP BY` and stream one element per row instead of `collect_list`-ing a whole array into one row. - -**Architecture:** Each mirrors an existing aggregator template and delegates finalize to an existing pure-compute method. `quadbin_cellunion_agg` → `Quadbin_CellUnion.execute`; `rst_rasterize_agg` → inline `VectorRasterBridge` (mirrors `RST_Rasterize.execute`, multi-feature); `rst_frombands_agg` → `RST_FromBands.execute` (sorted by an explicit streamed `band_index`). - -**Tech Stack:** Scala 2.13 / Spark 4.0 Catalyst `TypedImperativeAggregate`, JTS, GDAL. Tests + builds run in the `geobrix-dev` Docker container via `gbx:*`. - -**Conventions reminder:** -- Run Scala/Python tests via `gbx:*` IN THE FOREGROUND, wait for `BUILD SUCCESS`/`BUILD FAILURE` + `Tests: succeeded N, failed M` before reporting. Never host `mvn`. -- After Scala changes, the JAR is stale; rebuild via `gbx:docker:exec "mvn clean package -PskipScoverage -DskipTests"` before Python/doc tests. -- `gh auth switch --user mjohns-databricks` before any push. Use ASCII only in source (scalastyle `nonascii` warns on em-dashes etc.). -- The `binding-parity` QC check requires: every name in `registered_functions.txt` has a Scala `override def name = "gbx_..."` literal, a Python `call_function("gbx_...")` wrapper, and a `function-info.json` entry. - -**Design reference:** see `docs/superpowers/specs/` dtmfromgeoms design for the established `_agg` pattern. Templates to mirror: `.../gridx/bng/agg/BNG_CellUnionAgg.scala` (+ `UnionAcc.scala`), `.../rasterx/expressions/agg/RST_MergeAgg.scala`, and the just-built `.../rasterx/expressions/RST_DTMFromGeomsAgg.scala` (constant-expr handling + `ExpressionConfigExpr` child). - ---- - -## File Structure - -| File | Responsibility | -|---|---| -| `.../gridx/quadbin/agg/Quadbin_CellUnionAgg.scala` (+ `QuadbinUnionAcc.scala` or inline buffer) | Stream `BIGINT` cells; finalize `Quadbin_CellUnion.execute`. | -| `.../rasterx/expressions/agg/RST_RasterizeAgg.scala` | Stream `(geom_wkb, value)`; extent/srid as constant children; inline multi-feature rasterize. | -| `.../rasterx/expressions/agg/RST_FromBandsAgg.scala` | Stream `(tile, band_index)`; sort by band_index; finalize `RST_FromBands.execute`. | -| `.../rasterx/functions.scala`, `.../gridx/quadbin/functions.scala` | Register the three. | -| `docs/tests-function-info/registered_functions.txt` | Add 3 names. | -| `docs/tests/python/api/{rasterx,gridx}_functions_sql.py` | 3 `*_sql_example()`. | -| `src/main/resources/.../function-info.json` | Regenerated. | -| `python/geobrix/src/databricks/labs/gbx/{rasterx,gridx/quadbin}/functions.py` | 3 wrappers. | -| `src/test/scala/.../{quadbin,rasterx}/...AggTest.scala` | agg≡non-agg tests. | -| `python/geobrix/test/.../test_*_agg.py` | binding smoke tests. | - ---- - -## Task 1: `gbx_quadbin_cellunion_agg` - -**Files:** Create `src/main/scala/com/databricks/labs/gbx/gridx/quadbin/agg/Quadbin_CellUnionAgg.scala` (and a small buffer if needed); test `src/test/scala/com/databricks/labs/gbx/gridx/quadbin/Quadbin_CellUnionAggTest.scala`. - -**Design (verified):** `Quadbin_CellUnion` (non-agg) takes `ARRAY` and returns `BinaryType` (EWKB, SRID 4326) via the reusable `object Quadbin_CellUnion { def execute(cells: Array[Long]): Array[Byte] }`. The agg streams ONE `BIGINT` cell per row, buffers them, and calls `Quadbin_CellUnion.execute(buffer.toArray)` in `eval`. No per-group constants. Mirror `BNG_CellUnionAgg` structurally, but the buffer is just a `Long` accumulator (no chip struct / isCore). `UnaryLike[Expression]` (single child = the cell column). Return type `BinaryType`. - -- [ ] **Step 1: Read templates.** Read `.../gridx/bng/agg/BNG_CellUnionAgg.scala` + `UnionAcc.scala` (structure: TypedImperativeAggregate overrides, serde), and `.../gridx/quadbin/Quadbin_CellUnion.scala` (confirm `execute(Array[Long]): Array[Byte]` and the non-agg's return/SRID). Also read an existing quadbin test (e.g. find `Quadbin_CellUnion`'s test or any `src/test/.../quadbin/*Test.scala`) to learn how valid cell IDs are constructed in tests. - -- [ ] **Step 2: Write the failing test.** `Quadbin_CellUnionAggTest.scala` — an agg≡non-agg test: obtain a handful of valid quadbin cell IDs (construct them the same way the existing quadbin tests do — e.g. via the quadbin point→cell function or known-good literals), accumulate them into the agg's buffer, call `agg.eval(buf)`, and assert the resulting EWKB bytes equal `Quadbin_CellUnion.execute(sameCellsArray)`. Plus a buffer serialize/deserialize roundtrip test. Use `AnyFunSuite with Matchers`. Pattern the agg construction after the `RST_DTMFromGeomsAgg` test (build the case class with `Literal`/`null` child, call `.eval(buf)`). - -- [ ] **Step 3: Run test, verify it fails** (FOREGROUND, wait): `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.gridx.quadbin.Quadbin_CellUnionAggTest' --log qb-union-agg.log`. Expect compile-fail (`Quadbin_CellUnionAgg` missing). - -- [ ] **Step 4: Implement `Quadbin_CellUnionAgg`.** A `TypedImperativeAggregate[]` where the buffer accumulates `Long` cell ids (a small serializable acc with ByteBuffer serde `[count(4)][id*8...]`, OR reuse a simple `scala.collection.mutable.ArrayBuffer[Long]` wrapped in an acc class — match the serde rigor of `UnionAcc`). `update`: append `child.eval(input).asInstanceOf[Long]` (guard null). `merge`: concat. `eval`: `Quadbin_CellUnion.execute(buf.toArray)` (returns `Array[Byte]`; the agg's `dataType` is `BinaryType`, so return the bytes directly). `serialize`/`deserialize` via the acc. Companion: `name = "gbx_quadbin_cellunion_agg"`, `builder = c => Quadbin_CellUnionAgg(c.head)`. Place in new `agg/` subpackage mirroring `bng/agg/`. - -- [ ] **Step 5: Run test, verify pass** (FOREGROUND, wait). Expect 2 tests pass. - -- [ ] **Step 6: Commit** `git commit -m "feat(gridx): streaming gbx_quadbin_cellunion_agg (agg == non-agg)"` - ---- - -## Task 2: `gbx_rst_rasterize_agg` - -**Files:** Create `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/agg/RST_RasterizeAgg.scala`; test `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/agg/RST_RasterizeAggTest.scala`. - -**Design (verified):** `RST_Rasterize` (non-agg) signature `(geom_wkb BINARY, value DOUBLE, xmin, ymin, xmax, ymax DOUBLE, width_px, height_px, srid INT) → tile`, with `object RST_Rasterize { def execute(geomWkb, value, xmin..srid, conf): InternalRow }` that internally calls `VectorRasterBridge.buildOgrLayer(Seq((geomWkb, value)), srid)` (a single-element Seq). The agg STREAMS `(geom_wkb, value)`; extent/size/srid are PER-GROUP CONSTANTS modeled as constant child expressions (the `RST_DTMFromGeomsAgg`/`GridFromPointsAgg` pattern — read them via `InternalRow.empty` in `eval`). There is NO existing multi-feature execute, so `eval` inlines the same steps as `RST_Rasterize.execute` but passes the full accumulated `Seq[(wkb,value)]` to `buildOgrLayer`. Include `ExpressionConfigExpr()` as a child and `ExpressionConfig` init in `eval` (mirror `RST_MergeAgg`). Burn overlap = last-wins in layer order (documented; nondeterministic across the group — acceptable). Return tile struct `RST_ExpressionUtil.tileDataType(BinaryType)`. - -- [ ] **Step 1: Read** `RST_Rasterize.scala` (full — the `execute` body is the recipe), `VectorRasterBridge.scala` (`buildOgrLayer`, `buildEmptyRaster`, `toGTiffBytes`, and how RST_Rasterize.execute does the `gdal.RasterizeLayer` call), `RST_MergeAgg.scala` (TypedImperativeAggregate + `ExpressionConfigExpr` child + tile-row buffer serde), and `RST_DTMFromGeomsAgg.scala` (constant-expr `evalDouble`/`evalInt` readers, builder arg-count pattern). Read `RST_RasterizeTest.scala` for how to build geometries + read pixels back. - -- [ ] **Step 2: Write the failing test.** `RST_RasterizeAggTest.scala`: GDAL `beforeAll` setup (copy from `RST_DTMFromGeomsTest`/`RST_RasterizeTest`). agg≡non-agg-ish test: stream 2-3 non-overlapping polygons (WKB) with distinct burn values into the agg buffer over a known extent; assert the output raster has the expected burn value at a pixel inside each polygon and `no_data` outside. Since RST_Rasterize is single-geom, the equivalence anchor is: rasterizing features A and B via the agg yields a raster where A's pixels = A's value and B's pixels = B's value (i.e. both burned). Also a buffer serde roundtrip test. Build the agg case class with `Literal` constants for extent/size/srid. - -- [ ] **Step 3: Run, verify fail** (FOREGROUND, wait): `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.agg.RST_RasterizeAggTest' --log rasterize-agg.log`. - -- [ ] **Step 4: Implement `RST_RasterizeAgg`.** `TypedImperativeAggregate` with children `(geomWkbExpr, valueExpr, xminExpr, yminExpr, xmaxExpr, ymaxExpr, widthPxExpr, heightPxExpr, sridExpr, ExpressionConfigExpr())`. Buffer accumulates `(Array[Byte], Double)` features (acc class with ByteBuffer serde `[count][ (wkbLen, wkb, value) * N ]`). `update`: eval geomWkb (BINARY) + value (DOUBLE), append (skip nulls). `merge`: concat. `eval`: read constants via `InternalRow.empty` (Int/Long-tolerant readers), init ExpressionConfig, then `buildOgrLayer(buffer.features, srid)` → `buildEmptyRaster(xmin..srid, noData)` → `gdal.RasterizeLayer(...)` with `ATTRIBUTE=value` (replicate RST_Rasterize.execute's exact rasterize options) → `toGTiffBytes` → tile `InternalRow` (reuse the tile-row construction from RST_Rasterize.execute). Companion `name = "gbx_rst_rasterize_agg"`, builder accepting the 9 args (geom,value + 7 constants). Release GDAL datasets in `finally`. - -- [ ] **Step 5: Run, verify pass** (FOREGROUND, wait). - -- [ ] **Step 6: Commit** `git commit -m "feat(rasterx): streaming gbx_rst_rasterize_agg (burns many features per group)"` - ---- - -## Task 3: `gbx_rst_frombands_agg` - -**Files:** Create `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/agg/RST_FromBandsAgg.scala`; test `.../agg/RST_FromBandsAggTest.scala`. - -**Design (verified):** `RST_FromBands` (non-agg) takes `ARRAY` (band order = array position) and returns a single multiband tile via `object RST_FromBands { def execute(tiles: Seq[(Long, Dataset, Map[String,String])]): (Dataset, Map[String,String]) }` (uses `MergeBands.merge` → `gdalbuildvrt -separate`, band N = input N). **Band order matters and UDAF merge order is nondeterministic**, so the agg streams `(tile, band_index INT)` and SORTS by `band_index` ascending in `eval` before calling `execute`. Mirror `RST_MergeAgg`'s tile-buffer serde but extend each buffer element to a 2-field struct `(band_index: Int, tile: tileDataType)`. `BinaryLike[Expression]` (two children: tile + band_index) plus `ExpressionConfigExpr()`. Return tile struct (same rasterType as input). - -- [ ] **Step 1: Read** `RST_FromBands.scala` (full — confirm `execute(Seq[(Long,Dataset,Map)])` and that band order = Seq order; note how it derives output cellID/metadata from `tiles.head`), `RST_MergeAgg.scala` (full — buffer `ArrayBuffer[Any]` of tile `InternalRow`s, `UnsafeProjection`-based serialize/deserialize, `RasterSerializationUtil.rowToTile`/`tileToRow`). Read `RST_MergeAggTest` (or RST_FromBands test) for tile test-data construction. - -- [ ] **Step 2: Write the failing test.** `RST_FromBandsAggTest.scala`: construct 2-3 single-band tiles (reuse the band test-data construction from the RST_FromBands/RST_Merge tests). Stream them into the agg buffer WITH band_index values in SHUFFLED order (e.g. add band 3 first, then 1, then 2) to prove sorting works; call `agg.eval(buf)`; assert the output tile has the bands in band_index order — compare against `RST_FromBands.execute` on the tiles in correct (1,2,3) order. Assert output band count = number of inputs. Plus a buffer serde roundtrip test (with indices). - -- [ ] **Step 3: Run, verify fail** (FOREGROUND, wait): `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.agg.RST_FromBandsAggTest' --log frombands-agg.log`. - -- [ ] **Step 4: Implement `RST_FromBandsAgg`.** `TypedImperativeAggregate` with children `(tileExpr, bandIndexExpr, ExpressionConfigExpr())`. Buffer: `ArrayBuffer[Any]` where each element is an `InternalRow` of `(band_index: Int, tile: tileStruct)` (copy via `InternalRow.copyValue`). `update`: eval bandIndex (Int) + tile (struct), append `InternalRow(idx, tileCopy)`. `merge`: `++=`. `eval`: init ExpressionConfig; sort buffer by `row.getInt(0)`; extract each tile via `RasterSerializationUtil.rowToTile(row.getStruct(1, 3), rasterType)`; call `RST_FromBands.execute(sortedTiles)`; wrap result via `RasterSerializationUtil.tileToRow(...)`; release datasets. Serialize/deserialize: extend RST_MergeAgg's `UnsafeProjection` approach with element type `StructType(StructField("idx", IntegerType), StructField("tile", tileDataType))`. Companion `name = "gbx_rst_frombands_agg"`, builder accepting (tile, band_index). - -- [ ] **Step 5: Run, verify pass** (FOREGROUND, wait). - -- [ ] **Step 6: Commit** `git commit -m "feat(rasterx): streaming gbx_rst_frombands_agg (band_index-ordered band stacking)"` - ---- - -## Task 4: Register all three + rebuild JAR - -**Files:** `.../rasterx/functions.scala`, `.../gridx/quadbin/functions.scala`, (imports as needed). - -- [ ] **Step 1:** In `quadbin/functions.scala`, add `rd.register(Quadbin_CellUnionAgg)` near the other quadbin registrations (add import for the new `agg` subpackage class). In `rasterx/functions.scala`, add `rd.register(RST_RasterizeAgg)` and `rd.register(RST_FromBandsAgg)` near the other aggregator registrations (the `expressions._` wildcard likely covers `expressions.agg`? — verify; if not, add imports for the `agg` subpackage). -- [ ] **Step 2: Rebuild JAR** (FOREGROUND, wait): `gbx:docker:exec "mvn clean package -PskipScoverage -DskipTests"`. Expect BUILD SUCCESS (confirms all three register + compile). -- [ ] **Step 3: Commit** `git commit -m "feat: register quadbin_cellunion_agg, rst_rasterize_agg, rst_frombands_agg"` - ---- - -## Task 5: registered_functions.txt + SQL examples + function-info - -- [ ] **Step 1:** Add `gbx_quadbin_cellunion_agg`, `gbx_rst_rasterize_agg`, `gbx_rst_frombands_agg` to `docs/tests-function-info/registered_functions.txt`. -- [ ] **Step 2:** Add a `*_sql_example()` + `_output` for each, matching the file conventions (quadbin one goes in the gridx/quadbin SQL examples file — find where `gbx_quadbin_*` examples live; rasterize/frombands go in `rasterx_functions_sql.py`). Mirror the `rst_gridfrompoints_agg_sql_example` / `rst_dtmfromgeoms_agg_sql_example` style (illustrative `GROUP BY` SQL; placeholder tables are fine — they are display + structural-validation only, not executed). For frombands include the `band_index` column in the example. -- [ ] **Step 3: Regenerate** (FOREGROUND, wait): `gbx:docs:function-info`. Confirm all three appear in `function-info.json`. -- [ ] **Step 4: Verify coverage** (FOREGROUND, wait): `gbx:test:function-info --log three-agg-fninfo.log` — the `test_full_coverage_against_registered_list` test must pass (the pre-existing `No module named databricks` errors are unrelated baseline noise — confirm the coverage test itself passes). -- [ ] **Step 5: Commit** `git commit -m "docs: function-info examples for the three new _agg functions"` - ---- - -## Task 6: Python bindings + tests - -**Files:** `python/.../rasterx/functions.py`, `python/.../gridx/quadbin/functions.py`, new `test_*_agg.py` files. - -- [ ] **Step 1: Write failing Python tests** mirroring `test_dtmfromgeoms.py`'s session header. For each function a smoke test: build a small DataFrame, `groupBy`, call the wrapper, assert a non-null result. quadbin: stream cell BIGINTs (get cells via the quadbin point→cell binding or literal cell ids), assert union geometry returned. rasterize: stream `(wkb, value)` rows + constant extent, assert tile. frombands: stream `(tile, band_index)` rows, assert tile. -- [ ] **Step 2: Run, verify fail** (FOREGROUND, wait): `gbx:test:python --path --log three-agg-py.log`. -- [ ] **Step 3: Add wrappers.** `rst_rasterize_agg(geom_wkb, value, xmin, ymin, xmax, ymax, width_px, height_px, srid)` and `rst_frombands_agg(tile, band_index)` in `rasterx/functions.py`; `quadbin_cellunion_agg(cell)` in `gridx/quadbin/functions.py`. Each `return f.call_function("gbx_...", _col(...), ...)`. Match the existing wrapper style + docstrings. -- [ ] **Step 4: Run, verify pass** (FOREGROUND, wait). -- [ ] **Step 5: Commit** `git commit -m "feat(python): bindings + tests for the three new _agg functions"` - ---- - -## Task 7: Full verification + push - -- [ ] **Step 1: binding-parity** — `bash scripts/commands/gbx-test-bindings.sh --log three-agg-parity.log` → all three present in Scala/Python/function-info; parity green (count 144). -- [ ] **Step 2: Scala suites** (FOREGROUND/background, wait): `gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.*'` and `--suite 'com.databricks.labs.gbx.gridx.*'` → 0 failures. -- [ ] **Step 3: Python suites:** `gbx:test:python --path python/geobrix/test/rasterx/` and `--path python/geobrix/test/gridx/` → pass. -- [ ] **Step 4: scalastyle:** `gbx:lint:scalastyle` → 0 errors (ASCII-only; no `nonascii` warnings on new files). -- [ ] **Step 5: function-info coverage** → pass. -- [ ] **Step 6: Push** (`gh auth switch --user mjohns-databricks` first): `git push origin beta/0.4.0`. The QC `binding-parity` check gates the three new functions. - ---- - -## Self-review notes (author) -- **Coverage:** all three functions get impl+test (T1-3), registration (T4), function-info+examples (T5), Python bindings+tests (T6), full verification incl. binding-parity (T7). The `band_index` ordering decision is implemented (T3) and tested via shuffled-order input. Rasterize last-wins overlap documented (T2). -- **Type consistency:** finalize methods are verified to exist — `Quadbin_CellUnion.execute(Array[Long]): Array[Byte]`, `RST_FromBands.execute(Seq[(Long,Dataset,Map)]): (Dataset,Map)`; `RST_Rasterize.execute` is single-feature so `rst_rasterize_agg` inlines the multi-feature path via `VectorRasterBridge` (no nonexistent method referenced). -- **Risk:** the implementer must read the named template files for exact serde/TypedImperativeAggregate boilerplate (test-data construction for quadbin cells, band tiles) — flagged in each task's Step 1. diff --git a/docs/superpowers/plans/2026-05-29-custom-grid.md b/docs/superpowers/plans/2026-05-29-custom-grid.md deleted file mode 100644 index 58a01e94f..000000000 --- a/docs/superpowers/plans/2026-05-29-custom-grid.md +++ /dev/null @@ -1,244 +0,0 @@ -# Custom Grid (gbx_custom_*) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Steps use checkbox (`- [ ]`) syntax. - -**Goal:** Add a *bring-your-own regular cell-index grid* to GridX: a user defines a grid by bounds + root cell size + split factor (in any projected CRS), and gets the core cell-index vocabulary on it — `point→cellId`, `cellId→polygon/centroid`, `polyfill`, `kRing`. - -**Why (utility):** GeoBrix's built-in grids are each fixed — BNG (UK EPSG:27700 only), QuadBin/H3 (WGS84). None lets a user index into *their own* regular grid in *their own* CRS at *their own* cell size. Custom gridding enables spatial binning / aggregation / tiling on an arbitrary regular grid (e.g. a national grid in its native CRS, or a study-area analysis grid), reusing the same grid-op vocabulary as BNG. Distinct from raster/point gridding (`rst_gridfrompoints`, `st_interpolateelevation*`) which produce rasters/points, not a reusable cell index. - -**Architecture:** The complete core math already exists, commented out, in `gridx/grid/{GridConf.scala, CustomGridSystem.scala}` (it's correct and slightly *ahead* of the reference — `cellIdToBoundary`/`cellIdToCenter` are implemented). Work = uncomment + fix the one broken import, then add bespoke `gbx_custom_*` Spark expressions (GridX has no shared IndexSystem trait — BNG/QuadBin are bespoke; mirror that). The grid spec is passed per call as a **struct** built by a `gbx_custom_grid(...)` constructor, so op signatures stay small (`op(operand, grid[, res])`). Each op decodes the struct → `GridConf` → `CustomGridSystem` → method. - -**Polyfill semantic (clarification):** the existing core `polyfill` is a correct, standard **centroid-containment** polyfill — a cell is included iff its center falls inside the geometry (same semantic as H3 polyfill). This is NOT a bug; ship it as-is with the semantic documented + tested. (A BNG-style *intersects*-coverage flood-fill is a different semantic and an explicit future option, not this scope.) - -**Tech Stack:** Scala 2.13 / Spark 4.0 Catalyst expressions, JTS. Builds/tests in the `geobrix-dev` Docker container via `gbx:*`. - -**Conventions:** Run Scala/Python tests via `gbx:*` IN THE FOREGROUND, wait for `BUILD SUCCESS/FAILURE` + `Tests: succeeded N`. Never host `mvn`. Rebuild JAR after Scala changes before Python tests. ASCII-only source. `gh auth switch --user mjohns-databricks` before push. **Before pushing python changes run `gbx:lint:python --check`.** PySpark sends ints as Long → readers for int args (resolution, k, splits, sizes) must accept Int **or** Long. - ---- - -## File Structure - -| File | Responsibility | -|---|---| -| `gridx/grid/GridConf.scala` | Uncomment the `GridConf` case class (no logic change). | -| `gridx/grid/CustomGridSystem.scala` | Uncomment; fix `import JTS` → `com.databricks.labs.gbx.vectorx.jts.JTS`; the rest is correct. | -| `gridx/custom/Custom_GridSpec.scala` (new) | Shared: the grid-struct `StructType` schema + `gridConfFromRow(InternalRow): GridConf` decoder + Int/Long readers. | -| `gridx/custom/Custom_Grid.scala` (new) | `gbx_custom_grid(...)` constructor expression → grid struct (with validation). | -| `gridx/custom/Custom_PointAsCell.scala` (new) | `gbx_custom_pointascell(point_geom, grid, res) -> BIGINT` | -| `gridx/custom/Custom_AsWKB.scala`, `Custom_AsWKT.scala`, `Custom_Centroid.scala` (new) | `cellId, grid -> polygon WKB / polygon WKT / centroid-point WKB` | -| `gridx/custom/Custom_Polyfill.scala` (new) | `gbx_custom_polyfill(geom, grid, res) -> ARRAY` | -| `gridx/custom/Custom_KRing.scala` (new) | `gbx_custom_kring(cell, grid, k) -> ARRAY` | -| `gridx/custom/functions.scala` (new) | `register(spark)` for all `gbx_custom_*`; wired into GridX registration. | -| `gridx/functions.scala` (or wherever GridX aggregates registration) | Call custom `register`. | -| `docs/tests-function-info/registered_functions.txt` | Add the 7 names. | -| `docs/tests/python/api/gridx_functions_sql.py` | `*_sql_example()` for each. | -| `src/main/resources/.../function-info.json` | Regenerated. | -| `python/.../gridx/custom/functions.py` (new) | 7 wrappers. | -| `src/test/scala/.../gridx/...` | core math test + per-op tests. | -| `python/geobrix/test/gridx/custom/test_custom_grid.py` (new) | binding tests. | - ---- - -## Task 1: Uncomment + fix the core (GridConf + CustomGridSystem) + core unit test - -**Files:** `gridx/grid/GridConf.scala`, `gridx/grid/CustomGridSystem.scala`; test `src/test/scala/com/databricks/labs/gbx/gridx/grid/CustomGridSystemTest.scala` (new). - -- [ ] **Step 1: Write the failing test** — `CustomGridSystemTest.scala` (AnyFunSuite + Matchers). Use a known grid `GridConf(0, 100, 0, 100, cellSplits = 2, rootCellSizeX = 10, rootCellSizeY = 10, crsID = Some(32633))` and `val g = CustomGridSystem(conf)`. Assert: - - `g.pointToCellID(5.0, 5.0, 0)` returns a Long whose `g.getCellResolution(id) == 0` and whose `cellIdToGeometry(id)` is the rectangle `[0,10]×[0,10]` (check envelope min/max). Point (5,5) at res 0 (10×10 root cells) → cell (0,0). - - `g.pointToCellID(15.0, 25.0, 0)` → cell (1,2): envelope `[10,20]×[20,30]`. - - At res 1 (cellSplits=2 → 5×5 cells over the 10-unit root? NO: cellWidth(1) = 10/2^1 = 5; totalCellsX(1) = rootCellCountX * 2^1 = 10*2 = 20): `g.pointToCellID(2.5, 2.5, 1)` → cell width 5 → cell (0,0) envelope `[0,5]×[0,5]`. - - `cellIdToCenter` of the (0,0) res-0 cell ≈ (5,5). - - `g.polyfill(, 0)` returns the 9 cells whose centers (5,15,25 × 5,15,25) fall inside — assert size 9 (centroid semantic). - - `g.kRing(
, 1)` returns the 3×3 (or clipped) neighbourhood. - - Build the polygon for polyfill via `JTS.fromWKT("POLYGON ((0 0, 30 0, 30 30, 0 30, 0 0))")`. - -- [ ] **Step 2: Run, verify FAIL** (FOREGROUND, wait): `gbx:test:scala --suite 'com.databricks.labs.gbx.gridx.grid.CustomGridSystemTest' --log custom-core.log` — expect compile-fail (GridConf/CustomGridSystem are commented out). - -- [ ] **Step 3: Uncomment the core.** In `GridConf.scala`: uncomment the `case class GridConf(...)` block (remove the leading `//` on lines 4-34). No logic change. In `CustomGridSystem.scala`: uncomment everything (remove leading `//`), and FIX the broken import on (commented) line 5: `import JTS` → `import com.databricks.labs.gbx.vectorx.jts.JTS`. Keep `import org.apache.spark.unsafe.types.UTF8String`, `import org.locationtech.jts.geom.{Coordinate, Geometry}`, `import scala.util.{Success, Try}`. Verify `JTS.point(Double, Double)` and `JTS.polygonFromXYs(Array[(Double,Double)])` are used (they exist in JTS) — no change needed. - -- [ ] **Step 4: Run, verify PASS** (FOREGROUND, wait). Expect all core tests pass. If a cell-position/envelope assertion is off, re-derive the expected cell by hand from the formulas (`cellWidth(res) = rootCellSizeX / cellSplits^res`, `cellPosX = floor((x - boundXMin)/cellWidth)`, `totalCellsX(res) = rootCellCountX * cellSplits^res`) and correct the TEST's expected value (the core math is the reference) — do not change the core unless a real bug surfaces. - -- [ ] **Step 5: Commit** `git commit -m "feat(gridx): enable CustomGridSystem core (uncomment GridConf + CustomGridSystem, fix import)"` - ---- - -## Task 2: Grid-spec struct + `gbx_custom_grid` constructor - -**Files:** `gridx/custom/Custom_GridSpec.scala` (new, shared helpers), `gridx/custom/Custom_Grid.scala` (new constructor); test `src/test/scala/.../gridx/custom/Custom_GridTest.scala`. - -**Struct schema** (the grid spec carried between functions): -``` -StructType(Seq( - StructField("bound_x_min", LongType, false), - StructField("bound_x_max", LongType, false), - StructField("bound_y_min", LongType, false), - StructField("bound_y_max", LongType, false), - StructField("cell_splits", IntegerType, false), - StructField("root_cell_size_x", IntegerType, false), - StructField("root_cell_size_y", IntegerType, false), - StructField("srid", IntegerType, false) // -1 == no CRS (Option None) -)) -``` - -- [ ] **Step 1: Read** `gridx/bng/BNG_PointAsCell.scala` + `gridx/bng/functions.scala` for the expression base class + registration pattern, and `gridx/grid/CustomGridSystem.scala` (now uncommented) for `GridConf`/`CustomGridSystem`. - -- [ ] **Step 2: Write `Custom_GridSpec.scala`** (an object with shared helpers; no expression): -```scala -package com.databricks.labs.gbx.gridx.custom - -import com.databricks.labs.gbx.gridx.grid.{CustomGridSystem, GridConf} -import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.types._ - -object Custom_GridSpec { - /** Schema of the grid-spec struct produced by gbx_custom_grid and consumed by all ops. */ - val gridStructType: StructType = StructType(Seq( - StructField("bound_x_min", LongType, nullable = false), - StructField("bound_x_max", LongType, nullable = false), - StructField("bound_y_min", LongType, nullable = false), - StructField("bound_y_max", LongType, nullable = false), - StructField("cell_splits", IntegerType, nullable = false), - StructField("root_cell_size_x", IntegerType, nullable = false), - StructField("root_cell_size_y", IntegerType, nullable = false), - StructField("srid", IntegerType, nullable = false) - )) - - /** Decode the grid-spec struct InternalRow into a CustomGridSystem. */ - def systemFromRow(row: InternalRow): CustomGridSystem = { - require(row != null, "gbx_custom: grid spec must not be null") - val srid = row.getInt(7) - CustomGridSystem(GridConf( - boundXMin = row.getLong(0), boundXMax = row.getLong(1), - boundYMin = row.getLong(2), boundYMax = row.getLong(3), - cellSplits = row.getInt(4), - rootCellSizeX = row.getInt(5), rootCellSizeY = row.getInt(6), - crsID = if (srid < 0) None else Some(srid) - )) - } - - /** Int-or-Long tolerant read (PySpark sends Long). */ - def asInt(v: Any, label: String): Int = v match { - case i: Int => i - case l: Long => l.toInt - case null => throw new IllegalArgumentException(s"gbx_custom: $label must not be null") - case o => throw new IllegalArgumentException(s"gbx_custom: $label must be INT or LONG; got ${o.getClass.getName}") - } -} -``` - -- [ ] **Step 3: Write the failing constructor test** — `Custom_GridTest.scala`: build `Custom_Grid` with `Literal` args (0L,100L,0L,100L,2,10,10,32633), eval against `InternalRow.empty`, assert the returned `InternalRow` has the 8 fields with those values; assert `Custom_GridSpec.systemFromRow(result)` yields a `CustomGridSystem` whose `conf.maxResolution > 0`. Assert validation: `xmax<=xmin` (e.g. 100,0) throws; `cell_splits < 2` throws; `root_cell_size_x <= 0` throws. - -- [ ] **Step 4: Run, verify FAIL** (FOREGROUND, wait): suite `com.databricks.labs.gbx.gridx.custom.Custom_GridTest`. - -- [ ] **Step 5: Implement `Custom_Grid.scala`** — a Catalyst expression (extend `Expression with CodegenFallback`, or the simplest base that returns a struct; mirror how an existing GeoBrix expression returns a StructType — check BNG which returns chip structs). 8 children (bound_x_min..srid), `dataType = Custom_GridSpec.gridStructType`, `nullable = false`. `eval(input)`: read the 8 args (Long bounds via `asLong`-tolerant; Int splits/sizes/srid via `Custom_GridSpec.asInt`), validate (`xmax > xmin`, `ymax > ymin`, `cell_splits >= 2`, `root_cell_size_x > 0`, `root_cell_size_y > 0`), return `InternalRow(xmin, xmax, ymin, ymax, splits, rootX, rootY, srid)`. Companion `extends WithExpressionInfo`: `name = "gbx_custom_grid"`, builder accepting 7 args (srid defaulted to `Literal(-1)`) or 8 args. `withNewChildrenInternal` copies children. - -- [ ] **Step 6: Run, verify PASS** (FOREGROUND, wait). - -- [ ] **Step 7: Commit** `git commit -m "feat(gridx): gbx_custom_grid grid-spec constructor + shared decoder"` - ---- - -## Task 3: Cell-identity ops — `pointascell`, `aswkb`, `aswkt`, `centroid` - -**Files:** `gridx/custom/Custom_PointAsCell.scala`, `Custom_AsWKB.scala`, `Custom_AsWKT.scala`, `Custom_Centroid.scala` (new); test `Custom_OpsTest.scala`. - -Each op: read the grid struct via `Custom_GridSpec.systemFromRow`, then call the matching `CustomGridSystem` method. - -- [ ] **Step 1: Write the failing test** — `Custom_OpsTest.scala`: grid `gbx_custom_grid(0,100,0,100,2,10,10,32633)` (build the struct via `Custom_Grid` eval, or directly an `InternalRow` of the 8 fields). Construct each op expression with `Literal` children + the grid struct literal, eval, assert: - - `Custom_PointAsCell(point WKB at (5,5), grid, res=0)` → a Long; feeding that Long back to `Custom_AsWKB(cell, grid)` → polygon WKB whose envelope is `[0,10]×[0,10]`. - - `Custom_AsWKT(cell, grid)` → WKT string starting `POLYGON`. - - `Custom_Centroid(cell, grid)` → point WKB at ≈(5,5). - Build the input point via `JTS.toWKB(JTS.point(5.0, 5.0))`. - -- [ ] **Step 2: Run, verify FAIL** (FOREGROUND, wait): suite `com.databricks.labs.gbx.gridx.custom.Custom_OpsTest`. - -- [ ] **Step 3: Implement the four ops.** Mirror a BNG op's expression base. Decode geometry inputs with the typed pattern (`getBinary`/`getUTF8String` by declared element type, or `JTS.fromWKB`/`fromWKT` on the value — these are scalar geometry args, not arrays, so `geom.eval(input)` → `Array[Byte]`/`UTF8String` → `JTS.fromWKB`/`fromWKT`). Specs: - - **Custom_PointAsCell**(geomExpr, gridExpr, resExpr): `dataType = LongType`. eval: `sys = systemFromRow(gridExpr.eval(input).asInstanceOf[InternalRow])`; decode point geom → `c = geom.getCoordinate`; `res = asInt(resExpr.eval(input), "resolution")`; return `sys.pointToCellID(c.x, c.y, res)`. name `gbx_custom_pointascell`, 3-arg builder. - - **Custom_AsWKB**(cellExpr, gridExpr): `dataType = BinaryType`. eval: `sys.cellIdToGeometry(cell)` → `JTS.toWKB(_)`. name `gbx_custom_cellaswkb`, 2-arg. - - **Custom_AsWKT**(cellExpr, gridExpr): `dataType = StringType`. → `UTF8String.fromString(JTS.toWKT(sys.cellIdToGeometry(cell)))`. name `gbx_custom_cellaswkt`, 2-arg. - - **Custom_Centroid**(cellExpr, gridExpr): `dataType = BinaryType`. → `c = sys.cellIdToCenter(cell)`; `JTS.toWKB(JTS.point(c))`. name `gbx_custom_centroid`, 2-arg. - `cell` args are `Long` (read via `asInt`-style but Long: `cellExpr.eval(input).asInstanceOf[Long]`). Guard null grid/cell. - -- [ ] **Step 4: Run, verify PASS** (FOREGROUND, wait). - -- [ ] **Step 5: Commit** `git commit -m "feat(gridx): custom-grid cell-identity ops (pointascell, cellaswkb, cellaswkt, centroid)"` - ---- - -## Task 4: Coverage ops — `polyfill`, `kring` (array-returning) - -**Files:** `gridx/custom/Custom_Polyfill.scala`, `Custom_KRing.scala` (new); test `Custom_CoverageTest.scala`. - -- [ ] **Step 1: Write the failing test** — `Custom_CoverageTest.scala`, same grid: - - `Custom_Polyfill(POLYGON((0 0,30 0,30 30,0 30,0 0)) WKB, grid, res=0)` → `ARRAY` of size 9 (centroid-containment: the 9 cells with centers at {5,15,25}×{5,15,25}). Assert size 9 and that each returned cell's `cellIdToGeometry` envelope lies within `[0,30]×[0,30]`. - - `Custom_KRing(centerCell, grid, k=1)` for the (1,1) res-0 cell → the 3×3 = 9 neighbourhood (or clipped at the grid edge); assert it contains the center and its 8 neighbours' ids. - -- [ ] **Step 2: Run, verify FAIL** (FOREGROUND, wait): suite `com.databricks.labs.gbx.gridx.custom.Custom_CoverageTest`. - -- [ ] **Step 3: Implement.** Mirror `BNG_Polyfill`/`BNG_KRing` (array-returning) for the result encoding (`ArrayData`/`GenericArrayData` of Long). - - **Custom_Polyfill**(geomExpr, gridExpr, resExpr): `dataType = ArrayType(LongType, false)`. eval: decode geom → `sys.polyfill(geom, res)` → `ArrayData.toArrayData(seq.toArray)` (mirror how BNG_Polyfill builds its array result). name `gbx_custom_polyfill`, 3-arg. Scaladoc: documents **centroid-containment** semantic (cell included iff its center is inside the geometry). - - **Custom_KRing**(cellExpr, gridExpr, kExpr): `dataType = ArrayType(LongType, false)`. eval: `sys.kRing(cell, asInt(k))` → array. name `gbx_custom_kring`, 3-arg. - -- [ ] **Step 4: Run, verify PASS** (FOREGROUND, wait). - -- [ ] **Step 5: Commit** `git commit -m "feat(gridx): custom-grid coverage ops (polyfill centroid-containment, kring)"` - ---- - -## Task 5: Register all + rebuild JAR - -**Files:** `gridx/custom/functions.scala` (new), GridX registration aggregator. - -- [ ] **Step 1: Write `gridx/custom/functions.scala`** mirroring `gridx/bng/functions.scala`: an object with `def register(spark: SparkSession): Unit` that builds a `RegistryDelegate` and `rd.register(Custom_Grid)`, `rd.register(Custom_PointAsCell)`, `rd.register(Custom_AsWKB)`, `rd.register(Custom_AsWKT)`, `rd.register(Custom_Centroid)`, `rd.register(Custom_Polyfill)`, `rd.register(Custom_KRing)`. Match BNG's RegistryDelegate construction (prefix handling — note BNG names already include `gbx_bng_`; here companions' `name` already include `gbx_custom_*`, so follow BNG's exact prefix convention). - -- [ ] **Step 2: Wire into GridX registration.** Find where GridX registers grids (the top-level gridx registration, or how `bng`/`quadbin` `register` are called) and add a call to `custom.functions.register(spark)`. Mirror exactly. - -- [ ] **Step 3: Rebuild** (FOREGROUND, wait): `gbx:docker:exec "mvn clean package -PskipScoverage -DskipTests"` → BUILD SUCCESS. - -- [ ] **Step 4: Commit** `git commit -m "feat(gridx): register gbx_custom_* functions"` - ---- - -## Task 6: registered_functions.txt + SQL examples + function-info - -- [ ] **Step 1:** Add to `docs/tests-function-info/registered_functions.txt`: `gbx_custom_grid`, `gbx_custom_pointascell`, `gbx_custom_cellaswkb`, `gbx_custom_cellaswkt`, `gbx_custom_centroid`, `gbx_custom_polyfill`, `gbx_custom_kring`. -- [ ] **Step 2:** Add a `*_sql_example()` + `_output` for each to `docs/tests/python/api/gridx_functions_sql.py` (mirror the `quadbin_*` example style; placeholder tables OK — display + structural validation). Show the grid-spec usage, e.g.: - `SELECT gbx_custom_pointascell(geom, gbx_custom_grid(0, 1000000, 0, 1000000, 2, 1000, 1000), 5) AS cell FROM points;` - Descriptions framed by utility (no Mosaic references). -- [ ] **Step 3: Regenerate** (FOREGROUND, wait): `gbx:docs:function-info`; confirm all 7 in `function-info.json`. -- [ ] **Step 4: Verify coverage** (FOREGROUND, wait): `gbx:test:function-info --log custom-fninfo.log` — `test_full_coverage_against_registered_list` passes; the DESCRIBE step also validates the 7 register cleanly. -- [ ] **Step 5: Commit** `git commit -m "docs: function-info examples for gbx_custom_* grid functions"` - ---- - -## Task 7: Python bindings + tests - -- [ ] **Step 1: Write failing tests** — `python/geobrix/test/gridx/custom/test_custom_grid.py` (mirror an existing gridx python test's session header). Build a grid via `gbx_custom_grid`, then: point→cell (assert a BIGINT), cell→wkb (assert binary), polyfill (assert array of cells), kring (assert array). Use the wrappers. -- [ ] **Step 2: Run, verify FAIL** (FOREGROUND, wait): `gbx:test:python --path python/geobrix/test/gridx/custom/test_custom_grid.py --log custom-py.log`. -- [ ] **Step 3: Add wrappers** in `python/geobrix/src/databricks/labs/gbx/gridx/custom/functions.py` (new; mirror the quadbin functions.py module + add to package exports as needed): - - `custom_grid(bound_x_min, bound_x_max, bound_y_min, bound_y_max, cell_splits, root_cell_size_x, root_cell_size_y, srid=None)` - - `custom_pointascell(geom, grid, resolution)`, `custom_cellaswkb(cell, grid)`, `custom_cellaswkt(cell, grid)`, `custom_centroid(cell, grid)`, `custom_polyfill(geom, grid, resolution)`, `custom_kring(cell, grid, k)` - Each `return f.call_function("gbx_custom_...", _col(...), ...)`; `custom_grid` defaults srid to `f.lit(-1)` when None. Docstrings utility-framed. -- [ ] **Step 4: Run, verify PASS** (FOREGROUND, wait). -- [ ] **Step 5: Commit** `git commit -m "feat(python): gbx_custom_* grid bindings + tests"` - ---- - -## Task 8: Full verification + push - -- [ ] **Step 1: binding-parity** — `bash scripts/commands/gbx-test-bindings.sh --log custom-parity.log` → all 7 present in Scala/Python/function-info (count 154). -- [ ] **Step 2: Scala** (FOREGROUND/bg, wait): `gbx:test:scala --suite 'com.databricks.labs.gbx.gridx.*'` → 0 failures. -- [ ] **Step 3: Python:** `gbx:test:python --path python/geobrix/test/gridx/` → pass. -- [ ] **Step 4: Lint:** `gbx:lint:scalastyle` (0 errors) AND `gbx:lint:python --check` (clean — isort/black/flake8). -- [ ] **Step 5: function-info coverage** → pass. -- [ ] **Step 6: Push** (`gh auth switch --user mjohns-databricks` first): `git push origin beta/0.4.0`. QC binding-parity gates the 7. -- [ ] **Step 7:** Update `docs/docs/limitations.mdx` — remove/flip the "Custom Gridding - Not fully ported" line (now ported). Commit + (it'll go in the push, or a follow-up commit). Run `grep -rn "wave" docs/docs/` style internals-leak check is N/A; just ensure the limitations edit is utility-framed. - ---- - -## Self-review notes (author) -- **Rationale:** utility-framed (bring-your-own grid in any CRS); no Mosaic-parity framing in plan/examples/docstrings/limitations. -- **Polyfill:** centroid-containment semantic shipped as-is (correct + standard), documented + tested; NOT rewritten to flood-fill (that's a different semantic, out of scope). -- **Coverage:** core uncomment+test (T1); struct + constructor (T2); 4 identity ops (T3); 2 coverage ops (T4); register (T5); function-info (T6); python (T7); verify incl. both lints + limitations-doc update (T8). -- **Type consistency:** all int args (resolution, k, splits, sizes, srid) read Int-or-Long tolerant via `Custom_GridSpec.asInt`; grid struct schema is the single source `Custom_GridSpec.gridStructType`; ops decode via `Custom_GridSpec.systemFromRow`. -- **Risk:** core math is pre-written/correct; main new surface is the struct-spec plumbing + op expressions (mirror BNG). The `gbx_custom_grid` struct-return expression is the least-templated piece — T2 builds + tests it first so later ops rely on a verified spec. diff --git a/docs/superpowers/plans/2026-05-29-docs-consolidation.md b/docs/superpowers/plans/2026-05-29-docs-consolidation.md deleted file mode 100644 index 515f10f7c..000000000 --- a/docs/superpowers/plans/2026-05-29-docs-consolidation.md +++ /dev/null @@ -1,119 +0,0 @@ -# Docs Consolidation + Function Backfill + QC Guards Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Steps use checkbox (`- [ ]`) syntax. - -**Goal:** Eliminate the Packages-vs-Functions duplication (it already drifted), make the "Functions" pages the single source of truth for function docs, backfill the 15 recently-shipped functions with representative outputs, refresh the rasterx diagram + release notes, and add QC guards so this can't silently rot again. - -**Why:** The `docs/docs/packages/*.mdx` pages list functions by category AND the `docs/docs/api/*-functions.mdx` pages list them per-function — two hand-maintained sources → guaranteed drift (neither has the 15 new functions; example outputs are placeholder `...`). Consolidating to one source per package + deterministic QC checks fixes the root cause. - -**Architecture:** Merge each `packages/.mdx` (concepts) into the top of `api/-functions.mdx` (reference), streamlining verbose prose; merge `packages/overview.mdx` into `api/overview.mdx`; delete `docs/docs/packages/`; rename the sidebar "API Reference" category to **"Functions"** and drop the "Packages" category. Then backfill the 15 new functions into the consolidated pages with real outputs, refresh the diagram + release notes, and add QC checks (validated against the single source) + fix the git pre-push hook so QC actually runs. - -**Tech Stack:** Docusaurus MDX (`CodeFromTest` component reads `*_sql_example`/`_output` from `docs/tests/python/api/*.py` via raw-loader), `docs/sidebars.js`, the QC judge (`/.claude/qc-judge/config.json` + checks). Docs build via `gbx:docs:static-build` (Docusaurus `onBrokenLinks` fails the build on dangling links — the link-fix gate). Function/SQL example pipeline in Docker via `gbx:*`. - -**Conventions:** Build/verify docs via `gbx:docs:static-build` (FOREGROUND, wait). `gh auth switch --user mjohns-databricks` before push. Run `gbx:lint:python --check` before pushing python (test/example) changes. ASCII-only. Frame all docs by utility (no "Mosaic-faithful" framing). Commit per task. - -**Decisions (locked):** section name = **Functions** (drop Packages); **migrate** package concepts but **streamline** verbose prose; **consolidation-first**. Plan-author decisions: diagram = update its hardcoded list + add a QC staleness check (not a full data-driven rewrite — efficient); release-notes QC = **deterministic** (new `gbx_*` names added to `registered_functions.txt` in the push range must appear in `beta-release-notes.mdx`); binary-returning functions show a descriptor output (`[GTiff tile]` / ``), scalar/array/WKT/struct show real values. - ---- - -## Phase A — Consolidate Packages → Functions - -Each `packages/.mdx` has conceptual content (Overview, Key Features, package-specific concepts) + a "Function Categories" listing + Usage Examples. The matching `api/-functions.mdx` has the per-function reference. Merge: concepts → top of the functions page (streamlined), keep the per-function reference, drop the duplicated category-listing where it just re-lists functions (keep category *headings* as section organization if useful). - -### Task A1: Merge `packages/rasterx.mdx` → `api/rasterx-functions.mdx` -**Files:** read both; edit `docs/docs/api/rasterx-functions.mdx`; (later-deleted) `docs/docs/packages/rasterx.mdx`. -- [ ] **Step 1: Read** both pages fully. Identify CONCEPTUAL content in `packages/rasterx.mdx` not already on the functions page: Overview, Key Features, Tile payload, **VRT Python pixel functions** (setup/trusted-modules — important, keep), and the category structure. Identify pure function-category *listings* that duplicate the functions page. -- [ ] **Step 2:** At the TOP of `api/rasterx-functions.mdx` (after the existing intro/setup), add a streamlined **Overview + Key Features + concepts** section migrated from the package page (Tile payload, VRT pixel functions). Streamline verbose prose; preserve all unique technical content + any `CodeFromTest` examples (carry their imports). Do NOT duplicate per-function reference that already exists below. -- [ ] **Step 3:** Verify the page still imports everything it references (raw-loader imports for any migrated `CodeFromTest`). Do not delete `packages/rasterx.mdx` yet (Task A6 deletes the dir). -- [ ] **Step 4: Commit** `git commit -m "docs(functions): merge RasterX package concepts into rasterx-functions"` - -### Task A2: Merge `packages/gridx.mdx` → `api/gridx-functions.mdx` -- [ ] Same pattern. Migrate GridX Overview, Key Features, **BNG Structure / BNG Grid Reference Format / Precision Levels**, Quadbin concepts → top of `api/gridx-functions.mdx`, streamlined. Preserve unique concepts; drop duplicated category listings. Commit `docs(functions): merge GridX package concepts into gridx-functions`. - -### Task A3: Merge `packages/vectorx.mdx` → `api/vectorx-functions.mdx` -- [ ] Migrate the VectorX overview + the `gbx_st_asmvt` / `gbx_st_asmvt_pyramid` narrative sections (the package page has fuller MVT examples than the functions page — reconcile: keep the best single version on the functions page). Commit `docs(functions): merge VectorX package concepts into vectorx-functions`. - -### Task A4: Merge `packages/pmtiles.mdx` → `api/pmtiles-functions.mdx` -- [ ] Migrate the PMTiles UDAF-vs-DataSource narrative, schema contract, tile-type detection, compression, serving, limits. Commit `docs(functions): merge PMTiles package concepts into pmtiles-functions`. - -### Task A5: Merge `packages/overview.mdx` → `api/overview.mdx` -- [ ] Migrate Available Packages, **Package Comparison**, "Choosing the Right Package", **Function Naming Convention** into `api/overview.mdx` (streamlined). Commit `docs(functions): merge packages overview into Functions overview`. - -### Task A6: Sidebar + delete Packages + fix internal links + build-verify -**Files:** `docs/sidebars.js`, delete `docs/docs/packages/`, link fixes across `docs/docs/**`. -- [ ] **Step 1:** In `docs/sidebars.js`: remove the entire `Packages` category block; rename the `label: 'API Reference'` category to `label: 'Functions'`. (Keep its items: overview, tile-structure, Function Reference subcategory, scala/python/sql.) -- [ ] **Step 2:** `git rm docs/docs/packages/*.mdx` (the whole dir). -- [ ] **Step 3:** Fix internal links in `docs/docs/**` that point to `packages/` (markdown links like `(../packages/rasterx)`, `(/geobrix/docs/packages/...)`, `(./packages/...)`) → repoint to the corresponding `api/-functions` (or `api/overview` for the packages overview). Grep `docs/docs/` for `packages/` link targets; update each. (Scope to source `.mdx`; ignore any `docs/build/`.) -- [ ] **Step 4: Build-verify** (FOREGROUND, wait): `gbx:docs:static-build` (i.e. `bash scripts/commands/gbx-docs-static-build.sh`). Docusaurus `onBrokenLinks` FAILS the build on any dangling `/packages/...` link — fix every reported broken link until the build is GREEN. This is the definitive link gate. -- [ ] **Step 5: Commit** `git commit -m "docs: retire Packages section, fold into Functions; fix links; sidebar rename"` - ---- - -## Phase B — Backfill the 15 new functions + representative outputs - -For each new function: add an MDX reference section to its consolidated Functions page (mirror the existing per-function section format on that page: heading `## ` or `### gbx_(...)`, description, params, returns, ``), AND set a representative `*_sql_example_output` in the example file. Outputs: real values for scalar/array/WKT/struct; `[GTiff tile, 1 band]`-style descriptor for raster tiles; `` descriptor for binary geometry. Fix `st_triangulate`'s bare-string `_output`. - -### Task B1: RasterX new functions → `api/rasterx-functions.mdx` -- [ ] Add sections for `gbx_rst_dtmfromgeoms`, `gbx_rst_dtmfromgeoms_agg`, `gbx_rst_rasterize_agg`, `gbx_rst_frombands_agg` (params/returns/example via CodeFromTest). Set representative `_output` for each in `docs/tests/python/api/rasterx_functions_sql.py` (tiles → `+----+\n|dtm |\n+----+\n|[GTiff tile, 1 band]|\n...` descriptor, not bare `...`). Commit `docs(functions): document rst_dtmfromgeoms(+agg), rst_rasterize_agg, rst_frombands_agg`. - -### Task B2: GridX new functions → `api/gridx-functions.mdx` -- [ ] Add sections for `gbx_custom_grid`, `gbx_custom_pointascell`, `gbx_custom_cellaswkb`, `gbx_custom_cellaswkt`, `gbx_custom_centroid`, `gbx_custom_polyfill`, `gbx_custom_kring`, `gbx_quadbin_cellunion_agg`. Representative `_output` in `gridx_functions_sql.py`: `custom_pointascell`→a real cell-id integer; `custom_cellaswkt`→a real `POLYGON ((...))`; `custom_polyfill`/`custom_kring`→a real `[id, id, ...]` array; `custom_grid`→the struct values; `custom_cellaswkb`/`custom_centroid`/`quadbin_cellunion_agg`→`` descriptor. Commit `docs(functions): document gbx_custom_* + quadbin_cellunion_agg`. - -### Task B3: VectorX new functions → `api/vectorx-functions.mdx` -- [ ] Add sections for `gbx_st_triangulate`, `gbx_st_interpolateelevationbbox`, `gbx_st_interpolateelevationgeom`. Representative `_output` in `vectorx_functions_sql.py`: these emit rows of WKB geometries (generators) → `` / `` descriptor (fix `st_triangulate`'s bare `triangle`). Commit `docs(functions): document st_triangulate + st_interpolateelevation{bbox,geom}`. - -### Task B4: Regenerate function-info + verify outputs render -- [ ] Run `gbx:docs:function-info` (FOREGROUND, wait) to resync function-info.json with any example edits; `gbx:test:function-info` passes; `gbx:docs:static-build` GREEN (the new sections render, no MDX errors). Commit any regenerated `function-info.json`. `docs(functions): regenerate function-info after backfill`. - ---- - -## Phase C — Diagram + release notes - -### Task C1: Refresh RasterX function-categories diagram -**Files:** `resources/images/generators/rasterx-function-categories.py`, regenerated PNG. -- [ ] **Step 1:** Update the script's hardcoded `CARDS_LEFT`/`CARDS_RIGHT` function lists to include the 42 missing rst_ functions (categorize sensibly into existing/added cards) and fix the hardcoded count string (`"65 SQL functions"` → the current count). Keep ASCII. -- [ ] **Step 2:** Regenerate per the script docstring: `python3 resources/images/generators/rasterx-function-categories.py` then the Chrome-headless screenshot to `resources/images/diagrams/rasterx/rasterx-function-categories.png`. (Verify the PNG referenced by `docs/docs/api/rasterx-functions.mdx` updates.) -- [ ] **Step 3:** Build-verify the image renders. Commit `docs(images): refresh rasterx function-categories diagram for current function set`. - -### Task C2: Update beta release notes -- [ ] Add the new functions to `docs/docs/beta-release-notes.mdx` (v0.4.0 section): a concise entry per capability group — DTM-from-geoms (raster + agg), streaming aggregators (quadbin_cellunion_agg, rst_rasterize_agg, rst_frombands_agg), VectorX TIN (st_triangulate, st_interpolateelevation{bbox,geom}), custom grid (gbx_custom_*). Utility-framed, no Mosaic references. Commit `docs(release-notes): note dtmfromgeoms, streaming aggregators, TIN functions, custom grid`. - ---- - -## Phase D — QC guards + hook fix - -Each QC check: add to `/.claude/qc-judge/config.json` (project config), `command` type, with a backing deterministic script in the repo where logic is non-trivial (like `binding-parity` → `docs/scripts/check-binding-parity.py`). SELF-TEST each (inject a deliberate failure, confirm exit 1, restore). - -### Task D1: Q0 — make QC run on terminal pushes (hook fix) -- [ ] The geobrix repo's local `core.hooksPath=.git/hooks` (git-lfs pre-push) overrides the global QC chained hook, so terminal `git push` skips QC. Fix by chaining QC into the existing `.git/hooks/pre-push` (append `~/.claude/qc-judge/qc.py --git-pre-push` AFTER the git-lfs invocation, preserving git-lfs), so both run. This is a LOCAL `.git/hooks` change (not committed). Verify with a dry `git push --dry-run`-style or a no-op push that QC fires. Report the change (no commit — `.git/hooks` is not version-controlled). If chaining is fragile, document the exact manual step for the user instead. - -### Task D2: Q1 — every registered function has a Functions-page section -**Files:** `docs/scripts/check-doc-coverage.py` (new), `/.claude/qc-judge/config.json`. -- [ ] **Step 1:** Write `docs/scripts/check-doc-coverage.py` (stdlib): for each `gbx_*` name in `registered_functions.txt`, verify it (or its bare `` / `*_sql_example` constant) appears as a documented section in the matching `docs/docs/api/-functions.mdx` (map prefix → page: `gbx_rst_*`/`gbx_custom_*`→ which page; `gbx_bng_*`/`gbx_quadbin_*`/`gbx_custom_*`→gridx; `gbx_st_*`→vectorx; `gbx_pmtiles_*`→pmtiles). Detection: the function name appears in the page text OR its `outputConstant`/`functionName` is referenced. Exit 1 listing undocumented functions. Negative-test it. -- [ ] **Step 2:** Add a `doc-coverage` command check to the project qc config (`cmd: "[ -f docs/scripts/check-doc-coverage.py ] || exit 0; python3 docs/scripts/check-doc-coverage.py"`, expect_exit 0, severity warn). Confirm it PASSES now (after Phase B). Add a `gbx:test:doc-coverage` command wrapper (optional, mirror `gbx:test:bindings`). -- [ ] **Step 3: Commit** `feat(qc): doc-coverage check — every registered function documented on its Functions page`. - -### Task D3: Q2 — flag placeholder-only example outputs -- [ ] Add to `check-doc-coverage.py` (or a sibling) a check that each registered function's `*_sql_example_output` in `docs/tests/python/api/*.py` is NOT placeholder-only (a table whose only data row is `...`/empty, or a bare non-table string). Allow the binary descriptor convention (`[GTiff tile...]`, ``). Wire into the same/related qc check. Negative-test. Commit `feat(qc): flag placeholder-only SQL example outputs`. - -### Task D4: Q3 — rasterx diagram staleness -- [ ] Add `docs/scripts/check-diagram-coverage.py` (or extend): parse the function names listed in `resources/images/generators/rasterx-function-categories.py` and verify they cover all `gbx_rst_*` in `registered_functions.txt` (and the count string matches). Exit 1 on drift. Add a `diagram-coverage` qc command check. Negative-test. Commit `feat(qc): rasterx diagram coverage check`. - -### Task D5: Q4 — reliable release-notes check (deterministic) -- [ ] Replace/augment the project's `release-notes-current`: add a `release-notes-functions` command check — for each `gbx_*` name ADDED to `registered_functions.txt` within `$QC_RANGE` (`git diff $QC_RANGE -- docs/tests-function-info/registered_functions.txt | grep '^+gbx_'`), verify it appears in `docs/docs/beta-release-notes.mdx`; exit 1 listing unmentioned new functions. Deterministic (no LLM timeout/leniency). Add to qc config; in the project config, disable the flaky LLM `release-notes-current` (`{"enabled": false}`) in favor of this. Negative-test. Commit `feat(qc): deterministic release-notes-functions check; disable flaky LLM release-notes check`. - ---- - -## Phase E — Full verification + push - -- [ ] **Step 1: docs build** — `gbx:docs:static-build` GREEN (no broken links, all sections render). -- [ ] **Step 2: QC self-run** — run each new check's cmd from repo root; all exit 0 on the current tree (doc-coverage, placeholder-output, diagram-coverage, release-notes-functions). Confirm via the qc merge (like binding-parity verification) that they're registered + PASS. -- [ ] **Step 3:** `gbx:test:function-info` pass; `bash scripts/commands/gbx-test-bindings.sh` pass (parity unaffected); `gbx:lint:python --check` clean (example-file edits). -- [ ] **Step 4: Push** (`gh auth switch --user mjohns-databricks`): `git push origin beta/0.4.0`. With the hook fix (D1), QC runs; the new checks gate. Address findings. - ---- - -## Self-review notes (author) -- **Decisions honored:** section renamed Functions, Packages dropped; concepts migrated + streamlined; consolidation-first; diagram hardcoded-list-update + QC check (not full rewrite); deterministic release-notes check; binary-output descriptor convention. -- **Coverage:** consolidation (A) → backfill 15 funcs + real outputs (B) → diagram + release notes (C) → 4 QC checks + hook fix (D) → verify+push (E). The doc-coverage check (Q1) is the durable guard that would have caught the original gap; it's validated to PASS only AFTER Phase B backfills the 15. -- **Risk:** A6 link-fixing is broad — Docusaurus `onBrokenLinks` build failure is the gate (fix until green). Content migration is judgment-heavy (streamline without losing unique concepts) — per-package tasks let a subagent hold one page-pair in context. Diagram regen needs Chrome-headless (local, macOS) — if unavailable in the agent env, regenerate the SVG + report the manual screenshot step. diff --git a/docs/superpowers/plans/2026-05-29-st-triangulate-interpolateelevation.md b/docs/superpowers/plans/2026-05-29-st-triangulate-interpolateelevation.md deleted file mode 100644 index 138f1427c..000000000 --- a/docs/superpowers/plans/2026-05-29-st-triangulate-interpolateelevation.md +++ /dev/null @@ -1,213 +0,0 @@ -# VectorX TIN functions: st_triangulate + st_interpolateelevation{bbox,geom} - -> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Steps use checkbox (`- [ ]`) syntax. - -**Goal:** Add three VectorX geometry-generator functions that expose the constrained-Delaunay TIN pipeline (already used internally by `gbx_rst_dtmfromgeoms`) as first-class geometry output: -- `gbx_st_triangulate` — emits the TIN triangles as polygons. -- `gbx_st_interpolateelevationbbox` — interpolates Z onto a bbox+pixels grid, emits Z-valued points. -- `gbx_st_interpolateelevationgeom` — same, but the grid is given as an origin point + cell counts + cell sizes. - -**Why these (utility, not parity):** -- **st_triangulate** — the triangulated irregular network is currently locked inside the raster DTM path. Exposing the triangles as geometries lets users *inspect/visualize the mesh*, validate that breaklines were honored, and feed downstream mesh/contour/QC workflows. Useful on its own. -- **st_interpolateelevationbbox** — interpolating onto an extent+pixel grid returns elevation as **vector points** you can join, aggregate, or grid-index — and the bbox+pixels parameterization is **consistent with the rest of GeoBrix's grid functions** (`rst_dtmfromgeoms`, `rst_gridfrompoints`, `rst_rasterize`), so the same grid composes pixel-aligned across vector and raster. -- **st_interpolateelevationgeom** — lets users define the grid the way terrain practitioners think: an **origin corner + explicit cell size** ("10 m cells starting here"), instead of computing pixel counts from an extent. Resolution-first ergonomics; a distinct, genuinely useful convenience that coexists with the extent-first bbox form. Kept as a **separate clearly-named function** (not an overloaded signature) so each call site is unambiguous. -- **split_point_finder** — tunes how the conforming-Delaunay triangulation handles constraint (breakline) encroachment (`MIDPOINT` vs `NONENCROACHING`), trading triangle quality against constraint fidelity. A real quality knob for breakline-heavy terrain; the underlying builder already supports it (`JTSConformingDelaunayTriangulationBuilder.setSplitPointFinder`), it just isn't wired through yet. - -**Architecture:** The pure-JTS TIN math (`triangulate`, `interpolate`, `postProcessTriangulation`, grid helpers) is GDAL-free and conceptually pure geometry, so it moves from `rasterx.operations.InterpolateElevation` to `vectorx.jts` (the three new VectorX functions and the existing `rst_dtmfromgeoms` both consume it; `rasterx` already depends on `vectorx.jts`, so this removes a would-be `vectorx→rasterx` cycle). `split_point_finder` is threaded through as an optional param (default = current behavior, so `rst_dtmfromgeoms` is unchanged). The three functions are `CollectionGenerator` expressions (one input row → many geometry rows), mirroring `vectorx/expressions/ST_AsMvtPyramid`. - -**Tech Stack:** Scala 2.13 / Spark 4.0 Catalyst `CollectionGenerator`, JTS (`ConformingDelaunayTriangulationBuilder`, `Triangle.interpolateZ`), PySpark `call_function`. Builds/tests run in the `geobrix-dev` Docker container via `gbx:*`. - -**Conventions:** Run Scala/Python tests via `gbx:*` IN THE FOREGROUND, wait for `BUILD SUCCESS/FAILURE` + `Tests: succeeded N`. Never host `mvn`. Rebuild JAR after Scala changes before Python tests. ASCII-only source. `gh auth switch --user mjohns-databricks` before push. Encode/decode geometries with `JTS.fromWKB`/`fromWKT` and `JTS.toWKB3` (Z-preserving — `JTS.toWKB` strips Z). PySpark sends Python ints as `Long` → readers for int args must accept Int **or** Long. - -**Implementation reference:** the constrained-Delaunay + barycentric-Z algorithm already lives in our `InterpolateElevation` (`triangulate`/`interpolate`); the new work is mostly *exposing* it via generator expressions, not new algorithm code. - ---- - -## File Structure - -| File | Responsibility | -|---|---| -| `src/main/scala/.../vectorx/jts/InterpolateElevation.scala` (MOVED from `rasterx/operations/`) | Pure-JTS TIN math; `triangulate`/`interpolate` gain optional `splitPointFinder`; add `pointGridOrigin`. | -| `src/main/scala/.../rasterx/expressions/RST_DTMFromGeoms.scala` (edit imports) | Now imports `InterpolateElevation` from `vectorx.jts`. | -| `src/test/scala/.../vectorx/jts/InterpolateElevationTest.scala` (MOVED) | Follows the object. | -| `src/main/scala/.../vectorx/expressions/ST_Triangulate.scala` (new) | Generator → triangle polygons. | -| `src/main/scala/.../vectorx/expressions/ST_InterpolateElevationBBox.scala` (new) | Generator → Z-points, bbox+pixels grid. | -| `src/main/scala/.../vectorx/expressions/ST_InterpolateElevationGeom.scala` (new) | Generator → Z-points, origin+cell-size grid. | -| `src/main/scala/.../vectorx/functions.scala` (edit) | Register the three. | -| `docs/tests-function-info/registered_functions.txt` | Add 3 names. | -| `docs/tests/python/api/vectorx_functions_sql.py` | 3 `*_sql_example()`. | -| `src/main/resources/.../function-info.json` | Regenerated. | -| `python/.../vectorx/functions.py` | 3 wrappers. | -| `src/test/scala/.../vectorx/expressions/ST_*Test.scala` (new) | per-function tests. | -| `python/geobrix/test/vectorx/test_tin_functions.py` (new) | binding smoke tests. | - ---- - -## Task 1: Move TIN math to `vectorx.jts` + thread optional `split_point_finder` - -**Files:** move `src/main/scala/com/databricks/labs/gbx/rasterx/operations/InterpolateElevation.scala` → `src/main/scala/com/databricks/labs/gbx/vectorx/jts/InterpolateElevation.scala` (package `com.databricks.labs.gbx.vectorx.jts`); move its test similarly; edit `RST_DTMFromGeoms.scala` import. - -- [ ] **Step 1: Read** the current `InterpolateElevation.scala` (rasterx/operations), `RST_DTMFromGeoms.scala` (its `import ...operations.InterpolateElevation` + call sites: `triangulate`, `pointGridBBox`, `interpolate`), and `vectorx/jts/JTSConformingDelaunayTriangulationBuilder.scala` (the `setSplitPointFinder(TriangulationSplitPointTypeEnum.Value)` + `TriangulationSplitPointTypeEnum.fromString` API). Grep for any other references to `rasterx.operations.InterpolateElevation`. - -- [ ] **Step 2: Move + repackage.** Move the file to `vectorx/jts/InterpolateElevation.scala`, change its `package` to `com.databricks.labs.gbx.vectorx.jts`. It already imports `JTS` and `JTSConformingDelaunayTriangulationBuilder` from this package (now same-package). Move the test `InterpolateElevationTest.scala` to `src/test/scala/com/databricks/labs/gbx/vectorx/jts/` and update its `package`. - -- [ ] **Step 3: Thread optional `splitPointFinder`** (behavior-preserving). Change: -```scala -def triangulate(multiPoint: Geometry, breaklines: Seq[Geometry], - mergeTolerance: Double, snapTolerance: Double, - splitPointFinder: Option[TriangulationSplitPointTypeEnum.Value] = None): Seq[Geometry] = { - ... - val triangulator = JTSConformingDelaunayTriangulationBuilder(multiPoint) - if (breaklines.nonEmpty) triangulator.setConstraints(multiLineString) - triangulator.setTolerance(mergeTolerance) - splitPointFinder.foreach(triangulator.setSplitPointFinder) // only set when provided - ... -} -``` -and forward it through `interpolate`: -```scala -def interpolate(multipoint: MultiPoint, breaklines: Seq[LineString], gridPoints: MultiPoint, - mergeTolerance: Double, snapTolerance: Double, - splitPointFinder: Option[TriangulationSplitPointTypeEnum.Value] = None): Seq[Point] = { - val triangles = triangulate(multipoint, breaklines, mergeTolerance, snapTolerance, splitPointFinder) - ... -} -``` -The `= None` defaults mean `RST_DTMFromGeoms`'s existing 4-arg calls compile unchanged and behave identically (no `setSplitPointFinder` call). Import `TriangulationSplitPointTypeEnum` (same package now). - -- [ ] **Step 4: Add `pointGridOrigin`** (for the geom-form function in Task 4): -```scala -/** Grid of cell-center points from an origin corner + cell counts + per-cell sizes. - * Centers: x = originX + (i + 0.5)*cellSizeX, y = originY + (j + 0.5)*cellSizeY. - * cellSizeY is typically negative (y-down). Column-major (x slowest, y fastest). - */ -def pointGridOrigin(originX: Double, originY: Double, cols: Int, rows: Int, - cellSizeX: Double, cellSizeY: Double, srid: Int): MultiPoint = { - val pts = for (i <- 0 until cols; j <- 0 until rows) yield { - val p = JTS.point(new Coordinate(originX + (i + 0.5) * cellSizeX, originY + (j + 0.5) * cellSizeY)) - p.setSRID(srid); p - } - val mp = JTS.multiPoint(pts.toArray); mp.setSRID(srid); mp -} -``` - -- [ ] **Step 5: Update `RST_DTMFromGeoms.scala`** import from `...rasterx.operations.InterpolateElevation` to `...vectorx.jts.InterpolateElevation`. Fix any other references found in Step 1. - -- [ ] **Step 6: Verify no regression** (FOREGROUND, wait): run BOTH the moved unit test and the dtmfromgeoms suite: -``` -gbx:test:scala --suites 'com.databricks.labs.gbx.vectorx.jts.InterpolateElevationTest,com.databricks.labs.gbx.rasterx.expressions.RST_DTMFromGeomsTest' --log tin-move.log -``` -Expect all pass (InterpolateElevation tests + the 8 dtmfromgeoms tests). This proves the move + default-param threading didn't change dtmfromgeoms behavior. - -- [ ] **Step 7: Commit** `git commit -m "refactor(vectorx): move TIN math to vectorx.jts; optional split_point_finder; add pointGridOrigin"` - ---- - -## Task 2: `gbx_st_triangulate` - -**Files:** `src/main/scala/.../vectorx/expressions/ST_Triangulate.scala` (new); test `src/test/scala/.../vectorx/expressions/ST_TriangulateTest.scala` (new). - -**Utility:** exposes the TIN triangles as polygons so users can inspect/validate/visualize the mesh. - -- [ ] **Step 1: Read** `src/main/scala/com/databricks/labs/gbx/vectorx/expressions/ST_AsMvtPyramid.scala` (the `CollectionGenerator with CodegenFallback` pattern: `elementSchema`, `eval(input): IterableOnce[InternalRow]`, `children`, `withNewChildrenInternal`, companion `name`/`builder`), and how a VectorX expression is registered in `vectorx/functions.scala` + how geometries are decoded/encoded (`JTS.fromWKB`/`fromWKT`, `JTS.toWKB`). Read `InterpolateElevation.triangulate` (now in vectorx.jts) and `TriangulationSplitPointTypeEnum.fromString`. - -- [ ] **Step 2: Write the failing test.** `ST_TriangulateTest.scala` (AnyFunSuite + Matchers): build 4 corner points of a square (e.g. (0,0),(10,0),(0,10),(10,10)) as a geometry array, empty breaklines; construct `ST_Triangulate(...)` with `Literal` children; call `.eval(InternalRow)` and assert it yields **2 triangle rows** (Delaunay of a square = 2 triangles), each a valid Polygon WKB (parse via `JTS.fromWKB`, assert `.getNumPoints == 4` ring / `isValid`). Add a case with a breakline asserting it still triangulates (count > 0). - -- [ ] **Step 3: Run, verify FAIL** (FOREGROUND, wait): `gbx:test:scala --suite 'com.databricks.labs.gbx.vectorx.expressions.ST_TriangulateTest' --log st-triangulate.log`. - -- [ ] **Step 4: Implement `ST_Triangulate`.** `CollectionGenerator with CodegenFallback`, 5 children `(pointsArray, breaklinesArray, mergeTolerance, snapTolerance, splitPointFinder)`. `elementSchema = StructType(StructField("triangle", BinaryType) :: Nil)`. `eval`: decode arrays (WKB/WKT) to JTS geoms (mirror `RST_DTMFromGeoms.geomsFromArrayData`), build `JTS.multiPoint(points)`, parse `splitPointFinder` String via `TriangulationSplitPointTypeEnum.fromString`, call `InterpolateElevation.triangulate(mp, lines.map(_.asInstanceOf[LineString]), mergeTol, snapTol, Some(finder))`, map each triangle polygon to `InternalRow(JTS.toWKB(poly))` (triangles are 2D rings — `toWKB` is correct here; no Z needed). Companion `name = "gbx_st_triangulate"`, builder requiring 5 args. Register-ready (registration in Task 5). - -- [ ] **Step 5: Run, verify PASS** (FOREGROUND, wait). - -- [ ] **Step 6: Commit** `git commit -m "feat(vectorx): gbx_st_triangulate generator (TIN triangles as polygons)"` - ---- - -## Task 3: `gbx_st_interpolateelevationbbox` - -**Files:** `.../vectorx/expressions/ST_InterpolateElevationBBox.scala` (new); test (new). - -**Utility:** Z interpolation onto an extent+pixel grid, returned as vector points; bbox+pixels parameterization composes with the rest of GeoBrix's grid functions. - -- [ ] **Step 1: Read** `InterpolateElevation.{pointGridBBox, interpolate}` and the `ST_Triangulate` you just wrote (for the generator + decode pattern). Note PySpark Long handling for `width_px`/`height_px`/`srid`. - -- [ ] **Step 2: Write the failing test.** Known tilted plane `z = 2x + 3y + 5` at 4 corners of a 100×100 extent; grid 10×10 over (0,0)-(100,100); assert the generator yields Z-valued points whose Z equals `2x+3y+5` (within 1e-6) at each emitted point's (x,y); assert count = number of in-hull cells (100 for a fully-covered square). Construct via `Literal` children, `.eval(InternalRow)`, collect rows, parse each WKB point, check Z. - -- [ ] **Step 3: Run, verify FAIL** (FOREGROUND, wait): suite `com.databricks.labs.gbx.vectorx.expressions.ST_InterpolateElevationBBoxTest`. - -- [ ] **Step 4: Implement.** `CollectionGenerator`, 12 children `(points, breaklines, mergeTol, snapTol, splitPointFinder, xmin, ymin, xmax, ymax, widthPx, heightPx, srid)`. Read `widthPx/heightPx/srid` Int-or-Long tolerant (mirror `RST_DTMFromGeomsAgg.evalInt` style, but these are direct children so eval against the input row). `eval`: decode points/lines, `grid = InterpolateElevation.pointGridBBox(xmin,ymin,xmax,ymax,widthPx,heightPx,srid)`, `pts = InterpolateElevation.interpolate(mp, lines, grid, mergeTol, snapTol, Some(finder))`, emit `InternalRow(JTS.toWKB3(p))` per point (**toWKB3** — Z must be preserved). `elementSchema = StructType(StructField("elevation_point", BinaryType) :: Nil)`. Companion `name = "gbx_st_interpolateelevationbbox"`, builder requiring 12 args. - -- [ ] **Step 5: Run, verify PASS** (FOREGROUND, wait). - -- [ ] **Step 6: Commit** `git commit -m "feat(vectorx): gbx_st_interpolateelevationbbox generator (bbox+pixels grid)"` - ---- - -## Task 4: `gbx_st_interpolateelevationgeom` - -**Files:** `.../vectorx/expressions/ST_InterpolateElevationGeom.scala` (new); test (new). - -**Utility:** define the grid by origin corner + cell counts + cell sizes (resolution-first), the natural way to ask for "N-metre cells starting here." - -- [ ] **Step 1: Read** `InterpolateElevation.pointGridOrigin` (added in Task 1) and the `ST_InterpolateElevationBBox` you just wrote. - -- [ ] **Step 2: Write the failing test.** Same plane. Pick an origin + cell sizes that yield the SAME grid as a bbox case (e.g. origin (0,0), cols=10, rows=10, cell_size_x=10.0, cell_size_y=10.0 → centers at 5,15,...,95 — matching `pointGridBBox(0,0,100,100,10,10)`). Assert the emitted Z-points match `z=2x+3y+5` at their (x,y). Add an **equivalence assertion**: the set of (x,y,z) emitted by geom-form equals the set emitted by `ST_InterpolateElevationBBox` over the equivalent extent (sort both, compare) — proving the two functions are consistent. (Use positive cell_size_y here with origin at min-corner so centers match pointGridBBox; document that negative cell_size_y is y-down.) - -- [ ] **Step 3: Run, verify FAIL** (FOREGROUND, wait): suite `...ST_InterpolateElevationGeomTest`. - -- [ ] **Step 4: Implement.** `CollectionGenerator`, 10 children `(points, breaklines, mergeTol, snapTol, splitPointFinder, gridOrigin, gridCols, gridRows, cellSizeX, cellSizeY)`. `eval`: decode points/lines; decode `gridOrigin` geometry (WKB/WKT) → a JTS Point; `originX = origin.getX`, `originY = origin.getY`, `srid = origin.getSRID` (if 0, that's acceptable — document that origin should carry SRID); `gridCols/gridRows` Int-or-Long tolerant; `grid = InterpolateElevation.pointGridOrigin(originX, originY, cols, rows, cellSizeX, cellSizeY, srid)`; `pts = InterpolateElevation.interpolate(mp, lines, grid, mergeTol, snapTol, Some(finder))`; emit `InternalRow(JTS.toWKB3(p))`. `elementSchema = StructType(StructField("elevation_point", BinaryType) :: Nil)`. Companion `name = "gbx_st_interpolateelevationgeom"`, builder requiring 10 args. - -- [ ] **Step 5: Run, verify PASS** (FOREGROUND, wait) — including the bbox/geom equivalence assertion. - -- [ ] **Step 6: Commit** `git commit -m "feat(vectorx): gbx_st_interpolateelevationgeom generator (origin+cell-size grid)"` - ---- - -## Task 5: Register all three + rebuild JAR - -- [ ] **Step 1:** In `src/main/scala/com/databricks/labs/gbx/vectorx/functions.scala`, add `rd.register(ST_Triangulate)`, `rd.register(ST_InterpolateElevationBBox)`, `rd.register(ST_InterpolateElevationGeom)` near the other `ST_*` registrations; add imports if the file imports expressions individually (mirror existing style; check how `ST_AsMvtPyramid` is imported/registered). -- [ ] **Step 2: Rebuild** (FOREGROUND, wait): `gbx:docker:exec "mvn clean package -PskipScoverage -DskipTests"` → BUILD SUCCESS. -- [ ] **Step 3: Commit** `git commit -m "feat(vectorx): register st_triangulate + st_interpolateelevation{bbox,geom}"` - ---- - -## Task 6: registered_functions.txt + SQL examples + function-info - -- [ ] **Step 1:** Add `gbx_st_triangulate`, `gbx_st_interpolateelevationbbox`, `gbx_st_interpolateelevationgeom` to `docs/tests-function-info/registered_functions.txt`. -- [ ] **Step 2:** Add a `*_sql_example()` + `_output` for each to `docs/tests/python/api/vectorx_functions_sql.py` (find it; mirror existing `st_*` example style — placeholder tables OK, display+structural-validation only). Examples should show the streaming/generator usage (`SELECT gbx_st_triangulate(masspoints, breaklines, 0.01, 0.01, 'NONENCROACHING') FROM survey` etc.). Use clear inline values; for the geom form show `ST_Point(...)` origin + cell sizes. -- [ ] **Step 3: Regenerate** (FOREGROUND, wait): `gbx:docs:function-info`; confirm all three in `function-info.json`. -- [ ] **Step 4: Verify coverage** (FOREGROUND, wait): `gbx:test:function-info --log tin-fninfo.log` — `test_full_coverage_against_registered_list` passes (pre-existing `databricks`-module errors, if any, are baseline noise — confirm no NEW failure for the three). -- [ ] **Step 5: Commit** `git commit -m "docs: function-info examples for st_triangulate + st_interpolateelevation{bbox,geom}"` - ---- - -## Task 7: Python bindings + tests - -- [ ] **Step 1: Write failing tests** mirroring an existing vectorx python test's session header. `python/geobrix/test/vectorx/test_tin_functions.py`: for each function, build a small DataFrame of Z-valued point WKT/WKB (a square + corners), `select`/`lateral`-explode the generator, assert non-empty rows of geometry. (For generators in PySpark, the call returns multiple rows — use the generator in a `select` and `.collect()`; confirm how existing generator bindings like `st_asmvt_pyramid` are tested.) -- [ ] **Step 2: Run, verify FAIL** (FOREGROUND, wait): `gbx:test:python --path python/geobrix/test/vectorx/test_tin_functions.py --log tin-py.log`. -- [ ] **Step 3: Add wrappers** to `python/geobrix/src/databricks/labs/gbx/vectorx/functions.py`: - - `st_triangulate(points_geom, breaklines_geom, merge_tolerance, snap_tolerance, split_point_finder)` - - `st_interpolateelevationbbox(points_geom, breaklines_geom, merge_tolerance, snap_tolerance, split_point_finder, xmin, ymin, xmax, ymax, width_px, height_px, srid)` - - `st_interpolateelevationgeom(points_geom, breaklines_geom, merge_tolerance, snap_tolerance, split_point_finder, grid_origin, grid_cols, grid_rows, cell_size_x, cell_size_y)` - Each `return f.call_function("gbx_...", _col(...), ...)`. Match the existing vectorx wrapper style + docstrings (utility-framed, no Mosaic references). -- [ ] **Step 4: Run, verify PASS** (FOREGROUND, wait). -- [ ] **Step 5: Commit** `git commit -m "feat(python): bindings + tests for st_triangulate + st_interpolateelevation{bbox,geom}"` - ---- - -## Task 8: Full verification + push - -- [ ] **Step 1: binding-parity** — `bash scripts/commands/gbx-test-bindings.sh --log tin-parity.log` → all three present in Scala/Python/function-info; parity green (count 147). -- [ ] **Step 2: Scala suites** (FOREGROUND/bg, wait): `gbx:test:scala --suites 'com.databricks.labs.gbx.vectorx.*,com.databricks.labs.gbx.rasterx.*'` → 0 failures (rasterx included because the TIN math moved — confirms dtmfromgeoms still green). -- [ ] **Step 3: Python suites:** `gbx:test:python --path python/geobrix/test/vectorx/` and `--path python/geobrix/test/rasterx/` → pass. -- [ ] **Step 4: scalastyle:** `gbx:lint:scalastyle` → 0 errors (ASCII-only). -- [ ] **Step 5: function-info coverage** → pass. -- [ ] **Step 6: Push** (`gh auth switch --user mjohns-databricks` first): `git push origin beta/0.4.0`. QC `binding-parity` gates the three. - ---- - -## Self-review notes (author) -- **Rationale framing:** every function justified by user utility; no "Mosaic-faithful"/parity framing in plan, examples, docstrings, or function-info. -- **Coverage:** TIN extraction + split_point_finder threading (T1, behavior-preserving for dtmfromgeoms, re-verified); three generators (T2-4) with per-function tests incl. the bbox/geom **equivalence** test; registration (T5); function-info (T6); Python (T7); full verification incl. rasterx regression since TIN moved (T8). -- **Type consistency:** `InterpolateElevation.triangulate`/`interpolate` gain `splitPointFinder: Option[...] = None` (dtmfromgeoms calls unchanged); generators pass `Some(fromString(...))`; `toWKB3` used for Z-points, `toWKB` for triangle polygons; Int/Long tolerance on count/srid args. -- **Risk:** T1 moves shipped code — mitigated by re-running the dtmfromgeoms suite in T1 Step 6 and the rasterx suite in T8. diff --git a/docs/superpowers/plans/2026-06-11-light-pmtiles-writer.md b/docs/superpowers/plans/2026-06-11-light-pmtiles-writer.md deleted file mode 100644 index a4c540e0a..000000000 --- a/docs/superpowers/plans/2026-06-11-light-pmtiles-writer.md +++ /dev/null @@ -1,2007 +0,0 @@ -# Light Tiled-Output Framework + PMTiles Writer Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Ship a pure-Python/PySpark DataSource V2 PMTiles writer (`pmtiles_gbx`) on a new, shared, tier-neutral tiled-output framework at `databricks.labs.gbx.ds.tiles`, with distributed spatial sharding (fixed + adaptive), a separate overview archive, and a GeoJSON/STAC catalog. - -**Architecture:** A new tier-neutral package `databricks.labs.gbx.ds` (migrated from `pyrx/ds`) hosts the framework: `grid.py` (SlippyGrid tile math), `_header.py` (PMTiles header/sniff), `backend.py` (PMTilesBackend assembly), `catalog.py` (STAC/TileJSON), `shard.py` (entries-driven scratch + shard assignment). The `pmtiles_gbx` DataSource streams per-partition indexed scratch in `write()` and does all shard assignment + assembly from entries metadata in `commit()` — tile bytes never land on the driver, which is what enables both fixed `shardZoom` and adaptive `targetTilesPerShard` sharding and centralizes assembly (one writer per file, no overview write race). - -**Tech Stack:** Python 3.12, PySpark 4.x DataSource V2 (`pyspark.sql.datasource`), the Protomaps `pmtiles` PyPI package (`Writer`, `zxy_to_tileid`, `TileType`, `Compression`, `Reader`), pure-Python `os`/`open` I/O (Serverless-safe). Tests with pytest; local Spark for round-trips; Docker for doc-tests + heavy parity. - -**Reference spec:** `docs/superpowers/specs/2026-06-11-light-pmtiles-writer-design.md` - ---- - -## Repo conventions the engineer must know - -- **Light tier is Serverless-safe:** product code under `pyrx/` and the new `ds/` must NEVER use `._jvm`, `._jsc`, `.sparkContext`, `.rdd`, `.conf.set(`, `SparkConf`, `.setConf(`. A guard test enforces this (Task 3 extends it). The `bench/` package is exempt (harness-only) and is NOT scanned. -- **Namespace packages:** `databricks/`, `databricks/labs/`, `databricks/labs/gbx/` have **no** `__init__.py` (PEP 420). `pyrx/` and `pyrx/ds/` DO have `__init__.py`. The new `gbx/ds/` and `gbx/ds/tiles/` get `__init__.py`. -- **The canonical tile struct** is `databricks.labs.gbx.pyrx._serde.TILE_SCHEMA` = `(cellid: long, raster: binary, metadata: map)`. `_serde`, `_env`, and `core.tiling` **stay in `pyrx`** — the migrated `ds/` code keeps importing them from `databricks.labs.gbx.pyrx`. -- **Beta, no aliases:** the migration is a clean move — `pyrx/ds/` is deleted, no compat shim. -- **Commits:** before each commit run `chmod -R u+rwX .git/objects` (this env drops the execute bit on git object dirs). Commit message trailer is `Co-authored-by: Isaac` (never a human name). Keep subjects ≤72 chars with a WHY body for non-trivial commits. -- **Where tests live:** Python unit/integration tests under `python/geobrix/test/...`; doc-test example code under `docs/tests/python/...` (only runs in Docker); doc pages under `docs/docs/...`. -- **Run Python tests** from repo root with the project venv: `python/geobrix/.venv-pyrx/bin/python -m pytest -v` (or `gbx:test:python --path `). Framework unit tests need no Spark and no Docker; round-trip tests use a local SparkSession; heavy-parity + doc-tests run in Docker via `gbx:test:python-docs`. - ---- - -## File Structure - -**New framework package** `python/geobrix/src/databricks/labs/gbx/ds/`: -- `__init__.py` — package doc + auto-register on import (mirrors `pyrx/ds/__init__.py`). -- `register.py` — `register(spark)` registers ALL light DataSources (raster_gbx, gtiff_gbx, pmtiles_gbx). -- `raster.py`, `gtiff.py`, `writer.py`, `_write.py`, `_encode.py`, `_listing.py` — **migrated** from `pyrx/ds/` unchanged except import prefix. -- `pmtiles.py` — `PMTilesGbxDataSource` (write-only) + `PMTilesGbxWriter` + `PMTilesCommitMessage`. -- `tiles/__init__.py` — empty package marker. -- `tiles/grid.py` — `Grid` protocol + `SlippyGrid` (web-mercator XYZ tile math). -- `tiles/_header.py` — `sniff_tile_type`, `HeaderInfo`, `build_header_info`. -- `tiles/backend.py` — `TileArchiveBackend` protocol + `PMTilesBackend`. -- `tiles/catalog.py` — `CatalogWriter` protocol, `ShardInfo`, `STACManifestCatalog`, `TileJSONCatalog`. -- `tiles/shard.py` — `Entry`, `ScratchWriter`, `read_entries`, `assign_shards`, `stream_sorted`. - -**New/moved tests** `python/geobrix/test/ds/`: -- `conftest.py` — session-scoped local `spark` fixture. -- migrated `test_register.py`, `test_raster_datasource.py`, `test_gtiff_datasource.py`, `test_writer.py`, `test_encode.py`, `test_listing.py`, `test_write_helper.py`, `test_reader_parity.py`, `test_writer_parity.py`, `test_serverless_no_spark_config.py`. -- new `test_pmtiles.py` (round-trip single + sharded + overview + adaptive). -- new `test_pmtiles_parity.py` (Docker, skip-if-heavy-unavailable). -- `tiles/test_grid.py`, `tiles/test_header.py`, `tiles/test_backend.py`, `tiles/test_catalog.py`, `tiles/test_shard.py`. - -**Docs:** `docs/docs/writers/pmtiles_gbx.mdx`, `docs/tests/python/writers/pmtiles_gbx_examples.py` + `test_pmtiles_gbx_examples.py`, `docs/sidebars.js`, `docs/docs/writers/overview.mdx`. - -**Config:** `python/geobrix/pyproject.toml` (`[light]` extra), `python/geobrix/src/databricks/labs/gbx/bench/readers.py` (register import + pmtiles path). - ---- - -### Task 1: Add `pmtiles` dependency to the `[light]` extra - -**Files:** -- Modify: `python/geobrix/pyproject.toml` (the `light = [...]` block) - -- [ ] **Step 1: Add the dependency** - -In `python/geobrix/pyproject.toml`, inside the `light = [` list, add the `pmtiles` line next to `quadbin`: - -```toml - "h3>=4.0,<5", - "quadbin>=0.2,<0.3", - # Protomaps PMTiles archive writer/reader (pure-Python) for the pmtiles_gbx - # tiled-output backend. Writer needs ascending tileid; Reader used in tests. - "pmtiles>=3.4,<4", - "scikit-image>=0.22,<1", -``` - -- [ ] **Step 2: Install into the project venv and verify import** - -Run: -```bash -python/geobrix/.venv-pyrx/bin/pip install -e "python/geobrix[light]" >/dev/null 2>&1 -python/geobrix/.venv-pyrx/bin/python -c "from pmtiles.writer import Writer; from pmtiles.tile import zxy_to_tileid, TileType, Compression; from pmtiles.reader import Reader, MemorySource; print('pmtiles OK')" -``` -Expected: `pmtiles OK` - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/pyproject.toml -git commit -m "build(light): add pmtiles dependency for pmtiles_gbx backend - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: Migrate `pyrx/ds/` → `gbx/ds/` (move files, fix internal imports) - -This is the precursor step from the spec. Move the seven DataSource modules + the package init + tests into the tier-neutral package. `_serde`/`_env`/`core` stay in `pyrx`. - -**Files:** -- Move: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/{__init__,register,raster,gtiff,writer,_write,_encode,_listing}.py` → `python/geobrix/src/databricks/labs/gbx/ds/` -- Move: `python/geobrix/test/pyrx/ds/*` → `python/geobrix/test/ds/` - -- [ ] **Step 1: Create the new package dir and move source files with git** - -Run: -```bash -mkdir -p python/geobrix/src/databricks/labs/gbx/ds -git mv python/geobrix/src/databricks/labs/gbx/pyrx/ds/__init__.py python/geobrix/src/databricks/labs/gbx/ds/__init__.py -git mv python/geobrix/src/databricks/labs/gbx/pyrx/ds/register.py python/geobrix/src/databricks/labs/gbx/ds/register.py -git mv python/geobrix/src/databricks/labs/gbx/pyrx/ds/raster.py python/geobrix/src/databricks/labs/gbx/ds/raster.py -git mv python/geobrix/src/databricks/labs/gbx/pyrx/ds/gtiff.py python/geobrix/src/databricks/labs/gbx/ds/gtiff.py -git mv python/geobrix/src/databricks/labs/gbx/pyrx/ds/writer.py python/geobrix/src/databricks/labs/gbx/ds/writer.py -git mv python/geobrix/src/databricks/labs/gbx/pyrx/ds/_write.py python/geobrix/src/databricks/labs/gbx/ds/_write.py -git mv python/geobrix/src/databricks/labs/gbx/pyrx/ds/_encode.py python/geobrix/src/databricks/labs/gbx/ds/_encode.py -git mv python/geobrix/src/databricks/labs/gbx/pyrx/ds/_listing.py python/geobrix/src/databricks/labs/gbx/ds/_listing.py -rmdir python/geobrix/src/databricks/labs/gbx/pyrx/ds 2>/dev/null || true -``` - -- [ ] **Step 2: Fix the import prefix in the moved source files** - -Only the `pyrx.ds` self-reference changes; `pyrx._serde` / `pyrx._env` / `pyrx.core` stay. In every moved file, replace the package-internal prefix `databricks.labs.gbx.pyrx.ds` with `databricks.labs.gbx.ds`: - -```bash -grep -rl "databricks.labs.gbx.pyrx.ds" python/geobrix/src/databricks/labs/gbx/ds/ \ - | xargs sed -i '' 's/databricks\.labs\.gbx\.pyrx\.ds/databricks.labs.gbx.ds/g' -``` -(On Linux/Docker use `sed -i` without the `''`.) - -Then verify the only remaining `pyrx` references in the moved files are the legitimate `_serde`/`_env`/`core` ones: -```bash -grep -rn "gbx.pyrx" python/geobrix/src/databricks/labs/gbx/ds/ -``` -Expected: only lines importing `databricks.labs.gbx.pyrx._serde`, `...pyrx._env`, or `...pyrx.core...` — no `pyrx.ds`. - -- [ ] **Step 3: Move the tests and fix their import prefix** - -Run: -```bash -mkdir -p python/geobrix/test/ds -git mv python/geobrix/test/pyrx/ds/* python/geobrix/test/ds/ -rmdir python/geobrix/test/pyrx/ds 2>/dev/null || true -grep -rl "databricks.labs.gbx.pyrx.ds" python/geobrix/test/ds/ \ - | xargs sed -i '' 's/databricks\.labs\.gbx\.pyrx\.ds/databricks.labs.gbx.ds/g' -``` - -- [ ] **Step 4: Add a local Spark fixture for the ds test dir** - -Create `python/geobrix/test/ds/conftest.py` (only if one was not moved in Step 3; if `conftest.py` already exists there, skip this step): - -```python -"""Shared fixtures for the gbx.ds DataSource tests.""" - -import pytest -from pyspark.sql import SparkSession - - -@pytest.fixture(scope="session") -def spark(): - s = ( - SparkSession.builder.master("local[2]") - .appName("gbx-ds-tests") - .config("spark.sql.shuffle.partitions", "2") - .getOrCreate() - ) - yield s - s.stop() -``` - -- [ ] **Step 5: Run the migrated ds suite** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds -v -p no:cacheprovider` -Expected: PASS for all migrated tests **except** `test_serverless_no_spark_config.py`, which still roots its scan at `pyrx` and asserts the ds files live under pyrx — that test is fixed in Task 3. (If it fails on the file-list assertion, that is expected here.) - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add -A python/geobrix/src/databricks/labs/gbx/ds python/geobrix/test/ds python/geobrix/src/databricks/labs/gbx/pyrx -git commit -m "refactor(ds): migrate pyrx/ds raster reader+writer to tier-neutral gbx/ds - -Precursor for the tiled-output framework: the DataSource package is no -longer raster-specific. _serde/_env/core stay in pyrx and are imported -from there. Serverless guard + external refs updated in the next commit. - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: Re-point external references (bench, doc-tests, Serverless guard, sidebars) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/readers.py` (3 import sites) -- Modify: `python/geobrix/test/ds/test_serverless_no_spark_config.py` -- Modify: `docs/tests/python/{readers,writers}/*` (register imports in example code) -- Modify: `docs/docs/{readers,writers}/*.mdx` (any register-note text referencing `pyrx.ds.register`) - -- [ ] **Step 1: Update bench readers.py imports** - -In `python/geobrix/src/databricks/labs/gbx/bench/readers.py`, replace all three occurrences of: -```python -from databricks.labs.gbx.pyrx.ds.register import register -``` -with: -```python -from databricks.labs.gbx.ds.register import register -``` - -- [ ] **Step 2: Update doc-test example register imports** - -Replace the register import in the reader/writer example code so doc-tests still import a valid module: -```bash -grep -rl "databricks.labs.gbx.pyrx.ds.register" docs/tests docs/docs \ - | xargs sed -i '' 's/databricks\.labs\.gbx\.pyrx\.ds\.register/databricks.labs.gbx.ds.register/g' -grep -rl "databricks.labs.gbx.pyrx.ds" docs/tests docs/docs \ - | xargs sed -i '' 's/databricks\.labs\.gbx\.pyrx\.ds/databricks.labs.gbx.ds/g' -``` -Then verify nothing in docs still references the old path: -```bash -grep -rn "pyrx.ds" docs/ ; echo "exit:$?" -``` -Expected: no matches (`exit:1` from grep). - -- [ ] **Step 3: Rewrite the Serverless guard to scan pyrx AND gbx/ds** - -Replace the body of `python/geobrix/test/ds/test_serverless_no_spark_config.py` with a two-root scan (pyrx product code stays scanned; the migrated ds code is now a second root; `bench/` stays excluded): - -```python -"""Serverless safety guard: light product code must not mutate Spark config -or reach the JVM bridge. Scans pyrx + gbx.ds (NOT bench, which is harness-only).""" - -import re -from pathlib import Path - -import databricks.labs.gbx.ds as gbx_ds -import databricks.labs.gbx.pyrx as pyrx - -_FORBIDDEN = { - "spark config mutation": re.compile(r"\.conf\.set\s*\("), - "SparkConf": re.compile(r"\bSparkConf\b"), - "setConf": re.compile(r"\.setConf\s*\("), - "setSystemProperty": re.compile(r"\bsetSystemProperty\b"), - "JVM bridge (_jvm)": re.compile(r"\._jvm\b"), - "JVM bridge (_jsc)": re.compile(r"\._jsc\b"), - "sparkContext access": re.compile(r"\.sparkContext\b"), - "RDD API": re.compile(r"\.rdd\b"), -} - -_ROOTS = ( - Path(pyrx.__file__).resolve().parent, - Path(gbx_ds.__file__).resolve().parent, -) - - -def _source_files(): - for root in _ROOTS: - for p in root.rglob("*.py"): - if "__pycache__" in p.parts: - continue - yield p - - -def test_light_product_never_mutates_spark_config_or_uses_jvm_bridge(): - violations = [] - for path in _source_files(): - for i, line in enumerate(path.read_text().splitlines(), start=1): - code = line.split("#", 1)[0] - for label, pat in _FORBIDDEN.items(): - if pat.search(code): - violations.append(f"{path.name}:{i} [{label}] -> {line.strip()}") - assert not violations, ( - "light product code must be Serverless-safe. Found:\n " - + "\n ".join(violations) - ) - - -def test_serverless_scan_includes_ds_modules(): - """The migrated DataSource modules must be in scope of the scan.""" - files = {p.name for p in _source_files()} - for required in ( - "raster.py", - "gtiff.py", - "writer.py", - "_write.py", - "register.py", - "_encode.py", - "_listing.py", - ): - assert required in files, f"{required} not covered by Serverless scan" -``` - -- [ ] **Step 4: Run the guard + a bench import smoke check** - -Run: -```bash -python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/test_serverless_no_spark_config.py -v -p no:cacheprovider -python/geobrix/.venv-pyrx/bin/python -c "import databricks.labs.gbx.bench.readers; print('bench import OK')" -``` -Expected: both guard tests PASS; `bench import OK`. - -- [ ] **Step 5: Fix the sidebars register-note + commit** - -Search `docs/sidebars.js` and any `docs/docs/**/*.mdx` "register" notes for the string `pyrx.ds` and update to `ds` (the Step 2 grep already covered `.mdx`; confirm sidebars): -```bash -grep -rn "pyrx.ds" docs/sidebars.js ; echo "exit:$?" -``` -Expected: no matches. - -```bash -chmod -R u+rwX .git/objects -git add -A python/geobrix/src/databricks/labs/gbx/bench python/geobrix/test/ds docs -git commit -m "refactor(ds): re-point bench, doc-tests, and Serverless guard to gbx.ds - -Bench register imports, doc-test register imports + mdx notes now use -databricks.labs.gbx.ds.register. Serverless guard scans pyrx + gbx.ds -(bench stays excluded as harness-only). - -Co-authored-by: Isaac" -``` - ---- - -### Task 4: `tiles/grid.py` — Grid protocol + SlippyGrid - -Pure-Python web-mercator tile math. No Spark, no Docker. - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/tiles/__init__.py` -- Create: `python/geobrix/src/databricks/labs/gbx/ds/tiles/grid.py` -- Test: `python/geobrix/test/ds/tiles/test_grid.py` - -- [ ] **Step 1: Write the failing test** - -Create `python/geobrix/test/ds/tiles/__init__.py` (empty) and `python/geobrix/test/ds/tiles/test_grid.py`: - -```python -import math - -from databricks.labs.gbx.ds.tiles.grid import SlippyGrid - - -def test_tile_bbox_world_at_zoom0(): - g = SlippyGrid() - minlon, minlat, maxlon, maxlat = g.tile_bbox(0, 0, 0) - assert minlon == -180.0 - assert maxlon == 180.0 - # web-mercator clamps latitude near +/-85.0511 - assert math.isclose(maxlat, 85.0511, abs_tol=1e-3) - assert math.isclose(minlat, -85.0511, abs_tol=1e-3) - - -def test_tile_bbox_ordering_and_quadrant(): - g = SlippyGrid() - # z1 tile (1,0,0) is the NW quadrant: lon [-180,0], lat [0, ~85] - minlon, minlat, maxlon, maxlat = g.tile_bbox(1, 0, 0) - assert (minlon, maxlon) == (-180.0, 0.0) - assert minlat >= -0.001 and maxlat > minlat - - -def test_parent_clamps_and_shifts(): - g = SlippyGrid() - # a z8 tile's parent at shard zoom 6 drops 2 bits - assert g.parent(8, 130, 85, 6) == (6, 130 >> 2, 85 >> 2) - # parent at a zoom deeper than the tile clamps to the tile itself - assert g.parent(4, 3, 5, 6) == (4, 3, 5) - # parent at the same zoom is identity - assert g.parent(6, 12, 7, 6) == (6, 12, 7) - - -def test_tiles_for_bbox_covers_point(): - g = SlippyGrid() - # London ~ (-0.12, 51.5) at zoom 6 -> a single covering tile - tiles = list(g.tiles_for_bbox((-0.13, 51.49, -0.11, 51.51), 6)) - assert len(tiles) >= 1 - for z, x, y in tiles: - bb = g.tile_bbox(z, x, y) - assert bb[0] <= -0.12 <= bb[2] - - -def test_buffered_bbox_expands(): - g = SlippyGrid() - base = g.tile_bbox(6, 32, 21) - buf = g.buffered_bbox(6, 32, 21, 0.25) - assert buf[0] < base[0] and buf[2] > base[2] - assert buf[1] < base[1] and buf[3] > base[3] -``` - -- [ ] **Step 2: Run it to verify failure** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/tiles/test_grid.py -v -p no:cacheprovider` -Expected: FAIL — `ModuleNotFoundError: ... ds.tiles.grid`. - -- [ ] **Step 3: Implement grid.py** - -Create `python/geobrix/src/databricks/labs/gbx/ds/tiles/__init__.py`: -```python -"""gbx.ds.tiles — shared lightweight tiled-output framework (grid, sharding, -catalog, archive backends).""" -``` - -Create `python/geobrix/src/databricks/labs/gbx/ds/tiles/grid.py`: -```python -"""Grid-pluggable tile math. ``SlippyGrid`` is the web-mercator XYZ grid used by -PMTiles; future backends (COG-by-quadbin) add their own ``Grid`` implementation.""" - -from __future__ import annotations - -import math -from typing import Iterable, Protocol, Tuple, runtime_checkable - -BBox = Tuple[float, float, float, float] # (minlon, minlat, maxlon, maxlat) -TileKey = Tuple[int, int, int] # (z, x, y) - - -@runtime_checkable -class Grid(Protocol): - """The minimal tile math every tiled-output backend needs.""" - - def tile_bbox(self, z: int, x: int, y: int) -> BBox: ... - - def parent(self, z: int, x: int, y: int, shard_zoom: int) -> TileKey: ... - - def tiles_for_bbox(self, bbox: BBox, zoom: int) -> Iterable[TileKey]: ... - - def buffered_bbox(self, z: int, x: int, y: int, buffer: float) -> BBox: ... - - -class SlippyGrid: - """Web-mercator slippy-map (XYZ) grid.""" - - def tile_bbox(self, z: int, x: int, y: int) -> BBox: - n = 2 ** z - minlon = x / n * 360.0 - 180.0 - maxlon = (x + 1) / n * 360.0 - 180.0 - lat_top = self._lat(y, n) - lat_bot = self._lat(y + 1, n) - return (minlon, min(lat_top, lat_bot), maxlon, max(lat_top, lat_bot)) - - @staticmethod - def _lat(y: int, n: int) -> float: - return math.degrees(math.atan(math.sinh(math.pi * (1 - 2 * y / n)))) - - def parent(self, z: int, x: int, y: int, shard_zoom: int) -> TileKey: - sz = min(shard_zoom, z) - shift = z - sz - return (sz, x >> shift, y >> shift) - - def tiles_for_bbox(self, bbox: BBox, zoom: int) -> Iterable[TileKey]: - minlon, minlat, maxlon, maxlat = bbox - n = 2 ** zoom - x0 = int((minlon + 180.0) / 360.0 * n) - x1 = int((maxlon + 180.0) / 360.0 * n) - y0 = self._lat_to_y(maxlat, n) - y1 = self._lat_to_y(minlat, n) - for x in range(max(0, x0), min(n - 1, x1) + 1): - for y in range(max(0, y0), min(n - 1, y1) + 1): - yield (zoom, x, y) - - @staticmethod - def _lat_to_y(lat: float, n: int) -> int: - lat = max(min(lat, 85.05112878), -85.05112878) - rad = math.radians(lat) - return int((1.0 - math.asinh(math.tan(rad)) / math.pi) / 2.0 * n) - - def buffered_bbox(self, z: int, x: int, y: int, buffer: float) -> BBox: - minlon, minlat, maxlon, maxlat = self.tile_bbox(z, x, y) - dx = (maxlon - minlon) * buffer - dy = (maxlat - minlat) * buffer - return (minlon - dx, minlat - dy, maxlon + dx, maxlat + dy) -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/tiles/test_grid.py -v -p no:cacheprovider` -Expected: PASS (5 tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/ds/tiles/__init__.py \ - python/geobrix/src/databricks/labs/gbx/ds/tiles/grid.py \ - python/geobrix/test/ds/tiles/__init__.py \ - python/geobrix/test/ds/tiles/test_grid.py -git commit -m "feat(ds/tiles): add Grid protocol + SlippyGrid web-mercator tile math - -Co-authored-by: Isaac" -``` - ---- - -### Task 5: `tiles/_header.py` — tile-type sniff + PMTiles header assembly - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/tiles/_header.py` -- Test: `python/geobrix/test/ds/tiles/test_header.py` - -- [ ] **Step 1: Write the failing test** - -Create `python/geobrix/test/ds/tiles/test_header.py`: -```python -from pmtiles.tile import TileType, Compression - -from databricks.labs.gbx.ds.tiles.grid import SlippyGrid -from databricks.labs.gbx.ds.tiles._header import ( - sniff_tile_type, - build_header_info, -) - -PNG = b"\x89PNG\r\n\x1a\n" + b"\x00" * 8 -JPEG = b"\xff\xd8\xff\xe0" + b"\x00" * 8 -WEBP = b"RIFF\x00\x00\x00\x00WEBP" + b"\x00" * 4 -GZIP_MVT = b"\x1f\x8b\x08\x00" + b"\x00" * 8 - - -def test_sniff_known_types(): - assert sniff_tile_type(PNG) == TileType.PNG - assert sniff_tile_type(JPEG) == TileType.JPEG - assert sniff_tile_type(WEBP) == TileType.WEBP - assert sniff_tile_type(GZIP_MVT) == TileType.MVT - - -def test_build_header_info_zoom_and_bbox(): - g = SlippyGrid() - tiles = [(6, 32, 21), (6, 33, 21), (7, 64, 42)] - info = build_header_info( - tiles, g, TileType.PNG, Compression.NONE, {"name": "demo"} - ) - assert info.min_zoom == 6 - assert info.max_zoom == 7 - minlon, minlat, maxlon, maxlat = info.bbox - assert minlon < maxlon and minlat < maxlat - hd = info.header_dict() - assert hd["min_zoom"] == 6 and hd["max_zoom"] == 7 - assert hd["tile_type"] == TileType.PNG - assert hd["center_zoom"] == 6 - assert isinstance(hd["min_lon_e7"], int) -``` - -- [ ] **Step 2: Run it to verify failure** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/tiles/test_header.py -v -p no:cacheprovider` -Expected: FAIL — `ModuleNotFoundError: ... _header`. - -- [ ] **Step 3: Implement _header.py** - -Create `python/geobrix/src/databricks/labs/gbx/ds/tiles/_header.py`: -```python -"""PMTiles header assembly + tile-type sniffing from magic bytes.""" - -from __future__ import annotations - -from dataclasses import dataclass -from typing import Dict, Iterable, Tuple - -from pmtiles.tile import Compression, TileType - -from databricks.labs.gbx.ds.tiles.grid import BBox, Grid, TileKey - - -def sniff_tile_type(data: bytes) -> TileType: - """Detect tile encoding from magic bytes; default MVT for vector payloads.""" - if data[:8] == b"\x89PNG\r\n\x1a\n": - return TileType.PNG - if data[:3] == b"\xff\xd8\xff": - return TileType.JPEG - if data[:4] == b"RIFF" and data[8:12] == b"WEBP": - return TileType.WEBP - if data[4:8] == b"ftyp" and b"avif" in data[8:20]: - return TileType.AVIF - # MVT is protobuf (often gzipped) with no reliable magic. - return TileType.MVT - - -def _e7(v: float) -> int: - return int(round(v * 1e7)) - - -@dataclass -class HeaderInfo: - tile_type: TileType - tile_compression: Compression - min_zoom: int - max_zoom: int - bbox: BBox - metadata: Dict[str, object] - - def header_dict(self) -> Dict[str, object]: - minlon, minlat, maxlon, maxlat = self.bbox - clon = (minlon + maxlon) / 2.0 - clat = (minlat + maxlat) / 2.0 - return { - "tile_type": self.tile_type, - "tile_compression": self.tile_compression, - "min_zoom": self.min_zoom, - "max_zoom": self.max_zoom, - "min_lon_e7": _e7(minlon), - "min_lat_e7": _e7(minlat), - "max_lon_e7": _e7(maxlon), - "max_lat_e7": _e7(maxlat), - "center_zoom": self.min_zoom, - "center_lon_e7": _e7(clon), - "center_lat_e7": _e7(clat), - } - - -def build_header_info( - tiles: Iterable[TileKey], - grid: Grid, - tile_type: TileType, - tile_compression: Compression, - metadata: Dict[str, object], -) -> HeaderInfo: - """Compute min/max zoom + union bbox over a set of (z,x,y) tiles.""" - tiles = list(tiles) - if not tiles: - raise ValueError("build_header_info requires at least one tile") - zs = [z for z, _, _ in tiles] - minlon = minlat = float("inf") - maxlon = maxlat = float("-inf") - for z, x, y in tiles: - bb = grid.tile_bbox(z, x, y) - minlon, minlat = min(minlon, bb[0]), min(minlat, bb[1]) - maxlon, maxlat = max(maxlon, bb[2]), max(maxlat, bb[3]) - return HeaderInfo( - tile_type=tile_type, - tile_compression=tile_compression, - min_zoom=min(zs), - max_zoom=max(zs), - bbox=(minlon, minlat, maxlon, maxlat), - metadata=metadata, - ) -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/tiles/test_header.py -v -p no:cacheprovider` -Expected: PASS (2 tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/ds/tiles/_header.py \ - python/geobrix/test/ds/tiles/test_header.py -git commit -m "feat(ds/tiles): add tile-type sniff + PMTiles header assembly - -Co-authored-by: Isaac" -``` - ---- - -### Task 6: `tiles/backend.py` — TileArchiveBackend protocol + PMTilesBackend - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/tiles/backend.py` -- Test: `python/geobrix/test/ds/tiles/test_backend.py` - -- [ ] **Step 1: Write the failing test (round-trips via the pmtiles Reader)** - -Create `python/geobrix/test/ds/tiles/test_backend.py`: -```python -import os -import tempfile - -from pmtiles.reader import MmapSource, Reader -from pmtiles.tile import Compression, TileType, zxy_to_tileid - -from databricks.labs.gbx.ds.tiles.grid import SlippyGrid -from databricks.labs.gbx.ds.tiles._header import build_header_info -from databricks.labs.gbx.ds.tiles.backend import PMTilesBackend - -PNG = b"\x89PNG\r\n\x1a\n" - - -def test_pmtiles_backend_round_trip(): - g = SlippyGrid() - tiles = [(6, 32, 21), (6, 33, 21)] - # sorted-by-tileid stream of (tileid, bytes) - payload = {t: PNG + bytes([i]) for i, t in enumerate(tiles)} - stream = sorted( - ((zxy_to_tileid(z, x, y), payload[(z, x, y)]) for (z, x, y) in tiles) - ) - info = build_header_info(tiles, g, TileType.PNG, Compression.NONE, {"name": "t"}) - - with tempfile.TemporaryDirectory() as d: - out = os.path.join(d, "shard.pmtiles") - PMTilesBackend().assemble(iter(stream), info, out) - assert os.path.getsize(out) > 0 - with open(out, "rb") as f: - r = Reader(MmapSource(f)) - assert r.get(6, 32, 21) == payload[(6, 32, 21)] - assert r.get(6, 33, 21) == payload[(6, 33, 21)] - assert r.header()["min_zoom"] == 6 - assert r.metadata()["name"] == "t" -``` - -- [ ] **Step 2: Run it to verify failure** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/tiles/test_backend.py -v -p no:cacheprovider` -Expected: FAIL — `ModuleNotFoundError: ... backend`. - -- [ ] **Step 3: Implement backend.py** - -Create `python/geobrix/src/databricks/labs/gbx/ds/tiles/backend.py`: -```python -"""Tile-archive backends: turn one shard's sorted (tileid, bytes) stream into a -container file. ``PMTilesBackend`` is the first; MBTiles/MVT-dir slot in later.""" - -from __future__ import annotations - -from typing import Iterator, Protocol, Tuple - -from pmtiles.writer import Writer - -from databricks.labs.gbx.ds.tiles._header import HeaderInfo - -SortedTiles = Iterator[Tuple[int, bytes]] # ascending tileid - - -class TileArchiveBackend(Protocol): - def assemble( - self, sorted_tiles: SortedTiles, header_info: HeaderInfo, out_path: str - ) -> None: ... - - -class PMTilesBackend: - """Assemble a single ``.pmtiles`` archive from ascending-tileid tiles.""" - - def assemble( - self, sorted_tiles: SortedTiles, header_info: HeaderInfo, out_path: str - ) -> None: - with open(out_path, "wb") as f: - writer = Writer(f) - for tileid, data in sorted_tiles: - writer.write_tile(tileid, data) - writer.finalize(header_info.header_dict(), header_info.metadata) -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/tiles/test_backend.py -v -p no:cacheprovider` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/ds/tiles/backend.py \ - python/geobrix/test/ds/tiles/test_backend.py -git commit -m "feat(ds/tiles): add TileArchiveBackend protocol + PMTilesBackend - -Co-authored-by: Isaac" -``` - ---- - -### Task 7: `tiles/catalog.py` — CatalogWriter + STACManifestCatalog + TileJSONCatalog - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/tiles/catalog.py` -- Test: `python/geobrix/test/ds/tiles/test_catalog.py` - -- [ ] **Step 1: Write the failing test** - -Create `python/geobrix/test/ds/tiles/test_catalog.py`: -```python -import json -import os -import tempfile - -from databricks.labs.gbx.ds.tiles.catalog import ( - ShardInfo, - STACManifestCatalog, - TileJSONCatalog, -) - -SHARDS = [ - ShardInfo("6/32/21.pmtiles", 6, 14, (-0.5, 51.0, 0.0, 51.5)), - ShardInfo("6/33/21.pmtiles", 6, 14, (0.0, 51.0, 0.5, 51.5)), -] - - -def test_stac_manifest_shape(): - with tempfile.TemporaryDirectory() as d: - path = STACManifestCatalog().write(SHARDS, d) - assert os.path.basename(path) == "catalog.json" - doc = json.load(open(path)) - assert doc["type"] == "FeatureCollection" - assert len(doc["features"]) == 2 - feat = doc["features"][0] - assert feat["geometry"]["type"] == "Polygon" - assert feat["properties"]["pmtiles"] == "6/32/21.pmtiles" - assert feat["properties"]["minzoom"] == 6 - assert feat["properties"]["maxzoom"] == 14 - assert feat["bbox"] == [-0.5, 51.0, 0.0, 51.5] - - -def test_tilejson_shape(): - with tempfile.TemporaryDirectory() as d: - path = TileJSONCatalog().write(SHARDS, d) - doc = json.load(open(path)) - assert doc["tilejson"] == "3.0.0" - assert doc["minzoom"] == 6 and doc["maxzoom"] == 14 - # union bounds across shards - assert doc["bounds"] == [-0.5, 51.0, 0.5, 51.5] - assert len(doc["shards"]) == 2 -``` - -- [ ] **Step 2: Run it to verify failure** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/tiles/test_catalog.py -v -p no:cacheprovider` -Expected: FAIL — `ModuleNotFoundError: ... catalog`. - -- [ ] **Step 3: Implement catalog.py** - -Create `python/geobrix/src/databricks/labs/gbx/ds/tiles/catalog.py`: -```python -"""Catalog writers over a set of shards. Default = a GeoJSON/STAC-style manifest -(one feature per shard with bbox + relative URL); TileJSON is an option. VRT and -full STAC-spec catalogs slot in later (see spec).""" - -from __future__ import annotations - -import json -import os -from dataclasses import dataclass -from typing import List, Protocol, Tuple - -BBox = Tuple[float, float, float, float] - - -@dataclass -class ShardInfo: - rel_path: str # path relative to the catalog (e.g. "6/32/21.pmtiles") - min_zoom: int - max_zoom: int - bbox: BBox - - -class CatalogWriter(Protocol): - def write(self, shards: List[ShardInfo], out_dir: str) -> str: ... - - -def _bbox_polygon(bbox: BBox) -> dict: - minlon, minlat, maxlon, maxlat = bbox - return { - "type": "Polygon", - "coordinates": [ - [ - [minlon, minlat], - [maxlon, minlat], - [maxlon, maxlat], - [minlon, maxlat], - [minlon, minlat], - ] - ], - } - - -def _union(shards: List[ShardInfo]) -> BBox: - minlon = min(s.bbox[0] for s in shards) - minlat = min(s.bbox[1] for s in shards) - maxlon = max(s.bbox[2] for s in shards) - maxlat = max(s.bbox[3] for s in shards) - return (minlon, minlat, maxlon, maxlat) - - -class STACManifestCatalog: - """GeoJSON FeatureCollection (STAC-style): one feature per shard.""" - - def write(self, shards: List[ShardInfo], out_dir: str) -> str: - features = [ - { - "type": "Feature", - "bbox": list(s.bbox), - "geometry": _bbox_polygon(s.bbox), - "properties": { - "pmtiles": s.rel_path, - "minzoom": s.min_zoom, - "maxzoom": s.max_zoom, - }, - } - for s in shards - ] - doc = {"type": "FeatureCollection", "features": features} - path = os.path.join(out_dir, "catalog.json") - with open(path, "w") as f: - json.dump(doc, f) - return path - - -class TileJSONCatalog: - """Minimal TileJSON 3.0.0 over the shards (union bounds + a shards array).""" - - def write(self, shards: List[ShardInfo], out_dir: str) -> str: - bounds = _union(shards) - doc = { - "tilejson": "3.0.0", - "minzoom": min(s.min_zoom for s in shards), - "maxzoom": max(s.max_zoom for s in shards), - "bounds": list(bounds), - "shards": [ - { - "pmtiles": s.rel_path, - "bounds": list(s.bbox), - "minzoom": s.min_zoom, - "maxzoom": s.max_zoom, - } - for s in shards - ], - } - path = os.path.join(out_dir, "catalog.json") - with open(path, "w") as f: - json.dump(doc, f) - return path -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/tiles/test_catalog.py -v -p no:cacheprovider` -Expected: PASS (2 tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/ds/tiles/catalog.py \ - python/geobrix/test/ds/tiles/test_catalog.py -git commit -m "feat(ds/tiles): add CatalogWriter + STAC manifest + TileJSON catalogs - -Co-authored-by: Isaac" -``` - ---- - -### Task 8: `tiles/shard.py` — indexed scratch + entries-driven shard assignment - -The keystone: per-partition indexed scratch (`ScratchWriter`), driver-side `assign_shards` (fixed + adaptive) over entries metadata, and `stream_sorted` to read each shard's bytes back in tileid order. - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/tiles/shard.py` -- Test: `python/geobrix/test/ds/tiles/test_shard.py` - -- [ ] **Step 1: Write the failing test** - -Create `python/geobrix/test/ds/tiles/test_shard.py`: -```python -import os -import tempfile - -from pmtiles.tile import zxy_to_tileid - -from databricks.labs.gbx.ds.tiles.grid import SlippyGrid -from databricks.labs.gbx.ds.tiles.shard import ( - OVERVIEW, - ScratchWriter, - assign_shards, - read_entries, - stream_sorted, -) - - -def _write_scratch(scratch_dir, rows): - w = ScratchWriter(scratch_dir) - for z, x, y, data in rows: - w.add(z, x, y, zxy_to_tileid(z, x, y), data) - return w.close() # (bin_path, idx_path) - - -def test_scratch_round_trip_and_stream_sorted(): - g = SlippyGrid() - with tempfile.TemporaryDirectory() as d: - rows = [ - (6, 33, 21, b"B"), - (6, 32, 21, b"A"), - (7, 64, 42, b"C"), - ] - _bin, idx = _write_scratch(d, rows) - entries = read_entries(idx, d) - assert len(entries) == 3 - streamed = list(stream_sorted(entries)) - # ascending tileid - ids = [tid for tid, _ in streamed] - assert ids == sorted(ids) - assert {data for _, data in streamed} == {b"A", b"B", b"C"} - - -def test_fixed_assignment_and_overview_split(): - g = SlippyGrid() - with tempfile.TemporaryDirectory() as d: - rows = [ - (6, 32, 21, b"a"), # body shard (6,32,21) - (8, 130, 85, b"b"), # body, parent (6, 32, 21) - (3, 4, 2, b"o"), # overview (z<6) - ] - _bin, idx = _write_scratch(d, rows) - entries = read_entries(idx, d) - groups = assign_shards(entries, shard_zoom=6, grid=g) - assert OVERVIEW in groups - assert len(groups[OVERVIEW]) == 1 - body_keys = [k for k in groups if k != OVERVIEW] - # both body tiles share parent (6,32,21) - assert body_keys == [(6, 32, 21)] - assert len(groups[(6, 32, 21)]) == 2 - - -def test_adaptive_subdivides_dense_cells(): - g = SlippyGrid() - with tempfile.TemporaryDirectory() as d: - # 4 z8 tiles under (6,32,21) but in two distinct z7 children - rows = [ - (8, 128, 84, b"1"), - (8, 129, 84, b"2"), - (8, 130, 86, b"3"), - (8, 131, 86, b"4"), - ] - _bin, idx = _write_scratch(d, rows) - entries = read_entries(idx, d) - # target 2 per shard -> base z6 cell (4 tiles) must subdivide - groups = assign_shards( - entries, shard_zoom=6, grid=g, target_tiles_per_shard=2 - ) - assert all(len(v) <= 2 for v in groups.values()) - # variable zoom: at least one shard deeper than 6 - assert any(k[0] > 6 for k in groups) -``` - -- [ ] **Step 2: Run it to verify failure** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/tiles/test_shard.py -v -p no:cacheprovider` -Expected: FAIL — `ModuleNotFoundError: ... shard`. - -- [ ] **Step 3: Implement shard.py** - -Create `python/geobrix/src/databricks/labs/gbx/ds/tiles/shard.py`: -```python -"""Entries-driven sharding. ``write()`` (executor) appends tile bytes to a -per-partition indexed scratch (bytes file + entries index) with NO shard -assignment. ``commit()`` (driver) reads only the entries metadata, assigns each -tile to a shard (fixed or adaptive), then streams each shard's bytes back in -tileid order — tile bytes never load on the driver in bulk.""" - -from __future__ import annotations - -import json -import os -import uuid -from collections import defaultdict -from dataclasses import dataclass -from typing import Dict, Iterator, List, Optional, Tuple - -from databricks.labs.gbx.ds.tiles.grid import Grid, TileKey - -OVERVIEW = "overview" -_MAX_SHARD_ZOOM = 14 # cap adaptive subdivision depth - - -@dataclass -class Entry: - z: int - x: int - y: int - tileid: int - offset: int - length: int - bin_path: str - - -class ScratchWriter: - """Append tile bytes to one partition's scratch bin + collect an index.""" - - def __init__(self, scratch_dir: str): - os.makedirs(scratch_dir, exist_ok=True) - uid = uuid.uuid4().hex - self.bin_path = os.path.join(scratch_dir, f"part-{uid}.bin") - self.idx_path = os.path.join(scratch_dir, f"part-{uid}.idx") - self._f = open(self.bin_path, "wb") - self._entries: List[Tuple[int, int, int, int, int, int]] = [] - self._offset = 0 - - def add(self, z: int, x: int, y: int, tileid: int, data: bytes) -> None: - self._f.write(data) - self._entries.append((z, x, y, tileid, self._offset, len(data))) - self._offset += len(data) - - def close(self) -> Tuple[str, str]: - self._f.close() - with open(self.idx_path, "w") as fh: - json.dump( - {"bin": os.path.basename(self.bin_path), "entries": self._entries}, - fh, - ) - return self.bin_path, self.idx_path - - -def read_entries(idx_path: str, scratch_dir: str) -> List[Entry]: - with open(idx_path) as fh: - doc = json.load(fh) - bin_path = os.path.join(scratch_dir, doc["bin"]) - return [ - Entry(z, x, y, tid, off, length, bin_path) - for (z, x, y, tid, off, length) in doc["entries"] - ] - - -def assign_shards( - entries: List[Entry], - shard_zoom: int, - grid: Grid, - target_tiles_per_shard: Optional[int] = None, -) -> Dict[object, List[Entry]]: - """Group entries into shards. ``z < shard_zoom`` go to OVERVIEW; the rest are - keyed by fixed parent or adaptively subdivided. Keys are (sz,sx,sy) tuples.""" - overview = [e for e in entries if e.z < shard_zoom] - body = [e for e in entries if e.z >= shard_zoom] - - if target_tiles_per_shard is None: - groups: Dict[object, List[Entry]] = defaultdict(list) - for e in body: - groups[grid.parent(e.z, e.x, e.y, shard_zoom)].append(e) - result: Dict[object, List[Entry]] = dict(groups) - else: - result = _adaptive(body, shard_zoom, grid, target_tiles_per_shard) - - if overview: - result[OVERVIEW] = overview - return result - - -def _adaptive( - entries: List[Entry], base_zoom: int, grid: Grid, target: int -) -> Dict[object, List[Entry]]: - result: Dict[object, List[Entry]] = {} - - def recurse(zoom: int, cell_entries: List[Entry]) -> None: - if len(cell_entries) <= target or zoom >= _MAX_SHARD_ZOOM: - e0 = cell_entries[0] - result[grid.parent(e0.z, e0.x, e0.y, zoom)] = cell_entries - return - buckets: Dict[TileKey, List[Entry]] = defaultdict(list) - for e in cell_entries: - buckets[grid.parent(e.z, e.x, e.y, zoom + 1)].append(e) - # If subdivision did not actually split (all entries clamp to <= zoom), - # keep them here to avoid infinite recursion. - if len(buckets) == 1 and zoom + 1 > max(e.z for e in cell_entries): - e0 = cell_entries[0] - result[grid.parent(e0.z, e0.x, e0.y, zoom)] = cell_entries - return - for sub in buckets.values(): - recurse(zoom + 1, sub) - - base: Dict[TileKey, List[Entry]] = defaultdict(list) - for e in entries: - base[grid.parent(e.z, e.x, e.y, base_zoom)].append(e) - for cell in base.values(): - recurse(base_zoom, cell) - return result - - -def stream_sorted(entries: List[Entry]) -> Iterator[Tuple[int, bytes]]: - """Yield (tileid, bytes) in ascending tileid order, reading from scratch bins. - Duplicate tileids (should not occur in non-overlapping shards) are dropped.""" - ordered = sorted(entries, key=lambda e: e.tileid) - handles: Dict[str, object] = {} - seen = set() - try: - for e in ordered: - if e.tileid in seen: - continue - seen.add(e.tileid) - fh = handles.get(e.bin_path) - if fh is None: - fh = handles[e.bin_path] = open(e.bin_path, "rb") - fh.seek(e.offset) - yield e.tileid, fh.read(e.length) - finally: - for fh in handles.values(): - fh.close() -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/tiles/test_shard.py -v -p no:cacheprovider` -Expected: PASS (3 tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/ds/tiles/shard.py \ - python/geobrix/test/ds/tiles/test_shard.py -git commit -m "feat(ds/tiles): entries-driven indexed scratch + shard assignment - -ScratchWriter appends per-partition bytes + an index; assign_shards groups -from entries metadata (fixed parent or adaptive targetTilesPerShard, plus an -overview split); stream_sorted reads each shard's bytes in tileid order. - -Co-authored-by: Isaac" -``` - ---- - -### Task 9: `gbx/ds/pmtiles.py` — PMTilesGbxDataSource + PMTilesGbxWriter - -Wire the framework into a write-only DataSource V2. `write()` → scratch; `commit()` → assign + assemble (single / sharded / overview) + catalog; `abort()` → cleanup. - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/pmtiles.py` -- Test: `python/geobrix/test/ds/test_pmtiles.py` - -- [ ] **Step 1: Write the failing test (local Spark round-trip: single + sharded + overview)** - -Create `python/geobrix/test/ds/test_pmtiles.py`: -```python -import json -import os - -from pmtiles.reader import MmapSource, Reader - -from databricks.labs.gbx.ds.register import register - -PNG = b"\x89PNG\r\n\x1a\n" - - -def _png(tag: int) -> bytes: - return PNG + bytes([tag]) - - -def _rows(spark, tiles): - data = [(z, x, y, bytearray(_png(i))) for i, (z, x, y) in enumerate(tiles)] - return spark.createDataFrame(data, schema="z int, x int, y int, bytes binary") - - -def _read_tile(path, z, x, y): - with open(path, "rb") as f: - return Reader(MmapSource(f)).get(z, x, y) - - -def test_single_archive(spark, tmp_path): - register(spark) - out = str(tmp_path / "world.pmtiles") - tiles = [(6, 32, 21), (6, 33, 21), (7, 64, 42)] - _rows(spark, tiles).write.format("pmtiles_gbx").mode("overwrite").option( - "shardZoom", "0" - ).save(out) - assert os.path.isfile(out) - assert _read_tile(out, 6, 32, 21) is not None - assert _read_tile(out, 7, 64, 42) is not None - - -def test_sharded_with_overview_and_catalog(spark, tmp_path): - register(spark) - out = str(tmp_path / "tileset_out") - tiles = [(6, 32, 21), (8, 130, 85), (3, 4, 2)] # body, body(same parent), overview - _rows(spark, tiles).write.format("pmtiles_gbx").mode("overwrite").save(out) - - tileset = os.path.join(out, "tileset") - assert os.path.isfile(os.path.join(tileset, "6", "32", "21.pmtiles")) - assert os.path.isfile(os.path.join(tileset, "overview.pmtiles")) - catalog = json.load(open(os.path.join(tileset, "catalog.json"))) - assert catalog["type"] == "FeatureCollection" - # body tile reads back from its shard - assert _read_tile( - os.path.join(tileset, "6", "32", "21.pmtiles"), 6, 32, 21 - ) is not None - # overview tile reads back from overview archive - assert _read_tile(os.path.join(tileset, "overview.pmtiles"), 3, 4, 2) is not None - # scratch cleaned up - assert not os.path.isdir(os.path.join(out, "_scratch")) - - -def test_append_mode_rejected(spark, tmp_path): - register(spark) - out = str(tmp_path / "appendme") - os.makedirs(out, exist_ok=True) - open(os.path.join(out, "marker"), "w").close() - import pytest - - with pytest.raises(Exception): - _rows(spark, [(6, 32, 21)]).write.format("pmtiles_gbx").mode("append").save( - out - ) -``` - -- [ ] **Step 2: Run it to verify failure** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/test_pmtiles.py -v -p no:cacheprovider` -Expected: FAIL — `pmtiles_gbx` is not a registered format / `ModuleNotFoundError`. - -- [ ] **Step 3: Implement pmtiles.py** - -Create `python/geobrix/src/databricks/labs/gbx/ds/pmtiles.py`: -```python -"""``pmtiles_gbx`` — pure-Python DataSource V2 PMTiles writer on the shared -tiled-output framework. Write-only. Default = sharded (shardZoom=6) with a -separate overview.pmtiles + a STAC manifest; shardZoom=0 = single archive.""" - -from __future__ import annotations - -import json -import os -import shutil -from dataclasses import dataclass -from typing import Dict, Iterator, List, Optional - -from pmtiles.tile import Compression, TileType, zxy_to_tileid -from pyspark.sql.datasource import DataSource, DataSourceWriter, WriterCommitMessage -from pyspark.sql.types import ( - BinaryType, - IntegerType, - StructField, - StructType, -) - -from databricks.labs.gbx.ds.tiles import shard as _shard -from databricks.labs.gbx.ds.tiles._header import build_header_info, sniff_tile_type -from databricks.labs.gbx.ds.tiles.backend import PMTilesBackend -from databricks.labs.gbx.ds.tiles.catalog import ( - STACManifestCatalog, - ShardInfo, - TileJSONCatalog, -) -from databricks.labs.gbx.ds.tiles.grid import SlippyGrid - -INPUT_SCHEMA = StructType( - [ - StructField("z", IntegerType(), nullable=False), - StructField("x", IntegerType(), nullable=False), - StructField("y", IntegerType(), nullable=False), - StructField("bytes", BinaryType(), nullable=False), - ] -) - -_COMPRESSION = { - "none": Compression.NONE, - "gzip": Compression.GZIP, - "brotli": Compression.BROTLI, - "zstd": Compression.ZSTD, -} -_TILETYPE = { - "png": TileType.PNG, - "jpeg": TileType.JPEG, - "jpg": TileType.JPEG, - "webp": TileType.WEBP, - "avif": TileType.AVIF, - "mvt": TileType.MVT, -} -_CATALOGS = {"stac": STACManifestCatalog, "tilejson": TileJSONCatalog} - - -def assert_input_schema(schema: StructType) -> None: - names = [f.name for f in schema.fields] - if names != ["z", "x", "y", "bytes"]: - raise ValueError( - "pmtiles_gbx requires exactly columns (z:int, x:int, y:int, " - f"bytes:binary); got {names}" - ) - - -@dataclass -class PMTilesCommitMessage(WriterCommitMessage): - bin_path: str - idx_path: str - - -class PMTilesGbxDataSource(DataSource): - @classmethod - def name(cls) -> str: - return "pmtiles_gbx" - - def schema(self) -> StructType: - return INPUT_SCHEMA - - def writer(self, schema: StructType, overwrite: bool) -> DataSourceWriter: - assert_input_schema(schema) - path = self.options.get("path") - if not path: - raise ValueError("pmtiles_gbx writer requires an output path (.save(path)).") - return PMTilesGbxWriter(path, dict(self.options), overwrite) - - -class PMTilesGbxWriter(DataSourceWriter): - def __init__(self, path: str, options: Dict[str, str], overwrite: bool): - self.path = path - self.overwrite = overwrite - self.shard_zoom = int(options.get("shardZoom", "6")) - tps = options.get("targetTilesPerShard") - self.target_tiles_per_shard = int(tps) if tps else None - self.catalog_kind = options.get("catalog", "stac").lower() - if self.catalog_kind not in _CATALOGS and self.catalog_kind != "none": - raise ValueError(f"unknown catalog {self.catalog_kind!r}") - tt = options.get("tileType") - self.tile_type_override = _TILETYPE[tt.lower()] if tt else None - self.tile_compression = _COMPRESSION[options.get("tileCompression", "none").lower()] - self.metadata = json.loads(options["metadata"]) if options.get("metadata") else {} - self.scratch_dir = os.path.join(self.path, "_scratch") - self.grid = SlippyGrid() - - if not self.overwrite and self._target_exists(): - raise ValueError( - "pmtiles_gbx does not support append; a finalized archive cannot be " - "appended to. Use .mode('overwrite')." - ) - if self.overwrite: - self._clear_target() - - # ---- driver-side path helpers (no Spark internals; pure os) ---- - def _is_single(self) -> bool: - return self.shard_zoom == 0 - - def _target_exists(self) -> bool: - return os.path.exists(self.path) and ( - os.path.isfile(self.path) or bool(os.listdir(self.path)) - ) - - def _clear_target(self) -> None: - if os.path.isfile(self.path): - os.remove(self.path) - elif os.path.isdir(self.path): - shutil.rmtree(self.path) - - # ---- executor: stream bytes to indexed scratch ---- - def write(self, iterator: Iterator) -> WriterCommitMessage: - writer = _shard.ScratchWriter(self.scratch_dir) - for row in iterator: - z, x, y, data = int(row[0]), int(row[1]), int(row[2]), bytes(row[3]) - writer.add(z, x, y, zxy_to_tileid(z, x, y), data) - bin_path, idx_path = writer.close() - return PMTilesCommitMessage(bin_path=bin_path, idx_path=idx_path) - - # ---- driver: assemble shards + catalog from entries ---- - def commit(self, messages: List[Optional[WriterCommitMessage]]) -> None: - entries: List[_shard.Entry] = [] - for msg in messages: - if isinstance(msg, PMTilesCommitMessage): - entries.extend(_shard.read_entries(msg.idx_path, self.scratch_dir)) - try: - if not entries: - return - if self._is_single(): - self._assemble_single(entries) - else: - self._assemble_sharded(entries) - finally: - shutil.rmtree(self.scratch_dir, ignore_errors=True) - - def _tile_type(self, sample: bytes) -> TileType: - return self.tile_type_override or sniff_tile_type(sample) - - def _assemble_single(self, entries: List[_shard.Entry]) -> None: - os.makedirs(os.path.dirname(self.path) or ".", exist_ok=True) - tiles = [(e.z, e.x, e.y) for e in entries] - sample = next(iter(_shard.stream_sorted(entries[:1])))[1] - info = build_header_info( - tiles, self.grid, self._tile_type(sample), self.tile_compression, self.metadata - ) - PMTilesBackend().assemble(_shard.stream_sorted(entries), info, self.path) - - def _assemble_sharded(self, entries: List[_shard.Entry]) -> None: - tileset = os.path.join(self.path, "tileset") - os.makedirs(tileset, exist_ok=True) - groups = _shard.assign_shards( - entries, self.shard_zoom, self.grid, self.target_tiles_per_shard - ) - shard_infos: List[ShardInfo] = [] - for key, group in groups.items(): - sample = next(iter(_shard.stream_sorted(group[:1])))[1] - tiles = [(e.z, e.x, e.y) for e in group] - info = build_header_info( - tiles, self.grid, self._tile_type(sample), self.tile_compression, - self.metadata, - ) - if key == _shard.OVERVIEW: - rel = "overview.pmtiles" - else: - sz, sx, sy = key - rel = os.path.join(str(sz), str(sx), f"{sy}.pmtiles") - out_path = os.path.join(tileset, rel) - os.makedirs(os.path.dirname(out_path), exist_ok=True) - PMTilesBackend().assemble(_shard.stream_sorted(group), info, out_path) - shard_infos.append( - ShardInfo(rel, info.min_zoom, info.max_zoom, info.bbox) - ) - if self.catalog_kind != "none": - _CATALOGS[self.catalog_kind]().write(shard_infos, tileset) - - def abort(self, messages: List[Optional[WriterCommitMessage]]) -> None: - shutil.rmtree(self.scratch_dir, ignore_errors=True) - self._clear_target() -``` - -- [ ] **Step 4: Temporarily register pmtiles_gbx so the test can load it** - -The test calls `register(spark)` from `gbx.ds.register`, which does not yet include the new source. Add it now (this anticipates Task 10; keep it minimal): edit `python/geobrix/src/databricks/labs/gbx/ds/register.py` to import + include `PMTilesGbxDataSource`: - -```python -from typing import Optional - -from pyspark.sql import SparkSession - -from databricks.labs.gbx.ds.gtiff import GTiffGbxDataSource -from databricks.labs.gbx.ds.pmtiles import PMTilesGbxDataSource -from databricks.labs.gbx.ds.raster import RasterGbxDataSource - -_SOURCES = (RasterGbxDataSource, GTiffGbxDataSource, PMTilesGbxDataSource) - - -def register(spark: Optional[SparkSession] = None) -> None: - """Register all light DataSources. Uses the active session if not given.""" - if spark is None: - spark = SparkSession.builder.getOrCreate() - for source in _SOURCES: - spark.dataSource.register(source) -``` - -(If `register.py` had a `_try_register_on_import()` helper that lists sources, update it the same way.) - -- [ ] **Step 5: Run the test to verify it passes** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/test_pmtiles.py -v -p no:cacheprovider` -Expected: PASS (3 tests: single, sharded+overview+catalog, append rejected). - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/ds/pmtiles.py \ - python/geobrix/src/databricks/labs/gbx/ds/register.py \ - python/geobrix/test/ds/test_pmtiles.py -git commit -m "feat(ds): add pmtiles_gbx writer (single/sharded/overview + catalog) - -Write-only DataSource V2 on the tiles framework: write() streams indexed -scratch, commit() assigns shards + assembles archives + overview + catalog -from entries metadata, abort() cleans up. Registered in gbx.ds.register. - -Co-authored-by: Isaac" -``` - ---- - -### Task 10: Extend the Serverless guard to cover the framework + finalize register - -**Files:** -- Modify: `python/geobrix/test/ds/test_serverless_no_spark_config.py` - -- [ ] **Step 1: Add the new modules to the scan-coverage assertion** - -The two-root scan from Task 3 already walks `gbx/ds/` recursively (so `tiles/*.py` and `pmtiles.py` are scanned for forbidden patterns automatically). Extend the explicit coverage assertion so a future accidental deletion is caught. In `test_serverless_scan_includes_ds_modules`, extend the required list: - -```python - for required in ( - "raster.py", - "gtiff.py", - "writer.py", - "_write.py", - "register.py", - "_encode.py", - "_listing.py", - "pmtiles.py", - "grid.py", - "_header.py", - "backend.py", - "catalog.py", - "shard.py", - ): - assert required in files, f"{required} not covered by Serverless scan" -``` - -- [ ] **Step 2: Run the full ds suite green** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds -v -p no:cacheprovider` -Expected: PASS for everything (migrated raster/gtiff/writer tests, framework `tiles/` tests, `test_pmtiles.py`, both Serverless tests). - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/ds/test_serverless_no_spark_config.py -git commit -m "test(ds): cover tiles framework + pmtiles in Serverless guard scan - -Co-authored-by: Isaac" -``` - ---- - -### Task 11: Light-vs-heavy parity (Docker integration, skip-if-heavy-unavailable) - -Confirm `pmtiles_gbx` (single mode) and the heavy `pmtiles` writer produce archives that decode to the same `z/x/y → bytes` set. Heavy needs the JAR + GDAL, so the test is skipped when heavy is unavailable (as the reader/writer parity tests already do). - -**Files:** -- Create: `python/geobrix/test/ds/test_pmtiles_parity.py` - -- [ ] **Step 1: Write the parity test** - -Create `python/geobrix/test/ds/test_pmtiles_parity.py`: -```python -"""Light vs heavy PMTiles parity. Skipped unless the heavy `pmtiles` writer is -registered (JAR + GDAL present, i.e. on-cluster / Docker with heavy env).""" - -import os - -import pytest -from pmtiles.reader import MmapSource, Reader - -from databricks.labs.gbx.ds.register import register - -PNG = b"\x89PNG\r\n\x1a\n" - - -def _heavy_available(spark) -> bool: - try: - spark.read.format("pmtiles") - return True - except Exception: - return False - - -def _decode_all(path): - out = {} - with open(path, "rb") as f: - r = Reader(MmapSource(f)) - for z in range(0, 10): - for x in range(0, 2 ** z): - for y in range(0, 2 ** z): - t = r.get(z, x, y) - if t is not None: - out[(z, x, y)] = t - return out - - -def test_light_vs_heavy_single_archive(spark, tmp_path): - if not _heavy_available(spark): - pytest.skip("heavy pmtiles writer unavailable (no JAR/GDAL)") - register(spark) - tiles = [(2, 1, 1), (2, 2, 1), (3, 4, 3)] - rows = [(z, x, y, bytearray(PNG + bytes([i]))) for i, (z, x, y) in enumerate(tiles)] - df = spark.createDataFrame(rows, schema="z int, x int, y int, bytes binary") - - light_out = str(tmp_path / "light.pmtiles") - df.write.format("pmtiles_gbx").mode("overwrite").option("shardZoom", "0").save( - light_out - ) - - heavy_out = str(tmp_path / "heavy.pmtiles") - df.write.format("pmtiles").mode("overwrite").save(heavy_out) - - assert _decode_all(light_out) == _decode_all(heavy_out) -``` - -- [ ] **Step 2: Run in Docker (heavy env)** - -Run via the doc/integration test path inside the dev container (dispatch as a Task subagent — it touches Docker and may take minutes): -```bash -gbx:test:python --path test/ds/test_pmtiles_parity.py -``` -Expected: PASS on-cluster/Docker, or SKIP locally if heavy is unavailable. (Local host run: SKIP is acceptable and expected.) - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/ds/test_pmtiles_parity.py -git commit -m "test(ds): light-vs-heavy pmtiles parity (skip-if-heavy-unavailable) - -Co-authored-by: Isaac" -``` - ---- - -### Task 12: Bench — add a PMTiles write-timing path - -Extend the writer bench so `run_format_write` can time `pmtiles_gbx` (and the heavy `pmtiles`). Cluster execution is operator-driven (not part of the green-tests gate); this task adds the code path + a unit-level smoke check. - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/readers.py` -- Test: `python/geobrix/test/bench/test_readers_pmtiles.py` (create dir/file if absent) - -- [ ] **Step 1: Write a smoke test for the pmtiles write path** - -Create `python/geobrix/test/bench/test_readers_pmtiles.py` (create `python/geobrix/test/bench/` if it does not exist): -```python -"""Smoke test: the writer bench can run a pmtiles_gbx write and return a row.""" - -import os - -import pytest -from pyspark.sql import SparkSession - -from databricks.labs.gbx.bench.readers import run_pmtiles_write - - -@pytest.fixture(scope="module") -def spark(): - s = SparkSession.builder.master("local[2]").appName("bench-pmtiles").getOrCreate() - yield s - s.stop() - - -def test_run_pmtiles_write_returns_row(spark, tmp_path): - out = str(tmp_path / "bench_tiles") - row = run_pmtiles_write( - spark, - out_path=out, - run_id="t", - warmup=0, - measured=1, - n_tiles=8, - shard_zoom=0, - write_fmt="pmtiles_gbx", - ) - assert row.category == "writer" - assert row.fn == "pmtiles_gbx" - assert row.elapsed_s is not None - assert os.path.exists(out) -``` - -- [ ] **Step 2: Run it to verify failure** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/bench/test_readers_pmtiles.py -v -p no:cacheprovider` -Expected: FAIL — `ImportError: cannot import name 'run_pmtiles_write'`. - -- [ ] **Step 3: Implement run_pmtiles_write in bench/readers.py** - -Add to `python/geobrix/src/databricks/labs/gbx/bench/readers.py` (reuse the existing `time_iters` / `ResultRow` helpers already imported in that module — match their names exactly as used by `run_format_write`): -```python -def run_pmtiles_write( - spark, - out_path: str, - run_id: str, - warmup: int, - measured: int, - *, - n_tiles: int = 1000, - shard_zoom: int = 0, - write_fmt: str = "pmtiles_gbx", -): - """Time a PMTiles write of `n_tiles` synthetic PNG tiles. write_fmt is - 'pmtiles_gbx' (light) or 'pmtiles' (heavy). Returns a single writer ResultRow.""" - from databricks.labs.gbx.ds.register import register - - if write_fmt == "pmtiles_gbx": - register(spark) - - png = b"\x89PNG\r\n\x1a\n" - # generate a z-grid of n_tiles tiles at a zoom that fits them - z = max(1, (max(1, n_tiles) - 1).bit_length() // 2 + 1) - side = 2 ** z - rows = [] - for i in range(n_tiles): - x, y = i % side, (i // side) % side - rows.append((z, x, y, bytearray(png + i.to_bytes(4, "big")))) - df = spark.createDataFrame( - rows, schema="z int, x int, y int, bytes binary" - ).cache() - df.count() - - def _write(): - writer = df.write.format(write_fmt).mode("overwrite") - if write_fmt == "pmtiles_gbx": - writer = writer.option("shardZoom", str(shard_zoom)) - writer.save(out_path) - - elapsed = time_iters(_write, warmup=warmup, measured=measured) - return ResultRow( - run_id=run_id, - fn=write_fmt, - category="writer", - mode="spark-path", - elapsed_s=elapsed, - ) -``` - -NOTE: `time_iters` returns the median measured time in seconds and `ResultRow` is the dataclass already used by `run_format_write` in this file. If `ResultRow`'s required fields differ from the kwargs above, match them to the existing `run_format_write` construction (read the top of `readers.py`) — do not invent new fields. - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/bench/test_readers_pmtiles.py -v -p no:cacheprovider` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/bench/readers.py \ - python/geobrix/test/bench/test_readers_pmtiles.py -git commit -m "feat(bench): add run_pmtiles_write timing path (light + heavy) - -Co-authored-by: Isaac" -``` - ---- - -### Task 13: Docs — `pmtiles_gbx.mdx` + doc-test example + sidebar + overview - -Doc tests ARE the documentation source and run only in Docker. - -**Files:** -- Create: `docs/tests/python/writers/pmtiles_gbx_examples.py` -- Create: `docs/tests/python/writers/test_pmtiles_gbx_examples.py` -- Create: `docs/docs/writers/pmtiles_gbx.mdx` -- Modify: `docs/sidebars.js` (Lightweight → Writers → Named) -- Modify: `docs/docs/writers/overview.mdx` (cross-link) - -- [ ] **Step 1: Write the doc-test example code (string constants + verification functions)** - -Create `docs/tests/python/writers/pmtiles_gbx_examples.py`: -```python -"""Executable doc examples for the lightweight pmtiles_gbx writer (run in Docker).""" - -import json -import os -import tempfile - -from pmtiles.reader import MmapSource, Reader - -PNG = b"\x89PNG\r\n\x1a\n" - -WRITE_PMTILES_SHARDED = """# Lightweight PMTiles writer — distributed spatial sharding (default). -# Input is a tile pyramid: (z, x, y, bytes). shardZoom=6 emits one -# tileset/{z}/{x}/{y}.pmtiles per populated parent + overview.pmtiles + a -# STAC catalog.json. -from databricks.labs.gbx.ds.register import register -register(spark) -df.write.format("pmtiles_gbx").mode("overwrite").option("shardZoom", "6").save(OUT_DIR)""" - -WRITE_PMTILES_SINGLE = """# Single-archive PMTiles: shardZoom=0 packs every tile into one .pmtiles file. -from databricks.labs.gbx.ds.register import register -register(spark) -df.write.format("pmtiles_gbx").mode("overwrite").option("shardZoom", "0").save(OUT_FILE)""" - -OPTIONS_NOTE = """# Knobs (sensible defaults): -# shardZoom 6 -> sharded; 0 -> single archive -# targetTilesPerShard adaptive sharding (subdivide dense cells) -# catalog stac (default) | tilejson | none -# tileType auto-sniff (png/jpeg/webp/mvt); override if needed -# tileCompression none (default) | gzip | brotli | zstd -# metadata JSON string -> archive metadata""" - - -def _pyramid_df(spark, tiles): - rows = [(z, x, y, bytearray(PNG + bytes([i]))) for i, (z, x, y) in enumerate(tiles)] - return spark.createDataFrame(rows, schema="z int, x int, y int, bytes binary") - - -def write_pmtiles_single(spark): - """Verify WRITE_PMTILES_SINGLE: write one archive, read tiles back.""" - from databricks.labs.gbx.ds.register import register - - register(spark) - df = _pyramid_df(spark, [(2, 1, 1), (2, 2, 1), (3, 4, 3)]) - with tempfile.TemporaryDirectory() as d: - out = os.path.join(d, "world.pmtiles") - df.write.format("pmtiles_gbx").mode("overwrite").option( - "shardZoom", "0" - ).save(out) - with open(out, "rb") as f: - r = Reader(MmapSource(f)) - assert r.get(2, 1, 1) is not None - assert r.get(3, 4, 3) is not None - - -def write_pmtiles_sharded(spark): - """Verify WRITE_PMTILES_SHARDED: sharded output + overview + STAC catalog.""" - from databricks.labs.gbx.ds.register import register - - register(spark) - df = _pyramid_df(spark, [(6, 32, 21), (8, 130, 85), (3, 4, 2)]) - with tempfile.TemporaryDirectory() as d: - df.write.format("pmtiles_gbx").mode("overwrite").option( - "shardZoom", "6" - ).save(d) - tileset = os.path.join(d, "tileset") - assert os.path.isfile(os.path.join(tileset, "6", "32", "21.pmtiles")) - assert os.path.isfile(os.path.join(tileset, "overview.pmtiles")) - cat = json.load(open(os.path.join(tileset, "catalog.json"))) - assert cat["type"] == "FeatureCollection" and cat["features"] -``` - -Create `docs/tests/python/writers/test_pmtiles_gbx_examples.py`: -```python -"""Executes the pmtiles_gbx writer doc examples (Docker).""" - -import sys -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).parent)) - -import pmtiles_gbx_examples as ex # noqa: E402 - - -def test_write_pmtiles_single(spark): - ex.write_pmtiles_single(spark) - - -def test_write_pmtiles_sharded(spark): - ex.write_pmtiles_sharded(spark) -``` - -- [ ] **Step 2: Write the doc page** - -Create `docs/docs/writers/pmtiles_gbx.mdx` (match the frontmatter/`sidebar_position` convention of the sibling `gtiff_gbx.mdx`; pick the next free position in the Lightweight → Writers → Named group): -```jsx ---- -sidebar_position: 6 ---- - -import CodeFromTest from '@site/src/components/CodeFromTest'; -import pmtilesEx from '!!raw-loader!../../tests/python/writers/pmtiles_gbx_examples.py'; - -# Lightweight PMTiles Writer (`pmtiles_gbx`) - -Pure-Python, JAR-free, Serverless-safe writer that packages a tile pyramid -(`(z, x, y, bytes)`) into [PMTiles](https://docs.protomaps.com/pmtiles/) archives -using distributed **spatial sharding**: each populated parent tile becomes one -bounded, non-overlapping `.pmtiles` shard, plus a global `overview.pmtiles` and a -catalog over the shards. See the [spatial-sharding model](#spatial-sharding) below. - -## Sharded output (default) - - - -Output layout: - -``` -OUT_DIR/tileset/{z}/{x}/{y}.pmtiles # one per populated parent (Z >= shardZoom) -OUT_DIR/tileset/overview.pmtiles # Z < shardZoom global overview -OUT_DIR/tileset/catalog.json # STAC/GeoJSON manifest -``` - -## Single archive - - - -## Options - - - -## Spatial sharding {#spatial-sharding} - -The writer treats tiled output as immutable, spatially-indexed shards: partition -the world by a grid, emit one bounded `.pmtiles` per parent tile, and deliver a -catalog over the shards rather than one merged file. This keeps shards -independently regenerable and lets a browser fetch only the shard for the area in -view. Set `shardZoom=0` for a single merged archive. -``` - -- [ ] **Step 3: Add to the sidebar** - -In `docs/sidebars.js`, find the **Lightweight → Writers → Named** items array (the one containing `'writers/gtiff_gbx'`) and add `'writers/pmtiles_gbx'` beside it: -```javascript -{ type: 'category', label: 'Named', collapsed: false, items: ['writers/gtiff_gbx', 'writers/pmtiles_gbx'] }, -``` -(Read the current `docs/sidebars.js` to place it in the correct tier group — the writers nav is tiered `Overview | Lightweight | Heavyweight`; `pmtiles_gbx` is Lightweight.) - -- [ ] **Step 4: Cross-link in the writers overview** - -In `docs/docs/writers/overview.mdx`, add a one-line entry for the lightweight PMTiles writer in the lightweight writers list, e.g.: -```markdown -- **`pmtiles_gbx`** — package a tile pyramid into spatially-sharded PMTiles archives + a catalog. -``` - -- [ ] **Step 5: Run the doc tests + internals-leak check (Docker; dispatch as a Task subagent)** - -Run: -```bash -gbx:test:python-docs --path writers/ --log pmtiles-docs.log -grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/ ; echo "exit:$?" -``` -Expected: writer doc tests PASS (including the two new `pmtiles_gbx` tests); the wave-leak grep prints nothing (`exit:1`). - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add docs/tests/python/writers/pmtiles_gbx_examples.py \ - docs/tests/python/writers/test_pmtiles_gbx_examples.py \ - docs/docs/writers/pmtiles_gbx.mdx docs/sidebars.js docs/docs/writers/overview.mdx -git commit -m "docs(writers): add lightweight pmtiles_gbx writer page + doc-tests - -Co-authored-by: Isaac" -``` - ---- - -## Final verification (after all tasks) - -- [ ] **Full light ds suite:** `python/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/ds python/geobrix/test/bench/test_readers_pmtiles.py -v` — all green (heavy-parity SKIPs locally). -- [ ] **Serverless guard:** both tests in `test_serverless_no_spark_config.py` green; scan covers all `tiles/*.py` + `pmtiles.py`. -- [ ] **Python lint (CI gate):** `gbx:lint:python --check` (isort/black/flake8) — confirm in-container per the host-vs-Docker black caveat before push. -- [ ] **Doc tests + internals-leak (Docker):** `gbx:test:python-docs --path writers/`; `grep -rn -iE "wave [0-9]+" docs/docs/` prints nothing. -- [ ] **Binding parity unaffected:** `pmtiles_gbx` is a writer (a DataSource format), not a registered SQL/Scala function — it is NOT added to `registered_functions.txt`/`function-info.json`. Confirm `gbx:test:bindings` still passes unchanged. -- [ ] **Heavy parity on-cluster (optional, operator):** run `test_pmtiles_parity.py` in the heavy Docker/cluster env to confirm decoded-tile parity. -- [ ] **Bench on-cluster (optional, operator):** time `run_pmtiles_write` light vs heavy; record the ratio. - ---- - -## Self-Review notes (plan vs spec) - -- **Spec coverage:** framework `grid/_header/backend/catalog/shard` → Tasks 4–8; `pmtiles_gbx` DataSource (write/commit/abort, options, single+sharded+overview) → Task 9; entries-driven commit + fixed + adaptive sharding → Task 8 + 9; STAC default + tilejson → Task 7; overview.pmtiles → Task 9; nested layout → Task 9/13; migration precursor → Tasks 2–3; register unification → Task 9/10; pmtiles dep → Task 1; Serverless guard extension → Tasks 3,10; bench → Task 12; docs → Task 13; light-vs-heavy parity → Task 11. -- **Deferred per spec (not in this plan):** COG-by-quadbin + VRT catalog + `QuadbinGrid`; MVT-dir/MBTiles backends; bottom-up raster pyramid / global-scaling / sparse-skip / resampling buffer; light PMTiles reader; full STAC-spec compliance + meta-PMTiles; `pyvx`/`pygx`. `SlippyGrid` is the only `Grid` implementation built now (YAGNI — `QuadbinGrid` lands with the COG backend). -- **Type consistency:** `Entry`, `ScratchWriter`, `assign_shards`, `stream_sorted`, `OVERVIEW` (shard.py); `HeaderInfo.header_dict()`, `build_header_info`, `sniff_tile_type` (_header.py); `PMTilesBackend.assemble(sorted_tiles, header_info, out_path)` (backend.py); `ShardInfo(rel_path, min_zoom, max_zoom, bbox)`, `STACManifestCatalog`, `TileJSONCatalog` (catalog.py); `SlippyGrid.parent(z,x,y,shard_zoom)` clamps `sz=min(shard_zoom,z)` — used consistently in shard assignment and pmtiles.py. -- **Verify-during-impl (from spec) folded into steps:** pmtiles `finalize` header keys confirmed against the installed package (Tasks 5–6 round-trip); ascending-tileid requirement handled by `stream_sorted` (Task 8); scratch naming uuid-unique + cleaned on commit/abort (Tasks 8–9); empty-input returns without writing (Task 9 `commit` guard). diff --git a/docs/superpowers/plans/2026-06-11-light-raster-writer.md b/docs/superpowers/plans/2026-06-11-light-raster-writer.md deleted file mode 100644 index ad1c271b9..000000000 --- a/docs/superpowers/plans/2026-06-11-light-raster-writer.md +++ /dev/null @@ -1,641 +0,0 @@ -# Light Raster Writer Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Bring the pure-Python `pyrx` raster writer to functional parity with the heavy `gdal`/`gtiff_gdal` writer, on both light formats (`raster_gbx` catch-all + `gtiff_gbx` named). - -**Architecture:** A new `pyrx/ds/_write.py` holds the per-tile byte production (`tile_to_bytes`): hybrid — verbatim when the target driver is GTiff (the dominant case; `raster_gbx`/`gtiff_gbx` tiles are already GTiff), rasterio re-encode + `RASTERX_*` tag stamping when `tile.metadata` requests a non-GTiff driver. `RasterGbxWriter` (DataSource V2) stays thin: schema check, filename derivation (`nameCol` or content-hash+uuid), `append`/`overwrite` mode, delegating bytes to the helper. Both `RasterGbxDataSource` (driver from `tile.metadata`) and `GTiffGbxDataSource` (force GTiff) expose `writer()`. Writer `.option()`s are `path`/`nameCol`/`ext` only — encoding settings come from `tile.metadata`, exactly like heavy. - -**Tech Stack:** Python 3.12, PySpark 4.1.2 (`pyspark.sql.datasource`), rasterio, pytest. Docs are a separate plan. - -**Reference spec:** `docs/superpowers/specs/2026-06-11-light-raster-writer-design.md` (read "Heavy writer contract" + "Write contract (light)"). Companion reader spec/impl already merged on this branch. - ---- - -## Ground-truth facts (verified — do not re-derive) - -- **Heavy writer `.option()`s:** `path`, `nameCol`, `ext` (default `"tif"`). Nothing else. -- **Encoding from `tile.metadata`** (heavy `OperatorOptions.appendOptions(mtd)`): `format`/`driver` (default `GTiff`), `compression` (default `DEFLATE`), `blocksize` (default `"512"`), `zlevel` (default `"6"`), `zstd_level` (default `"9"`); PREDICTOR=3 for float dtypes else 2. -- **Heavy always re-encodes** via `gdal_translate` and stamps `RASTERX_` (each metadata entry) + `RASTERX_CELL` = cellid. -- **Filenames:** `nameCol` → `{row[nameCol]}.{ext}`; else `{MurmurHash3(tile)}_{pid}_{tid}.{ext}`. Flat dir, one file per row. -- **Light tile schema (single source):** `pyrx._serde.TILE_SCHEMA` — `(cellid: long, raster: binary, metadata: map)`. Reader/writer schema `(source, tile)` via `pyrx.ds.raster.reader_schema()` — **import it, never redeclare.** -- **Parity is decoded-pixel within tolerance, NOT byte-for-byte** (independent GDAL stacks). -- **Current `writer.py`** (post-revert): `RasterGbxWriter` writes `tile.raster` verbatim to `raster_{uuid}.tif`; `assert_write_schema`; overwrite clears `*.tif` in `__init__`; commit no-op; abort removes written files. Wired only into `gtiff_gbx`. - -## File structure - -| File | Responsibility | -|---|---| -| `python/geobrix/src/databricks/labs/gbx/pyrx/ds/_write.py` | **New.** `tile_to_bytes()` (hybrid verbatim/re-encode + RASTERX_* tags) + encoding/creation-option helpers. No Spark. | -| `.../pyrx/ds/writer.py` | **Rework.** `RasterGbxWriter` with `nameCol`/`ext`/`force_driver`/mode; filename derivation; delegates bytes to `_write`. | -| `.../pyrx/ds/raster.py` | **Modify.** `RasterGbxDataSource.writer()` (catch-all, `force_driver=None`). | -| `.../pyrx/ds/gtiff.py` | **Modify.** `GTiffGbxDataSource.writer()` passes `force_driver="GTiff"`, `nameCol`, `ext`. | -| `python/geobrix/test/pyrx/ds/test_write_helper.py` | **New.** Unit tests for `_write.tile_to_bytes` (no Spark). | -| `.../test/pyrx/ds/test_writer.py` | **Extend.** Integration: nameCol, ext, catch-all-vs-named, re-encode, mode. | -| `.../test/pyrx/ds/test_writer_parity.py` | **New.** Light-vs-heavy round-trip (Docker/integration, skip-if-heavy-unavailable). | -| `.../test/pyrx/test_serverless_no_spark_config.py` | **Modify.** Add `_write.py` to the covered-files guard. | - -Local venv for non-Docker tests: `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate`. **Before EVERY commit:** `chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects`. Do NOT push. Pre-commit banner is normal. - ---- - -## Task 1: `_write.tile_to_bytes` (per-tile byte production) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/_write.py` -- Test: `python/geobrix/test/pyrx/ds/test_write_helper.py` - -- [ ] **Step 1: Write the failing test** - -Create `python/geobrix/test/pyrx/ds/test_write_helper.py`: - -```python -"""Unit tests for the writer's per-tile byte production (no Spark).""" -import numpy as np -from rasterio.io import MemoryFile - -from databricks.labs.gbx.pyrx.ds import _write - - -def _gtiff_bytes(width=4, height=3, dtype="float32"): - from rasterio.transform import from_origin - data = np.arange(width * height, dtype=dtype).reshape(height, width) - profile = dict(driver="GTiff", width=width, height=height, count=1, dtype=dtype, - crs="EPSG:4326", transform=from_origin(10.0, 50.0, 0.5, 0.5)) - with MemoryFile() as mf: - with mf.open(**profile) as ds: - ds.write(data, 1) - return mf.read() - - -def test_gtiff_target_is_verbatim(): - raw = _gtiff_bytes() - meta = {"driver": "GTiff", "format": "GTiff", "compression": "DEFLATE"} - out = _write.tile_to_bytes(cellid=-1, raster_bytes=raw, metadata=meta, force_driver=None) - assert out == raw # GTiff target -> bytes passed through unchanged - - -def test_force_gtiff_is_verbatim_even_if_metadata_says_otherwise(): - raw = _gtiff_bytes() - meta = {"driver": "COG", "format": "COG"} - out = _write.tile_to_bytes(cellid=-1, raster_bytes=raw, metadata=meta, force_driver="GTiff") - assert out == raw # gtiff_gbx forces GTiff -> verbatim - - -def test_non_gtiff_target_reencodes_same_pixels_with_tags(): - raw = _gtiff_bytes() - meta = {"driver": "COG", "format": "COG", "compression": "DEFLATE"} - out = _write.tile_to_bytes(cellid=7, raster_bytes=raw, metadata=meta, force_driver=None) - assert out != raw # re-encoded - with MemoryFile(out) as mf, mf.open() as ds: - assert ds.driver in ("COG", "GTiff") # COG driver writes a GTiff-structured file - arr = ds.read(1) - tags = ds.tags() - np.testing.assert_allclose(arr, np.arange(12, dtype="float32").reshape(3, 4), rtol=1e-6) - assert tags.get("RASTERX_CELL") == "7" - assert tags.get("RASTERX_driver") == "COG" -``` - -- [ ] **Step 2: Run it and confirm it fails** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_write_helper.py -v` -Expected: FAIL — no module `pyrx.ds._write`. - -- [ ] **Step 3: Implement `_write.py`** - -Create `python/geobrix/src/databricks/labs/gbx/pyrx/ds/_write.py`: - -```python -"""Per-tile byte production for the raster writer. - -Hybrid, mirroring the heavy gdal writer's intent (encoding from tile.metadata): -- target driver GTiff (the common case; raster_gbx/gtiff_gbx tiles are already - GTiff) -> pass tile.raster bytes through VERBATIM. Pixel-identical to heavy; - heavy's specific creation options differ but our contract is decoded-pixel. -- non-GTiff target -> rasterio re-encode to that driver applying the - metadata-derived compression/blocksize/zlevel/zstd, and stamp RASTERX_ - (each metadata entry) + RASTERX_CELL (cellid), matching heavy SetMetadataItem. - -Writer .option()s never carry encoding; only tile.metadata does (like heavy). -""" -from __future__ import annotations - -from typing import Dict - - -def _is_float(dtype: str) -> bool: - return str(dtype).startswith("float") - - -def _creation_opts(driver: str, meta: Dict[str, str], dtype: str) -> Dict[str, str]: - """GTiff/COG creation options from tile metadata, mirroring OperatorOptions.appendOptions.""" - compression = str(meta.get("compression", "DEFLATE")).upper() - opts: Dict[str, str] = {"compress": compression} - if compression == "DEFLATE": - opts["zlevel"] = str(meta.get("zlevel", "6")) - opts["predictor"] = "3" if _is_float(dtype) else "2" - elif compression == "ZSTD": - opts["zstd_level"] = str(meta.get("zstd_level", "9")) - elif compression == "LZW": - opts["predictor"] = "3" if _is_float(dtype) else "2" - # blocksize: floor to mult-of-16, clamped >=64, applied for COG/GTiff family - try: - blk = int(meta.get("blocksize", "512")) - except ValueError: - blk = 512 - blk = max(64, (blk // 16) * 16) - if driver.upper() == "COG": - opts["blocksize"] = str(blk) - return opts - - -def tile_to_bytes( - cellid: int, - raster_bytes: bytes, - metadata: Dict[str, str], - force_driver: str = None, -) -> bytes: - """Return the on-disk bytes for one tile (verbatim GTiff, else re-encode).""" - driver = force_driver or metadata.get("driver") or metadata.get("format") or "GTiff" - if str(driver).upper() == "GTIFF": - return raster_bytes # already GTiff -> verbatim (fast, pixel-exact) - - import rasterio - from rasterio.io import MemoryFile - - with MemoryFile(raster_bytes) as src_mf, src_mf.open() as src: - data = src.read() - profile = src.profile.copy() - profile["driver"] = driver - profile.update(_creation_opts(driver, metadata, src.dtypes[0])) - with MemoryFile() as out_mf: - with out_mf.open(**profile) as dst: - dst.write(data) - tags = {f"RASTERX_{k}": str(v) for k, v in (metadata or {}).items()} - tags["RASTERX_CELL"] = str(cellid) - dst.update_tags(**tags) - return out_mf.read() -``` - -- [ ] **Step 4: Run it and confirm it passes** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_write_helper.py -v` -Expected: PASS (3 tests). If COG driver is unavailable in the local GDAL, the re-encode test will error on `out_mf.open(driver="COG")` — if so, change the test's `driver` to `"GTiff"` is NOT valid (that's verbatim); instead skip with `pytest.importorskip`-style guard: wrap the COG open in try/except and `pytest.skip("COG driver unavailable")`. Note which path you took. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add python/geobrix/src/databricks/labs/gbx/pyrx/ds/_write.py python/geobrix/test/pyrx/ds/test_write_helper.py -git commit -m "feat(pyrx-ds): writer byte production (verbatim GTiff / re-encode + RASTERX tags)" -``` - ---- - -## Task 2: Rework `RasterGbxWriter` (options, filenames, mode) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/writer.py` -- Test: `python/geobrix/test/pyrx/ds/test_writer.py` (extend) - -- [ ] **Step 1: Write the failing tests** (append to `test_writer.py`) - -First read the existing `test_writer.py` to reuse its `_write_sample` + imports. Append: - -```python -def test_namecol_controls_filenames(spark, tmp_path): - from pyspark.sql import functions as F - src = tmp_path / "in.tif" - _write_sample(str(src)) - out_dir = tmp_path / "out_named" - spark.dataSource.register(RasterGbxDataSource) - spark.dataSource.register(GTiffGbxDataSource) - df = spark.read.format("raster_gbx").load(str(src)).withColumn("source", F.lit("mytile")) - df.write.format("gtiff_gbx").mode("overwrite").option("nameCol", "source").save(str(out_dir)) - import os - files = [f for f in os.listdir(out_dir) if f.endswith(".tif")] - assert files == ["mytile.tif"] - - -def test_ext_option_controls_suffix(spark, tmp_path): - src = tmp_path / "in.tif" - _write_sample(str(src)) - out_dir = tmp_path / "out_ext" - spark.dataSource.register(RasterGbxDataSource) - spark.dataSource.register(GTiffGbxDataSource) - df = spark.read.format("raster_gbx").load(str(src)) - df.write.format("gtiff_gbx").mode("overwrite").option("ext", "tiff").save(str(out_dir)) - import os - assert all(f.endswith(".tiff") for f in os.listdir(out_dir)) -``` - -- [ ] **Step 2: Run and confirm failure** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_writer.py -v` -Expected: FAIL — `nameCol`/`ext` not honored (writer ignores them today) and/or `GTiffGbxDataSource` not imported in test (add `from databricks.labs.gbx.pyrx.ds.gtiff import GTiffGbxDataSource` and `from databricks.labs.gbx.pyrx.ds.raster import RasterGbxDataSource` at top if missing). - -- [ ] **Step 3: Rework `writer.py`** - -Replace the body of `python/geobrix/src/databricks/labs/gbx/pyrx/ds/writer.py` with: - -```python -"""gtiff_gbx / raster_gbx writer (DataSource V2 write path). - -Enforces the exact (source, tile) schema like the heavy GDAL writer. Writer -options are path/nameCol/ext only; the on-disk encoding comes from tile.metadata -(see _write.tile_to_bytes). Pure Python (Serverless). -""" -from __future__ import annotations - -import glob -import hashlib -import os -import uuid -from dataclasses import dataclass -from typing import Iterator, List, Optional - -from pyspark.sql.datasource import DataSourceWriter, WriterCommitMessage -from pyspark.sql.types import StructType - -from databricks.labs.gbx.pyrx.ds import _write -from databricks.labs.gbx.pyrx.ds.raster import reader_schema - - -@dataclass -class RasterCommitMessage(WriterCommitMessage): - paths: List[str] - - -def assert_write_schema(schema: StructType) -> None: - """Exact (source, tile) — extras OR missing both fail (matches GDAL writer).""" - expected = reader_schema() - if [f.name for f in schema.fields] != [f.name for f in expected.fields]: - raise ValueError( - f"raster writer requires exactly columns " - f"{[f.name for f in expected.fields]}, got {[f.name for f in schema.fields]}" - ) - - -def _safe_name(raster_bytes: bytes, cellid: int) -> str: - """Opaque, collision-free fallback name when no nameCol: content hash + uuid. - - PySpark's DataSourceWriter does not expose partition/task ids (Scala uses - pid_tid), so the uuid suffix keeps names unique across partitions. NOT - byte-identical to heavy's MurmurHash3_pid_tid -- use nameCol for control. - """ - h = hashlib.sha1(raster_bytes).hexdigest()[:12] - return f"{h}_{uuid.uuid4().hex[:8]}" - - -class RasterGbxWriter(DataSourceWriter): - def __init__( - self, - path: str, - schema: StructType, - overwrite: bool, - name_col: Optional[str] = None, - ext: str = "tif", - force_driver: Optional[str] = None, - ): - assert_write_schema(schema) - if name_col and name_col not in [f.name for f in schema.fields]: - raise ValueError( - f"nameCol {name_col!r} is not a column; available: " - f"{[f.name for f in schema.fields]} (overwrite 'source')." - ) - self.path = path - self.overwrite = overwrite - self.name_col = name_col - self.ext = ext - self.force_driver = force_driver - if overwrite and os.path.isdir(path): - for stale in glob.glob(os.path.join(path, f"*.{ext}")): - try: - os.remove(stale) - except OSError: - pass - - def write(self, iterator: Iterator) -> WriterCommitMessage: - os.makedirs(self.path, exist_ok=True) - written: List[str] = [] - for row in iterator: - tile = row["tile"] - cellid = tile["cellid"] - raster_bytes = bytes(tile["raster"]) - metadata = dict(tile["metadata"] or {}) - name = row[self.name_col] if self.name_col else _safe_name(raster_bytes, cellid) - out_bytes = _write.tile_to_bytes(cellid, raster_bytes, metadata, self.force_driver) - out = os.path.join(self.path, f"{name}.{self.ext}") - with open(out, "wb") as fh: - fh.write(out_bytes) - written.append(out) - return RasterCommitMessage(paths=written) - - def commit(self, messages: List[Optional[WriterCommitMessage]]) -> None: - return None - - def abort(self, messages: List[Optional[WriterCommitMessage]]) -> None: - for msg in messages: - if isinstance(msg, RasterCommitMessage): - for p in msg.paths: - try: - os.remove(p) - except OSError: - pass -``` - -- [ ] **Step 4: Run and confirm pass** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_writer.py -v` -Expected: PASS (existing round-trip/strict/overwrite tests + new nameCol/ext). Note: `gtiff.py` already wires the writer; Task 3 makes `raster_gbx` writable + passes nameCol/ext/force_driver through, so some new assertions depend on Task 3 — if the new tests fail on options not being passed, proceed to Task 3 then re-run. (Order: do Task 3 immediately after if needed.) - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add python/geobrix/src/databricks/labs/gbx/pyrx/ds/writer.py python/geobrix/test/pyrx/ds/test_writer.py -git commit -m "feat(pyrx-ds): writer nameCol/ext options + content-hash fallback names" -``` - ---- - -## Task 3: Wire `writer()` on both DataSources - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/gtiff.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/raster.py` -- Test: `python/geobrix/test/pyrx/ds/test_writer.py` (catch-all vs named + re-encode) - -- [ ] **Step 1: Write the failing tests** (append to `test_writer.py`) - -```python -def test_raster_gbx_catch_all_writer_round_trips(spark, tmp_path): - import numpy as np, rasterio - src = tmp_path / "in.tif" - _write_sample(str(src)) - out_dir = tmp_path / "out_catchall" - spark.dataSource.register(RasterGbxDataSource) - df = spark.read.format("raster_gbx").load(str(src)) - df.write.format("raster_gbx").mode("overwrite").save(str(out_dir)) # catch-all writer - import os - written = [f for f in os.listdir(out_dir) if f.endswith(".tif")] - assert len(written) == 1 - with rasterio.open(os.path.join(out_dir, written[0])) as ds: - arr = ds.read(1) - np.testing.assert_allclose(arr, np.arange(12, dtype="float32").reshape(3, 4), rtol=1e-6) -``` - -- [ ] **Step 2: Run and confirm failure** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_writer.py::test_raster_gbx_catch_all_writer_round_trips -v` -Expected: FAIL — `raster_gbx` has no writer (`DataSource.writer` not implemented). - -- [ ] **Step 3: Add `writer()` to both DataSources** - -In `python/geobrix/src/databricks/labs/gbx/pyrx/ds/raster.py`, add to `RasterGbxDataSource` (import `RasterGbxWriter` LAZILY inside the method to avoid the writer↔raster import cycle): - -```python - def writer(self, schema: StructType, overwrite: bool) -> "DataSourceWriter": - from pyspark.sql.datasource import DataSourceWriter # noqa: F401 - from databricks.labs.gbx.pyrx.ds.writer import RasterGbxWriter - - path = self.options.get("path") - if not path: - raise ValueError("raster_gbx writer requires an output path (.save(path)).") - return RasterGbxWriter( - path, schema, overwrite, - name_col=self.options.get("nameCol"), - ext=self.options.get("ext", "tif"), - force_driver=None, # catch-all: driver from tile.metadata - ) -``` - -In `python/geobrix/src/databricks/labs/gbx/pyrx/ds/gtiff.py`, replace the existing `writer()` so it threads nameCol/ext and forces GTiff: - -```python - def writer(self, schema: StructType, overwrite: bool) -> DataSourceWriter: - path = self.options.get("path") - if not path: - raise ValueError("gtiff_gbx writer requires an output path (.save(path)).") - return RasterGbxWriter( - path, schema, overwrite, - name_col=self.options.get("nameCol"), - ext=self.options.get("ext", "tif"), - force_driver="GTiff", # named writer forces GTiff output - ) -``` - -(Confirm `gtiff.py` imports `DataSourceWriter` and `RasterGbxWriter`; they were added when the writer was first wired — keep them.) - -- [ ] **Step 4: Run and confirm pass** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_writer.py -v` -Expected: PASS (all writer tests, both formats). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add python/geobrix/src/databricks/labs/gbx/pyrx/ds/raster.py python/geobrix/src/databricks/labs/gbx/pyrx/ds/gtiff.py python/geobrix/test/pyrx/ds/test_writer.py -git commit -m "feat(pyrx-ds): raster_gbx catch-all writer + gtiff_gbx forces GTiff" -``` - ---- - -## Task 4: Serverless guard covers `_write.py` - -**Files:** -- Modify: `python/geobrix/test/pyrx/test_serverless_no_spark_config.py` - -- [ ] **Step 1: Add `_write.py` to the required-files list** - -In `test_serverless_scan_includes_ds_subpackage`, add `"_write.py"` to the `required` tuple (alongside `raster.py`, `writer.py`, etc.). - -- [ ] **Step 2: Run** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/test_serverless_no_spark_config.py -v` -Expected: PASS — both the forbidden-pattern scan (now also covering `_write.py`; it must use no `.conf.set`/`._jvm`/`.sparkContext`/`.rdd` — the Task 1 code uses none) and the coverage guard. - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add python/geobrix/test/pyrx/test_serverless_no_spark_config.py -git commit -m "test(pyrx-ds): Serverless scan covers _write.py" -``` - ---- - -## Task 5: Re-encode integration test (non-GTiff via tile.metadata) - -**Files:** -- Test: `python/geobrix/test/pyrx/ds/test_writer.py` (append) - -- [ ] **Step 1: Write the test** - -```python -def test_metadata_driver_cog_triggers_reencode(spark, tmp_path): - import os - import numpy as np - import rasterio - from pyspark.sql import functions as F - src = tmp_path / "in.tif" - _write_sample(str(src)) - out_dir = tmp_path / "out_cog" - spark.dataSource.register(RasterGbxDataSource) - df = spark.read.format("raster_gbx").load(str(src)) - # Override the tile.metadata driver to COG so the catch-all writer re-encodes. - df2 = df.withColumn( - "tile", - F.col("tile").withField("metadata", F.map_concat( - F.col("tile.metadata"), F.create_map(F.lit("driver"), F.lit("COG")) - )), - ) - try: - df2.write.format("raster_gbx").mode("overwrite").save(str(out_dir)) - except Exception as e: - import pytest - pytest.skip(f"COG driver unavailable in this env: {str(e)[:80]}") - written = [f for f in os.listdir(out_dir) if f.endswith(".tif")] - assert len(written) == 1 - with rasterio.open(os.path.join(out_dir, written[0])) as ds: - arr = ds.read(1) - tags = ds.tags() - np.testing.assert_allclose(arr, np.arange(12, dtype="float32").reshape(3, 4), rtol=1e-6) - assert tags.get("RASTERX_CELL") == "-1" -``` - -- [ ] **Step 2: Run** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_writer.py::test_metadata_driver_cog_triggers_reencode -v` -Expected: PASS (or SKIP if COG unavailable locally; it runs in Docker where GDAL 3.11 has COG). - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add python/geobrix/test/pyrx/ds/test_writer.py -git commit -m "test(pyrx-ds): tile.metadata driver=COG triggers writer re-encode" -``` - ---- - -## Task 6: Light-vs-heavy round-trip parity (Docker / integration) - -**Files:** -- Create: `python/geobrix/test/pyrx/ds/test_writer_parity.py` - -- [ ] **Step 1: Write the parity test** (model the fixture on `test_reader_parity.py`) - -```python -"""Light-vs-heavy writer round-trip parity (Docker; needs JAR + sample data).""" -import logging -import os -from pathlib import Path - -import numpy as np -import pytest -import rasterio -from rasterio.io import MemoryFile - -pytestmark = pytest.mark.integration - -SAMPLE = os.environ.get( - "GBX_PARITY_SAMPLE", - "/Volumes/main/default/test-data/geobrix-examples/london/sentinel2/london_sentinel2_red.tif", -) -REL_TOL = 1e-3 -ABS_TOL = 1e-3 -_HERE = Path(__file__).resolve() -_JARS = sorted((_HERE.parents[3] / "lib").glob("geobrix-*-jar-with-dependencies.jar")) - - -@pytest.fixture(scope="module") -def spark_with_jar(): - if not _JARS: - pytest.skip("no geobrix JAR staged") - if not os.path.exists(SAMPLE): - pytest.skip(f"sample not mounted at {SAMPLE}") - from pyspark.sql import SparkSession - logging.getLogger("py4j").setLevel(logging.ERROR) - s = (SparkSession.builder.master("local[2]").appName("pyrx-ds-writer-parity") - .config("spark.driver.extraJavaOptions", - "-Djava.library.path=/usr/local/lib:/usr/lib:/usr/java/packages/lib:" - "/usr/lib64:/lib64:/lib:/usr/local/hadoop/lib/native") - .config("spark.jars", str(_JARS[-1])).getOrCreate()) - from databricks.labs.gbx.pyrx.ds.register import register - register(s) - yield s - - -def _decode(path): - with rasterio.open(path) as ds: - return ds.read() - - -def test_light_write_roundtrips_to_same_pixels_as_heavy(spark_with_jar): - light_dir = "/tmp/gbx_parity_light_out" - # Light: raster_gbx read -> gtiff_gbx write -> re-read - light = spark_with_jar.read.format("raster_gbx").load(SAMPLE) - light.write.format("gtiff_gbx").mode("overwrite").save(light_dir) - lf = [f for f in os.listdir(light_dir) if f.endswith(".tif")] - assert lf, "light writer produced no files" - la = _decode(os.path.join(light_dir, lf[0])) - # Ground truth: the source pixels (heavy write also re-encodes to same pixels) - with rasterio.open(SAMPLE) as src: - truth = src.read() - assert la.shape == truth.shape - np.testing.assert_allclose(la, truth, rtol=REL_TOL, atol=ABS_TOL) -``` - -NOTE: a true side-by-side vs the heavy `gtiff_gdal` writer is skipped because the heavy GDAL path doesn't run in the local dev container (documented in the reader parity work); the source raster is the pixel ground truth both tiers must match. If a JAR-backed heavy write *does* work in the target env, extend this to compare against `spark.read.format("gdal").load(SAMPLE).write.format("gtiff_gdal")` output and skip-on-failure like the reader parity test. - -- [ ] **Step 2: Run in Docker** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/ds/test_writer_parity.py --with-integration --log writer-parity.log` -Expected: PASS. First confirm the sample path exists in-container (`docker exec geobrix-dev ls `); adjust `GBX_PARITY_SAMPLE`/the default if needed (the reader parity work uses this same london sentinel2 path). - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add python/geobrix/test/pyrx/ds/test_writer_parity.py -git commit -m "test(pyrx-ds): light writer round-trip pixel parity (integration)" -``` - ---- - -## Task 7: Full `ds/` suite + lint in Docker - -- [ ] **Step 1: Run the suite (incl. integration) in Docker** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/ds/ --with-integration --log ds-writer-suite.log` -Expected: all pass (re-encode/COG + writer-parity may SKIP if a driver/sample is unavailable; otherwise pass). Fix any Docker-vs-venv discrepancy in the module, not the test. - -- [ ] **Step 2: Lint (CI parity)** - -Run: `bash scripts/commands/gbx-lint-python.sh --check --log writer-lint.log` -Expected: clean. If host formatting drifts, reformat in-container per the host-vs-Docker black caveat and re-run. - -- [ ] **Step 3: Commit any fixes** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add -A python/geobrix -git commit -m "chore(pyrx-ds): writer suite green + lint in Docker" -``` - ---- - -## Out of scope (this plan) - -- **Docs** (light reader+writer pages; heavy `gdal`/`gtiff_gdal` reader+writer option audit) — separate plan per the spec's "Implementation plans" note. -- Vector writers; byte-identical filename parity with heavy's `MurmurHash3_pid_tid`. - -## Self-review notes (for the executor) - -- **Import cycle:** `writer.py` imports `reader_schema` from `raster.py`; `raster.py.writer()` imports `RasterGbxWriter` **lazily** (inside the method) to avoid a cycle. `gtiff.py` imports `RasterGbxWriter` at module level (gtiff→writer→raster, no cycle). -- **Writer options are path/nameCol/ext only.** Encoding (driver/compression/blocksize/zlevel/zstd) comes from `tile.metadata` — never add them as `.option()`s. -- **Verbatim is the GTiff path; re-encode only for non-GTiff** `tile.metadata` driver. `gtiff_gbx` forces GTiff (always verbatim). -- **No `_jvm`/`sparkContext`/`.rdd`/`.conf.set`** in `_write.py`/`writer.py` (Serverless; Task 4 enforces). -- **Row access:** the writer receives PySpark `Row`s — `row["tile"]["cellid"]`, `bytes(row["tile"]["raster"])`, `row[name_col]`. If a Row-access form fails at runtime, adapt to the correct accessor and note it. diff --git a/docs/superpowers/plans/2026-06-11-light-readers-raster.md b/docs/superpowers/plans/2026-06-11-light-readers-raster.md deleted file mode 100644 index 9400cdec0..000000000 --- a/docs/superpowers/plans/2026-06-11-light-readers-raster.md +++ /dev/null @@ -1,1341 +0,0 @@ -# Light Readers — Raster (Python DataSource V2) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add pure-Python/PySpark raster readers (`raster_gbx` catch-all, `gtiff_gbx` named) and a writer built on PySpark 4.x DataSource V2, as a 1:1 swap-out for the GDAL-backed Scala `gdal`/`gtiff_gdal` readers, with a bench reader mode to prove perf + distribution. - -**Architecture:** A new `pyrx/ds/` subpackage. Pure-Python primitives (tile-grid sizing, GTiff re-encode + metadata, recursive listing) are composed by a `DataSourceReader` whose `partitions()` emits one partition per file and whose `read()` tiles+re-encodes each file into `(source, tile)` rows matching `pyrx._serde.TILE_SCHEMA`. The named reader subclasses the catch-all and injects a driver preset (mirrors Scala `dsExtraMap`). A `DataSourceWriter` writes `tile.raster` GTiff bytes back out. Everything is pure Python (no `_jvm`/`sparkContext`/`.rdd`) for Serverless. A new `bench/readers.py` mode times light vs heavy reads over a corpus. - -**Tech Stack:** Python 3.12, PySpark 4.1.2 (`pyspark.sql.datasource`), rasterio (already in the `[pyrx]` extra), numpy, pytest. Heavy comparison runs in the `geobrix-dev` Docker container (needs the JAR + GDAL). - -**Reference spec:** `docs/superpowers/specs/2026-06-11-light-readers-raster-design.md` (read the "Revision 2026-06-11" + "Parity contract" sections — the parity model is decoded-pixel-array within tolerance, NOT byte-for-byte). - ---- - -## Execution status — COMPLETE (2026-06-11) - -All tasks done. 19 commits on `light-readers`; nothing pushed. - -- **T1–T8 — DONE.** Primitives + `raster_gbx`/`gtiff_gbx`/writer/`register` + Serverless guard. TDD, code-reviewed (critical fix: reuse `core.tiling` for split parity instead of a divergent raw-byte model), 21 tests green, lint clean. Commits `90192d7`→`3026ec4`, `340a69d`. -- **T9 — DONE.** `test_reader_parity.py` (`f6d40e6`,`1e7ccc9`,`4a18f34`): ground-truth pixel parity (light vs rasterio direct read) **passes in Docker**; light-vs-heavy compare **skips** locally (heavy `gdal` yields 0 tiles in local Docker — GDAL-init quirk) and runs on cluster. -- **T10 — DONE.** Full `ds/` suite in Docker: **20 passed, 1 skipped**; CI-parity lint (Docker black) clean. -- **T11 — DONE.** `bench/readers.py` + `gbx:bench:readers` + unit test. Commit `0168d96`. fn unified to `raster_read` for cross-tier pairing (`2818aba`). -- **T12 — DONE.** On-cluster light-vs-heavy reader bench wired into the cluster runner (`2818aba`) + launcher venv fix (`feed6ad`). Ran `run_id=readers-20260611` at 1000 tiles, both tiers, SUCCESS: **heavy `gdal` 8.98 s vs light `raster_gbx` 24.13 s → light ~2.7× slower** (perf follow-up; see spec Performance section). Heavy reader confirmed working on-cluster. - ---- - -## Ground-truth facts (verified against Scala source — do not re-derive) - -- **Schema:** `source: string` + `tile: struct{cellid: long, raster: binary, metadata: map}`. Single source of truth: `python/geobrix/src/databricks/labs/gbx/pyrx/_serde.py::TILE_SCHEMA` (`cellid` LongType non-null, `raster` BinaryType non-null, `metadata` MapType(String,String) nullable). **Always import it — never redeclare.** -- **cellid:** `-1` on every emitted tile (`GDAL_Reader.scala:30` writes `-1L`). -- **tile.raster:** re-encoded **GTiff, DEFLATE** (`RasterDriver.writeToBytes` coerces to GTiff regardless of source). Not raw bytes. -- **One row per tile.** Tiling = `BalancedSubdivision.getTileSize` (power-of-4 split by `sizeInMB`, default 16) → grid of windows. Sub-16MB raster → 1 tile → 1 row. -- **metadata (11 keys):** `path, sourcePath, driver, format, last_command, last_error, all_parents, size, compression, isZipped, isSubset`. Fixed values from heavy: `driver="GTiff"`, `format="GTiff"`, `last_error=""`, `size="-1"`, `compression="DEFLATE"`, `isZipped="false"`, `isSubset="false"`, `last_command="windowed_extract -srcwin "`, `all_parents=";"`, `sourcePath=`, `path=`. -- **Options:** `path` (required), `sizeInMB` (default `"16"`), `filterRegex` (default `".*"`). Recursive **regex** match on full path, not glob. -- **Corrupt file:** fail-fast, no `ignoreCorruptFiles` (`GDAL_Reader.scala:17`). -- **Named reader:** Scala `GTiff_DataSource.shortName()=="gtiff_gdal"`, injects `driver->"GTiff"`. Our light names: catch-all `raster_gbx`, named `gtiff_gbx`. - -## PySpark DataSource V2 API (verified, PySpark 4.1.2) - -`from pyspark.sql.datasource import DataSource, DataSourceReader, DataSourceWriter, InputPartition, WriterCommitMessage` - -- `DataSource.__init__(self, options: Dict[str,str])` → stores `self.options`. -- `@classmethod DataSource.name(cls) -> str` -- `DataSource.schema(self) -> Union[StructType, str]` -- `DataSource.reader(self, schema: StructType) -> DataSourceReader` -- `DataSource.writer(self, schema: StructType, overwrite: bool) -> DataSourceWriter` -- `DataSourceReader.partitions(self) -> Sequence[InputPartition]` -- `@abstractmethod DataSourceReader.read(self, partition) -> Iterator[Tuple]` (yields **tuples**, not Row; tuple field order = schema field order) -- `InputPartition(value)` — subclass with extra attrs; **must be picklable**. -- `@abstractmethod DataSourceWriter.write(self, iterator: Iterator[Row]) -> WriterCommitMessage` (receives **Row** objects) -- `DataSourceWriter.commit(self, messages) -> None` / `abort(self, messages) -> None` -- Register: `spark.dataSource.register(MyDataSourceClass)` (pass the **class**). - -## File structure - -| File | Responsibility | -|---|---| -| `python/geobrix/src/databricks/labs/gbx/pyrx/ds/__init__.py` | Subpackage marker + re-exports | -| `.../pyrx/ds/_tiling.py` | Port of `BalancedSubdivision` tile-grid math → list of windows | -| `.../pyrx/ds/_encode.py` | Windowed-read + GTiff(DEFLATE) re-encode + 11-key metadata → `(cellid, bytes, meta)` | -| `.../pyrx/ds/_listing.py` | Recursive path listing with `filterRegex` | -| `.../pyrx/ds/raster.py` | `RasterGbxDataSource` (`raster_gbx`) + reader + partition | -| `.../pyrx/ds/gtiff.py` | `GTiffGbxDataSource` (`gtiff_gbx`) — subclass + driver preset | -| `.../pyrx/ds/writer.py` | `RasterGbxWriter` + `RasterCommitMessage` | -| `.../pyrx/ds/register.py` | `register(spark)` + opportunistic-on-import guard | -| `python/geobrix/src/databricks/labs/gbx/bench/readers.py` | Reader bench mode (light vs heavy) | -| Tests under `python/geobrix/test/pyrx/ds/` | one test file per module | - -All test commands run in Docker: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/ds/ --log .log`. For a single node, append `::test_name` to the path. Local-venv runs (`source .venv-pyrx/bin/activate && pytest ...`) are fine for the pure-Python phases (1-3) that need no JAR. - ---- - -## Task 1: Tile-grid math (`_tiling.py`) - -Port `BalancedSubdivision.getTileSize` + grid enumeration as a pure function. No rasterio, no Spark — just integer math, so it's trivially testable and matches heavy row-count. - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/__init__.py` -- Create: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/_tiling.py` -- Test: `python/geobrix/test/pyrx/ds/__init__.py`, `python/geobrix/test/pyrx/ds/test_tiling.py` - -- [ ] **Step 1: Write the failing test** - -Create `python/geobrix/test/pyrx/ds/__init__.py` (empty) and `python/geobrix/test/pyrx/ds/test_tiling.py`: - -```python -"""Unit tests for the BalancedSubdivision port (pure integer math).""" -from databricks.labs.gbx.pyrx.ds import _tiling - - -def _bytes_per_pixel(dtype: str) -> int: - import numpy as np - return np.dtype(dtype).itemsize - - -def test_small_raster_is_single_tile(): - # 4x3 float32 single band = 48 bytes << 16 MiB -> one tile covering whole raster - windows = _tiling.plan_windows(width=4, height=3, bands=1, dtype="float32", size_mib=16) - assert windows == [(0, 0, 4, 3)] - - -def test_tile_count_is_power_of_four_when_split(): - # Force a split: 4096x4096 float64 x4 bands ~= 512 MiB, size limit 16 MiB. - windows = _tiling.plan_windows(width=4096, height=4096, bands=4, dtype="float64", size_mib=16) - n = len(windows) - # nx == ny == 2^k -> count is a perfect square AND a power of four - side = int(round(n ** 0.5)) - assert side * side == n, f"{n} tiles is not a square grid" - assert (side & (side - 1)) == 0, f"side {side} is not a power of two" - - -def test_windows_tile_the_full_raster_without_gaps_or_overlap(): - width, height = 1000, 700 - windows = _tiling.plan_windows(width=width, height=height, bands=2, dtype="uint8", size_mib=1) - # Reconstruct coverage: every pixel covered exactly once. - covered = 0 - for col_off, row_off, win_w, win_h in windows: - assert col_off + win_w <= width - assert row_off + win_h <= height - covered += win_w * win_h - assert covered == width * height - - -def test_get_tile_size_matches_scala_ceil_div(): - # getTileSize(ds, destMiB) returns (tileX, tileY) = ceil(x/nx), ceil(y/ny) - nx, ny, tile_x, tile_y = _tiling.tile_grid(width=1000, height=700, bands=2, dtype="uint8", size_mib=1) - assert tile_x == -(-1000 // nx) # ceil div - assert tile_y == -(-700 // ny) -``` - -- [ ] **Step 2: Run it and confirm it fails** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_tiling.py -v` -Expected: FAIL — `ModuleNotFoundError: ... pyrx.ds._tiling` (or `__init__` missing). - -- [ ] **Step 3: Implement `_tiling.py`** - -Create `python/geobrix/src/databricks/labs/gbx/pyrx/ds/__init__.py`: - -```python -"""pyrx.ds — pure-Python/PySpark DataSource V2 raster readers + writer. - -Light-tier swap-out for the GDAL-backed Scala readers. See -docs/superpowers/specs/2026-06-11-light-readers-raster-design.md. -""" -``` - -Create `python/geobrix/src/databricks/labs/gbx/pyrx/ds/_tiling.py`: - -```python -"""Port of Scala BalancedSubdivision tile-grid math (power-of-4 split). - -Pure integer math so the light reader emits the SAME number of tiles per -raster as the heavy reader (row-count parity). Mirrors -``BalancedSubdivision.getTileSize`` in -src/main/scala/.../rasterx/operations/BalancedSubdivision.scala. -""" -from __future__ import annotations - -from typing import List, Tuple - -import numpy as np - - -def _mem_size_bytes(width: int, height: int, bands: int, dtype: str) -> int: - """In-memory size of the raster, matching RasterAccessors.memSize.""" - return width * height * bands * int(np.dtype(dtype).itemsize) - - -def _num_splits_k(width: int, height: int, bands: int, dtype: str, size_mib: int) -> int: - """Number of quad-split rounds k (nx=ny=2^k, tiles=4^k). Mirrors the Scala while-loop.""" - size_bytes = _mem_size_bytes(width, height, bands, dtype) - limit = size_mib * 1024 * 1024 - k = 0 - # while k<9 and (sizeBytes >> 2k) > limit and 4^(k+1) <= 512 - while k < 9 and (size_bytes >> (2 * k)) > limit and (1 << (2 * (k + 1))) <= 512: - k += 1 - return k - - -def tile_grid(width: int, height: int, bands: int, dtype: str, size_mib: int) -> Tuple[int, int, int, int]: - """Return (nx, ny, tile_x, tile_y): grid divisions and per-tile pixel dims (ceil-div).""" - k = _num_splits_k(width, height, bands, dtype, size_mib) - nx = 1 << k - ny = 1 << k - tile_x = -(-width // nx) # ceil div - tile_y = -(-height // ny) - return nx, ny, tile_x, tile_y - - -def plan_windows(width: int, height: int, bands: int, dtype: str, size_mib: int) -> List[Tuple[int, int, int, int]]: - """List of (col_off, row_off, win_w, win_h) windows tiling the raster, no gaps/overlap.""" - _nx, _ny, tile_x, tile_y = tile_grid(width, height, bands, dtype, size_mib) - windows: List[Tuple[int, int, int, int]] = [] - row_off = 0 - while row_off < height: - win_h = min(tile_y, height - row_off) - col_off = 0 - while col_off < width: - win_w = min(tile_x, width - col_off) - windows.append((col_off, row_off, win_w, win_h)) - col_off += tile_x - row_off += tile_y - return windows -``` - -- [ ] **Step 4: Run it and confirm it passes** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_tiling.py -v` -Expected: PASS (4 tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyrx/ds/__init__.py \ - python/geobrix/src/databricks/labs/gbx/pyrx/ds/_tiling.py \ - python/geobrix/test/pyrx/ds/__init__.py \ - python/geobrix/test/pyrx/ds/test_tiling.py -git commit -m "feat(pyrx-ds): port BalancedSubdivision tile-grid math" -``` - ---- - -## Task 2: Tile re-encode + metadata (`_encode.py`) - -Given an open rasterio dataset and one window, windowed-read the pixels, write an in-memory GTiff (DEFLATE), and build the 11-key metadata map. Returns `(cellid=-1, gtiff_bytes, metadata)`. - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/_encode.py` -- Test: `python/geobrix/test/pyrx/ds/test_encode.py` - -- [ ] **Step 1: Write the failing test** - -Create `python/geobrix/test/pyrx/ds/test_encode.py`: - -```python -"""Unit tests for windowed GTiff re-encode + metadata.""" -import numpy as np -import rasterio -from rasterio.io import MemoryFile - -from databricks.labs.gbx.pyrx.ds import _encode - -# 11 keys the heavy reader emits (WindowedExtract.scala:108-119) -EXPECTED_METADATA_KEYS = { - "path", "sourcePath", "driver", "format", "last_command", "last_error", - "all_parents", "size", "compression", "isZipped", "isSubset", -} - - -def test_encode_tile_roundtrips_pixels(gtiff_bytes): - # gtiff_bytes fixture (conftest): 4x3 float32 single band, values arange(12) - with MemoryFile(gtiff_bytes) as mf, mf.open() as ds: - cellid, raster_bytes, meta = _encode.encode_tile( - ds, window=(0, 0, 4, 3), source_path="/data/sample.tif", all_parents="" - ) - assert cellid == -1 - with MemoryFile(raster_bytes) as mf2, mf2.open() as ds2: - assert ds2.count == 1 - assert (ds2.width, ds2.height) == (4, 3) - out = ds2.read(1) - expected = np.arange(12, dtype="float32").reshape(3, 4) - np.testing.assert_allclose(out, expected, rtol=1e-6) - - -def test_encode_metadata_key_set(gtiff_bytes): - with MemoryFile(gtiff_bytes) as mf, mf.open() as ds: - _cellid, _b, meta = _encode.encode_tile( - ds, window=(0, 0, 4, 3), source_path="/data/sample.tif", all_parents="" - ) - assert set(meta.keys()) == EXPECTED_METADATA_KEYS - # fixed-value parity with heavy - assert meta["driver"] == "GTiff" - assert meta["format"] == "GTiff" - assert meta["compression"] == "DEFLATE" - assert meta["isZipped"] == "false" - assert meta["isSubset"] == "false" - assert meta["last_error"] == "" - assert meta["sourcePath"] == "/data/sample.tif" - assert meta["last_command"] == "windowed_extract -srcwin 0 0 4 3" - - -def test_encode_subwindow_reads_only_that_window(gtiff_bytes): - with MemoryFile(gtiff_bytes) as mf, mf.open() as ds: - _c, raster_bytes, _m = _encode.encode_tile( - ds, window=(2, 0, 2, 3), source_path="/data/sample.tif", all_parents="" - ) - with MemoryFile(raster_bytes) as mf2, mf2.open() as ds2: - assert (ds2.width, ds2.height) == (2, 3) - out = ds2.read(1) - full = np.arange(12, dtype="float32").reshape(3, 4) - np.testing.assert_allclose(out, full[:, 2:4], rtol=1e-6) -``` - -NOTE: the `gtiff_bytes` fixture already exists in `python/geobrix/test/pyrx/conftest.py` (session-scoped, 4x3 float32). To make it visible under `test/pyrx/ds/`, conftest fixtures in a parent dir are auto-inherited by pytest — no action needed. - -- [ ] **Step 2: Run it and confirm it fails** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_encode.py -v` -Expected: FAIL — `_encode` has no `encode_tile`. - -- [ ] **Step 3: Implement `_encode.py`** - -Create `python/geobrix/src/databricks/labs/gbx/pyrx/ds/_encode.py`: - -```python -"""Windowed GTiff(DEFLATE) re-encode + 11-key metadata, matching the heavy reader. - -Mirrors RasterDriver.writeToBytes (always GTiff/DEFLATE on the wire) and -WindowedExtract metadata. tile.raster is NOT raw source bytes. -""" -from __future__ import annotations - -from typing import Dict, Tuple - -import rasterio -from rasterio.io import MemoryFile -from rasterio.windows import Window - -CELLID_FRESH = -1 # GDAL_Reader.scala:30 writes -1L for un-tessellated tiles - - -def encode_tile( - ds: "rasterio.DatasetReader", - window: Tuple[int, int, int, int], - source_path: str, - all_parents: str, - compression: str = "DEFLATE", -) -> Tuple[int, bytes, Dict[str, str]]: - """Read one window, re-encode it as an in-memory GTiff, return (cellid, bytes, metadata).""" - col_off, row_off, win_w, win_h = window - rio_window = Window(col_off, row_off, win_w, win_h) - data = ds.read(window=rio_window) # (bands, h, w) - - profile = ds.profile.copy() - profile.update( - driver="GTiff", - width=win_w, - height=win_h, - compress=compression.lower(), - transform=ds.window_transform(rio_window), - ) - # nodata/dtype/count/crs carried from source profile. - - with MemoryFile() as mf: - with mf.open(**profile) as out: - out.write(data) - raster_bytes = mf.read() - - metadata = { - "path": f"/vsimem/light_{abs(hash((source_path, col_off, row_off))) & 0xffffffff}.tif", - "sourcePath": source_path, - "driver": "GTiff", - "format": "GTiff", - "last_command": f"windowed_extract -srcwin {col_off} {row_off} {win_w} {win_h}", - "last_error": "", - "all_parents": f"{source_path};{all_parents}", - "size": "-1", - "compression": compression, - "isZipped": "false", - "isSubset": "false", - } - return CELLID_FRESH, raster_bytes, metadata -``` - -- [ ] **Step 4: Run it and confirm it passes** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_encode.py -v` -Expected: PASS (3 tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyrx/ds/_encode.py \ - python/geobrix/test/pyrx/ds/test_encode.py -git commit -m "feat(pyrx-ds): windowed GTiff re-encode + 11-key metadata" -``` - ---- - -## Task 3: Recursive path listing (`_listing.py`) - -Mirror `HadoopUtils.listAllHadoopFiles(path, conf, filterRegex)`: recursively walk a dir (or accept a single file), keep paths whose full string matches `filterRegex`. - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/_listing.py` -- Test: `python/geobrix/test/pyrx/ds/test_listing.py` - -- [ ] **Step 1: Write the failing test** - -Create `python/geobrix/test/pyrx/ds/test_listing.py`: - -```python -"""Unit tests for recursive path listing with regex filter.""" -import os - -import pytest - -from databricks.labs.gbx.pyrx.ds import _listing - - -@pytest.fixture -def tree(tmp_path): - (tmp_path / "a").mkdir() - (tmp_path / "a" / "one.tif").write_bytes(b"x") - (tmp_path / "a" / "two.tif").write_bytes(b"x") - (tmp_path / "a" / "skip.txt").write_bytes(b"x") - (tmp_path / "b").mkdir() - (tmp_path / "b" / "three.tif").write_bytes(b"x") - return tmp_path - - -def test_lists_all_files_recursively_default_regex(tree): - files = _listing.list_files(str(tree), filter_regex=".*") - assert len(files) == 4 - assert all(os.path.isabs(f) for f in files) - - -def test_regex_filters_by_full_path(tree): - files = _listing.list_files(str(tree), filter_regex=r".*\.tif$") - assert len(files) == 3 - assert all(f.endswith(".tif") for f in files) - - -def test_single_file_path_returns_that_file(tree): - target = str(tree / "a" / "one.tif") - files = _listing.list_files(target, filter_regex=".*") - assert files == [target] - - -def test_no_match_raises(tree): - with pytest.raises(FileNotFoundError): - _listing.list_files(str(tree), filter_regex=r".*\.nope$") -``` - -- [ ] **Step 2: Run it and confirm it fails** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_listing.py -v` -Expected: FAIL — no `list_files`. - -- [ ] **Step 3: Implement `_listing.py`** - -Create `python/geobrix/src/databricks/labs/gbx/pyrx/ds/_listing.py`: - -```python -"""Recursive file listing with a regex filter (mirrors HadoopUtils.listAllHadoopFiles). - -Local-filesystem only — fits FUSE-mounted UC Volumes (/Volumes/...). Returns -sorted absolute paths so partition ordering is deterministic. -""" -from __future__ import annotations - -import os -import re -from typing import List - - -def list_files(path: str, filter_regex: str = ".*") -> List[str]: - """Return sorted absolute file paths under ``path`` whose full path matches ``filter_regex``.""" - pattern = re.compile(filter_regex) - abspath = os.path.abspath(path) - - if os.path.isfile(abspath): - candidates = [abspath] if pattern.match(abspath) else [] - else: - candidates = [] - for root, _dirs, names in os.walk(abspath): - for name in names: - full = os.path.join(root, name) - if pattern.match(full): - candidates.append(full) - - if not candidates: - raise FileNotFoundError( - f"No files under {path!r} matched filterRegex {filter_regex!r}" - ) - return sorted(candidates) -``` - -- [ ] **Step 4: Run it and confirm it passes** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_listing.py -v` -Expected: PASS (4 tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyrx/ds/_listing.py \ - python/geobrix/test/pyrx/ds/test_listing.py -git commit -m "feat(pyrx-ds): recursive path listing with regex filter" -``` - ---- - -## Task 4: Catch-all `raster_gbx` DataSource - -Compose the primitives into a `DataSource`/`DataSourceReader`. `partitions()` lists files (driver), `read()` tiles+encodes one file (executor), yielding tuples `(source, (cellid, raster, metadata))` in `TILE_SCHEMA` order. - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/raster.py` -- Test: `python/geobrix/test/pyrx/ds/test_raster_datasource.py` - -- [ ] **Step 1: Write the failing test** - -Create `python/geobrix/test/pyrx/ds/test_raster_datasource.py`: - -```python -"""Integration tests for the raster_gbx DataSource (uses local Spark).""" -import numpy as np -import rasterio -from rasterio.io import MemoryFile - -from databricks.labs.gbx.pyrx.ds.raster import RasterGbxDataSource - -EXPECTED_METADATA_KEYS = { - "path", "sourcePath", "driver", "format", "last_command", "last_error", - "all_parents", "size", "compression", "isZipped", "isSubset", -} - - -def _write_sample(path, width=4, height=3): - from rasterio.transform import from_origin - data = np.arange(width * height, dtype="float32").reshape(height, width) - profile = dict(driver="GTiff", width=width, height=height, count=1, - dtype="float32", crs="EPSG:4326", - transform=from_origin(10.0, 50.0, 0.5, 0.5), nodata=-9999.0) - with rasterio.open(path, "w", **profile) as ds: - ds.write(data, 1) - - -def test_schema_matches_tile_schema(): - from databricks.labs.gbx.pyrx import _serde - ds = RasterGbxDataSource(options={"path": "/tmp/none"}) - schema = ds.schema() - assert [f.name for f in schema.fields] == ["source", "tile"] - assert schema["tile"].dataType == _serde.TILE_SCHEMA - - -def test_read_single_file_yields_one_row(spark, tmp_path): - f = tmp_path / "sample.tif" - _write_sample(str(f)) - spark.dataSource.register(RasterGbxDataSource) - df = spark.read.format("raster_gbx").load(str(f)) - rows = df.collect() - assert len(rows) == 1 # < 16 MiB -> 1 tile - row = rows[0] - assert row["source"] == str(f) - assert row["tile"]["cellid"] == -1 - assert set(row["tile"]["metadata"].keys()) == EXPECTED_METADATA_KEYS - # decode the re-encoded GTiff and check pixels - with MemoryFile(bytes(row["tile"]["raster"])) as mf, mf.open() as out: - arr = out.read(1) - np.testing.assert_allclose(arr, np.arange(12, dtype="float32").reshape(3, 4), rtol=1e-6) - - -def test_read_directory_one_partition_per_file(spark, tmp_path): - for i in range(3): - _write_sample(str(tmp_path / f"s{i}.tif")) - spark.dataSource.register(RasterGbxDataSource) - df = spark.read.format("raster_gbx").option("filterRegex", r".*\.tif$").load(str(tmp_path)) - assert df.rdd.getNumPartitions() == 3 # one partition per file - assert df.count() == 3 - - -def test_corrupt_file_fails_fast(spark, tmp_path): - bad = tmp_path / "bad.tif" - bad.write_bytes(b"not a raster") - spark.dataSource.register(RasterGbxDataSource) - df = spark.read.format("raster_gbx").load(str(bad)) - import pytest - with pytest.raises(Exception): - df.collect() -``` - -- [ ] **Step 2: Run it and confirm it fails** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_raster_datasource.py -v` -Expected: FAIL — no `raster` module / `RasterGbxDataSource`. - -- [ ] **Step 3: Implement `raster.py`** - -Create `python/geobrix/src/databricks/labs/gbx/pyrx/ds/raster.py`: - -```python -"""raster_gbx — catch-all pure-Python DataSource V2 raster reader. - -1:1 swap-out for the Scala ``gdal`` reader: recursively lists files, splits each -into BalancedSubdivision tiles, re-encodes each tile as GTiff, emits -(source, tile) rows matching pyrx._serde.TILE_SCHEMA. Pure Python (Serverless). -""" -from __future__ import annotations - -from typing import Dict, Iterator, Sequence, Tuple - -from pyspark.sql.datasource import DataSource, DataSourceReader, InputPartition -from pyspark.sql.types import StructField, StructType, StringType - -from databricks.labs.gbx.pyrx import _serde -from databricks.labs.gbx.pyrx.ds import _encode, _listing, _tiling - - -def reader_schema() -> StructType: - """(source, tile) — tile from the single-source TILE_SCHEMA.""" - return StructType([ - StructField("source", StringType(), nullable=False), - StructField("tile", _serde.TILE_SCHEMA, nullable=False), - ]) - - -class _FilePartition(InputPartition): - """One source file = one partition (picklable).""" - - def __init__(self, file_path: str, size_mib: int): - self.file_path = file_path - self.size_mib = size_mib - - -class RasterGbxReader(DataSourceReader): - def __init__(self, options: Dict[str, str]): - self.path = options.get("path") - if not self.path: - raise ValueError("raster_gbx requires a 'path' (e.g. .load(path)).") - self.size_mib = int(options.get("sizeInMB", "16")) - self.filter_regex = options.get("filterRegex", ".*") - - def partitions(self) -> Sequence[InputPartition]: - files = _listing.list_files(self.path, self.filter_regex) - return [_FilePartition(f, self.size_mib) for f in files] - - def read(self, partition: "_FilePartition") -> Iterator[Tuple]: - # rasterio imported inside read() so the import lands on executors. - import rasterio - - from databricks.labs.gbx.pyrx import _env - _env.configure_gdal_env() # worker-side GDAL/PROJ env (matches pyrx UDF pattern) - - with rasterio.open(partition.file_path) as ds: - windows = _tiling.plan_windows( - width=ds.width, height=ds.height, bands=ds.count, - dtype=ds.dtypes[0], size_mib=partition.size_mib, - ) - for win in windows: - cellid, raster_bytes, meta = _encode.encode_tile( - ds, window=win, source_path=partition.file_path, all_parents="", - ) - yield (partition.file_path, (cellid, raster_bytes, meta)) - - -class RasterGbxDataSource(DataSource): - @classmethod - def name(cls) -> str: - return "raster_gbx" - - def schema(self) -> StructType: - return reader_schema() - - def reader(self, schema: StructType) -> DataSourceReader: - return RasterGbxReader(self.options) -``` - -- [ ] **Step 4: Run it and confirm it passes** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_raster_datasource.py -v` -Expected: PASS (4 tests). If the local `spark` fixture is missing under `test/pyrx/ds/`, it is inherited from `test/pyrx/conftest.py` (module-scoped `local[2]`); confirm by running. If Arrow/typing errors appear, ensure the tuple field order is `(source, (cellid, raster, metadata))`. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyrx/ds/raster.py \ - python/geobrix/test/pyrx/ds/test_raster_datasource.py -git commit -m "feat(pyrx-ds): raster_gbx catch-all DataSource V2 reader" -``` - ---- - -## Task 5: Named `gtiff_gbx` reader - -Subclass the catch-all, override `name()`, and inject a `driver="GTiff"` preset into options (the light analogue of Scala `dsExtraMap`). - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/gtiff.py` -- Test: `python/geobrix/test/pyrx/ds/test_gtiff_datasource.py` - -- [ ] **Step 1: Write the failing test** - -Create `python/geobrix/test/pyrx/ds/test_gtiff_datasource.py`: - -```python -"""gtiff_gbx named reader: same output as raster_gbx with driver preset.""" -import numpy as np -import rasterio -from rasterio.transform import from_origin - -from databricks.labs.gbx.pyrx.ds.gtiff import GTiffGbxDataSource - - -def _write_sample(path): - data = np.arange(12, dtype="float32").reshape(3, 4) - profile = dict(driver="GTiff", width=4, height=3, count=1, dtype="float32", - crs="EPSG:4326", transform=from_origin(10.0, 50.0, 0.5, 0.5)) - with rasterio.open(path, "w", **profile) as ds: - ds.write(data, 1) - - -def test_name_is_gtiff_gbx(): - assert GTiffGbxDataSource.name() == "gtiff_gbx" - - -def test_driver_preset_injected(): - ds = GTiffGbxDataSource(options={"path": "/tmp/x"}) - reader = ds.reader(ds.schema()) - assert reader.driver == "GTiff" - - -def test_reads_geotiff_like_catch_all(spark, tmp_path): - f = tmp_path / "s.tif" - _write_sample(str(f)) - spark.dataSource.register(GTiffGbxDataSource) - df = spark.read.format("gtiff_gbx").load(str(f)) - rows = df.collect() - assert len(rows) == 1 - assert rows[0]["tile"]["metadata"]["driver"] == "GTiff" - assert rows[0]["tile"]["cellid"] == -1 -``` - -- [ ] **Step 2: Run it and confirm it fails** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_gtiff_datasource.py -v` -Expected: FAIL — no `gtiff` module. - -- [ ] **Step 3: Implement `gtiff.py`** - -Create `python/geobrix/src/databricks/labs/gbx/pyrx/ds/gtiff.py`: - -```python -"""gtiff_gbx — named GeoTIFF reader. Light analogue of Scala GTiff_DataSource: -extends the catch-all and presets driver="GTiff" (the dsExtraMap mirror). -""" -from __future__ import annotations - -from typing import Dict - -from pyspark.sql.datasource import DataSourceReader -from pyspark.sql.types import StructType - -from databricks.labs.gbx.pyrx.ds.raster import RasterGbxDataSource, RasterGbxReader - - -class GTiffGbxReader(RasterGbxReader): - def __init__(self, options: Dict[str, str]): - super().__init__(options) - self.driver = "GTiff" # preset; rasterio detects the driver, kept for parity/metadata - - -class GTiffGbxDataSource(RasterGbxDataSource): - @classmethod - def name(cls) -> str: - return "gtiff_gbx" - - def reader(self, schema: StructType) -> DataSourceReader: - return GTiffGbxReader(self.options) -``` - -- [ ] **Step 4: Run it and confirm it passes** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_gtiff_datasource.py -v` -Expected: PASS (3 tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyrx/ds/gtiff.py \ - python/geobrix/test/pyrx/ds/test_gtiff_datasource.py -git commit -m "feat(pyrx-ds): gtiff_gbx named reader (driver preset)" -``` - ---- - -## Task 6: Writer (`writer.py`) - -DataSource write path: enforce the exact `(source, tile)` schema, write each row's `tile.raster` GTiff bytes to a file under the output path, `commit`/`abort`. - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/writer.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/gtiff.py` (add `writer()` + `schema` validation) -- Test: `python/geobrix/test/pyrx/ds/test_writer.py` - -- [ ] **Step 1: Write the failing test** - -Create `python/geobrix/test/pyrx/ds/test_writer.py`: - -```python -"""Round-trip: raster_gbx read -> gtiff_gbx write -> re-read; + strict schema.""" -import os - -import numpy as np -import rasterio -from rasterio.io import MemoryFile -from rasterio.transform import from_origin - -from databricks.labs.gbx.pyrx.ds.gtiff import GTiffGbxDataSource -from databricks.labs.gbx.pyrx.ds.raster import RasterGbxDataSource - - -def _write_sample(path): - data = np.arange(12, dtype="float32").reshape(3, 4) - profile = dict(driver="GTiff", width=4, height=3, count=1, dtype="float32", - crs="EPSG:4326", transform=from_origin(10.0, 50.0, 0.5, 0.5)) - with rasterio.open(path, "w", **profile) as ds: - ds.write(data, 1) - - -def test_round_trip(spark, tmp_path): - src = tmp_path / "in.tif" - _write_sample(str(src)) - out_dir = tmp_path / "out" - spark.dataSource.register(RasterGbxDataSource) - spark.dataSource.register(GTiffGbxDataSource) - - df = spark.read.format("raster_gbx").load(str(src)) - df.write.format("gtiff_gbx").mode("overwrite").save(str(out_dir)) - - written = [f for f in os.listdir(out_dir) if f.endswith(".tif")] - assert len(written) == 1 - with rasterio.open(os.path.join(out_dir, written[0])) as ds: - arr = ds.read(1) - np.testing.assert_allclose(arr, np.arange(12, dtype="float32").reshape(3, 4), rtol=1e-6) - - -def test_strict_schema_rejects_extra_columns(spark, tmp_path): - import pytest - from pyspark.sql import functions as F - src = tmp_path / "in.tif" - _write_sample(str(src)) - spark.dataSource.register(RasterGbxDataSource) - spark.dataSource.register(GTiffGbxDataSource) - df = spark.read.format("raster_gbx").load(str(src)).withColumn("extra", F.lit(1)) - with pytest.raises(Exception): - df.write.format("gtiff_gbx").mode("overwrite").save(str(tmp_path / "o2")) -``` - -- [ ] **Step 2: Run it and confirm it fails** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_writer.py -v` -Expected: FAIL — `gtiff_gbx` has no writer. - -- [ ] **Step 3: Implement `writer.py` and wire it into `gtiff.py`** - -Create `python/geobrix/src/databricks/labs/gbx/pyrx/ds/writer.py`: - -```python -"""gtiff_gbx writer (DataSource V2 write path). - -Enforces the exact (source, tile) schema like the heavy GDAL writer, writes each -row's tile.raster GTiff bytes to a file under the output path. Pure Python. -""" -from __future__ import annotations - -import os -import uuid -from dataclasses import dataclass -from typing import Iterator, List, Optional - -from pyspark.sql.datasource import DataSourceWriter, WriterCommitMessage -from pyspark.sql.types import StructType - -from databricks.labs.gbx.pyrx.ds.raster import reader_schema - - -@dataclass -class RasterCommitMessage(WriterCommitMessage): - paths: List[str] - - -def assert_write_schema(schema: StructType) -> None: - """Exact (source, tile) — extras OR missing both fail (matches GDAL writer).""" - expected = reader_schema() - if [f.name for f in schema.fields] != [f.name for f in expected.fields]: - raise ValueError( - f"gtiff_gbx writer requires exactly columns " - f"{[f.name for f in expected.fields]}, got {[f.name for f in schema.fields]}" - ) - - -class RasterGbxWriter(DataSourceWriter): - def __init__(self, path: str, schema: StructType, overwrite: bool): - assert_write_schema(schema) - self.path = path - self.overwrite = overwrite - - def write(self, iterator: Iterator) -> WriterCommitMessage: - os.makedirs(self.path, exist_ok=True) - written: List[str] = [] - for row in iterator: - raster_bytes = bytes(row["tile"]["raster"]) - out = os.path.join(self.path, f"raster_{uuid.uuid4().hex}.tif") - with open(out, "wb") as fh: - fh.write(raster_bytes) - written.append(out) - return RasterCommitMessage(paths=written) - - def commit(self, messages: List[Optional[WriterCommitMessage]]) -> None: - return None # files already durable; nothing to finalize - - def abort(self, messages: List[Optional[WriterCommitMessage]]) -> None: - for msg in messages: - if isinstance(msg, RasterCommitMessage): - for p in msg.paths: - try: - os.remove(p) - except OSError: - pass -``` - -Modify `python/geobrix/src/databricks/labs/gbx/pyrx/ds/gtiff.py` — add the writer method to `GTiffGbxDataSource` (append the import and method): - -```python -# add at top imports: -from pyspark.sql.datasource import DataSourceWriter -from databricks.labs.gbx.pyrx.ds.writer import RasterGbxWriter - -# add inside GTiffGbxDataSource: - def writer(self, schema: StructType, overwrite: bool) -> DataSourceWriter: - path = self.options.get("path") - if not path: - raise ValueError("gtiff_gbx writer requires an output path (.save(path)).") - return RasterGbxWriter(path, schema, overwrite) -``` - -- [ ] **Step 4: Run it and confirm it passes** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_writer.py -v` -Expected: PASS (2 tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyrx/ds/writer.py \ - python/geobrix/src/databricks/labs/gbx/pyrx/ds/gtiff.py \ - python/geobrix/test/pyrx/ds/test_writer.py -git commit -m "feat(pyrx-ds): gtiff_gbx DataSource V2 writer + strict schema" -``` - ---- - -## Task 7: `register(spark)` + opportunistic import - -Mirror `pyrx.functions.register`: a `register(spark)` that registers all light DataSources, plus an opportunistic attempt on `pyrx.ds` import guarded for no-active-session. - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/register.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/ds/__init__.py` (re-export + opportunistic register) -- Test: `python/geobrix/test/pyrx/ds/test_register.py` - -- [ ] **Step 1: Write the failing test** - -Create `python/geobrix/test/pyrx/ds/test_register.py`: - -```python -"""register(spark) makes all light raster formats resolvable.""" -import numpy as np -import rasterio -from rasterio.transform import from_origin - -from databricks.labs.gbx.pyrx.ds import register as ds_register - - -def _write_sample(path): - data = np.arange(12, dtype="float32").reshape(3, 4) - profile = dict(driver="GTiff", width=4, height=3, count=1, dtype="float32", - crs="EPSG:4326", transform=from_origin(10.0, 50.0, 0.5, 0.5)) - with rasterio.open(path, "w", **profile) as ds: - ds.write(data, 1) - - -def test_register_makes_both_formats_loadable(spark, tmp_path): - f = tmp_path / "s.tif" - _write_sample(str(f)) - ds_register.register(spark) - assert spark.read.format("raster_gbx").load(str(f)).count() == 1 - assert spark.read.format("gtiff_gbx").load(str(f)).count() == 1 -``` - -- [ ] **Step 2: Run it and confirm it fails** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_register.py -v` -Expected: FAIL — no `register` module. - -- [ ] **Step 3: Implement `register.py` and update `__init__.py`** - -Create `python/geobrix/src/databricks/labs/gbx/pyrx/ds/register.py`: - -```python -"""Register the light raster DataSources with a Spark session. - -Mirrors pyrx.functions.register: call once, consciously. The format strings -raster_gbx / gtiff_gbx do not collide with the Scala-registered gdal / -gtiff_gdal, so both tiers coexist. -""" -from __future__ import annotations - -from pyspark.sql import SparkSession - -from databricks.labs.gbx.pyrx.ds.gtiff import GTiffGbxDataSource -from databricks.labs.gbx.pyrx.ds.raster import RasterGbxDataSource - -_SOURCES = (RasterGbxDataSource, GTiffGbxDataSource) - - -def register(spark: SparkSession = None) -> None: - """Register raster_gbx + gtiff_gbx. Uses the active session if not given.""" - if spark is None: - spark = SparkSession.builder.getOrCreate() - for source in _SOURCES: - spark.dataSource.register(source) - - -def _try_register_on_import() -> None: - """Best-effort register if a session is already live (no-op otherwise).""" - try: - spark = SparkSession.getActiveSession() - if spark is not None: - register(spark) - except Exception: - pass # never fail an import; explicit register() remains available -``` - -Append to `python/geobrix/src/databricks/labs/gbx/pyrx/ds/__init__.py`: - -```python -from databricks.labs.gbx.pyrx.ds.register import register # noqa: E402,F401 -from databricks.labs.gbx.pyrx.ds.register import _try_register_on_import - -_try_register_on_import() -``` - -- [ ] **Step 4: Run it and confirm it passes** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/ds/test_register.py -v` -Expected: PASS (1 test). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyrx/ds/register.py \ - python/geobrix/src/databricks/labs/gbx/pyrx/ds/__init__.py \ - python/geobrix/test/pyrx/ds/test_register.py -git commit -m "feat(pyrx-ds): register(spark) + opportunistic import" -``` - ---- - -## Task 8: Serverless guard covers `ds/` - -Confirm the existing Serverless scan (`test_serverless_no_spark_config.py`) walks `pyrx/ds/*.py`. The scan globs all `*.py` under `pyrx/`; add an explicit assertion that the new dir is covered so a future refactor can't silently drop it. - -**Files:** -- Modify: `python/geobrix/test/pyrx/test_serverless_no_spark_config.py` - -- [ ] **Step 1: Read the current scan + add a guard test** - -First inspect how `_pyrx_source_files()` collects files (it uses `rglob("*.py")` over the pyrx package root — confirm). Then append: - -```python -def test_serverless_scan_includes_ds_subpackage(): - files = {p.name for p in _pyrx_source_files()} - # the new DataSource modules must be in scope of the Serverless scan - for required in ("raster.py", "gtiff.py", "writer.py", "register.py", - "_tiling.py", "_encode.py", "_listing.py"): - assert required in files, f"{required} not covered by Serverless scan" -``` - -- [ ] **Step 2: Run it** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/pyrx/test_serverless_no_spark_config.py -v` -Expected: PASS — both the existing forbidden-pattern test (now also scanning `ds/`) and the new coverage guard. If the forbidden-pattern test FAILS, a `ds/` module used a banned call (`.conf.set`, `._jvm`, `.sparkContext`, `.rdd`) — fix the module (none of the Task 1-7 code uses these). - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/pyrx/test_serverless_no_spark_config.py -git commit -m "test(pyrx-ds): assert Serverless scan covers ds/ subpackage" -``` - ---- - -## Task 9: Light-vs-heavy parity test (Docker / integration) - -The real swap-out proof: same sample file through heavy `gdal` and light `raster_gbx`, decode both and compare pixel arrays within tolerance. Needs the JAR + GDAL → Docker only; mark `integration`. - -**Files:** -- Create: `python/geobrix/test/pyrx/ds/test_reader_parity.py` - -- [ ] **Step 1: Write the parity test** - -Create `python/geobrix/test/pyrx/ds/test_reader_parity.py`: - -```python -"""Light vs heavy reader parity (Docker; needs JAR + sample data).""" -import numpy as np -import pytest -import rasterio -from rasterio.io import MemoryFile - -pytestmark = pytest.mark.integration - -SAMPLE = "/Volumes/main/geobrix_samples/geobrix-examples/london/sentinel2.tif" # adjust to an existing sample -REL_TOL = 1e-3 -ABS_TOL = 1e-3 - - -def _decode(raster_bytes): - with MemoryFile(bytes(raster_bytes)) as mf, mf.open() as ds: - return ds.read() # (bands, h, w) - - -def test_raster_gbx_matches_gdal(spark_with_jar): - # spark_with_jar: a session with the geobrix JAR on the classpath (Docker conftest). - from databricks.labs.gbx.pyrx.ds.register import register - register(spark_with_jar) - - heavy = spark_with_jar.read.format("gdal").load(SAMPLE).orderBy("source").collect() - light = spark_with_jar.read.format("raster_gbx").load(SAMPLE).orderBy("source").collect() - - assert len(light) == len(heavy), "tile/row count differs" - for h, l in zip(heavy, light): - assert l["tile"]["cellid"] == -1 == h["tile"]["cellid"] - assert set(l["tile"]["metadata"].keys()) == set(h["tile"]["metadata"].keys()) - ha, la = _decode(h["tile"]["raster"]), _decode(l["tile"]["raster"]) - assert ha.shape == la.shape, "tile pixel dims differ" - np.testing.assert_allclose(la, ha, rtol=REL_TOL, atol=ABS_TOL) -``` - -NOTE for the implementer: (a) pick an actual sample file that exists in the Volume — list `/Volumes/main/geobrix_samples/...` first and fix `SAMPLE`. (b) If no `spark_with_jar` fixture exists, the Docker conftest builds a session with the JAR; reuse however other heavy/integration pyrx tests obtain a JAR-backed session (grep `test/pyrx` for `.format("gdal")` usage). (c) If row counts differ, the tiling port (Task 1) needs reconciliation with `BalancedSubdivision` for that file's size/dtype — debug `_num_splits_k` against `memSize`. - -- [ ] **Step 2: Run it in Docker** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/ds/test_reader_parity.py --with-integration --log reader-parity.log` -Expected: PASS. Watch the log; report progress every ~30s per the repo convention. - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/pyrx/ds/test_reader_parity.py -git commit -m "test(pyrx-ds): light-vs-heavy raster reader pixel-parity (integration)" -``` - ---- - -## Task 10: Full ds/ suite green in Docker - -Run the entire new suite in the canonical Docker path (the parity definition of done for unit tests). - -- [ ] **Step 1: Run the suite** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/ds/ --log ds-suite.log` -Expected: all non-integration tests PASS. Fix any Docker-vs-venv discrepancy (e.g. GDAL env) in the relevant module, not in the test. - -- [ ] **Step 2: Lint** - -Run: `bash scripts/commands/gbx-lint-python.sh --fix` then verify with the in-container check per the host-vs-Docker black caveat. Re-run the suite if reformatted. - -- [ ] **Step 3: Commit any fixes** - -```bash -chmod -R u+rwX .git/objects -git add -A python/geobrix -git commit -m "chore(pyrx-ds): lint + Docker suite green" -``` - ---- - -## Task 11: Bench reader mode (`bench/readers.py`) - -Add a reader bench reusing `results.ResultRow` / `store` / `compare`. Two surfaces: pure-local (single-file open+tile+encode, light rasterio vs heavy via the JAR path) and spark-path (N files distributed). Light meaningfully slower than heavy is a deprecation blocker (record the ratio like the function bench). - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/bench/readers.py` -- Create: `scripts/commands/gbx-bench-readers.md`, `scripts/commands/gbx-bench-readers.sh` -- Test: `python/geobrix/test/bench/test_readers_bench.py` - -- [ ] **Step 1: Write a unit test for the pure-local timing path** - -First grep `python/geobrix/test/bench/` for the existing bench test style and the `time_iters` import path. Create `python/geobrix/test/bench/test_readers_bench.py`: - -```python -"""Unit test: reader bench pure-local path produces a ResultRow with timing.""" -import numpy as np -import rasterio -from rasterio.transform import from_origin - -from databricks.labs.gbx.bench import readers - - -def _write_sample(path): - data = np.arange(12, dtype="float32").reshape(3, 4) - profile = dict(driver="GTiff", width=4, height=3, count=1, dtype="float32", - crs="EPSG:4326", transform=from_origin(10.0, 50.0, 0.5, 0.5)) - with rasterio.open(path, "w", **profile) as ds: - ds.write(data, 1) - - -def test_pure_local_reader_bench_emits_result(tmp_path): - f = tmp_path / "s.tif" - _write_sample(str(f)) - rows = readers.run_pure_local_reader( - files=[str(f)], run_id="t", warmup=1, measured=3, size_mib=16, - ) - assert len(rows) == 1 - r = rows[0] - assert r.api == "lightweight" - assert r.fn == "raster_gbx_read" - assert r.mode == "pure-core" - assert r.iter_median_s >= 0.0 - assert r.status == "ok" -``` - -- [ ] **Step 2: Run it and confirm it fails** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/bench/test_readers_bench.py -v` -Expected: FAIL — no `bench.readers`. - -- [ ] **Step 3: Implement `bench/readers.py`** - -Create `python/geobrix/src/databricks/labs/gbx/bench/readers.py`. Reuse `time_iters` from `runner` and `ResultRow` from `results` (confirm exact import paths by reading those modules). Skeleton with the real timing call: - -```python -"""Reader bench: time the light raster reader (and compare to heavy gdal). - -Pure-local surface times the per-file open+tile+encode in-process (no Spark). -Spark-path surface times a distributed read over a corpus. Emits results.ResultRow -so it shares store/compare plumbing with the function bench. -""" -from __future__ import annotations - -from typing import List - -from databricks.labs.gbx.bench import results -from databricks.labs.gbx.bench.runner import time_iters -from databricks.labs.gbx.bench.env import describe_env # confirm this helper name in env/results -from databricks.labs.gbx.pyrx.ds import _encode, _tiling - - -def _read_one_file_light(file_path: str, size_mib: int) -> int: - """Open + tile + re-encode one file; return tile count (forces the work).""" - import rasterio - n = 0 - with rasterio.open(file_path) as ds: - windows = _tiling.plan_windows(ds.width, ds.height, ds.count, ds.dtypes[0], size_mib) - for win in windows: - _encode.encode_tile(ds, window=win, source_path=file_path, all_parents="") - n += 1 - return n - - -def run_pure_local_reader(files: List[str], run_id: str, warmup: int, measured: int, - size_mib: int = 16) -> List[results.ResultRow]: - rows: List[results.ResultRow] = [] - for f in files: - stats = time_iters(lambda: _read_one_file_light(f, size_mib), warmup, measured) - rows.append(_to_result_row(run_id, f, stats)) # build ResultRow from stats + env - return rows -``` - -The implementer must fill `_to_result_row` using the actual `ResultRow` field names from `results.py` (run_id, api="lightweight", fn="raster_gbx_read", category="reader", mode="pure-core", iter_median_s from `stats["iter_median_ms"]/1000`, status="ok", plus the env_* fields from the env helper). Read `runner.py::run_pure_core` for the exact ResultRow construction pattern and copy it. Add a `run_spark_path_reader(spark, path, run_id, ...)` that times `spark.read.format("raster_gbx").load(path).count()` (or `.foreach`), recording `rows` and `iter_median_s`, mirroring how `runner.py` times spark-path functions. - -- [ ] **Step 4: Run the unit test and confirm it passes** - -Run: `source .venv-pyrx/bin/activate && python -m pytest python/geobrix/test/bench/test_readers_bench.py -v` -Expected: PASS. - -- [ ] **Step 5: Add the `gbx:bench:readers` command** - -Create `scripts/commands/gbx-bench-readers.md` (title, description, usage `bash scripts/commands/gbx-bench-readers.sh [OPTIONS]`, options `--corpus`, `--mode {pure-local|spark-path|both}`, `--out`, `--run-id`, `--size-mib`, `--warmup`, `--measured`, `--with-heavy`, `--log`, `--help`, two examples). - -Create `scripts/commands/gbx-bench-readers.sh` modeled on `scripts/commands/gbx-bench-lightweight.sh`: source `common.sh`, resolve `SCRIPT_DIR`/`PROJECT_ROOT`, support `--help`, `--log` via `resolve_log_path`, `check_docker` if `--mode spark-path`/`--with-heavy` (needs JAR), and invoke a Python entry (`python -m databricks.labs.gbx.bench.readers ...` — add a small `__main__`/argparse to `readers.py`) via `run_in_pyrx_venv`. No placeholders — implement real arg parsing and the python invocation. - -`chmod +x scripts/commands/gbx-bench-readers.sh`. - -- [ ] **Step 6: Smoke-run the command** - -Run: `bash scripts/commands/gbx-bench-readers.sh --mode pure-local --corpus --warmup 1 --measured 3 --log bench-readers.log` -Expected: writes ResultRows; prints a summary path. Verify the result store has `fn="raster_gbx_read"` rows. - -- [ ] **Step 7: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/bench/readers.py \ - python/geobrix/test/bench/test_readers_bench.py \ - scripts/commands/gbx-bench-readers.md scripts/commands/gbx-bench-readers.sh -git commit -m "feat(bench): reader bench mode (light raster reader timing)" -``` - ---- - -## Task 12: Cluster spark-path reader bench + perf gate (validation) - -Run the distributed reader bench on the bench cluster (light `raster_gbx` vs heavy `gdal`) at the standard scale, confirm distribution (partition count tracks file count) and the light-vs-heavy ratio. This is the perf-viability evidence for the swap-out. - -- [ ] **Step 1: Stage the wheel + run** - -Per the cluster-bench memories: build/stage the `[pyrx]` wheel (`gbx:data:push-wheel`), confirm the bench cluster is up (poll libraries INSTALLED), run `gbx:bench:readers --mode spark-path --with-heavy --corpus --run-id `. Give the run's `bench-out//summary.md` link at the end (per repo convention). Do NOT launch duplicate runs. - -- [ ] **Step 2: Record findings** - -Append a short findings note to the spec's Performance section (or a `prompts/refactoring/2026-06-..-reader-bench-findings.md`, gitignored): light-vs-heavy ratio for pure-local and spark-path, parallel-efficiency (per-partition spread), and whether the light reader meets the no-regression bar. If light is meaningfully slower, file a perf follow-up (deprecation blocker per the perf-parity memory). - -- [ ] **Step 3: Commit any spec/doc update** - -```bash -chmod -R u+rwX .git/objects -git add docs/superpowers/specs/2026-06-11-light-readers-raster-design.md -git commit -m "docs(spec): record reader bench light-vs-heavy findings" -``` - ---- - -## Out of scope (own follow-up plans) - -- Vector readers (`vector_gbx`, `shapefile_gbx`, `geojson_gbx`, `gpkg_gbx`, `file_gdb_gbx`) via pyogrio. -- `pygx` grid I/O. -- User-facing docs pages for the light readers (doc-tests under `docs/tests/`); add once the API stabilizes. The unit + parity tests here are the contract in the interim. -- Refactoring the 2 test-only DDL strings in `test_functions_spark.py` to import `TILE_SCHEMA` (latent drift; noted, not blocking). - -## Self-review notes (for the executor) - -- **Tuple order matters:** reader `read()` yields `(source, (cellid, raster, metadata))` — the inner tuple order must match `TILE_SCHEMA` field order (cellid, raster, metadata). -- **Import rasterio inside `read()`/`write()`** so it resolves on executors, and call `_env.configure_gdal_env()` worker-side (matches the pyrx UDF pattern). -- **Never redeclare the tile schema** — always `from databricks.labs.gbx.pyrx import _serde; _serde.TILE_SCHEMA`. -- **No `_jvm`/`sparkContext`/`.rdd`/`.conf.set`** anywhere in `ds/` (Serverless; Task 8 enforces). -- **Parity is pixel-level within tolerance, never byte-equal** (Task 9). diff --git a/docs/superpowers/plans/2026-06-11-raster-io-docs.md b/docs/superpowers/plans/2026-06-11-raster-io-docs.md deleted file mode 100644 index e461c0650..000000000 --- a/docs/superpowers/plans/2026-06-11-raster-io-docs.md +++ /dev/null @@ -1,557 +0,0 @@ -# Raster I/O Documentation Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Document the geobrix raster readers/writers across both tiers — net-new light `raster_gbx`/`gtiff_gbx` reader + writer pages, and an audit/fill of the heavy `gdal`/`gtiff_gdal` reader + writer option docs (esp. the tile-metadata-driven write encoding). - -**Architecture:** Per repo convention, **doc-tests are the documentation source**: real, asserting Python in `docs/tests/python/{readers,writers}/_examples.py` (display constants `FOO`/`FOO_output` + executable `test_*`/helper functions), imported into `.mdx` pages via `!!raw-loader!` + the `CodeFromTest` component. Doc-tests run in Docker (`gbx:test:python-docs`) on a JAR-backed Spark session; the light DataSources register on that session via `pyrx.ds.register.register(spark)`. - -**Tech Stack:** Docusaurus MDX + `CodeFromTest`, pytest doc-tests (Docker, `local[*]`), rasterio, the light `pyrx.ds` package (already implemented + merged on this branch). - -**Reference spec:** `docs/superpowers/specs/2026-06-11-light-raster-writer-design.md` → "Documentation" section. Companion: the reader spec `2026-06-11-light-readers-raster-design.md`. - ---- - -## Ground-truth facts (verified — do not re-derive) - -- **MDX import pattern:** `import xExamples from '!!raw-loader!../../tests/python//_examples.py';` then ``. `CodeFromTest` extracts the named constant by substring; `outputConstant` (optional) renders an "Example output" block. -- **Doc-test file:** display constants `READ_X = """..."""` + optional `READ_X_output = """..."""`; helper/`test_*` functions take the `spark` fixture and do real reads/writes. Sample path via `from path_config import SAMPLE_DATA_BASE`. -- **Sample raster:** `f"{SAMPLE_DATA_BASE}/nyc/sentinel2/nyc_sentinel2_red.tif"` (heavy docs use this; exists in the doc-test Volume mount). -- **Runner:** `bash scripts/commands/gbx-test-python-docs.sh --suite --skip-build` → `pytest docs/tests/python/... -m 'not integration'` in Docker. conftest `spark` fixture = JAR-backed `local[*]`, rasterx registered. The light DS registers fine on it (`spark.dataSource.register(...)`); the `[pyrx]` wheel is installed in the doc-test env by the runner build. -- **Sidebar:** `docs/sidebars.js` "Readers & Writers" category — `readers/*` + `writers/*` lists. **Overview tables:** `docs/docs/readers/overview.mdx`, `docs/docs/writers/overview.mdx`. -- **Heavy reader options already documented** (`readers/gdal.mdx`): `sizeInMB`, `filterRegex`, `readSubdatasets`, `rasterAsGrid`, `retile`, `tileSize`, `driver`. **Heavy writer options documented:** `ext`, `nameCol` + a "format comes from the tile" section. **Gap:** the tile-metadata write-encoding keys (`compression`/`blocksize`/`zlevel`/`zstd_level`/`format`) and an explicit `gtiff_gdal` *writer* example are NOT documented. -- **Stale line:** `docs/docs/api/execution-tiers.mdx` Readers/Writers row says "native Python Data Source readers planned" — now shipped. -- **QC gate:** `internals-leak` blocks `wave\s*\d+` in `docs/docs/`. Doc-tests must pass; MDX must build (`gbx:ci:docs`/`gbx:docs`). - -All commands run from repo root. **Before EVERY commit:** `chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects`. Do NOT push. Doc-tests run in Docker only. - -## File structure - -| File | Responsibility | -|---|---| -| `docs/tests/python/readers/raster_gbx_examples.py` | **New.** Light reader display constants + `test_*` (raster_gbx + gtiff_gbx reads). | -| `docs/tests/python/readers/test_raster_gbx_examples.py` | **New.** pytest entry calling the helpers. | -| `docs/docs/readers/raster_gbx.mdx` | **New.** Light reader page (both formats). | -| `docs/tests/python/writers/raster_gbx_examples.py` | **New.** Light writer constants + `test_*` (verbatim, nameCol, round-trip). | -| `docs/tests/python/writers/test_raster_gbx_examples.py` | **New.** pytest entry. | -| `docs/docs/writers/raster_gbx.mdx` | **New.** Light writer page. | -| `docs/docs/writers/gdal.mdx` | **Modify.** Document tile-metadata encoding keys + `gtiff_gdal` writer example. | -| `docs/tests/python/writers/gdal_examples.py` | **Modify.** Add a `gtiff_gdal` write display-constant + helper. | -| `docs/docs/readers/gdal.mdx`, `readers/gtiff.mdx` | **Modify (audit).** Ensure `path`/`driver` documented; complete the options tables. | -| `docs/docs/api/execution-tiers.mdx` | **Modify.** Un-stale the readers/writers row. | -| `docs/sidebars.js` | **Modify.** Add `readers/raster_gbx`, `writers/raster_gbx`. | -| `docs/docs/readers/overview.mdx`, `writers/overview.mdx` | **Modify.** Add a row each. | - ---- - -## Task 1: Light reader doc-test (`raster_gbx_examples.py`) - -**Files:** -- Create: `docs/tests/python/readers/raster_gbx_examples.py`, `docs/tests/python/readers/test_raster_gbx_examples.py` - -- [ ] **Step 1: Create the examples module** - -`docs/tests/python/readers/raster_gbx_examples.py`: - -```python -"""raster_gbx / gtiff_gbx (lightweight) Reader Examples — single source of truth. - -Code shown in docs/docs/readers/raster_gbx.mdx is imported from here. Pure-Python -DataSource V2 readers; no JAR required (registered via pyrx.ds.register). -""" -from path_config import SAMPLE_DATA_BASE - -SAMPLE_RASTER_PATH = f"{SAMPLE_DATA_BASE}/nyc/sentinel2/nyc_sentinel2_red.tif" - -REGISTER = """# Register the lightweight raster DataSources (once per session) -from databricks.labs.gbx.pyrx.ds.register import register -register(spark)""" - -READ_RASTER_GBX = """# Catch-all lightweight reader (any rasterio-readable raster) -df = spark.read.format("raster_gbx").load("{SAMPLE_RASTER_PATH}") -df.show()""" - -READ_RASTER_GBX_output = """+--------------------------------------------------+-----+ -|source |tile | -+--------------------------------------------------+-----+ -|/Volumes/.../nyc_sentinel2_red.tif |{...}| -+--------------------------------------------------+-----+""" - -READ_GTIFF_GBX = """# Named lightweight GeoTIFF reader (preset for GeoTIFF) -df = spark.read.format("gtiff_gbx").load("{SAMPLE_RASTER_PATH}")""" - -READ_WITH_OPTIONS = """# Options: sizeInMB (tile split threshold) + filterRegex (directory listing) -df = (spark.read.format("raster_gbx") - .option("sizeInMB", "16") - .option("filterRegex", r".*\\.tif$") - .load("{SAMPLE_RASTER_PATH}"))""" - - -def _register(spark): - from databricks.labs.gbx.pyrx.ds.register import register - register(spark) - - -def read_raster_gbx(spark, path=None): - """Verify READ_RASTER_GBX: catch-all reader yields (source, tile) rows.""" - _register(spark) - df = spark.read.format("raster_gbx").load(path or SAMPLE_RASTER_PATH) - assert [f.name for f in df.schema.fields] == ["source", "tile"] - rows = df.collect() - assert len(rows) >= 1 - assert rows[0]["tile"]["cellid"] == -1 - return df - - -def read_gtiff_gbx(spark, path=None): - """Verify READ_GTIFF_GBX: named reader reads a GeoTIFF identically.""" - _register(spark) - df = spark.read.format("gtiff_gbx").load(path or SAMPLE_RASTER_PATH) - assert df.count() >= 1 - assert df.collect()[0]["tile"]["metadata"]["driver"] == "GTiff" - return df -``` - -- [ ] **Step 2: Create the test entry** - -`docs/tests/python/readers/test_raster_gbx_examples.py`: - -```python -"""Executes the raster_gbx reader doc examples against real sample data (Docker).""" -import raster_gbx_examples as ex - - -def test_read_raster_gbx(spark): - ex.read_raster_gbx(spark) - - -def test_read_gtiff_gbx(spark): - ex.read_gtiff_gbx(spark) -``` - -- [ ] **Step 3: Run in Docker** - -Run: `bash scripts/commands/gbx-test-python-docs.sh --path readers/test_raster_gbx_examples.py --skip-build` -Expected: 2 passed. (`--skip-build` reuses the built JAR/wheel; if the wheel lacks `pyrx.ds`, drop `--skip-build` once to rebuild.) If `register` import fails, confirm the `[pyrx]` wheel is installed in-container (`docker exec geobrix-dev python3 -c "import databricks.labs.gbx.pyrx.ds.register"`). - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add docs/tests/python/readers/raster_gbx_examples.py docs/tests/python/readers/test_raster_gbx_examples.py -git commit -m "docs(test): light raster_gbx/gtiff_gbx reader doc examples" -``` - ---- - -## Task 2: Light reader MDX page + nav - -**Files:** -- Create: `docs/docs/readers/raster_gbx.mdx` -- Modify: `docs/sidebars.js`, `docs/docs/readers/overview.mdx` - -- [ ] **Step 1: Create the page** - -`docs/docs/readers/raster_gbx.mdx`: - -```mdx ---- -sidebar_position: 9 ---- - -import CodeFromTest from '@site/src/components/CodeFromTest'; -import rasterGbxExamples from '!!raw-loader!../../tests/python/readers/raster_gbx_examples.py'; - -# Lightweight Raster Readers (`raster_gbx` / `gtiff_gbx`) - -Pure-Python/PySpark raster readers built on Spark DataSource V2 — the lightweight -tier's drop-in for the GDAL-backed [`gdal`](./gdal) / [`gtiff_gdal`](./gtiff) -readers. They require no JAR (powered by `rasterio`) and run on Serverless. Output -is the same `(source, tile)` schema as the heavy readers, so downstream code is -unchanged — swapping tiers is a one-line `format(...)` change. - -## Register - - - -## Read (catch-all) - - - -## Read GeoTIFF (named) - - - -## Options - -| Option | Default | Description | -|--------|---------|-------------| -| `sizeInMB` | `"16"` | Split threshold (MB on disk) for tiling large rasters into multiple tiles. | -| `filterRegex` | `".*"` | When loading a directory, keep files whose full path matches this regex. | - - - -`gtiff_gbx` is `raster_gbx` with the GeoTIFF driver preset. See -[Choosing an Execution Tier](../api/execution-tiers) for when to use the -lightweight vs heavyweight readers. -``` - -- [ ] **Step 2: Add to sidebar + overview** - -In `docs/sidebars.js`, add `'readers/raster_gbx',` to the Readers `items` list (after `'readers/gtiff'`). - -In `docs/docs/readers/overview.mdx`, add rows to the reader table: - -```markdown -| [Lightweight Raster Reader](./raster_gbx) | `raster_gbx` | Pure-Python catch-all raster reader (no JAR; DataSource V2) | -| [Lightweight GeoTIFF Reader](./raster_gbx) | `gtiff_gbx` | Pure-Python GeoTIFF reader (preset `driver="GTiff"`) | -``` - -- [ ] **Step 3: Verify the MDX builds** - -Run: `bash scripts/commands/gbx-ci-docs.sh 2>/dev/null || bash scripts/commands/gbx-docs-start.sh --build-only 2>/dev/null || (cd docs && npm run build)` -Expected: build succeeds, no broken-link/import error for `readers/raster_gbx`. (Use whichever docs-build command exists; check `scripts/commands/` for `gbx-ci-docs.sh` / `gbx-docs-*.sh`. If none builds headless, at minimum confirm the raw-loader path resolves: the import path `../../tests/python/readers/raster_gbx_examples.py` is correct relative to `docs/docs/readers/`.) - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add docs/docs/readers/raster_gbx.mdx docs/sidebars.js docs/docs/readers/overview.mdx -git commit -m "docs: lightweight raster reader page (raster_gbx/gtiff_gbx) + nav" -``` - ---- - -## Task 3: Light writer doc-test (`writers/raster_gbx_examples.py`) - -**Files:** -- Create: `docs/tests/python/writers/raster_gbx_examples.py`, `docs/tests/python/writers/test_raster_gbx_examples.py` - -- [ ] **Step 1: Create the examples module** - -`docs/tests/python/writers/raster_gbx_examples.py`: - -```python -"""raster_gbx / gtiff_gbx (lightweight) Writer Examples — single source of truth. - -Code shown in docs/docs/writers/raster_gbx.mdx is imported from here. Writer -options are path/nameCol/ext; on-disk encoding comes from tile.metadata. -""" -import os -import tempfile - -from path_config import SAMPLE_DATA_BASE - -SAMPLE_RASTER_PATH = f"{SAMPLE_DATA_BASE}/nyc/sentinel2/nyc_sentinel2_red.tif" - -WRITE_GTIFF_GBX = """# Read then write GeoTIFF tiles (lightweight) -from databricks.labs.gbx.pyrx.ds.register import register -register(spark) -df = spark.read.format("raster_gbx").load("{SAMPLE_RASTER_PATH}") -df.write.format("gtiff_gbx").mode("overwrite").save(OUT_DIR)""" - -WRITE_WITH_NAMECOL = """# Control output filenames: overwrite 'source', set nameCol -from pyspark.sql.functions import concat, lit, monotonically_increasing_id -(df.withColumn("source", concat(lit("tile_"), monotonically_increasing_id())) - .write.format("gtiff_gbx").mode("overwrite") - .option("nameCol", "source").option("ext", "tif").save(OUT_DIR))""" - -ENCODING_NOTE = """# On-disk format/compression come from tile.metadata, NOT writer options -# driver/format -> output driver (default GTiff; GTiff = passed through verbatim) -# compression/blocksize/zlevel/zstd_level -> applied when re-encoding (non-GTiff) -# Change them via upstream transforms, then write.""" - - -def _register(spark): - from databricks.labs.gbx.pyrx.ds.register import register - register(spark) - - -def write_gtiff_gbx(spark, path=None): - """Verify WRITE_GTIFF_GBX: round-trip read -> write -> re-read, same pixels.""" - import numpy as np - import rasterio - _register(spark) - df = spark.read.format("raster_gbx").load(path or SAMPLE_RASTER_PATH) - with tempfile.TemporaryDirectory() as out_dir: - df.write.format("gtiff_gbx").mode("overwrite").save(out_dir) - files = [f for f in os.listdir(out_dir) if f.endswith(".tif")] - assert files, "no output written" - with rasterio.open(os.path.join(out_dir, files[0])) as w: - written = w.read() - with rasterio.open(path or SAMPLE_RASTER_PATH) as src: - truth = src.read() - # whole-file GTiff pass-through -> identical pixels - assert written.shape == truth.shape - np.testing.assert_allclose(written, truth, rtol=1e-3, atol=1e-3) - - -def write_with_namecol(spark, path=None): - """Verify WRITE_WITH_NAMECOL: nameCol controls output filenames.""" - from pyspark.sql.functions import lit - _register(spark) - df = spark.read.format("raster_gbx").load(path or SAMPLE_RASTER_PATH) - with tempfile.TemporaryDirectory() as out_dir: - (df.withColumn("source", lit("mytile")) - .write.format("gtiff_gbx").mode("overwrite") - .option("nameCol", "source").save(out_dir)) - assert "mytile.tif" in os.listdir(out_dir) -``` - -- [ ] **Step 2: Create the test entry** - -`docs/tests/python/writers/test_raster_gbx_examples.py`: - -```python -"""Executes the raster_gbx writer doc examples (Docker).""" -import raster_gbx_examples as ex - - -def test_write_gtiff_gbx(spark): - ex.write_gtiff_gbx(spark) - - -def test_write_with_namecol(spark): - ex.write_with_namecol(spark) -``` - -- [ ] **Step 3: Run in Docker** - -Run: `bash scripts/commands/gbx-test-python-docs.sh --path writers/test_raster_gbx_examples.py --skip-build` -Expected: 2 passed. (Note: two `raster_gbx_examples.py` files now exist — one under `readers/`, one under `writers/`. pytest imports by module name; confirm no import collision — if pytest complains about duplicate module basenames, the doc-test suite uses `rootdir`/`importmode` that disambiguates by path; if it errors, set unique names is NOT desired — instead verify `docs/tests/python` has an `__init__.py`-free layout with `--import-mode=importlib` (check conftest/pyproject) and note the resolution.) - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add docs/tests/python/writers/raster_gbx_examples.py docs/tests/python/writers/test_raster_gbx_examples.py -git commit -m "docs(test): light raster_gbx/gtiff_gbx writer doc examples" -``` - ---- - -## Task 4: Light writer MDX page + nav - -**Files:** -- Create: `docs/docs/writers/raster_gbx.mdx` -- Modify: `docs/sidebars.js`, `docs/docs/writers/overview.mdx` - -- [ ] **Step 1: Create the page** - -`docs/docs/writers/raster_gbx.mdx`: - -```mdx ---- -sidebar_position: 4 ---- - -import CodeFromTest from '@site/src/components/CodeFromTest'; -import rasterGbxWrite from '!!raw-loader!../../tests/python/writers/raster_gbx_examples.py'; - -# Lightweight Raster Writer (`raster_gbx` / `gtiff_gbx`) - -Pure-Python/PySpark raster writer — the lightweight tier's drop-in for the -GDAL-backed [`gdal`](./gdal) writer. Requires the exact `(source, tile)` schema, -the same as the heavy writer. Writer options are `path` / `nameCol` / `ext`; the -on-disk format and compression come from `tile.metadata`, not writer options. - -## Write GeoTIFF tiles - - - -A whole-file GeoTIFF tile is written through verbatim (no re-encode); a tile whose -`metadata["driver"]` is non-GTiff (e.g. `COG`) is re-encoded via `rasterio`. - -## Control filenames (`nameCol`) - - - -## Output format & compression - - - -| Option | Default | Description | -|--------|---------|-------------| -| `nameCol` | _unset_ | Existing string column whose value is the output filename (overwrite `source`). When unset, an opaque unique name is used. | -| `ext` | `"tif"` | Filename suffix. Does **not** change the on-disk format. | - -The driver / `compression` / `blocksize` / `zlevel` / `zstd_level` are read from -`tile.metadata` (same as the heavy [`gdal`](./gdal) writer). -``` - -- [ ] **Step 2: Sidebar + overview** - -In `docs/sidebars.js`, add `'writers/raster_gbx',` to the Writers `items` (after `'writers/gdal'`). - -In `docs/docs/writers/overview.mdx`, add a row: - -```markdown -| [Lightweight Raster Writer](./raster_gbx) | `raster_gbx`, `gtiff_gbx` | Pure-Python raster writer (no JAR; DataSource V2) | -``` - -- [ ] **Step 3: Verify MDX builds** (same command as Task 2 Step 3). - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add docs/docs/writers/raster_gbx.mdx docs/sidebars.js docs/docs/writers/overview.mdx -git commit -m "docs: lightweight raster writer page + nav" -``` - ---- - -## Task 5: Heavy `gdal`/`gtiff_gdal` writer option docs - -**Files:** -- Modify: `docs/docs/writers/gdal.mdx` -- Modify: `docs/tests/python/writers/gdal_examples.py` - -- [ ] **Step 1: Add a `gtiff_gdal` write example to the doc-test** - -Append to `docs/tests/python/writers/gdal_examples.py`: - -```python -WRITE_GTIFF_GDAL = """# Named GeoTIFF writer (gtiff_gdal = gdal writer with driver preset) -spark.read.format("gtiff_gdal").load(SAMPLE_RASTER_PATH) \\ - .write.format("gtiff_gdal").mode("append").option("ext", "tif").save(OUT_DIR)""" - -ENCODING_FROM_METADATA = """# Output encoding is read from tile.metadata, not writer options: -# format/driver (default GTiff), compression (DEFLATE), blocksize (512), -# zlevel (6), zstd_level (9). Set them upstream (e.g. RST_AsFormat), then write.""" -``` - -(These are display constants — no new test function needed; the existing `gdal` write test already covers the write path. If the file has a `_run` harness that asserts every constant is a non-empty string, these satisfy it.) - -- [ ] **Step 2: Document the encoding keys + gtiff_gdal writer in `writers/gdal.mdx`** - -In `docs/docs/writers/gdal.mdx`, after the existing options table, add an "Output encoding (from tile metadata)" subsection importing the new constants and a table: - -```mdx -## Output encoding (from tile metadata) - -The output **driver, compression, and block layout are read from -`tile.metadata`**, not from writer options. They are set when the tile is read or -produced (e.g. via `RST_AsFormat`); the writer honors them on serialization. - -| Metadata key | Default | Effect | -|--------------|---------|--------| -| `driver` / `format` | `GTiff` | GDAL output driver. | -| `compression` | `DEFLATE` | `DEFLATE` / `ZSTD` / `LZW` / … creation compression. | -| `blocksize` | `512` | Tile/block size in pixels (floored to a multiple of 16, clamped to the raster size). | -| `zlevel` | `6` | DEFLATE level. | -| `zstd_level` | `9` | ZSTD level. | - - - -### Named GeoTIFF writer (`gtiff_gdal`) - -`gtiff_gdal` is the `gdal` writer with the GeoTIFF driver preset — use it to make -GeoTIFF output explicit. - - -``` - -(Confirm `gdal.mdx` already imports `gdalExamples` via raw-loader — it does, per the existing page; reuse that import.) - -- [ ] **Step 3: Run the writer doc-tests + build** - -Run: `bash scripts/commands/gbx-test-python-docs.sh --path writers/ --skip-build` (the gdal write test still passes; new constants are strings). Then verify MDX builds (Task 2 Step 3 command). - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add docs/docs/writers/gdal.mdx docs/tests/python/writers/gdal_examples.py -git commit -m "docs: document heavy writer tile-metadata encoding keys + gtiff_gdal writer" -``` - ---- - -## Task 6: Heavy reader option audit + execution-tiers un-stale - -**Files:** -- Modify: `docs/docs/readers/gdal.mdx`, `docs/docs/readers/gtiff.mdx` (audit only) -- Modify: `docs/docs/api/execution-tiers.mdx` - -- [ ] **Step 1: Audit the heavy reader options tables** - -Read `docs/docs/readers/gdal.mdx` and `docs/docs/readers/gtiff.mdx`. Confirm the options table documents at least: `path` (load arg), `driver`, `sizeInMB`, `filterRegex`. Add any missing row (the recon found `sizeInMB`/`filterRegex`/`driver` present; if `path`/`driver` lack a one-line description, add it). Do NOT remove existing rows. If the tables are already complete, make no change and note it in the commit. - -- [ ] **Step 2: Un-stale `execution-tiers.mdx`** - -In `docs/docs/api/execution-tiers.mdx`, change the Readers/Writers row (currently `... binaryFile + rst_fromcontent today; native Python Data Source readers planned`) to reflect that the lightweight tier now has native readers + a writer: - -```markdown -| Readers / Writers | `gtiff_gdal`, `gdal`, OGR readers | `raster_gbx` / `gtiff_gbx` native Python DataSource V2 reader + writer (no JAR); vector OGR readers still heavy-only | -``` - -And update the prose around "You need the GDAL/OGR readers" so it no longer says lightweight native readers are unavailable — note they exist for raster (`raster_gbx`/`gtiff_gbx`), with vector still heavy-only. - -- [ ] **Step 3: internals-leak guard** - -Run: `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/ 2>/dev/null` — must print nothing (QC `internals-leak` gate). Fix any hit you introduced. - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add docs/docs/readers/gdal.mdx docs/docs/readers/gtiff.mdx docs/docs/api/execution-tiers.mdx -git commit -m "docs: audit heavy raster reader options; un-stale lightweight tier readers/writers" -``` - ---- - -## Task 7: Full docs verification (Docker) - -- [ ] **Step 1: Run the full reader+writer doc-test suite in Docker** - -Run: `bash scripts/commands/gbx-test-python-docs.sh --suite readers --skip-build` then `... --suite writers --skip-build` -Expected: all pass (the new `raster_gbx` reader/writer tests + existing gdal tests). Report progress per the repo convention for long runs. - -- [ ] **Step 2: Build the docs site (link/import check)** - -Run the docs build (`bash scripts/commands/gbx-ci-docs.sh` or `cd docs && npm run build`). -Expected: build succeeds; no broken links to the new pages, no unresolved raw-loader imports. - -- [ ] **Step 3: internals-leak final check** - -Run: `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/ 2>/dev/null` → nothing. - -- [ ] **Step 4: Commit any fixes** - -```bash -chmod -R u+rwX /Users/mjohns/IdeaProjects/geobrix/.git/objects -git add -A docs -git commit -m "docs: raster I/O docs suite green + site builds" -``` - ---- - -## Out of scope - -- Vector reader/writer docs (`*_ogr`, future `*_gbx` vector). -- Rewriting heavy reader/writer behavior docs beyond the option audit (e.g. the `(source, tile)` schema framing) unless an audited option is wrong. - -## Self-review notes (for the executor) - -- **Two `raster_gbx_examples.py` files** (readers/ + writers/) — same basename. If pytest collection errors on duplicate module names, check `docs/tests/python` import mode (likely `importlib` via pyproject `addopts`); if not, the safe fix is unique basenames (`raster_gbx_read_examples.py` / `raster_gbx_write_examples.py`) — adjust the `import` lines in the `.mdx` and `test_*` files to match. Decide based on the actual collection behavior in Task 3 Step 3. -- **`CodeFromTest functionName`** does substring extraction — keep constant names unique within a file (e.g. don't have both `READ` and `READ_GTIFF` where one is a prefix of the other in a way that mis-extracts; the existing files use distinct names like `READ_GDAL`/`READ_WITH_DRIVER`). -- **Light DS registration in doc-tests:** call `register(spark)` inside each helper (idempotent) — the doc-test session is shared/session-scoped, re-register is safe (DataSourceManager replaces). -- **No `wave N`** or internal vocabulary in any `.mdx` (QC `internals-leak`). -- Doc-tests run **in Docker only** (need `/Volumes` sample data + the wheel). -``` diff --git a/docs/superpowers/plans/2026-06-12-heavy-geojsonl-writer.md b/docs/superpowers/plans/2026-06-12-heavy-geojsonl-writer.md deleted file mode 100644 index 13a4a50e2..000000000 --- a/docs/superpowers/plans/2026-06-12-heavy-geojsonl-writer.md +++ /dev/null @@ -1,53 +0,0 @@ -# Heavy `geojsonl` writer — implementation plan - -> Subagent-driven, TDD. Builds the first heavyweight vector writer: a multi-file GeoJSONL -> directory writer (`geojsonl`) matching the lightweight `geojsonl_gbx` (one shard per -> partition, no driver merge, `maxRecordsPerFile`), so the "any-scale, sharded" writer is -> available in both tiers. - -**Goal:** a Scala DataSource V2 writer `geojsonl` that writes a directory of newline-delimited -GeoJSONL shards (one per partition; `maxRecordsPerFile` splits a partition), NO driver merge, -round-tripping with the `geojson_ogr`/`geojson_gbx` (`multi=true`/GeoJSONSeq) directory reader. - -**Architecture:** DataSource V2 write set modeled on the **PMTiles writer** (`pmtiles/PMTiles_*`) -but *simpler* — each partition's `DataWriter` writes its own shard(s) and there is **no -consolidation in `commit()`**. The per-shard encoding uses **OGR `GeoJSONSeq`** (model -`vectorx/mvt/MvtWriter.scala`, which already writes via an OGR driver): registration guarded by -`GDALManager.initOgr()`, write to node-local temp, then `HadoopUtils.copyToPath` to the Volume. - -**Tech:** Scala 2.13 / Spark 4.0 / GDAL-OGR JNI, in the `geobrix-dev` Docker container (mvn). - ---- - -## Components (new files under `src/main/scala/com/databricks/labs/gbx/vectorx/ds/geojsonl/`) -Model the `pmtiles/PMTiles_*` set: -- `GeoJSONL_DataSource.scala` — `TableProvider`/`DataSourceRegister`, `shortName = "geojsonl"`; `META-INF/services` entry. -- `GeoJSONL_Table.scala` — `SupportsWrite`; exposes `newWriteBuilder`. -- `GeoJSONL_WriteBuilder.scala` — validates the write schema (geom + `*_srid` + attrs, mirror the light writer / PMTiles exact-schema policy); requires `overwrite` mode (reject append for v1, matching the light writer); reads the `maxRecordsPerFile` option. -- `GeoJSONL_BatchWrite.scala` — `BatchWrite`; `createBatchWriterFactory`; `commit(msgs)` does **no merge** (optionally write `_SUCCESS`); `abort` best-effort deletes shards listed in the messages. On overwrite, clear the target dir once before tasks (in the builder/driver, like the light writer's `__init__`). -- `GeoJSONL_DataWriterFactory.scala` + `GeoJSONL_RowWriter.scala` — per-partition `DataWriter[InternalRow]`: buffer rows; when the buffer hits `maxRecordsPerFile` (or at `commit()`), flush a shard: `GDALManager.initOgr()` → `ogr.GetDriverByName("GeoJSONSeq").CreateDataSource(localTmpShard)` → `CreateLayer(name, srs, geomType)` → per row build a `Feature` (geometry via `ogr.CreateGeometryFromWkb(wkb)`, attrs via `SetField`), `layer.CreateFeature(f)` → close DS → `HadoopUtils.copyToPath(localShard, OUTDIR/part-.geojsonl, hConf)` → delete local. Unique shard name per flush (`uuid`). Return a `GeoJSONL_WriterMsg(shardPaths)`. -- `GeoJSONL_WriterMsg.scala` — `WriterCommitMessage` carrying the written shard paths. -- Register in `ds/register/RegisterBatch.scala` + `META-INF/services/org.apache.spark.sql.sources.DataSourceRegister`. - -## Schema contract -Input = the vector writer schema the readers emit: a geometry column (WKB binary) named ``, its `_srid` (string/int) and `_srid_proj`, plus attribute columns. Reuse/mirror the light `_writer_col_roles` logic (the column paired with `*_srid` is the geometry; `*_srid_proj` is proj4; the rest are attributes). SRS from the srid/proj. - -## Tests (TDD — Scala, run in Docker via `gbx:test:scala --suite '...geojsonl...'`) -- One shard per partition: write a small DF repartitioned to N → assert OUTDIR has exactly N `.geojsonl` shards (count matches expected — explicit). -- `maxRecordsPerFile`: M rows in one partition with k → exactly `ceil(M/k)` shards. -- Round-trip: read OUTDIR back with `geojson_ogr` (`multi=true`) → feature count == input; geometry + an attribute value match. -- Overwrite clears stale shards; append rejected. -- (Concurrency sanity: 2+ partitions write concurrently without a driver-registry race — covered by a multi-partition write test.) - -## Verification -- `gbx:lint:scalastyle` (CI gate) + the Scala suite green in Docker. -- Build + stage the JAR (`gbx:data:push-jar`), restart the bench cluster, and **round-trip on the cluster**: write a few-M-row table → `geojsonl` directory on a Volume → read back via `geojson_gbx(multi=true)` → **confirm shard count == expected** and row count matches. -- Optionally bench it (light vs heavy geojsonl writer) like the other writers. - -## Docs -- Add a **Heavyweight** tab to `docs/docs/writers/geojsonl.mdx` (`geojsonl` format name; same multi-file/`maxRecordsPerFile` semantics; classic-x86 + JAR requirement note). Keep the page template (Options near top, How-it-scales, Next Steps last). - -## Risks / notes -- OGR thread-safety: register only via `GDALManager.initOgr()`; each task writes its own node-local file via the `GeoJSONSeq` driver (not the shared MEM driver), so no `GetDriverByName` race. [[gdal-ogr-register-via-guard]] [[vectorraster-bridge-not-threadsafe]] -- Node-local → Volume write-back via `HadoopUtils.copyToPath` (sequential; FUSE-safe). Each tests.jar change needs a cluster restart [[jar-stage-before-cluster-start]]. -- First heavy vector writer → sets the pattern for future heavy vector writers (shapefile/gpkg). diff --git a/docs/superpowers/plans/2026-06-12-light-vector-readers.md b/docs/superpowers/plans/2026-06-12-light-vector-readers.md deleted file mode 100644 index c9a385c91..000000000 --- a/docs/superpowers/plans/2026-06-12-light-vector-readers.md +++ /dev/null @@ -1,937 +0,0 @@ -# Light Vector Readers (`*_gbx`, pyogrio) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build five pyogrio-backed PySpark DataSource V2 vector readers (`ogr_gbx` + `shapefile_gbx`/`geojson_gbx`/`gpkg_gbx`/`file_gdb_gbx`) that emit the exact same schema as the heavy Scala OGR readers, so the light tier reaches reader parity for vector data. - -**Architecture:** One module `databricks.labs.gbx.ds.vector` holds the generic `OgrGbxReader`/`OgrGbxDataSource` (schema from `pyogrio.read_info`, partitions = `chunkSize`-feature slices via `read_arrow(skip_features, max_features)`, rows = arrow→WKB tuples) plus four ~3-line named presets (subclass + `driverName`). Pure-Python (pyogrio + pyproj + shapely + pyarrow), Serverless-safe, registered via `gbx.ds.register`. - -**Tech Stack:** PySpark DataSource V2, `pyogrio` 0.12.x (`read_info`/`read_arrow`), `pyproj` (CRS→srid/proj4), `shapely` (WKT branch), `pyarrow`. - -**Reference spec:** `docs/superpowers/specs/2026-06-12-light-vector-readers-design.md` - ---- - -## Conventions / key facts (read before starting) - -- **Run tests** with the repo venv: `/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pytest -v -p no:cacheprovider`. Light/unit tests need no Docker; light-vs-heavy parity is Docker/integration (skip-if-heavy). -- **Serverless-safe:** product code in `gbx/ds/` must NOT use `._jvm`/`.sparkContext`/`.rdd`/`.conf.set`/`SparkConf`. Import `pyogrio`/`pyproj`/`shapely` **inside `read()`** (lazy), like `raster.py` imports rasterio lazily — keeps partitions picklable + Serverless-safe. -- **Commit hygiene:** before each `git commit` run `chmod -R u+rwX .git/objects`; trailer EXACTLY `Co-authored-by: Isaac` (repo convention; a security linter may warn — ignore; never a human name); subjects ≤72 chars; NO push. -- **Heavy schema to match (parity bar)** — from `OGR_SchemaInference.scala`: - - Field order: **attributes first, then geometry columns.** - - Attribute column = OGR field name (or `field_` if empty), Spark type per `getType`: - `Boolean→BooleanType, Integer→IntegerType, Integer64→LongType, Real→DoubleType, String/WideString→StringType, Date→DateType, Time→TimestampType, DateTime→TimestampType, Binary→BinaryType, IntegerList→ArrayType(IntegerType), RealList→ArrayType(DoubleType), StringList/WideStringList→ArrayType(StringType), else→StringType.` - - Geometry (single field `j=0` in v1): `geom_0` (`BinaryType` WKB if `asWKB=true` default, else `StringType` WKT) using the OGR geom field name if present else `geom_0`; then `geom_0_srid` (`StringType`, authority code e.g. `"4326"`, fallback `"0"`); then `geom_0_srid_proj` (`StringType`, PROJ4, fallback `""`). All nullable. - - Heavy options (exact names): `driverName` (default `""`=auto), `asWKB` (default `"true"`), `chunkSize` (default `"10000"`), `layerNumber` (default `"0"`), `layerName` (default `""`). -- **pyogrio facts (verified, 0.12.1):** - - `read_info(path, layer=…)` → dict with `crs` (e.g. `"EPSG:4326"`), `fields` (list of names), `ogr_types` (list like `"OFTString"`/`"OFTInteger"`/`"OFTReal"`/`"OFTInteger64"`/`"OFTDate"`/`"OFTDateTime"`/`"OFTTime"`/`"OFTBinary"`/list variants), `ogr_subtypes` (e.g. `"OFSTBoolean"`/`"OFSTNone"`), `geometry_name` (often `""`), `features` (count), `layer_name`. - - `read_arrow(path, layer=…, skip_features=, max_features=, read_geometry=True, datetime_as_string=False)` → `(meta, pyarrow.Table)`. The geometry column in the table is named `meta["geometry_name"] or "wkb_geometry"`, type `binary` (WKB). Attribute columns precede it. - - CRS→srid/proj4: `from pyproj import CRS; c = CRS.from_user_input(info["crs"]); auth = c.to_authority() # ('EPSG','4326') or None; proj4 = c.to_proj4()`. -- **Corpus paths** (doc-tests import `SAMPLE_DATA_BASE` from `docs/tests/python/readers/path_config.py`): - - GeoJSON: `{SAMPLE_DATA_BASE}/nyc/boroughs/nyc_boroughs.geojson` - - GeoJSONSeq: `{SAMPLE_DATA_BASE}/nyc/boroughs/nyc_boroughs.geojsonl` - - Shapefile (zip): `{SAMPLE_DATA_BASE}/nyc/subway/nyc_subway.shp.zip` - - GeoPackage: `{SAMPLE_DATA_BASE}/nyc/geopackage/nyc_complete.gpkg` - - FileGDB (zip): `{SAMPLE_DATA_BASE}/nyc/filegdb/NYC_Sample.gdb.zip` - - For local unit tests (no Volumes), tasks below generate tiny in-memory/temp vector files instead. - -## File map - -- Create: `python/geobrix/src/databricks/labs/gbx/ds/vector.py` — all five readers + `_vector_schema` + `_ogr_to_spark` + `_crs_to_srid_proj` + `_zip_vsi`. -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/register.py` — add five sources to `_SOURCES`. -- Modify: `python/geobrix/pyproject.toml` (`[light]`), `requirements-pyrx-ci.in/.txt`, `requirements-dev-container.in/.txt` — add `pyogrio` + `pyproj`. -- Modify: `python/geobrix/test/pyrx/test_serverless_no_spark_config.py` — add `vector.py` to the coverage list. -- Create tests: `python/geobrix/test/ds/test_vector_schema.py`, `test_vector_reader.py`, `test_vector_named.py`, `test_vector_parity.py`. -- Modify: `bench/readers.py` (+ `run_format_read` already exists), `bench/cluster.py` (+ `_CELL_VECTOR`), launcher (`--benchmark-vector`); `docs/docs/api/benchmarking.mdx`. -- Modify docs: `docs/docs/readers/{ogr,shapefile,geojson,geopackage,filegdb}.mdx` (add lightweight tab, drop note) + new `docs/tests/python/readers/*_gbx_examples.py`. - ---- - -### Task 1: Add `pyogrio` + `pyproj` to the light deps - -**Files:** `python/geobrix/pyproject.toml`, `requirements-pyrx-ci.in`, `requirements-dev-container.in` (+ regenerate the `.txt` locks) - -- [ ] **Step 1: Add to the `[light]` extra** in `python/geobrix/pyproject.toml`, after the `quadbin`/`pmtiles` lines: - -```toml - "quadbin>=0.2,<0.3", - "pmtiles>=3.4,<4", - # OGR-free vector reading for the light *_gbx vector DataSources. pyogrio - # bundles its own libgdal; pyproj (pulled by pyogrio) maps CRS->srid/proj4. - "pyogrio>=0.8,<1", - "pyproj>=3.6", -``` - -- [ ] **Step 2: Add to both lock sources.** In `python/geobrix/requirements-pyrx-ci.in` and `python/geobrix/requirements-dev-container.in`, add (pin a current version, e.g. `0.12.1` / `3.7.2` — use whatever resolves) near the geospatial stack: - -``` -pyogrio==0.12.1 -pyproj==3.7.2 -``` - -- [ ] **Step 3: Regenerate the hash-pinned locks** in the dev container (PyPI is firewalled; use the Databricks proxy). Run via the dev container (`bash scripts/commands/gbx-docker-start.sh` if not running): - -```bash -bash scripts/commands/gbx-docker-exec.sh "cd /root/geobrix/python/geobrix && \ - uv pip compile --generate-hashes --python-version 3.12 \ - --index-url "$PIP_INDEX_URL" \ - --output-file requirements-pyrx-ci.txt requirements-pyrx-ci.in && \ - uv pip compile --generate-hashes --python-version 3.12 \ - --index-url "$PIP_INDEX_URL" \ - --output-file requirements-dev-container.txt requirements-dev-container.in" -``` - -- [ ] **Step 4: Install into the repo venv + verify import** - -```bash -/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pip install --require-hashes -r python/geobrix/requirements-pyrx-ci.txt >/dev/null 2>&1 || \ - /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/pip install "pyogrio>=0.8,<1" "pyproj>=3.6" -/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -c "import pyogrio, pyproj; print('pyogrio', pyogrio.__version__, 'pyproj', pyproj.__version__)" -``` -Expected: prints versions. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/pyproject.toml python/geobrix/requirements-pyrx-ci.in python/geobrix/requirements-pyrx-ci.txt python/geobrix/requirements-dev-container.in python/geobrix/requirements-dev-container.txt -git commit -m "build(light): add pyogrio + pyproj for vector readers - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: Schema builder + type map (pure, unit-tested) - -**Files:** Create `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (schema parts only); Test `python/geobrix/test/ds/test_vector_schema.py` - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/ds/test_vector_schema.py -from pyspark.sql.types import ( - BinaryType, BooleanType, DoubleType, IntegerType, LongType, StringType, StructType, -) -from databricks.labs.gbx.ds.vector import _ogr_to_spark, _vector_schema, _crs_to_srid_proj - - -def test_ogr_to_spark_map(): - assert isinstance(_ogr_to_spark("OFTString", "OFSTNone"), StringType) - assert isinstance(_ogr_to_spark("OFTInteger", "OFSTNone"), IntegerType) - assert isinstance(_ogr_to_spark("OFTInteger", "OFSTBoolean"), BooleanType) - assert isinstance(_ogr_to_spark("OFTInteger64", "OFSTNone"), LongType) - assert isinstance(_ogr_to_spark("OFTReal", "OFSTNone"), DoubleType) - assert isinstance(_ogr_to_spark("OFTUnknownFuture", "OFSTNone"), StringType) # default - - -def test_vector_schema_matches_heavy_layout(): - info = { - "fields": ["name", "pop", "area"], - "ogr_types": ["OFTString", "OFTInteger", "OFTReal"], - "ogr_subtypes": ["OFSTNone", "OFSTNone", "OFSTNone"], - "geometry_name": "", - } - schema = _vector_schema(info, as_wkb=True) - names = [f.name for f in schema.fields] - # attributes first, then geom_0 + srid + proj - assert names == ["name", "pop", "area", "geom_0", "geom_0_srid", "geom_0_srid_proj"] - by = {f.name: f for f in schema.fields} - assert isinstance(by["pop"].dataType, IntegerType) - assert isinstance(by["area"].dataType, DoubleType) - assert isinstance(by["geom_0"].dataType, BinaryType) - assert isinstance(by["geom_0_srid"].dataType, StringType) - assert all(f.nullable for f in schema.fields) - - -def test_vector_schema_wkt_is_string(): - info = {"fields": [], "ogr_types": [], "ogr_subtypes": [], "geometry_name": ""} - schema = _vector_schema(info, as_wkb=False) - assert isinstance({f.name: f for f in schema.fields}["geom_0"].dataType, StringType) - - -def test_crs_to_srid_proj(): - srid, proj4 = _crs_to_srid_proj("EPSG:4326") - assert srid == "4326" - assert "+proj=longlat" in proj4 - assert _crs_to_srid_proj(None) == ("0", "") -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/test_vector_schema.py -v -p no:cacheprovider` -Expected: FAIL (ModuleNotFoundError `...ds.vector`). - -- [ ] **Step 3: Create `vector.py` with the schema parts** - -```python -# python/geobrix/src/databricks/labs/gbx/ds/vector.py -"""Light vector readers (*_gbx) — pyogrio-backed PySpark DataSource V2, emitting -the same schema as the heavyweight Scala OGR readers (geom_j WKB + srid + proj4 + -typed attributes). Pure-Python / Serverless-safe (no JVM).""" - -from __future__ import annotations - -from typing import Dict, List, Optional, Sequence, Tuple - -from pyspark.sql.datasource import DataSource, DataSourceReader, InputPartition -from pyspark.sql.types import ( - ArrayType, - BinaryType, - BooleanType, - DateType, - DoubleType, - IntegerType, - LongType, - StringType, - StructField, - StructType, - TimestampType, -) - -# OGR field type (+ subtype) -> Spark type, matching heavy OGR_SchemaInference.getType. -_OGR_TO_SPARK = { - "OFTInteger": IntegerType, - "OFTInteger64": LongType, - "OFTReal": DoubleType, - "OFTString": StringType, - "OFTWideString": StringType, - "OFTDate": DateType, - "OFTTime": TimestampType, - "OFTDateTime": TimestampType, - "OFTBinary": BinaryType, -} -_OGR_LIST_TO_SPARK = { - "OFTIntegerList": IntegerType, - "OFTRealList": DoubleType, - "OFTStringList": StringType, - "OFTWideStringList": StringType, -} - - -def _ogr_to_spark(ogr_type: str, subtype: str): - if subtype == "OFSTBoolean": - return BooleanType() - if ogr_type in _OGR_LIST_TO_SPARK: - return ArrayType(_OGR_LIST_TO_SPARK[ogr_type]()) - return _OGR_TO_SPARK.get(ogr_type, StringType)() - - -def _geom_name(info: Dict) -> str: - # Heavy uses the OGR geom field name if present, else geom_0 (single-geom v1). - return info.get("geometry_name") or "geom_0" - - -def _vector_schema(info: Dict, as_wkb: bool) -> StructType: - fields: List[StructField] = [] - names = list(info.get("fields", [])) - ogr_types = list(info.get("ogr_types", [])) - subtypes = list(info.get("ogr_subtypes", [])) - for j, name in enumerate(names): - col = name if name else f"field_{j}" - ot = ogr_types[j] if j < len(ogr_types) else "OFTString" - st = subtypes[j] if j < len(subtypes) else "OFSTNone" - fields.append(StructField(col, _ogr_to_spark(ot, st), True)) - gname = _geom_name(info) - geom_type = BinaryType() if as_wkb else StringType() - fields.append(StructField(gname, geom_type, True)) - fields.append(StructField(gname + "_srid", StringType(), True)) - fields.append(StructField(gname + "_srid_proj", StringType(), True)) - return StructType(fields) - - -def _crs_to_srid_proj(crs) -> Tuple[str, str]: - """(authority code string e.g. '4326' or '0', PROJ4 string or '').""" - if not crs: - return "0", "" - try: - from pyproj import CRS - - c = CRS.from_user_input(crs) - auth = c.to_authority() - srid = auth[1] if auth else "0" - try: - proj4 = c.to_proj4() or "" - except Exception: - proj4 = "" - return srid, proj4 - except Exception: - return "0", "" -``` - -- [ ] **Step 4: Run to verify pass** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/test_vector_schema.py -v -p no:cacheprovider` -Expected: 4 passed. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_vector_schema.py -git commit -m "feat(ds): vector schema builder + OGR->Spark type map (heavy parity) - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: Generic `ogr_gbx` reader + DataSource (+ register) - -**Files:** Modify `vector.py` (add reader/datasource + zip helper), `register.py`; Test `python/geobrix/test/ds/test_vector_reader.py` - -- [ ] **Step 1: Write the failing test** (generates a tiny GeoJSON locally so it needs no Volumes) - -```python -# python/geobrix/test/ds/test_vector_reader.py -import json -import os -import tempfile - -from shapely import from_wkb -from databricks.labs.gbx.ds.register import register - -_GJ = { - "type": "FeatureCollection", - "crs": {"type": "name", "properties": {"name": "urn:ogc:def:crs:EPSG::4326"}}, - "features": [ - {"type": "Feature", "properties": {"name": "a", "pop": 10}, - "geometry": {"type": "Point", "coordinates": [-73.9, 40.7]}}, - {"type": "Feature", "properties": {"name": "b", "pop": 20}, - "geometry": {"type": "Point", "coordinates": [-0.1, 51.5]}}, - ], -} - - -def _gj_path(tmp): - p = os.path.join(tmp, "pts.geojson") - with open(p, "w") as f: - json.dump(_GJ, f) - return p - - -def test_ogr_gbx_reads_wkb_schema(spark, tmp_path): - register(spark) - p = _gj_path(str(tmp_path)) - df = spark.read.format("ogr_gbx").load(p) - assert df.columns == ["name", "pop", "geom_0", "geom_0_srid", "geom_0_srid_proj"] - rows = df.orderBy("name").collect() - assert rows[0]["name"] == "a" and rows[0]["pop"] == 10 - assert rows[0]["geom_0_srid"] == "4326" - # geom_0 is valid WKB - assert from_wkb(bytes(rows[0]["geom_0"])).geom_type == "Point" - assert df.count() == 2 - - -def test_ogr_gbx_wkt_option(spark, tmp_path): - register(spark) - p = _gj_path(str(tmp_path)) - df = spark.read.format("ogr_gbx").option("asWKB", "false").load(p) - g = df.orderBy("name").collect()[0]["geom_0"] - assert isinstance(g, str) and g.upper().startswith("POINT") - - -def test_ogr_gbx_chunksize_partitions(spark, tmp_path): - register(spark) - p = _gj_path(str(tmp_path)) - df = spark.read.format("ogr_gbx").option("chunkSize", "1").load(p) - assert df.rdd.getNumPartitions() >= 2 # 2 features / chunk 1 - assert df.count() == 2 -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/test_vector_reader.py -v -p no:cacheprovider` -Expected: FAIL (`ogr_gbx` not registered). - -- [ ] **Step 3: Add the reader + DataSource + zip helper to `vector.py`** (append): - -```python -def _zip_vsi(path: str) -> str: - """Map a zipped vector source to a GDAL /vsizip/ path (so OGR reads it in place).""" - low = path.lower() - if low.endswith(".zip"): - return "/vsizip/" + path - return path - - -class _ChunkPartition(InputPartition): - """One contiguous feature slice of one layer (picklable).""" - - def __init__(self, path, driver, layer, as_wkb, skip, count, field_names): - self.path = path - self.driver = driver - self.layer = layer - self.as_wkb = as_wkb - self.skip = skip - self.count = count - self.field_names = field_names - - -class OgrGbxReader(DataSourceReader): - _DRIVER = "" # named subclasses override - - def __init__(self, options: Dict[str, str]): - self.path = options.get("path") - if not self.path: - raise ValueError("ogr_gbx requires a 'path' (e.g. .load(path)).") - self.driver = options.get("driverName", "") or self._DRIVER - self.as_wkb = options.get("asWKB", "true").lower() != "false" - self.chunk_size = int(options.get("chunkSize", "10000")) - self.layer_number = int(options.get("layerNumber", "0")) - self.layer_name = options.get("layerName", "") - - def _layer(self): - return self.layer_name if self.layer_name else self.layer_number - - def _info(self): - import pyogrio - - kw = {"layer": self._layer()} - if self.driver: - kw["driver"] = self.driver - return pyogrio.read_info(_zip_vsi(self.path), **kw) - - def schema(self) -> StructType: - return _vector_schema(self._info(), self.as_wkb) - - def partitions(self) -> Sequence[InputPartition]: - info = self._info() - n = int(info.get("features", 0) or 0) - names = list(info.get("fields", [])) - names = [nm if nm else f"field_{j}" for j, nm in enumerate(names)] - chunk = max(1, self.chunk_size) - parts = [] - skip = 0 - # at least one partition even for empty/unknown-count sources - while skip < n or (n == 0 and skip == 0): - parts.append( - _ChunkPartition(self.path, self.driver, self._layer(), - self.as_wkb, skip, chunk, names) - ) - skip += chunk - if n == 0: - break - return parts - - def read(self, partition: "_ChunkPartition"): - import pyogrio - - kw = { - "layer": partition.layer, - "skip_features": partition.skip, - "max_features": partition.count, - "read_geometry": True, - "datetime_as_string": False, - } - if partition.driver: - kw["driver"] = partition.driver - meta, tbl = pyogrio.read_arrow(_zip_vsi(partition.path), **kw) - gcol = meta.get("geometry_name") or "wkb_geometry" - srid, proj4 = _crs_to_srid_proj(meta.get("crs")) - attr_cols = [c for c in tbl.column_names if c != gcol] - # column-wise to python, then row tuples (attrs..., geom, srid, proj) - cols = {c: tbl.column(c).to_pylist() for c in tbl.column_names} - geom = cols.get(gcol, [None] * tbl.num_rows) - for i in range(tbl.num_rows): - g = geom[i] - if g is not None and not partition.as_wkb: - from shapely import from_wkb - - g = from_wkb(bytes(g)).wkt - elif g is not None: - g = bytes(g) - row = tuple(cols[c][i] for c in attr_cols) + (g, srid, proj4) - yield row - - -class OgrGbxDataSource(DataSource): - @classmethod - def name(cls) -> str: - return "ogr_gbx" - - _READER = OgrGbxReader - - def schema(self) -> StructType: - return self._READER(self.options).schema() - - def reader(self, schema: StructType) -> DataSourceReader: - return self._READER(self.options) -``` - -NOTE on attribute order: `read_arrow`'s table lists attribute columns in the same order as `read_info["fields"]`, then the geometry column — matching `_vector_schema` (attributes then geom). `attr_cols` (table order minus geom) preserves it. If an attribute name was empty and `_vector_schema` renamed it to `field_`, pyogrio still returns it under its arrow name; for the corpus all attribute fields are named, so this matches — the parity test (Task 6) is the gate for any rename edge. - -- [ ] **Step 4: Register `ogr_gbx`** — edit `register.py`: - -```python -from databricks.labs.gbx.ds.gtiff import GTiffGbxDataSource -from databricks.labs.gbx.ds.pmtiles import PMTilesGbxDataSource -from databricks.labs.gbx.ds.raster import RasterGbxDataSource -from databricks.labs.gbx.ds.vector import OgrGbxDataSource - -_SOURCES = ( - RasterGbxDataSource, - GTiffGbxDataSource, - PMTilesGbxDataSource, - OgrGbxDataSource, -) -``` - -- [ ] **Step 5: Run to verify pass** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/test_vector_reader.py -v -p no:cacheprovider` -Expected: 3 passed. (If `chunkSize` partition count assertion is environment-flaky, confirm `df.count()==2` and ≥1 partition; tune the partition test to assert the union, not an exact count, if Spark coalesces.) - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/src/databricks/labs/gbx/ds/register.py python/geobrix/test/ds/test_vector_reader.py -git commit -m "feat(ds): generic ogr_gbx vector reader (pyogrio, chunked, WKB/WKT) - -Co-authored-by: Isaac" -``` - ---- - -### Task 4: Named presets (`shapefile_gbx`/`geojson_gbx`/`gpkg_gbx`/`file_gdb_gbx`) - -**Files:** Modify `vector.py` (add 4 subclasses), `register.py`; Test `python/geobrix/test/ds/test_vector_named.py` - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/ds/test_vector_named.py -import json -import os - -from databricks.labs.gbx.ds.register import register -from databricks.labs.gbx.ds import vector as V - - -def test_named_drivers_preset(): - assert V.ShapefileGbxDataSource.name() == "shapefile_gbx" - assert V.GeoJSONGbxDataSource.name() == "geojson_gbx" - assert V.GpkgGbxDataSource.name() == "gpkg_gbx" - assert V.FileGdbGbxDataSource.name() == "file_gdb_gbx" - assert V.ShapefileGbxDataSource._READER._DRIVER == "ESRI Shapefile" - assert V.GeoJSONGbxDataSource._READER._DRIVER == "GeoJSON" - assert V.GpkgGbxDataSource._READER._DRIVER == "GPKG" - assert V.FileGdbGbxDataSource._READER._DRIVER == "OpenFileGDB" - - -def test_geojson_gbx_reads(spark, tmp_path): - register(spark) - p = os.path.join(str(tmp_path), "pts.geojson") - with open(p, "w") as f: - json.dump({"type": "FeatureCollection", - "features": [{"type": "Feature", "properties": {"k": 1}, - "geometry": {"type": "Point", "coordinates": [0, 0]}}]}, f) - df = spark.read.format("geojson_gbx").load(p) - assert "geom_0" in df.columns and df.count() == 1 -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/test_vector_named.py -v -p no:cacheprovider` -Expected: FAIL (`ShapefileGbxDataSource` undefined / `geojson_gbx` not registered). - -- [ ] **Step 3: Add the four presets to `vector.py`** (append): - -```python -class _ShapefileReader(OgrGbxReader): - _DRIVER = "ESRI Shapefile" - - -class _GeoJSONReader(OgrGbxReader): - _DRIVER = "GeoJSON" - - -class _GpkgReader(OgrGbxReader): - _DRIVER = "GPKG" - - -class _FileGdbReader(OgrGbxReader): - _DRIVER = "OpenFileGDB" - - -class ShapefileGbxDataSource(OgrGbxDataSource): - _READER = _ShapefileReader - - @classmethod - def name(cls) -> str: - return "shapefile_gbx" - - -class GeoJSONGbxDataSource(OgrGbxDataSource): - _READER = _GeoJSONReader - - @classmethod - def name(cls) -> str: - return "geojson_gbx" - - -class GpkgGbxDataSource(OgrGbxDataSource): - _READER = _GpkgReader - - @classmethod - def name(cls) -> str: - return "gpkg_gbx" - - -class FileGdbGbxDataSource(OgrGbxDataSource): - _READER = _FileGdbReader - - @classmethod - def name(cls) -> str: - return "file_gdb_gbx" -``` - -- [ ] **Step 4: Register the four** — extend `_SOURCES` in `register.py`: - -```python -from databricks.labs.gbx.ds.vector import ( - FileGdbGbxDataSource, - GeoJSONGbxDataSource, - GpkgGbxDataSource, - OgrGbxDataSource, - ShapefileGbxDataSource, -) - -_SOURCES = ( - RasterGbxDataSource, - GTiffGbxDataSource, - PMTilesGbxDataSource, - OgrGbxDataSource, - ShapefileGbxDataSource, - GeoJSONGbxDataSource, - GpkgGbxDataSource, - FileGdbGbxDataSource, -) -``` - -- [ ] **Step 5: Run to verify pass** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/test_vector_named.py -v -p no:cacheprovider` -Expected: 2 passed. - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/src/databricks/labs/gbx/ds/register.py python/geobrix/test/ds/test_vector_named.py -git commit -m "feat(ds): named vector readers (shapefile/geojson/gpkg/file_gdb _gbx) - -Co-authored-by: Isaac" -``` - ---- - -### Task 5: Serverless guard coverage - -**Files:** Modify `python/geobrix/test/pyrx/test_serverless_no_spark_config.py` - -- [ ] **Step 1: Add `vector.py` to the coverage assertion** — in `test_serverless_scan_includes_ds_modules`, append `"vector.py"` to the required tuple (after `"shard.py"`). - -- [ ] **Step 2: Run the guard + the full vector + ds suite** - -```bash -.venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/test_serverless_no_spark_config.py python/geobrix/test/ds -q -p no:cacheprovider -``` -Expected: PASS (the guard's forbidden-pattern scan over `vector.py` must pass — confirms no `_jvm`/`.conf.set`/`.rdd`; pyogrio/pyproj/shapely imports are inside `read()`/`_info()`). - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/pyrx/test_serverless_no_spark_config.py -git commit -m "test(ds): cover vector.py in the Serverless guard scan - -Co-authored-by: Isaac" -``` - ---- - -### Task 6: Light-vs-heavy parity test (Docker / integration, skip-if-heavy) - -**Files:** Create `python/geobrix/test/ds/test_vector_parity.py` - -- [ ] **Step 1: Write the parity test** (mirrors `test/ds/test_writer_parity.py`'s `spark_with_jar` skip pattern; reuses the corpus) - -```python -# python/geobrix/test/ds/test_vector_parity.py -"""Light vs heavy vector reader parity (Docker / integration). - -Same source -> light *_gbx vs heavy *_ogr produce the same schema (columns + -types), row count, and decoded geometries. Skips unless the geobrix JAR is staged -+ sample data is mounted.""" - -import logging -import os -from pathlib import Path - -import pytest - -pytestmark = pytest.mark.integration - -_HERE = Path(__file__).resolve() -_JARS = sorted((_HERE.parents[3] / "lib").glob("geobrix-*-jar-with-dependencies.jar")) -SAMPLE = os.environ.get( - "GBX_SAMPLE_DATA_ROOT", - "/Volumes/main/default/test-data", -).rstrip("/") + "/geobrix-examples" - -_CASES = [ - ("geojson_gbx", "geojson_ogr", f"{SAMPLE}/nyc/boroughs/nyc_boroughs.geojson"), - ("shapefile_gbx", "shapefile_ogr", f"{SAMPLE}/nyc/subway/nyc_subway.shp.zip"), - ("gpkg_gbx", "gpkg_ogr", f"{SAMPLE}/nyc/geopackage/nyc_complete.gpkg"), - ("file_gdb_gbx", "file_gdb_ogr", f"{SAMPLE}/nyc/filegdb/NYC_Sample.gdb.zip"), -] - - -@pytest.fixture(scope="module") -def spark_with_jar(): - if not _JARS: - pytest.skip("no geobrix JAR staged") - from pyspark.sql import SparkSession - - logging.getLogger("py4j").setLevel(logging.ERROR) - s = ( - SparkSession.builder.master("local[2]") - .appName("gbx-ds-vector-parity") - .config( - "spark.driver.extraJavaOptions", - "-Djava.library.path=/usr/local/lib:/usr/lib:/usr/java/packages/lib:" - "/usr/lib64:/lib64:/lib:/usr/local/hadoop/lib/native", - ) - .config("spark.jars", str(_JARS[-1])) - .getOrCreate() - ) - from databricks.labs.gbx.ds.register import register - - register(s) - yield s - - -@pytest.mark.parametrize("light_fmt,heavy_fmt,path", _CASES) -def test_vector_reader_parity(spark_with_jar, light_fmt, heavy_fmt, path): - if not os.path.exists(path): - pytest.skip(f"sample not mounted: {path}") - spark = spark_with_jar - light = spark.read.format(light_fmt).load(path) - heavy = spark.read.format(heavy_fmt).load(path) - # same schema (names + types) - assert [(f.name, f.dataType.simpleString()) for f in light.schema.fields] == [ - (f.name, f.dataType.simpleString()) for f in heavy.schema.fields - ] - assert light.count() == heavy.count() - # decoded geometry sets match (compare WKB geom_0 bytes) - lg = {bytes(r["geom_0"]) for r in light.select("geom_0").collect()} - hg = {bytes(r["geom_0"]) for r in heavy.select("geom_0").collect()} - assert lg == hg -``` - -- [ ] **Step 2: Run locally (skips without JAR)** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/ds/test_vector_parity.py -v -p no:cacheprovider --no-header -rs` -Expected: skipped (no JAR locally) — NOT errored. Runs/asserts in Docker/cluster. - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/ds/test_vector_parity.py -git commit -m "test(ds): light-vs-heavy vector reader parity (skip-if-heavy) - -Co-authored-by: Isaac" -``` - ---- - -### Task 7: Bench — `--benchmark-vector` (light `*_gbx` vs heavy `*_ogr` + parity) - -> **Phasing note:** this is the **initial** vector benchmark — limited to the -> sample-data formats already on the Volume. The **scaled/final** benchmark (varied -> formats + larger feature counts) comes in a later phase once the **light vector -> writers** exist to *generate* that corpus. Wire the harness + numbers now; the -> bigger run is a follow-on. - -**Files:** Modify `bench/cluster.py` (PREAMBLE flags + `_CELL_VECTOR` + `build_bench_notebook`), `notebooks/tests/push_and_run_bench_on_cluster.py`, `scripts/commands/gbx-bench-cluster.sh`; Test `python/geobrix/test/bench/test_cluster_vector.py` - -This mirrors the existing `--benchmark-readers` / `--benchmark-pmtiles` wiring exactly (read those first). - -- [ ] **Step 1: Add PREAMBLE flags** in `cluster.py` next to `BENCHMARK_PMTILES`/`PMTILES_ONLY`: - -``` -BENCHMARK_VECTOR = {benchmark_vector!r} -VECTOR_ONLY = {vector_only!r} -``` - -- [ ] **Step 2: Add `_CELL_VECTOR`** in `cluster.py`, modeled on `_CELL_READERS`. For each light/heavy vector pair, time `run_format_read` and assert row-count + geom parity: - -```python -_CELL_VECTOR = """# Vector reader benchmark: light *_gbx vs heavy *_ogr (+ parity) -from databricks.labs.gbx.bench import readers as _rd -_vbase = f"{CORPUS}/vector" # operator stages the vector corpus here -_vcases = [ - ("geojson_gbx", "geojson_ogr", _vbase + "/nyc_boroughs.geojson"), - ("shapefile_gbx", "shapefile_ogr", _vbase + "/nyc_subway.shp.zip"), - ("gpkg_gbx", "gpkg_ogr", _vbase + "/nyc_complete.gpkg"), - ("file_gdb_gbx", "file_gdb_ogr", _vbase + "/NYC_Sample.gdb.zip"), -] -_vrows = [] -for _lfmt, _hfmt, _vp in _vcases: - if LIGHTWEIGHT: - _r = _rd.run_format_read(spark, _vp, RUN_ID, SPARK_WARMUP, SPARK_MEASURED, - api="lightweight", fmt=_lfmt, where="cluster") - _sink([_r]); lw.append(_r); _vrows.append(_r) - if HEAVYWEIGHT: - _r = _rd.run_format_read(spark, _vp, RUN_ID, SPARK_WARMUP, SPARK_MEASURED, - api="heavyweight", fmt=_hfmt, where="cluster") - _sink([_r]); hw.append(_r); _vrows.append(_r) - if LIGHTWEIGHT and HEAVYWEIGHT: - _lc = spark.read.format(_lfmt).load(_vp).count() - _hc = spark.read.format(_hfmt).load(_vp).count() - print(f"VECTOR PARITY {_lfmt}: light={_lc} heavy={_hc} {'PASS' if _lc==_hc else 'FAIL'}") - assert _lc == _hc, f"row-count parity FAIL for {_lfmt}: {_lc} != {_hc}" -if _vrows: - _md = results.summarize(_vrows) - _show_md(f"vector reader benchmark -- {RUN_ID}", _md) -""" -``` - -- [ ] **Step 3: Wire into `build_bench_notebook`** — add `benchmark_vector=bool(cfg.get("benchmark_vector"))` and `vector_only=bool(cfg.get("vector_only"))` to the `_PREAMBLE.format(...)` call; add the locals; gate the fn cells with `and not vector_only`; append `if benchmark_vector or vector_only: cells.append(_cell(_CELL_VECTOR))` (after the pmtiles cell). - -- [ ] **Step 4: Wire the launcher** (`push_and_run_bench_on_cluster.py`) mirroring `benchmark_pmtiles`/`pmtiles_only`: argv parse `--benchmark-vector`/`--vector-only`, add to cfg, pass to `_expected_rows`, and `if vector_only: cfg["modes"]="spark-path"`. - -- [ ] **Step 5: Document the flags** in `scripts/commands/gbx-bench-cluster.sh` help (it forwards unknown args). - -- [ ] **Step 6: Smoke test** (mirror `test/bench/test_cluster_pmtiles.py`): assert `build_bench_notebook` with `benchmark_vector=True` includes a cell containing `run_format_read` + `VECTOR PARITY`, and `vector_only=True` omits the per-function cells. Run: - -```bash -.venv-pyrx/bin/python -m pytest python/geobrix/test/bench/test_cluster_vector.py -q -p no:cacheprovider -``` -Expected: pass. - -- [ ] **Step 7: Add a benchmarking.mdx section** — `docs/docs/api/benchmarking.mdx`, a "Results — vector readers" subsection (placeholder table to be filled from a cluster run, mirroring the PMTiles section's prose: light `*_gbx` vs heavy `*_ogr`, row-count parity, run with `gbx:bench:cluster --benchmark-vector`). - -- [ ] **Step 8: Lint + commit** - -```bash -bash scripts/commands/gbx-lint-python.sh --check # fix any findings in changed files -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/bench/cluster.py notebooks/tests/push_and_run_bench_on_cluster.py scripts/commands/gbx-bench-cluster.sh python/geobrix/test/bench/test_cluster_vector.py docs/docs/api/benchmarking.mdx -git commit -m "feat(bench): --benchmark-vector (light *_gbx vs heavy *_ogr + parity) - -Co-authored-by: Isaac" -``` - ---- - -### Task 8: Docs — add Lightweight tab to the five vector reader pages - -**Files:** Modify `docs/docs/readers/{ogr,shapefile,geojson,geopackage,filegdb}.mdx`; Create `docs/tests/python/readers/{ogr,shapefile,geojson,geopackage,filegdb}_gbx_examples.py` (+ matching `test_*` runners). - -For EACH of the five pages (they currently have the `:::note No lightweight equivalent yet` admonition + heavy body): - -- [ ] **Step 1: Create the lightweight doc-test example.** e.g. `docs/tests/python/readers/geojson_gbx_examples.py`: - -```python -"""Executable doc example for the lightweight geojson_gbx reader (Docker).""" - -import path_config # noqa: F401 (sets SAMPLE_DATA_BASE) -from path_config import SAMPLE_DATA_BASE - -READ_GEOJSON_GBX = """# Lightweight GeoJSON reader (pyogrio; no JAR) -from databricks.labs.gbx.ds.register import register -register(spark) -df = spark.read.format("geojson_gbx").load(SAMPLE) # same (geom_0, *_srid, attrs) schema as geojson_ogr -df.show()""" - -SAMPLE = f"{SAMPLE_DATA_BASE}/nyc/boroughs/nyc_boroughs.geojson" - - -def read_geojson_gbx(spark): - from databricks.labs.gbx.ds.register import register - - register(spark) - df = spark.read.format("geojson_gbx").load(SAMPLE) - assert "geom_0" in df.columns and "geom_0_srid" in df.columns - assert df.count() > 0 -``` -(Analogous files for ogr/shapefile/geopackage/filegdb with their corpus paths + format names `ogr_gbx`/`shapefile_gbx`/`gpkg_gbx`/`file_gdb_gbx`; the shapefile/filegdb ones use the `.shp.zip`/`.gdb.zip` paths.) Add a `test__gbx_examples.py` runner per file that calls the verification function (mirror an existing `docs/tests/python/readers/test_*_examples.py`). - -- [ ] **Step 2: Restructure each reader page to a tabbed page** — convert the heavy-only page to the same `` form used by `readers/raster.mdx`: a **Lightweight tab first** (the `*_gbx` reader, importing the new example via raw-loader + ``), then a **Heavyweight tab** holding the page's current body. **Remove** the `:::note No lightweight equivalent yet` admonition. Keep `sidebar_label` as the format name. Example head for `readers/geojson.mdx`: - -```mdx ---- -sidebar_position: 5 -sidebar_label: GeoJSON ---- - -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import CodeFromTest from '@site/src/components/CodeFromTest'; -import geojsonExamples from '!!raw-loader!../../tests/python/readers/geojson_examples.py'; -import geojsonScala from '!!raw-loader!../../tests/scala/readers/GeoJSONExamples.scala'; -import geojsonGbx from '!!raw-loader!../../tests/python/readers/geojson_gbx_examples.py'; - -# GeoJSON Reader - -Read GeoJSON into the shared OGR schema (`geom_0` WKB + `geom_0_srid` + -`geom_0_srid_proj` + attributes). The **lightweight** `geojson_gbx` reader -(pyogrio, JAR-free) and the **heavyweight** `geojson_ogr` reader are -interchangeable — see [Choosing an Execution Tier](../api/execution-tiers#the-one-line-swap). - - - - - - - - - - - - - -``` -(Analogous for ogr/shapefile/geopackage/filegdb — tab labels `Lightweight · ogr_gbx`/`shapefile_gbx`/`gpkg_gbx`/`file_gdb_gbx` and `Heavyweight · ogr`/`shapefile_ogr`/`gpkg_ogr`/`file_gdb_ogr`.) - -- [ ] **Step 3: Run the reader doc-tests (Docker) + internals-leak** - -```bash -gbx:test:python-docs --path readers/ --log vector-gbx-docs.log -grep -rn -iE "wave [0-9]+" docs/docs/ ; echo "exit:$?" -``` -Expected: the five new `*_gbx` reader doc-tests pass; internals-leak clean (`exit:1`). - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX .git/objects -git add docs/docs/readers/ogr.mdx docs/docs/readers/shapefile.mdx docs/docs/readers/geojson.mdx docs/docs/readers/geopackage.mdx docs/docs/readers/filegdb.mdx docs/tests/python/readers/ -git commit -m "docs(readers): lightweight tab for the five vector readers - -Co-authored-by: Isaac" -``` - ---- - -## Final verification (after all tasks) - -- [ ] Full light suite: `.venv-pyrx/bin/python -m pytest python/geobrix/test/ds python/geobrix/test/bench/test_cluster_vector.py python/geobrix/test/pyrx/test_serverless_no_spark_config.py -v` — green (vector parity SKIPs locally). -- [ ] Python lint (CI gate): `gbx:lint:python --check` (Docker) — clean. -- [ ] Docs build: Docusaurus build succeeds (no broken links / sidebar ids); the five vector pages render the lightweight tab + dropped the note. -- [ ] Binding parity unaffected: the `*_gbx` vector readers are DataSource formats, not registered functions — `gbx:test:bindings` unchanged. -- [ ] On-cluster (operator): `gbx:bench:cluster --benchmark-vector` runs the 5 light-vs-heavy reader pairs + row-count parity; fill the benchmarking.mdx vector table; the Docker `test_vector_parity.py` confirms schema+geometry parity. - -## Self-Review notes (plan vs spec) - -- **Spec coverage:** 5 readers → Tasks 3 (generic) + 4 (named); schema parity + type map → Task 2 (gated by Task 6 parity); options (driverName/asWKB/chunkSize/layerNumber/layerName) → Task 3; chunked partitioning + `/vsizip/` → Task 3; pyogrio/pyproj dep → Task 1; Serverless guard → Task 5; docs tabs + drop note → Task 8; per-reader bench + parity → Task 7; light-vs-heavy parity test → Task 6. -- **Naming consistency:** `OgrGbxReader`/`OgrGbxDataSource` + `_DRIVER` preset + `ShapefileGbxDataSource`/`GeoJSONGbxDataSource`/`GpkgGbxDataSource`/`FileGdbGbxDataSource`; format names `ogr_gbx`/`shapefile_gbx`/`geojson_gbx`/`gpkg_gbx`/`file_gdb_gbx`; heavy option names `driverName`/`asWKB`/`chunkSize`/`layerNumber`/`layerName`; `_vector_schema`/`_ogr_to_spark`/`_crs_to_srid_proj`/`_zip_vsi`/`_ChunkPartition` — used identically across tasks. -- **Known edges (gated by the parity test, Task 6), documented:** OGR→Spark type fidelity (int width, date/timestamp, bool subtype) — the map in Task 2 mirrors heavy's `getType`; if a corpus format reveals a mismatch, fix `_OGR_TO_SPARK`. Empty geometry / null geom rows → emit `None`/empty per heavy ("0"/""). Multi-geometry-field is out of scope (single `geom_0`, per spec). -- **Heavy option-name note:** the heavy reader parses `layerNumber` (per `OGR_Batch.scala`); the `ogr.mdx` options table historically said `layerN`. The light reader uses the heavy code's actual names (`layerNumber`/`layerName`); if the docs table is stale it can be corrected in Task 8. -- **Placeholder scan:** the only `` markers are explicit "move the current page body verbatim" instructions (Task 8) with the exact source named — not placeholders. The benchmarking.mdx vector table is intentionally filled from an operator cluster run (Task 7 step 7 / final verification), as the PMTiles section was. diff --git a/docs/superpowers/plans/2026-06-12-light-vector-writers.md b/docs/superpowers/plans/2026-06-12-light-vector-writers.md deleted file mode 100644 index ed2fa5a72..000000000 --- a/docs/superpowers/plans/2026-06-12-light-vector-writers.md +++ /dev/null @@ -1,1094 +0,0 @@ -# Light Vector Writers (`*_gbx`, pyogrio) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add net-new pure-Python vector writers (`ogr_gbx` + `shapefile_gbx`, `geojson_gbx`, `gpkg_gbx`, `file_gdb_gbx`) as PySpark DataSource V2 writers backed by pyogrio, so `read(ogr_gbx) → write(_gbx)` round-trips and the writers double as the Phase-3 benchmark-corpus generator. - -**Architecture:** Two-phase merge mirroring the PMTiles writer (`ds/pmtiles.py`): each executor `write()` serializes its partition's Arrow table (attributes + WKB geometry + srid/proj metadata columns) to one lossless Arrow-IPC fragment under a shared-FS scratch dir; the driver `commit()` reads every fragment, infers `geometry_type` + `crs` once over the combined data, then writes them into a single output file via `pyogrio.write_arrow` (first table plain, the rest with `append=True`). `abort()` cleans scratch + partial output. All writers live in the existing `ds/vector.py`; the named writers reuse the reader subclasses' `_DRIVER`. Pure-Python / Serverless-safe (pyogrio/pyarrow/shapely/pyproj lazy-imported inside methods). - -**Tech Stack:** Python 3.12, PySpark DataSource V2 (`pyspark.sql.datasource`), pyogrio 0.12.1 (`write_arrow`/`read_arrow`/`read_info`), pyarrow (`feather`), shapely (WKB↔WKT + geom_type), pyproj (CRS), pytest + local Spark (`local[2]`). - ---- - -## File Structure - -- **`python/geobrix/src/databricks/labs/gbx/ds/vector.py`** (MODIFY) — append writer classes/helpers below the existing readers. One file: readers + writers for the vector tier change together and share `_zip_vsi`, the OGR/CRS maps, and the geom-column conventions. - - `_VectorCommitMessage` (dataclass) — carries one fragment's Arrow-IPC path. - - `_geometry_type_of(wkb: bytes) -> str` — OGR geometry-type name from a WKB blob (shapely `geom_type`). - - `_srid_to_crs(srid: str, proj4: str) -> Optional[str]` — inverse of the reader's `_crs_to_spark`; `"4326"`→`"EPSG:4326"`, else proj4, else `None`. - - `_writer_col_roles(schema) -> (geom_col, srid_col, proj_col, attr_cols)` — derive column roles from the schema (the column `X` paired with `X_srid`). - - `OgrGbxWriter(DataSourceWriter)` — `write()`/`commit()`/`abort()`. - - `OgrGbxDataSource.writer(self, schema, overwrite)` — validates schema, builds the writer with the driver from `self._READER._DRIVER` (or `driverName` option). The four named `*GbxDataSource` subclasses inherit `.writer()` unchanged. -- **`python/geobrix/test/ds/test_vector_writer.py`** (CREATE) — unit + local-Spark round-trip / merge / CRS / mode tests. -- **`python/geobrix/test/ds/test_vector_writer_parity.py`** (CREATE) — Docker integration round-trip against the real corpus (`@pytest.mark.integration`). -- **`python/geobrix/test/pyrx/test_serverless_no_spark_config.py`** (MODIFY) — `vector.py` is already in the scanned list (readers); confirm it still passes after the writer additions. -- **`python/geobrix/src/databricks/labs/gbx/bench/readers.py`** (MODIFY) — add `run_vector_write(spark, light_fmt, path, out_dir)` returning `(seconds, roundtrip_ok)`. -- **`python/geobrix/src/databricks/labs/gbx/bench/cluster.py`** (MODIFY) — extend `_CELL_VECTOR` to call the writer timing + round-trip per format. -- **`docs/docs/writers/vector.mdx`** (CREATE) — lightweight-only vector writer page (a `:::note` that heavy has no vector writer). -- **`docs/docs/writers/overview.mdx`** (MODIFY) — add the vector writer to the lightweight Available-Writers table. -- **`docs/tests/python/api/vectorx_functions_*.py`** area / **`docs/tests/python/writers/`** (CREATE doc-test) — a real write→read round-trip exercised by `gbx:test:python-docs`. -- **`scripts/commands/gbx-data-generate-vector-corpus.{md,sh}`** (CREATE) — corpus-generator command wrapping the writers (Phase-3 enabler). - ---- - -## Task 1: Writer helpers (pure functions, no Spark) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (append after line 98, near `_zip_vsi`) -- Test: `python/geobrix/test/ds/test_vector_writer.py` (create) - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/ds/test_vector_writer.py -from shapely import Point, LineString, to_wkb - -from databricks.labs.gbx.ds.vector import ( - _geometry_type_of, - _srid_to_crs, - _writer_col_roles, -) -from pyspark.sql.types import ( - BinaryType, - IntegerType, - StringType, - StructField, - StructType, -) - - -def test_geometry_type_of_point_and_line(): - assert _geometry_type_of(to_wkb(Point(1, 2))) == "Point" - assert _geometry_type_of(to_wkb(LineString([(0, 0), (1, 1)]))) == "LineString" - - -def test_srid_to_crs(): - assert _srid_to_crs("4326", "") == "EPSG:4326" - assert _srid_to_crs("0", "+proj=longlat +datum=WGS84 +no_defs") == ( - "+proj=longlat +datum=WGS84 +no_defs" - ) - assert _srid_to_crs("0", "") is None - assert _srid_to_crs("", "") is None - - -def test_writer_col_roles_named_geom(): - schema = StructType( - [ - StructField("name", StringType()), - StructField("pop", IntegerType()), - StructField("SHAPE", BinaryType()), - StructField("SHAPE_srid", StringType()), - StructField("SHAPE_srid_proj", StringType()), - ] - ) - geom, srid, proj, attrs = _writer_col_roles(schema) - assert (geom, srid, proj) == ("SHAPE", "SHAPE_srid", "SHAPE_srid_proj") - assert attrs == ["name", "pop"] -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py -v` -Expected: FAIL — `ImportError: cannot import name '_geometry_type_of'`. - -- [ ] **Step 3: Write minimal implementation** - -Append to `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (after `_zip_vsi`, before `class _ChunkPartition`): - -```python -def _geometry_type_of(wkb: bytes) -> str: - """OGR geometry-type name (e.g. 'Point', 'MultiPolygon') from a WKB blob.""" - from shapely import from_wkb - - return from_wkb(bytes(wkb)).geom_type - - -def _srid_to_crs(srid: str, proj4: str): - """Inverse of the reader's CRS encoding: authority code -> 'EPSG:', - else the PROJ4 string, else None (CRS-less).""" - if srid and srid != "0": - return f"EPSG:{srid}" - if proj4: - return proj4 - return None - - -def _writer_col_roles(schema): - """(geom_col, srid_col, proj_col, attr_cols) derived from the reader schema: - the column X paired with X_srid is the geometry; X_srid_proj is its proj4; - everything else is an attribute. Mirrors how the parity test finds geom.""" - names = [f.name for f in schema.fields] - srid_cols = [n for n in names if n.endswith("_srid")] - if not srid_cols: - raise ValueError( - "vector writer input needs a geometry/'*_srid' column pair " - f"(from a *_gbx reader); got columns {names}" - ) - srid_col = srid_cols[0] - geom_col = srid_col[: -len("_srid")] - proj_col = geom_col + "_srid_proj" - if geom_col not in names: - raise ValueError(f"no geometry column '{geom_col}' for srid '{srid_col}'") - attr_cols = [n for n in names if n not in (geom_col, srid_col, proj_col)] - return geom_col, srid_col, proj_col, attr_cols -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py -v` -Expected: PASS (3 passed). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_vector_writer.py -git commit -m "feat(ds): vector writer helpers (geom-type, crs, col roles) - -Pure-function building blocks for the light vector writers: OGR geometry-type -name from WKB, srid/proj4 -> pyogrio crs, and schema column-role derivation -(the column paired with *_srid is the geometry). - -Co-authored-by: Isaac" -``` - ---- - -## Task 2: `OgrGbxWriter` + `OgrGbxDataSource.writer()` — generic GeoJSON round-trip (single partition) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (append writer class + `.writer()` on `OgrGbxDataSource`) -- Test: `python/geobrix/test/ds/test_vector_writer.py` - -- [ ] **Step 1: Write the failing test** - -Append to `test/ds/test_vector_writer.py`: - -```python -from shapely import from_wkb as _from_wkb - -from databricks.labs.gbx.ds.register import register - - -def _wkb_df(spark): - rows = [ - ("a", 10, bytearray(to_wkb(Point(-73.9, 40.7))), "4326", ""), - ("b", 20, bytearray(to_wkb(Point(-0.1, 51.5))), "4326", ""), - ] - return spark.createDataFrame( - rows, schema="name string, pop int, geom_0 binary, " - "geom_0_srid string, geom_0_srid_proj string" - ) - - -def test_geojson_roundtrip_single_partition(spark, tmp_path): - register(spark) - out = str(tmp_path / "out.geojson") - _wkb_df(spark).coalesce(1).write.format("ogr_gbx").mode("overwrite").option( - "driverName", "GeoJSON" - ).save(out) - - back = spark.read.format("ogr_gbx").load(out) - assert back.count() == 2 - got = {r["name"]: r["pop"] for r in back.collect()} - assert got == {"a": 10, "b": 20} - # geometry survives (derive geom col from schema, like parity tests) - gcol = [f.name for f in back.schema.fields if f.name.endswith("_srid")][0][:-5] - geoms = {_from_wkb(bytes(r[gcol])).geom_type for r in back.select(gcol).collect()} - assert geoms == {"Point"} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py::test_geojson_roundtrip_single_partition -v` -Expected: FAIL — `pyspark...` error that `ogr_gbx` has no writer (`writer() not implemented` / unsupported save). - -- [ ] **Step 3: Write minimal implementation** - -Add imports at the top of `vector.py` (extend the existing import lines): - -```python -import os -import shutil -import uuid -from dataclasses import dataclass -from typing import Dict, Iterator, List, Optional, Sequence, Tuple - -from pyspark.sql.datasource import ( - DataSource, - DataSourceReader, - DataSourceWriter, - InputPartition, - WriterCommitMessage, -) -``` - -Append to `vector.py` (after the named reader subclasses): - -```python -@dataclass -class _VectorCommitMessage(WriterCommitMessage): - frag_path: str - - -class OgrGbxWriter(DataSourceWriter): - """Two-phase vector writer: each partition -> one Arrow-IPC fragment in a - shared-FS scratch dir; the driver merges fragments into one output file via - pyogrio.write_arrow (first plain, rest append=True). Mirrors the PMTiles - writer's executor-scratch / driver-merge shape.""" - - def __init__(self, path, schema, driver, options, overwrite): - opts = {k.lower(): v for k, v in options.items()} - self.path = path - self.driver = options.get("driverName", "") or driver - if not self.driver: - raise ValueError( - "ogr_gbx writer requires a 'driverName' option (e.g. 'GeoJSON')." - ) - self.overwrite = overwrite - self.geometry_type_override = opts.get("geometrytype") - self.layer_name = opts.get("layername") - self.geom_col, self.srid_col, self.proj_col, self.attr_cols = ( - _writer_col_roles(schema) - ) - self._col_order = [f.name for f in schema.fields] - self._geom_is_wkb = any( - f.name == self.geom_col and isinstance(f.dataType, BinaryType) - for f in schema.fields - ) - parent = os.path.dirname(self.path) or "." - self.scratch_dir = os.path.join(parent, "_vec_scratch") - if not self.overwrite and self._target_exists(): - raise ValueError( - "ogr_gbx does not support append; use .mode('overwrite')." - ) - - def _target_exists(self) -> bool: - return os.path.exists(self.path) and ( - os.path.isfile(self.path) or bool(os.listdir(self.path)) - ) - - # ---- executor: partition rows -> one Arrow-IPC fragment ---- - def write(self, iterator: Iterator) -> WriterCommitMessage: - import pyarrow as pa - import pyarrow.feather as feather - from shapely import from_wkt, to_wkb - - idx = {n: i for i, n in enumerate(self._col_order)} - cols: Dict[str, list] = {n: [] for n in self._col_order} - for row in iterator: - for n in self._col_order: - v = row[idx[n]] - if n == self.geom_col and v is not None and not self._geom_is_wkb: - v = to_wkb(from_wkt(v)) # WKT input -> WKB - elif n == self.geom_col and v is not None: - v = bytes(v) - cols[n].append(v) - if not cols[self.geom_col]: - return _VectorCommitMessage(frag_path="") # empty partition - os.makedirs(self.scratch_dir, exist_ok=True) - tbl = pa.table({n: cols[n] for n in self._col_order}) - frag = os.path.join(self.scratch_dir, f"frag-{uuid.uuid4().hex}.arrow") - feather.write_feather(tbl, frag) - return _VectorCommitMessage(frag_path=frag) - - # ---- driver: merge fragments into one output file ---- - def commit(self, messages: List[Optional[WriterCommitMessage]]) -> None: - import pyarrow.feather as feather - import pyogrio - - frags = [ - m.frag_path - for m in messages - if isinstance(m, _VectorCommitMessage) and m.frag_path - ] - try: - if not frags: - return - self._prepare_target() - tables = [feather.read_table(f) for f in frags] - geom_type, crs = self._infer_geom_crs(tables) - kw = dict( - driver=self.driver, - geometry_name=self.geom_col, - geometry_type=geom_type, - crs=crs, - ) - if self.layer_name: - kw["layer"] = self.layer_name - for n, tbl in enumerate(tables): - out_tbl = tbl.drop_columns( - [c for c in (self.srid_col, self.proj_col) if c in tbl.column_names] - ) - pyogrio.write_arrow(out_tbl, self.path, append=(n > 0), **kw) - finally: - shutil.rmtree(self.scratch_dir, ignore_errors=True) - - def _infer_geom_crs(self, tables) -> Tuple[str, Optional[str]]: - geom_type, crs = self.geometry_type_override, None - for tbl in tables: - g = tbl.column(self.geom_col).to_pylist() - s = tbl.column(self.srid_col).to_pylist() if self.srid_col in tbl.column_names else [] - p = tbl.column(self.proj_col).to_pylist() if self.proj_col in tbl.column_names else [] - for i, gv in enumerate(g): - if gv is None: - continue - if geom_type is None: - geom_type = _geometry_type_of(gv) - if crs is None: - crs = _srid_to_crs( - s[i] if i < len(s) else "", p[i] if i < len(p) else "" - ) - break - if geom_type is not None and crs is not None: - break - return geom_type or "Unknown", crs - - def _prepare_target(self) -> None: - # PySpark may pre-create self.path as a directory; vector output is a - # single file (or driver-managed dir). Clear it and write directly — - # no os.rename (FUSE-unsafe on DBFS/Volumes); write_arrow writes - # sequentially so a direct write to a FUSE path is safe. - parent = os.path.dirname(self.path) or "." - os.makedirs(parent, exist_ok=True) - if os.path.isdir(self.path): - shutil.rmtree(self.path) - elif os.path.isfile(self.path): - os.remove(self.path) - - def abort(self, messages: List[Optional[WriterCommitMessage]]) -> None: - shutil.rmtree(self.scratch_dir, ignore_errors=True) - if os.path.isfile(self.path): - os.remove(self.path) - elif os.path.isdir(self.path): - shutil.rmtree(self.path, ignore_errors=True) -``` - -Add `.writer()` to `OgrGbxDataSource` (insert after its `reader()` method): - -```python - def writer(self, schema: StructType, overwrite: bool) -> DataSourceWriter: - path = self.options.get("path") - if not path: - raise ValueError("ogr_gbx writer requires an output path (.save(path)).") - return OgrGbxWriter( - path, schema, self._READER._DRIVER, dict(self.options), overwrite - ) -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py::test_geojson_roundtrip_single_partition -v` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_vector_writer.py -git commit -m "feat(ds): OgrGbxWriter two-phase vector writer (GeoJSON round-trip) - -Executor writes each partition to an Arrow-IPC scratch fragment; driver merges -into one file via pyogrio.write_arrow (append for fragments 2..n), inferring -geometry_type + crs from the data and dropping the srid/proj metadata columns. - -Co-authored-by: Isaac" -``` - ---- - -## Task 3: Multi-partition merge (no lost/duplicated rows) - -**Files:** -- Test: `python/geobrix/test/ds/test_vector_writer.py` - -- [ ] **Step 1: Write the failing test** - -Append to `test/ds/test_vector_writer.py`: - -```python -def test_multi_partition_merge(spark, tmp_path): - register(spark) - out = str(tmp_path / "multi.geojson") - rows = [ - (str(i), i, bytearray(to_wkb(Point(float(i) / 10.0, 40.0))), "4326", "") - for i in range(50) - ] - df = spark.createDataFrame( - rows, - schema="name string, pop int, geom_0 binary, " - "geom_0_srid string, geom_0_srid_proj string", - ).repartition(4) - assert df.rdd.getNumPartitions() == 4 - df.write.format("ogr_gbx").mode("overwrite").option( - "driverName", "GeoJSON" - ).save(out) - back = spark.read.format("ogr_gbx").load(out) - assert back.count() == 50 - assert {r["name"] for r in back.collect()} == {str(i) for i in range(50)} -``` - -- [ ] **Step 2: Run test to verify it fails or passes** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py::test_multi_partition_merge -v` -Expected: PASS if the Task-2 merge is correct. If it FAILS (e.g. only the last partition's rows present), the append loop in `commit()` is the bug — fix so every fragment is appended. - -- [ ] **Step 3: Fix if needed** - -If the test fails, confirm `commit()` iterates ALL fragments with `append=(n > 0)` and that `write()` returns a fragment per non-empty partition. No new code expected if Task 2 is correct. - -- [ ] **Step 4: Re-run to confirm PASS** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py -v` -Expected: PASS (all writer tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/ds/test_vector_writer.py -git commit -m "test(ds): vector writer multi-partition merge keeps all rows - -Co-authored-by: Isaac" -``` - ---- - -## Task 4: CRS + geometry_type inference + override, and `append` rejection - -**Files:** -- Test: `python/geobrix/test/ds/test_vector_writer.py` - -- [ ] **Step 1: Write the failing test** - -Append to `test/ds/test_vector_writer.py`: - -```python -import pytest -from shapely import Polygon - - -def test_crs_roundtrips(spark, tmp_path): - register(spark) - out = str(tmp_path / "crs.geojson") - _wkb_df(spark).coalesce(1).write.format("ogr_gbx").mode("overwrite").option( - "driverName", "GeoJSON" - ).save(out) - back = spark.read.format("ogr_gbx").load(out) - scol = [f.name for f in back.schema.fields if f.name.endswith("_srid")][0] - assert {r[scol] for r in back.select(scol).collect()} == {"4326"} - - -def test_geometry_type_override(spark, tmp_path): - register(spark) - out = str(tmp_path / "poly.geojson") - poly = to_wkb(Polygon([(0, 0), (0, 1), (1, 1), (0, 0)])) - df = spark.createDataFrame( - [("p", bytearray(poly), "4326", "")], - schema="name string, geom_0 binary, geom_0_srid string, " - "geom_0_srid_proj string", - ) - df.coalesce(1).write.format("ogr_gbx").mode("overwrite").option( - "driverName", "GeoJSON" - ).option("geometryType", "Polygon").save(out) - back = spark.read.format("ogr_gbx").load(out) - gcol = [f.name for f in back.schema.fields if f.name.endswith("_srid")][0][:-5] - assert _from_wkb(bytes(back.collect()[0][gcol])).geom_type == "Polygon" - - -def test_append_mode_rejected(spark, tmp_path): - register(spark) - out = str(tmp_path / "exists.geojson") - _wkb_df(spark).coalesce(1).write.format("ogr_gbx").mode("overwrite").option( - "driverName", "GeoJSON" - ).save(out) - with pytest.raises(Exception) as ei: - _wkb_df(spark).write.format("ogr_gbx").mode("append").option( - "driverName", "GeoJSON" - ).save(out) - assert "append" in str(ei.value).lower() -``` - -- [ ] **Step 2: Run test to verify it fails or passes** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py -k "crs or geometry_type or append_mode" -v` -Expected: `crs` and `geometry_type_override` PASS from Task 2; `append_mode_rejected` PASS because `OgrGbxWriter.__init__` raises when `not overwrite and target exists`. - -- [ ] **Step 3: Fix if needed** - -If `append` is not rejected (Spark may pass `overwrite=False` without an existing target on first save), confirm the guard triggers only when the target exists; the test pre-creates it, so the guard must fire. No new code expected. - -- [ ] **Step 4: Re-run to confirm** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py -v` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/ds/test_vector_writer.py -git commit -m "test(ds): vector writer CRS/geometry-type/override + append-rejection - -Co-authored-by: Isaac" -``` - ---- - -## Task 5: Named writers (`shapefile_gbx`, `geojson_gbx`, `gpkg_gbx`) round-trip - -**Files:** -- Test: `python/geobrix/test/ds/test_vector_writer.py` -- (No source change expected — named `*GbxDataSource` inherit `.writer()`; `self._READER._DRIVER` supplies the driver.) - -- [ ] **Step 1: Write the failing test** - -Append to `test/ds/test_vector_writer.py`: - -```python -@pytest.mark.parametrize( - "fmt,target", - [ - ("geojson_gbx", "named.geojson"), - ("gpkg_gbx", "named.gpkg"), - ("shapefile_gbx", "named.shp"), - ], -) -def test_named_writer_roundtrip(spark, tmp_path, fmt, target): - register(spark) - out = str(tmp_path / target) - _wkb_df(spark).coalesce(1).write.format(fmt).mode("overwrite").save(out) - back = spark.read.format(fmt).load(out) - assert back.count() == 2 - gcol = [f.name for f in back.schema.fields if f.name.endswith("_srid")][0][:-5] - assert { - _from_wkb(bytes(r[gcol])).geom_type for r in back.select(gcol).collect() - } == {"Point"} -``` - -- [ ] **Step 2: Run test to verify it passes (or surfaces a driver gap)** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py::test_named_writer_roundtrip -v` -Expected: `geojson_gbx` + `gpkg_gbx` PASS. `shapefile_gbx` writes sidecar files next to `named.shp`; the `shapefile_gbx` reader reads the `.shp` directly (its `_zip_vsi` only rewrites `.zip` paths, so a plain `.shp` passes through). If the named writer can't find a driver, the bug is `OgrGbxDataSource.writer()` not reading `self._READER._DRIVER` — fix it. - -- [ ] **Step 3: Fix if needed** - -If `shapefile_gbx` fails because Shapefile attribute-name truncation breaks the round-trip count/geometry, keep attributes short in the test (already `name`, `pop`) — no source change. If a named driver is missing, ensure `OgrGbxDataSource.writer()` passes `self._READER._DRIVER`. - -- [ ] **Step 4: Re-run to confirm** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py -v` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/ds/test_vector_writer.py -git commit -m "test(ds): named vector writers (geojson/gpkg/shapefile) round-trip - -Named *GbxDataSource subclasses inherit .writer(); the driver comes from the -paired reader's _DRIVER, so no per-writer subclass is needed. - -Co-authored-by: Isaac" -``` - ---- - -## Task 6: Serverless guard still green - -**Files:** -- Modify (verify): `python/geobrix/test/pyrx/test_serverless_no_spark_config.py` - -- [ ] **Step 1: Confirm `vector.py` is in the scanned modules** - -Run: `cd python/geobrix && grep -n "vector" test/pyrx/test_serverless_no_spark_config.py` -Expected: `vector.py` already listed (added when the readers landed). If absent, add it to the module list. - -- [ ] **Step 2: Run the guard** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/pyrx/test_serverless_no_spark_config.py -v` -Expected: PASS — the writer additions use only `pyogrio`/`pyarrow`/`shapely`/`os`/`shutil`/`uuid`, no `_jvm`/`.conf.set`/`.rdd`/`SparkConf`. - -- [ ] **Step 3: Fix if needed** - -If the guard flags a banned pattern, remove it (the writer must not touch Spark internals). No expected violations. - -- [ ] **Step 4: Commit (only if the test file changed)** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/pyrx/test_serverless_no_spark_config.py -git commit -m "test(pyrx): keep vector.py under the Serverless no-Spark-config guard - -Co-authored-by: Isaac" -``` - ---- - -## Task 7: FileGDB writer (best-effort, version-gated) - -**Files:** -- Test: `python/geobrix/test/ds/test_vector_writer.py` - -- [ ] **Step 1: Write the failing/skipping test** - -Append to `test/ds/test_vector_writer.py`: - -```python -def _ogr_can_create(driver: str) -> bool: - try: - import tempfile - - from shapely import Point, to_wkb - import pyarrow as pa - import pyogrio - - d = tempfile.mkdtemp() - path = d + ("/t.gdb" if driver == "OpenFileGDB" else "/t.out") - tbl = pa.table({"g": [to_wkb(Point(0, 0))]}) - pyogrio.write_arrow( - tbl, path, driver=driver, geometry_name="g", - geometry_type="Point", crs="EPSG:4326", - ) - return True - except Exception: - return False - - -def test_file_gdb_writer_roundtrip(spark, tmp_path): - register(spark) - if not _ogr_can_create("OpenFileGDB"): - pytest.skip("installed GDAL OpenFileGDB driver cannot create datasets") - out = str(tmp_path / "out.gdb") - _wkb_df(spark).coalesce(1).write.format("file_gdb_gbx").mode("overwrite").save(out) - back = spark.read.format("file_gdb_gbx").load(out) - assert back.count() == 2 -``` - -- [ ] **Step 2: Run test** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py::test_file_gdb_writer_roundtrip -v` -Expected: PASS or SKIP (skip if local GDAL OpenFileGDB is read-only). The Docker container's GDAL (DBR-aligned) is the authoritative check — see Task 11. - -- [ ] **Step 3: Fix if needed** - -No source change expected; `file_gdb_gbx` inherits `.writer()` and `_FileGdbReader._DRIVER = "OpenFileGDB"`. If create fails everywhere, leave the skip (FileGDB write is documented best-effort). - -- [ ] **Step 4: Re-run to confirm** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py -v` -Expected: PASS/SKIP, no errors. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/ds/test_vector_writer.py -git commit -m "test(ds): file_gdb_gbx writer round-trip (version-gated skip) - -Co-authored-by: Isaac" -``` - ---- - -## Task 8: Docker integration round-trip parity against the real corpus - -**Files:** -- Create: `python/geobrix/test/ds/test_vector_writer_parity.py` - -- [ ] **Step 1: Write the test** - -```python -"""Light vector writer round-trip (Docker / integration). - -read(_gbx) -> write(_gbx) -> read(_gbx) is feature-count and -geometry stable against the real corpus. Writer is light-only (heavy has no -vector writer); this is the writer's correctness gate. Skips unless sample -data is mounted.""" - -import os -from pathlib import Path - -import pytest - -pytestmark = pytest.mark.integration - -SAMPLE = ( - os.environ.get("GBX_SAMPLE_DATA_ROOT", "/Volumes/main/default/test-data").rstrip( - "/" - ) - + "/geobrix-examples" -) - -_CASES = [ - ("geojson_gbx", f"{SAMPLE}/nyc/boroughs/nyc_boroughs.geojson", "rt.geojson"), - ("gpkg_gbx", f"{SAMPLE}/nyc/geopackage/nyc_complete.gpkg", "rt.gpkg"), - ("shapefile_gbx", f"{SAMPLE}/nyc/subway/nyc_subway.shp.zip", "rt.shp"), -] - - -@pytest.fixture(scope="module") -def spark(): - import logging - - from pyspark.sql import SparkSession - - logging.getLogger("py4j").setLevel(logging.ERROR) - s = ( - SparkSession.builder.master("local[2]") - .appName("gbx-ds-vector-writer-parity") - .getOrCreate() - ) - from databricks.labs.gbx.ds.register import register - - register(s) - yield s - - -@pytest.mark.parametrize("fmt,src,target", _CASES) -def test_vector_writer_roundtrip(spark, tmp_path, fmt, src, target): - if not os.path.exists(src): - pytest.skip(f"sample not mounted: {src}") - src_df = spark.read.format(fmt).load(src) - n = src_df.count() - out = str(tmp_path / target) - src_df.coalesce(1).write.format(fmt).mode("overwrite").save(out) - back = spark.read.format(fmt).load(out) - assert back.count() == n - gcol = [f.name for f in back.schema.fields if f.name.endswith("_srid")][0][:-5] - assert back.where(f"{gcol} is not null").count() == n -``` - -- [ ] **Step 2: Run in Docker (dispatch a Task subagent — long-running)** - -Start the volumes container and run the integration test (sample data + `GBX_SAMPLE_DATA_ROOT` full bundle). Per `docker-volumes-for-integration-tests` memory: - -```bash -./scripts/docker/start_docker_with_volumes.sh -docker exec -e GBX_SAMPLE_DATA_ROOT=/Volumes/main/default/geobrix_samples geobrix-dev \ - bash -lc "cd /root/geobrix/python/geobrix && python -m pytest test/ds/test_vector_writer_parity.py -v" -``` - -Expected: 3 round-trips PASS (or SKIP if a corpus file is absent in the mounted bundle). - -- [ ] **Step 3: Fix if needed** - -If shapefile round-trip drops features due to field-name truncation, that is OGR Shapefile behavior, not a writer bug — assert on geometry count only (already done). If a format fails to write, fix the writer. - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/ds/test_vector_writer_parity.py -git commit -m "test(ds): Docker round-trip parity for light vector writers - -Co-authored-by: Isaac" -``` - ---- - -## Task 9: Bench — vector writer timing + round-trip gate - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/readers.py` (add `run_vector_write`) -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/cluster.py` (`_CELL_VECTOR` calls the writer path) - -- [ ] **Step 1: Add `run_vector_write` to `python/geobrix/src/databricks/labs/gbx/bench/readers.py`** - -Read `python/geobrix/src/databricks/labs/gbx/bench/readers.py` first to match the existing `run_format_read`/`run_pmtiles_write` style (timing helper, return shape). Then add: - -```python -def run_vector_write(spark, fmt, src_path, out_path, warmups=1, measured=1): - """Time read(fmt)->write(fmt) and assert read-back parity. Light-only - (no heavy vector writer). Returns (median_seconds, roundtrip_ok).""" - import time - - src = spark.read.format(fmt).load(src_path) - n = src.count() - - def _once(target): - t0 = time.perf_counter() - src.coalesce(1).write.format(fmt).mode("overwrite").save(target) - return time.perf_counter() - t0 - - for w in range(warmups): - _once(out_path + f".warm{w}") - times = [] - ok = True - for m in range(measured): - target = out_path + f".m{m}" - times.append(_once(target)) - back = spark.read.format(fmt).load(target) - gcol = [f.name for f in back.schema.fields if f.name.endswith("_srid")][0][:-5] - ok = ok and back.where(f"{gcol} is not null").count() == n - times.sort() - return times[len(times) // 2], ok -``` - -- [ ] **Step 2: Wire into `_CELL_VECTOR` in `python/geobrix/src/databricks/labs/gbx/bench/cluster.py`** - -Read `python/geobrix/src/databricks/labs/gbx/bench/cluster.py`'s `_CELL_VECTOR` block. After the existing reader parity per format, add a writer leg guarded by the existing `BENCHMARK_VECTOR` flag — for each corpus format, call `run_vector_write` and record `light_write_s` + `roundtrip_ok` into the same results row the readers use. Keep the heavy columns null for the writer leg (no heavy vector writer). - -- [ ] **Step 3: Smoke-test the bench cell locally (no cluster)** - -Run a 1-tile/1-format local smoke through the existing bench smoke entrypoint (mirror `test-logs/bench-readers-smoke.log`'s invocation) to confirm `run_vector_write` imports and returns a tuple. Expected: a printed `(seconds, True)`. - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX .git/objects -git add bench/readers.py bench/cluster.py -git commit -m "feat(bench): vector writer timing + round-trip gate in --benchmark-vector - -Light-only write timing per format with a read-back parity assertion (the -writer's correctness gate; no heavy vector writer to compare). - -Co-authored-by: Isaac" -``` - ---- - -## Task 10: Docs — lightweight-only vector writer page + concurrency framing - -**Files:** -- Create: `docs/docs/writers/vector.mdx` -- Modify: `docs/docs/writers/overview.mdx` (add the row + a prominent concurrency/perf section) -- Modify: `docs/docs/readers/overview.mdx` (add the same concurrency/perf section) -- Create: a doc-test under `docs/tests/python/writers/` exercised by `gbx:test:python-docs` - -**User directive — make the distributed advantage a "big deal" (esp. on the overview pages).** -The light readers/writers are NOT single-node rasterio/pyogrio wrappers — they are Spark -DataSource V2 connectors that parallelize across the cluster. State this prominently and -concretely (factual, not marketing) in BOTH overview pages' lightweight tab, e.g. an -admonition / "Why this scales" subsection covering: -- **Readers — partitioned parallel reads:** vector readers slice features by `chunkSize` - into partitions read concurrently across executors; raster readers split large files by - `sizeInMB`. A single-node `pyogrio.read_*` / `rasterio.open` call reads one file - sequentially on one machine; these readers fan the work across the cluster and yield a - distributed DataFrame ready for joins/aggregations with no driver-side `collect`. This - scales past a single machine's memory. -- **Writers — per-partition parallel writes + driver merge:** each executor writes its - partition concurrently to a scratch fragment; the driver merges into the final output - (two-phase). A single-node `pyogrio.write_*` serializes one file on one machine. -- **PMTiles writer — distributed spatial sharding:** partitions tiles into bounded - per-shard archives written in parallel, then catalogs them — horizontal scaling vs a - single memory-bound archive on one node. The merge is FUSE/object-store-safe (sequential, - no `os.rename`) so it works on UC Volumes / DBFS. -Keep it concrete and tie each claim to the actual mechanism/option. No internal vocab -(no "wave"); no empty superlatives — describe the mechanism, let it speak. - -- [ ] **Step 1: Write the doc-test (the doc's source of truth)** - -Create `docs/tests/python/writers/vector_gbx_write.py` with a real write→read round-trip against the corpus (follow the existing reader doc-test pattern in `docs/tests/python/readers/`; read `GBX_SAMPLE_DATA_ROOT`). It must execute and assert (feature count stable), not just compile. - -```python -# docs/tests/python/writers/vector_gbx_write.py -import os - -from pyspark.sql import SparkSession - -from databricks.labs.gbx.ds.register import register - - -def write_vector_gbx_example(): - spark = SparkSession.builder.getOrCreate() - register(spark) - root = os.environ.get("GBX_SAMPLE_DATA_ROOT", "/Volumes/main/default/test-data") - src = f"{root}/geobrix-examples/nyc/boroughs/nyc_boroughs.geojson" - out = "/tmp/boroughs_out.geojson" - - df = spark.read.format("geojson_gbx").load(src) - df.coalesce(1).write.format("geojson_gbx").mode("overwrite").save(out) - - back = spark.read.format("geojson_gbx").load(out) - assert back.count() == df.count() - return out -``` - -- [ ] **Step 2: Create `docs/docs/writers/vector.mdx`** - -Lightweight-only page (no heavyweight tab; a `:::note` says heavy has no vector writer), importing the doc-test via raw-loader. Match the structure of `docs/docs/writers/pmtiles.mdx`: - -```mdx ---- -sidebar_position: 5 ---- - -import CodeBlock from '@theme/CodeBlock'; -import VectorWrite from '!!raw-loader!../../tests/python/writers/vector_gbx_write.py'; - -# Vector Writer - -`geojson_gbx` / `shapefile_gbx` / `gpkg_gbx` / `file_gdb_gbx` / `ogr_gbx` — pure-Python -DataSource V2 writers (pyogrio). They take the light vector reader's schema -(`…attributes, geom_0` WKB, `geom_0_srid`, `geom_0_srid_proj`), so -`read → write` round-trips. - -:::note No heavyweight equivalent -The heavyweight tier has no vector writer; vector output flows through Spark's -built-in writers. These `*_gbx` writers are lightweight-only. -::: - -:::note Register first -Call `register(spark)` once before using any `*_gbx` format (see the -[Writers Overview](./overview)). -::: - -## Options - -| Option | Default | Behavior | -|---|---|---| -| `driverName` | required for `ogr_gbx`; preset by named writers | OGR driver. | -| `mode` | `overwrite` | `overwrite` only; `append` is rejected. | -| `geometryType` | inferred from the data | Override the OGR geometry type. | -| `layerName` | driver default | Output layer name where supported. | - -## Example - -{VectorWrite} -``` - -- [ ] **Step 3: Add the row to `docs/docs/writers/overview.mdx`** - -In the lightweight `### Available Writers` table (after the PMTiles row at line 34), add: - -``` -| [Vector Writer](./vector) | `geojson_gbx` / `shapefile_gbx` / `gpkg_gbx` / `file_gdb_gbx` / `ogr_gbx` | Pure-Python vector writers (pyogrio); round-trip with the `*_gbx` readers. | -``` - -- [ ] **Step 4: Run the doc-test in Docker + build docs (dispatch a Task subagent)** - -Run: `gbx:test:python-docs --path docs/tests/python/writers/` and `gbx:docs:start` build check. -Expected: doc-test PASSES; docs build clean; `grep -rn -iE "wave [0-9]+" docs/docs/writers/vector.mdx` prints nothing. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add docs/docs/writers/vector.mdx docs/docs/writers/overview.mdx docs/tests/python/writers/vector_gbx_write.py -git commit -m "docs(writers): lightweight-only Vector writer page + overview row - -Co-authored-by: Isaac" -``` - ---- - -## Task 11: Corpus-generator command (Phase-3 enabler) - -**Files:** -- Create: `scripts/commands/gbx-data-generate-vector-corpus.md` -- Create: `scripts/commands/gbx-data-generate-vector-corpus.sh` - -- [ ] **Step 1: Write the `.md` registration** - -Create `scripts/commands/gbx-data-generate-vector-corpus.md` (follow `scripts/commands/gbx-data-generate-minimal-bundle.md`): title, 1-2 sentence description, usage `bash scripts/commands/gbx-data-generate-vector-corpus.sh [OPTIONS]`, options (`--format`, `--features`, `--geometry`, `--out`, `--log`, `--help`), 2 examples. - -- [ ] **Step 2: Write the `.sh` implementation** - -Create `scripts/commands/gbx-data-generate-vector-corpus.sh` sourcing `common.sh` (for `check_docker`, `resolve_log_path`, `setup_log_file`, `show_banner`, `SCRIPT_DIR`/`PROJECT_ROOT` per the CLAUDE.md procedure). It runs inside the dev container and invokes a small Python that uses the `*_gbx` writer to emit N synthetic features of a chosen geometry type/format to `--out`. Real behavior, no placeholders; non-zero exit on failure. The generator Python: - -```python -# emitted/run inside the container by the .sh -import sys - -from pyspark.sql import SparkSession -from shapely import Point, to_wkb - -from databricks.labs.gbx.ds.register import register - -fmt, n, out = sys.argv[1], int(sys.argv[2]), sys.argv[3] -spark = SparkSession.builder.getOrCreate() -register(spark) -rows = [ - (str(i), i, bytearray(to_wkb(Point(float(i % 360) - 180.0, float(i % 170) - 85.0))), - "4326", "") - for i in range(n) -] -df = spark.createDataFrame( - rows, - schema="name string, val int, geom_0 binary, geom_0_srid string, " - "geom_0_srid_proj string", -) -df.write.format(fmt).mode("overwrite").save(out) -print(f"wrote {n} features to {out} as {fmt}") -``` - -- [ ] **Step 3: Make executable + smoke-test** - -```bash -chmod +x scripts/commands/gbx-data-generate-vector-corpus.sh -bash scripts/commands/gbx-data-generate-vector-corpus.sh --help -``` -Expected: prints usage, exit 0. Then a tiny in-container run (`--format geojson_gbx --features 100 --out /tmp/corpus_test.geojson`) prints the wrote-line and exits 0. - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX .git/objects -git add scripts/commands/gbx-data-generate-vector-corpus.md scripts/commands/gbx-data-generate-vector-corpus.sh -git commit -m "feat(data): gbx:data:generate-vector-corpus command (writer-backed) - -Generates synthetic vector data via the *_gbx writers for Phase-3 scaled -benchmarking; runs in the dev container. - -Co-authored-by: Isaac" -``` - ---- - -## Task 12: Lint + full vector test sweep before handoff - -**Files:** none (verification) - -- [ ] **Step 1: Python lint (CI gate)** - -Run (per `run-python-lint-before-push` + `host-vs-docker-black-mismatch` memories — verify with the Docker check, not just host `--fix`): -`gbx:lint:python --check` -Expected: isort/black/flake8 clean for `vector.py`, the new tests, and `bench/*`. - -- [ ] **Step 2: Full local vector DataSource sweep** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_writer.py test/ds/test_vector_reader.py test/ds/test_vector_named.py test/ds/test_vector_schema.py test/ds/test_register.py -v` -Expected: all PASS (FileGDB write may SKIP locally). - -- [ ] **Step 3: Serverless guard final** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/pyrx/test_serverless_no_spark_config.py -v` -Expected: PASS. - -- [ ] **Step 4: Commit any lint fixes** - -```bash -chmod -R u+rwX .git/objects -git add -A -git commit -m "chore(ds): lint fixes for light vector writers - -Co-authored-by: Isaac" -``` - ---- - -## Self-Review - -**1. Spec coverage** (spec: `docs/superpowers/specs/2026-06-12-light-vector-writers-design.md`): -- Two-phase merge to one file → Tasks 2, 3 (executor fragment + driver merge; shared-FS scratch; FUSE-safe direct write; abort cleanup). -- Geometry + CRS handling (WKB geom col, WKT→WKB, inferred geometry_type + override, crs from srid/proj, srid/proj consumed not written) → Tasks 2, 4. -- Options table (`driverName`, `mode` overwrite-only/append-rejected, `geometryType`, `layerName`) → Tasks 2, 4. -- Architecture/files (`OgrGbxWriter`, `.writer()`, helpers, Serverless-safe lazy imports) → Tasks 1, 2, 6. -- Five writers (`ogr_gbx` + 4 named) → Tasks 2, 5, 7. -- Docs (lightweight-only page + note + doc-test round-trip) → Task 10. -- Benchmark (light-only writer timing + round-trip gate, in `--benchmark-vector`; benchmarking.mdx) → Task 9 (benchmarking.mdx vector-writer subsection is appended when the Task-9 numbers land — tracked, not a placeholder, since timings don't exist until the bench runs). -- Corpus generator (Phase-3 enabler) → Task 11. -- Testing (round-trip per format, multi-partition, CRS/geom-type, mode, Serverless, Docker) → Tasks 2–8. -- Out of scope (multi-geom field, heavy vector writer) → honored (single `geom_0`/derived geom col only). - -**2. Placeholder scan:** No "TBD/TODO". The only deferred item is the benchmarking.mdx writer numbers, which legitimately do not exist until Task 9's bench runs; the doc edit is part of the bench-run follow-up, consistent with how reader numbers were handled. - -**3. Type consistency:** `_VectorCommitMessage.frag_path` (str) used in `write`/`commit`; `_writer_col_roles` returns `(geom_col, srid_col, proj_col, attr_cols)` consistently; `OgrGbxWriter.__init__(path, schema, driver, options, overwrite)` matches the `OgrGbxDataSource.writer()` call; `run_vector_write(...) -> (seconds, ok)` consistent between `readers.py` and `cluster.py`. Driver source-of-truth is `self._READER._DRIVER` in both reader and writer paths. - ---- - -## Execution Handoff - -Plan complete. Recommended: subagent-driven-development (fresh subagent per task, two-stage review). Tasks 8, 10 dispatch Docker subagents (long-running); Task 9's cluster bench number-fill follows the next `--benchmark-vector` run. diff --git a/docs/superpowers/plans/2026-06-12-readers-writers-tabbed-tiers.md b/docs/superpowers/plans/2026-06-12-readers-writers-tabbed-tiers.md deleted file mode 100644 index 0cf5a29e3..000000000 --- a/docs/superpowers/plans/2026-06-12-readers-writers-tabbed-tiers.md +++ /dev/null @@ -1,677 +0,0 @@ -# Readers & Writers Tabbed Tiers Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Reorganize the Readers & Writers docs by **format** (not tier): one page per format with synced **Lightweight (default/first) / Heavyweight** tabs; single-tier formats stay single pages with a note. - -**Architecture:** Merge each light+heavy page-pair into one format-named `.mdx` using Docusaurus `` (the convention already used by quick-start / the overviews, so the tier choice syncs site-wide). The heavy/light bodies move verbatim into their tabs — the imported example code (and the executable doc-tests) are unchanged. Single-tier (vector) pages get a "no lightweight equivalent yet" note. Sidebars regroup under Readers → General/Named and Writers → General/Named. - -**Tech Stack:** Docusaurus MDX, `@theme/Tabs`/`@theme/TabItem`, the repo's `CodeFromTest` component, the `gbx:test:python-docs` doc-test harness, `gbx:docs:*` build commands. - -**Reference spec:** `docs/superpowers/specs/2026-06-12-readers-writers-tabbed-tiers-design.md` - ---- - -## Conventions for this plan - -- **Tabs convention (match the existing site):** every tier tab group uses - `` with **`` FIRST** then ``. The shared `groupId` + `queryString` sync the choice across all pages and the URL; **lightweight first = lightweight default** on a fresh visit. (The spec wrote `groupId="tier"`/`light`/`heavy`; we use the established `gbx-tier`/`lightweight`/`heavyweight` so the new pages sync with quick-start and the overviews.) -- **Tab labels** carry the format/engine name: `label="Lightweight · raster_gbx"`, `label="Heavyweight · gdal"`. (Labels may differ per page; the `value` is what syncs.) -- **"Move the body verbatim"** means: take everything in the source page **after** its frontmatter and `import` lines, and paste it unchanged inside the target `` — keep every `` call and its props exactly. Do NOT edit the imported example `.py`/`.scala` files. -- **No redirects** (beta/WIP — old URLs may break). -- **Commits:** before each `git commit` run `chmod -R u+rwX .git/objects`; trailer is exactly `Co-authored-by: Isaac` (repo convention; a security linter may warn — ignore it; never a human name); subjects ≤72 chars. No push (the operator pushes at the end). -- **Docs build/test:** `bash scripts/commands/gbx-docs-start.sh` (or `gbx:docs:dev`) for a local build; `gbx:test:python-docs --path readers/` / `--path writers/` for doc-tests (Docker). Run doc-test/build steps via a Task subagent (they touch Docker / take minutes). -- **MDX gotcha:** a `` body that starts with a Markdown heading or import must have a blank line after the `` tag and before ``. Keep the existing pages' blank-line spacing when moving bodies. - -## File map - -**Create (merged, tabbed):** -- `docs/docs/readers/raster.mdx` ← `raster_gbx.mdx` (light) + `gdal.mdx` (heavy) -- `docs/docs/readers/geotiff.mdx` ← `gtiff_gbx.mdx` (light) + `gtiff.mdx` (heavy) -- `docs/docs/writers/raster.mdx` ← `raster_gbx.mdx` (light) + `gdal.mdx` (heavy) -- `docs/docs/writers/geotiff.mdx` ← `gtiff_gbx.mdx` (light) + new heavy tab (`WRITE_GTIFF_GDAL` from `gdal_examples.py`) - -**Modify in place (merge into existing id):** -- `docs/docs/writers/pmtiles.mdx` ← add light `pmtiles_gbx` tab (from `pmtiles_gbx.mdx`) above the existing heavy body - -**Delete (folded into the above):** -- `docs/docs/readers/raster_gbx.mdx`, `docs/docs/readers/gdal.mdx`, `docs/docs/readers/gtiff_gbx.mdx`, `docs/docs/readers/gtiff.mdx` -- `docs/docs/writers/raster_gbx.mdx`, `docs/docs/writers/gdal.mdx`, `docs/docs/writers/gtiff_gbx.mdx`, `docs/docs/writers/pmtiles_gbx.mdx` - -**Modify (labels + note):** `docs/docs/readers/{ogr,shapefile,geojson,geopackage,filegdb}.mdx` - -**Modify (light-first + relink):** `docs/docs/readers/overview.mdx`, `docs/docs/writers/overview.mdx` - -**Modify (light-first sweep):** `docs/docs/installation.mdx`, `docs/docs/api/raster-functions.mdx` (+ confirm `quick-start.mdx` is already light-first) - -**Modify:** `docs/sidebars.js` (Readers & Writers block), plus cross-links across `docs/docs/**`. - ---- - -### Task 1: Merged `readers/raster.mdx` (Raster reader — light + heavy tabs) - -**Files:** -- Create: `docs/docs/readers/raster.mdx` -- Read (sources to fold): `docs/docs/readers/raster_gbx.mdx`, `docs/docs/readers/gdal.mdx` - -- [ ] **Step 1: Create the merged page scaffold** - -Create `docs/docs/readers/raster.mdx` with this exact head, then fill the two tab bodies per Steps 2–3: - -```mdx ---- -sidebar_position: 1 -sidebar_label: Raster ---- - -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import CodeFromTest from '@site/src/components/CodeFromTest'; -import rasterGbxExamples from '!!raw-loader!../../tests/python/readers/raster_gbx_read_examples.py'; -import gdalExamples from '!!raw-loader!../../tests/python/readers/gdal_examples.py'; -import gdalScala from '!!raw-loader!../../tests/scala/readers/GDALExamples.scala'; - -# Raster Reader - -Read rasters into the shared `(source, tile)` schema. GeoBrix offers two -interchangeable tiers: a **lightweight** pure-Python/PySpark reader (`raster_gbx`, -`rasterio`-backed, JAR-free, Serverless-safe) and a **heavyweight** GDAL-backed -reader (`gdal`). They emit the same schema, so swapping is a one-line -`format(...)` change — see [Choosing an Execution Tier](../api/execution-tiers#the-one-line-swap). - -> The heavyweight `gdal` reader supports the full set of GDAL drivers (NetCDF, -> HDF5, COG, …); the lightweight reader covers the common raster path. The pairing -> is a corresponding *general raster reader* per tier, not a feature-identical one. - - - - - - - - - - - - - -``` - -- [ ] **Step 2: Fill the Lightweight tab** - -Replace `` with the **body of `docs/docs/readers/raster_gbx.mdx` moved verbatim** — everything after its `import` lines (i.e. from `## Register` onward, including the `:::note Lightweight readers and writers are not auto-registered` admonition, `## Read (catch-all)`, `## Options`, and `## Performance vs the heavyweight reader`). Keep all `` calls exactly. Drop its old `# Lightweight Raster Reader (\`raster_gbx\`)` H1 (the page H1 + tab already establish context); start the tab body at `## Register`. Re-point any in-body relative links per Task 10 (do not worry about them now). - -- [ ] **Step 3: Fill the Heavyweight tab** - -Replace `` with the **body of `docs/docs/readers/gdal.mdx` moved verbatim** — everything after its `import` lines (drop its `# GDAL Reader` H1; start at its first section). Keep all `` and `gdalScala` usages exactly. - -- [ ] **Step 4: Delete the two source pages** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -git rm docs/docs/readers/raster_gbx.mdx docs/docs/readers/gdal.mdx -``` - -- [ ] **Step 5: Sanity-check the merged page parses (imports + tabs balanced)** - -Run: -```bash -grep -c "" docs/docs/readers/raster.mdx # expect 2 -grep -c "raw-loader" docs/docs/readers/raster.mdx # expect 3 (rasterGbx, gdal py, gdal scala) -``` -Expected: `2`, `2`, `3`. - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add docs/docs/readers/raster.mdx docs/docs/readers/raster_gbx.mdx docs/docs/readers/gdal.mdx -git commit -m "docs(readers): merge raster_gbx + gdal into tabbed Raster reader - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: Merged `readers/geotiff.mdx` (GeoTIFF reader — light + heavy tabs) - -**Files:** -- Create: `docs/docs/readers/geotiff.mdx` -- Read (sources): `docs/docs/readers/gtiff_gbx.mdx`, `docs/docs/readers/gtiff.mdx` - -- [ ] **Step 1: Create the merged page scaffold** - -```mdx ---- -sidebar_position: 1 -sidebar_label: GeoTIFF ---- - -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import CodeFromTest from '@site/src/components/CodeFromTest'; -import rasterGbxExamples from '!!raw-loader!../../tests/python/readers/raster_gbx_read_examples.py'; -import gtiffExamples from '!!raw-loader!../../tests/python/readers/gtiff_examples.py'; -import gtiffScala from '!!raw-loader!../../tests/scala/readers/GTiffExamples.scala'; - -# GeoTIFF Reader - -Read GeoTIFFs into the shared `(source, tile)` schema. The **lightweight** -`gtiff_gbx` reader (the `raster_gbx` catch-all with the GeoTIFF driver preset, -`rasterio`-backed, JAR-free) and the **heavyweight** `gtiff_gdal` reader are -interchangeable — see [Choosing an Execution Tier](../api/execution-tiers#the-one-line-swap). - - - - - - - - - - - - - -``` - -- [ ] **Step 2: Fill the Lightweight tab** with the body of `docs/docs/readers/gtiff_gbx.mdx` moved verbatim (everything after its imports; drop its H1). Keep the `` call. Replace its "Register the lightweight DataSources first (see …)" link target per Task 10. - -- [ ] **Step 3: Fill the Heavyweight tab** with the body of `docs/docs/readers/gtiff.mdx` moved verbatim (after imports; drop its `# GeoTIFF Reader` H1). Keep all `gtiffExamples`/`gtiffScala` `` calls. - -- [ ] **Step 4: Delete sources** - -```bash -git rm docs/docs/readers/gtiff_gbx.mdx docs/docs/readers/gtiff.mdx -``` - -- [ ] **Step 5: Sanity-check** - -```bash -grep -c "" docs/docs/readers/geotiff.mdx # 2 -``` -Expected: `2`, `2`. - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add docs/docs/readers/geotiff.mdx docs/docs/readers/gtiff_gbx.mdx docs/docs/readers/gtiff.mdx -git commit -m "docs(readers): merge gtiff_gbx + gtiff_gdal into tabbed GeoTIFF reader - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: Merged `writers/raster.mdx` (Raster writer — light + heavy tabs) - -**Files:** -- Create: `docs/docs/writers/raster.mdx` -- Read (sources): `docs/docs/writers/raster_gbx.mdx`, `docs/docs/writers/gdal.mdx` - -- [ ] **Step 1: Create the merged page scaffold** - -```mdx ---- -sidebar_position: 1 -sidebar_label: Raster ---- - -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import CodeFromTest from '@site/src/components/CodeFromTest'; -import rasterGbxWrite from '!!raw-loader!../../tests/python/writers/raster_gbx_write_examples.py'; -import gdalWriteExamples from '!!raw-loader!../../tests/python/writers/gdal_examples.py'; -import gdalWriteScala from '!!raw-loader!../../tests/scala/writers/GDALWriteExamples.scala'; - -# Raster Writer - -Write raster tiles from the shared `(source, tile)` schema. The **lightweight** -`raster_gbx` writer (`rasterio`-backed, JAR-free, supports `overwrite`) and the -**heavyweight** GDAL-backed `gdal` writer (append-only) take the same schema — see -[Choosing an Execution Tier](../api/execution-tiers#the-one-line-swap). - - - - - - - - - - - - - -``` - -- [ ] **Step 2: Fill the Lightweight tab** with the body of `docs/docs/writers/raster_gbx.mdx` moved verbatim (after imports; drop its H1; start at `## Write raster tiles`). Keep its `:::note Register before writing` admonition and all `` calls. - -- [ ] **Step 3: Fill the Heavyweight tab** with the body of `docs/docs/writers/gdal.mdx` moved verbatim (after imports; drop its `# GDAL Writer` H1). Keep all `gdalWriteExamples`/`gdalWriteScala` `` calls. - -- [ ] **Step 4: Delete sources** - -```bash -git rm docs/docs/writers/raster_gbx.mdx docs/docs/writers/gdal.mdx -``` - -- [ ] **Step 5: Sanity-check** - -```bash -grep -c " - - - - - - - -`gtiff_gdal` is the GDAL writer restricted to the GeoTIFF driver — it reads and -writes `.tif` rasters and is **append-only** (use the lightweight writer for -`overwrite`). It takes the same `(source, tile)` schema as the other writers. - - - -See the [Raster Writer → Heavyweight](./raster?tier=heavyweight) tab for the full -GDAL-writer options (`path` / `nameCol` / `ext`, format & compression from -`tile.metadata`). - - - -``` - -- [ ] **Step 2: Fill the Lightweight tab** with the body of `docs/docs/writers/gtiff_gbx.mdx` moved verbatim (after imports; drop its H1). Keep its `` call. Re-point its `./raster_gbx` links per Task 10. - -- [ ] **Step 3: Delete the light source page** - -```bash -git rm docs/docs/writers/gtiff_gbx.mdx -``` - -- [ ] **Step 4: Sanity-check + confirm the heavy example exists** - -```bash -grep -c " - -# PMTiles Writer - -Package a tile pyramid (`(z, x, y, bytes)`) into [PMTiles](https://docs.protomaps.com/pmtiles/) -archives. The **lightweight** `pmtiles_gbx` writer (pure-Python, Serverless-safe, -distributed spatial sharding) and the **heavyweight** `pmtiles` writer take the -same input and produce decoded-tile-identical archives (verified in the -[benchmark](../api/benchmarking#results--tiled-output-pmtiles-writer)). - - - - - - - - - - - - - -``` -Move the `pmtiles_gbx.mdx` body (after its imports, drop its H1) into the Lightweight tab, and the **current** `pmtiles.mdx` body (after its frontmatter + `# PMTiles Writer` H1) into the Heavyweight tab. Hoist any raw-loader import the heavy body needs into the import block. - -- [ ] **Step 3: Delete the light source page** - -```bash -git rm docs/docs/writers/pmtiles_gbx.mdx -``` - -- [ ] **Step 4: Sanity-check** - -```bash -grep -c "` before ``. **Reorder so the `lightweight` TabItem comes first** (move the whole ` … ` block above the `heavyweight` one). Do not change the tab contents yet beyond reordering. - -- [ ] **Step 2: Repoint format links + tables** in both overviews to the new page ids: - - `readers/raster_gbx` and `readers/gdal` → `readers/raster` - - `readers/gtiff_gbx` and `readers/gtiff` → `readers/geotiff` - - `writers/raster_gbx` and `writers/gdal` → `writers/raster` - - `writers/gtiff_gbx` → `writers/geotiff` - - `writers/pmtiles_gbx` → `writers/pmtiles` - - vector readers (`ogr`/`shapefile`/`geojson`/`geopackage`/`filegdb`) unchanged. - In each overview's reader/writer **table**, collapse the separate light/heavy rows for raster & GeoTIFF into one row per format that links to the merged page (the tier is now a tab, not a page). - -- [ ] **Step 3: Verify no dangling links to deleted pages** remain in the overviews: -```bash -grep -nE "raster_gbx|writers/gdal|readers/gdal|gtiff_gbx|readers/gtiff[^a-z]|pmtiles_gbx" docs/docs/readers/overview.mdx docs/docs/writers/overview.mdx ; echo "exit:$?" -``` -Expected: no matches (`exit:1`). - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX .git/objects -git add docs/docs/readers/overview.mdx docs/docs/writers/overview.mdx -git commit -m "docs(readers/writers): light-first overviews relinked to merged pages - -Co-authored-by: Isaac" -``` - ---- - -### Task 8: Light-first sweep for the remaining `gbx-tier` tab groups - -**Files:** -- Modify: `docs/docs/installation.mdx`, `docs/docs/api/raster-functions.mdx` -- Verify only: `docs/docs/quick-start.mdx` (already light-first) - -- [ ] **Step 1: For each `` in `installation.mdx` and `raster-functions.mdx`, ensure the `` block is FIRST.** If a group has `heavyweight` first, move the `lightweight` `…` block above it (content unchanged). This makes lightweight the default on a fresh visit, consistently with every other tier tab group. - -- [ ] **Step 2: Confirm quick-start is already light-first** (no change expected): -```bash -awk '/` markers are explicit move instructions (move the named source page's body verbatim), not placeholders; each has an exact source file + what to drop (frontmatter/imports/H1). No TBD/TODO. -- **No-doc-test-churn:** confirmed — every `CodeFromTest` source/example file referenced by the merged pages already exists and is unchanged; the only "new" content (Task 4 heavy GeoTIFF tab) reuses the existing tested `WRITE_GTIFF_GDAL` example. diff --git a/docs/superpowers/plans/2026-06-12-scaled-vector-bench-corpus.md b/docs/superpowers/plans/2026-06-12-scaled-vector-bench-corpus.md deleted file mode 100644 index b40521d54..000000000 --- a/docs/superpowers/plans/2026-06-12-scaled-vector-bench-corpus.md +++ /dev/null @@ -1,592 +0,0 @@ -# Scaled Vector Benchmark Corpus + Bench Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Generate a realistic at-scale vector benchmark corpus (1M-polygon seed per format → GeoJSON/Shapefile/GeoPackage/FileGDB → replicate ×100), wire it into the cluster vector bench, run it, and fill the per-page + central benchmark numbers with real light-vs-heavy figures. - -**Architecture:** A pure-Python corpus generator (`bench/corpus_vector.py`) mints a 1M-row polygon DataFrame in the writer schema, transcodes it to each format via the existing `*_gbx` writers (FileGDB via the native-osgeo hybrid — cluster only), and replicates each seed into a per-format directory on the bench Volume. A `gbx:bench:generate-vector-corpus` command runs the pipeline locally (small-scale validation) and on the cluster (full scale). The bench's `_CELL_VECTOR` gains a scaled mode that reads the 1M seed + writes 1M rows per format, light vs heavy, recording the real numbers. - -**Tech Stack:** PySpark, the `*_gbx` vector readers/writers, pyogrio, shapely, the cluster bench harness (`databricks.labs.gbx.bench`). - ---- - -## Reframe (2026-06-12): pipeline-shaped bench + format-capacity sizing - -After Tasks 1–6 landed, the scenario and sizing were refined (supersedes the relevant parts of Tasks 6–8 below): - -**Two-leg pipeline (not a `read.format(x)→write.format(x)` round-trip):** -- **Reader = ingest.** Read *enough files with enough rows* and **write a Delta table** (the common "load vector data into Delta" pipeline). Timed = the full read→Delta materialization (forces a real read, not a lazy `count()` that games the parser). Light `*_gbx` vs heavy `*_ogr`. Source = a directory of N vector files (the replicated copies; shapefile/FileGDB copies are zipped `.shp.zip`/`.gdb.zip` so both tiers dir-read them). -- **Writer = export.** Start from a **Delta table** of vector data and **write a single file** in format x. Timed = table→single-file write (the writer's driver-side merge). Light only (heavy has no named-vector writers — documented gap). Multi-file output ("subdivide a table into a handful of files") is the **future API capability**, noted not benchmarked. - -**Capacity-driven sizing — push toward each format's ceiling; the limit must read as the *format's*, not GeoBrix's (never let a round number look like our cap):** - -| Format | Hard ceiling | Cause | ≈ box-polygon features at ceiling | -|---|---|---|---| -| Shapefile | **2 GB** per `.shp` (and per `.dbf`) | 32-bit offsets in 16-bit words (OGR caps 4 GB; 2 GB compat norm) | **~15.8 M** | -| GeoPackage | **~17.6 TB** (4 KB pages; ~281 TB max) | SQLite `page_size × max_page_count` | billions (no practical cap) | -| FileGDB | **2.1 B rows** / 1 TB per FC (→256 TB) | OBJECTID = signed int32 | ~2.1 billion | -| GeoJSON | **none** (RFC 7946) | text; bounded only by disk/parse memory | unbounded | - -- **Writer-export (capacity demo):** source = a single ~14 M-polygon Delta table (generated directly via `generate_polygon_seed`, not from the vector corpus) → one file per format. ~14 M ≈ 1.9 GB shapefile — near (safely under) its 2 GB ceiling, **no deliberate break**; the others carry it and are *documented* (cited) to go to billions/TB. Single-file is driver-bound → measure on the small validation; if too heavy (esp. GPKG/FileGDB), cap the writer leg at the largest size that completes cleanly and document the figure there. -- **Reader-ingest (scale demo):** read a directory of N copies (1 M each) → Delta; N chosen for "enough files with enough rows" (distributed, scales fine). Final N picked from the small-validation throughput (light GPKG read uses the read-only-Volume in-memory fallback and is the slow path — size so it completes). - -**Harness shape:** `run_format_read` → read dir + `write.format("delta").saveAsTable` (timed); `run_vector_write` → read a source Delta table + write one file (timed). `_CELL_VECTOR` materializes the ~14 M writer-source table once (untimed), runs the reader leg (copies dir → Delta, both tiers) and the writer leg (source table → single file, light). - -**Benchmarking.mdx (Task 8):** fill the two legs' numbers AND state each format's true ceiling (cited) so the capacity story is explicit. - ---- - -## File structure - -- **Create `python/geobrix/src/databricks/labs/gbx/bench/corpus_vector.py`** — the generator: `generate_polygon_seed`, `transcode_vector_seed`, `replicate_vector_seed`, `build_vector_corpus` (orchestrator). Pure functions over a SparkSession; no bench-harness coupling. -- **Create `python/geobrix/test/bench/test_corpus_vector.py`** — local small-scale tests (1000 rows, ×3 copies; FileGDB skips without osgeo). -- **Create `scripts/commands/gbx-bench-generate-vector-corpus.{md,sh}`** — CLI wrapper (params `--rows`, `--copies`, `--formats`, `--out`, `--log`), runs in the dev container / on the cluster. -- **Modify `python/geobrix/src/databricks/labs/gbx/ds/vector.py`** — `OgrGbxReader` gains directory enumeration (one partition per vector file in a dir) so the bench can read the ×100 corpus and users can read a folder of files. -- **Modify `python/geobrix/src/databricks/labs/gbx/bench/cluster.py`** (`_CELL_VECTOR`) + **`bench/readers.py`** — a scaled-corpus path: read the 1M seed + the ×N directory, write 1M rows, per format, light vs heavy. -- **Modify `docs/docs/api/benchmarking.mdx`** + the 15 reader/writer page `Benchmark & tradeoff` callouts — fill the real numbers. - -The corpus lives at `{CORPUS}/vector-scale//seed.` (the 1M seed) and `{CORPUS}/vector-scale//copies/copy_.` (the ×N replicas). ``: `.geojson`, `.shp`, `.gpkg`, `.gdb`. - ---- - -## Task 1: Polygon seed generator - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/bench/corpus_vector.py` -- Test: `python/geobrix/test/bench/test_corpus_vector.py` - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/bench/test_corpus_vector.py -import logging - -import pytest -from shapely import from_wkb - - -@pytest.fixture(scope="module") -def spark(): - logging.getLogger("py4j").setLevel(logging.ERROR) - from pyspark.sql import SparkSession - - s = ( - SparkSession.builder.master("local[2]") - .appName("gbx-corpus-vector") - .config("spark.sql.shuffle.partitions", "2") - .getOrCreate() - ) - yield s - - -def test_generate_polygon_seed(spark): - from databricks.labs.gbx.bench.corpus_vector import generate_polygon_seed - - df = generate_polygon_seed(spark, 200, srid="4326") - assert [f.name for f in df.schema.fields] == [ - "geom_0", - "geom_0_srid", - "geom_0_srid_proj", - "id", - "name", - ] - assert df.count() == 200 - row = df.orderBy("id").first() - assert row["geom_0_srid"] == "4326" - g = from_wkb(bytes(row["geom_0"])) - assert g.geom_type == "Polygon" - assert -180.0 <= g.bounds[0] <= 180.0 and -90.0 <= g.bounds[1] <= 90.0 -``` - -- [ ] **Step 2: Run it to verify it fails** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/bench/test_corpus_vector.py::test_generate_polygon_seed -v` -Expected: FAIL — `ModuleNotFoundError`/`ImportError` (corpus_vector / generate_polygon_seed not defined). - -- [ ] **Step 3: Implement `generate_polygon_seed`** - -```python -# python/geobrix/src/databricks/labs/gbx/bench/corpus_vector.py -"""Scaled vector benchmark corpus generator. Mints a 1M-polygon seed in the light -vector-writer schema, transcodes it to each format via the *_gbx writers, and -replicates each seed into a per-format directory on the bench Volume. Runs locally -(small scale) and on the bench cluster (full scale). FileGDB writing needs the -heavyweight GDAL natives (native osgeo) -- cluster only.""" - -from __future__ import annotations - -import os -import shutil -from typing import List - - -def generate_polygon_seed(spark, n_rows: int, srid: str = "4326"): - """A DataFrame of ``n_rows`` synthetic polygons in the light vector-writer schema - (geom_0 WKB, geom_0_srid, geom_0_srid_proj, id, name). Polygons are small axis- - aligned boxes at deterministic pseudo-random lon/lat from the row id.""" - from pyspark.sql import functions as F - from pyspark.sql.types import BinaryType - - @F.udf(BinaryType()) - def _poly(i): - from shapely import box, to_wkb - - lon = (int(i) * 73 % 35900) / 100.0 - 179.0 - lat = (int(i) * 37 % 17800) / 100.0 - 89.0 - d = 0.01 - return bytes(to_wkb(box(lon, lat, lon + d, lat + d))) - - return spark.range(n_rows).select( - _poly(F.col("id")).alias("geom_0"), - F.lit(srid).alias("geom_0_srid"), - F.lit("").alias("geom_0_srid_proj"), - F.col("id").cast("int").alias("id"), - F.concat(F.lit("feat_"), F.col("id").cast("string")).alias("name"), - ) -``` - -- [ ] **Step 4: Run it to verify it passes** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/bench/test_corpus_vector.py::test_generate_polygon_seed -v` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/bench/corpus_vector.py python/geobrix/test/bench/test_corpus_vector.py -git commit -m "feat(bench): polygon seed generator for the scaled vector corpus - -Co-authored-by: Isaac" -``` - ---- - -## Task 2: Transcode the seed to each vector format - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/corpus_vector.py` -- Test: `python/geobrix/test/bench/test_corpus_vector.py` - -- [ ] **Step 1: Write the failing test** - -```python -def test_transcode_vector_seed(spark, tmp_path): - from databricks.labs.gbx.bench.corpus_vector import ( - generate_polygon_seed, - transcode_vector_seed, - ) - from databricks.labs.gbx.ds.register import register - - register(spark) - seed = generate_polygon_seed(spark, 100) - # file_gdb needs native osgeo (heavy natives) -> exclude locally - fmts = ["geojson_gbx", "shapefile_gbx", "gpkg_gbx"] - out = transcode_vector_seed(spark, seed, fmts, str(tmp_path / "vec")) - for fmt in fmts: - assert fmt in out - back = spark.read.format(fmt).load(out[fmt]) - assert back.count() == 100 -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/bench/test_corpus_vector.py::test_transcode_vector_seed -v` -Expected: FAIL — `transcode_vector_seed` not defined. - -- [ ] **Step 3: Implement `transcode_vector_seed`** - -Append to `corpus_vector.py`: - -```python -_EXT = { - "geojson_gbx": "geojson", - "shapefile_gbx": "shp", - "gpkg_gbx": "gpkg", - "file_gdb_gbx": "gdb", - "vector_gbx": "geojson", -} - - -def transcode_vector_seed(spark, seed_df, formats: List[str], out_base: str) -> dict: - """Write the seed DataFrame to each format's seed file via the *_gbx writers. - Returns {fmt: seed_path}. The seed is cached so each write reuses it. FileGDB - requires the native osgeo (heavyweight GDAL natives).""" - seed_df = seed_df.cache() - seed_df.count() # materialize the cache - out: dict = {} - for fmt in formats: - ext = _EXT.get(fmt, "out") - path = f"{out_base}/{fmt}/seed.{ext}" - writer = seed_df.coalesce(1).write.format(fmt).mode("overwrite") - if fmt in ("vector_gbx", "ogr_gbx"): - writer = writer.option("driverName", "GeoJSON") - writer.save(path) - out[fmt] = path - return out -``` - -- [ ] **Step 4: Run to verify it passes** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/bench/test_corpus_vector.py::test_transcode_vector_seed -v` -Expected: PASS (geojson/shapefile/gpkg round-trip). - -- [ ] **Step 5: Commit** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/bench/corpus_vector.py python/geobrix/test/bench/test_corpus_vector.py -git commit -m "feat(bench): transcode the vector seed to each *_gbx format - -Co-authored-by: Isaac" -``` - ---- - -## Task 3: Replicate each seed ×N - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/corpus_vector.py` -- Test: `python/geobrix/test/bench/test_corpus_vector.py` - -- [ ] **Step 1: Write the failing test** - -```python -def test_replicate_vector_seed(spark, tmp_path): - import os - - from databricks.labs.gbx.bench.corpus_vector import replicate_vector_seed - - # a fake single-file seed - seed = str(tmp_path / "seed.geojson") - with open(seed, "w") as fh: - fh.write('{"type":"FeatureCollection","features":[]}') - copies_dir = str(tmp_path / "copies") - paths = replicate_vector_seed(seed, 5, copies_dir) - assert len(paths) == 5 - assert all(os.path.exists(p) for p in paths) - assert sorted(os.listdir(copies_dir)) == [f"copy_{i}.geojson" for i in range(5)] -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/bench/test_corpus_vector.py::test_replicate_vector_seed -v` -Expected: FAIL — `replicate_vector_seed` not defined. - -- [ ] **Step 3: Implement `replicate_vector_seed`** - -Append to `corpus_vector.py`: - -```python -def replicate_vector_seed(seed_path: str, n_copies: int, copies_dir: str) -> List[str]: - """Copy a per-format seed (a file, a `.shp` + sidecars, or a `.gdb` dir) ``n_copies`` - times into ``copies_dir`` as ``copy_.``. Sequential copies (FUSE-safe). - Returns the copy paths.""" - os.makedirs(copies_dir, exist_ok=True) - base = os.path.basename(seed_path.rstrip("/")) - stem, _, ext = base.partition(".") - paths: List[str] = [] - for i in range(n_copies): - dst = os.path.join(copies_dir, f"copy_{i}.{ext}" if ext else f"copy_{i}") - if os.path.isdir(seed_path): # FileGDB .gdb directory - shutil.copytree(seed_path, dst, dirs_exist_ok=True) - else: - shutil.copy(seed_path, dst) - # Shapefile sidecars (.shx/.dbf/.prj) share the stem -- copy them too. - src_dir = os.path.dirname(seed_path) or "." - src_stem = base.split(".")[0] - for sib in os.listdir(src_dir): - if sib.startswith(src_stem + ".") and sib != base: - sib_ext = sib[len(src_stem) + 1 :] - shutil.copy( - os.path.join(src_dir, sib), - os.path.join(copies_dir, f"copy_{i}.{sib_ext}"), - ) - paths.append(dst) - return paths -``` - -- [ ] **Step 4: Run to verify it passes** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/bench/test_corpus_vector.py::test_replicate_vector_seed -v` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/bench/corpus_vector.py python/geobrix/test/bench/test_corpus_vector.py -git commit -m "feat(bench): replicate per-format vector seeds (incl. sidecars/.gdb dir) - -Co-authored-by: Isaac" -``` - ---- - -## Task 4: Orchestrator `build_vector_corpus` + CLI command - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/corpus_vector.py` -- Create: `scripts/commands/gbx-bench-generate-vector-corpus.md`, `scripts/commands/gbx-bench-generate-vector-corpus.sh` -- Test: `python/geobrix/test/bench/test_corpus_vector.py` - -- [ ] **Step 1: Write the failing test** - -```python -def test_build_vector_corpus(spark, tmp_path): - import os - - from databricks.labs.gbx.bench.corpus_vector import build_vector_corpus - from databricks.labs.gbx.ds.register import register - - register(spark) - out = build_vector_corpus( - spark, rows=50, copies=3, - formats=["geojson_gbx", "gpkg_gbx"], out_base=str(tmp_path / "vc"), - ) - for fmt in ("geojson_gbx", "gpkg_gbx"): - assert os.path.exists(out[fmt]["seed"]) - assert len(out[fmt]["copies"]) == 3 - assert spark.read.format(fmt).load(out[fmt]["seed"]).count() == 50 -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/bench/test_corpus_vector.py::test_build_vector_corpus -v` -Expected: FAIL — `build_vector_corpus` not defined. - -- [ ] **Step 3: Implement `build_vector_corpus`** - -Append to `corpus_vector.py`: - -```python -def build_vector_corpus( - spark, rows: int, copies: int, formats: List[str], out_base: str, srid: str = "4326" -) -> dict: - """Full pipeline: generate the polygon seed -> transcode to each format -> - replicate ×copies. Returns {fmt: {"seed": path, "copies": [paths]}}.""" - from databricks.labs.gbx.ds.register import register - - register(spark) - seed_df = generate_polygon_seed(spark, rows, srid=srid) - seeds = transcode_vector_seed(spark, seed_df, formats, out_base) - result: dict = {} - for fmt, seed_path in seeds.items(): - copies_dir = f"{out_base}/{fmt}/copies" - result[fmt] = { - "seed": seed_path, - "copies": replicate_vector_seed(seed_path, copies, copies_dir), - } - return result -``` - -- [ ] **Step 4: Run to verify it passes** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/bench/test_corpus_vector.py -v` -Expected: all PASS. - -- [ ] **Step 5: Create the CLI command** - -`scripts/commands/gbx-bench-generate-vector-corpus.md`: title, description, usage `bash scripts/commands/gbx-bench-generate-vector-corpus.sh [OPTIONS]`, options (`--rows` default `1000000`, `--copies` default `100`, `--formats` default `geojson_gbx,shapefile_gbx,gpkg_gbx,file_gdb_gbx`, `--out` default `/Volumes/.../bench-corpus/vector-scale`, `--log`, `--help`), 2 examples (small local, full cluster). Mirror `scripts/commands/gbx-data-generate-vector-corpus.md`. - -`scripts/commands/gbx-bench-generate-vector-corpus.sh`: source `common.sh`; parse the options; run in the dev container (or note cluster execution); invoke a small inline Python that calls `build_vector_corpus` with a `SparkSession.builder.getOrCreate()`. Mirror the structure of `scripts/commands/gbx-data-generate-vector-corpus.sh` (which already runs a writer-backed generator in the container). `chmod +x` it. - -- [ ] **Step 6: Smoke-test** - -```bash -bash scripts/commands/gbx-bench-generate-vector-corpus.sh --help # prints usage, exit 0 -# small local run (geojson/gpkg only — no osgeo locally): -bash scripts/commands/gbx-bench-generate-vector-corpus.sh --rows 500 --copies 2 --formats geojson_gbx,gpkg_gbx --out /tmp/vc_smoke -``` -Expected: the `--help` exits 0; the small run reports the seeds + copies created. - -- [ ] **Step 7: Commit** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/bench/corpus_vector.py python/geobrix/test/bench/test_corpus_vector.py scripts/commands/gbx-bench-generate-vector-corpus.md scripts/commands/gbx-bench-generate-vector-corpus.sh -git commit -m "feat(bench): gbx:bench:generate-vector-corpus (seed->transcode->replicate) - -Co-authored-by: Isaac" -``` - ---- - -## Task 5: Light vector reader — directory enumeration - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (`OgrGbxReader.partitions`/`schema`) -- Test: `python/geobrix/test/ds/test_vector_reader.py` - -So the bench (and users) can read the ×N corpus folder: when `path` is a directory, enumerate the vector files in it (one `_ChunkPartition` per file × its feature chunks), and infer the schema from the first file. - -- [ ] **Step 1: Write the failing test** - -```python -def test_ogr_gbx_reads_directory(spark, tmp_path): - register(spark) - import os - d = os.path.join(str(tmp_path), "many") - os.makedirs(d) - for k in range(3): - with open(os.path.join(d, f"p{k}.geojson"), "w") as f: - json.dump(_GJ, f) # _GJ is the 2-feature FeatureCollection in this file - df = spark.read.format("geojson_gbx").load(d) - assert df.count() == 6 # 3 files x 2 features -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_reader.py::test_ogr_gbx_reads_directory -v` -Expected: FAIL — the reader treats the dir as one path and errors or returns 0. - -- [ ] **Step 3: Implement directory enumeration** - -In `OgrGbxReader`, add a helper that lists the member files when `self.path` is a directory (matching the format's extensions; for shapefile use `.shp`/`.shz`/`.zip`, geojson `.geojson`/`.json`/`.geojsonl`, gpkg `.gpkg`, filegdb `.gdb` dirs), and make `schema()` read the first member and `partitions()` emit chunk partitions per member. Single-file paths keep current behavior. Show the full method bodies: - -```python - _EXT_FOR_DRIVER = { - "GeoJSON": (".geojson", ".json"), - "GeoJSONSeq": (".geojsonl", ".geojsons"), - "ESRI Shapefile": (".shp", ".shz", ".zip"), - "GPKG": (".gpkg",), - "OpenFileGDB": (".gdb",), - } - - def _members(self) -> List[str]: - """If self.path is a directory of vector files, the member paths; else [self.path].""" - if not os.path.isdir(self.path) or self.path.lower().endswith(".gdb"): - return [self.path] - exts = self._EXT_FOR_DRIVER.get(self.driver) or () - names = sorted(os.listdir(self.path)) - members = [ - os.path.join(self.path, n) - for n in names - if (exts and n.lower().endswith(exts)) or n.lower().endswith(".gdb") - ] - return members or [self.path] - - def schema(self) -> StructType: - first = self._members()[0] - return _vector_schema(self._info_for(first), self.as_wkb) -``` - -Refactor `_info()` into `_info_for(path)` (the current `_info` body parameterized by path, keeping the read-only in-memory fallback), and make `partitions()` loop over `self._members()`, emitting `_ChunkPartition(member, ...)` chunks for each. (`read()` already takes the partition's `path`, so it is unchanged.) - -- [ ] **Step 4: Run to verify it passes** - -Run: `cd python/geobrix && ../../.venv-pyrx/bin/python -m pytest test/ds/test_vector_reader.py -v` -Expected: all PASS (existing single-file tests + the new directory test). - -- [ ] **Step 5: Commit** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_vector_reader.py -git commit -m "feat(ds): light vector reader enumerates a directory of files - -Co-authored-by: Isaac" -``` - ---- - -## Task 6: Scaled-corpus mode in the vector bench - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/cluster.py` (`_CELL_VECTOR` + a `VECTOR_SCALE` preamble flag), `notebooks/tests/push_and_run_bench_on_cluster.py` (parse `--vector-scale`) -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/readers.py` (no signature change expected; `run_format_read`/`run_vector_write` already take a path) - -- [ ] **Step 1: Add the `--vector-scale` flag (launcher)** - -In `push_and_run_bench_on_cluster.py`, parse `vector_scale = "--vector-scale" in sys.argv` near the other flags, pass it through `build_bench_notebook(cfg)` (add `vector_scale` to `cfg`), and into the PREAMBLE as `VECTOR_SCALE = {vector_scale!r}`. - -- [ ] **Step 2: Branch `_CELL_VECTOR` on `VECTOR_SCALE`** - -When `VECTOR_SCALE` is true, point the cases at the scaled corpus and read the **copies directory** (the reader now enumerates it) for the reader leg, and write the **seed read-back DataFrame** for the writer leg. Concretely, the scaled cases use `f"{CORPUS}/vector-scale/{fmt}/copies"` for the read path and `f"{CORPUS}/vector-scale/{fmt}/seed."` for the writer's source. Keep the existing tiny-corpus cases for the default (non-scale) run. Show the scaled `_vcases` block and that the reader path is the `copies` dir while the writer reads the seed and writes it back. Heavy geojson keeps `multi=false`. - -- [ ] **Step 3: Smoke-test the notebook builder locally** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix && .venv-pyrx/bin/python -c " -import sys; sys.path.insert(0,'python/geobrix/src') -from databricks.labs.gbx.bench.cluster import build_bench_notebook -nb=build_bench_notebook({'corpus':'/Volumes/x','out_dir':'/Volumes/x/o','table':'t','run_id':'r','functions':'','set':'core','modes':'spark-path','row_counts':'1000','warmup':1,'measured':1,'spark_warmup':1,'spark_measured':1,'partition_size':0,'truncate':False,'truncate_all':False,'resume':False,'fix_errors':True,'redo_functions':'','lightweight':True,'heavyweight':True,'explain_only':False,'benchmark_readers':False,'readers_only':False,'benchmark_pmtiles':False,'pmtiles_only':False,'benchmark_vector':True,'vector_only':True,'vector_scale':True,'wheel':'/Volumes/x/w.whl'}) -print('cells:', len(nb['cells'])) -print('VECTOR_SCALE' in str(nb)) -" -``` -Expected: prints a cell count and `True` (the flag threads through). Adjust the cfg keys to match the real `build_bench_notebook` signature if it differs (read it first). - -- [ ] **Step 4: Commit** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/bench/cluster.py notebooks/tests/push_and_run_bench_on_cluster.py -git commit -m "feat(bench): --vector-scale mode reads the 1M-seed corpus + copies dir - -Co-authored-by: Isaac" -``` - ---- - -## Task 7: Generate the corpus + run the scaled bench (cluster, operational) - -**Files:** none (operational). Requires the heavy GDAL natives on the cluster (for the FileGDB seed) and a rebuilt+staged wheel. - -- [ ] **Step 1: Rebuild + stage the wheel** (it carries `corpus_vector.py` + the reader/bench changes): -`GBX_BUNDLE_SKIP_JAR_UPLOAD=1 bash scripts/commands/gbx-data-push-wheel.sh` → "Done: …/geobrix-0.4.0-py3-none-any.whl". - -- [ ] **Step 2: Generate the corpus on the cluster** (1M × 100, polygons, all 4 formats) via a one-off notebook/job that calls `build_vector_corpus(spark, rows=1_000_000, copies=100, formats=[…4…], out_base=f"{CORPUS}/vector-scale")`. (FileGDB seed needs the natives — confirm `osgeo` imports on the cluster, as proven earlier.) Verify the seeds + 100 copies exist per format on the Volume. - -- [ ] **Step 3: Run the scaled vector bench**: -`export GBX_BUNDLE_WHEEL_VOLUME_PATH=…; bash scripts/commands/gbx-bench-cluster.sh --vector-only --vector-scale --row-counts 1000 --log vector-scale.log` -Expected: `run_id=cluster-vector` rows for each format's reader (light vs heavy, ~1M-scale) + writer (light, 1M rows), all `status=ok`. - -- [ ] **Step 4: Capture the numbers** — query `geospatial_docs.geobrix.bench_results` for `run_id='cluster-vector'`, record per-format light/heavy `iter_median_s` + `throughput_rows_s`. - ---- - -## Task 8: Fill the benchmark numbers — CENTRAL Benchmarking page only - -**Decision (user, 2026-06-12):** benchmarks are **consolidated to the single -[Benchmarking](../api/benchmarking) page**, NOT duplicated per page. The per-format reader/ -writer pages keep ONLY their prominent `Benchmark & tradeoff` note + link (already in place -— do NOT add per-page numbers). So this task touches `benchmarking.mdx` alone. - -**Files:** -- Modify: `docs/docs/api/benchmarking.mdx` (vector reader + writer results tables — replace the `—` placeholders with the Task-7 numbers). - -- [ ] **Step 1** — put the Task-7 per-format numbers into `benchmarking.mdx`'s vector reader + writer tables: light vs heavy median + throughput where both tiers have an implementation (readers: all 4 formats; writers: light `*_gbx` for all, heavy only for PMTiles/raster — vector heavy has no writer; FileGDB writer is the native-osgeo hybrid). Method line: 1M-polygon seed, ×100 copies, cluster, median of measured iters. -- [ ] **Step 2** — `cd docs && npm run build` → SUCCESS; `grep -rn -iE "wave [0-9]+" docs/docs/` empty. Confirm the 15 per-page `Benchmark & tradeoff` callouts are UNCHANGED (note + link only). -- [ ] **Step 3: Commit** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -chmod -R u+rwX .git/objects -git add docs/docs/api/benchmarking.mdx -git commit -m "docs(bench): fill scaled vector reader/writer numbers on the Benchmarking page - -Co-authored-by: Isaac" -``` - ---- - -## Self-Review - -**Spec coverage:** seed generation (T1), transcode to geojson/shapefile/gpkg/filegdb via writers (T2), replicate ×100 (T3), orchestrator + CLI runnable locally & on cluster (T4), reader directory support to read the copies (T5), bench wiring scaled mode (T6), cluster generate+run at 1M×100 (T7), fill per-page + benchmarking.mdx numbers (T8). FileGDB-needs-natives is called out in T2/T7. Local small-scale validation in T1–T4 (FileGDB excluded locally — no osgeo). ✓ - -**Placeholder scan:** no TBD/TODO. T6/T8 reference reading the real `build_bench_notebook` signature / inserting real numbers from T7 (numbers don't exist until the run) — consistent with how the reader numbers were handled; the code steps (T1–T5) carry full implementations. - -**Type consistency:** `generate_polygon_seed`→`transcode_vector_seed`→`replicate_vector_seed`→`build_vector_corpus` signatures align; seed schema `(geom_0, geom_0_srid, geom_0_srid_proj, id, name)` matches the writer's `_writer_col_roles` contract (geom + `*_srid`); `_EXT` map consistent across transcode/replicate; the reader directory enumeration reuses the existing `_ChunkPartition`/`read()`. - ---- - -## Execution Handoff - -Recommended: subagent-driven-development (fresh subagent per task, two-stage review). Tasks 1–6 are local/code (TDD, FileGDB-write steps skip locally without osgeo); Task 7 is the cluster generate+run; Task 8 fills the numbers from Task 7. diff --git a/docs/superpowers/plans/2026-06-13-h3-tessellate-modes.md b/docs/superpowers/plans/2026-06-13-h3-tessellate-modes.md deleted file mode 100644 index 675a21da0..000000000 --- a/docs/superpowers/plans/2026-06-13-h3-tessellate-modes.md +++ /dev/null @@ -1,289 +0,0 @@ -# rst_h3_tessellate covering/centroid Modes — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax. - -**Goal:** Add a `mode` parameter (`"covering"` default / `"centroid"`) to `rst_h3_tessellate` in BOTH tiers, aligning light and heavy on the same per-mode semantics: `covering` = the true overlapping cell set, each clipped to its hexagon (all-touched); `centroid` = pixel-centroid single-assignment partition. - -**Architecture:** Light (`pyrx`) uses h3-py 4.4.2 `polygon_to_cells_experimental(contain='overlap')` for covering and per-pixel `latlng_to_cell` for centroid. Heavy (Scala, H3-Java **3.7.0**, no v4 covering primitive) hand-rolls the covering set via a **JTS hexagon∩bbox overlap test** (replacing the nodata keep-test, which is what over-includes a disjoint fringe today) and per-pixel `pointToCellID` for centroid. Parity is enforced by per-mode light-vs-heavy tests, not an identical API call. Backward-compatible: SQL arity 2 (default `covering`) and 3. - -**Tech Stack:** Python 3.12 / PySpark UDTF / h3-py 4.4.2 / shapely / rasterio (light); Scala 2.13 / Spark 4 / H3-Java 3.7.0 / JTS / GDAL OGR (heavy), in the `geobrix-dev` Docker container. - -**Spec:** `docs/superpowers/specs/2026-06-13-h3-raster-tessellation-modes-design.md` (§6–10). - ---- - -## File Structure - -**Light (modify):** -- `python/geobrix/src/databricks/labs/gbx/pyrx/core/tessellate.py` — the H3 cell-selection + chip core (`iter_tessellate_h3` and helpers). Add `mode`; covering→`contain='overlap'`; centroid→per-pixel assignment; fix `all_touched` asymmetry. -- `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` — `_RstH3TessellateUDTF.eval` (+`mode`), the `rst_h3_tessellate` wrapper (`mode: ColLike = "covering"`). -- Tests: `python/geobrix/test/pyrx/` (core + UDTF), and a JAR-gated cross-tier parity test under `python/geobrix/test/pyvx/` or `test/pyrx/`. - -**Heavy (modify):** -- `src/main/scala/com/databricks/labs/gbx/rasterx/.../RST_H3_Tessellate.scala` — add `modeExpr`; `FunctionBuilder` arity 2+3. -- `.../RasterTessellate.scala` (`tessellateH3Iter`) — covering (JTS overlap keep-test) + centroid (per-pixel) paths. -- `.../H3.scala` — `getBufferRadius` default arm; any covering helper. -- `.../functions.scala` — Scala API overloads with `mode`. -- `python/geobrix/src/databricks/labs/gbx/rasterx/functions.py` — heavy binding `mode="covering"`. -- Tests: `src/test/scala/com/databricks/labs/gbx/rasterx/.../` (RST_H3_Tessellate / RasterTessellate). - -**Docs (create/modify):** -- `docs/docs/` — new H3 explainer page + sidebar wiring. -- `function-info.json` usage example for `gbx_rst_h3_tessellate` (mode arg). - -> Implementer note: exact Scala paths/line numbers — locate via `grep -rn "RST_H3_Tessellate\|tessellateH3Iter\|getBufferRadius" src/main/scala`. The behavioral summary in the spec's §3 cites the current logic. - ---- - -## Task 1: Light — `centroid` mode in the core (pixel-centroid partition) - -**Files:** Modify `pyrx/core/tessellate.py`; Test `python/geobrix/test/pyrx/test_core_tessellate_modes.py` (create). - -- [ ] **Step 1: Failing test** (Spark-free core, real raster): - -```python -# python/geobrix/test/pyrx/test_core_tessellate_modes.py -import numpy as np -from rasterio.io import MemoryFile -from databricks.labs.gbx.pyrx.core import tessellate as T -from databricks.labs.gbx.pyrx import _serde - - -def _tile_4326(size=64, res_deg=0.01, origin=(-0.1, 51.5)): - data = np.arange(size * size, dtype="float32").reshape(size, size) - prof = dict(driver="GTiff", height=size, width=size, count=1, dtype="float32", - crs="EPSG:4326", - transform=__import__("rasterio").transform.from_origin(origin[0], origin[1], res_deg, res_deg)) - with MemoryFile() as mf: - with mf.open(**prof) as dst: - dst.write(data, 1) - return mf.read() - - -def test_centroid_mode_partitions_pixels(): - """centroid: every valid pixel assigned to exactly one cell; union == all pixels; no overlap.""" - tile = _tile_4326() - cells = list(T.iter_tessellate_h3(_serde.open_tile(tile), resolution=9, mode="centroid")) - # collect the set of valid pixels covered by each cell's chip - seen = [] - for cell in cells: - with _serde.open_tile(cell["raster"]) as ds: - arr = ds.read(1, masked=True) - seen.append(int((~arr.mask).sum())) - total_valid = 64 * 64 - assert sum(seen) == total_valid, "centroid chips must partition all pixels exactly once" -``` - -- [ ] **Step 2: Run, verify FAIL** — `PYSPARK_PYTHON=.venv-pyrx/bin/python .venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/test_core_tessellate_modes.py::test_centroid_mode_partitions_pixels -v` → FAIL (`iter_tessellate_h3` has no `mode`). - -- [ ] **Step 3: Implement centroid path** — read the current `iter_tessellate_h3` in `tessellate.py`; add a `mode: str = "covering"` param with validation (`{"covering","centroid"}` → `ValueError` listing valid values). For `mode == "centroid"`: reproject the raster to 4326 (or assert 4326), then for each valid pixel compute its lon/lat centroid → `h3.latlng_to_cell(lat, lon, resolution)`; group pixels by cell; for each cell emit a chip = a raster with only that cell's pixels (others nodata) — reuse the existing tile-build/serde helpers. Do NOT build the covering set in this path (cells emerge from pixels). Keep the existing covering behavior under `mode == "covering"` for now (Task 2 replaces it). - -- [ ] **Step 4: Run, verify PASS** (same command) → PASS. - -- [ ] **Step 5: Commit** -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyrx/core/tessellate.py python/geobrix/test/pyrx/test_core_tessellate_modes.py -git commit -m "feat(pyrx): h3 tessellate centroid mode (pixel-centroid partition)" -``` - -## Task 2: Light — `covering` mode via `contain='overlap'` (replace ring+prune) - -**Files:** Modify `pyrx/core/tessellate.py`; Test same file as Task 1. - -- [ ] **Step 1: Failing test** — covering = the true overlapping set + all-touched chips: - -```python -def test_covering_mode_is_overlap_set(): - import h3 - from shapely.geometry import box - tile = _tile_4326() - with _serde.open_tile(tile) as ds: - cells = {c["index"] if "index" in c else c["cellid"] for c in - __import__("databricks.labs.gbx.pyrx.core.tessellate", fromlist=["iter_tessellate_h3"]).iter_tessellate_h3(ds, resolution=9, mode="covering")} - # oracle: h3-py overlap containment over the same 4326 bbox - shp = h3.geo_to_h3shape(box(-0.1, 51.5 - 0.64, -0.1 + 0.64, 51.5).__geo_interface__) - oracle = set(h3.polygon_to_cells_experimental(shp, 9, contain="overlap")) - assert cells == oracle -``` - -(Adjust the chip's cell-id field name to match the tile schema; adjust the bbox to the test raster's actual 4326 extent.) - -- [ ] **Step 2: Run, verify FAIL** (current covering = seed+grid_disk+prune, not exactly the overlap set). - -- [ ] **Step 3: Implement** — in the `mode == "covering"` path, replace the `h3shape_to_cells` seed + `grid_disk(1)` ring + prune with a single `h3.polygon_to_cells_experimental(bbox_shape_4326, resolution, contain="overlap")`. Keep the hexagon clip → chip, and make the clip use **`all_touched=True`** (fixing the prune-vs-clip asymmetry: the old prune used True, the clip used False — both must be True now). - -- [ ] **Step 4: Run, verify PASS.** - -- [ ] **Step 5: Commit** -```bash -git add python/geobrix/src/databricks/labs/gbx/pyrx/core/tessellate.py python/geobrix/test/pyrx/test_core_tessellate_modes.py -git commit -m "feat(pyrx): h3 tessellate covering mode via contain=overlap (+all_touched fix)" -``` - -## Task 3: Light — wire `mode` through the UDTF + wrapper + SQL - -**Files:** Modify `pyrx/functions.py`; Test `python/geobrix/test/pyrx/test_functions_spark.py` (extend). - -- [ ] **Step 1: Failing test** — LATERAL call with + without mode: - -```python -def test_h3_tessellate_mode_sql(spark): - from databricks.labs.gbx.pyrx import functions as rx - rx.register(spark) - # build a tiny 4326 tile view (reuse existing tile-fixture helper in this test module) - df = _h3_tile_df(spark) # existing helper that yields a (tile) column - df.createOrReplaceTempView("ras") - n_default = spark.sql("SELECT t.* FROM ras, LATERAL gbx_rst_h3_tessellate(tile, 9) t").count() - n_cover = spark.sql("SELECT t.* FROM ras, LATERAL gbx_rst_h3_tessellate(tile, 9, 'covering') t").count() - n_centroid = spark.sql("SELECT t.* FROM ras, LATERAL gbx_rst_h3_tessellate(tile, 9, 'centroid') t").count() - assert n_default == n_cover and n_cover > 0 and n_centroid > 0 - import pytest - with pytest.raises(Exception): - spark.sql("SELECT t.* FROM ras, LATERAL gbx_rst_h3_tessellate(tile, 9, 'bogus') t").count() -``` - -- [ ] **Step 2: Run, verify FAIL** (`eval` takes only `(tile, resolution)`). - -- [ ] **Step 3: Implement** — `_RstH3TessellateUDTF.eval(self, tile, resolution, mode=None)`: default `"covering"` when `mode is None`; validate `{"covering","centroid"}` (ValueError); pass to `iter_tessellate_h3(ds, resolution, mode=...)`. Update the `rst_h3_tessellate` wrapper signature to `(tile, resolution, mode: ColLike = "covering")` (keep the NotImplementedError-LATERAL-guidance body; mention `mode` in the docstring). The UDTF registration is unchanged (positional args). - -- [ ] **Step 4: Run, verify PASS**; then the full pyrx suite + Serverless guard: -`PYSPARK_PYTHON=.venv-pyrx/bin/python PYSPARK_DRIVER_PYTHON=.venv-pyrx/bin/python .venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/ -v` (green; note skips). - -- [ ] **Step 5: Commit** -```bash -git add python/geobrix/src/databricks/labs/gbx/pyrx/functions.py python/geobrix/test/pyrx/test_functions_spark.py -git commit -m "feat(pyrx): rst_h3_tessellate mode param (covering default / centroid)" -``` - -## Task 4: Heavy — `covering` mode: JTS overlap keep-test (fix disjoint fringe) - -**Files (Docker):** Modify `RasterTessellate.scala`, `H3.scala`; Test `src/test/scala/.../RST_H3_TessellateTest.scala` (extend or create). - -- [ ] **Step 1: Failing Scala test** — covering produces no disjoint cells (every emitted cell's hexagon intersects the tile bbox): - -```scala -test("h3 tessellate covering emits only cells whose hexagon overlaps the tile") { - rasterx.functions.register(spark) - import rasterx.functions._ - val df = h3TileDf(spark) // small 4326 raster tile (reuse existing test fixture) - val cells = df.select(rst_h3_tessellate(col("tile"), lit(9))).collect() // default covering - // For each emitted cell, assert its H3 hexagon (JTS) intersects the tile bbox geometry. - assert(cells.nonEmpty) - assert(TessTestUtil.allHexagonsOverlapBbox(cells, df)) // helper: no disjoint cell -} -``` - -Add a `TessTestUtil.allHexagonsOverlapBbox` helper (build each cell's hexagon via `H3.cellIdToGeometry`, the tile bbox via `BoundingBox.bbox`, assert JTS `intersects`). - -- [ ] **Step 2: Run, verify FAIL** (current heavy over-includes the disjoint fringe): -`bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.*.RST_H3_TessellateTest' --log h3tess.log` → FAIL. - -- [ ] **Step 3: Implement** — in `RasterTessellate.tessellateH3Iter`, for the covering path keep the polyfill+buffer candidate generation but **replace the per-cell nodata keep-test (`RasterAccessors.isEmpty`) with a JTS overlap test**: keep the cell iff `cellHexagon.intersects(bboxGeom4326)` (the hexagon, not its bbox). Give `H3.getBufferRadius` a default match arm (return 0 or a safe default) so non-Polygon inputs don't `MatchError`. (The chip clip via `ClipToGeom` with `CUTLINE_ALL_TOUCHED=TRUE` is unchanged.) - -- [ ] **Step 4: Run, verify PASS** (covering test green; existing tessellate tests still green). - -- [ ] **Step 5: Commit** -```bash -chmod -R u+rwX .git/objects -git add src/main/scala/com/databricks/labs/gbx/rasterx/ src/test/scala/com/databricks/labs/gbx/rasterx/ -git commit -m "fix(rasterx): h3 tessellate covering uses JTS hexagon-overlap keep-test" -``` - -## Task 5: Heavy — `centroid` mode + `mode` param (arity 2+3) + bindings - -**Files (Docker):** `RST_H3_Tessellate.scala`, `RasterTessellate.scala`, `functions.scala`, `rasterx/functions.py`; Test `RST_H3_TessellateTest.scala`. - -- [ ] **Step 1: Failing Scala test** — centroid partitions pixels + mode arity: - -```scala -test("h3 tessellate centroid partitions pixels; mode arity 2 and 3 both work") { - rasterx.functions.register(spark) - import rasterx.functions._ - val df = h3TileDf(spark) - val cover = df.select(rst_h3_tessellate(col("tile"), lit(9), lit("covering"))).count() - val cent = df.select(rst_h3_tessellate(col("tile"), lit(9), lit("centroid"))).count() - val dflt = df.select(rst_h3_tessellate(col("tile"), lit(9))).count() // arity-2 == covering - assert(dflt == cover && cover > 0 && cent > 0) - assert(TessTestUtil.centroidPartitionsAllPixels(df, 9)) // helper: each valid pixel in exactly one chip -} -``` - -- [ ] **Step 2: Run, verify FAIL** (no `mode`; arity strict-2). - -- [ ] **Step 3: Implement** — add `modeExpr` to the `RST_H3_Tessellate` case class; `builder()` arity **2** (`Literal("covering")`) and **3** (`c(2)`), else `IllegalArgumentException` ("takes 2 or 3 arguments…"); validate the mode string (`require(Set("covering","centroid").contains(...))`). In `RasterTessellate.tessellateH3Iter`, add the centroid path: per valid pixel → `H3.pointToCellID(lon, lat, res)`; group pixels by cell; emit one chip per cell with only its pixels. Add `functions.scala` overloads (`rst_h3_tessellate(tile, res, mode: String)` + `(tile, Int res, mode: String = "covering")`, mirroring the existing `resolution: Int` overload). Update the heavy Python binding `rasterx/functions.py` `rst_h3_tessellate(tile, resolution, mode: ColLike = "covering")` (mirror `rst_resample`'s string-default handling). - -- [ ] **Step 4: Run, verify PASS** — `bash scripts/commands/gbx-test-scala.sh --suite '...RST_H3_TessellateTest' --log h3tess.log` green; `bash scripts/commands/gbx-lint-scalastyle.sh` 0 errors. - -- [ ] **Step 5: Commit** -```bash -git add src/main/scala/com/databricks/labs/gbx/ python/geobrix/src/databricks/labs/gbx/rasterx/functions.py src/test/scala/com/databricks/labs/gbx/rasterx/ -git commit -m "feat(rasterx): rst_h3_tessellate centroid mode + mode param (arity 2+3)" -``` - -## Task 6: Cross-tier parity tests (both modes, border tile) - -**Files:** `python/geobrix/test/pyvx/test_parity_h3_tessellate.py` (create) — JAR-gated like `test_parity_mvt.py`. - -- [ ] **Step 1: Write the parity test** — for a small 4326 raster containing a tile border, for EACH mode assert light and heavy produce the **same cell set** and matching per-cell pixel counts: - -```python -import os, pytest -mvt = None # not needed -pytestmark = pytest.mark.skipif(not os.environ.get("GBX_HEAVY_JAR"), - reason="needs heavyweight JAR; run in geobrix-dev Docker") - -@pytest.mark.parametrize("mode", ["covering", "centroid"]) -def test_light_vs_heavy_h3_tessellate(spark_with_jar, mode): - from databricks.labs.gbx.pyrx import functions as rx - from databricks.labs.gbx.rasterx import functions as hx - rx.register(spark_with_jar); hx.register(spark_with_jar) - df = _border_tile_df(spark_with_jar); df.createOrReplaceTempView("ras") - light = {(r["index"]) for r in spark_with_jar.sql( - f"SELECT t.index FROM ras, LATERAL gbx_rst_h3_tessellate(tile, 9, '{mode}') t").collect()} - heavy = {r[0] for r in df.select(hx.rst_h3_tessellate(__import__("pyspark.sql.functions", fromlist=['col']).col("tile"), 9, mode)).collect()} - # heavy returns a tile struct; extract its cell-id field. Adjust accessors to the actual schema. - assert light == heavy and len(light) > 0 -``` - -(Reuse `test_parity_mvt.py`'s `spark_with_jar` fixture pattern; adjust the cell-id extraction to the tile struct's field. For `centroid`, also assert the partition property holds in both tiers.) - -- [ ] **Step 2: Run in Docker (JAR staged)** — `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/test_parity_h3_tessellate.py --log h3-parity.log` → both modes PASS (rebuild the lib JAR first so it has Tasks 4–5). - -- [ ] **Step 3: Commit** -```bash -git add python/geobrix/test/pyvx/test_parity_h3_tessellate.py -git commit -m "test: light-vs-heavy rst_h3_tessellate parity (covering + centroid)" -``` - -## Task 7: Bench — per-mode h3_tessellate leg - -**Files:** Modify `python/geobrix/src/databricks/labs/gbx/bench/readers.py` (`run_fanout_udtf` / `_fanout_spec`) + `_CELL_FANOUT` in `cluster.py`. - -- [ ] **Step 1: Update the h3_tessellate fan-out leg** to pass `mode` (default `covering`) on both tiers' SQL and compare per-mode; the heavy generator call is `LATERAL VIEW gbx_rst_h3_tessellate(tile, res, 'covering') t AS tile` (no explode). Optionally add a `centroid` variant. -- [ ] **Step 2: Local smoke** (light path) — `run_fanout_udtf(..., api="lightweight", fn="rst_h3_tessellate")` returns ok rows for both modes. -- [ ] **Step 3: Commit** (`feat(bench): h3_tessellate fan-out leg is mode-aware`). (The cluster re-bench runs when a cluster is next up — coordinate with the operator; the parity is already proven by Task 6.) - -## Task 8: Docs — H3 explainer page + function-info - -**Files:** new `docs/docs/.../h3.mdx` (or per the existing IA), `docs/sidebars.js`, `function-info.json` (or the doc-test source that generates it). - -- [ ] **Step 1: Write the H3 explainer page** per spec §9: lineage (Mosaic → `h3_coverash3`/`h3_tessellateaswkb`); the two tessellation modes (`covering` full-coverage/shareable vs `centroid` pixel-centroid partition/de-duped) with a "when to use which" + a border-behavior visual; relationship to `rst_h3_rastertogrid*`; CRS expectations; cross-tier parity (with the H3-Java-3.7.0 mechanism note framed as a defensible divergence). Doc-voice rules (no marketing, no Mosaic-as-rationale, no "wave N"). Wire into `docs/sidebars.js`. -- [ ] **Step 2: Update the `gbx_rst_h3_tessellate` usage example** (function-info source / doc-test) to show the `mode` arg. -- [ ] **Step 3: Build + checks** — `cd docs && npm run build` SUCCESS; `grep -rn -iE "wave [0-9]+" docs/docs/` empty. -- [ ] **Step 4: Commit** (`docs: H3 tessellation explainer page + mode usage`). - ---- - -## Self-Review (against spec §6–9) - -- §6.1 mode param (string, default covering, arity 2+3, validation) → Tasks 3, 5. ✓ -- §6.2 covering (true overlapping set; light `contain='overlap'`, heavy JTS overlap test; all_touched chip) → Tasks 2, 4. ✓ -- §6.3 centroid (pixel-centroid partition, both tiers) → Tasks 1, 5. ✓ -- §6.4 CRS internal reproject → preserved (Tasks 1–2, 4–5 keep current reproject). ✓ -- §6.5 parity by definition + tests → Task 6. ✓ -- §7 impl scope (heavy + light + bindings + function-info) → Tasks 1–5, 8. ✓ -- §8 testing (per-mode parity, covering-no-disjoint, centroid-partition) → Tasks 4, 5, 6. ✓ -- §9 explainer page → Task 8. ✓ - -**Known soft spots (acceptable, test-gated):** the heavy Scala steps reference the current code by file/grep + give the target behavior + a Scala test as the gate (verbatim line numbers shift); the cell-id field accessor in the parity/heavy tests must be matched to the actual tile struct schema by the implementer. JAR rebuild precedes Task 6; the cluster re-bench (Task 7) is deferred to the next cluster session (parity already proven locally). diff --git a/docs/superpowers/plans/2026-06-13-pyvx-mvt-light-tier.md b/docs/superpowers/plans/2026-06-13-pyvx-mvt-light-tier.md deleted file mode 100644 index 934acd28e..000000000 --- a/docs/superpowers/plans/2026-06-13-pyvx-mvt-light-tier.md +++ /dev/null @@ -1,946 +0,0 @@ -# pyvx MVT Light Tier — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Ship the MVT slice of the pure-Python/PySpark light VectorX tier (`pyvx`) — `st_asmvt` (aggregator) + `st_asmvt_pyramid` (generator) — as a drop-in swap for the heavyweight `vectorx` MVT functions, and upgrade **both** tiers to encode MVT attributes with native protobuf value types. - -**Architecture:** A new `pyvx` package mirrors `pyrx`: pure-Python encode/tiling helpers (Spark-free, unit-tested) wrapped by Serverless-safe Spark wiring (`spark.udf.register` for the aggregator, `spark.udtf.register` for the pyramid — no `_jvm`/`conf.set`/`.rdd`). `mapbox-vector-tile` + `shapely` + `pyproj` do the encoding. The heavyweight Scala `MvtWriter` is upgraded from all-`OFTString` to native OGR field types so the tiers emit byte-equivalent typed tiles; parity is checked at the decoded-feature level. - -**Tech Stack:** Python 3.12, PySpark (Spark Connect / Serverless), `mapbox-vector-tile`, `shapely 2`, `pyproj`; Scala 2.13 / Spark 4 / OGR (heavy MVT). Tests: pytest (light), ScalaTest in the `geobrix-dev` Docker container (heavy), `gbx:*` commands. - -**Spec:** `docs/superpowers/specs/2026-06-13-pyvx-mvt-light-tier-design.md`. - ---- - -## File Structure - -**New — light package** (`python/geobrix/src/databricks/labs/gbx/pyvx/`): -- `__init__.py` — package marker + docstring. -- `_env.py` — assert `mapbox_vector_tile` / `shapely` importable (mirrors `pyrx/_env.py`). -- `_serde.py` — WKB↔shapely; `attrs` struct/Row → native-typed property dict; the `(z,x,y,mvt_bytes)` tile struct schema. -- `_mvt.py` — Spark-free encode helpers: `encode_layer(features, layer_name, extent)` and `pyramid_tiles(geom, attrs, min_z, max_z, layer_name, extent)` (generator of `(z,x,y,bytes)`). -- `functions.py` — `register(spark)`, `SQL_REGISTRY`, the `st_asmvt` grouped-agg UDF + Column wrapper, and the `st_asmvt_pyramid` UDTF + wrapper. Signatures mirror `vectorx/functions.py`. - -**New — light tests** (`python/geobrix/test/pyvx/`): -- `conftest.py`, `test_mvt_encode.py` (Spark-free), `test_asmvt.py`, `test_asmvt_pyramid.py`, `test_parity_mvt.py` (Docker/JAR integration). - -**Modified — heavy (Scala):** -- `src/main/scala/com/databricks/labs/gbx/vectorx/mvt/MvtWriter.scala` — native OGR field types. -- `src/main/scala/com/databricks/labs/gbx/vectorx/expressions/ST_AsMvt.scala` — ensure `encodeAttrs`/`decodeAttrs` preserve typed values (not stringified). -- `src/test/scala/com/databricks/labs/gbx/vectorx/expressions/ST_AsMvtTest.scala` — assert native-typed decoded values. - -**Modified — packaging / bench / docs:** -- `python/geobrix/pyproject.toml` — add `mapbox-vector-tile` to the `light` extra. -- `python/geobrix/test/pyrx/test_serverless_no_spark_config.py` — extend `_source_files()` to cover `pyvx`. -- `python/geobrix/src/databricks/labs/gbx/bench/readers.py`, `.../bench/cluster.py`, `notebooks/tests/push_and_run_bench_on_cluster.py` — MVT light-vs-heavy bench + `--mvt-only`. -- `docs/docs/` — pyvx MVT page + Benchmarking **Vector** tab. - ---- - -## Task 1: De-risk — verify Python UDTF on Serverless / Spark Connect - -**Files:** -- Create (scratch, not committed): `/tmp/udtf_probe.py` - -**Goal:** Confirm the generator can be a Python UDTF (approach 2B). If UDTFs don't register/run on the target (Serverless / Spark Connect), fall back to approach 2A (`pandas_udf(ArrayType(tile_struct))` + caller `explode`) for Task 5. This is a spike, not TDD. - -- [ ] **Step 1: Write a trivial UDTF probe** - -```python -# /tmp/udtf_probe.py -from pyspark.sql import SparkSession -from pyspark.sql.functions import udtf - -@udtf(returnType="z int, x int, y int") -class Fan: - def eval(self, n: int): - for i in range(n): - yield (0, i, i) - -spark = SparkSession.builder.getOrCreate() -spark.udtf.register("fan", Fan) -spark.sql("SELECT f.* FROM (SELECT 3 AS n) t, LATERAL fan(t.n) f").show() -print("UDTF_OK") -``` - -- [ ] **Step 2: Run it on the local venv AND record the Serverless/Connect answer** - -Run locally: `.venv-pyrx/bin/python /tmp/udtf_probe.py` — expect `UDTF_OK` and 3 rows. -Then confirm on the real target: run the same probe in a Serverless / Spark Connect notebook (or the bench cluster via a one-off). Record whether `spark.udtf.register` + `LATERAL` works over Connect. - -- [ ] **Step 3: Record the decision in the plan + spec** - -Append a one-line note to `docs/superpowers/specs/2026-06-13-pyvx-mvt-light-tier-design.md` under "Risks": either "UDTF verified on Serverless/Connect — Task 5 uses 2B" or "UDTF unsupported — Task 5 uses 2A fallback (pandas_udf(ArrayType)+explode)". All later steps that say "UDTF (2B)" switch to the array+explode form if 2A was chosen. - -- [ ] **Step 4: Commit the decision note** - -```bash -git add docs/superpowers/specs/2026-06-13-pyvx-mvt-light-tier-design.md -git commit -m "docs(spec): record pyvx pyramid generator approach (UDTF vs explode)" -``` - ---- - -## Task 2: Package skeleton + dependency + env guard - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyvx/__init__.py` -- Create: `python/geobrix/src/databricks/labs/gbx/pyvx/_env.py` -- Modify: `python/geobrix/pyproject.toml` (the `light = [...]` block) -- Test: `python/geobrix/test/pyvx/test_env.py` - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/pyvx/test_env.py -def test_pyvx_imports_and_env_ok(): - import databricks.labs.gbx.pyvx as pyvx # noqa: F401 - from databricks.labs.gbx.pyvx import _env - # Raises a clear ImportError if mapbox-vector-tile / shapely are missing. - _env.assert_mvt_available() -``` - -- [ ] **Step 2: Run it to verify it fails** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_env.py -v` -Expected: FAIL (`ModuleNotFoundError: databricks.labs.gbx.pyvx`). - -- [ ] **Step 3: Add the dependency** - -In `python/geobrix/pyproject.toml`, add to the `light = [...]` list (after `pyproj>=3.6`): - -```toml - "mapbox-vector-tile>=2.0,<3", -``` - -Then install into the venv: `.venv-pyrx/bin/pip install 'mapbox-vector-tile>=2.0,<3'`. - -- [ ] **Step 4: Create the package + env guard** - -```python -# python/geobrix/src/databricks/labs/gbx/pyvx/__init__.py -"""pyvx — pure-Python/PySpark light VectorX tier (Serverless-safe). - -Mirrors the heavyweight ``vectorx`` MVT functions (``gbx_st_*``) with no JVM, -no JAR, and no native GDAL. See databricks.labs.gbx.pyvx.functions. -""" -``` - -```python -# python/geobrix/src/databricks/labs/gbx/pyvx/_env.py -"""Environment checks for the pyvx light tier.""" - - -def assert_mvt_available() -> None: - """Raise a clear ImportError if the MVT light deps are missing.""" - missing = [] - try: - import mapbox_vector_tile # noqa: F401 - except Exception: # noqa: BLE001 - missing.append("mapbox-vector-tile") - try: - import shapely # noqa: F401 - except Exception: # noqa: BLE001 - missing.append("shapely") - if missing: - raise ImportError( - "pyvx requires the [light] extra; missing: " - + ", ".join(missing) - + ". Install with: pip install 'geobrix[light]'" - ) -``` - -Also create empty `python/geobrix/test/pyvx/__init__.py` if the test package needs it (match the `test/pyrx/` layout). - -- [ ] **Step 5: Run the test to verify it passes** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_env.py -v` -Expected: PASS. - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/pyproject.toml python/geobrix/src/databricks/labs/gbx/pyvx/ python/geobrix/test/pyvx/ -git commit -m "feat(pyvx): package skeleton + mapbox-vector-tile dep + env guard" -``` - ---- - -## Task 3: Pure-Python MVT encode core (`_serde.py` + `_mvt.py`) — native typing - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyvx/_serde.py` -- Create: `python/geobrix/src/databricks/labs/gbx/pyvx/_mvt.py` -- Test: `python/geobrix/test/pyvx/test_mvt_encode.py` - -This is the heart: encoding features (geometry + **native-typed** attributes) to MVT bytes, Spark-free. - -- [ ] **Step 1: Write the failing tests (decode-and-assert native types)** - -```python -# python/geobrix/test/pyvx/test_mvt_encode.py -import mapbox_vector_tile as mvt -from shapely.geometry import Point -from shapely import to_wkb - -from databricks.labs.gbx.pyvx import _mvt - - -def _decode(blob, layer="layer"): - tile = mvt.decode(blob) - return tile[layer]["features"] - - -def test_encode_layer_preserves_native_attr_types(): - feats = [ - {"geometry": to_wkb(Point(10, 20)), "properties": {"name": "a", "pop": 42, "h": 3.5, "ok": True}}, - ] - blob = _mvt.encode_layer(feats, layer_name="layer", extent=4096) - props = _decode(blob)[0]["properties"] - assert props["name"] == "a" - assert props["pop"] == 42 and isinstance(props["pop"], int) - assert props["h"] == 3.5 and isinstance(props["h"], float) - assert props["ok"] is True - - -def test_encode_layer_unsupported_type_falls_back_to_string(): - feats = [{"geometry": to_wkb(Point(1, 1)), "properties": {"b": b"\x00\x01"}}] - blob = _mvt.encode_layer(feats, layer_name="layer", extent=4096) - props = _decode(blob)[0]["properties"] - assert isinstance(props["b"], str) # bytes -> str fallback - - -def test_pyramid_tiles_caps_and_schema(): - # A point at lon/lat 0,0 over zooms 0..2 -> one tile per zoom (3 rows). - rows = list(_mvt.pyramid_tiles(to_wkb(Point(0.0, 0.0)), {"id": 7}, 0, 2, "layer", 4096)) - zs = sorted(r[0] for r in rows) - assert zs == [0, 1, 2] - for (z, x, y, blob) in rows: - assert isinstance(z, int) and isinstance(x, int) and isinstance(y, int) - assert isinstance(blob, (bytes, bytearray)) and len(blob) > 0 - - -def test_pyramid_rejects_too_many_tiles(): - import pytest - with pytest.raises(ValueError): - # whole-world polygon at high zoom blows the 10^6 cap - from shapely.geometry import box - list(_mvt.pyramid_tiles(to_wkb(box(-179, -85, 179, 85)), {}, 0, 20, "layer", 4096)) -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_mvt_encode.py -v` -Expected: FAIL (`_mvt` has no `encode_layer`/`pyramid_tiles`). - -- [ ] **Step 3: Implement `_serde.py`** - -```python -# python/geobrix/src/databricks/labs/gbx/pyvx/_serde.py -"""Geometry + attribute marshalling for pyvx MVT encoding (Spark-free).""" -from typing import Any, Dict - -from pyspark.sql.types import ( - BinaryType, - IntegerType, - LongType, - StructField, - StructType, -) - -# Output tile struct, identical to the heavy generator's row shape. -TILE_SCHEMA = StructType( - [ - StructField("z", IntegerType(), False), - StructField("x", IntegerType(), False), - StructField("y", IntegerType(), False), - StructField("mvt_bytes", BinaryType(), True), - ] -) - -# Python native types that map to a native MVT Value; everything else -> str(). -_NATIVE = (bool, int, float, str) - - -def to_native_props(attrs: Any) -> Dict[str, Any]: - """Coerce an attrs mapping/Row into a dict of MVT-native property values. - - bool/int/float/str pass through (mapbox-vector-tile picks the matching MVT - Value field); any other type (bytes, datetime, list, dict) is str()-ified; - None values are dropped (no field emitted), matching the heavy writer. - """ - if attrs is None: - return {} - items = attrs.asDict().items() if hasattr(attrs, "asDict") else dict(attrs).items() - out: Dict[str, Any] = {} - for k, v in items: - if v is None: - continue - out[str(k)] = v if isinstance(v, _NATIVE) else str(v) - return out -``` - -- [ ] **Step 4: Implement `_mvt.py`** - -```python -# python/geobrix/src/databricks/labs/gbx/pyvx/_mvt.py -"""Pure-Python MVT encoding + XYZ pyramid tiling (Spark-free, Serverless-safe).""" -import math -from typing import Any, Dict, Iterator, List, Tuple - -import mapbox_vector_tile as mvt -from shapely import from_wkb -from shapely.geometry import box -from shapely.ops import transform - -from ._serde import to_native_props - -MAX_ZOOM = 20 -MAX_TILES = 1_000_000 -DEFAULT_EXTENT = 4096 - - -def encode_layer(features: List[Dict[str, Any]], layer_name: str, extent: int = DEFAULT_EXTENT) -> bytes: - """Encode features (each {'geometry': WKB bytes, 'properties': dict}) into one MVT layer. - - Geometry is expected in tile-local coordinates (caller transformed). Property - values keep their native Python type; non-native types are str()-ified. - """ - layer_feats = [] - for f in features: - geom = f["geometry"] - shp = from_wkb(bytes(geom)) if isinstance(geom, (bytes, bytearray)) else geom - if shp is None or shp.is_empty: - continue - layer_feats.append({"geometry": shp, "properties": to_native_props(f.get("properties"))}) - return mvt.encode( - {"name": layer_name, "features": layer_feats}, - default_options={"extents": extent}, - ) - - -def _lonlat_to_tile(lon: float, lat: float, z: int) -> Tuple[int, int]: - n = 2 ** z - x = int((lon + 180.0) / 360.0 * n) - lat_r = math.radians(lat) - y = int((1.0 - math.asinh(math.tan(lat_r)) / math.pi) / 2.0 * n) - return max(0, min(n - 1, x)), max(0, min(n - 1, y)) - - -def _tile_bounds(z: int, x: int, y: int) -> Tuple[float, float, float, float]: - n = 2 ** z - lon1 = x / n * 360.0 - 180.0 - lon2 = (x + 1) / n * 360.0 - 180.0 - lat1 = math.degrees(math.atan(math.sinh(math.pi * (1 - 2 * y / n)))) - lat2 = math.degrees(math.atan(math.sinh(math.pi * (1 - 2 * (y + 1) / n)))) - return lon1, min(lat1, lat2), lon2, max(lat1, lat2) - - -def _to_tile_local(geom, z: int, x: int, y: int, extent: int): - """Project a 4326 geometry into [0, extent] tile-pixel space for tile (z,x,y).""" - minx, miny, maxx, maxy = _tile_bounds(z, x, y) - sx = extent / (maxx - minx) - sy = extent / (maxy - miny) - return transform(lambda xs, ys, zs=None: ((xs - minx) * sx, (maxy - ys) * sy), geom) - - -def pyramid_tiles( - geom_wkb, - attrs: Any, - min_z: int, - max_z: int, - layer_name: str, - extent: int = DEFAULT_EXTENT, -) -> Iterator[Tuple[int, int, int, bytes]]: - """Yield (z, x, y, mvt_bytes) for every tile a 4326 feature intersects across [min_z, max_z]. - - Yields incrementally (no buffering) to keep the worker memory flat. Caps: - max_z <= MAX_ZOOM; total intersecting tiles <= MAX_TILES (raises ValueError). - """ - if max_z > MAX_ZOOM: - raise ValueError(f"max_z {max_z} exceeds MAX_ZOOM {MAX_ZOOM}") - shp = from_wkb(bytes(geom_wkb)) if isinstance(geom_wkb, (bytes, bytearray)) else geom_wkb - if shp is None or shp.is_empty: - return - props = to_native_props(attrs) - minx, miny, maxx, maxy = shp.bounds - # Pre-count tiles to enforce the cap before emitting anything. - total = 0 - spans = {} - for z in range(min_z, max_z + 1): - x0, y1 = _lonlat_to_tile(minx, miny, z) - x1, y0 = _lonlat_to_tile(maxx, maxy, z) - xr, yr = range(min(x0, x1), max(x0, x1) + 1), range(min(y0, y1), max(y0, y1) + 1) - spans[z] = (xr, yr) - total += len(xr) * len(yr) - if total > MAX_TILES: - raise ValueError(f"pyramid would emit > {MAX_TILES} tiles; narrow the zoom range") - for z in range(min_z, max_z + 1): - xr, yr = spans[z] - for x in xr: - for y in yr: - tb = box(*_tile_bounds(z, x, y)) - clipped = shp.intersection(tb) - if clipped.is_empty: - continue - local = _to_tile_local(clipped, z, x, y, extent) - blob = encode_layer( - [{"geometry": local, "properties": props}], layer_name, extent - ) - yield (z, x, y, blob) -``` - -- [ ] **Step 5: Run the tests to verify they pass** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_mvt_encode.py -v` -Expected: PASS (all 4). If `mapbox_vector_tile.encode` signature differs in the installed version, adjust the `default_options`/`extents` kwarg to match that version's API (verify with `python -c "import mapbox_vector_tile, inspect; print(inspect.signature(mapbox_vector_tile.encode))"`). - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/pyvx/_serde.py python/geobrix/src/databricks/labs/gbx/pyvx/_mvt.py python/geobrix/test/pyvx/test_mvt_encode.py -git commit -m "feat(pyvx): pure-Python MVT encode + XYZ pyramid (native attr types)" -``` - ---- - -## Task 4: `st_asmvt` aggregator + `register(spark)` - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py` -- Test: `python/geobrix/test/pyvx/test_asmvt.py`, `python/geobrix/test/pyvx/conftest.py` - -- [ ] **Step 1: Write `conftest.py` (spark fixture)** - -```python -# python/geobrix/test/pyvx/conftest.py -import pytest -from pyspark.sql import SparkSession - - -@pytest.fixture(scope="session") -def spark(): - s = ( - SparkSession.builder.master("local[2]") - .appName("pyvx-tests") - .config("spark.sql.shuffle.partitions", "2") - .getOrCreate() - ) - yield s - s.stop() -``` - -- [ ] **Step 2: Write the failing aggregator test** - -```python -# python/geobrix/test/pyvx/test_asmvt.py -import mapbox_vector_tile as mvt -from shapely import to_wkb -from shapely.geometry import Point - -from databricks.labs.gbx.pyvx import functions as vx - - -def test_st_asmvt_aggregates_group_to_one_tile(spark): - vx.register(spark) - rows = [ - (0, 0, 0, bytearray(to_wkb(Point(100.0, 200.0))), "a", 1), - (0, 0, 0, bytearray(to_wkb(Point(300.0, 400.0))), "b", 2), - ] - df = spark.createDataFrame(rows, "z int, x int, y int, geom binary, name string, pop int") - from pyspark.sql import functions as f - - out = ( - df.groupBy("z", "x", "y") - .agg(vx.st_asmvt(f.col("geom"), f.struct("name", "pop"), "layer").alias("mvt")) - .collect() - ) - assert len(out) == 1 - blob = bytes(out[0]["mvt"]) - feats = mvt.decode(blob)["layer"]["features"] - assert len(feats) == 2 - pops = sorted(ff["properties"]["pop"] for ff in feats) - assert pops == [1, 2] - assert all(isinstance(ff["properties"]["pop"], int) for ff in feats) -``` - -- [ ] **Step 3: Run to verify it fails** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_asmvt.py -v` -Expected: FAIL (`functions` module missing / `st_asmvt` undefined). - -- [ ] **Step 4: Implement `functions.py` (aggregator + register)** - -```python -# python/geobrix/src/databricks/labs/gbx/pyvx/functions.py -"""pyvx light VectorX API — MVT functions (Serverless-safe). - -Signatures mirror databricks.labs.gbx.vectorx.functions so light <-> heavy is a -one-line import swap. Register once with vx.register(spark), then use on columns. -""" -from typing import Union - -import pandas as pd -from pyspark.sql import Column, SparkSession -from pyspark.sql import functions as f -from pyspark.sql.functions import pandas_udf -from pyspark.sql.types import BinaryType - -from . import _env, _mvt - -ColLike = Union[Column, str, bool, int, float, bytes] - - -def _col(x: ColLike) -> Union[Column, str]: - if isinstance(x, Column) or isinstance(x, str): - return x - return f.lit(x) - - -# --- st_asmvt: grouped-aggregate pandas UDF ------------------------------------------------- -@pandas_udf(BinaryType()) -def _asmvt_udf(geom: pd.Series, attrs: pd.Series, layer: pd.Series) -> bytes: - """Grouped-agg: encode one group's features into a single MVT layer blob.""" - layer_name = "layer" - if layer is not None and len(layer) > 0 and layer.iloc[0] is not None: - layer_name = str(layer.iloc[0]) - feats = [ - {"geometry": bytes(g), "properties": a} - for g, a in zip(geom, attrs) - if g is not None and len(bytes(g)) > 0 - ] - return _mvt.encode_layer(feats, layer_name=layer_name) - - -def register(spark: SparkSession = None) -> None: - """Register the pyvx MVT SQL functions (Serverless-safe: udf/udtf only).""" - _env.assert_mvt_available() - if spark is None: - spark = SparkSession.builder.getOrCreate() - spark.udf.register("gbx_st_asmvt", _asmvt_udf) - # st_asmvt_pyramid registration is added in Task 5. - - -def st_asmvt(geom_wkb: ColLike, attrs: ColLike, layer_name: ColLike) -> Column: - """Aggregator: encode a group of features into an MVT protobuf blob (BINARY). - - geom_wkb: per-row WKB geometry in tile-local coordinates. - attrs: per-row attribute struct (native-typed in the output tile). - layer_name: constant MVT layer name (plain str -> literal). - """ - if isinstance(layer_name, str): - layer_name = f.lit(layer_name) - return _asmvt_udf(_col(geom_wkb), _col(attrs), _col(layer_name)) -``` - -Note: `_asmvt_udf` is a grouped-aggregate pandas UDF — receives each group's columns as `pd.Series`. Registering it via `spark.udf.register("gbx_st_asmvt", _asmvt_udf)` exposes the SQL name; the Python `st_asmvt(...)` wrapper calls the UDF object directly so `df.groupBy(...).agg(vx.st_asmvt(...))` works. - -- [ ] **Step 5: Run the test to verify it passes** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_asmvt.py -v` -Expected: PASS. (If the installed pandas-UDF API requires an explicit `functionType`/`PandasUDFType.GROUPED_AGG`, add it; on Spark 3.5+/4.0 the type-hinted `pandas_udf` returning a scalar from `Series` inputs is treated as a grouped-agg in `.agg(...)`.) - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/pyvx/functions.py python/geobrix/test/pyvx/test_asmvt.py python/geobrix/test/pyvx/conftest.py -git commit -m "feat(pyvx): st_asmvt grouped-agg UDF + register" -``` - ---- - -## Task 5: `st_asmvt_pyramid` generator (UDTF — 2B; or 2A fallback) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py` -- Test: `python/geobrix/test/pyvx/test_asmvt_pyramid.py` - -> Use the approach chosen in Task 1. The **2B (UDTF)** form is below; if Task 1 selected **2A**, implement `st_asmvt_pyramid` as a `pandas_udf(ArrayType(_serde.TILE_SCHEMA))` returning the list from `_mvt.pyramid_tiles`, and the test calls it as `df.select(f.explode(vx.st_asmvt_pyramid(...)).alias("t"))`. - -- [ ] **Step 1: Write the failing test (2B / UDTF)** - -```python -# python/geobrix/test/pyvx/test_asmvt_pyramid.py -import mapbox_vector_tile as mvt -from shapely import to_wkb -from shapely.geometry import Point - -from databricks.labs.gbx.pyvx import functions as vx - - -def test_st_asmvt_pyramid_fans_out_per_tile(spark): - vx.register(spark) - df = spark.createDataFrame( - [(bytearray(to_wkb(Point(0.0, 0.0))), "a", 7)], - "geom binary, name string, id int", - ) - df.createOrReplaceTempView("feats") - out = spark.sql( - "SELECT t.z, t.x, t.y, t.mvt_bytes " - "FROM feats, LATERAL gbx_st_asmvt_pyramid(geom, struct(name, id), 0, 2, 'layer', 4096) t" - ).collect() - zs = sorted({r["z"] for r in out}) - assert zs == [0, 1, 2] - blob = bytes([r for r in out if r["z"] == 2][0]["mvt_bytes"]) - feats = mvt.decode(blob)["layer"]["features"] - assert feats[0]["properties"]["id"] == 7 -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_asmvt_pyramid.py -v` -Expected: FAIL (`gbx_st_asmvt_pyramid` not registered). - -- [ ] **Step 3: Implement the UDTF + register + wrapper (2B)** - -Add to `functions.py`: - -```python -from pyspark.sql.functions import udtf - - -@udtf(returnType=_mvt_tile_return()) # defined below -class _AsMvtPyramidUDTF: - def eval(self, geom_wkb, attrs, min_z: int, max_z: int, layer_name=None, extent=None): - ln = "layer" if layer_name is None else str(layer_name) - ex = _mvt.DEFAULT_EXTENT if extent is None else int(extent) - # yield incrementally — never build the full list (fan-out OOM guard) - for z, x, y, blob in _mvt.pyramid_tiles(geom_wkb, attrs, int(min_z), int(max_z), ln, ex): - yield (z, x, y, blob) -``` - -Add the return-type helper and extend `register`: - -```python -def _mvt_tile_return(): - from ._serde import TILE_SCHEMA - return TILE_SCHEMA - - -# inside register(spark), after the st_asmvt line: - spark.udtf.register("gbx_st_asmvt_pyramid", _AsMvtPyramidUDTF) -``` - -And the Python convenience wrapper (DataFrame ergonomics + parity with the heavy signature): - -```python -def st_asmvt_pyramid( - geom_wkb: ColLike, - attrs: ColLike, - min_z: ColLike, - max_z: ColLike, - layer_name: Union[ColLike, None] = None, - extent: Union[ColLike, None] = None, -): - """Generator: one (z,x,y,mvt_bytes) row per intersecting tile across [min_z,max_z]. - - Light tier is a Python UDTF — invoke as a table function: - SELECT t.* FROM features, LATERAL gbx_st_asmvt_pyramid(geom, attrs, 0, 12, 'layer', 4096) t - The output schema (z,x,y,mvt_bytes) matches the heavyweight generator and feeds gbx_pmtiles_agg. - """ - raise NotImplementedError( - "Invoke the registered UDTF as a SQL LATERAL table function: " - "SELECT t.* FROM , LATERAL gbx_st_asmvt_pyramid(geom, attrs, min_z, max_z, layer, extent) t" - ) -``` - -(The `st_asmvt_pyramid` Python function documents the UDTF call form; the usable surface is the registered `gbx_st_asmvt_pyramid` SQL table function. If Task 1 chose 2A, instead make `st_asmvt_pyramid` return `_pyramid_array_udf(...)` and drop the `raise`.) - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_asmvt_pyramid.py -v` -Expected: PASS. - -- [ ] **Step 5: Add a cap-enforcement test + run full pyvx suite** - -```python -def test_pyramid_cap_raises(spark): - vx.register(spark) - from shapely.geometry import box - df = spark.createDataFrame([(bytearray(to_wkb(box(-179, -85, 179, 85))),)], "geom binary") - df.createOrReplaceTempView("big") - import pytest - with pytest.raises(Exception): - spark.sql( - "SELECT t.z FROM big, LATERAL gbx_st_asmvt_pyramid(geom, struct(), 0, 20, 'l', 4096) t" - ).collect() -``` - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/ -v` -Expected: all pyvx tests PASS. - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/pyvx/functions.py python/geobrix/test/pyvx/test_asmvt_pyramid.py -git commit -m "feat(pyvx): st_asmvt_pyramid generator (UDTF, incremental yield)" -``` - ---- - -## Task 6: Serverless-safety guard covers pyvx - -**Files:** -- Modify: `python/geobrix/test/pyrx/test_serverless_no_spark_config.py` - -- [ ] **Step 1: Extend the guard's source-file set to include pyvx** - -In `_source_files()`, add the pyvx package dir alongside the existing pyrx/ds dirs. The function currently globs the light source roots; add: - -```python - roots.append(Path(__file__).parents[2] / "src" / "databricks" / "labs" / "gbx" / "pyvx") -``` - -(Match the existing `roots`/`Path(...)` construction in that file — read it first and append the `pyvx` root the same way the `pyrx` root is added.) - -- [ ] **Step 2: Run the guard** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/test_serverless_no_spark_config.py -v` -Expected: PASS (pyvx uses only `spark.udf.register`/`spark.udtf.register` + Column/UDF exprs — no `_jvm`/`conf.set`/`.rdd`). - -- [ ] **Step 3: Commit** - -```bash -git add python/geobrix/test/pyrx/test_serverless_no_spark_config.py -git commit -m "test(pyvx): cover pyvx in the Serverless-safety guard" -``` - ---- - -## Task 7: Heavy `MvtWriter` — native OGR field types - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/vectorx/mvt/MvtWriter.scala` (attr field-type logic, ~lines 94–127) -- Modify (if needed): `src/main/scala/com/databricks/labs/gbx/vectorx/expressions/ST_AsMvt.scala` (`encodeAttrs`/`decodeAttrs`, ~lines 109–153) to preserve typed values -- Test: `src/test/scala/com/databricks/labs/gbx/vectorx/expressions/ST_AsMvtTest.scala` - -Runs in the `geobrix-dev` Docker container. - -- [ ] **Step 1: Write the failing Scala test (native-typed decode)** - -Add to `ST_AsMvtTest.scala` a test that encodes an int + double attr, re-reads the MVT bytes with the OGR `MVT` driver, and asserts the field comes back numeric (not string): - -```scala -test("st_asmvt encodes numeric attributes with native MVT value types") { - vectorx.functions.register(spark) - import vectorx.functions._ - import org.apache.spark.sql.functions.{col, struct, lit} - - val gf = new org.locationtech.jts.geom.GeometryFactory() - val pt = gf.createPoint(new org.locationtech.jts.geom.Coordinate(0.5, 0.5)) - val df = spark.createDataFrame(Seq((JTS.toWKB(pt), 42, 3.5))) - .toDF("geom_wkb", "pop", "h") - - val mvtBytes = df.agg( - st_asmvt(col("geom_wkb"), struct(col("pop"), col("h")), lit("layer1")).as("mvt") - ).collect().head.getAs[Array[Byte]]("mvt") - - // Decode with OGR MVT driver and assert field types are numeric, not string. - val (popType, hType) = MvtTestUtil.readFieldTypes(mvtBytes, "layer1", Seq("pop", "h")) - assert(popType == org.gdal.ogr.ogrConstants.OFTInteger || popType == org.gdal.ogr.ogrConstants.OFTInteger64) - assert(hType == org.gdal.ogr.ogrConstants.OFTReal) -} -``` - -Add a small `MvtTestUtil.readFieldTypes` test helper (in the test tree) that writes the bytes to a `/vsimem/` path, opens with the OGR `MVT` driver, and returns the layer's field types. If introducing a helper is undesirable, inline the `/vsimem/` open in the test. (Use `GDALManager.initOgr()` for driver registration, per the repo's thread-safety rule.) - -- [ ] **Step 2: Run to verify it fails** - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.vectorx.expressions.ST_AsMvtTest' --log mvt-native.log` -Expected: FAIL (current writer makes every field `OFTString`). - -- [ ] **Step 3: Make `MvtWriter` infer native field types** - -Replace the all-`OFTString` field creation (line ~105) and the `v.toString` SetField (line ~124) with type-aware logic. For each field, infer the OGR type from the first non-null value's runtime type, then set with the typed setter: - -```scala -// helper: OGR field type for a Scala attribute value -private def ogrFieldType(v: Any): Int = v match { - case _: Int | _: java.lang.Integer => ogrConstants.OFTInteger - case _: Long | _: java.lang.Long => ogrConstants.OFTInteger64 - case _: Double | _: Float - | _: java.lang.Double | _: java.lang.Float => ogrConstants.OFTReal - case _: Boolean | _: java.lang.Boolean => ogrConstants.OFTInteger // subtype Boolean below - case _ => ogrConstants.OFTString -} - -// field creation: type per field from first non-null value across features -schema.foreach { fieldName => - val firstVal = features.iterator.map(_._2).filter(_ != null) - .flatMap(m => m.get(fieldName)).find(_ != null) - val ft = firstVal.map(ogrFieldType).getOrElse(ogrConstants.OFTString) - val fd = new FieldDefn(fieldName, ft) - if (firstVal.exists(_.isInstanceOf[Boolean])) fd.SetSubType(ogrConstants.OFSTBoolean) - layer.CreateField(fd) - fd.delete() -} - -// per-feature SetField: typed setters, str fallback -attrs.get(fieldName).foreach { - case v: Int => feat.SetField(fieldName, v) - case v: Long => feat.SetFieldInteger64(feat.GetFieldIndex(fieldName), v) - case v: Double => feat.SetField(fieldName, v) - case v: Float => feat.SetField(fieldName, v.toDouble) - case v: Boolean => feat.SetField(fieldName, if (v) 1 else 0) - case null => () - case v => feat.SetField(fieldName, v.toString) -} -``` - -Adjust to the exact OGR Java binding method names/imports present in the file (read the current imports; `ogrConstants` may be imported as `ogr` constants). Update the v0.4.0 "all OFTString" comment (lines 25–26) to describe native typing. - -- [ ] **Step 4: Ensure `ST_AsMvt.encodeAttrs`/`decodeAttrs` preserve types** - -Read `ST_AsMvt.scala` lines ~109–153. If `encodeAttrs` serializes attribute values to strings (so `decodeAttrs` yields a `Map[String, String]`), change the (de)serialization to preserve the Catalyst field types — carry the struct `dataType` so `decodeAttrs` returns `Map[String, Any]` with `Int`/`Long`/`Double`/`Boolean`/`String` runtime values that `MvtWriter.ogrFieldType` can switch on. The Task-7 test is the definition of done: if it passes, types are preserved end to end. - -- [ ] **Step 5: Run the suite to verify it passes** - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.vectorx.expressions.ST_AsMvtTest' --log mvt-native.log` -Expected: all ST_AsMvt tests PASS (including the new native-type test; existing "non-empty blob"/"layer name" tests still hold). - -- [ ] **Step 6: Update any heavy MVT docs that say "stringified"** - -`grep -rn -i "OFTString\|stringif\|as string" src/main/scala/com/databricks/labs/gbx/vectorx/mvt/ docs/docs/` and update the doc/comment wording to "native value types". Re-run the wave-leak check is not needed; just keep wording accurate. - -- [ ] **Step 7: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/vectorx/mvt/MvtWriter.scala src/main/scala/com/databricks/labs/gbx/vectorx/expressions/ST_AsMvt.scala src/test/scala/com/databricks/labs/gbx/vectorx/expressions/ -git commit -m "feat(vectorx): native MVT attribute value types (was OFTString)" -``` - ---- - -## Task 8: Light-vs-heavy decoded-feature parity test - -**Files:** -- Test: `python/geobrix/test/pyvx/test_parity_mvt.py` - -Integration test — needs the heavy JAR in `python/geobrix/lib/` and the Docker container with `/Volumes` (per the corpus-parity convention). Guard with a skip when the JAR/Spark-with-JVM isn't present (like the existing corpus-parity tests). - -- [ ] **Step 1: Write the parity test** - -```python -# python/geobrix/test/pyvx/test_parity_mvt.py -import os - -import mapbox_vector_tile as mvt -import pytest -from shapely import to_wkb -from shapely.geometry import Point - -pytestmark = pytest.mark.skipif( - not os.environ.get("GBX_HEAVY_JAR"), - reason="needs heavyweight JAR (set GBX_HEAVY_JAR); run in geobrix-dev Docker", -) - - -def _feats(blob, layer="layer"): - return {(round(p["properties"]["id"])): p["properties"] for p in mvt.decode(blob)[layer]["features"]} - - -def test_light_vs_heavy_asmvt_decoded_parity(spark): - from databricks.labs.gbx.pyvx import functions as vx - from databricks.labs.gbx.vectorx import functions as hx - from pyspark.sql import functions as f - - vx.register(spark) - hx.register(spark) - rows = [(bytearray(to_wkb(Point(100.0, 200.0))), 1, 3.5), (bytearray(to_wkb(Point(300.0, 400.0))), 2, 9.0)] - df = spark.createDataFrame(rows, "geom binary, id int, h double") - - light = bytes(df.agg(vx.st_asmvt(f.col("geom"), f.struct("id", "h"), "layer")).collect()[0][0]) - heavy = bytes(df.agg(hx.st_asmvt(f.col("geom"), f.struct("id", "h"), "layer")).collect()[0][0]) - - lf, hf = _feats(light), _feats(heavy) - assert lf.keys() == hf.keys() - for k in lf: - assert lf[k]["id"] == hf[k]["id"] and isinstance(lf[k]["id"], int) - assert abs(float(lf[k]["h"]) - float(hf[k]["h"])) < 1e-9 -``` - -- [ ] **Step 2: Run it in Docker (JAR present)** - -Run in the container (which stages the JAR + a JVM Spark): the existing `gbx:test:python` path for integration, e.g. -`bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/test_parity_mvt.py --log pyvx-parity.log` -(start the container with the volumes script if the test reads corpus; for this synthetic test only the JAR is needed). Expected: PASS — decoded features (ids + native-typed values) match across tiers. - -- [ ] **Step 3: Commit** - -```bash -git add python/geobrix/test/pyvx/test_parity_mvt.py -git commit -m "test(pyvx): light-vs-heavy decoded-feature MVT parity" -``` - ---- - -## Task 9: Bench — `st_asmvt` light-vs-heavy timing + parity - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/readers.py` (add `run_mvt_agg`) -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/cluster.py` (add `_CELL_MVT`, wire `mvt_only`) -- Modify: `notebooks/tests/push_and_run_bench_on_cluster.py` (add `--mvt-only` flag + run_id suffix) - -Mirror the existing PMTiles bench (`run_pmtiles_write` + `_CELL_PMTILES` + `--pmtiles-only`) — it is the closest precedent (a non-reader op with a parity check). - -- [ ] **Step 1: Add `run_mvt_agg` to `readers.py`** - -Implement `run_mvt_agg(spark, run_id, warmup, measured, *, api, where="cluster")` that: builds (or reads) a synthetic features DataFrame keyed by `(z,x,y)`, times `groupBy("z","x","y").agg(.st_asmvt(...))` (light = `pyvx`, heavy = `vectorx`), and returns a `ResultRow(category="mvt", fn="st_asmvt", mode="spark-path", ...)`. Follow `run_pmtiles_write`'s structure (timing via `time_iters`, env via `capture_env`, registration per tier). - -- [ ] **Step 2: Add `_CELL_MVT` to `cluster.py` and a `mvt_only` branch** - -Mirror `_CELL_PMTILES`: when `MVT_ONLY`/`BENCHMARK_MVT`, run `run_mvt_agg` for light and heavy, `_sink` the rows, decode both tiers' tiles and assert decoded-feature parity (like the PMTiles cell), then display the `category='mvt'` rows. - -- [ ] **Step 3: Add the `--mvt-only` flag to the launcher** - -In `push_and_run_bench_on_cluster.py`: parse `--mvt-only`/`--benchmark-mvt`, give `mvt_only` its own run_id suffix (`-mvt`), thread `mvt_only` into `cfg` and the cell selection (mirror `pmtiles_only`). Ensure the row-pool guard skip already covers `*-only` runs (it does, after the earlier fix). - -- [ ] **Step 4: Run the bench on the cluster** - -Run: `bash scripts/commands/gbx-bench-cluster.sh --mvt-only --spark-warmup 0 --spark-measured 1` -Expected: 2 rows (`api=lightweight` + `heavyweight`, `category=mvt`), parity PASS, light/heavy timings. - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/bench/ notebooks/tests/push_and_run_bench_on_cluster.py -git commit -m "feat(bench): st_asmvt light-vs-heavy MVT bench + parity (--mvt-only)" -``` - ---- - -## Task 10: Docs — pyvx MVT page + Benchmarking Vector tab + binding consistency - -**Files:** -- Create: `docs/docs/api/` (or the VectorX docs location) pyvx MVT page -- Modify: `docs/docs/api/benchmarking.mdx` (the **Vector (soon)** tab) -- Verify: `docs/tests-function-info/registered_functions.txt` + `function-info.json` + `gbx:test:bindings` - -- [ ] **Step 1: Write the pyvx MVT doc page** - -Mirror the readers/writers doc template (Options near the top if any, Next Steps last, light+heavy tabs via `groupId="gbx-tier"`). Show: `register(spark)`, `groupBy(...).agg(pyvx.st_asmvt(geom, attrs, "layer"))`, and the `LATERAL gbx_st_asmvt_pyramid(...)` table-function usage; note native attribute typing and Serverless support. Use real example code consistent with the doc-test convention if adding a doc test; otherwise show prose examples (per the light-vector-pages precedent). - -- [ ] **Step 2: Fill the Benchmarking Vector tab** - -In `benchmarking.mdx`, replace the **Vector (soon)** placeholder TabItem content with the `st_asmvt` light-vs-heavy numbers from Task 9 (timing table + parity note), matching the Readers & Writers tab style. - -- [ ] **Step 3: Verify binding parity + function-info** - -`gbx_st_asmvt` / `gbx_st_asmvt_pyramid` already exist in `registered_functions.txt` (heavy). Run `bash scripts/commands/gbx-test-bindings.sh` (or the equivalent) to confirm Scala name + Python binding(s) + `function-info.json` stay consistent now that `pyvx` adds a second Python binding. Fix any gap upstream (no placeholders). - -- [ ] **Step 4: Build the docs + wave-leak check** - -Run: `cd docs && npm run build` (expect SUCCESS) and `grep -rn -iE "wave [0-9]+" docs/docs/` (expect empty). - -- [ ] **Step 5: Commit** - -```bash -git add docs/ -git commit -m "docs(pyvx): MVT light tier page + benchmarking Vector tab" -``` - ---- - -## Final review - -After all tasks: dispatch a final code review across the branch (light pyvx package, heavy MVT native-typing change, bench, docs), confirm the full pyvx pytest suite + the ST_AsMvt Scala suite are green, then finish the branch (PR into `beta/0.4.0`) per the project's PR flow. diff --git a/docs/superpowers/plans/2026-06-13-pyvx-vectorx-tin-legacy.md b/docs/superpowers/plans/2026-06-13-pyvx-vectorx-tin-legacy.md deleted file mode 100644 index c351a14c3..000000000 --- a/docs/superpowers/plans/2026-06-13-pyvx-vectorx-tin-legacy.md +++ /dev/null @@ -1,1314 +0,0 @@ -# pyvx VectorX TIN + Legacy Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Port the 4 remaining heavy VectorX `gbx_st_*` functions (`st_legacyaswkb`, `st_triangulate`, `st_interpolateelevationbbox`, `st_interpolateelevationgeom`) to the pyvx light tier, with cross-tier alignment, so the light tier is a genuine exit from heavy. - -**Architecture:** Pure-Python/PySpark light tier (Serverless/Connect-safe: `udf`/`udtf` + Column only — never `_jvm`/`spark.conf.set`/`.rdd`). TIN triangulation = scipy `Delaunay` + a hand-rolled Sloan constraint-recovery core in `pyvx/_tin.py`; legacy decode = shapely in `pyvx/_legacy.py`. Heavy gains a `mode` param (`"constrained"` default / `"conforming"` opt-in) and two legacy bug-fixes (preserve Z, preserve holes). - -**Tech Stack:** Python 3.12, scipy `Delaunay`, shapely 2.x (WKB/WKT I/O), numpy, PySpark `@udf`/`@udtf`; Scala 2.13 / JTS (heavy). - -**Spec:** `docs/superpowers/specs/2026-06-13-pyvx-vectorx-tin-legacy-light-tier-design.md` - -**Branch:** `pyvx-light`. **PR:** #38 (draft — won't merge until this lands). - -**No new dependencies** — `scipy>=1.11.0`, `shapely>=2.0.0`, `mapbox-vector-tile` are already in the `[light]` / `[test]` extras of `python/geobrix/pyproject.toml`. - ---- - -## Conventions every task follows - -- Run Python tests via the dev container: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/ --log .log`. Spark-free unit tests (`_tin.py`, `_legacy.py`) also run on the host venv `.venv-pyrx/bin/python -m pytest`. -- Run Scala tests: `bash scripts/commands/gbx-test-scala.sh --suites '' --log .log`. -- Commit messages: subject ≤72 chars + a WHY body for non-trivial commits; trailer exactly `Co-authored-by: Isaac`. Before commit: `chmod -R u+rwX .git/objects`. -- Serverless-safety: the pyvx package must never reference `_jvm`, `sparkContext`, `.rdd`, or `spark.conf.set`. `test/pyvx/test_serverless_no_spark_config.py` already guards this — keep it green. - -## Geometry input contract (cross-cutting — applies to EVERY geom-accepting function) - -Every pyvx function that accepts a geometry argument MUST accept the **same set of encodings**, consistent with the heavy ST surface: **WKB, EWKB, WKT, and EWKT** (`SRID=;`). This is centralized in one helper so the contract can't drift between functions. - -- Create `python/geobrix/src/databricks/labs/gbx/pyvx/_geom.py` with: - -```python -"""Shared geometry-input parsing for the pyvx light tier. - -Every geom-accepting pyvx function uses parse_geom so the accepted encodings -(WKB / EWKB / WKT / EWKT) stay consistent across the ST surface and match the -heavyweight tier (which accepts BINARY|STRING for geometry inputs). -""" -from typing import Any, Optional -from shapely import from_wkb, from_wkt, set_srid -from shapely.geometry.base import BaseGeometry - - -def parse_geom(x: Any) -> Optional[BaseGeometry]: - """Parse a geometry from WKB/EWKB bytes or WKT/EWKT text. None -> None. - - shapely.from_wkb already reads EWKB (SRID embedded). shapely.from_wkt does - NOT understand the EWKT 'SRID=;' prefix, so strip+apply it here. - """ - if x is None: - return None - if isinstance(x, (bytes, bytearray)): - return from_wkb(bytes(x)) # handles WKB and EWKB - s = str(x).strip() - if s[:5].upper() == "SRID=": - srid_part, _, wkt_part = s.partition(";") - geom = from_wkt(wkt_part) - try: - return set_srid(geom, int(srid_part[5:])) - except ValueError: - return geom - return from_wkt(s) -``` - -- All geom-accepting functions call `parse_geom` (TIN `points`/`breaklines` via `_geoms_from_array`, the geom-grid `grid_origin`). Where a task below shows inline `from_wkb`/`from_wkt`, replace it with `parse_geom`. -- **Consistency audit (Task 12):** confirm the existing `st_asmvt` geom input (`_asmvt_udf`) accepts the same set (today it takes WKB bytes only) and align it to `parse_geom` if heavy `gbx_st_asmvt` accepts WKT/EWKT too; verify all `gbx_st_*` geom params share the contract. - ---- - -## File Structure - -| File | Responsibility | -|---|---| -| `python/geobrix/src/databricks/labs/gbx/pyvx/_geom.py` | **new** — shared `parse_geom` (WKB/EWKB/WKT/EWKT) used by every geom-accepting pyvx function | -| `python/geobrix/src/databricks/labs/gbx/pyvx/_legacy.py` | **new** — decode legacy Mosaic struct → shapely geom (Z + holes preserved) | -| `python/geobrix/src/databricks/labs/gbx/pyvx/_tin.py` | **new** — scipy Delaunay + Sloan constraint recovery + Z-snap + barycentric interp + grid generators (Spark-free) | -| `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py` | **modify** — add `st_legacyaswkb` UDF, 3 TIN `@udtf` classes + wrappers, extend `register()` | -| `python/geobrix/src/databricks/labs/gbx/pyvx/_env.py` | **modify** — add a `assert_tin_available()` guard (scipy + shapely) | -| `python/geobrix/test/pyvx/test_legacy.py` | **new** — Spark-free `_legacy` unit tests | -| `python/geobrix/test/pyvx/test_tin_core.py` | **new** — Spark-free `_tin` unit tests (the Sloan core, hardest) | -| `python/geobrix/test/pyvx/test_legacy_udf.py` | **new** — registered `st_legacyaswkb` via spark fixture | -| `python/geobrix/test/pyvx/test_tin_udtf.py` | **new** — registered TIN UDTFs via spark fixture | -| `python/geobrix/test/pyvx/test_parity_legacy.py` | **new** — light↔heavy parity (JAR-gated) | -| `python/geobrix/test/pyvx/test_parity_tin.py` | **new** — light↔heavy TIN parity (JAR-gated) | -| `src/main/scala/com/databricks/labs/gbx/vectorx/jts/legacy/InternalGeometry.scala` | **modify** — fix dropped-holes TODO | -| `src/main/scala/com/databricks/labs/gbx/vectorx/jts/legacy/expressions/ST_LegacyAsWKB.scala` | **modify** — `toWKB` → `toWKB3` (preserve Z) | -| `src/main/scala/com/databricks/labs/gbx/vectorx/jts/InterpolateElevation.scala` | **modify** — add `mode` (constrained vs conforming) to `triangulate` | -| `src/main/scala/com/databricks/labs/gbx/vectorx/expressions/ST_Triangulate.scala` | **modify** — add trailing `mode` arg (arity 5→5/6) | -| `src/main/scala/com/databricks/labs/gbx/vectorx/expressions/ST_InterpolateElevationBBox.scala` | **modify** — add trailing `mode` arg (arity 12→12/13) | -| `src/main/scala/com/databricks/labs/gbx/vectorx/expressions/ST_InterpolateElevationGeom.scala` | **modify** — add trailing `mode` arg (arity 10→10/11) | -| `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py` (bindings) + `docs/tests/python/api/vectorx_functions_sql.py` + `docs/docs/api/vectorx-functions.mdx` | **modify** — bindings, function-info examples, docs | - ---- - -# PHASE 1 — Legacy (`st_legacyaswkb`), both tiers - -Independent, low-risk, establishes the non-MVT scalar-UDF pattern. - -## Task 1: Light legacy decode core (`_legacy.py`) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyvx/_legacy.py` -- Test: `python/geobrix/test/pyvx/test_legacy.py` - -The input is the legacy Mosaic struct, which arrives in a UDF as a `pyspark.sql.Row` (or a nested dict). Shape: -`{typeId: int, srid: int, boundaries: list[list[list[float]]], holes: list[list[list[list[float]]]]}`. -typeId map: 1 POINT, 2 MULTIPOINT, 3 LINESTRING, 4 MULTILINESTRING, 5 POLYGON, 6 MULTIPOLYGON, 7 LINEARRING, 8 GEOMETRYCOLLECTION (→ raise). Each coordinate is `[x, y]` or `[x, y, z]`. - -- [ ] **Step 1: Write the failing tests** - -```python -# test/pyvx/test_legacy.py -import pytest - -shapely = pytest.importorskip("shapely") -from shapely.geometry import Point, LineString, Polygon, MultiPolygon # noqa: E402 -from shapely import wkb # noqa: E402 - -from databricks.labs.gbx.pyvx import _legacy - - -def _row(type_id, boundaries, holes=None, srid=0): - return {"typeId": type_id, "srid": srid, "boundaries": boundaries, "holes": holes or []} - - -def test_point_xy(): - g = _legacy.legacy_to_geom(_row(1, [[[30.0, 10.0]]])) - assert g.equals(Point(30.0, 10.0)) - - -def test_point_xyz_preserves_z(): - g = _legacy.legacy_to_geom(_row(1, [[[30.0, 10.0, 5.0]]])) - assert g.has_z and abs(g.z - 5.0) < 1e-9 - - -def test_linestring(): - g = _legacy.legacy_to_geom(_row(3, [[[0.0, 0.0], [1.0, 1.0]]])) - assert g.equals(LineString([(0, 0), (1, 1)])) - - -def test_polygon_preserves_holes(): - # outer 0..10 square, one inner hole 2..4 square - outer = [[0.0, 0.0], [10.0, 0.0], [10.0, 10.0], [0.0, 10.0], [0.0, 0.0]] - hole = [[2.0, 2.0], [4.0, 2.0], [4.0, 4.0], [2.0, 4.0], [2.0, 2.0]] - g = _legacy.legacy_to_geom(_row(5, [outer], holes=[[hole]])) - assert len(g.interiors) == 1 - assert abs(g.area - (100.0 - 4.0)) < 1e-9 - - -def test_multipolygon_preserves_holes(): - sq = lambda o, s: [[o, o], [o + s, o], [o + s, o + s], [o, o + s], [o, o]] - poly0 = [sq(0.0, 10.0)] - hole0 = [sq(2.0, 2.0)] - poly1 = [sq(20.0, 5.0)] - g = _legacy.legacy_to_geom(_row(6, [poly0[0], poly1[0]], holes=[[hole0[0]], []])) - assert isinstance(g, MultiPolygon) - assert sum(len(p.interiors) for p in g.geoms) == 1 - - -def test_geometrycollection_raises(): - with pytest.raises(ValueError, match="GeometryCollection"): - _legacy.legacy_to_geom(_row(8, [])) - - -def test_aswkb_preserves_z_iso(): - out = _legacy.legacy_to_wkb(_row(1, [[[30.0, 10.0, 5.0]]])) - assert wkb.loads(out).has_z -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_legacy.py -v` -Expected: FAIL (`module databricks.labs.gbx.pyvx has no attribute _legacy` / `legacy_to_geom`). - -- [ ] **Step 3: Implement `_legacy.py`** - -```python -"""Legacy Mosaic geometry decode for the pyvx light tier. - -Decodes the legacy internal struct {typeId, srid, boundaries, holes} into a -shapely geometry, preserving Z and polygon holes, then serializes to WKB. -Heavy parity target: databricks.labs.gbx.vectorx.jts.legacy (with the Z-drop -and holes-drop bugs fixed in both tiers). -""" -from typing import Any, List, Optional, Sequence - -from shapely import to_wkb -from shapely.geometry import ( - LinearRing, - LineString, - MultiLineString, - MultiPoint, - MultiPolygon, - Point, - Polygon, -) - -# typeId -> name (mirrors GeometryTypeEnum in jts/legacy) -_POINT, _MULTIPOINT, _LINESTRING, _MULTILINESTRING = 1, 2, 3, 4 -_POLYGON, _MULTIPOLYGON, _LINEARRING, _GEOMETRYCOLLECTION = 5, 6, 7, 8 - - -def _field(row: Any, name: str, idx: int) -> Any: - """Read a field from a Row or dict-like legacy struct.""" - if hasattr(row, "__fields__"): # pyspark Row - return row[name] - if isinstance(row, dict): - return row.get(name) - return row[idx] # positional tuple fallback - - -def _ring(coords: Sequence[Sequence[float]]) -> List[tuple]: - return [tuple(float(v) for v in c) for c in coords] - - -def legacy_to_geom(row: Any): - """Decode the legacy struct into a shapely geometry (Z + holes preserved).""" - type_id = int(_field(row, "typeId", 0)) - boundaries = _field(row, "boundaries", 2) or [] - holes = _field(row, "holes", 3) or [] - - if type_id == _POINT: - return Point(*_ring(boundaries[0])[0]) - if type_id == _MULTIPOINT: - return MultiPoint([_ring(p)[0] for p in boundaries[0]]) - if type_id in (_LINESTRING, _LINEARRING): - return LineString(_ring(boundaries[0])) - if type_id == _MULTILINESTRING: - return MultiLineString([_ring(ls) for ls in boundaries]) - if type_id == _POLYGON: - shell = _ring(boundaries[0]) - rings = holes[0] if holes else [] - return Polygon(shell, [_ring(h) for h in rings]) - if type_id == _MULTIPOLYGON: - polys = [] - for i, shell_coords in enumerate(boundaries): - shell = _ring(shell_coords) - rings = holes[i] if i < len(holes) and holes[i] else [] - polys.append(Polygon(shell, [_ring(h) for h in rings])) - return MultiPolygon(polys) - if type_id == _GEOMETRYCOLLECTION: - raise ValueError("GeometryCollection is not supported by st_legacyaswkb") - raise ValueError(f"unknown legacy geometry typeId: {type_id}") - - -def legacy_to_wkb(row: Any) -> Optional[bytes]: - """Decode the legacy struct and return ISO WKB (Z preserved when present).""" - if row is None: - return None - geom = legacy_to_geom(row) - # shapely defaults: flavor="iso", output_dimension=3 -> Z written when present. - return to_wkb(geom) -``` - -- [ ] **Step 4: Run to verify they pass** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_legacy.py -v` -Expected: PASS (7 tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyvx/_legacy.py python/geobrix/test/pyvx/test_legacy.py -git commit -m "feat(pyvx): legacy geometry decode core (Z + holes preserved) - -Pure shapely decode of the legacy Mosaic struct; preserves polygon holes and -Z (ISO WKB), which the heavy tier currently drops. Spark-free, TDD." -``` - -## Task 2: Light `st_legacyaswkb` UDF + register + wrapper - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/pyvx/_env.py` -- Test: `python/geobrix/test/pyvx/test_legacy_udf.py` - -- [ ] **Step 1: Write the failing test** - -```python -# test/pyvx/test_legacy_udf.py -import pytest - -shapely = pytest.importorskip("shapely") -from shapely import wkb # noqa: E402 - -from databricks.labs.gbx.pyvx import functions as vx - - -def test_st_legacyaswkb_roundtrips_polygon_with_hole(spark): - vx.register(spark) - outer = [[0.0, 0.0], [10.0, 0.0], [10.0, 10.0], [0.0, 10.0], [0.0, 0.0]] - hole = [[2.0, 2.0], [4.0, 2.0], [4.0, 4.0], [2.0, 4.0], [2.0, 2.0]] - schema = "g struct>>,holes:array>>>>" - df = spark.createDataFrame([({"typeId": 5, "srid": 0, "boundaries": [outer], "holes": [[hole]]},)], schema) - out = df.selectExpr("gbx_st_legacyaswkb(g) AS wkb").collect() - geom = wkb.loads(bytes(out[0]["wkb"])) - assert len(geom.interiors) == 1 - assert abs(geom.area - 96.0) < 1e-9 -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/test_legacy_udf.py --log legacy-udf.log` -Expected: FAIL (`gbx_st_legacyaswkb` not registered). - -- [ ] **Step 3: Add the env guard** - -In `python/geobrix/src/databricks/labs/gbx/pyvx/_env.py`, add after `assert_mvt_available`: - -```python -def assert_tin_available() -> None: - """Raise a clear ImportError if the TIN/legacy light deps are missing.""" - missing = [] - try: - import scipy # noqa: F401 - except Exception: # noqa: BLE001 - missing.append("scipy") - try: - import shapely # noqa: F401 - except Exception: # noqa: BLE001 - missing.append("shapely") - if missing: - raise ImportError( - "pyvx TIN/legacy requires the [light] extra; missing: " - + ", ".join(missing) - + ". Install with: pip install 'geobrix[light]'" - ) -``` - -- [ ] **Step 4: Implement the UDF + register + wrapper in `functions.py`** - -Add the import (top, alongside `from . import _env, _mvt`): change to `from . import _env, _mvt, _legacy`. - -Add the UDF near `_asmvt_udf`: - -```python -def _legacyaswkb_impl(geom): - """Scalar: decode a legacy Mosaic struct row to ISO WKB (Z + holes).""" - return _legacy.legacy_to_wkb(geom) -``` - -In `register(spark)`, after the MVT registrations add: - -```python - _env.assert_tin_available() - spark.udf.register("gbx_st_legacyaswkb", _legacyaswkb_impl, BinaryType()) -``` - -Add the Column wrapper (bottom of file, mirroring `st_asmvt`): - -```python -def st_legacyaswkb(geom: ColLike) -> Column: - """Decode a legacy Mosaic geometry struct to ISO WKB (Z + holes preserved).""" - return f.call_function("gbx_st_legacyaswkb", _col(geom)) -``` - -- [ ] **Step 5: Run to verify it passes** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/test_legacy_udf.py --log legacy-udf.log` -Expected: PASS. - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyvx/functions.py python/geobrix/src/databricks/labs/gbx/pyvx/_env.py python/geobrix/test/pyvx/test_legacy_udf.py -git commit -m "feat(pyvx): register gbx_st_legacyaswkb scalar UDF - -Decode legacy Mosaic geometry struct to ISO WKB (Z + holes preserved), -Serverless-safe (spark.udf.register only). Adds assert_tin_available guard." -``` - -## Task 3: Heavy legacy fixes — preserve holes + Z - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/vectorx/jts/legacy/InternalGeometry.scala:21-39` -- Modify: `src/main/scala/com/databricks/labs/gbx/vectorx/jts/legacy/expressions/ST_LegacyAsWKB.scala:33` -- Test: `src/test/scala/com/databricks/labs/gbx/vectorx/jts/legacy/InternalGeometryTest.scala` (create if absent; otherwise add cases) - -- [ ] **Step 1: Write the failing Scala test** - -Add a test asserting a POLYGON with one hole round-trips through `InternalGeometry(row).toJTS` with `getNumInteriorRing == 1`, and that `ST_LegacyAsWKB.eval` on a Z-valued POINT yields WKB whose `WKBReader` reads a 3-D coordinate (`!coord.getZ.isNaN`). - -```scala -// InternalGeometryTest.scala (sketch — use the project's WithExpressionInfo test base + JTS reads) -test("toJTS preserves polygon holes") { - val ig = InternalGeometry(/* polygon InternalRow with one hole */) - val poly = ig.toJTS.asInstanceOf[org.locationtech.jts.geom.Polygon] - assert(poly.getNumInteriorRing == 1) -} -test("ST_LegacyAsWKB preserves Z") { - val wkb = ST_LegacyAsWKB.eval(/* POINT Z InternalRow */) - val g = new org.locationtech.jts.io.WKBReader().read(wkb) - assert(!g.getCoordinate.getZ.isNaN) -} -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `bash scripts/commands/gbx-test-scala.sh --suites 'com.databricks.labs.gbx.vectorx.jts.legacy.InternalGeometryTest' --log legacy-scala.log` -Expected: FAIL (holes dropped → `getNumInteriorRing == 0`; Z dropped → `getZ.isNaN`). - -- [ ] **Step 3: Fix `InternalGeometry.toJTS` (holes) — replace the POLYGON/MULTIPOLYGON branches** - -```scala - case GeometryTypeEnum.POLYGON => - val shell = boundaries.head.map(c => c.toCoordinate) - val rings = if (holes.nonEmpty) holes.head.map(_.map(_.toCoordinate)) else Seq.empty - JTS.polygonWithHoles(shell, rings) - case GeometryTypeEnum.MULTIPOLYGON => - val polys = boundaries.indices.map { i => - val shell = boundaries(i).map(c => c.toCoordinate) - val rings = if (i < holes.length && holes(i).nonEmpty) holes(i).map(_.map(_.toCoordinate)) else Seq.empty - JTS.polygonWithHoles(shell, rings) - } - JTS.multiPolygon(polys) -``` - -If `JTS.polygonWithHoles` / `JTS.multiPolygon(Seq[Polygon])` helpers do not exist, add them to `JTS.scala` (use `geometryFactory.createPolygon(shell, holes.toArray)` / `createMultiPolygon`). Preserve the existing helper-naming style in `JTS.scala`. - -- [ ] **Step 4: Fix `ST_LegacyAsWKB.eval` (Z) — `src/.../ST_LegacyAsWKB.scala:33`** - -```scala - JTS.toWKB3(geom) -``` -(was `JTS.toWKB(geom)`.) - -- [ ] **Step 5: Run to verify it passes** - -Run: `bash scripts/commands/gbx-test-scala.sh --suites 'com.databricks.labs.gbx.vectorx.jts.legacy.InternalGeometryTest' --log legacy-scala.log` -Expected: PASS. - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add src/main/scala/com/databricks/labs/gbx/vectorx/jts/legacy/ src/test/scala/com/databricks/labs/gbx/vectorx/jts/legacy/ -git commit -m "fix(vectorx): legacy decode preserves holes + Z - -InternalGeometry.toJTS now keeps polygon/multipolygon interior rings (was a -TODO that dropped them); ST_LegacyAsWKB emits toWKB3 so Z survives. Aligns -heavy with the new pyvx light st_legacyaswkb." -``` - -## Task 4: Cross-tier legacy parity + bindings + function-info - -**Files:** -- Test: `python/geobrix/test/pyvx/test_parity_legacy.py` -- Modify: `docs/tests/python/api/vectorx_functions_sql.py` (the `st_legacyaswkb_sql_example_output`) - -- [ ] **Step 1: Write the JAR-gated parity test** (copy the gating block from `test_parity_mvt.py` verbatim — `pytestmark = pytest.mark.integration`, the `_JARS` glob on `parents[2] / "lib"`, the `spark_with_jar` fixture with the active-session skip). - -```python -def test_legacy_parity_polygon_with_hole_and_z(spark_with_jar): - spark = spark_with_jar - from databricks.labs.gbx.pyvx import functions as vx - from databricks.labs.gbx.vectorx.jts.legacy import functions as hx - vx.register(spark) - hx.register(spark) - outer = [[0.0, 0.0, 1.0], [10.0, 0.0, 1.0], [10.0, 10.0, 1.0], [0.0, 10.0, 1.0], [0.0, 0.0, 1.0]] - hole = [[2.0, 2.0, 1.0], [4.0, 2.0, 1.0], [4.0, 4.0, 1.0], [2.0, 4.0, 1.0], [2.0, 2.0, 1.0]] - schema = "g struct>>,holes:array>>>>" - df = spark.createDataFrame([({"typeId": 5, "srid": 0, "boundaries": [outer], "holes": [[hole]]},)], schema) - light = bytes(df.selectExpr("gbx_st_legacyaswkb(g) AS w").collect()[0]["w"]) - heavy = bytes(df.selectExpr("gbx_st_legacyaswkb(g) AS w").collect()[0]["w"]) # heavy reg overwrote? -> register separately - from shapely import wkb - lg, hg = wkb.loads(light), wkb.loads(heavy) - assert lg.equals(hg) - assert len(lg.interiors) == 1 and len(hg.interiors) == 1 -``` -Note: because both tiers register the *same* SQL name `gbx_st_legacyaswkb`, register and collect the light result first, then register heavy and collect — or compare the light WKB against shapely-decoded heavy WKB obtained in a separate session. Keep the assertion: decoded geometries equal, both retain the hole, both retain Z (`lg.has_z`). - -- [ ] **Step 2: Run in Docker (JAR present)** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/test_parity_legacy.py --with-integration --log parity-legacy.log` -Expected: PASS (skips only if no JAR staged). - -- [ ] **Step 3: Update the function-info example output** (`docs/tests/python/api/vectorx_functions_sql.py`) so `st_legacyaswkb_sql_example_output` shows a `[BINARY]` table (add it if absent, canonically aligned per the D5 checker — `+` borders sized to max cell width). - -- [ ] **Step 4: Add the pyvx binding to the canonical list check** - -Run `bash scripts/commands/gbx-test-bindings.sh --log bindings-legacy.log`. `gbx_st_legacyaswkb` already exists in `registered_functions.txt`, Scala, heavy Python, and `function-info.json`; this confirms the light binding doesn't break parity. Fix any failure upstream (not by editing the canonical list). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/pyvx/test_parity_legacy.py docs/tests/python/api/vectorx_functions_sql.py -git commit -m "test(pyvx): light-vs-heavy legacy parity (holes + Z) - -Cross-tier decoded-geometry equality for gbx_st_legacyaswkb incl. a holed, -Z-valued polygon; JAR-gated. Refresh the function-info example output." -``` - ---- - -# PHASE 2 — Light TIN block - -Pure-Python `_tin.py` core (TDD the Sloan recovery hardest), then the 3 `@udtf` functions with the `mode` param. - -## Task 5: `_tin.py` — unconstrained triangulation + vertex merge - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyvx/_tin.py` -- Test: `python/geobrix/test/pyvx/test_tin_core.py` - -- [ ] **Step 1: Write the failing tests** - -```python -# test/pyvx/test_tin_core.py -import numpy as np -import pytest - -pytest.importorskip("scipy") -from databricks.labs.gbx.pyvx import _tin - - -def test_triangulate_square_gives_two_triangles(): - pts = np.array([[0.0, 0.0, 0.0], [1.0, 0.0, 0.0], [1.0, 1.0, 0.0], [0.0, 1.0, 0.0]]) - tris = _tin.triangulate(pts, breaklines=[], merge_tolerance=0.0, snap_tolerance=0.0) - assert len(tris) == 2 # two triangles cover the unit square - # each triangle is a (3,3) array of XYZ vertices - assert all(t.shape == (3, 3) for t in tris) - - -def test_merge_tolerance_dedups_near_coincident(): - pts = np.array([[0, 0, 0], [1, 0, 0], [1, 1, 0], [0, 1, 0], [1e-9, 1e-9, 0]], dtype=float) - tris = _tin.triangulate(pts, breaklines=[], merge_tolerance=1e-6, snap_tolerance=0.0) - assert len(tris) == 2 # the near-duplicate vertex is merged away - - -def test_empty_or_too_few_points(): - assert _tin.triangulate(np.zeros((0, 3)), [], 0.0, 0.0) == [] - assert _tin.triangulate(np.array([[0.0, 0.0, 0.0], [1.0, 1.0, 0.0]]), [], 0.0, 0.0) == [] -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_tin_core.py -v` -Expected: FAIL (`_tin` missing). - -- [ ] **Step 3: Implement the unconstrained core + merge** - -```python -"""Pure-Python TIN engine for the pyvx light tier (Serverless-safe). - -scipy Delaunay + Sloan constraint recovery (constrained, no Steiner points), -Z-snap to breaklines, barycentric Z interpolation, and grid generators. -Heavy parity target: vectorx.jts.InterpolateElevation (mode="constrained"). -""" -from typing import List, Sequence, Tuple - -import numpy as np -from scipy.spatial import Delaunay - - -def _merge_vertices(pts: np.ndarray, tol: float) -> np.ndarray: - """Snap near-coincident XY vertices (within tol) to a single representative.""" - if tol <= 0.0 or len(pts) == 0: - return pts - keys = np.round(pts[:, :2] / tol).astype(np.int64) - _, idx = np.unique(keys, axis=0, return_index=True) - return pts[np.sort(idx)] - - -def triangulate( - points: np.ndarray, - breaklines: Sequence[np.ndarray], - merge_tolerance: float, - snap_tolerance: float, -) -> List[np.ndarray]: - """Constrained Delaunay over XYZ points. Returns a list of (3,3) XYZ triangles. - - breaklines: sequence of (N,2|3) constraint polylines whose segments are forced - as triangle edges (Sloan recovery). Empty -> plain Delaunay. - """ - pts = _merge_vertices(np.asarray(points, dtype=float), merge_tolerance) - if len(pts) < 3: - return [] - tri = Delaunay(pts[:, :2]) - simplices = tri.simplices.copy() - if breaklines: - simplices = _recover_constraints(pts[:, :2], simplices, tri, breaklines) - z = pts[:, 2] - out = [np.column_stack([pts[s, 0], pts[s, 1], z[s]]) for s in simplices] - if snap_tolerance > 0.0 and breaklines: - out = _zsnap(out, breaklines, snap_tolerance) - return out -``` - -(`_recover_constraints` and `_zsnap` are defined in Task 6 — for this task, guard the `if breaklines:` branch behind a `raise NotImplementedError` placeholder is NOT allowed; instead implement Task 5 with `breaklines=[]` only paths exercised, and define `_recover_constraints`/`_zsnap` as the real functions in Task 6. To keep Task 5 self-contained and green, temporarily define them as identity/no-op stubs *with a test that only uses empty breaklines*, then replace in Task 6.) - -To keep Task 5 green without breakline tests, add minimal definitions: - -```python -def _recover_constraints(xy, simplices, tri, breaklines): - return simplices # replaced with real Sloan recovery in the next task - - -def _zsnap(triangles, breaklines, tol): - return triangles # replaced in the next task -``` - -- [ ] **Step 4: Run to verify they pass** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_tin_core.py -v` -Expected: PASS (3 tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyvx/_tin.py python/geobrix/test/pyvx/test_tin_core.py -git commit -m "feat(pyvx): TIN unconstrained Delaunay + vertex merge core - -scipy Delaunay triangulation with mergeTolerance vertex dedup; constraint -recovery + Z-snap are stubbed pending the next task." -``` - -## Task 6: `_tin.py` — Sloan constraint recovery + Z-snap (the hard core) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyvx/_tin.py` -- Test: `python/geobrix/test/pyvx/test_tin_core.py` - -- [ ] **Step 1: Write the failing tests** - -```python -def test_breakline_appears_as_triangle_edges(): - # square + center diagonal breakline that the plain Delaunay might not use - pts = np.array([[0,0,0],[4,0,0],[4,4,0],[0,4,0],[1,3,0],[3,1,0]], dtype=float) - bl = [np.array([[1.0, 3.0], [3.0, 1.0]])] - tris = _tin.triangulate(pts, bl, 0.0, 0.0) - edges = set() - for t in tris: - xy = [tuple(np.round(p[:2], 6)) for p in t] - for a, b in [(0, 1), (1, 2), (2, 0)]: - edges.add(frozenset([xy[a], xy[b]])) - assert frozenset([(1.0, 3.0), (3.0, 1.0)]) in edges - - -def test_recovery_terminates_on_dense_constraints(): - rng = np.random.default_rng(0) - pts = np.column_stack([rng.random(40), rng.random(40), np.zeros(40)]) - bl = [np.array([[0.05, 0.05], [0.95, 0.95]])] - tris = _tin.triangulate(pts, bl, 0.0, 0.0) # must not hang - assert len(tris) > 0 - - -def test_zsnap_sets_vertex_z_along_constraint(): - # flat points at z=0, a breakline carrying z=10 at its endpoints - pts = np.array([[0,0,0],[4,0,0],[4,4,0],[0,4,0]], dtype=float) - bl = [np.array([[0.0, 2.0, 10.0], [4.0, 2.0, 10.0]])] - tris = _tin.triangulate(pts, bl, 0.0, 1e-6) - # any vertex landing on the breakline keeps z≈10 (within snap) - near = [p[2] for t in tris for p in t if abs(p[1] - 2.0) < 1e-9] - # at minimum the recovery + snap must not crash and must yield triangles - assert len(tris) > 0 -``` - -- [ ] **Step 2: Run to verify the breakline test fails** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_tin_core.py::test_breakline_appears_as_triangle_edges -v` -Expected: FAIL (stub returns plain Delaunay; diagonal not guaranteed an edge). - -- [ ] **Step 3: Replace the stubs with the real Sloan recovery + Z-snap** - -```python -def _orient2d(a, b, c) -> float: - """>0 if a->b->c is counter-clockwise.""" - return (b[0] - a[0]) * (c[1] - a[1]) - (b[1] - a[1]) * (c[0] - a[0]) - - -def _segments_intersect(p1, p2, p3, p4) -> bool: - d1 = _orient2d(p3, p4, p1); d2 = _orient2d(p3, p4, p2) - d3 = _orient2d(p1, p2, p3); d4 = _orient2d(p1, p2, p4) - return ((d1 > 0) != (d2 > 0)) and ((d3 > 0) != (d4 > 0)) - - -def _vertex_index(xy: np.ndarray, p, tol=1e-9) -> int: - d = np.hypot(xy[:, 0] - p[0], xy[:, 1] - p[1]) - i = int(np.argmin(d)) - return i if d[i] <= max(tol, 1e-9) else -1 - - -def _recover_constraints(xy: np.ndarray, simplices: np.ndarray, tri, breaklines): - """Sloan constraint recovery: force each breakline segment to be an edge. - - Operates on a mutable triangle list with an edge->triangles adjacency map, - flipping intersected diagonals (only across convex quads) until each - constraint segment is present as a triangle edge. Bounded flip budget. - """ - triangles = [list(s) for s in simplices.tolist()] - - def edges_of(t): - return [frozenset((t[0], t[1])), frozenset((t[1], t[2])), frozenset((t[2], t[0]))] - - def build_adj(): - adj = {} - for ti, t in enumerate(triangles): - for e in edges_of(t): - adj.setdefault(e, []).append(ti) - return adj - - for bl in breaklines: - seg_pts = np.asarray(bl, dtype=float) - for k in range(len(seg_pts) - 1): - ia = _vertex_index(xy, seg_pts[k]); ib = _vertex_index(xy, seg_pts[k + 1]) - if ia < 0 or ib < 0 or ia == ib: - continue - target = frozenset((ia, ib)) - budget = 50 * len(triangles) + 100 - while target not in build_adj() and budget > 0: - budget -= 1 - adj = build_adj() - flipped = False - for e, ts in adj.items(): - if len(ts) != 2: - continue - (u, v) = tuple(e) - if not _segments_intersect(xy[ia], xy[ib], xy[u], xy[v]): - continue - t0, t1 = triangles[ts[0]], triangles[ts[1]] - w0 = next(x for x in t0 if x not in e) - w1 = next(x for x in t1 if x not in e) - # convex quad test: diagonal (w0,w1) must cross (u,v) - if not _segments_intersect(xy[w0], xy[w1], xy[u], xy[v]): - continue - triangles[ts[0]] = [w0, w1, u] - triangles[ts[1]] = [w0, w1, v] - flipped = True - break - if not flipped: - break # cannot recover this segment with convex flips; leave as-is - if budget <= 0: - raise RuntimeError("Sloan constraint recovery did not terminate") - return np.array(triangles, dtype=np.int64) - - -def _zsnap(triangles, breaklines, tol): - """Overwrite vertex Z with linear interpolation along any constraint line - within tol (mirrors heavy LengthIndexedLine post-process).""" - snapped = [] - for t in triangles: - t = t.copy() - for vi in range(3): - p = t[vi] - for bl in breaklines: - bl = np.asarray(bl, dtype=float) - if bl.shape[1] < 3: - continue - for k in range(len(bl) - 1): - a, b = bl[k], bl[k + 1] - ab = b[:2] - a[:2] - L2 = float(ab @ ab) - if L2 == 0.0: - continue - s = float((p[:2] - a[:2]) @ ab) / L2 - if 0.0 <= s <= 1.0: - proj = a[:2] + s * ab - if np.hypot(*(p[:2] - proj)) <= tol: - t[vi, 2] = a[2] + s * (b[2] - a[2]) - snapped.append(None) # placeholder; replaced below - snapped[-1] = t - # rebuild cleanly (one entry per triangle) - return [t for t in (snapped[i] for i in range(len(triangles)))] -``` - -Note for the implementer: simplify the `_zsnap` accumulation to one appended `t` per triangle (the sketch's placeholder bookkeeping is illustrative — append each finished `t` once). Keep the projection-onto-segment math exactly. - -- [ ] **Step 4: Run to verify they pass** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_tin_core.py -v` -Expected: PASS (all 6 tests, incl. breakline-edge + termination + z-snap). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyvx/_tin.py python/geobrix/test/pyvx/test_tin_core.py -git commit -m "feat(pyvx): Sloan constraint recovery + breakline Z-snap - -Force breakline segments as triangle edges via convex-quad edge flips (bounded -budget, raises on non-termination); snap vertex Z onto constraint lines within -snapTolerance. Constrained (no Steiner) Delaunay." -``` - -## Task 7: `_tin.py` — barycentric interpolation + grid generators - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyvx/_tin.py` -- Test: `python/geobrix/test/pyvx/test_tin_core.py` - -- [ ] **Step 1: Write the failing tests** - -```python -def test_grid_bbox_centers_column_major(): - cells = list(_tin.grid_bbox(0.0, 0.0, 2.0, 2.0, 2, 2)) - # column-major: i (x) slowest, j (y) fastest; centers at 0.5/1.5 - assert cells == [(0.5, 0.5), (0.5, 1.5), (1.5, 0.5), (1.5, 1.5)] - - -def test_grid_geom_negative_celly(): - cells = list(_tin.grid_geom(0.0, 10.0, 2, 2, 5.0, -5.0)) - assert cells == [(2.5, 7.5), (2.5, 2.5), (7.5, 7.5), (7.5, 2.5)] - - -def test_interpolate_known_plane_and_outside_hull(): - # plane z = x + y over unit square - pts = np.array([[0,0,0],[1,0,1],[1,1,2],[0,1,1]], dtype=float) - tris = _tin.triangulate(pts, [], 0.0, 0.0) - z = _tin.interpolate_z(tris, 0.5, 0.5) - assert abs(z - 1.0) < 1e-9 - assert _tin.interpolate_z(tris, 5.0, 5.0) is None # outside hull -> None (dropped) -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_tin_core.py -k "grid or interpolate" -v` -Expected: FAIL (`grid_bbox`/`grid_geom`/`interpolate_z` missing). - -- [ ] **Step 3: Implement** - -```python -def grid_bbox(xmin, ymin, xmax, ymax, width_px, height_px): - """Yield (x, y) cell centers, column-major (matches heavy pointGridBBox).""" - xres = (xmax - xmin) / width_px - yres = (ymax - ymin) / height_px - for i in range(int(width_px)): - for j in range(int(height_px)): - yield (xmin + (i + 0.5) * xres, ymin + (j + 0.5) * yres) - - -def grid_geom(origin_x, origin_y, cols, rows, cell_x, cell_y): - """Yield (x, y) cell centers from origin + cell sizes (matches pointGridOrigin). - cell_y may be negative (y-down).""" - for i in range(int(cols)): - for j in range(int(rows)): - yield (origin_x + (i + 0.5) * cell_x, origin_y + (j + 0.5) * cell_y) - - -def interpolate_z(triangles: List[np.ndarray], x: float, y: float): - """Barycentric Z at (x,y) within the TIN. None if outside all triangles.""" - p = np.array([x, y]) - for t in triangles: - a, b, c = t[0, :2], t[1, :2], t[2, :2] - d = _orient2d(a, b, c) - if d == 0.0: - continue - l1 = _orient2d(p, b, c) / d - l2 = _orient2d(a, p, c) / d - l3 = 1.0 - l1 - l2 - if l1 >= -1e-12 and l2 >= -1e-12 and l3 >= -1e-12: - z = l1 * t[0, 2] + l2 * t[1, 2] + l3 * t[2, 2] - return None if np.isnan(z) else float(z) - return None -``` - -- [ ] **Step 4: Run to verify they pass** - -Run: `.venv-pyrx/bin/python -m pytest python/geobrix/test/pyvx/test_tin_core.py -v` -Expected: PASS (all `_tin` tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyvx/_tin.py python/geobrix/test/pyvx/test_tin_core.py -git commit -m "feat(pyvx): TIN barycentric interpolation + grid generators - -Column-major bbox/geom grid centers (matches heavy pointGrid*); barycentric Z -with outside-hull -> None (heavy's silent drop). Negative cell_y supported." -``` - -## Task 8: Light `st_triangulate` UDTF + `mode` - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/pyvx/_serde.py` (add `TRIANGLE_SCHEMA`, `ELEVATION_SCHEMA`) -- Test: `python/geobrix/test/pyvx/test_tin_udtf.py` - -- [ ] **Step 1: Add output schemas to `_serde.py`** - -```python -TRIANGLE_SCHEMA = StructType([StructField("triangle", BinaryType(), False)]) -ELEVATION_SCHEMA = StructType([StructField("elevation_point", BinaryType(), False)]) -``` - -- [ ] **Step 2: Write the failing test** - -```python -# test/pyvx/test_tin_udtf.py -import pytest - -pytest.importorskip("scipy") -shapely = pytest.importorskip("shapely") -from shapely import to_wkb, wkb # noqa: E402 -from shapely.geometry import Point # noqa: E402 - -from databricks.labs.gbx.pyvx import functions as vx - - -def _pts_wkb(coords): - return [bytearray(to_wkb(Point(*c))) for c in coords] - - -def test_st_triangulate_emits_triangles(spark): - vx.register(spark) - pts = _pts_wkb([(0, 0, 0), (1, 0, 0), (1, 1, 0), (0, 1, 0)]) - df = spark.createDataFrame([(pts, [], 0.0, 0.0, "NONENCROACHING")], - "pts array, bl array, mt double, st double, spf string") - rows = df.selectExpr( - "t.* FROM {df} JOIN LATERAL gbx_st_triangulate(pts, bl, mt, st, spf, 'constrained') t" - ) if False else spark.sql( - "SELECT t.triangle FROM v, LATERAL gbx_st_triangulate(pts, bl, mt, st, spf, 'constrained') t" - ) - df.createOrReplaceTempView("v") - rows = spark.sql("SELECT t.triangle FROM v, LATERAL gbx_st_triangulate(pts, bl, mt, st, spf, 'constrained') t").collect() - assert len(rows) == 2 - assert all(wkb.loads(bytes(r["triangle"])).geom_type == "Polygon" for r in rows) - - -def test_st_triangulate_conforming_raises(spark): - vx.register(spark) - pts = _pts_wkb([(0, 0, 0), (1, 0, 0), (1, 1, 0)]) - df = spark.createDataFrame([(pts, [], 0.0, 0.0, "MIDPOINT")], - "pts array, bl array, mt double, st double, spf string") - df.createOrReplaceTempView("v2") - with pytest.raises(Exception, match="conforming"): - spark.sql("SELECT t.* FROM v2, LATERAL gbx_st_triangulate(pts, bl, mt, st, spf, 'conforming') t").collect() -``` - -- [ ] **Step 3: Run to verify it fails** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/test_tin_udtf.py --log tin-udtf.log` -Expected: FAIL (UDTF not registered). - -- [ ] **Step 4: Implement the UDTF + helpers + register** - -In `functions.py`, add a shared geometry-array decoder and the UDTF: - -```python -def _geoms_from_array(arr): - """Decode an ARRAY of geometries via the shared parse_geom - contract (WKB/EWKB/WKT/EWKT).""" - from ._geom import parse_geom - out = [] - for g in arr or []: - geom = parse_geom(g) - if geom is not None: - out.append(geom) - return out - - -def _validate_mode(mode): - m = (mode or "constrained").lower() - if m == "conforming": - raise NotImplementedError( - "mode='conforming' (Steiner-point conforming Delaunay) is heavy-only; " - "use the heavyweight vectorx tier, or mode='constrained' in light." - ) - if m != "constrained": - raise ValueError(f"mode must be 'constrained' or 'conforming'; got {mode!r}") - return m - - -def _triangulate_schema(): - from ._serde import TRIANGLE_SCHEMA - return TRIANGLE_SCHEMA - - -@udtf(returnType=_triangulate_schema()) -class _TriangulateUDTF: - def eval(self, points, breaklines, merge_tolerance, snap_tolerance, split_point_finder, mode=None): - _validate_mode(mode) - from shapely import to_wkb - from shapely.geometry import Polygon - import numpy as np - pt_geoms = _geoms_from_array(points) - if not pt_geoms: - return - coords = np.array([[*(c if len(c) == 3 else (c[0], c[1], 0.0))] - for g in pt_geoms for c in g.coords], dtype=float) - bls = [np.array(g.coords, dtype=float) for g in _geoms_from_array(breaklines)] - for t in _tin.triangulate(coords, bls, float(merge_tolerance), float(snap_tolerance)): - yield (to_wkb(Polygon([(p[0], p[1]) for p in t])),) # 2D triangle WKB -``` - -Add `from . import _env, _mvt, _legacy, _tin` at the top. In `register(spark)` add: - -```python - spark.udtf.register("gbx_st_triangulate", _TriangulateUDTF) -``` - -Add the Python Column wrapper (SQL-LATERAL-only, like the pyramid): - -```python -def st_triangulate(points_geom, breaklines_geom, merge_tolerance, snap_tolerance, - split_point_finder, mode: ColLike = "constrained"): - """Triangulate mass points (constrained Delaunay). Invoke via SQL LATERAL: - SELECT t.* FROM , LATERAL gbx_st_triangulate(points, breaklines, mt, st, spf, mode) t - mode='conforming' is heavy-only.""" - raise NotImplementedError( - "Light st_triangulate has no Python Column form; invoke the registered UDTF via SQL LATERAL." - ) -``` - -- [ ] **Step 5: Run to verify it passes** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/test_tin_udtf.py --log tin-udtf.log` -Expected: PASS (2 tests). - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyvx/functions.py python/geobrix/src/databricks/labs/gbx/pyvx/_serde.py python/geobrix/test/pyvx/test_tin_udtf.py -git commit -m "feat(pyvx): gbx_st_triangulate UDTF (constrained mode) - -Streaming UDTF emitting one 2D-WKB triangle per row; mode='constrained' -(default) via scipy+Sloan, mode='conforming' raises (heavy-only). SQL LATERAL." -``` - -## Task 9: Light `st_interpolateelevationbbox` + `st_interpolateelevationgeom` UDTFs - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py` -- Test: `python/geobrix/test/pyvx/test_tin_udtf.py` - -- [ ] **Step 1: Write the failing tests** - -```python -def test_interpolateelevationbbox_emits_points(spark): - vx.register(spark) - pts = _pts_wkb([(0, 0, 0), (10, 0, 10), (10, 10, 20), (0, 10, 10)]) - df = spark.createDataFrame( - [(pts, [], 0.0, 0.0, "NONENCROACHING", 0.0, 0.0, 10.0, 10.0, 5, 5, 0)], - "pts array, bl array, mt double, st double, spf string, " - "xmin double, ymin double, xmax double, ymax double, w int, h int, srid int") - df.createOrReplaceTempView("vb") - rows = spark.sql("SELECT t.elevation_point FROM vb, LATERAL " - "gbx_st_interpolateelevationbbox(pts, bl, mt, st, spf, xmin, ymin, xmax, ymax, w, h, srid, 'constrained') t").collect() - assert len(rows) == 25 # 5x5 grid, all inside hull - from shapely import wkb - assert wkb.loads(bytes(rows[0]["elevation_point"])).has_z - - -def test_interpolateelevationgeom_emits_points(spark): - vx.register(spark) - pts = _pts_wkb([(0, 0, 0), (10, 0, 10), (10, 10, 20), (0, 10, 10)]) - origin = bytearray(to_wkb(Point(0.0, 10.0))) - df = spark.createDataFrame( - [(pts, [], 0.0, 0.0, "NONENCROACHING", origin, 5, 5, 2.0, -2.0)], - "pts array, bl array, mt double, st double, spf string, " - "origin binary, cols int, rows int, cx double, cy double") - df.createOrReplaceTempView("vg") - rows = spark.sql("SELECT t.elevation_point FROM vg, LATERAL " - "gbx_st_interpolateelevationgeom(pts, bl, mt, st, spf, origin, cols, rows, cx, cy, 'constrained') t").collect() - assert len(rows) == 25 -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/test_tin_udtf.py --log tin-udtf.log` -Expected: FAIL (UDTFs not registered). - -- [ ] **Step 3: Implement both UDTFs + register** - -```python -def _elevation_schema(): - from ._serde import ELEVATION_SCHEMA - return ELEVATION_SCHEMA - - -def _emit_elevation(points, breaklines, mt, st, spf, mode, cell_iter, srid): - _validate_mode(mode) - from shapely import to_wkb - from shapely.geometry import Point - from shapely import set_srid - import numpy as np - pt_geoms = _geoms_from_array(points) - if not pt_geoms: - return - coords = np.array([[*(c if len(c) == 3 else (c[0], c[1], 0.0))] - for g in pt_geoms for c in g.coords], dtype=float) - bls = [np.array(g.coords, dtype=float) for g in _geoms_from_array(breaklines)] - tris = _tin.triangulate(coords, bls, float(mt), float(st)) - for (x, y) in cell_iter: - z = _tin.interpolate_z(tris, x, y) - if z is None: - continue - p = Point(x, y, z) - if srid: - p = set_srid(p, int(srid)) - yield (to_wkb(p, output_dimension=3),) # POINT Z - - -@udtf(returnType=_elevation_schema()) -class _InterpElevBBoxUDTF: - def eval(self, points, breaklines, merge_tolerance, snap_tolerance, split_point_finder, - xmin, ymin, xmax, ymax, width_px, height_px, srid, mode=None): - yield from _emit_elevation( - points, breaklines, merge_tolerance, snap_tolerance, split_point_finder, mode, - _tin.grid_bbox(float(xmin), float(ymin), float(xmax), float(ymax), int(width_px), int(height_px)), - int(srid), - ) - - -@udtf(returnType=_elevation_schema()) -class _InterpElevGeomUDTF: - def eval(self, points, breaklines, merge_tolerance, snap_tolerance, split_point_finder, - grid_origin, grid_cols, grid_rows, cell_size_x, cell_size_y, mode=None): - from shapely import get_srid - from ._geom import parse_geom - og = parse_geom(grid_origin) # WKB/EWKB/WKT/EWKT - ox, oy = (og.x, og.y) if og is not None else (0.0, 0.0) - srid = get_srid(og) if og is not None else 0 - yield from _emit_elevation( - points, breaklines, merge_tolerance, snap_tolerance, split_point_finder, mode, - _tin.grid_geom(ox, oy, int(grid_cols), int(grid_rows), float(cell_size_x), float(cell_size_y)), - int(srid), - ) -``` - -In `register(spark)` add: - -```python - spark.udtf.register("gbx_st_interpolateelevationbbox", _InterpElevBBoxUDTF) - spark.udtf.register("gbx_st_interpolateelevationgeom", _InterpElevGeomUDTF) -``` - -Add SQL-LATERAL-only Column wrappers `st_interpolateelevationbbox(...)` and `st_interpolateelevationgeom(...)` raising `NotImplementedError` (same pattern as `st_triangulate`), with the full positional signatures + trailing `mode="constrained"`. - -- [ ] **Step 4: Run to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/test_tin_udtf.py --log tin-udtf.log` -Expected: PASS (4 tests). - -- [ ] **Step 5: Run the full pyvx suite + Serverless guard** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/ --log pyvx-all.log` -Expected: PASS (MVT + legacy + TIN units; parity tests skip without JAR). `test_serverless_no_spark_config.py` green. - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyvx/functions.py python/geobrix/test/pyvx/test_tin_udtf.py -git commit -m "feat(pyvx): interpolateelevation bbox + geom UDTFs - -Barycentric Z over the constrained TIN at column-major grid centers; POINT Z -WKB out, outside-hull cells dropped. SRID from param (bbox) / origin (geom)." -``` - ---- - -# PHASE 3 — Heavy `mode` alignment + cross-tier parity + docs - -## Task 10: Heavy `mode` param + constrained path (Scala) - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/vectorx/jts/InterpolateElevation.scala` -- Modify: `src/.../expressions/ST_Triangulate.scala`, `ST_InterpolateElevationBBox.scala`, `ST_InterpolateElevationGeom.scala` -- Test: `src/test/scala/com/databricks/labs/gbx/vectorx/expressions/ST_TriangulateTest.scala` (+ existing TIN suites) - -This is a **cross-language port** of the Python Sloan core (Tasks 5–7) to Scala. The Python in `_tin.py` is the algorithm reference; reproduce its `triangulate` + `_recover_constraints` + `_zsnap` semantics in JTS. - -- [ ] **Step 1: Add a `mode` to `InterpolateElevation.triangulate` + a constrained path** - -Extend the signature with `mode: String = "conforming"` and branch: - -```scala -def triangulate( - multiPoint: Geometry, - breaklines: Seq[Geometry], - mergeTolerance: Double, - snapTolerance: Double, - splitPointFinder: Option[TriangulationSplitPointTypeEnum.Value] = None, - mode: String = "constrained" -): Seq[Geometry] = mode.toLowerCase match { - case "conforming" => triangulateConforming(multiPoint, breaklines, mergeTolerance, snapTolerance, splitPointFinder) - case "constrained" => triangulateConstrained(multiPoint, breaklines, mergeTolerance, snapTolerance) - case other => throw new IllegalArgumentException( - s"mode must be 'constrained' or 'conforming'; got '$other'") -} -``` - -`triangulateConforming` = today's body (verbatim). `triangulateConstrained` = build the initial Delaunay via `DelaunayTriangulationBuilder` (no constraints), then port `_recover_constraints` (edge-flip recovery on a triangle-index structure) + `_zsnap`. Use JTS only for the initial triangulation, geometry construction, and the `LengthIndexedLine` Z-snap. - -- [ ] **Step 2: Add the `mode` arg to the three expressions (builder arity arms)** - -For `ST_Triangulate` (currently fixed arity 5), follow the `RST_H3_Tessellate` pattern: - -```scala -override def builder(): FunctionBuilder = (c: Seq[Expression]) => c.length match { - case 5 => ST_Triangulate(c(0), c(1), c(2), c(3), c(4), Literal("constrained")) - case 6 => ST_Triangulate(c(0), c(1), c(2), c(3), c(4), c(5)) - case n => throw new IllegalArgumentException( - s"gbx_st_triangulate takes 5 or 6 arguments (points, breaklines, mergeTol, snapTol, splitPointFinder, [mode]); got $n") -} -``` - -Add a `modeExpr: Expression` field to the case class; in `eval`, read it (`modeExpr.eval(...).toString`) and pass to `InterpolateElevation.triangulate(..., mode = modeStr)`. Do the same for `ST_InterpolateElevationBBox` (12→12/13) and `ST_InterpolateElevationGeom` (10→10/11). - -- [ ] **Step 3: Write + run the Scala tests** - -Add `ST_TriangulateTest`: same point set, `mode="constrained"` vs `"conforming"` both produce valid triangle covers; with a breakline, both contain the constraint segment as an edge; unknown mode throws. Run: - -`bash scripts/commands/gbx-test-scala.sh --suites 'com.databricks.labs.gbx.vectorx.expressions.ST_TriangulateTest,com.databricks.labs.gbx.vectorx.expressions.ST_InterpolateElevationBBoxTest,com.databricks.labs.gbx.vectorx.expressions.ST_InterpolateElevationGeomTest' --log tin-scala.log` -Expected: PASS. - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX .git/objects -git add src/main/scala/com/databricks/labs/gbx/vectorx/ src/test/scala/com/databricks/labs/gbx/vectorx/ -git commit -m "feat(vectorx): constrained/conforming mode on TIN functions - -Add a trailing mode arg (default 'constrained') to st_triangulate + -interpolateelevation{bbox,geom}; constrained (no Steiner) ports the pyvx -Sloan recovery to JTS, conforming keeps ConformingDelaunayTriangulator." -``` - -## Task 11: Cross-tier TIN parity tests - -**Files:** -- Test: `python/geobrix/test/pyvx/test_parity_tin.py` - -- [ ] **Step 1: Write the JAR-gated parity tests** (reuse the `spark_with_jar` fixture pattern). - -```python -def test_triangulate_parity_no_breaklines(spark_with_jar): - # Delaunay ~unique: light constrained == heavy constrained triangle set (within tol) - ... - # assert same number of triangles and matching sorted centroid coordinates within 1e-6 - -def test_interpolate_parity_surface_closeness(spark_with_jar): - # same points + bbox grid, mode='constrained' both tiers - # assert per-cell interpolated Z within 1e-6 (no breaklines) - -def test_triangulate_breakline_edges_present_both(spark_with_jar): - # with a breakline: assert the constraint segment is a triangle edge in BOTH tiers - # (NOT triangle-identity) - -def test_conforming_is_heavy_only(spark_with_jar): - # heavy mode='conforming' returns rows; light mode='conforming' raises -``` - -Assertion bar per the spec: **no-breakline → near-exact** (triangle set / surface within `1e-6`); **with-breakline → surface-closeness + constraint-edges-present**, not triangle-identity. - -- [ ] **Step 2: Run in Docker (JAR present)** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/test_parity_tin.py --with-integration --log parity-tin.log` -Expected: PASS (skips without JAR). - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/pyvx/test_parity_tin.py -git commit -m "test(pyvx): light-vs-heavy TIN parity (constrained mode) - -No-breakline near-exact surface/triangle parity; with-breakline asserts -constraint edges present + surface closeness, not triangle identity; conforming -is heavy-only." -``` - -## Task 12: Bindings, function-info, docs - -**Files:** -- Modify: `docs/tests/python/api/vectorx_functions_sql.py` (add `mode` to the 3 TIN examples) -- Modify: `docs/docs/api/vectorx-functions.mdx` -- Modify: `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py` (ensure all 4 wrappers exist for import-parity) - -- [ ] **Step 1: Update function-info examples** to show the `mode` arg, e.g. `gbx_st_triangulate(masspoints, breaklines, 0.01, 0.01, 'NONENCROACHING', 'constrained')`. Regenerate: `bash scripts/commands/gbx-docs-function-info.sh` (or `gbx:test:function-info`). Keep output tables canonically aligned (D5). - -- [ ] **Step 2: Update `vectorx-functions.mdx`** — add light/heavy tabs for the 4 functions, the `mode` param, and a "constrained vs conforming" + breakline divergence explainer (defensible-divergence framing, like the H3-Java 3.7.0 note). Legacy section: `st_legacyaswkb` preserves Z + holes; SRID applied separately at ingestion; M out of scope. No "wave N" / internal vocabulary. - -- [ ] **Step 3: Geometry-input consistency audit** - -Verify every `gbx_st_*` geom-accepting function shares the `parse_geom` contract (WKB/EWKB/WKT/EWKT): the TIN `points`/`breaklines`/`grid_origin` (done in T8/T9) **and the MVT functions `st_asmvt` + `st_asmvt_pyramid`** (both decode geom WKB today — `_asmvt_udf` and `_mvt.pyramid_tiles`/`_AsMvtPyramidUDTF`). Route their geom inputs through `_geom.parse_geom` (decode → the encoder re-`to_wkb`s as needed), and add tests that both `st_asmvt` and `st_asmvt_pyramid` accept a WKT geom (and EWKB). Verify against heavy `gbx_st_asmvt`/`gbx_st_asmvt_pyramid` accepted encodings. Document the accepted-encodings contract once on the VectorX page. - -- [ ] **Step 4: Build docs + checks** - -Run: `cd docs && npm run build` → SUCCESS. `grep -rn -iE "wave [0-9]+" docs/docs/` → empty. `bash scripts/commands/gbx-test-bindings.sh --log bindings-tin.log` → PASS. - -- [ ] **Step 5: Full Docker verification** - -Run the pyvx suite + the TIN/legacy Scala suites in Docker (mirror the MVT verification): light units green, parity green (JAR staged), Scala green. - -- [ ] **Step 6: Commit + update the PR checklist** - -```bash -chmod -R u+rwX .git/objects -git add docs/ -git commit -m "docs(pyvx): TIN + legacy functions, modes, divergence explainer - -Light/heavy tabs + mode param for the 4 VectorX functions; constrained-vs- -conforming + breakline divergence note; legacy Z+holes/SRID framing. Update -function-info examples." -``` - -Then check the 4 boxes in PR #38's description (legacy + the 3 TIN functions) and mark the PR ready for review. - ---- - -## Self-Review - -**Spec coverage:** ✅ Legacy Z+holes (T1–T4, both tiers); ✅ TIN engine scipy+Sloan (T5–T7); ✅ 3 UDTFs (T8–T9); ✅ `mode` both tiers, conforming heavy-only/light-raises (T8, T10); ✅ parity posture no-breakline-exact / breakline-surface (T11); ✅ scipy dep (already present — noted); ✅ Serverless-safe (T9 step 5 guard); ✅ docs + divergence explainer (T12); ✅ phasing legacy→TIN→heavy. - -**Placeholder scan:** The only soft spots are the deliberately-staged `_recover_constraints`/`_zsnap` stubs in T5 (replaced with real code in T6 — an intentional TDD sequencing, not a shipped placeholder) and the `_zsnap` accumulation note in T6 (flagged with the exact correction). The Scala constrained port in T10 references the Python core as the algorithm spec (a cross-language port, not a placeholder). - -**Type consistency:** schema names (`TRIANGLE_SCHEMA`, `ELEVATION_SCHEMA`, `TILE_SCHEMA`), function names (`triangulate`, `interpolate_z`, `grid_bbox`, `grid_geom`, `legacy_to_geom`, `legacy_to_wkb`, `_validate_mode`, `_geoms_from_array`), and SQL names (`gbx_st_*`) are consistent across tasks. diff --git a/docs/superpowers/plans/2026-06-14-pmtiles-agg-light-tier.md b/docs/superpowers/plans/2026-06-14-pmtiles-agg-light-tier.md deleted file mode 100644 index 5cab63c98..000000000 --- a/docs/superpowers/plans/2026-06-14-pmtiles-agg-light-tier.md +++ /dev/null @@ -1,832 +0,0 @@ -# Lightweight `gbx_pmtiles_agg` Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a lightweight, Serverless-safe implementation of `gbx_pmtiles_agg` — a grouped aggregate that folds a group of `(bytes, z, x, y)` map tiles into one PMTiles v3 archive (BINARY) — behind the same SQL name and Column-wrapper import as the heavy tier. - -**Architecture:** A tier-neutral `_agg_light.py` in the existing `databricks.labs.gbx.pmtiles` package holds a `pandas_udf` grouped aggregate that reuses the `ds.tiles` archive assembler (`SlippyGrid` + `build_header_info` + `pmtiles.writer.Writer`), writing to an in-memory `BytesIO`. A shared `register_pmtiles_agg(spark)` is wired into both `pyrx.register` and `pyvx.register` (PMTiles is format-agnostic — raster *or* vector tiles — so it belongs to neither raster nor vector). No new SQL name is introduced (`gbx_pmtiles_agg` already exists for heavy). Parity is decoded-tile parity (not byte-identical: heavy uses NONE internal-compression, light GZIP). - -**Tech Stack:** Python 3.12, PySpark `pandas_udf` (GROUPED_AGG), the `pmtiles` PyPI lib (`>=3.4,<4`, in the `[light]` extra), `ds.tiles` assembler, pytest (Docker for Spark fixtures, JAR-gated for cross-tier parity). - -**Spec:** `docs/superpowers/specs/2026-06-14-pmtiles-agg-light-tier-design.md` -**Branch:** `pygx-light` (add-on; does not "finish" the branch — pygx BNG Phase 2 follows separately). - ---- - -## File Structure - -| File | Responsibility | -|---|---| -| `python/geobrix/src/databricks/labs/gbx/pmtiles/_agg_light.py` (create) | `_assemble_archive(...)` BytesIO assembler, `_pmtiles_agg_udf` grouped-agg `pandas_udf`, `register_pmtiles_agg(spark)`, `_MAX_ARCHIVE_BYTES`, `_LIGHT_REGISTERED` flag | -| `python/geobrix/src/databricks/labs/gbx/pmtiles/__init__.py` (modify) | re-export `register_pmtiles_agg` | -| `python/geobrix/src/databricks/labs/gbx/pmtiles/functions.py` (modify, fallback only) | tier-aware `pmtiles_agg` wrapper *iff* Task 2's `call_function` test fails | -| `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` (modify) | call `register_pmtiles_agg(spark)` at end of `register` | -| `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py` (modify) | call `register_pmtiles_agg(spark)` at end of `register` | -| `python/geobrix/test/pmtiles/__init__.py` + `conftest.py` (create) | light no-JAR `spark` fixture | -| `python/geobrix/test/pmtiles/test_agg_light_core.py` (create) | Spark-free assembler tests | -| `python/geobrix/test/pmtiles/test_agg_light_udf.py` (create) | registered-UDF + wrapper + dual-register tests | -| `python/geobrix/test/pmtiles/test_serverless_safety.py` (create) | no `_jvm`/`conf`/`rdd` guard | -| `python/geobrix/test/ds/test_pmtiles_agg_parity.py` (create) | JAR-gated cross-tier decoded parity | -| `python/geobrix/src/databricks/labs/gbx/bench/readers.py` (modify) | `run_pmtiles_agg(...)` bench leg | -| `python/geobrix/src/databricks/labs/gbx/bench/cluster.py` (modify) | `_CELL_PMTILES_AGG` dispatch cell | -| `notebooks/tests/push_and_run_bench_on_cluster.py` (modify) | `--benchmark-pmtiles-agg` / `--pmtiles-agg-only` flags | -| `docs/docs/api/pmtiles-functions.mdx` (modify) | per-function tier: `gbx_pmtiles_agg` → ` ` + lib note | -| `docs/docs/api/execution-tiers.mdx` (modify) | move `gbx_pmtiles_agg` out of heavy-only | -| `docs/docs/api/performance.mdx` + `benchmarking.mdx` (modify) | pmtiles_agg light-vs-heavy result | - ---- - -## Task 1: Spark-free archive assembler core - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pmtiles/_agg_light.py` -- Test: `python/geobrix/test/pmtiles/test_agg_light_core.py` -- Create: `python/geobrix/test/pmtiles/__init__.py` (empty) - -- [ ] **Step 1: Write the failing tests** - -`python/geobrix/test/pmtiles/test_agg_light_core.py`: -```python -"""Spark-free tests for the light PMTiles archive assembler.""" -import io - -import pytest -from pmtiles.reader import MmapSource, Reader - -from databricks.labs.gbx.pmtiles._agg_light import _assemble_archive, _MAX_ARCHIVE_BYTES - -_PNG = b"\x89PNG\r\n\x1a\n" + b"\x00" * 16 # sniffs as PNG -def _mvt(i): # arbitrary non-magic bytes => sniffs as MVT - return b"mvt-payload-" + bytes([i % 256]) + b"\x00\x01\x02" - - -def _decode(blob, tmp_path): - p = tmp_path / "a.pmtiles" - p.write_bytes(blob) - out = {} - with open(p, "rb") as f: - r = Reader(MmapSource(f)) - for z in range(0, 6): - n = 2 ** z - for x in range(n): - for y in range(n): - t = r.get(z, x, y) - if t is not None: - out[(z, x, y)] = t - return out - - -def test_single_tile_roundtrip(tmp_path): - blob = _assemble_archive([_mvt(1)], [3], [2], [4], {}) - assert blob is not None - assert _decode(blob, tmp_path) == {(3, 2, 4): _mvt(1)} - - -def test_multi_zoom_roundtrip(tmp_path): - data = [_mvt(1), _mvt(2), _mvt(3)] - zs, xs, ys = [2, 3, 3], [1, 2, 5], [1, 4, 6] - got = _decode(_assemble_archive(data, zs, xs, ys, {}), tmp_path) - assert got == {(2, 1, 1): _mvt(1), (3, 2, 4): _mvt(2), (3, 5, 6): _mvt(3)} - - -def test_png_payload_roundtrip(tmp_path): - got = _decode(_assemble_archive([_PNG], [1], [0], [0], {}), tmp_path) - assert got == {(1, 0, 0): _PNG} - - -def test_metadata_roundtrip(tmp_path): - blob = _assemble_archive([_mvt(1)], [0], [0], [0], {"name": "demo", "n": 1}) - p = tmp_path / "m.pmtiles" - p.write_bytes(blob) - with open(p, "rb") as f: - r = Reader(MmapSource(f)) - assert r.metadata().get("name") == "demo" - - -def test_null_payloads_skipped(tmp_path): - got = _decode(_assemble_archive([None, _mvt(2), None], [0, 1, 0], [0, 1, 0], [0, 1, 0], {}), tmp_path) - assert got == {(1, 1, 1): _mvt(2)} - - -def test_empty_group_returns_none(): - assert _assemble_archive([], [], [], [], {}) is None - assert _assemble_archive([None], [0], [0], [0], {}) is None - - -def test_duplicate_tileid_dropped(tmp_path): - # two rows for the same (z,x,y): keep first, no Writer error - got = _decode(_assemble_archive([_mvt(1), _mvt(9)], [2, 2], [1, 1], [1, 1], {}), tmp_path) - assert got == {(2, 1, 1): _mvt(1)} - - -def test_cap_exceeded_raises(): - big = b"\x00" * (_MAX_ARCHIVE_BYTES + 1) - with pytest.raises(ValueError, match="exceeds"): - _assemble_archive([big], [0], [0], [0], {}) -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `cd python/geobrix && python -m pytest test/pmtiles/test_agg_light_core.py -q` -Expected: FAIL — `ModuleNotFoundError: ... pmtiles._agg_light`. - -- [ ] **Step 3: Implement the assembler** - -`python/geobrix/src/databricks/labs/gbx/pmtiles/_agg_light.py`: -```python -"""Lightweight gbx_pmtiles_agg — tier-neutral grouped aggregate. - -PMTiles archives raster OR vector tiles, so this lives in the pmtiles package -(not pyrx/pyvx) and is registered from BOTH light tiers. Reuses the ds.tiles -assembler; writes to an in-memory BytesIO. Serverless-safe: spark.udf.register + -Column expressions only (no _jvm / spark.conf / rdd). -""" - -from __future__ import annotations - -import io -import json -from typing import Optional, Sequence - -import pandas as pd -from pyspark.sql import Column, SparkSession -from pyspark.sql import functions as f -from pyspark.sql.functions import pandas_udf -from pyspark.sql.types import BinaryType -from pmtiles.tile import Compression, zxy_to_tileid -from pmtiles.writer import Writer - -from databricks.labs.gbx.ds.tiles._header import build_header_info, sniff_tile_type -from databricks.labs.gbx.ds.tiles.grid import SlippyGrid - -# Mirror heavy PMTilesAcc's 100 MiB accumulation cap so the failure mode matches. -_MAX_ARCHIVE_BYTES = 100 * 1024 * 1024 - -# Set True by register_pmtiles_agg; only consulted by the fallback wrapper path. -_LIGHT_REGISTERED = False - - -def _assemble_archive( - data: Sequence, - zs: Sequence, - xs: Sequence, - ys: Sequence, - metadata: Optional[dict] = None, -) -> Optional[bytes]: - """Fold a group's (bytes, z, x, y) tiles into one PMTiles v3 archive (bytes). - - Null payloads are skipped; an all-null/empty group returns None. Tiles are - written in ascending Hilbert TileID order; duplicate (z,x,y) keep the first. - """ - tiles = [] - seen = set() - total = 0 - first_payload = None - for d, z, x, y in zip(data, zs, xs, ys): - if d is None: - continue - b = bytes(d) - total += len(b) - if total > _MAX_ARCHIVE_BYTES: - raise ValueError( - f"pmtiles_agg group payload exceeds {_MAX_ARCHIVE_BYTES} bytes; " - "split into more groups or fewer tiles per archive" - ) - tileid = zxy_to_tileid(int(z), int(x), int(y)) - if tileid in seen: - continue - seen.add(tileid) - if first_payload is None: - first_payload = b - tiles.append((int(z), int(x), int(y), tileid, b)) - if not tiles: - return None - - tile_type = sniff_tile_type(first_payload) - info = build_header_info( - [(z, x, y) for (z, x, y, _, _) in tiles], - SlippyGrid(), - tile_type, - Compression.NONE, - metadata or {}, - ) - buf = io.BytesIO() - writer = Writer(buf) - for (_, _, _, tileid, b) in sorted(tiles, key=lambda t: t[3]): - writer.write_tile(tileid, b) - writer.finalize(info.header_dict(), info.metadata) - return buf.getvalue() -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `cd python/geobrix && python -m pytest test/pmtiles/test_agg_light_core.py -q` -Expected: PASS (8 passed). If `MmapSource`/`Reader` import fails, confirm the `pmtiles` lib is installed in the venv (`pip show pmtiles`). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pmtiles/_agg_light.py python/geobrix/test/pmtiles/__init__.py python/geobrix/test/pmtiles/test_agg_light_core.py -git commit -m "feat(pmtiles): light pmtiles_agg archive assembler core - -Co-authored-by: Isaac" -``` - ---- - -## Task 2: Grouped-agg UDF, register helper, and wrapper resolution - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pmtiles/_agg_light.py` (append the UDF + register helper) -- Modify: `python/geobrix/src/databricks/labs/gbx/pmtiles/__init__.py` -- Modify (fallback only): `python/geobrix/src/databricks/labs/gbx/pmtiles/functions.py` -- Create: `python/geobrix/test/pmtiles/conftest.py` -- Test: `python/geobrix/test/pmtiles/test_agg_light_udf.py` - -This task RESOLVES the open question from the spec: does the existing `pmtiles.functions.pmtiles_agg` wrapper (which uses `f.call_function("gbx_pmtiles_agg", ...)`) compose with a registered pandas GROUPED_AGG UDF inside `.agg()`? The test decides; the fallback is implemented only if it fails. - -- [ ] **Step 1: Create the light spark fixture** - -`python/geobrix/test/pmtiles/conftest.py` (mirror `test/pyvx/conftest.py`): -```python -import pytest -from pyspark.sql import SparkSession - - -@pytest.fixture(scope="session") -def spark(): - s = ( - SparkSession.builder.master("local[2]") - .appName("pmtiles-light-tests") - .config("spark.sql.shuffle.partitions", "2") - .config("spark.sql.execution.arrow.pyspark.enabled", "true") - .getOrCreate() - ) - yield s - s.stop() -``` - -- [ ] **Step 2: Write the failing tests** - -`python/geobrix/test/pmtiles/test_agg_light_udf.py`: -```python -"""Registered-UDF tests for the light gbx_pmtiles_agg (no JAR).""" -import io - -from pmtiles.reader import MmapSource, Reader -from pyspark.sql import functions as F - -from databricks.labs.gbx.pmtiles import functions as pt -from databricks.labs.gbx.pmtiles._agg_light import register_pmtiles_agg - -_MVT_A = b"mvt-a\x00\x01" -_MVT_B = b"mvt-b\x00\x02" - - -def _decode(blob, tmp_path): - p = tmp_path / "r.pmtiles" - p.write_bytes(blob) - out = {} - with open(p, "rb") as fh: - r = Reader(MmapSource(fh)) - for z in range(0, 6): - n = 2 ** z - for x in range(n): - for y in range(n): - t = r.get(z, x, y) - if t is not None: - out[(z, x, y)] = t - return out - - -def _rows(spark): - return spark.createDataFrame( - [("grp", _MVT_A, 3, 2, 4), ("grp", _MVT_B, 3, 5, 6)], - ["g", "tile", "z", "x", "y"], - ) - - -def test_wrapper_in_agg(spark, tmp_path): - register_pmtiles_agg(spark) - df = _rows(spark) - out = df.groupBy("g").agg(pt.pmtiles_agg("tile", "z", "x", "y").alias("arc")) - blob = out.collect()[0]["arc"] - assert _decode(blob, tmp_path) == {(3, 2, 4): _MVT_A, (3, 5, 6): _MVT_B} - - -def test_sql_name_in_agg(spark, tmp_path): - register_pmtiles_agg(spark) - _rows(spark).createOrReplaceTempView("tiles_v") - blob = spark.sql( - "SELECT gbx_pmtiles_agg(tile, z, x, y) AS arc FROM tiles_v GROUP BY g" - ).collect()[0]["arc"] - assert _decode(blob, tmp_path) == {(3, 2, 4): _MVT_A, (3, 5, 6): _MVT_B} - - -def test_metadata_passthrough(spark, tmp_path): - register_pmtiles_agg(spark) - df = _rows(spark).withColumn("meta", F.lit('{"name": "demo"}')) - out = df.groupBy("g").agg( - pt.pmtiles_agg("tile", "z", "x", "y", "meta").alias("arc") - ) - blob = out.collect()[0]["arc"] - p = tmp_path / "md.pmtiles" - p.write_bytes(blob) - with open(p, "rb") as fh: - assert Reader(MmapSource(fh)).metadata().get("name") == "demo" -``` - -- [ ] **Step 3: Run tests to verify they fail** - -Run (in Docker — needs Spark + the light env; see CLAUDE.md doc-test note): -`bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pmtiles/test_agg_light_udf.py --log pmtiles-agg-udf.log` -Expected: FAIL — `register_pmtiles_agg` import error (not yet defined). - -- [ ] **Step 4: Implement the UDF + register helper** - -Append to `python/geobrix/src/databricks/labs/gbx/pmtiles/_agg_light.py`: -```python -@pandas_udf(BinaryType()) -def _pmtiles_agg_udf( - data: pd.Series, - z: pd.Series, - x: pd.Series, - y: pd.Series, - metadata_json: pd.Series, -) -> Optional[bytes]: - """GROUPED_AGG: fold one group's tiles into a PMTiles archive (BINARY).""" - meta = {} - if metadata_json is not None and len(metadata_json) > 0: - for m in metadata_json: - if m is not None and str(m).strip(): - meta = json.loads(m) - break - return _assemble_archive(data, z, x, y, meta) - - -def register_pmtiles_agg(spark: SparkSession = None) -> None: - """Register the light gbx_pmtiles_agg grouped aggregate (Serverless-safe). - - Called by both pyrx.register and pyvx.register, and usable standalone. - """ - global _LIGHT_REGISTERED - if spark is None: - spark = SparkSession.builder.getOrCreate() - spark.udf.register("gbx_pmtiles_agg", _pmtiles_agg_udf) - _LIGHT_REGISTERED = True -``` - -`python/geobrix/src/databricks/labs/gbx/pmtiles/__init__.py` (replace empty file): -```python -from databricks.labs.gbx.pmtiles._agg_light import register_pmtiles_agg - -__all__ = ["register_pmtiles_agg"] -``` - -- [ ] **Step 5: Run tests** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pmtiles/test_agg_light_udf.py --log pmtiles-agg-udf.log` -Expected outcomes: -- **All 3 PASS** → the `call_function` wrapper composes; **do nothing to `functions.py`**. Skip Step 6. -- **`test_sql_name_in_agg` passes but `test_wrapper_in_agg` FAILS** (the wrapper's `call_function` doesn't resolve the registered pandas UDAF in the DataFrame `.agg()`) → apply Step 6 (fallback), then re-run. - -- [ ] **Step 6 (FALLBACK — apply only if `test_wrapper_in_agg` failed): tier-aware wrapper** - -Edit `python/geobrix/src/databricks/labs/gbx/pmtiles/functions.py` — make `pmtiles_agg` prefer the directly-callable light UDF when the light tier is registered (mirrors the pygx `quadbin_cellunion_agg` direct-object pattern), else fall back to `call_function` for heavy: -```python -def pmtiles_agg(bytes_col, z, x, y, metadata_json=None): - meta = f.lit("{}") if metadata_json is None else _col(metadata_json) - from databricks.labs.gbx.pmtiles import _agg_light - if _agg_light._LIGHT_REGISTERED: - return _agg_light._pmtiles_agg_udf( - _col(bytes_col), _col(z), _col(x), _col(y), meta - ) - return f.call_function( - "gbx_pmtiles_agg", _col(bytes_col), _col(z), _col(x), _col(y), meta - ) -``` -(Preserve the existing `meta` defaulting logic already in `functions.py`; only add the `_LIGHT_REGISTERED` branch.) Re-run Step 5 → all 3 PASS. - -- [ ] **Step 7: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pmtiles/ python/geobrix/test/pmtiles/conftest.py python/geobrix/test/pmtiles/test_agg_light_udf.py -git commit -m "feat(pmtiles): light pmtiles_agg grouped-agg UDF + register helper - -Resolve the wrapper path (call_function vs direct udf object) by test. - -Co-authored-by: Isaac" -``` - ---- - -## Task 3: Wire into pyrx + pyvx register (and standalone) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` (end of `register`, ~line 101) -- Modify: `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py` (end of `register`, ~line 279) -- Test: `python/geobrix/test/pmtiles/test_agg_light_udf.py` (append) - -- [ ] **Step 1: Write the failing tests** - -Append to `python/geobrix/test/pmtiles/test_agg_light_udf.py`: -```python -def _agg_registered(spark): - names = {r.function for r in spark.sql("SHOW USER FUNCTIONS").collect()} - return any(n.endswith("gbx_pmtiles_agg") for n in names) - - -def test_pyrx_register_installs_pmtiles_agg(spark): - from databricks.labs.gbx.pyrx import functions as rx - rx.register(spark) - assert _agg_registered(spark) - - -def test_pyvx_register_installs_pmtiles_agg(spark): - from databricks.labs.gbx.pyvx import functions as vx - vx.register(spark) - assert _agg_registered(spark) -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path "python/geobrix/test/pmtiles/test_agg_light_udf.py::test_pyrx_register_installs_pmtiles_agg python/geobrix/test/pmtiles/test_agg_light_udf.py::test_pyvx_register_installs_pmtiles_agg" --log pmtiles-reg.log` -Expected: FAIL (the registers don't yet install `gbx_pmtiles_agg`). - -- [ ] **Step 3: Add the hook to pyrx** - -In `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py`, at the END of `register` (after the last `spark.udtf.register("gbx_rst_xyzpyramid", _RstXyzPyramidUDTF)` line ~101), append: -```python - # PMTiles archive aggregate is format-agnostic (raster or vector tiles); - # register it from the light raster tier too. - from databricks.labs.gbx.pmtiles import register_pmtiles_agg - register_pmtiles_agg(spark) -``` - -- [ ] **Step 4: Add the hook to pyvx** - -In `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py`, at the END of `register` (after the last `spark.udtf.register("gbx_st_interpolateelevationgeom", _InterpElevGeomUDTF)` line ~279), append: -```python - from databricks.labs.gbx.pmtiles import register_pmtiles_agg - register_pmtiles_agg(spark) -``` - -- [ ] **Step 5: Run tests** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pmtiles/ --log pmtiles-all.log` -Expected: PASS (all pmtiles light tests). - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pyrx/functions.py python/geobrix/src/databricks/labs/gbx/pyvx/functions.py python/geobrix/test/pmtiles/test_agg_light_udf.py -git commit -m "feat(pmtiles): register pmtiles_agg from pyrx and pyvx - -Co-authored-by: Isaac" -``` - ---- - -## Task 4: Serverless-safety guard - -**Files:** -- Test: `python/geobrix/test/pmtiles/test_serverless_safety.py` (create) - -- [ ] **Step 1: Write the test** - -Mirror the established Serverless guard (no `_jvm` / `sparkContext._jsc` / `.rdd` / `spark.conf.set` in the light source): -```python -"""The light pmtiles_agg module must be Serverless/Connect-safe.""" -import inspect - -from databricks.labs.gbx.pmtiles import _agg_light - -_FORBIDDEN = ("_jvm", "sparkContext", ".rdd", "spark.conf.set", "_jsc") - - -def test_no_spark_internal_access(): - src = inspect.getsource(_agg_light) - for bad in _FORBIDDEN: - assert bad not in src, f"Serverless-unsafe access: {bad}" -``` - -- [ ] **Step 2: Run** - -Run: `cd python/geobrix && python -m pytest test/pmtiles/test_serverless_safety.py -q` -Expected: PASS (the module uses only `spark.udf.register` + Column exprs). - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/pmtiles/test_serverless_safety.py -git commit -m "test(pmtiles): Serverless-safety guard for light pmtiles_agg - -Co-authored-by: Isaac" -``` - ---- - -## Task 5: JAR-gated cross-tier decoded parity - -**Files:** -- Test: `python/geobrix/test/ds/test_pmtiles_agg_parity.py` (create) - -Mirror the JAR gating + decode helper from `test/ds/test_pmtiles_parity.py`. Register light, capture archive; register heavy, capture archive; assert decoded tile-dicts + metadata equal (NOT byte-identical). - -- [ ] **Step 1: Write the test** - -```python -"""Cross-tier decoded parity for gbx_pmtiles_agg (light vs heavy). JAR-gated.""" -from pathlib import Path - -import pytest -from pmtiles.reader import MmapSource, Reader - -pytestmark = pytest.mark.integration - -_HERE = Path(__file__).resolve() -_JARS = sorted((_HERE.parents[3] / "lib").glob("geobrix-*-jar-with-dependencies.jar")) - -# Two MVT-ish payloads + one POLYGON-derived payload across two zooms. -_TILES = [ - ("g", b"tile-point-0\x00", 3, 2, 4), - ("g", b"tile-point-1\x00", 3, 5, 6), - ("g", b"tile-polygon-\x07\x08\x09", 2, 1, 1), -] - - -def _decode(path): - out = {} - with open(path, "rb") as f: - r = Reader(MmapSource(f)) - for z in range(0, 8): - n = 2 ** z - for x in range(n): - for y in range(n): - t = r.get(z, x, y) - if t is not None: - out[(z, x, y)] = t - return out, r.metadata() - - -@pytest.fixture(scope="module") -def spark_with_jar(): - if not _JARS: - pytest.skip("no geobrix JAR staged") - from pyspark.sql import SparkSession - session = ( - SparkSession.builder.master("local[2]") - .appName("pmtiles-agg-parity") - .config("spark.jars", str(_JARS[0])) - .config("spark.sql.shuffle.partitions", "2") - .config("spark.sql.execution.arrow.pyspark.enabled", "true") - .getOrCreate() - ) - yield session - session.stop() - - -def _archive(spark, register_fn, tmp_path, name): - register_fn(spark) - from databricks.labs.gbx.pmtiles import functions as pt - df = spark.createDataFrame(_TILES, ["g", "tile", "z", "x", "y"]) - blob = ( - df.groupBy("g") - .agg(pt.pmtiles_agg("tile", "z", "x", "y").alias("arc")) - .collect()[0]["arc"] - ) - p = tmp_path / f"{name}.pmtiles" - p.write_bytes(blob) - return _decode(p) - - -def test_decoded_tile_parity(spark_with_jar, tmp_path): - from databricks.labs.gbx.pmtiles._agg_light import register_pmtiles_agg - from databricks.labs.gbx.pmtiles import functions as heavy_pt - - light_tiles, _ = _archive(spark_with_jar, register_pmtiles_agg, tmp_path, "light") - heavy_tiles, _ = _archive(spark_with_jar, heavy_pt.register, tmp_path, "heavy") - assert light_tiles == heavy_tiles -``` - -- [ ] **Step 2: Run (requires a staged JAR; otherwise skips)** - -Build + stage the JAR into `python/geobrix/lib/` (see CLAUDE.md / `gbx:data:push-jar` builds the fat jar; for local parity, copy the `*-jar-with-dependencies.jar` into `python/geobrix/lib/`). Then: -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_pmtiles_agg_parity.py --log pmtiles-agg-parity.log` -Expected: PASS, or SKIP ("no geobrix JAR staged") when no JAR is present. - -- [ ] **Step 3: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/ds/test_pmtiles_agg_parity.py -git commit -m "test(pmtiles): JAR-gated cross-tier decoded parity for pmtiles_agg - -Co-authored-by: Isaac" -``` - ---- - -## Task 6: Documentation (tier flip + lib note) - -**Files:** -- Modify: `docs/docs/api/pmtiles-functions.mdx` -- Modify: `docs/docs/api/execution-tiers.mdx` - -- [ ] **Step 1: Per-function tier on the PMTiles functions page** - -`docs/docs/api/pmtiles-functions.mdx` is page-level `` (line ~11). It can no longer be page-level heavy. Add `import { Impl } from '@site/src/components/Tier';` if not present, change the page-level statement to scope it to the still-heavy entries, and under the `gbx_pmtiles_agg` heading place: -```mdx - - -:::note Lightweight tier -Powered by the **pmtiles** package. Grouped aggregate — `groupBy(...).agg(pt.pmtiles_agg("tile", "z", "x", "y"))` folds a group's `(bytes, z, x, y)` tiles into one PMTiles v3 archive (BINARY). Registered by both `pyrx.register` and `pyvx.register` (PMTiles archives raster or vector tiles). -::: -``` -Leave any genuinely heavy-only PMTiles entries as ``. - -- [ ] **Step 2: execution-tiers.mdx** - -Move `gbx_pmtiles_agg` out of the heavy-only column/list into the both-tiers grouping (grep `pmtiles_agg` in `docs/docs/api/execution-tiers.mdx`; if it's listed under a heavy-only section, relocate it). - -- [ ] **Step 3: Verify the docs build** - -Run: `cd docs && npm run build` -Expected: SUCCESS, no broken links. Then `grep -rn -iE "wave [0-9]+" docs/docs/` → empty. - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX .git/objects -git add docs/docs/api/pmtiles-functions.mdx docs/docs/api/execution-tiers.mdx -git commit -m "docs(pmtiles): gbx_pmtiles_agg now both tiers (grouped-agg + lib note) - -Co-authored-by: Isaac" -``` - ---- - -## Task 7: Bench leg — `run_pmtiles_agg` - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/readers.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/cluster.py` -- Modify: `notebooks/tests/push_and_run_bench_on_cluster.py` - -Mirror `run_mvt_agg` (`bench/readers.py:857-1120`) — a grouped-agg leg — NOT `run_pmtiles_write`. Read `run_mvt_agg` in full and copy its structure (tier register → synthetic corpus → validation pass → `time_iters` → `ResultRow`), changing the aggregate and the corpus to PMTiles tiles. - -- [ ] **Step 1: Add `run_pmtiles_agg` to `bench/readers.py`** - -```python -def run_pmtiles_agg( - spark, - run_id: str, - warmup: int, - measured: int, - *, - api: str, # "lightweight" or "heavyweight" - n_tiles: int = 1000, - n_groups: int = 1, - where: str = "cluster", -) -> "ResultRow": - """Grouped-agg bench: fold n_tiles synthetic PNG tiles into PMTiles archive(s).""" - from pyspark.sql import functions as F - - if api == "lightweight": - from databricks.labs.gbx.pmtiles import register_pmtiles_agg - register_pmtiles_agg(spark) - from databricks.labs.gbx.pmtiles import functions as pt - else: - from databricks.labs.gbx.pmtiles import functions as pt - pt.register(spark) - - # Synthetic PNG tiles at zoom z over a 2^z grid, split into n_groups. - z = 6 - n = 2 ** z - png = b"\x89PNG\r\n\x1a\n" + b"\x00" * 256 - rows = [] - for i in range(n_tiles): - x, y = i % n, (i // n) % n - rows.append((i % n_groups, bytearray(png), z, x, y)) - df = ( - spark.createDataFrame(rows, ["g", "tile", "z", "x", "y"]) - .withColumn("tile", F.col("tile").cast("binary")) - .cache() - ) - df.count() - - def _job(): - return ( - df.groupBy("g") - .agg(pt.pmtiles_agg("tile", "z", "x", "y").alias("arc")) - .count() - ) - - _job() # validation pass (untimed) - median = time_iters(_job, warmup, measured) - df.unpersist() - return ResultRow( - run_id=run_id, - category="pmtiles_agg", - mode="spark-path", - api=api, - fn="pmtiles_agg", - row_count=n_tiles, - median_seconds=median, - parity_status="n/a", - where=where, - ) -``` -**Note:** match `ResultRow(...)`'s exact keyword set to the one `run_mvt_agg` uses (read its return statement and copy the field names verbatim — adjust only `category`/`fn`/`row_count`). If `time_iters`/`ResultRow` need imports, they're already imported at the top of `readers.py` (used by `run_mvt_agg`). - -- [ ] **Step 2: Add the cluster dispatch cell** - -In `bench/cluster.py`, add `_CELL_PMTILES_AGG` modeled on `_CELL_PMTILES` (lines 640-754) but calling `run_pmtiles_agg` for both `api="lightweight"` and `api="heavyweight"` and appending both ResultRows. Wire it near line 1779: -```python - if benchmark_pmtiles_agg or pmtiles_agg_only: - cells.append(_cell(_CELL_PMTILES_AGG)) -``` -Add `benchmark_pmtiles_agg=False, pmtiles_agg_only=False` to `build_notebook(...)`'s signature alongside the existing `benchmark_pmtiles`/`pmtiles_only` params. - -- [ ] **Step 3: Add launcher flags** - -In `notebooks/tests/push_and_run_bench_on_cluster.py`, alongside `--benchmark-pmtiles`/`--pmtiles-only` (parsed ~lines 254-267), add: -```python - parser.add_argument("--benchmark-pmtiles-agg", action="store_true") - parser.add_argument("--pmtiles-agg-only", action="store_true") -``` -and thread `benchmark_pmtiles_agg=args.benchmark_pmtiles_agg, pmtiles_agg_only=args.pmtiles_agg_only` into the `cluster.build_notebook(...)` call. - -- [ ] **Step 4: Local smoke test** - -Run: `cd python/geobrix && python -c "from databricks.labs.gbx.bench.readers import run_pmtiles_agg; print('import ok')"` -Expected: `import ok` (no syntax/name errors). The full leg runs on cluster in Task 8. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/bench/readers.py python/geobrix/src/databricks/labs/gbx/bench/cluster.py notebooks/tests/push_and_run_bench_on_cluster.py -git commit -m "feat(bench): light-vs-heavy pmtiles_agg grouped-agg leg - -Co-authored-by: Isaac" -``` - ---- - -## Task 8: Cluster bench run + benchmarking docs - -**Files:** -- Modify: `docs/docs/api/benchmarking.mdx` -- Modify: `docs/docs/api/performance.mdx` - -- [ ] **Step 1: Build + stage artifacts, then run the leg on the warm bench cluster** - -Per the bench memories: build+stage BOTH the fat jar + tests.jar before cluster start (`gbx:data:push-jar` / `push-wheel`), keep the standing 0519 bench cluster warm. Run ONLY the pmtiles_agg leg at 1000 tiles, both tiers: -```bash -python notebooks/tests/push_and_run_bench_on_cluster.py --pmtiles-agg-only --row-counts 1000 --spark-measured 5 -``` -Verify exactly one geobrix-bench run on the cluster (don't double-launch). Capture the light-vs-heavy median + decoded parity status. - -- [ ] **Step 2: Update benchmarking.mdx + performance.mdx** - -Add the pmtiles_agg result to `docs/docs/api/benchmarking.mdx` (#results) and the execution-shape/narrative to `performance.mdx`, framed noise-aware (grouped-agg, decoded parity, the GZIP-vs-NONE internal-compression note). Per the standing rule, any bench change must be reflected in `benchmarking.mdx` in the same stroke. Give the run's `bench-out//summary.md` link. - -- [ ] **Step 3: Verify docs build + commit** - -```bash -cd docs && npm run build # SUCCESS, no broken links -grep -rn -iE "wave [0-9]+" docs/docs/ # empty -cd .. && chmod -R u+rwX .git/objects -git add docs/docs/api/benchmarking.mdx docs/docs/api/performance.mdx -git commit -m "docs(bench): pmtiles_agg light-vs-heavy result - -Co-authored-by: Isaac" -``` - -- [ ] **Step 4: Push the branch** - -```bash -gh auth switch --user mjohns-databricks -export QC_OVERRIDE=1 -git push origin pygx-light -``` - ---- - -## Final review - -- [ ] Dispatch a final code-reviewer over the whole add-on (the `_agg_light` module, the two register hooks, all tests, the bench leg, the docs). Confirm: decoded parity (not byte) is the asserted contract; no new SQL name leaked into `registered_functions.txt`/`function-info.json`; Serverless-safe; `gbx:test:bindings` still green; no wave/internal vocab in docs. -- [ ] Confirm `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pmtiles/` is green and `test/ds/test_pmtiles_agg_parity.py` passes (or skips cleanly without a JAR). - ---- - -## Self-review notes (author) - -- **Spec coverage:** neutral home (Task 1-2), dual register + standalone (Task 3), decoded parity (Task 5), no-new-SQL-name (called out, Task 6 confirms bindings untouched), Serverless safety (Task 4), docs tier flip + lib note + execution-tiers + benchmarking (Task 6, 8), bench leg (Task 7-8), the `call_function` wrapper question resolved by test with documented fallback (Task 2). 100 MiB cap + magic-byte sniff + Hilbert order + leaf-dir capability all in the Task-1 assembler. All spec sections map to a task. -- **Type/name consistency:** `_assemble_archive(data, zs, xs, ys, metadata)`, `_pmtiles_agg_udf(data, z, x, y, metadata_json)`, `register_pmtiles_agg(spark)`, `_MAX_ARCHIVE_BYTES`, `_LIGHT_REGISTERED` — used identically across Tasks 1-7. `SlippyGrid`, `build_header_info`, `sniff_tile_type`, `zxy_to_tileid`, `Compression.NONE`, `pmtiles.writer.Writer`, `pmtiles.reader.{MmapSource,Reader}` match the recon'd real symbols. -- **No placeholders:** every code step has real code; the one "mirror `run_mvt_agg`" step in Task 7 includes a complete runnable function plus the explicit delta (copy `ResultRow` kwargs verbatim) because the bench leg intentionally mirrors an existing 260-line function. diff --git a/docs/superpowers/plans/2026-06-14-pygx-bng.md b/docs/superpowers/plans/2026-06-14-pygx-bng.md deleted file mode 100644 index c2b44d679..000000000 --- a/docs/superpowers/plans/2026-06-14-pygx-bng.md +++ /dev/null @@ -1,1301 +0,0 @@ -# pygx Phase 2 — BNG Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Ship the 23 lightweight BNG (British National Grid) GridX functions in the existing pure-Python `databricks.labs.gbx.pygx` package, at exact cell-ID / cell-set parity with the heavy Scala tier. BNG is a faithful pure-Python port of `gridx/grid/BNG.scala` (no PyPI BNG library exists). On completion, all 23 `gbx_bng_*` SQL names run in both tiers as a one-line import swap, and every doc surface flips BNG heavy→both (custom grids stay heavyweight-only). - -**Architecture:** Pure-Python/PySpark, Serverless/Connect-safe (only `spark.udf.register`/`spark.udtf.register` + Column exprs — never `_jvm`/`spark.conf.set`/`.rdd`). All BNG cell math is ported from `BNG.scala` into `pygx/_bng.py`; geometry via shapely → **WKB, no SRID** (heavy BNG uses `JTS.toWKB`, *not* `toEWKB` — unlike quadbin which is EWKB SRID 4326). Mirrors the just-completed `pygx` quadbin (Phase 1) and the `pyvx` patterns. - -**Tech Stack:** Python 3.12, `shapely` 2.x, `numpy`, PySpark `@udf`/`pandas_udf`/`@udtf`. **No new dependencies** (`shapely`/`numpy` already in the `[light]` extra; BNG is pure Python — no `quadbin`/BNG lib involved). - -**Spec:** `docs/superpowers/specs/2026-06-14-pygx-light-tier-design.md` (Phase 2). **Branch:** `pygx-light`. Out of scope: quadbin (Phase 1, complete), `gbx_custom_*` (stays heavy-only), the `h3` GridX subpackage (native H3 covers hex). - ---- - -## The 23 functions (parity targets, all `gbx_bng_*`) - -From `docs/tests-function-info/registered_functions.txt` (lines 111–133), 23 names: - -| Function | Shape | Output | Light source (port of `BNG.scala`) | -|---|---|---|---| -| `pointascell(geom, res)` | scalar | STRING cellid | `pointToCellID(centroid.x, centroid.y, res)` → `format` | -| `eastnorthasbng(e, n, res)` | scalar | STRING cellid | `pointToCellID(e, n, res)` → `format` | -| `cellarea(cellid)` | scalar | DOUBLE (km²) | `area` = `(edgeSize/1000)²` | -| `distance(c1, c2)` | scalar | LONG | `distance` (Manhattan, edge-size units) | -| `euclideandistance(c1, c2)` | scalar | LONG | `euclideanDistance` (Chebyshev / max) | -| `aswkb(cellid)` | scalar | BINARY (WKB polygon, **no SRID**) | `cellIdToGeometry` → shapely → `to_wkb` | -| `aswkt(cellid)` | scalar | STRING (WKT polygon) | `cellIdToGeometry` → shapely `wkt` | -| `centroid(cellid)` | scalar | BINARY (WKB point, no SRID) | `cellIdToGeometry().centroid` | -| `kring(cellid, k)` | scalar | ARRAY\ | `kRing` → `format` | -| `kloop(cellid, k)` | scalar | ARRAY\ | `kLoop` → `format` | -| `polyfill(geom, res)` | scalar | ARRAY\ | `polyfill` BFS flood-fill → `format` | -| `tessellate(geom, res)` | scalar | ARRAY\\> | `tessellate` core/border split | -| `cellintersection(left, right)` | scalar | STRUCT\ | per-chip JTS-style intersection (left-hand rule) | -| `cellunion(left, right)` | scalar | STRUCT\ | per-chip union (same-cell) | -| `geomkring(geom, res, k)` | scalar | ARRAY\ | `geometryKRing` (chips + per-border kRing) | -| `geomkloop(geom, res, k)` | scalar | ARRAY\ | `geometryKLoop` (chips + per-border kLoop minus nRing) | -| `cellintersection_agg(chip)` | grouped-agg | STRUCT\ | accumulate intersection per cell | -| `cellunion_agg(chip)` | grouped-agg | STRUCT\ | accumulate union per cell | -| `kringexplode(cellid, k)` | UDTF | rows of STRING cellid | `kRing` exploded | -| `kloopexplode(cellid, k)` | UDTF | rows of STRING cellid | `kLoop` exploded | -| `geomkringexplode(geom, res, k)` | UDTF | rows of STRING cellid | `geometryKRing` exploded | -| `geomkloopexplode(geom, res, k)` | UDTF | rows of STRING cellid | `geometryKLoop` exploded | -| `tessellateexplode(geom, res)` | UDTF | rows of STRUCT\ | `tessellate` exploded | - -Heavy reference: `src/main/scala/com/databricks/labs/gbx/gridx/bng/` (the per-function wrappers) + the **canonical algorithm** `gridx/grid/BNG.scala` (≈816 lines). Per-function wrappers just parse the WKT/WKB input, look up `resolutionMap` for string res, and call into `BNG.*` then `BNG.format`. - -### Key heavy facts to bake in (from `BNG.scala`) -- **Cell ids are STRING in the SQL surface** (e.g. `"TQ3080"`, `"NE"`). Internally `BNG` works on a `Long` digit-id; `format(Long)→String` and `parse(String)→Long` round-trip it. The pygx UDFs accept/return the STRING form (call `parse` on input, `format` on output) to match heavy exactly. -- **WKB has NO SRID** (`JTS.toWKB`, not `toEWKB`). Quadbin used EWKB SRID 4326; **BNG must use plain WKB** (`shapely.to_wkb(geom)` with `include_srid=False`). `aswkt` is plain WKT. Coordinates are EPSG:27700 eastings/northings, but no SRID is stamped — do NOT call `set_srid`. -- **Resolution** = Int index in `{1,-1,2,-2,3,-3,4,-4,5,-5,6,-6}` (1=100km, 2=10km, 3=1km, 4=100m, 5=10m, 6=1m; negatives = quadrant resolutions 500km/50km/5km/500m/50m/5m) **or** a `resolutionMap` string key (`"500km","100km","50km","10km","5km","1km","500m","100m","50m","10m","5m","1m"`). **Never** metres-as-Int (1000). A bare Int is the index; a string is mapped via `resolutionMap`. Reject anything else with a clear `ValueError`. -- **`cellarea` returns square KILOMETRES** (`(edgeSize/1000)²`), not m². Edge sizes via `sizeMap`. -- **`pointascell` expects EPSG:27700 eastings/northings**, not WGS84 — it reads the centroid x/y of the input geometry as planar BNG coords. Doc examples must use BNG coords (`POINT(530000 180000)` for London → `"TQ3080"`-family). -- **Encode/format digit-length rule** (the #580 disambiguation, see below): `getResolution(digits)` returns `-1` (500km) only when `digits.length < 6`; otherwise `quadrant = digits.last`, `k = (n-6)/2`, and resolution is `-(k+2)` if `quadrant>0` else `k+1`. The trailing digit is the quadrant marker (0 = no quadrant). `getX`/`getY` reconstruct eastings/northings from the digit slices and the edge size, applying half-cell offsets for quadrants 2/3 (y) and 3/4 (x). -- **`format`** (`< 6` digits): prefix letter only (500km). (`>= 6`): `prefix + xStr + yStr + qStr` where the bins are split half/half and `qStr` is the quadrant suffix from `quadrants = Seq("","SW","NW","NE","SE")`. - -### BNG known-issue validation (Mosaic lineage) — findings during plan authoring - -Both bugs were checked against the current `gridx/grid/BNG.scala` while writing this plan. **Both fixes are present in the geobrix port.** The Phase-2 tasks must port the FIXED behavior and lock it with parity tests: - -- **[mosaic#434](https://github.com/databrickslabs/mosaic/issues/434)** — fixed upstream in [mosaic#580](https://github.com/databrickslabs/mosaic/pull/580). **CONFIRMED FIXED in geobrix `BNG.scala`.** Verified by hand: encoding the 100km NE cell (`encode(eLetter=4, nLetter=9, 0,0, quadrant=0, nPositions=1, resolution=1)`) yields the Long `104090` — a **6-digit** id, so `getResolution(digits)` takes the `length >= 6` branch with `quadrant = digits.last = 0`, returning `k+1 = 1` (= 100km), **not** a negative quadrant resolution. The `"NE"/"NW"/"SE"/"SW"` two-letter prefixes are therefore never misread as quadrant markers, and `cellIdToGeometry` builds a full 100km × 100km tile. The light port (`_bng.py`) replicates the exact digit-length + trailing-quadrant-digit logic, so it inherits the fix. **Task 9's parity test MUST include these four 100km cells and assert `cellarea == 10000.0` km² (= 100km × 100km) in both tiers.** -- **[mosaic#423](https://github.com/databrickslabs/mosaic/issues/423)** — degenerate (POINT/LINESTRING) chips on grid-aligned polygons. **CONFIRMED FIXED in geobrix `BNG.scala`.** Verified by reading `BNG.tessellate` (line 813): the final result is `(coreChips ++ borderChips).filter(t => t._3 == null || t._3.getGeometryType == inGeomType)` — border-chip intersections whose geometry type differs from the input geometry type (e.g. a POINT/LINESTRING touching a Polygon input at a shared vertex/edge) are dropped. The light port must replicate this **input-geometry-type filter** on border chips. **Task 6's parity test MUST include a grid-aligned polygon and assert NO non-areal (POINT/LINESTRING) chips in either tier, and equal surviving chip cell-sets.** - -No heavy change is required for either bug (both already correct). Reference both issues in the `_bng.py` docstrings/comments at the relevant code (per the "reference issues when validated/fixed" rule) and note in the beta release notes that BNG light matches heavy on the corrected #434/#423 behavior. - -## Conventions (every task) -- Spark-free core tests (`_bng.py`) run on host: `.venv-pyrx/bin/python -m pytest -v`. Registered-fn + parity tests run in the `geobrix-dev` container: `bash scripts/commands/gbx-test-python.sh --path --log .log`. -- Commit (no push unless a task says so): `chmod -R u+rwX .git/objects`; subject ≤72 + WHY body; trailer exactly `Co-authored-by: Isaac`. -- Serverless guard: never add `_jvm`/`sparkContext`/`.rdd`/`spark.conf.set` to `pygx`. -- Impl rule (from Phase 1, apply to BNG identically): **scalar/bounded-output → `pandas_udf`** (pointascell/eastnorthasbng/cellarea/distance/euclideandistance + single-geometry aswkb/aswkt/centroid + the chip-pair cellintersection/cellunion); **ARRAY-returning → plain `@udf`** (kring/kloop/polyfill/tessellate/geomkring/geomkloop — variable-length output, row-by-row for OOM safety); **explode → `@udtf`** (no Python Column form; raise `NotImplementedError` pointing to SQL `LATERAL`, like pyvx pyramid); **grouped-agg → `pandas_udf` returning STRUCT** (cellunion_agg/cellintersection_agg). -- Note: the pygx test package already lives in `test/pygx/` (not named after a PyPI lib), so the test-package-shadows-installed-lib gotcha that bit the pmtiles work does not apply here. - -## File Structure -| File | Responsibility | New? | -|---|---|---| -| `python/geobrix/src/databricks/labs/gbx/pygx/_bng.py` | pure-Python port of `BNG.scala`: codec (`encode`/`decode`/`format`/`parse`, `resolutionMap`/`sizeMap`/`letterMap`/quadrants), `get_resolution`, `point_to_cell_id`, `cell_id_to_geometry`/`cell_id_to_center`, `area`, `distance`/`euclidean_distance`, `k_ring`/`k_loop`, `polyfill`, `tessellate`, `geometry_k_ring`/`geometry_k_loop`, chip intersection/union | new | -| `pygx/_serde.py` | **add** `BNG_CHIP_SCHEMA` (`StructType`) alongside `QUADBIN_CELL_SCHEMA` | extend | -| `pygx/_env.py` | **add** `assert_bng_available()` (shapely + numpy only; no quadbin) | extend | -| `pygx/functions.py` | **add** BNG UDFs/UDTFs/aggs + Column wrappers + extend `register(spark)` to install all 23 `gbx_bng_*` | extend | -| `python/geobrix/test/pygx/test_bng_codec.py` | Spark-free `_bng` codec round-trip + every resolution + quadrants + #434/#423 cases | new | -| `python/geobrix/test/pygx/test_bng_cellmath.py` | Spark-free `_bng` geometry/distance/kring/kloop/polyfill/tessellate/geomk* unit tests | new | -| `python/geobrix/test/pygx/test_bng_udf.py` | registered-fn tests via the spark fixture (UDF/UDTF/agg) | new | -| `python/geobrix/test/pygx/test_parity_bng.py` | JAR-gated cross-tier exact parity (cells + WKB geom within 1e-6) | new | - ---- - -## Task 1: `_bng.py` codec — resolutionMap/sizeMap/letterMap, encode/format/parse, get_resolution - -**Files:** create `pygx/_bng.py`, `test/pygx/test_bng_codec.py`. - -Port the constants and the Long-digit codec verbatim from `BNG.scala`: `resolutionMap`, `sizeMap`, `letterMap` (14×8), `quadrants = ["", "SW", "NW", "NE", "SE"]`, `RESOLUTIONS = {1,-1,2,-2,3,-3,4,-4,5,-5,6,-6}`, and `encode`, `cell_digits`, `get_resolution(digits)`, `get_x`, `get_y`, `format`, `parse`, `get_resolution(res: Any)` (Int-index or resolutionMap-string), `get_edge_size`. - -- [ ] **Step 1: failing test** `test/pygx/test_bng_codec.py` -```python -import pytest - -shapely = pytest.importorskip("shapely") # _bng imports shapely at module load -from databricks.labs.gbx.pygx import _bng - - -def test_resolution_index_and_string_keys(): - # Int index passes through if a valid BNG resolution. - assert _bng.get_resolution(1) == 1 - assert _bng.get_resolution(-2) == -2 - # String keys map via resolutionMap. - assert _bng.get_resolution("100km") == 1 - assert _bng.get_resolution("1km") == 3 - assert _bng.get_resolution("100m") == 4 - assert _bng.get_resolution("1m") == 6 - - -def test_resolution_rejects_metres_as_int_and_junk(): - with pytest.raises(ValueError): - _bng.get_resolution(1000) # metres-as-Int is NOT a resolution - with pytest.raises(ValueError): - _bng.get_resolution(7) - with pytest.raises(ValueError): - _bng.get_resolution("nope") - - -def test_eastnorth_encode_format_london_1km(): - # London TQ 30 80 at 1km: easting 530000, northing 180000. - cell_long = _bng.point_to_cell_id(530000.0, 180000.0, _bng.get_resolution("1km")) - s = _bng.format(cell_long) - assert s == "TQ3080" - - -def test_format_parse_roundtrip_all_resolutions(): - # Encode a fixed BNG point at every supported resolution; format->parse->format is stable. - e, n = 530000.0, 180000.0 - for res in sorted(_bng.RESOLUTIONS): - cid = _bng.point_to_cell_id(e, n, res) - s = _bng.format(cid) - reparsed = _bng.parse(s) - assert _bng.format(reparsed) == s, f"roundtrip failed at res={res}: {s}" - - -def test_500km_prefix_only(): - # 500km (res -1) formats to a single prefix letter. - cid = _bng.point_to_cell_id(530000.0, 180000.0, -1) - s = _bng.format(cid) - assert len(s) == 1 and s.isalpha() - - -def test_100km_NE_family_not_quadrant(): - # mosaic#434 fix: a 100km cell whose 2-letter prefix is NE/NW/SE/SW is res 1, - # NOT a quadrant. Its id is 6 digits with trailing quadrant-digit 0. - # NE region centre (e.g. easting 450000, northing 950000 -> "NE"). - cid = _bng.point_to_cell_id(450000.0, 950000.0, 1) - digits = _bng.cell_digits(cid) - assert digits[-1] == 0 # quadrant marker is 0 - assert _bng.get_resolution(digits) == 1 # 100km, not a negative quadrant res - assert _bng.format(cid)[:2] in {"NE", "NW", "SE", "SW", "OA", "NA"} -``` - -- [ ] **Step 2: run → FAIL** (`.venv-pyrx/bin/python -m pytest python/geobrix/test/pygx/test_bng_codec.py -v` — no `_bng`). - -- [ ] **Step 3: implement** `pygx/_bng.py` codec. Port directly from `BNG.scala`. Module docstring must state it is a pure-Python port of `gridx/grid/BNG.scala`, WKB has no SRID, and reference mosaic#434/#580 + #423 on the relevant functions. -```python -"""Pure-Python British National Grid (BNG) core for the pygx light tier. - -A faithful port of the heavy ``com.databricks.labs.gbx.gridx.grid.BNG`` Scala -object (gridx/grid/BNG.scala). No PyPI BNG library exists; this module reproduces -the codec, cell geometry, neighborhood walks, polyfill, and tessellation EXACTLY -so light and heavy share bit-identical cell ids and cell sets. - -Coordinates are EPSG:27700 eastings/northings. Cell ids are STRING in the public -surface (``format``/``parse`` round-trip a Long digit-id internally). Geometry is -emitted as plain WKB (NO SRID) and WKT, matching heavy ``JTS.toWKB``/``toWKT`` -(BNG does NOT stamp an SRID, unlike quadbin which is EWKB SRID 4326). - -Mosaic-lineage bug status (validated against this port): - * mosaic#434 (fixed upstream in mosaic#580): 100km NE/NW/SE/SW cells are res 1, - not quadrant resolutions. Carried here by the digit-length + trailing-quadrant - logic in ``get_resolution``/``format`` (a 100km id is 6 digits, trailing 0). - * mosaic#423: grid-aligned polygons must not emit POINT/LINESTRING chips. Carried - here by the input-geometry-type filter on border chips in ``tessellate``. -""" - -import math -from typing import Any - -CRS_ID = 27700 -NAME = "BNG" - -QUADRANTS = ["", "SW", "NW", "NE", "SE"] - -RESOLUTION_MAP = { - "500km": -1, "100km": 1, "50km": -2, "10km": 2, "5km": -3, "1km": 3, - "500m": -4, "100m": 4, "50m": -5, "10m": 5, "5m": -6, "1m": 6, -} -SIZE_MAP = { - "500km": 500000, "100km": 100000, "50km": 50000, "10km": 10000, - "5km": 5000, "1km": 1000, "500m": 500, "100m": 100, "50m": 50, - "10m": 10, "5m": 5, "1m": 1, -} -RESOLUTIONS = {1, -1, 2, -2, 3, -3, 4, -4, 5, -5, 6, -6} - -LETTER_MAP = [ - ["SV", "SW", "SX", "SY", "SZ", "TV", "TW", "TX"], - ["SQ", "SR", "SS", "ST", "SU", "TQ", "TR", "TS"], - ["SL", "SM", "SN", "SO", "SP", "TL", "TM", "TN"], - ["SF", "SG", "SH", "SJ", "SK", "TF", "TG", "TH"], - ["SA", "SB", "SC", "SD", "SE", "TA", "TB", "TC"], - ["NV", "NW", "NX", "NY", "NZ", "OV", "OW", "OX"], - ["NQ", "NR", "NS", "NT", "NU", "OQ", "OR", "OS"], - ["NL", "NM", "NN", "NO", "NP", "OL", "OM", "ON"], - ["NF", "NG", "NH", "NJ", "NK", "OF", "OG", "OH"], - ["NA", "NB", "NC", "ND", "NE", "OA", "OB", "OC"], - ["HV", "HW", "HX", "HY", "HZ", "JV", "JW", "JX"], - ["HQ", "HR", "HS", "HT", "HU", "JQ", "JR", "JS"], - ["HL", "HM", "HN", "HO", "HP", "JL", "JM", "JN"], - ["HF", "HG", "HH", "HJ", "HK", "JF", "JG", "JH"], -] - - -def get_resolution(res: Any) -> int: - """Resolution Int (BNG.getResolution(Any)): Int index or resolutionMap string key. - - NEVER accepts metres-as-Int (e.g. 1000). A bare int must be in RESOLUTIONS. - """ - if isinstance(res, bool): - raise ValueError(f"BNG resolution not supported; found {res!r}") - if isinstance(res, int): - if res in RESOLUTIONS: - return res - raise ValueError( - f"BNG resolution index must be one of {sorted(RESOLUTIONS)} " - f"(1=100km..6=1m, negatives=quadrants); got {res}. " - "Metres-as-Int (e.g. 1000) is NOT a resolution." - ) - if isinstance(res, str) and res in RESOLUTION_MAP: - return RESOLUTION_MAP[res] - raise ValueError(f"BNG resolution not supported; found {res!r}") - - -def get_edge_size(resolution: int) -> int: - """Edge size (metres) for an Int resolution (BNG.getEdgeSize(Int)).""" - res_str = get_resolution_str(resolution) - return SIZE_MAP[res_str] - - -def get_resolution_str(resolution: int) -> str: - for k, v in RESOLUTION_MAP.items(): - if v == resolution: - return k - return "" - - -def cell_digits(cell_id: int) -> list: - """Cell id Long -> list of decimal digits (BNG.cellDigits).""" - return [int(c) for c in str(cell_id)] - - -def _safe_digit_index(digit_slice, max_idx: int) -> int: - s = "".join(str(d) for d in digit_slice) - n = 0 if s == "" else int(s) - return max(0, min(max_idx, n)) - - -def get_resolution_from_digits(digits) -> int: - """BNG.getResolution(Seq[Int]) — resolution implied by digit length + quadrant.""" - if len(digits) < 6: - return -1 # 500km - quadrant = digits[-1] - k = (len(digits) - 6) // 2 - return -(k + 2) if quadrant > 0 else k + 1 - - -def get_x(digits, edge_size: int) -> int: - n = len(digits) - k = (n - 6) // 2 - x_digits = digits[1:3] + digits[5 : 5 + k] - quadrant = digits[-1] - edge_adj = 2 * edge_size if quadrant > 0 else edge_size - x_offset = edge_size if quadrant in (3, 4) else 0 - return int("".join(str(d) for d in x_digits)) * edge_adj + x_offset - - -def get_y(digits, edge_size: int) -> int: - n = len(digits) - k = (n - 6) // 2 - y_digits = digits[3:5] + digits[5 + k : 5 + 2 * k] - quadrant = digits[-1] - edge_adj = 2 * edge_size if quadrant > 0 else edge_size - y_offset = edge_size if quadrant in (2, 3) else 0 - return int("".join(str(d) for d in y_digits)) * edge_adj + y_offset - - -def get_quadrant(resolution: int, eastings: float, northings: float, divisor: float) -> int: - if resolution < -1: - e_q = eastings / divisor - n_q = northings / divisor - e_dec = e_q - math.floor(e_q) - n_dec = n_q - math.floor(n_q) - if e_dec < 0.5 and n_dec < 0.5: - return 1 # SW - if e_dec < 0.5: - return 2 # NW - if n_dec < 0.5: - return 4 # SE - return 3 # NE - return 0 - - -def encode(e_letter, n_letter, e_bin, n_bin, quadrant, n_positions, resolution) -> int: - id_placeholder = 10 ** (5 + 2 * n_positions - 2) - e_letter_shift = 10 ** (3 + 2 * n_positions - 2) - n_letter_shift = 10 ** (1 + 2 * n_positions - 2) - e_shift = 10 ** n_positions - n_shift = 10 - if resolution == -1: - val = (id_placeholder + e_letter * e_letter_shift) / 100 + quadrant - else: - val = ( - id_placeholder - + e_letter * e_letter_shift - + n_letter * n_letter_shift - + e_bin * e_shift - + n_bin * n_shift - + quadrant - ) - return int(val) - - -def point_to_cell_id(eastings: float, northings: float, resolution: int) -> int: - if math.isnan(eastings) or math.isnan(northings): - raise ValueError("NaN coordinates are not supported.") - e_int = int(eastings) - n_int = int(northings) - e_letter = math.floor(e_int / 100000) - n_letter = math.floor(n_int / 100000) - if resolution < 0: - divisor = 10 ** (6 - abs(resolution) + 1) - else: - divisor = 10 ** (6 - resolution) - quadrant = get_quadrant(resolution, e_int, n_int, divisor) - n_positions = abs(resolution) if resolution >= -1 else abs(resolution) - 1 - e_bin = math.floor((e_int % 100000) / divisor) - n_bin = math.floor((n_int % 100000) / divisor) - return encode(e_letter, n_letter, e_bin, n_bin, quadrant, n_positions, resolution) - - -def format(cell_id: int) -> str: - digits = cell_digits(cell_id) - if len(digits) < 6: - x_idx = _safe_digit_index(digits[3:5], len(LETTER_MAP) - 1) - y_idx = _safe_digit_index(digits[1:3], len(LETTER_MAP[0]) - 1) - return LETTER_MAP[x_idx][y_idx][0] - q_idx = max(0, min(len(QUADRANTS) - 1, digits[-1])) - x_idx = _safe_digit_index(digits[3:5], len(LETTER_MAP) - 1) - y_idx = _safe_digit_index(digits[1:3], len(LETTER_MAP[0]) - 1) - prefix = LETTER_MAP[x_idx][y_idx] - coords = digits[5:-1] - k = len(coords) // 2 - if not coords: - x_str = y_str = "" - else: - x_str = "".join(str(d) for d in (coords[:k] + [0] * (k - len(coords[:k])))) - y_str = "".join(str(d) for d in (coords[k : 2 * k] + [0] * (k - len(coords[k : 2 * k])))) - return f"{prefix}{x_str}{y_str}{QUADRANTS[q_idx]}" - - -def parse(cell_id: str) -> int: - prefix = cell_id[:2] if len(cell_id) >= 2 else f"{cell_id}V" - letter_row = next(row for row in LETTER_MAP if prefix in row) - e_letter = letter_row.index(prefix) - n_letter = LETTER_MAP.index(letter_row) - if len(cell_id) == 1: - return encode(e_letter, 0, 0, 0, 0, 1, -1) - suffix = cell_id[-2:] - quadrant = QUADRANTS.index(suffix) if (suffix in QUADRANTS and len(cell_id) > 2) else 0 - bin_digits = cell_id[2:-2] if quadrant > 0 else cell_id[2:] - if not bin_digits: - return encode(e_letter, n_letter, 0, 0, quadrant, 1, -2) - half = len(bin_digits) // 2 - e_bin = int(bin_digits[:half] or "0") - n_bin = int(bin_digits[half:] or "0") - n_positions = half + 1 - resolution = (n_positions + 1) if quadrant == 0 else -n_positions - return encode(e_letter, n_letter, e_bin, n_bin, quadrant, n_positions, resolution) -``` -*(Verify `format("TQ3080")`-family and the `_safe_digit_index` padding behavior in Step 4; the `format` `coords` padding mirrors `padTo(k, 0)` — pad the right side to length k.)* - -- [ ] **Step 4: run → PASS** (codec tests). - -- [ ] **Step 5: commit** (`feat(pygx): BNG codec port (resolutionMap/letterMap/encode/format/parse)`). - ---- - -## Task 2: `_bng.py` cell math — pointascell/eastnorthasbng, cellarea, distance, euclideandistance - -**Files:** modify `pygx/_bng.py`, create `test/pygx/test_bng_cellmath.py`. - -Add `area`, `distance`, `euclidean_distance`, and convenience wrappers `point_as_cell`/`east_north_as_bng` (returning the STRING cellid) ported from `BNG.area`/`BNG.distance`/`BNG.euclideanDistance`. `distance` is Manhattan in edge-size units at the **min** of the two resolutions; `euclidean_distance` is Chebyshev (max of dx, dy) / edge_size, also at the min resolution. - -- [ ] **Step 1: failing tests** `test/pygx/test_bng_cellmath.py` -```python -import pytest - -shapely = pytest.importorskip("shapely") -from databricks.labs.gbx.pygx import _bng - - -def test_cellarea_100km_is_10000_sqkm(): - # 100km cell -> (100000/1000)^2 = 10000 km^2. - cid = _bng.parse("NE") # a 100km grid square - assert _bng.area(cid) == pytest.approx(10000.0) - - -def test_cellarea_1km_is_1_sqkm(): - cid = _bng.parse("TQ3080") # a 1km cell - assert _bng.area(cid) == pytest.approx(1.0) - - -def test_east_north_as_bng_string_and_int_res_agree(): - s_int = _bng.east_north_as_bng(530000.0, 180000.0, 3) - s_str = _bng.east_north_as_bng(530000.0, 180000.0, "1km") - assert s_int == s_str == "TQ3080" - - -def test_distance_manhattan_one_cell_east(): - a = _bng.east_north_as_bng(530000.0, 180000.0, "1km") # TQ3080 - b = _bng.east_north_as_bng(531000.0, 180000.0, "1km") # one cell east - assert _bng.distance(_bng.parse(a), _bng.parse(b)) == 1 - - -def test_euclidean_distance_diagonal_is_one(): - a = _bng.east_north_as_bng(530000.0, 180000.0, "1km") - b = _bng.east_north_as_bng(531000.0, 181000.0, "1km") # diagonal neighbour - # Manhattan would be 2; Chebyshev (max) is 1. - assert _bng.euclidean_distance(_bng.parse(a), _bng.parse(b)) == 1 - assert _bng.distance(_bng.parse(a), _bng.parse(b)) == 2 -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** (append to `_bng.py`). -```python -def area(cell_id: int) -> float: - """Cell area in square KILOMETRES (BNG.area): (edgeSize/1000)^2.""" - resolution = get_resolution_from_digits(cell_digits(cell_id)) - edge = float(get_edge_size(resolution)) - return (edge / 1000.0) ** 2 - - -def distance(cell_id: int, cell_id2: int) -> int: - """Manhattan grid distance in edge-size units (BNG.distance).""" - d1, d2 = cell_digits(cell_id), cell_digits(cell_id2) - edge = get_edge_size(min(get_resolution_from_digits(d1), get_resolution_from_digits(d2))) - x1, x2 = get_x(d1, edge), get_x(d2, edge) - y1, y2 = get_y(d1, edge), get_y(d2, edge) - return abs((x1 - x2) // edge) + abs((y1 - y2) // edge) - - -def euclidean_distance(cell_id: int, cell_id2: int) -> int: - """Chebyshev (max of dx, dy) grid distance in edge-size units (BNG.euclideanDistance).""" - d1, d2 = cell_digits(cell_id), cell_digits(cell_id2) - edge = get_edge_size(min(get_resolution_from_digits(d1), get_resolution_from_digits(d2))) - x1, x2 = get_x(d1, edge), get_x(d2, edge) - y1, y2 = get_y(d1, edge), get_y(d2, edge) - return max(abs(x1 - x2), abs(y1 - y2)) // edge - - -def point_as_cell(eastings: float, northings: float, resolution) -> str: - """EPSG:27700 (eastings, northings) -> STRING cellid (BNG_EastNorthAsBNG core).""" - res = get_resolution(resolution) - return format(point_to_cell_id(float(eastings), float(northings), res)) - - -# east_north_as_bng is an alias of point_as_cell at the coordinate level; the SQL -# split (pointascell takes a POINT geom, eastnorthasbng takes scalar e/n) happens -# in functions.py. Both call this. -east_north_as_bng = point_as_cell -``` -Note the Scala `distance`/`euclideanDistance` use integer division `/`; mirror with `//` on the metre deltas (deltas are exact multiples of `edge`). Confirm the `//` matches `math.abs((x1-x2)/edge)` for the negative case in Step 4 (Scala truncates toward zero; Python `//` floors — guard by dividing the `abs` first, as written above, so both are non-negative). - -- [ ] **Step 4: run → PASS.** - -- [ ] **Step 5: commit** (`feat(pygx): BNG cellarea/distance/euclideandistance + point_as_cell`). - ---- - -## Task 3: `_bng.py` cell→geometry — cell_id_to_geometry, aswkb/aswkt/centroid - -**Files:** modify `pygx/_bng.py`, `test/pygx/test_bng_cellmath.py`. - -Heavy emits **plain WKB (no SRID)** via `JTS.toWKB` and plain WKT via `JTS.toWKT`. The cell polygon is the closed ring `(x,y),(x+e,y),(x+e,y+e),(x,y+e),(x,y)` from `getX`/`getY`/`getEdgeSize`. Use shapely `box(x, y, x+e, y+e)` (NOT `set_srid`). - -- [ ] **Step 1: failing tests** (append) -```python -from shapely import from_wkb, from_wkt, get_srid # noqa: E402 - - -def test_aswkb_is_wkb_polygon_no_srid(): - cid = _bng.parse("TQ3080") # 1km cell - g = from_wkb(_bng.cell_aswkb(cid)) - assert g.geom_type == "Polygon" - assert get_srid(g) == 0 # BNG WKB carries NO SRID (heavy uses toWKB, not toEWKB) - minx, miny, maxx, maxy = g.bounds - # 1km cell, easting bin 30 -> 530000, northing bin 80 -> 180000. - assert (minx, miny, maxx, maxy) == (530000.0, 180000.0, 531000.0, 181000.0) - - -def test_aswkt_is_polygon_text(): - cid = _bng.parse("TQ3080") - g = from_wkt(_bng.cell_aswkt(cid)) - assert g.geom_type == "Polygon" - - -def test_centroid_is_wkb_point_no_srid(): - cid = _bng.parse("TQ3080") - g = from_wkb(_bng.cell_centroid(cid)) - assert g.geom_type == "Point" and get_srid(g) == 0 - assert (g.x, g.y) == (530500.0, 180500.0) # cell centre -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** (append to `_bng.py`; import `box`, `to_wkb` from shapely at top of file). -```python -from shapely import to_wkb as _to_wkb # at top of file -from shapely.geometry import box as _box - - -def cell_id_to_geometry(cell_id: int): - """Closed BNG cell polygon (shapely), NO SRID (BNG.cellIdToGeometry).""" - digits = cell_digits(cell_id) - resolution = get_resolution_from_digits(digits) - edge = get_edge_size(resolution) - x = get_x(digits, edge) - y = get_y(digits, edge) - return _box(x, y, x + edge, y + edge) - - -def cell_aswkb(cell_id: int) -> bytes: - return _to_wkb(cell_id_to_geometry(cell_id)) # include_srid defaults False -> no SRID - - -def cell_aswkt(cell_id: int) -> str: - return cell_id_to_geometry(cell_id).wkt - - -def cell_centroid(cell_id: int) -> bytes: - return _to_wkb(cell_id_to_geometry(cell_id).centroid) -``` - -- [ ] **Step 4: run → PASS.** - -- [ ] **Step 5: commit** (`feat(pygx): BNG cell geometry — aswkb/aswkt/centroid (WKB, no SRID)`). - ---- - -## Task 4: `_bng.py` neighborhood — k_ring, k_loop - -**Files:** modify `pygx/_bng.py`, `test/pygx/test_bng_cellmath.py`. - -Port `BNG.kRing` and `BNG.kLoop`. `k_loop(cellId, k)` walks the hollow square ring at radius `k*edgeSize` (corners + the four edge runs, `pointToCellID` each); `k_ring(cellId, n)` = center + union of `k_loop(.,k)` for k in 1..n. Return STRING cellids via `format`. - -- [ ] **Step 1: failing tests** (append) -```python -def test_kring_k1_is_nine_cells_incl_center(): - cid_s = _bng.east_north_as_bng(530000.0, 180000.0, "1km") # TQ3080 - ring = _bng.k_ring_str(cid_s, 1) - assert cid_s in ring - assert len(set(ring)) == 9 # 3x3 block - - -def test_kloop_k1_is_eight_cells_excl_center(): - cid_s = _bng.east_north_as_bng(530000.0, 180000.0, "1km") - loop = _bng.k_loop_str(cid_s, 1) - assert cid_s not in loop - assert len(set(loop)) == 8 - - -def test_kring_contains_all_kloops(): - cid_s = _bng.east_north_as_bng(530000.0, 180000.0, "1km") - ring2 = set(_bng.k_ring_str(cid_s, 2)) - loop1 = set(_bng.k_loop_str(cid_s, 1)) - loop2 = set(_bng.k_loop_str(cid_s, 2)) - assert loop1 <= ring2 and loop2 <= ring2 - assert cid_s in ring2 -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** (append). Operate on the Long id internally; the `_str` wrappers `parse`→walk→`format`. -```python -def k_loop(cell_id: int, k: int) -> list: - """Hollow square ring of Long cell ids at radius k (BNG.kLoop).""" - digits = cell_digits(cell_id) - resolution = get_resolution_from_digits(digits) - edge = get_edge_size(resolution) - x = get_x(digits, edge) - y = get_y(digits, edge) - xmin, xmax = x - k * edge, x + k * edge - ymin, ymax = y - k * edge, y + k * edge - pts = [(xmin, ymin), (xmin, ymax), (xmax, ymax), (xmax, ymin)] - pts += [(xmin, yy) for yy in range(ymin + edge, ymax, edge)] # left - pts += [(xmax, yy) for yy in range(ymin + edge, ymax, edge)] # right - pts += [(xx, ymax) for xx in range(xmin + edge, xmax, edge)] # up - pts += [(xx, ymin) for xx in range(xmin + edge, xmax, edge)] # down - return [point_to_cell_id(px, py, resolution) for (px, py) in pts] - - -def k_ring(cell_id: int, n: int) -> list: - """Center + all k-loops 1..n of Long cell ids (BNG.kRing).""" - if n == 1: - return [cell_id] + k_loop(cell_id, 1) - out = [cell_id] - for k in range(1, n + 1): - out += k_loop(cell_id, k) - return out - - -def k_ring_str(cell_id_str: str, n: int) -> list: - return [format(c) for c in k_ring(parse(cell_id_str), int(n))] - - -def k_loop_str(cell_id_str: str, k: int) -> list: - return [format(c) for c in k_loop(parse(cell_id_str), int(k))] -``` -Note: the Scala edge runs use `until ... by edgeSize` (exclusive upper bound). Python `range(start, stop, step)` is also exclusive — matches. Confirm the corner+edge cell-set equals heavy's in the parity test (Task 9). - -- [ ] **Step 4: run → PASS.** - -- [ ] **Step 5: commit** (`feat(pygx): BNG kring/kloop square-grid neighborhood walks`). - ---- - -## Task 5: `_bng.py` coverage — polyfill (centroid BFS flood-fill) - -**Files:** modify `pygx/_bng.py`, `test/pygx/test_bng_cellmath.py`. - -Port `BNG.polyfill`: seed the queue with the cell ids of all geometry coordinates + the centroid; BFS via `kLoop(.,1)`; a cell is included iff its **centroid** is `contains`-ed by the geometry (shapely `geometry.contains(cell_centroid_point)`). Return STRING cellids. Input via `parse_geom` (WKB/EWKB/WKT/EWKT). - -- [ ] **Step 1: failing tests** (append) -```python -from shapely.geometry import box as _box2 # noqa: E402 -from shapely import to_wkb as _towkb # noqa: E402 - - -def test_polyfill_small_box_1km(): - # A 3km x 3km box aligned to the 1km grid around TQ3080. - geom = _towkb(_box2(530000.0, 180000.0, 533000.0, 183000.0)) - cells = _bng.polyfill_str(geom, _bng.get_resolution("1km")) - assert len(cells) > 0 - assert all(isinstance(c, str) for c in cells) - # Cells are 1km (6-char TQ#### form). - assert all(c.startswith("TQ") for c in cells) - - -def test_polyfill_empty_geom_is_empty(): - assert _bng.polyfill_str(None, 3) == [] -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** (append; import `parse_geom` from `._geom`). -```python -from ._geom import parse_geom # at top of file -from shapely.geometry import Point as _Point - - -def polyfill(geometry, resolution: int) -> list: - """Long cell ids whose centroid is contained by the geometry (BNG.polyfill BFS).""" - if geometry is None or geometry.is_empty: - return [] - coords = list(geometry.exterior.coords) if geometry.geom_type == "Polygon" else list( - geometry.coords if hasattr(geometry, "coords") else [] - ) - # Fall back to representative + all boundary coords via shapely .coords across parts. - seeds = _seed_coords(geometry) + [(geometry.centroid.x, geometry.centroid.y)] - queue = [point_to_cell_id(px, py, resolution) for (px, py) in seeds] - visited = set() - out = [] - while queue: - current = queue.pop(0) - if current in visited: - continue - visited.add(current) - center = cell_id_to_geometry(current).centroid - if geometry.contains(_Point(center.x, center.y)): - out.append(current) - for nb in k_loop(current, 1): - if nb not in visited: - queue.append(nb) - return out - - -def _seed_coords(geometry) -> list: - """All vertex coords across any geometry type (mirrors geometry.getCoordinates).""" - from shapely.geometry import mapping - - pts = [] - - def _walk(obj): - if isinstance(obj, (list, tuple)): - if obj and isinstance(obj[0], (int, float)): - pts.append((obj[0], obj[1])) - else: - for o in obj: - _walk(o) - - _walk(mapping(geometry).get("coordinates", [])) - return pts - - -def polyfill_str(geom, resolution) -> list: - res = get_resolution(resolution) - parsed = parse_geom(geom) - return [format(c) for c in polyfill(parsed, res)] -``` -*(Replace the placeholder `coords` line above with `_seed_coords` — shown for clarity; the implementation must use `_seed_coords(geometry) + centroid`, which handles Point/Line/Polygon/Multi* uniformly. Confirm the seed-coords + BFS yields the same cell set as heavy in Task 9 — that parity test is the definition of done for the flood-fill.)* - -- [ ] **Step 4: run → PASS.** - -- [ ] **Step 5: commit** (`feat(pygx): BNG polyfill (centroid BFS flood-fill from boundary coords)`). - ---- - -## Task 6: `_bng.py` tessellation — tessellate (core/border split, #423 chip filter) - -**Files:** modify `pygx/_bng.py`, `test/pygx/test_bng_cellmath.py`. - -Port `BNG.tessellate`: buffer-erode the geometry by `radius = edgeSize*sqrt(2)/2` to get the carved core; polyfill the carved geometry → core cells; polyfill a buffered boundary (`radius*1.01`, simplified) → border candidates minus core; per border cell intersect `cell_id_to_geometry` with the input geom, use `equals_exact(cell_geom, 0.1)` to promote a near-full chip to core, else keep the clipped chip; **filter all chips to `chip is None or chip.geom_type == input.geom_type`** (the #423 fix). Emit `(cellid_str, core_bool, chip_wkb_or_None)`. - -- [ ] **Step 1: failing tests** (append) -```python -def test_tessellate_box_has_core_and_border(): - # 5km box on the 1km grid -> interior core cells + clipped border cells. - geom = _towkb(_box2(530000.0, 180000.0, 535000.0, 185000.0)) - chips = _bng.tessellate_str(geom, _bng.get_resolution("1km")) - assert len(chips) > 0 - cores = [c for c in chips if c[1]] - borders = [c for c in chips if not c[1]] - assert cores and borders - # core chip geom is None (keepCoreGeom false default for the array form) - # border chips carry a WKB polygon. - for cell, core, chip in borders: - assert chip is not None - g = from_wkb(chip) - assert g.geom_type in ("Polygon", "MultiPolygon") - - -def test_tessellate_grid_aligned_no_degenerate_chips(): - # mosaic#423: a polygon aligned exactly to the 1km grid must NOT emit - # POINT/LINESTRING chips at shared edges. Box edges land on 1km lines. - geom = _towkb(_box2(530000.0, 180000.0, 533000.0, 183000.0)) - chips = _bng.tessellate_str(geom, _bng.get_resolution("1km")) - for cell, core, chip in chips: - if chip is not None: - g = from_wkb(chip) - assert g.geom_type in ("Polygon", "MultiPolygon"), ( - f"degenerate chip {g.geom_type} for {cell} (mosaic#423)" - ) -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** (append). Reference mosaic#423 in the comment on the type filter. -```python -def get_buffer_radius(resolution: int) -> float: - return get_edge_size(resolution) * math.sqrt(2) / 2.0 - - -def tessellate(geometry, resolution: int, keep_core_geom: bool = False) -> list: - """List of (Long cellid, core: bool, chip shapely|None) (BNG.tessellate).""" - if geometry is None or geometry.is_empty: - return [] - in_type = geometry.geom_type - radius = get_buffer_radius(resolution) - carved = geometry.buffer(-radius) - if carved.is_empty: - border_geom = geometry.buffer(radius * 1.01).simplify(0.01 * radius) - else: - border_geom = geometry.boundary.buffer(radius * 1.01).simplify(0.01 * radius) - - core_set = set(polyfill(carved, resolution)) - border = [c for c in polyfill(border_geom, resolution) if c not in core_set] - - chips = [] - for cell in core_set: - chips.append((cell, True, cell_id_to_geometry(cell) if keep_core_geom else None)) - for cell in border: - cell_geom = cell_id_to_geometry(cell) - inter = cell_geom.intersection(geometry) - if inter.is_empty: - continue - if inter.geom_type == "GeometryCollection": - inter = inter.difference(cell_geom.boundary) - is_core = inter.equals_exact(cell_geom, 0.1) # 0.1 m tolerance (heavy parity) - if is_core: - chips.append((cell, True, cell_geom if keep_core_geom else None)) - else: - chips.append((cell, False, inter)) - # mosaic#423: drop degenerate (non-areal) chips by keeping only chips whose - # type matches the input geometry type (heavy: getGeometryType == inGeomType). - return [ - (c, core, chip) - for (c, core, chip) in chips - if chip is None or chip.geom_type == in_type - ] - - -def tessellate_str(geom, resolution, keep_core_geom: bool = False) -> list: - res = get_resolution(resolution) - parsed = parse_geom(geom) - return [ - (format(c), core, (_to_wkb(chip) if chip is not None else None)) - for (c, core, chip) in tessellate(parsed, res, keep_core_geom) - ] -``` - -- [ ] **Step 4: run → PASS.** - -- [ ] **Step 5: commit** (`feat(pygx): BNG tessellate (core/border split + mosaic#423 chip-type filter)`). - ---- - -## Task 7: `_bng.py` geometry-centric neighborhood + chip ops — geomkring/geomkloop, cellintersection/cellunion - -**Files:** modify `pygx/_bng.py`, `test/pygx/test_bng_cellmath.py`. - -Port `BNG.geometryKRing`/`geometryKLoop` (depend on `getChips` → for Polygon use `tessellate`; the chips split into core IDs + border cells, then border per-cell `kRing`/`kLoop`, filtered by `isValid`). For the lightweight tier, `getChips` on Polygon/MultiPolygon = `tessellate(geom, res, keepCoreGeom=False)`; Point/MultiPoint = `pointToCellID` per point (core=false); Line/MultiLine = `lineFill` (`lineDecompose` BFS walk along the line via `kRing`). Also port `is_valid` (bounds check) and the two chip-pair ops `cell_intersection`/`cell_union` (operate on `(cellid, core, chip)` tuples; left-hand rule, same-cell only). - -- [ ] **Step 1: failing tests** (append) -```python -def test_geomkring_box_superset_of_polyfill(): - geom = _towkb(_box2(530000.0, 180000.0, 533000.0, 183000.0)) - res = _bng.get_resolution("1km") - fill = set(_bng.polyfill_str(geom, res)) - gkr = set(_bng.geometry_k_ring_str(geom, res, 1)) - # k-ring around the geometry includes (at least) the tessellated coverage. - assert fill <= gkr or len(gkr) >= len(fill) - - -def test_geomkloop_excludes_inner_ring(): - geom = _towkb(_box2(530000.0, 180000.0, 535000.0, 185000.0)) - res = _bng.get_resolution("1km") - gkr1 = set(_bng.geometry_k_ring_str(geom, res, 1)) - gkl2 = set(_bng.geometry_k_loop_str(geom, res, 2)) - # k-loop at 2 is disjoint from the k-ring at 1 (hollow outer ring). - assert gkl2.isdisjoint(gkr1) or len(gkl2) > 0 - - -def test_cell_union_same_cell_merges_chips(): - cid_s = _bng.east_north_as_bng(530000.0, 180000.0, "1km") - cid = _bng.parse(cid_s) - full = _bng.cell_id_to_geometry(cid) - left = (cid_s, False, full.buffer(0).intersection(_box2(530000, 180000, 530500, 181000))) - right = (cid_s, False, full.buffer(0).intersection(_box2(530500, 180000, 531000, 181000))) - cell, core, chip = _bng.cell_union(left, right) - assert cell == cid_s - assert chip.equals_exact(full, 0.5) or chip.area == pytest.approx(full.area, rel=1e-6) - - -def test_cell_intersection_different_cells_is_empty(): - a = (_bng.east_north_as_bng(530000.0, 180000.0, "1km"), False, _box2(530000, 180000, 531000, 181000)) - b = (_bng.east_north_as_bng(540000.0, 180000.0, "1km"), False, _box2(540000, 180000, 541000, 181000)) - cell, core, chip = _bng.cell_intersection(a, b) - assert chip.is_empty -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** (append). Port `is_valid`, `get_chips`, `point_chip`, `multi_point_chips`, `line_fill`/`line_decompose`, `geometry_k_ring`/`geometry_k_loop`, `cell_intersection`/`cell_union`, plus the `_str`/wkb wrappers used by the UDFs. The chip ops take `(cellid_str, core, shapely_or_None)` and return the same. `cell_union`/`cell_intersection` left-hand rule: different cell ids → empty polygon; if either chip is core → that chip; else `.union()` / `.intersection()`. -```python -def is_valid(cell_id: int) -> bool: - digits = cell_digits(cell_id) - x_letter = int("".join(str(d) for d in digits[3:5])) - y_letter = int("".join(str(d) for d in digits[1:3])) - resolution = get_resolution_from_digits(digits) - edge = get_edge_size(resolution) - x = get_x(digits, edge) - y = get_y(digits, edge) - return ( - 0 <= x <= 700000 and 0 <= y <= 1300000 - and x_letter < len(LETTER_MAP) and y_letter < len(LETTER_MAP[0]) - ) - - -def get_chips(geometry, resolution: int): - """(Long cellid, core, shapely|None) per BNG.getChips, dispatched on geom type.""" - gt = geometry.geom_type - if gt == "Point": - return [(point_to_cell_id(geometry.x, geometry.y, resolution), False, None)] - if gt == "MultiPoint": - return [ - (point_to_cell_id(p.x, p.y, resolution), False, None) for p in geometry.geoms - ] - if gt in ("LineString", "MultiLineString"): - return _line_fill(geometry, resolution) - return [(c, core, chip) for (c, core, chip) in tessellate(geometry, resolution, False)] - - -def _line_fill(geometry, resolution: int): - lines = [geometry] if geometry.geom_type == "LineString" else list(geometry.geoms) - out = [] - for line in lines: - out += _line_decompose(line, resolution) - return out - - -def _line_decompose(line, resolution: int): - """BFS walk along a LineString, yielding (cellid, False, segment) (BNG.lineDecompose).""" - start = line.coords[0] - start_cell = point_to_cell_id(start[0], start[1], resolution) - queue = [start_cell] - traversed = set() - chips = [] - while queue: - traversed |= set(queue) - next_queue = [] - for current in queue: - cell_geom = cell_id_to_geometry(current) - seg = line.intersection(cell_geom) - if not seg.is_empty: - chips.append((current, False, seg)) - for nb in k_ring(current, 1): - if nb not in traversed and nb not in next_queue: - next_queue.append(nb) - elif len(traversed) == 1: - for nb in k_ring(current, 1): - if nb not in traversed: - next_queue.append(nb) - queue = next_queue - return chips - - -def geometry_k_ring(geometry, resolution: int, k: int) -> set: - chips = get_chips(geometry, resolution) - core_ids = {c for (c, core, _) in chips if core} - border = [c for (c, core, _) in chips if not core] - border_kring = {x for c in border for x in k_ring(c, k)} - return {c for c in (core_ids | border_kring) if is_valid(c)} - - -def geometry_k_loop(geometry, resolution: int, k: int) -> set: - n = k - 1 - chips = get_chips(geometry, resolution) - core_ids = {c for (c, core, _) in chips if core} - border = [c for (c, core, _) in chips if not core] - border_nring = {x for c in border for x in k_ring(c, n)} - n_ring = core_ids | border_nring - border_kloop = {x for c in border for x in k_loop(c, k)} - return {c for c in (border_kloop - n_ring) if is_valid(c)} - - -def geometry_k_ring_str(geom, resolution, k) -> list: - res = get_resolution(resolution) - return [format(c) for c in geometry_k_ring(parse_geom(geom), res, int(k))] - - -def geometry_k_loop_str(geom, resolution, k) -> list: - res = get_resolution(resolution) - return [format(c) for c in geometry_k_loop(parse_geom(geom), res, int(k))] - - -from shapely.geometry import Polygon as _Polygon # for empty polygon - - -def _empty_polygon(): - return _Polygon() - - -def cell_union(left, right): - """Union two chips (cellid_str, core, shapely) with the left-hand rule (BNG_CellUnion).""" - lc, lcore, lg = left - rc, rcore, rg = right - if lc != rc: - return (lc, lcore, _empty_polygon()) - if lcore: - return left - if rcore: - return right - return (lc, lcore, lg.union(rg)) - - -def cell_intersection(left, right): - """Intersect two chips with the left-hand rule (BNG_CellIntersection).""" - lc, lcore, lg = left - rc, rcore, rg = right - if lc != rc: - return (lc, lcore, _empty_polygon()) - if lcore: - return left - if rcore: - return right - return (lc, lcore, lg.intersection(rg)) -``` - -- [ ] **Step 4: run → PASS.** - -- [ ] **Step 5: commit** (`feat(pygx): BNG geomkring/geomkloop + chip cellunion/cellintersection`). - ---- - -## Task 8: `functions.py` — register all 23 BNG functions (UDF + UDTF + agg) + Column wrappers - -**Files:** modify `pygx/functions.py`, `pygx/_serde.py`, `pygx/_env.py`, create `test/pygx/test_bng_udf.py`. - -Add `BNG_CHIP_SCHEMA` to `_serde.py`; `assert_bng_available()` to `_env.py` (shapely only). In `functions.py`: add BNG UDFs/UDTFs/aggs and Column wrappers, and extend `register(spark)` (or add a sibling that the existing `register` calls) to install all 23 `gbx_bng_*` names. Apply the impl rule: scalar/bounded → pandas_udf; array → plain `@udf`; explode → `@udtf`; grouped-agg → pandas_udf returning STRUCT. - -- [ ] **Step 1: failing test** `test/pygx/test_bng_udf.py` -```python -import pytest - -shapely = pytest.importorskip("shapely") -from shapely import from_wkb, get_srid, to_wkb # noqa: E402 -from shapely.geometry import box # noqa: E402 - -from databricks.labs.gbx.pygx import functions as gx - - -def test_eastnorthasbng_and_cellarea(spark): - gx.register(spark) - row = spark.sql( - "SELECT gbx_bng_eastnorthasbng(530000.0, 180000.0, '1km') AS c" - ).collect()[0] - assert row["c"] == "TQ3080" - a = spark.sql("SELECT gbx_bng_cellarea('TQ3080') AS a").collect()[0] - assert a["a"] == pytest.approx(1.0) - - -def test_pointascell_wkt_uses_bng_coords(spark): - gx.register(spark) - # EPSG:27700 eastings/northings (NOT WGS84). London at 1km. - row = spark.sql( - "SELECT gbx_bng_pointascell('POINT(530000 180000)', '1km') AS c" - ).collect()[0] - assert row["c"] == "TQ3080" - - -def test_aswkb_no_srid_polygon(spark): - gx.register(spark) - out = spark.sql("SELECT gbx_bng_aswkb('TQ3080') AS w").collect()[0] - g = from_wkb(bytes(out["w"])) - assert g.geom_type == "Polygon" and get_srid(g) == 0 - - -def test_kring_array_of_strings(spark): - gx.register(spark) - rows = spark.sql("SELECT gbx_bng_kring('TQ3080', 1) AS r").collect()[0] - assert "TQ3080" in rows["r"] and len(set(rows["r"])) == 9 - - -def test_polyfill_and_tessellate(spark): - gx.register(spark) - df = spark.createDataFrame( - [(bytearray(to_wkb(box(530000.0, 180000.0, 533000.0, 183000.0))),)], "g binary" - ) - df.createOrReplaceTempView("bv") - pf = spark.sql("SELECT size(gbx_bng_polyfill(g, 3)) AS n FROM bv").collect()[0] - assert pf["n"] > 0 - chips = spark.sql( - "SELECT t.cellid, t.core, t.chip " - "FROM bv LATERAL VIEW explode(gbx_bng_tessellate(g, 3)) AS t" - ).collect() - assert len(chips) > 0 and isinstance(chips[0]["cellid"], str) - - -def test_kringexplode_udtf(spark): - gx.register(spark) - rows = spark.sql( - "SELECT cellid FROM gbx_bng_kringexplode('TQ3080', 1)" - ).collect() - assert len(rows) == 9 and all(isinstance(r["cellid"], str) for r in rows) - - -def test_cellunion_agg(spark): - gx.register(spark) - df = spark.createDataFrame( - [(bytearray(to_wkb(box(530000.0, 180000.0, 535000.0, 185000.0))),)], "g binary" - ) - df.createOrReplaceTempView("bv2") - # explode tessellate chips, then dissolve per cell via the agg. - out = spark.sql( - "WITH chips AS (SELECT t.* FROM bv2 LATERAL VIEW explode(gbx_bng_tessellate(g,3)) AS t) " - "SELECT gbx_bng_cellunion_agg(struct(cellid, core, chip)) AS u FROM chips GROUP BY cellid" - ).collect() - assert len(out) > 0 -``` - -- [ ] **Step 2: run → FAIL** (`bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pygx/test_bng_udf.py --log bng-udf.log`). - -- [ ] **Step 3: implement.** - - `_serde.py` add: -```python -from pyspark.sql.types import BooleanType, StringType # extend imports - -BNG_CHIP_SCHEMA = StructType( - [ - StructField("cellid", StringType(), False), - StructField("core", BooleanType(), False), - StructField("chip", BinaryType(), True), - ] -) -``` - - `_env.py` add: -```python -def assert_bng_available() -> None: - """Raise a clear ImportError if shapely (the only pygx BNG dep) is missing.""" - try: - import shapely # noqa: F401 - except Exception: # noqa: BLE001 - raise ImportError( - "pygx BNG requires the [light] extra (shapely). " - "Install with: pip install 'geobrix[light]'" - ) -``` - - `functions.py` add BNG section. Scalar pandas_udfs wrap `_bng` scalar fns; array plain `@udf`s; `@udtf` classes for the five explode fns; grouped-agg pandas_udfs returning `BNG_CHIP_SCHEMA`. Sketch (full impl, no placeholders — every fn registered): -```python -from . import _bng -from ._serde import BNG_CHIP_SCHEMA -from pyspark.sql.types import BooleanType, DoubleType, StringType -from pyspark.sql.functions import udtf - -# --- scalar / bounded-output -> pandas_udf ------------------------------------- -@pandas_udf(StringType()) -def _bng_pointascell_udf(geom: pd.Series, res: pd.Series) -> pd.Series: - out = [] - for g, r in zip(geom, res): - if g is None: - out.append(None); continue - parsed = _bng.parse_geom(g) if False else None # use _geom.parse_geom - from ._geom import parse_geom - pg = parse_geom(g) - c = pg.centroid - out.append(_bng.point_as_cell(c.x, c.y, _to_res(r))) - return pd.Series(out) - -# (similar pandas_udf wrappers for: eastnorthasbng (e,n,res); cellarea(cellid); -# distance(c1,c2); euclideandistance(c1,c2); aswkb(cellid); aswkt(cellid); -# centroid(cellid). All accept STRING cellids and call parse internally where the -# _bng fn expects a Long.) - -def _to_res(r): - # res may arrive as int (index) or str (resolutionMap key). _bng.get_resolution - # validates both and rejects metres-as-Int. - return r - -# --- chip-pair scalar ops -> pandas_udf returning BNG_CHIP_SCHEMA -------------- -# cellintersection / cellunion take two chip structs; loop the batch, call -# _bng.cell_union/_intersection on (cellid, core, parse_geom(chip)) tuples, -# re-encode chip to WKB. - -# --- array-output -> plain @udf ----------------------------------------------- -def _bng_kring(cellid, k): - if cellid is None or k is None: return None - return _bng.k_ring_str(cellid, int(k)) -def _bng_kloop(cellid, k): - if cellid is None or k is None: return None - return _bng.k_loop_str(cellid, int(k)) -def _bng_polyfill(geom, res): - if geom is None: return None - return _bng.polyfill_str(geom, res) -def _bng_geomkring(geom, res, k): - if geom is None: return None - return sorted(_bng.geometry_k_ring_str(geom, res, k)) -def _bng_geomkloop(geom, res, k): - if geom is None: return None - return sorted(_bng.geometry_k_loop_str(geom, res, k)) -def _bng_tessellate(geom, res): - if geom is None: return None - return [ - {"cellid": c, "core": core, "chip": chip} - for (c, core, chip) in _bng.tessellate_str(geom, res) - ] - -# --- explode UDTFs ------------------------------------------------------------ -@udtf(returnType="cellid: string") -class _BngKRingExplode: - def eval(self, cellid, k): - if cellid is None or k is None: return - for c in _bng.k_ring_str(cellid, int(k)): - yield (c,) -# (likewise _BngKLoopExplode, _BngGeomKRingExplode, _BngGeomKLoopExplode; -# _BngTessellateExplode returns "cellid: string, core: boolean, chip: binary".) - -# --- grouped-agg pandas_udf returning BNG_CHIP_SCHEMA ------------------------- -@pandas_udf(BNG_CHIP_SCHEMA) -def _bng_cellunion_agg_udf(chip: pd.Series): - # chip is a Series of struct rows {cellid, core, chip}. Fold via _bng.cell_union. - ... -@pandas_udf(BNG_CHIP_SCHEMA) -def _bng_cellintersection_agg_udf(chip: pd.Series): - ... -``` - Extend `register(spark)` (after the quadbin block) with all 23: -```python - _env.assert_bng_available() - # scalar -> pandas_udf - spark.udf.register("gbx_bng_pointascell", _bng_pointascell_udf) - spark.udf.register("gbx_bng_eastnorthasbng", _bng_eastnorthasbng_udf) - spark.udf.register("gbx_bng_cellarea", _bng_cellarea_udf) - spark.udf.register("gbx_bng_distance", _bng_distance_udf) - spark.udf.register("gbx_bng_euclideandistance", _bng_euclideandistance_udf) - spark.udf.register("gbx_bng_aswkb", _bng_aswkb_udf) - spark.udf.register("gbx_bng_aswkt", _bng_aswkt_udf) - spark.udf.register("gbx_bng_centroid", _bng_centroid_udf) - spark.udf.register("gbx_bng_cellintersection", _bng_cellintersection_udf) - spark.udf.register("gbx_bng_cellunion", _bng_cellunion_udf) - # array -> plain @udf - spark.udf.register("gbx_bng_kring", _bng_kring, ArrayType(StringType())) - spark.udf.register("gbx_bng_kloop", _bng_kloop, ArrayType(StringType())) - spark.udf.register("gbx_bng_polyfill", _bng_polyfill, ArrayType(StringType())) - spark.udf.register("gbx_bng_geomkring", _bng_geomkring, ArrayType(StringType())) - spark.udf.register("gbx_bng_geomkloop", _bng_geomkloop, ArrayType(StringType())) - spark.udf.register("gbx_bng_tessellate", _bng_tessellate, ArrayType(BNG_CHIP_SCHEMA)) - # explode -> UDTF - spark.udtf.register("gbx_bng_kringexplode", _BngKRingExplode) - spark.udtf.register("gbx_bng_kloopexplode", _BngKLoopExplode) - spark.udtf.register("gbx_bng_geomkringexplode", _BngGeomKRingExplode) - spark.udtf.register("gbx_bng_geomkloopexplode", _BngGeomKLoopExplode) - spark.udtf.register("gbx_bng_tessellateexplode", _BngTessellateExplode) - # grouped agg - spark.udf.register("gbx_bng_cellunion_agg", _bng_cellunion_agg_udf) - spark.udf.register("gbx_bng_cellintersection_agg", _bng_cellintersection_agg_udf) -``` - Then Column wrappers mirroring heavy `gridx.bng.functions` (and the quadbin wrapper style): `bng_pointascell(geom, res)`, `bng_eastnorthasbng(e, n, res)`, `bng_cellarea(cellid)`, `bng_distance`, `bng_euclideandistance`, `bng_aswkb`, `bng_aswkt`, `bng_centroid`, `bng_cellintersection`, `bng_cellunion`, `bng_kring`, `bng_kloop`, `bng_polyfill`, `bng_geomkring`, `bng_geomkloop`, `bng_tessellate`, `bng_cellunion_agg(chip)`, `bng_cellintersection_agg(chip)` — each via `f.call_function("gbx_bng_*", _col(...))`. The five `*explode` wrappers raise `NotImplementedError("gbx_bng_*explode is SQL-LATERAL-only in the light tier; use spark.sql(... LATERAL ...)")` (mirroring the pyvx pyramid contract); leave a docstring pointing to SQL. - -- [ ] **Step 4: run → PASS** (all `test_bng_udf.py`). Also run the Serverless guard test (`test/pyrx/test_serverless_no_spark_config.py` if it scans pygx; else grep-confirm no `_jvm`/`conf`/`.rdd` added to pygx). - -- [ ] **Step 5: commit** (`feat(pygx): register 23 BNG functions (udf + udtf + grouped-agg)`). - ---- - -## Task 9: cross-tier exact parity (JAR-gated) + #434/#423 lock-in - -**Files:** create `test/pygx/test_parity_bng.py`. - -- [ ] **Step 1: rebuild + stage the JAR** (heavy unchanged, but the parity test needs a JAR present): `bash scripts/commands/gbx-data-push-jar.sh` (stages both fat jar + tests.jar) and confirm `python/geobrix/lib/geobrix-0.4.0-jar-with-dependencies.jar` exists. (A present prior JAR suffices if heavy is unchanged.) - -- [ ] **Step 2: write the JAR-gated parity test** — copy the gating block from `test/pygx/test_parity_quadbin.py` (the `_JARS` glob on `parents[2]/"lib"`, the `spark_with_jar` fixture with the active-session skip + `appName="gbx-pygx-bng-parity"`). Register light then heavy under the SAME `gbx_bng_*` SQL names (`from databricks.labs.gbx.pygx import functions as gx; gx.register(spark)`; heavy via `from databricks.labs.gbx.gridx.bng import functions as hx; hx.register(spark)` — collect ALL light results first, then heavy overwrites the names, then collect heavy). Assert per function group, over a deterministic BNG fixture (EPSG:27700 coords: London `530000,180000` + a multi-cell box + the four 100km NE/NW/SE/SW cells): - - **Exact cell-ID / set**: `pointascell` & `eastnorthasbng` (same STRING), `cellarea` (same DOUBLE), `distance` / `euclideandistance` (same LONG), `kring` / `kloop` (sorted set equality), `polyfill` (sorted cell-set equality), `tessellate` (cellid set equality), `geomkring` / `geomkloop` (sorted set equality), and the five `*explode` UDTFs collected via `SELECT ... FROM gbx_bng_*explode(...)` vs heavy `LATERAL VIEW` (sorted cellid set equality). - - **Geometry WKB within 1e-6**: decode both tiers' WKB (`shapely.from_wkb`), assert `get_srid == 0` in BOTH (BNG carries no SRID), and `equals_exact(lg.normalize(), hg.normalize(), 1e-6)` for `aswkb`, `centroid`, surviving `tessellate` chips, `cellunion`/`cellintersection` and the two aggs. `aswkt` decoded via `from_wkt` and compared the same way. - - **mosaic#434 lock-in**: include the four 100km cells (`"NE","NW","SE","SW"`-prefix family — derive via `eastnorthasbng` at `"100km"` over points in those quadrants); assert `cellarea == 10000.0` km² in BOTH tiers and `aswkb` decodes to a 100km × 100km polygon (bounds span 100000 m) in BOTH. Comment-reference mosaic#434/#580. - - **mosaic#423 lock-in**: include a grid-aligned polygon (box on exact 1km lines, e.g. `box(530000,180000,533000,183000)`); assert NO chip in either tier decodes to POINT/LINESTRING, and the surviving chip cell-sets match. Comment-reference mosaic#423. - - **Contingency**: if any cell set diverges (digit-codec or BFS/tessellate ordering), fix the `_bng.py` port until exact — EXACT parity is the bar (no tolerance on cell IDs). - -- [ ] **Step 3: run in Docker** `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pygx/test_parity_bng.py --with-integration --log parity-bng.log` → green (skips only without JAR). - -- [ ] **Step 4: commit** (`test(pygx): light-vs-heavy BNG exact parity (cells + WKB) + #434/#423 lock-in`). - ---- - -## Task 10: function-info + binding parity - -**Files:** verify `docs/tests/python/api/gridx_functions_sql.py` (the `bng_*_sql_example()` functions exist — confirmed present), confirm all 23 `gbx_bng_*` Column wrappers exist in `functions.py` for import parity. - -- [ ] **Step 1:** confirm the 23 `gbx_bng_*` are in `docs/tests-function-info/registered_functions.txt` (they are — lines 111–133) and each has a `*_sql_example()` in `gridx_functions_sql.py` (they do). No new entries needed — this task verifies parity, not adds. Doc SQL examples must use BNG coords (`POINT(530000 180000)`, `530000, 180000`) and `cellarea` framed as km² — confirm the existing examples already do (they reference `'TQ3080'` and `gbx_bng_eastnorthasbng(530000, 180000, '1km')`). -- [ ] **Step 2:** run `bash scripts/commands/gbx-test-bindings.sh --log bindings-bng.log` → PASS (every `gbx_bng_*` present in Scala `override def name` + Python `functions.py` binding + `function-info.json` key). Fix upstream if it fails (a missing Python wrapper is the likely gap; the explode wrappers must still EXIST as importable names even though they raise `NotImplementedError`). -- [ ] **Step 3: commit** if any wrapper/example changed (`docs(pygx): BNG binding/function-info parity`); otherwise note "no change — parity already holds" and skip. - ---- - -## Task 11: bench harness — BNG legs + `--grid-bng-only` - -**Files:** modify `python/geobrix/src/databricks/labs/gbx/bench/` (`corpus_vector.py` add BNG generators; `readers.py` add `run_bng_*`; `cluster.py` add `_CELL_GRID_BNG` + `benchmark_grid_bng`/`grid_bng_only` config); `notebooks/tests/push_and_run_bench_on_cluster.py` (`--benchmark-grid-bng` / `--grid-bng-only` flags). - -- [ ] **Step 1:** add BNG corpus generators to `corpus_vector.py` mirroring the quadbin block (`generate_bng_points` → EPSG:27700 e/n points for `bng_pointascell`/`eastnorthasbng`; `generate_bng_polygons` → BNG-coord WKT polygons for `polyfill`/`tessellate`/`geomk*`; `generate_bng_cells` → single STRING cellids for `cellarea`/`aswkb`/`aswkt`/`centroid`/`kring`/`kloop`; `generate_bng_cell_pairs` → STRING cellid pairs for `distance`/`euclideandistance`). Cells computed via pure-Python `_bng` so both tiers consume identical inputs. -- [ ] **Step 2:** add `run_bng_pointascell` / `run_bng_polyfill` / `run_bng_tessellate` / `run_bng_kring` / `run_bng_cellunion_agg` (representative coverage: a scalar encode, a geom→array, a struct-array, a scalar cell-in array, an agg) to `readers.py`, mirroring `run_quadbin_*` signatures (same `(spark, run_id, warmup, measured, *, api, n_rows, res, where)` shape, `category="grid"`), with light-vs-heavy timing + **exact cell-set / decoded-geom parity** verdicts. Add a `_register_bng(spark, api)` helper paralleling `_register_quadbin`. -- [ ] **Step 3:** add `_CELL_GRID_BNG` to `cluster.py` (mirror `_CELL_GRID_QUADBIN`: light leg collected before heavy registration), wire `benchmark_grid_bng` / `grid_bng_only` config keys (parallel to `benchmark_grid_quadbin` / `grid_quadbin_only` at the `cfg.get(...)` sites + the `BENCHMARK_GRID_BNG`/`GRID_BNG_ONLY` template constants), and the launcher `--benchmark-grid-bng` / `--grid-bng-only` flags in `push_and_run_bench_on_cluster.py` (mirror the `--benchmark-grid-quadbin` / `--grid-quadbin-only` wiring at lines ~272–510 + the `run_id` suffix `-grid-bng` + the skip-fn-benchmarks branch). -- [ ] **Step 4: local smoke** at tiny scale in the `geobrix-dev` container (resolve SQL, both tiers run, parity verdicts PASS). Do NOT run the cluster here. -- [ ] **Step 5: commit** (`feat(bench): BNG light-vs-heavy bench legs (--grid-bng-only)`). - ---- - -## Task 12: cluster bench run - -- [ ] **Step 1:** controller-orchestrated (not a subagent): build+stage JAR+wheel to the sample-data Volume; restart the bench cluster (the standing 0519 bench cluster — restart, don't auto-terminate mid-iteration); poll libs INSTALLED; run `gbx:bench:cluster --grid-bng-only` once (pass `--row-counts 1000`; verify exactly one geobrix-bench run on the cluster's cluster_id); verify rows; fetch `summary.md` (give the user the `bench-out//summary.md` link, unprompted). -- [ ] **Step 2:** record the light-vs-heavy medians + exact-parity verdicts (for the docs in Task 13). No commit (bench writes to the Volume/table). Note any BNG function materially slower than heavy (per the perf-parity-blocker rule — the WKB-UDF-boundary tax applies to `aswkb`/`tessellate`/`cellunion`; cell-id STRING ops should be competitive). Terminate the cluster only if it was started FRESH for this run; if it was the standing 0519 cluster, suggest termination, don't auto-kill. - ---- - -## Task 13: docs — all surfaces (flip BNG heavy→both; keep custom heavy-only) - -**Files:** `docs/docs/api/gridx-functions.mdx`, `execution-tiers.mdx`, `performance.mdx`, `benchmarking.mdx`, `README.md`, `docs/src/pages/index.js`, `docs/docs/intro.mdx`; `function-info`. - -- [ ] **Step 1: `gridx-functions.mdx`** — flip the **BNG section** badges heavy→both: the section header (`## British National Grid (BNG)`, currently ` ... heavyweight-only`) becomes `` with a lightweight note ("Powered by a pure-Python port of the BNG codec + shapely; identical `gbx_bng_*` SQL names — a drop-in swap. Cell ids are STRING; geometry outputs are plain WKB (EPSG:27700 coordinates, no SRID)."), and EVERY per-function `### bng_*` `` → `` (all 23: aswkb, aswkt, cellarea, centroid, distance, euclideandistance, cellintersection, cellunion, eastnorthasbng, pointascell, kring, kloop, geomkring, geomkloop, polyfill, tessellate, cellintersection_agg, cellunion_agg, kringexplode, kloopexplode, geomkringexplode, geomkloopexplode, tessellateexplode). Update the top-of-page summary line (line 15) so BNG is no longer in the heavyweight-only list (leave **custom grids** heavy-only). KEEP the `## Custom Grid Functions` section + all `gbx_custom_*` at ``. Document the BNG resolution convention (Int index ±1..±6 or `resolutionMap` string; never metres-as-Int), the EPSG:27700 expectation for `pointascell`/`eastnorthasbng`, and that `cellarea` returns km². -- [ ] **Step 2: `execution-tiers.mdx`** — in the heavyweight-only reasons (lines 45/47), move BNG out (now both-tier) while KEEPING `gbx_custom_*` (custom grids) heavyweight-only. Update the GridX framing: "GridX BNG and quadbin are now available in both tiers; only custom grids remain heavyweight-only." -- [ ] **Step 3: `performance.mdx`** — extend the "GridX (pygx)" subsection (currently quadbin-only) with the BNG cell ops: a BNG execution-shapes table (scalar pandas-UDF cell math + the array `@udf` kring/kloop/polyfill/tessellate/geomk* + the `@udtf` explodes + the `cellunion_agg`/`cellintersection_agg` grouped-aggs), the `pygx/_bng.py` module row in the modules table (pure-Python BNG codec + shapely), and the perf narrative from the Task 12 numbers (note STRING cell-id ops competitive; WKB-geometry ops carry the UDF-boundary tax). -- [ ] **Step 4: `benchmarking.mdx`** — extend the **Grid tab** (currently quadbin-only, lines ~608–620) with a BNG subsection: light (pygx) vs heavy (gridx.bng) with exact-output parity across the representative shapes (scalar encode `gbx_bng_eastnorthasbng`, geom→cell-array `gbx_bng_polyfill`, geom→chip struct-array `gbx_bng_tessellate`, scalar cell-in `gbx_bng_kring`, grouped agg `gbx_bng_cellunion_agg`), with the Task 12 medians + parity verdicts, and the `gbx:bench:cluster --grid-bng-only` invocation. (Per the bench-changes-update-docs rule, the numbers go here in the same stroke as Task 12.) -- [ ] **Step 5: README / `index.js` / `intro.mdx`** — flip BNG to lightweight-available: - - `README.md` line 26: "GridX — BNG, Quadbin, and custom grids ... **BNG and Quadbin in both tiers** (lightweight `pygx` + heavyweight Scala); **custom grids** heavyweight." - - `docs/src/pages/index.js` GridX card (line 55) + the heavyweight-only line (lines 105–107): BNG + quadbin lightweight (pygx); only custom grids heavyweight. - - `docs/docs/intro.mdx` line 9: lightweight tier now covers RasterX, VectorX, and **GridX BNG + quadbin**; only GridX custom grids heavyweight-only. -- [ ] **Step 6:** `function-info` regen if any example changed (`bash scripts/commands/gbx-docs-function-info.sh`); `cd docs && npm run build` → SUCCESS; `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/` → empty (QC internals-leak gate). -- [ ] **Step 7: commit** (`docs(pygx): BNG lightweight tier across all surfaces`). - ---- - -## Self-Review - -**Spec coverage:** ✅ all 23 BNG functions (Tasks 1–8, enumerated against `registered_functions.txt` lines 111–133); ✅ pure-Python port of `BNG.scala`, no PyPI BNG lib (Tasks 1–7); ✅ WKB no-SRID for BNG (vs quadbin EWKB) — Tasks 3, 8, 9; ✅ resolution rules: Int ±1..±6 or resolutionMap string, reject metres-as-Int (Task 1); ✅ EPSG:27700 + cellarea km² (Tasks 2, 8, 13); ✅ exact cell-set parity (Task 9); ✅ mosaic#434 validated FIXED + locked in parity (Task 9); ✅ mosaic#423 validated FIXED + locked in parity (Tasks 6, 9); ✅ Serverless-safe udf/udtf-only (Task 8 guard); ✅ no new deps (shapely/numpy only); ✅ explode UDTFs + NotImplementedError Column contract (Task 8); ✅ grouped-agg pandas_udf returning STRUCT (Task 8); ✅ bench legs + `--grid-bng-only` (Tasks 11–12); ✅ all doc surfaces incl. performance.mdx + KEEP custom-grid heavy-only (Task 13); ✅ function-info/binding parity (Task 10); ✅ TDD ordering: Spark-free codec/cellmath FIRST (Tasks 1–7), registered-fn (Task 8), JAR-gated parity (Task 9). - -**Mosaic-bug findings (authored into the plan):** #434 — CONFIRMED FIXED in geobrix `BNG.scala` (100km id is 6 digits with trailing quadrant-digit 0 → `get_resolution` returns 1, not a quadrant; verified by hand-computing `encode(...res=1)` = `104090`). #423 — CONFIRMED FIXED (tessellate line 813 filters border chips to `getGeometryType == inGeomType`, dropping POINT/LINESTRING). No heavy change needed for either; both locked by Task 9 parity assertions and referenced in code/release-notes. - -**Placeholder scan:** the soft spots are intentional faithful-port markers, not TBDs: (a) `_bng.format` `coords` right-padding mirrors Scala `padTo(k,0)` (verify-in-step-4); (b) `polyfill` seed-coords + BFS and `tessellate` core/border split reference `BNG.scala` with the Task-9 parity test as the exact definition of done (same pattern as the quadbin polyfill / pyvx Sloan); (c) the `functions.py` pandas_udf/UDTF/agg bodies in Task 8 give one full example each + an explicit "similar wrappers for: ..." enumeration of every remaining name — all 23 registrations are listed verbatim in the `register` block, so nothing is left unnamed. - -**Name consistency:** `_bng` fn names (`get_resolution`, `point_to_cell_id`, `point_as_cell`/`east_north_as_bng`, `area`, `distance`, `euclidean_distance`, `cell_id_to_geometry`, `cell_aswkb`/`cell_aswkt`/`cell_centroid`, `k_ring`/`k_loop` + `_str` wrappers, `polyfill`/`polyfill_str`, `tessellate`/`tessellate_str`, `geometry_k_ring`/`geometry_k_loop` + `_str`, `cell_union`/`cell_intersection`, `is_valid`, `get_chips`), SQL names (`gbx_bng_*`, exactly the 23 from `registered_functions.txt`), schema (`BNG_CHIP_SCHEMA`), and the `register`/wrapper shapes are consistent across Tasks 1–13. diff --git a/docs/superpowers/plans/2026-06-14-pygx-custom-gridding.md b/docs/superpowers/plans/2026-06-14-pygx-custom-gridding.md deleted file mode 100644 index c13b7cf48..000000000 --- a/docs/superpowers/plans/2026-06-14-pygx-custom-gridding.md +++ /dev/null @@ -1,1065 +0,0 @@ -# pygx Phase 3 — Custom-Gridding Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Ship the 7 lightweight custom-gridding GridX functions (`gbx_custom_*`) in the existing pure-Python `databricks.labs.gbx.pygx` package at EXACT cell-ID / cell-set parity with the heavy Scala tier. Custom gridding is a faithful, bit-exact pure-Python port of `gridx/grid/CustomGridSystem.scala` + `GridConf.scala` + `custom/Custom_GridSpec.scala` (no PyPI library exists). On completion **GridX reaches full 1:1 light↔heavy parity** (quadbin + BNG + custom all in both tiers) and every doc surface flips the custom-grid family from heavyweight-only to both-tier. A one-line `import` swap (`pygx` vs `gridx.custom`) changes tiers — the SQL names are identical. - -**Architecture:** Pure-Python/PySpark, Serverless/Connect-safe (only `spark.udf.register` + Column exprs — never `_jvm`/`spark.conf.set`/`sparkContext`/`.rdd`). All custom-grid cell math is ported from `CustomGridSystem.scala`/`GridConf.scala` into `pygx/_custom.py`; geometry via shapely → **WKB, no SRID** (heavy uses `JTS.toWKB`, line 159, the 2D no-SRID variant — the grid's `srid` field is metadata only and is NEVER stamped into output geometry). The grid spec is a `STRUCT` built by `gbx_custom_grid` (a validating `@udf`) and consumed by the other six functions as a struct column. Mirrors the just-completed `pygx` quadbin (Phase 1) and BNG (Phase 2). - -**Tech Stack:** Python 3.12, `shapely` 2.x, `numpy`, PySpark `@udf`/`pandas_udf`. **No new dependencies** (`shapely`/`numpy` already in the `[light]` extra; custom gridding is integer/coordinate arithmetic plus shapely for the rectangle/point geometry and the `contains` test). No `@udtf`, no aggregators — the heavy custom family has none. - -**Spec:** `docs/superpowers/specs/2026-06-14-pygx-custom-gridding-light-tier-design.md` (APPROVED 2026-06-14, the four Resolved decisions are baked into the tasks below). **Branch:** `pygx-light`. Out of scope: quadbin (Phase 1, complete), BNG (Phase 2, complete), the `h3` GridX subpackage (native H3 covers hex), any heavy behavior change EXCEPT the approved `pointToCellID` Y-NaN typo fix (Resolved decision 3). - ---- - -## The 7 functions (parity targets, all `gbx_custom_*`) - -From `docs/tests-function-info/registered_functions.txt` (lines 144–150), 7 names: - -| # | Function (SQL) | Heavy signature | Return type | Light shape | Light source (port of `CustomGridSystem`) | -|---|---|---|---|---|---| -| 1 | `gbx_custom_grid` | `(xMin, xMax, yMin, yMax, cellSplits, rootCellSizeX, rootCellSizeY[, srid])` — 7 or 8 INT/LONG args | `STRUCT` (8 fields) | **validating `@udf`** (eager validation, returns `CUSTOM_GRID_SCHEMA`) | `Custom_Grid.eval` validation (`xMax>xMin`, `yMax>yMin`, `cell_splits>=2`, rootX/Y `>0`); no cell math | -| 2 | `gbx_custom_pointascell` | `(point BINARY\|STRING, grid STRUCT, resolution INT\|LONG)` | `BIGINT` | **`pandas_udf`** → `LongType` | `parse_geom`→coord→`point_to_cell_id` (bit-pack). NULL point/grid → NULL | -| 3 | `gbx_custom_cellaswkb` | `(cell BIGINT, grid STRUCT)` | `BINARY` (WKB polygon, no SRID) | **`pandas_udf`** → `BinaryType` | `cell_id_to_polygon`→shapely→`to_wkb()` (no SRID) | -| 4 | `gbx_custom_cellaswkt` | `(cell BIGINT, grid STRUCT)` | `STRING` (WKT polygon) | **`pandas_udf`** → `StringType` | same polygon → `.wkt` | -| 5 | `gbx_custom_centroid` | `(cell BIGINT, grid STRUCT)` | `BINARY` (WKB point, no SRID) | **`pandas_udf`** → `BinaryType` | `cell_id_to_center`→shapely Point→`to_wkb()` (no SRID) | -| 6 | `gbx_custom_polyfill` | `(geom BINARY\|STRING, grid STRUCT, resolution INT\|LONG)` | `ARRAY` | **plain `@udf`** → `ArrayType(LongType())` | bbox over-scan (`first..last+1`) + centroid-containment filter (exact port of `CustomGridSystem.polyfill`). NULL geom → NULL | -| 7 | `gbx_custom_kring` | `(cell BIGINT, grid STRUCT, k INT\|LONG)` | `ARRAY` | **plain `@udf`** → `ArrayType(LongType())` | decode cell→posX/posY→Chebyshev square clamped to `[0, totalCells]`→map back to cell IDs (exact port of `CustomGridSystem.kRing`) | - -Heavy reference: `src/main/scala/com/databricks/labs/gbx/gridx/custom/Custom_*.scala` (the 7 expression classes) + the canonical algorithm `gridx/grid/CustomGridSystem.scala` (≈340 lines) + `gridx/grid/GridConf.scala` + `gridx/custom/Custom_GridSpec.scala`. The per-function expressions just decode the inputs and call into `CustomGridSystem.*`. - -### Cell-ID encoding — must port BIT-EXACT (verbatim from `CustomGridSystem.scala` / `GridConf.scala`) - -``` -# GridConf (idBits = 56, resBits = 8): -subCellsCount = cell_splits * cell_splits -bitsPerResolution = ceil(log10(subCellsCount) / log10(2)) # GridConf.scala:25 -maxResolution = min(20, floor(56 / bitsPerResolution)) # GridConf.scala:28 -rootCellCountX = ceil((bound_x_max - bound_x_min) / root_cell_size_x) # ceil(span/size).toInt -rootCellCountY = ceil((bound_y_max - bound_y_min) / root_cell_size_y) - -# CustomGridSystem cell math: -totalCellsX(res) = rootCellCountX * pow(cell_splits, res).toLong # (Y analogous) -cellWidth(res) = root_cell_size_x / pow(cell_splits, res) # FLOAT division (height analogous) -getCellId(cellPos, res) = cellPos | (res.toLong << 56) # CustomGridSystem:310-315 -getCellPositionFromPositions(x,y,r) = posY * totalCellsX(r) + posX # row-major :317-321 -getCellResolution(cellId) = (cellId >> 56).toInt # :180-182 -getCellPosition(cellId) = cellId & 0x00ffffffffffffffL # :184-186 -getCellPositionX(idNum,r) = idNum % totalCellsX(r) # :188-190 -getCellPositionY(idNum,r) = floor(idNum / totalCellsX(r)).toLong # :192-194 - -# coordinate -> cell position (getCellPositionFromCoordinates, :268-272): -cellPosX = ((x - bound_x_min) / cellWidth(res)).toLong # Scala Double->Long = TRUNCATE toward zero -cellPosY = ((y - bound_y_min) / cellHeight(res)).toLong - -# cell -> polygon (cellIdToGeometry, :213-235): -x = cellX * cellWidth(res) + bound_x_min ; y = cellY * cellHeight(res) + bound_y_min -ring = [(x,y),(x+w,y),(x+w,y+h),(x,y+h),(x,y)] # closed 5-point ring - -# cell center (getCellCenterX/Y, :296-308): -centerX = cellPosX * cellWidth + cellWidth/2 + bound_x_min # (Y analogous) -``` - -`pointToCellID` (`:249-266`) enforces, in order: **no-NaN** (the buggy line — see Resolved decision 3), `resolution <= maxResolution`, `bound_x_min <= x < bound_x_max`, `bound_y_min <= y < bound_y_max` — raising `IllegalStateException` on violation. Light must match each as a `ValueError`. - -`polyfill` (`:145-178`): bbox `getCellPositionFromCoordinates(minX,minY)` / `(maxX,maxY)` → iterate `firstCellPosX to lastCellPosX + 1` × `firstCellPosY to lastCellPosY + 1` (the **`+1` over-scan is INTENTIONAL** — keep it; Scala `a to b` is INCLUSIVE so the Python range is `range(first, last + 2)`) → map each `(x,y)` to its cell-center coordinate (`getCellCenterX/Y`) → keep centers with `geometry.contains(point(cx,cy))` → `pointToCellID(cx, cy, res)`. - -`kRing` (`:38-60`): `res = getCellResolution(cellID)`; decode `posX`/`posY`; `fromX=max(posX-k,0)`, `toX=min(posX+k, totalCellsX(res))`, `fromY`/`toY` analogous; iterate `fromX to toX` × `fromY to toY` INCLUSIVE → `getCellPositionFromPositions` → `getCellId`. (Note the clamp uses `totalCellsX` itself, **not** `totalCellsX-1` — port that exactly even though it admits one out-of-range column; the parity test locks heavy's set.) - -### Resolved decisions baked into tasks (spec, 2026-06-14) - -1. **`gbx_custom_grid` = validating `@udf`** returning `CUSTOM_GRID_SCHEMA`, eager validation matching heavy `require(...)` (error-at-build-time parity). Task 6. -2. **Geom inputs accept `[E]WKB` + `[E]WKT` in BOTH tiers.** Light via `pygx/_geom.py` `parse_geom`. **Heavy already does** — `Custom_PointAsCell.decodeGeom` (`Custom_PointAsCell.scala:60-66`) is `case b: Array[Byte] => JTS.fromWKB(b); case UTF8String/String => JTS.fromWKT(...)`, and `fromWKB`/`fromWKT` auto-strip EWKB/EWKT SRID; `Custom_Polyfill` reuses `decodeGeom`. **No heavy geom-decoder extension is required.** Output geometry stays plain WKB no-SRID (`to_wkb()` / `JTS.toWKB`); `srid` is metadata only. Task 7 + Task 9. -3. **FIX the `pointToCellID` Y-NaN typo in BOTH tiers.** `CustomGridSystem.scala:250` is `require(!x.isNaN && !x.isNaN, ...)` — the second clause repeats `x`, so a NaN **Y** is unguarded. Heavy task fixes it to `!x.isNaN && !y.isNaN`; the light port guards both. Reference in the commit + beta release notes. The `polyfill` `+1` over-scan is intentional (NOT a fix). Task 8 (heavy) + Tasks 5/9 (light + parity lock-in). -4. **`maxResolution` computed identically** (`min(20, floor(56 / bitsPerResolution))`, cell_splits-dependent); light rejects `res > maxResolution`. Task 1. - -## Conventions (every task) -- Spark-free core tests (`_custom.py`) run on host: `.venv-pyrx/bin/python -m pytest -v`. Registered-fn + parity tests run in the `geobrix-dev` container: `bash scripts/commands/gbx-test-python.sh --path --log .log`. -- Commit (no push unless a task says so): `chmod -R u+rwX .git/objects`; subject ≤72 + WHY body; trailer exactly `Co-authored-by: Isaac`. -- Serverless guard: never add `_jvm`/`sparkContext`/`.rdd`/`spark.conf.set` to `pygx`. -- Impl rule (from Phases 1–2, apply identically): **scalar/bounded-output → `pandas_udf`** (pointascell + cellaswkb/cellaswkt/centroid); **variable-length array output → plain `@udf`** (polyfill/kring — OOM-safe row-by-row; a scalar `pandas_udf` would buffer a whole Arrow batch of arrays); **grid spec builder → validating `@udf` returning the STRUCT**. No `@udtf`, no grouped-agg. -- Int/Long tolerance: PySpark sends `Long` for integer literals; the grid-struct fields and `resolution`/`k` arrive as Python `int` in the UDF — coerce defensively (mirror `Custom_GridSpec.asInt`/`asLong`). -- The pygx test package lives in `test/pygx/` (not named after a PyPI lib), so the test-package-shadows-installed-lib gotcha does not apply. - -## File Structure -| File | Responsibility | New? | -|---|---|---| -| `python/geobrix/src/databricks/labs/gbx/pygx/_custom.py` | pure-Python port of `CustomGridSystem`/`GridConf`: a `CustomGridConf` dataclass (8 fields + derived `bits_per_resolution`, `max_resolution`, `root_cell_count_x/y`, `id_bits=56`), `conf_from_row`, and grid-math fns `point_to_cell_id`, `cell_id_to_polygon`, `cell_id_to_centroid`, `polyfill`, `k_ring`, plus bit-pack/unpack helpers (`get_cell_id`, `get_cell_resolution`, `get_cell_position`, `get_cell_position_x/y`, `total_cells_x/y`, `cell_width/height`, `get_cell_center_x/y`, `get_cell_position_from_coordinates`, `get_cell_position_from_positions`). Spark-free; shapely for geometry | new | -| `pygx/_serde.py` | **add** `CUSTOM_GRID_SCHEMA` (8-field STRUCT matching `Custom_GridSpec.gridStructType` field names/types: `bound_x_min/x_max/y_min/y_max` LONG, `cell_splits`/`root_cell_size_x`/`root_cell_size_y`/`srid` INT) | extend | -| `pygx/_env.py` | **add** `assert_custom_available()` (shapely only) | extend | -| `pygx/functions.py` | **add** the 7 `gbx_custom_*` UDFs + Column wrappers + extend `register(spark)` to install all 7 | extend | -| `python/geobrix/test/pygx/test_custom_core.py` | Spark-free `_custom` unit tests (bit-pack/unpack round-trips, maxResolution/rootCellCount formulas, point_to_cell_id, cell→geometry/centroid, polyfill over-scan, k_ring clamp, validation raises incl. Y-NaN) | new | -| `python/geobrix/test/pygx/test_custom_functions.py` | registered-fn tests via the spark fixture (grid struct shape, NULL propagation, round-trip) | new | -| `python/geobrix/test/pygx/test_parity_custom.py` | JAR-gated cross-tier EXACT parity (cells + WKB geom within 1e-6 + all-4-encodings geom-input + Y-NaN lock-in) | new | - ---- - -## Task 1: `_custom.py` — `CustomGridConf` (GridConf port) + bit-pack/unpack + maxResolution - -**Files:** create `pygx/_custom.py`, `test/pygx/test_custom_core.py`. - -Port `GridConf` (the derived quantities) and the cell-ID codec from `CustomGridSystem`. The dataclass holds the 8 fields; computed properties `bits_per_resolution`, `max_resolution`, `root_cell_count_x/y` mirror `GridConf.scala` EXACTLY. The codec functions are pure integer/float math. - -- [ ] **Step 1: failing test** `test/pygx/test_custom_core.py` -```python -import math - -import pytest - -shapely = pytest.importorskip("shapely") # _custom imports shapely at module load -from databricks.labs.gbx.pygx import _custom - - -def _conf(splits=2, rootx=1000, rooty=1000, srid=-1): - # A 0..1,000,000 grid (mirrors the doc SQL example grid). - return _custom.CustomGridConf( - bound_x_min=0, bound_x_max=1_000_000, bound_y_min=0, bound_y_max=1_000_000, - cell_splits=splits, root_cell_size_x=rootx, root_cell_size_y=rooty, srid=srid, - ) - - -def test_gridconf_derived_quantities_match_scala(): - c = _conf(splits=2, rootx=1000, rooty=1000) - # subCellsCount = 4 -> ceil(log10(4)/log10(2)) = ceil(2.0) = 2 - assert c.bits_per_resolution == 2 - # min(20, floor(56/2)) = 20 (the 20 cap binds here) - assert c.max_resolution == 20 - # ceil(1_000_000 / 1000) = 1000 - assert c.root_cell_count_x == 1000 - assert c.root_cell_count_y == 1000 - - -def test_max_resolution_is_cell_splits_dependent(): - # cell_splits=4 -> subCells=16 -> ceil(log10(16)/log10(2)) = ceil(4.0) = 4 - # -> min(20, floor(56/4)) = 14 - c = _conf(splits=4) - assert c.bits_per_resolution == 4 - assert c.max_resolution == 14 - # cell_splits=8 -> subCells=64 -> bitsPerRes = ceil(log10(64)/log10(2)) = 6 - # -> min(20, floor(56/6)=9) = 9 - c8 = _conf(splits=8) - assert c8.bits_per_resolution == 6 - assert c8.max_resolution == 9 - - -def test_cell_id_pack_unpack_roundtrip(): - c = _conf() - for res in (0, 1, 5, 10): - for (px, py) in [(0, 0), (3, 7), (123, 456)]: - pos = _custom.get_cell_position_from_positions(c, px, py, res) - cid = _custom.get_cell_id(pos, res) - assert _custom.get_cell_resolution(cid) == res - decoded = _custom.get_cell_position(cid) - assert _custom.get_cell_position_x(c, decoded, res) == px - assert _custom.get_cell_position_y(c, decoded, res) == py - - -def test_total_cells_and_cell_width(): - c = _conf(splits=2, rootx=1000) - assert _custom.total_cells_x(c, 0) == 1000 # rootCellCountX * 2^0 - assert _custom.total_cells_x(c, 1) == 2000 # * 2^1 - assert _custom.cell_width(c, 0) == 1000.0 - assert _custom.cell_width(c, 1) == 500.0 # 1000 / 2^1 (FLOAT division) - - -def test_conf_from_row_int_long_tolerant(): - # Simulate the struct arriving as a dict (PySpark Row.asDict) with Long bounds. - row = { - "bound_x_min": 0, "bound_x_max": 1_000_000, - "bound_y_min": 0, "bound_y_max": 1_000_000, - "cell_splits": 2, "root_cell_size_x": 1000, "root_cell_size_y": 1000, - "srid": 27700, - } - c = _custom.conf_from_row(row) - assert c.srid == 27700 and c.cell_splits == 2 and c.bound_x_max == 1_000_000 -``` - -- [ ] **Step 2: run → FAIL** (`.venv-pyrx/bin/python -m pytest python/geobrix/test/pygx/test_custom_core.py -v` — no `_custom`). - -- [ ] **Step 3: implement** `pygx/_custom.py`. Module docstring must state it is a pure-Python port of `gridx/grid/CustomGridSystem.scala` + `GridConf.scala`, that WKB has no SRID, and reference Resolved decision 3 (Y-NaN fix) on `point_to_cell_id`. -```python -"""Pure-Python custom-grid core for the pygx light tier. - -A faithful, BIT-EXACT port of the heavy -``com.databricks.labs.gbx.gridx.grid.CustomGridSystem`` + ``GridConf`` Scala -objects (gridx/grid/CustomGridSystem.scala, GridConf.scala). No PyPI library -exists; this module reproduces the cell-ID bit-packing, coordinate<->cell -mapping, polyfill (centroid-containment), and k-ring (Chebyshev clamp) EXACTLY -so light and heavy share bit-identical cell ids and cell sets. - -A custom grid is a user-defined regular rectangular grid: extent, root cell -size, and a recursive ``cell_splits`` factor (each resolution level subdivides -into ``cell_splits x cell_splits`` sub-cells). Cell ids are BIGINT (the top 8 -bits hold the resolution, the low 56 hold the row-major cell position). - -Geometry is emitted as plain WKB (NO SRID) / WKT, matching heavy ``JTS.toWKB`` -(line 159, the 2D no-SRID variant). The grid ``srid`` is metadata only and is -NOT stamped into output geometry. - -Resolved decision 3 (spec 2026-06-14): heavy ``pointToCellID`` had a -``require(!x.isNaN && !x.isNaN, ...)`` typo that left a NaN Y unguarded; -``point_to_cell_id`` here (and the heavy fix) guards BOTH x and y. -""" - -import math -from dataclasses import dataclass -from typing import Any, List - -ID_BITS = 56 # GridConf.idBits — low 56 bits hold the cell position -RES_BITS = 8 # GridConf.resBits — top 8 bits hold the resolution -_POSITION_MASK = 0x00FFFFFFFFFFFFFF - - -def _as_int(v: Any) -> int: - if isinstance(v, bool): # bool is an int subclass; reject explicitly - raise ValueError(f"gbx_custom: expected INT/LONG, got bool {v!r}") - if isinstance(v, int): - return v - if isinstance(v, float) and v.is_integer(): - return int(v) - raise ValueError(f"gbx_custom: expected INT/LONG, got {v!r}") - - -@dataclass(frozen=True) -class CustomGridConf: - bound_x_min: int - bound_x_max: int - bound_y_min: int - bound_y_max: int - cell_splits: int - root_cell_size_x: int - root_cell_size_y: int - srid: int = -1 # -1 == no CRS - - @property - def sub_cells_count(self) -> int: - return self.cell_splits * self.cell_splits - - @property - def bits_per_resolution(self) -> int: - # GridConf.scala:25 — ceil(log10(subCellsCount) / log10(2)) - return math.ceil(math.log10(self.sub_cells_count) / math.log10(2)) - - @property - def max_resolution(self) -> int: - # GridConf.scala:28 — min(20, floor(56 / bitsPerResolution)) - return min(20, math.floor(ID_BITS / self.bits_per_resolution)) - - @property - def root_cell_count_x(self) -> int: - span = self.bound_x_max - self.bound_x_min - return math.ceil(span / self.root_cell_size_x) - - @property - def root_cell_count_y(self) -> int: - span = self.bound_y_max - self.bound_y_min - return math.ceil(span / self.root_cell_size_y) - - -def conf_from_row(row: Any) -> CustomGridConf: - """Reconstruct a CustomGridConf from a grid-spec struct (Row/dict). - - Mirrors Custom_GridSpec.systemFromRow; Int/Long tolerant (PySpark sends Long - for INT literals). - """ - if row is None: - raise ValueError("gbx_custom: grid spec must not be null") - g = row.asDict() if hasattr(row, "asDict") else dict(row) - return CustomGridConf( - bound_x_min=_as_int(g["bound_x_min"]), - bound_x_max=_as_int(g["bound_x_max"]), - bound_y_min=_as_int(g["bound_y_min"]), - bound_y_max=_as_int(g["bound_y_max"]), - cell_splits=_as_int(g["cell_splits"]), - root_cell_size_x=_as_int(g["root_cell_size_x"]), - root_cell_size_y=_as_int(g["root_cell_size_y"]), - srid=_as_int(g["srid"]), - ) - - -# --- cell-ID codec + grid math (CustomGridSystem) ----------------------------- - -def total_cells_x(conf: CustomGridConf, resolution: int) -> int: - return conf.root_cell_count_x * int(math.pow(conf.cell_splits, resolution)) - - -def total_cells_y(conf: CustomGridConf, resolution: int) -> int: - return conf.root_cell_count_y * int(math.pow(conf.cell_splits, resolution)) - - -def cell_width(conf: CustomGridConf, resolution: int) -> float: - return conf.root_cell_size_x / math.pow(conf.cell_splits, resolution) - - -def cell_height(conf: CustomGridConf, resolution: int) -> float: - return conf.root_cell_size_y / math.pow(conf.cell_splits, resolution) - - -def get_cell_id(cell_position: int, resolution: int) -> int: - return cell_position | (resolution << ID_BITS) - - -def get_cell_resolution(cell_id: int) -> int: - return cell_id >> ID_BITS - - -def get_cell_position(cell_id: int) -> int: - return cell_id & _POSITION_MASK - - -def get_cell_position_x(conf: CustomGridConf, id_number: int, resolution: int) -> int: - return id_number % total_cells_x(conf, resolution) - - -def get_cell_position_y(conf: CustomGridConf, id_number: int, resolution: int) -> int: - return int(math.floor(id_number / total_cells_x(conf, resolution))) - - -def get_cell_position_from_positions(conf, cell_pos_x: int, cell_pos_y: int, resolution: int) -> int: - return cell_pos_y * total_cells_x(conf, resolution) + cell_pos_x - - -def _trunc_long(v: float) -> int: - # Scala Double->Long truncates toward zero (NOT math.floor). - return int(v) - - -def get_cell_position_from_coordinates(conf, x: float, y: float, resolution: int): - cell_pos_x = _trunc_long((x - conf.bound_x_min) / cell_width(conf, resolution)) - cell_pos_y = _trunc_long((y - conf.bound_y_min) / cell_height(conf, resolution)) - return cell_pos_x, cell_pos_y, get_cell_position_from_positions(conf, cell_pos_x, cell_pos_y, resolution) - - -def get_cell_center_x(conf, cell_position_x: int, resolution: int) -> float: - w = cell_width(conf, resolution) - return cell_position_x * w + (w / 2) + conf.bound_x_min - - -def get_cell_center_y(conf, cell_position_y: int, resolution: int) -> float: - h = cell_height(conf, resolution) - return cell_position_y * h + (h / 2) + conf.bound_y_min -``` - -- [ ] **Step 4: run → PASS** (core formula tests). - -- [ ] **Step 5: commit** (`feat(pygx): custom-grid GridConf port + cell-ID bit-pack/unpack`). - ---- - -## Task 2: `_custom.py` — `point_to_cell_id` (+ the Y-NaN guard, Resolved decision 3) - -**Files:** modify `pygx/_custom.py`, `test/pygx/test_custom_core.py`. - -Port `pointToCellID` (`CustomGridSystem.scala:249-266`) with all four `require` guards, in order. Fix the heavy Y-NaN typo: guard BOTH x and y (Resolved decision 3). - -- [ ] **Step 1: failing tests** (append to `test_custom_core.py`) -```python -def test_point_to_cell_id_known_fixture(): - c = _conf(splits=2, rootx=1000, rooty=1000) - # res 0: 1000m root cells. Point (530000, 180000) -> posX=530, posY=180. - cid = _custom.point_to_cell_id(c, 530000.0, 180000.0, 0) - assert _custom.get_cell_resolution(cid) == 0 - pos = _custom.get_cell_position(cid) - assert _custom.get_cell_position_x(c, pos, 0) == 530 - assert _custom.get_cell_position_y(c, pos, 0) == 180 - - -def test_point_to_cell_id_rejects_nan_x_and_y(): - c = _conf() - with pytest.raises(ValueError): - _custom.point_to_cell_id(c, float("nan"), 180000.0, 0) - # Resolved decision 3: a NaN Y must ALSO raise (heavy typo left Y unguarded). - with pytest.raises(ValueError): - _custom.point_to_cell_id(c, 530000.0, float("nan"), 0) - - -def test_point_to_cell_id_rejects_out_of_bounds_and_over_max_res(): - c = _conf(splits=2) # max_resolution == 20 - with pytest.raises(ValueError): - _custom.point_to_cell_id(c, -1.0, 180000.0, 0) # x < bound_x_min - with pytest.raises(ValueError): - _custom.point_to_cell_id(c, 1_000_000.0, 180000.0, 0) # x == bound_x_max (exclusive) - with pytest.raises(ValueError): - _custom.point_to_cell_id(c, 530000.0, 180000.0, 21) # res > max_resolution -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** (append to `_custom.py`). -```python -def point_to_cell_id(conf: CustomGridConf, x: float, y: float, resolution: int) -> int: - """Cell ID containing (x, y) at `resolution` (CustomGridSystem.pointToCellID). - - Resolved decision 3: guard BOTH x and y for NaN (the heavy Scala had a - ``!x.isNaN && !x.isNaN`` typo that left Y unguarded; fixed in both tiers). - """ - if math.isnan(x) or math.isnan(y): - raise ValueError("gbx_custom: NaN coordinates are not supported.") - if resolution > conf.max_resolution: - raise ValueError( - f"gbx_custom: resolution ({resolution}) exceeds maximum " - f"resolution of {conf.max_resolution}." - ) - if not (conf.bound_x_min <= x < conf.bound_x_max): - raise ValueError( - f"gbx_custom: X coordinate ({x}) out of bounds " - f"{conf.bound_x_min}-{conf.bound_x_max}" - ) - if not (conf.bound_y_min <= y < conf.bound_y_max): - raise ValueError( - f"gbx_custom: Y coordinate ({y}) out of bounds " - f"{conf.bound_y_min}-{conf.bound_y_max}" - ) - _, _, cell_pos = get_cell_position_from_coordinates(conf, x, y, resolution) - return get_cell_id(cell_pos, resolution) -``` - -- [ ] **Step 4: run → PASS.** - -- [ ] **Step 5: commit** (`feat(pygx): custom-grid point_to_cell_id (guards both x and y NaN)`). - ---- - -## Task 3: `_custom.py` cell→geometry — `cell_id_to_polygon`, `cell_id_to_centroid`, aswkb/aswkt/centroid (no SRID) - -**Files:** modify `pygx/_custom.py`, `test/pygx/test_custom_core.py`. - -Heavy emits plain WKB (no SRID) via `JTS.toWKB` and plain WKT via `JTS.toWKT`. The cell polygon is the closed ring `(x,y),(x+w,y),(x+w,y+h),(x,y+h),(x,y)` from `cellIdToGeometry` (`:213-235`); the centroid is the polygon centroid (`cellIdToCenter`, `:332-338`). Use shapely `box(x, y, x+w, y+h)`; `to_wkb()` defaults to no SRID — do NOT `set_srid`. - -- [ ] **Step 1: failing tests** (append) -```python -from shapely import from_wkb, from_wkt, get_srid # noqa: E402 - - -def test_cell_aswkb_is_polygon_no_srid(): - c = _conf(splits=2, rootx=1000, rooty=1000) - cid = _custom.point_to_cell_id(c, 530000.0, 180000.0, 0) # res 0 -> 1000m cell - g = from_wkb(_custom.cell_aswkb(c, cid)) - assert g.geom_type == "Polygon" - assert get_srid(g) == 0 # custom WKB carries NO SRID - assert g.bounds == (530000.0, 180000.0, 531000.0, 181000.0) - - -def test_cell_aswkt_is_polygon_text(): - c = _conf() - cid = _custom.point_to_cell_id(c, 530000.0, 180000.0, 0) - g = from_wkt(_custom.cell_aswkt(c, cid)) - assert g.geom_type == "Polygon" - - -def test_cell_centroid_is_point_no_srid(): - c = _conf() - cid = _custom.point_to_cell_id(c, 530000.0, 180000.0, 0) - g = from_wkb(_custom.cell_centroid(c, cid)) - assert g.geom_type == "Point" and get_srid(g) == 0 - assert (g.x, g.y) == (530500.0, 180500.0) # cell center -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** (append; import `box`, `to_wkb`, `Point` from shapely at top of file). -```python -from shapely import to_wkb as _to_wkb # at top of file -from shapely.geometry import Point as _Point -from shapely.geometry import box as _box - - -def cell_id_to_polygon(conf: CustomGridConf, cell_id: int): - """Closed custom-grid cell polygon (shapely), NO SRID (cellIdToGeometry).""" - resolution = get_cell_resolution(cell_id) - cell_number = get_cell_position(cell_id) - cell_x = get_cell_position_x(conf, cell_number, resolution) - cell_y = get_cell_position_y(conf, cell_number, resolution) - w = cell_width(conf, resolution) - h = cell_height(conf, resolution) - x = cell_x * w + conf.bound_x_min - y = cell_y * h + conf.bound_y_min - return _box(x, y, x + w, y + h) - - -def cell_id_to_centroid(conf: CustomGridConf, cell_id: int): - return cell_id_to_polygon(conf, cell_id).centroid - - -def cell_aswkb(conf: CustomGridConf, cell_id: int) -> bytes: - return _to_wkb(cell_id_to_polygon(conf, cell_id)) # include_srid defaults False - - -def cell_aswkt(conf: CustomGridConf, cell_id: int) -> str: - return cell_id_to_polygon(conf, cell_id).wkt - - -def cell_centroid(conf: CustomGridConf, cell_id: int) -> bytes: - return _to_wkb(cell_id_to_centroid(conf, cell_id)) -``` - -- [ ] **Step 4: run → PASS.** - -- [ ] **Step 5: commit** (`feat(pygx): custom-grid cell geometry — aswkb/aswkt/centroid (no SRID)`). - ---- - -## Task 4: `_custom.py` — `k_ring` (Chebyshev square, clamp to `totalCells`) - -**Files:** modify `pygx/_custom.py`, `test/pygx/test_custom_core.py`. - -Port `CustomGridSystem.kRing` (`:38-60`) EXACTLY: decode `res`/`posX`/`posY`; `fromX=max(posX-k,0)`, `toX=min(posX+k, totalCellsX(res))` (clamp uses `totalCellsX` itself, not `-1`); iterate `fromX..toX` × `fromY..toY` INCLUSIVE (Scala `a to b`); each `(x,y)` → `get_cell_position_from_positions` → `get_cell_id`. - -- [ ] **Step 1: failing tests** (append) -```python -def test_kring_interior_k1_is_nine_cells(): - c = _conf(splits=2, rootx=1000, rooty=1000) - cid = _custom.point_to_cell_id(c, 530000.0, 180000.0, 0) # interior cell - ring = _custom.k_ring(c, cid, 1) - assert cid in ring - assert len(set(ring)) == 9 # 3x3 block, far from edges - - -def test_kring_k0_is_self_only(): - c = _conf() - cid = _custom.point_to_cell_id(c, 530000.0, 180000.0, 0) - assert _custom.k_ring(c, cid, 0) == [cid] - - -def test_kring_clamps_at_origin(): - c = _conf(splits=2, rootx=1000, rooty=1000) - cid = _custom.point_to_cell_id(c, 100.0, 100.0, 0) # posX=posY=0 (origin cell) - ring = _custom.k_ring(c, cid, 1) - # fromX/fromY clamp to 0; the block does not extend to negative positions. - assert cid in ring - assert all(_custom.get_cell_position(rc) >= 0 for rc in ring) -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** (append). -```python -def k_ring(conf: CustomGridConf, cell_id: int, k: int) -> List[int]: - """Chebyshev (square) k-ring of cell IDs around cell_id (CustomGridSystem.kRing). - - Includes the center cell. Clamped to [0, totalCells] per the heavy port (the - upper clamp uses totalCellsX/Y itself, INCLUSIVE iteration — ported verbatim). - """ - if k < 0: - raise ValueError("gbx_custom: k must be at least 0") - res = get_cell_resolution(cell_id) - cell_position = get_cell_position(cell_id) - pos_x = get_cell_position_x(conf, cell_position, res) - pos_y = get_cell_position_y(conf, cell_position, res) - from_x = max(pos_x - k, 0) - to_x = min(pos_x + k, total_cells_x(conf, res)) - from_y = max(pos_y - k, 0) - to_y = min(pos_y + k, total_cells_y(conf, res)) - out = [] - for x in range(from_x, to_x + 1): # Scala `a to b` is INCLUSIVE - for y in range(from_y, to_y + 1): - pos = get_cell_position_from_positions(conf, x, y, res) - out.append(get_cell_id(pos, res)) - return out -``` - -- [ ] **Step 4: run → PASS.** - -- [ ] **Step 5: commit** (`feat(pygx): custom-grid k_ring (Chebyshev square, clamp to totalCells)`). - ---- - -## Task 5: `_custom.py` — `polyfill` (bbox `+1` over-scan + centroid containment) - -**Files:** modify `pygx/_custom.py`, `test/pygx/test_custom_core.py`. - -Port `CustomGridSystem.polyfill` (`:145-178`) EXACTLY, including the **intentional `+1` over-scan** (Resolved decision 3 confirms it is NOT a bug): bbox `getCellPositionFromCoordinates(minX,minY)`/`(maxX,maxY)` → iterate `firstCellPosX to lastCellPosX + 1` × `firstCellPosY to lastCellPosY + 1` (INCLUSIVE Scala → Python `range(first, last + 2)`) → map each `(x,y)` to its cell-center coordinate → keep centers `contains`-ed by the geometry → `point_to_cell_id(cx, cy, res)`. Empty/None geom → `[]`. - -- [ ] **Step 1: failing tests** (append) -```python -from shapely import to_wkb as _towkb # noqa: E402 -from shapely.geometry import box as _box2 # noqa: E402 - - -def test_polyfill_small_box_res0(): - c = _conf(splits=2, rootx=1000, rooty=1000) # 1000m root cells - # A 3000m x 3000m box aligned to the grid -> 9 cell centers fall inside. - geom = _box2(530000.0, 180000.0, 533000.0, 183000.0) - cells = _custom.polyfill(c, geom, 0) - assert len(cells) == 9 - assert all(_custom.get_cell_resolution(cid) == 0 for cid in cells) - - -def test_polyfill_empty_geom_is_empty(): - c = _conf() - assert _custom.polyfill(c, None, 0) == [] - from shapely.geometry import Polygon - assert _custom.polyfill(c, Polygon(), 0) == [] - - -def test_polyfill_centroid_containment_only(): - c = _conf(splits=2, rootx=1000, rooty=1000) - # A box smaller than one cell, off-center, contains NO cell center -> empty. - geom = _box2(530100.0, 180100.0, 530400.0, 180400.0) - assert _custom.polyfill(c, geom, 0) == [] -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** (append). -```python -def polyfill(conf: CustomGridConf, geometry, resolution: int) -> List[int]: - """Cell IDs whose CENTER is contained by the geometry (CustomGridSystem.polyfill). - - Mirrors heavy EXACTLY incl. the intentional ``first..last + 1`` bbox over-scan - (Resolved decision 3: over-scan is by design, not a bug — the centroid filter - discards the extra cells). - """ - if geometry is None or geometry.is_empty: - return [] - min_x, min_y, max_x, max_y = geometry.bounds - first_x, first_y, _ = get_cell_position_from_coordinates(conf, min_x, min_y, resolution) - last_x, last_y, _ = get_cell_position_from_coordinates(conf, max_x, max_y, resolution) - out = [] - # `first to last + 1` INCLUSIVE -> Python range(first, (last + 1) + 1) - for x in range(first_x, last_x + 2): - for y in range(first_y, last_y + 2): - cx = get_cell_center_x(conf, x, resolution) - cy = get_cell_center_y(conf, y, resolution) - if geometry.contains(_Point(cx, cy)): - out.append(point_to_cell_id(conf, cx, cy, resolution)) - return out -``` - -- [ ] **Step 4: run → PASS.** - -- [ ] **Step 5: commit** (`feat(pygx): custom-grid polyfill (centroid containment + intentional +1 over-scan)`). - ---- - -## Task 6: `_serde.py` `CUSTOM_GRID_SCHEMA` + the validating `gbx_custom_grid` `@udf` - -**Files:** modify `pygx/_serde.py`, `pygx/_env.py`, `pygx/functions.py`, `test/pygx/test_custom_functions.py` (create — the grid-builder test only). - -Add `CUSTOM_GRID_SCHEMA` to `_serde.py` matching `Custom_GridSpec.gridStructType` field names/types EXACTLY (`bound_x_min/x_max/y_min/y_max` LONG, `cell_splits`/`root_cell_size_x`/`root_cell_size_y`/`srid` INT). Add `assert_custom_available()` to `_env.py` (shapely only). In `functions.py` add `gbx_custom_grid` as a validating `@udf` (Resolved decision 1) returning the schema, eager-validating like `Custom_Grid.eval` (`xMax>xMin`, `yMax>yMin`, `cell_splits>=2`, rootX/Y `>0`), 7-arg form defaults `srid=-1`. - -- [ ] **Step 1: failing test** `test/pygx/test_custom_functions.py` -```python -import pytest - -shapely = pytest.importorskip("shapely") -from pyspark.sql.utils import PythonException # noqa: E402 - -from databricks.labs.gbx.pygx import functions as gx # noqa: E402 - - -def test_custom_grid_struct_shape(spark): - gx.register(spark) - row = spark.sql( - "SELECT gbx_custom_grid(0, 1000000, 0, 1000000, 2, 1000, 1000, 27700) AS g" - ).collect()[0] - g = row["g"].asDict() - assert g["bound_x_min"] == 0 and g["bound_x_max"] == 1000000 - assert g["cell_splits"] == 2 and g["root_cell_size_x"] == 1000 - assert g["srid"] == 27700 - - -def test_custom_grid_7arg_defaults_srid_minus1(spark): - gx.register(spark) - row = spark.sql( - "SELECT gbx_custom_grid(0, 1000000, 0, 1000000, 2, 1000, 1000) AS g" - ).collect()[0] - assert row["g"].asDict()["srid"] == -1 - - -def test_custom_grid_validation_raises(spark): - gx.register(spark) - with pytest.raises(Exception): # PythonException wrapping ValueError - spark.sql( - "SELECT gbx_custom_grid(1000000, 0, 0, 1000000, 2, 1000, 1000)" - ).collect() # xMax <= xMin - with pytest.raises(Exception): - spark.sql( - "SELECT gbx_custom_grid(0, 1000000, 0, 1000000, 1, 1000, 1000)" - ).collect() # cell_splits < 2 -``` - -- [ ] **Step 2: run → FAIL** (`bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pygx/test_custom_functions.py --log custom-fns.log` — no `gbx_custom_grid` registered). - -- [ ] **Step 3: implement.** - - `_serde.py` add: -```python -from pyspark.sql.types import IntegerType # extend imports - -# Grid-spec struct produced by gbx_custom_grid, consumed by all gbx_custom_* ops. -# Field names/types match heavy Custom_GridSpec.gridStructType exactly so the SAME -# struct flows into both light and heavy consumers. srid == -1 means no CRS. -CUSTOM_GRID_SCHEMA = StructType( - [ - StructField("bound_x_min", LongType(), False), - StructField("bound_x_max", LongType(), False), - StructField("bound_y_min", LongType(), False), - StructField("bound_y_max", LongType(), False), - StructField("cell_splits", IntegerType(), False), - StructField("root_cell_size_x", IntegerType(), False), - StructField("root_cell_size_y", IntegerType(), False), - StructField("srid", IntegerType(), False), - ] -) -``` - - `_env.py` add: -```python -def assert_custom_available() -> None: - """Raise a clear ImportError if shapely (the only pygx custom dep) is missing. - - Custom gridding is a pure-Python port of CustomGridSystem.scala; it needs only - shapely (geometry + WKB/WKT I/O), no quadbin/BNG PyPI library. - """ - try: - import shapely # noqa: F401 - except Exception: # noqa: BLE001 - raise ImportError( - "pygx custom gridding requires the [light] extra (shapely). " - "Install with: pip install 'geobrix[light]'" - ) -``` - - `functions.py` add (imports: `from . import _custom`; `from ._serde import CUSTOM_GRID_SCHEMA`; `from pyspark.sql.functions import udf`): -```python -# --- custom-grid spec builder -> validating @udf returning CUSTOM_GRID_SCHEMA --- -# Resolved decision 1 (spec 2026-06-14): eager validation matches heavy -# Custom_Grid.eval's require(...) (error-at-build-time parity). The 7-arg form -# defaults srid to -1 (no CRS), as in Custom_Grid.builder. -@udf(returnType=CUSTOM_GRID_SCHEMA) -def _custom_grid_udf(x_min, x_max, y_min, y_max, splits, root_x, root_y, srid): - x_min, x_max = int(x_min), int(x_max) - y_min, y_max = int(y_min), int(y_max) - splits, root_x, root_y = int(splits), int(root_x), int(root_y) - srid = -1 if srid is None else int(srid) - if not x_max > x_min: - raise ValueError( - f"gbx_custom_grid: bound_x_max ({x_max}) must be greater than " - f"bound_x_min ({x_min})" - ) - if not y_max > y_min: - raise ValueError( - f"gbx_custom_grid: bound_y_max ({y_max}) must be greater than " - f"bound_y_min ({y_min})" - ) - if splits < 2: - raise ValueError(f"gbx_custom_grid: cell_splits must be >= 2; got {splits}") - if root_x <= 0: - raise ValueError(f"gbx_custom_grid: root_cell_size_x must be > 0; got {root_x}") - if root_y <= 0: - raise ValueError(f"gbx_custom_grid: root_cell_size_y must be > 0; got {root_y}") - return (x_min, x_max, y_min, y_max, splits, root_x, root_y, srid) -``` - Register both arities under one SQL name. Spark UDFs are fixed-arity, so register a single 8-arg UDF and let SQL pass an explicit srid OR register a separate 7-arg lambda; the simplest parity path mirrors `Custom_Grid.builder` by registering one name whose Python function takes a defaulted 8th arg. PySpark `spark.udf.register` of a `@udf` cannot default-fill a missing SQL arg, so register a tiny dispatcher: -```python - # register BOTH the 7- and 8-arg call shapes under gbx_custom_grid (Spark UDFs - # are fixed-arity; a 7-arg call must default srid=-1 like Custom_Grid.builder). - spark.udf.register("gbx_custom_grid", _custom_grid_udf) # 8-arg - # 7-arg form: a thin wrapper UDF that injects srid=-1. - spark.udf.register( - "gbx_custom_grid", _custom_grid_udf - ) # see note: handle 7-arg in the SQL example / Column wrapper -``` - Note for the implementer: confirm in Step 4 whether `spark.udf.register("gbx_custom_grid", _custom_grid_udf)` accepts a 7-arg SQL call. PySpark `register` of a fixed-8-arg UDF will REJECT a 7-arg call (`wrong number of arguments`). To match `Custom_Grid.builder`'s 7-or-8 contract, give `_custom_grid_udf` a Python default (`srid=-1`) AND register via the Python-function path so the wrapper accepts both; if PySpark still requires fixed arity, register a second SQL name-free dispatcher is NOT allowed (one canonical SQL name). The robust solution: make `_custom_grid_udf` a plain Python function decorated `@udf` whose signature is `(x_min, x_max, y_min, y_max, splits, root_x, root_y, srid=-1)` — PySpark binds by position and a 7-arg SQL call uses the default. Verify; if PySpark rejects, fall back to requiring 8 args in SQL and document the 7-arg form is light-tier-via-`custom_grid()` Column wrapper only (the wrapper supplies `-1`). The Column wrapper (below) ALWAYS supplies all 8. - -- [ ] **Step 4: run → PASS** (grid-builder tests; resolve the 7-vs-8-arg arity per the note — the test asserts BOTH arities work via SQL). - -- [ ] **Step 5: commit** (`feat(pygx): custom-grid CUSTOM_GRID_SCHEMA + validating gbx_custom_grid @udf`). - ---- - -## Task 7: `functions.py` — register the 6 consuming functions + all 7 Column wrappers - -**Files:** modify `pygx/functions.py`, append to `test/pygx/test_custom_functions.py`. - -Add the six consuming UDFs (pointascell/cellaswkb/cellaswkt/centroid → `pandas_udf`; polyfill/kring → plain `@udf`), extend `register(spark)` to install all 7 `gbx_custom_*`, and add the 7 `custom_*` Column wrappers (mirror heavy `gridx.custom.functions` + the quadbin/BNG wrapper style via `f.call_function`). The consuming UDFs receive the grid spec as a struct column → arrives as a `Row`/dict → `_custom.conf_from_row`. Geom inputs use `parse_geom` (Resolved decision 2, all 4 encodings). - -- [ ] **Step 1: failing tests** (append to `test_custom_functions.py`) -```python -from shapely import from_wkb, get_srid, to_wkb # noqa: E402 -from shapely.geometry import box # noqa: E402 - -_GRID = "gbx_custom_grid(0, 1000000, 0, 1000000, 2, 1000, 1000, 27700)" - - -def test_pointascell_wkt_and_wkb(spark): - gx.register(spark) - # WKT input (Resolved decision 2: all 4 encodings accepted). - r1 = spark.sql( - f"SELECT gbx_custom_pointascell('POINT(530000 180000)', {_GRID}, 0) AS c" - ).collect()[0] - assert r1["c"] is not None - # WKB input must give the SAME cell id. - df = spark.createDataFrame( - [(bytearray(to_wkb(box(530000, 180000, 530001, 180001).centroid)),)], "g binary" - ) - df.createOrReplaceTempView("pv") - r2 = spark.sql( - f"SELECT gbx_custom_pointascell(g, {_GRID}, 0) AS c FROM pv" - ).collect()[0] - assert r2["c"] == r1["c"] - - -def test_cellaswkb_no_srid_and_centroid(spark): - gx.register(spark) - cell = spark.sql( - f"SELECT gbx_custom_pointascell('POINT(530000 180000)', {_GRID}, 0) AS c" - ).collect()[0]["c"] - out = spark.sql( - f"SELECT gbx_custom_cellaswkb({cell}L, {_GRID}) AS w" - ).collect()[0] - g = from_wkb(bytes(out["w"])) - assert g.geom_type == "Polygon" and get_srid(g) == 0 # no SRID stamped - cen = spark.sql( - f"SELECT gbx_custom_centroid({cell}L, {_GRID}) AS w" - ).collect()[0] - assert from_wkb(bytes(cen["w"])).geom_type == "Point" - wkt = spark.sql( - f"SELECT gbx_custom_cellaswkt({cell}L, {_GRID}) AS w" - ).collect()[0] - assert wkt["w"].startswith("POLYGON") - - -def test_polyfill_and_kring_arrays(spark): - gx.register(spark) - df = spark.createDataFrame( - [(bytearray(to_wkb(box(530000.0, 180000.0, 533000.0, 183000.0))),)], "g binary" - ) - df.createOrReplaceTempView("rv") - pf = spark.sql( - f"SELECT size(gbx_custom_polyfill(g, {_GRID}, 0)) AS n FROM rv" - ).collect()[0] - assert pf["n"] == 9 - cell = spark.sql( - f"SELECT gbx_custom_pointascell('POINT(530000 180000)', {_GRID}, 0) AS c" - ).collect()[0]["c"] - kr = spark.sql( - f"SELECT gbx_custom_kring({cell}L, {_GRID}, 1) AS r" - ).collect()[0] - assert cell in kr["r"] and len(set(kr["r"])) == 9 - - -def test_null_propagation(spark): - gx.register(spark) - df = spark.createDataFrame([(None,)], "g binary") - df.createOrReplaceTempView("nv") - r = spark.sql( - f"SELECT gbx_custom_pointascell(g, {_GRID}, 0) AS c FROM nv" - ).collect()[0] - assert r["c"] is None -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** the six UDFs + wrappers. -```python -# --- consuming scalar UDFs -> pandas_udf (bounded scalar) ------------------- -@pandas_udf(LongType()) -def _custom_pointascell_udf(point: pd.Series, grid: pd.Series, res: pd.Series) -> pd.Series: - out = [] - for g, spec, r in zip(point, grid, res): - if g is None or spec is None or r is None: - out.append(None) - continue - pg = parse_geom(g) - if pg is None or pg.is_empty: - out.append(None) - continue - conf = _custom.conf_from_row(spec) - c = pg.representative_point() if pg.geom_type != "Point" else pg - # heavy uses geom.getCoordinate (first coord); for a Point that's its xy. - coord = pg.coords[0] if pg.geom_type == "Point" else list(pg.coords)[0] \ - if hasattr(pg, "coords") else (pg.centroid.x, pg.centroid.y) - out.append(_custom.point_to_cell_id(conf, float(coord[0]), float(coord[1]), int(r))) - return pd.Series(out) - - -@pandas_udf(BinaryType()) -def _custom_cellaswkb_udf(cell: pd.Series, grid: pd.Series) -> pd.Series: - return pd.Series([ - _custom.cell_aswkb(_custom.conf_from_row(s), int(c)) if c is not None and s is not None else None - for c, s in zip(cell, grid) - ]) - - -@pandas_udf(StringType()) -def _custom_cellaswkt_udf(cell: pd.Series, grid: pd.Series) -> pd.Series: - return pd.Series([ - _custom.cell_aswkt(_custom.conf_from_row(s), int(c)) if c is not None and s is not None else None - for c, s in zip(cell, grid) - ]) - - -@pandas_udf(BinaryType()) -def _custom_centroid_udf(cell: pd.Series, grid: pd.Series) -> pd.Series: - return pd.Series([ - _custom.cell_centroid(_custom.conf_from_row(s), int(c)) if c is not None and s is not None else None - for c, s in zip(cell, grid) - ]) - - -# --- array-output -> plain @udf (row-by-row, scale-safe) -------------------- -def _custom_polyfill(geom, grid, res): - if geom is None or grid is None or res is None: - return None - return _custom.polyfill(_custom.conf_from_row(grid), parse_geom(geom), int(res)) - - -def _custom_kring(cell, grid, k): - if cell is None or grid is None or k is None: - return None - return _custom.k_ring(_custom.conf_from_row(grid), int(cell), int(k)) -``` - Note on `pointascell` coordinate extraction: heavy uses `geom.getCoordinate` (the FIRST coordinate of the geometry), NOT the centroid (unlike BNG). Port that exactly — for a `POINT` it's the point's xy; for any other geom it's the first vertex. Simplify the Step-3 body to `coord = pg.coords[0]` for Point else the first coordinate via `shapely.get_coordinates(pg)[0]`; verify against heavy in Task 9. - - Extend `register(spark)` (after the BNG block): -```python - _env.assert_custom_available() - spark.udf.register("gbx_custom_grid", _custom_grid_udf) # (+ 7-arg handling, Task 6) - spark.udf.register("gbx_custom_pointascell", _custom_pointascell_udf) - spark.udf.register("gbx_custom_cellaswkb", _custom_cellaswkb_udf) - spark.udf.register("gbx_custom_cellaswkt", _custom_cellaswkt_udf) - spark.udf.register("gbx_custom_centroid", _custom_centroid_udf) - spark.udf.register("gbx_custom_polyfill", _custom_polyfill, ArrayType(LongType())) - spark.udf.register("gbx_custom_kring", _custom_kring, ArrayType(LongType())) -``` - Column wrappers (mirror heavy `gridx.custom.functions`; `custom_grid` ALWAYS supplies all 8 args, defaulting `srid=-1`): -```python -def custom_grid(x_min, x_max, y_min, y_max, cell_splits, root_x, root_y, srid: ColLike = -1) -> Column: - """Build a custom-grid spec STRUCT (validated eagerly). srid=-1 means no CRS.""" - return f.call_function( - "gbx_custom_grid", _col(x_min), _col(x_max), _col(y_min), _col(y_max), - _col(cell_splits), _col(root_x), _col(root_y), _col(srid), - ) - - -def custom_pointascell(point: ColLike, grid: ColLike, res: ColLike) -> Column: - """Custom-grid cell ID (BIGINT) for a point geometry at `res`.""" - return f.call_function("gbx_custom_pointascell", _col(point), _col(grid), _col(res)) - - -def custom_cellaswkb(cell: ColLike, grid: ColLike) -> Column: - """Cell footprint polygon as plain WKB (no SRID) BINARY.""" - return f.call_function("gbx_custom_cellaswkb", _col(cell), _col(grid)) - - -def custom_cellaswkt(cell: ColLike, grid: ColLike) -> Column: - """Cell footprint polygon as WKT (STRING).""" - return f.call_function("gbx_custom_cellaswkt", _col(cell), _col(grid)) - - -def custom_centroid(cell: ColLike, grid: ColLike) -> Column: - """Cell centroid point as plain WKB (no SRID) BINARY.""" - return f.call_function("gbx_custom_centroid", _col(cell), _col(grid)) - - -def custom_polyfill(geom: ColLike, grid: ColLike, res: ColLike) -> Column: - """ARRAY of cells whose center is contained by the geometry.""" - return f.call_function("gbx_custom_polyfill", _col(geom), _col(grid), _col(res)) - - -def custom_kring(cell: ColLike, grid: ColLike, k: ColLike) -> Column: - """ARRAY of cells within Chebyshev ring distance `k` (includes center).""" - return f.call_function("gbx_custom_kring", _col(cell), _col(grid), _col(k)) -``` - -- [ ] **Step 4: run → PASS** (all `test_custom_functions.py`). Confirm no `_jvm`/`conf`/`.rdd` added to pygx (grep) — the existing Serverless guard test covers `functions.py`. - -- [ ] **Step 5: commit** (`feat(pygx): register 6 custom-grid consuming fns + 7 Column wrappers`). - ---- - -## Task 8: HEAVY Scala — fix the `pointToCellID` Y-NaN typo + scalastyle + suite green - -**Files:** modify `src/main/scala/com/databricks/labs/gbx/gridx/grid/CustomGridSystem.scala`; rebuild guidance. - -Resolved decision 3: fix the `pointToCellID` Y-NaN typo. **No geom-decoder change is needed** — `Custom_PointAsCell.decodeGeom` already accepts BINARY (`JTS.fromWKB`) and STRING (`JTS.fromWKT`), both of which strip EWKB/EWKT SRID, so all four encodings already work in heavy (confirmed by reading `Custom_PointAsCell.scala:60-66`; `Custom_Polyfill` reuses it). - -- [ ] **Step 1:** edit `CustomGridSystem.scala:250` — change -```scala - require(!x.isNaN && !x.isNaN, throw new IllegalStateException("NaN coordinates are not supported.")) -``` -to -```scala - require(!x.isNaN && !y.isNaN, throw new IllegalStateException("NaN coordinates are not supported.")) -``` - Add a one-line comment referencing the fix (the second clause was a duplicate-`x` typo that left a NaN Y unguarded). -- [ ] **Step 2:** dispatch a Task subagent (Docker, long-running) to run scalastyle + the custom Scala suite: - - `bash scripts/commands/gbx-lint-scalastyle.sh --log scalastyle-custom.log` - - `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.gridx.custom.*' --log scala-custom.log` (and any `CustomGridSystem`/`GridConf` suite under `gridx.grid.*`). If a heavy test asserts the OLD (Y-unguarded) behavior, update it to expect the NaN-Y rejection (the fix is approved). Surface a one-line progress update ~every 30s while it runs. -- [ ] **Step 3:** rebuild + stage the JAR so Task 9 parity can run heavy against the FIXED behavior: `bash scripts/commands/gbx-data-push-jar.sh` (stages both fat jar + tests.jar); confirm `python/geobrix/lib/geobrix-0.4.0-jar-with-dependencies.jar` exists. -- [ ] **Step 4:** add a beta-release-notes line (`docs/docs/beta-release-notes.mdx`) noting custom gridding is now available in both tiers AND that the `gbx_custom_pointascell`/`polyfill` NaN-Y guard was corrected (was silently unguarded). Keep user-facing voice (no wave/internal vocabulary). -- [ ] **Step 5: commit** (`fix(gridx): guard NaN Y in CustomGridSystem.pointToCellID (was duplicate-x typo)`). - ---- - -## Task 9: cross-tier EXACT parity (JAR-gated) — cells + WKB geom + all-4-encodings + Y-NaN lock-in - -**Files:** create `test/pygx/test_parity_custom.py` (mirror `test_parity_bng.py`). - -- [ ] **Step 1:** confirm the JAR is staged (Task 8 Step 3). The test auto-skips without it. -- [ ] **Step 2: write the JAR-gated parity test** — copy the gating block + `spark_with_jar` fixture from `test/pygx/test_parity_bng.py` (the `_JARS` glob on `parents[2]/"lib"`, the active-session skip, `appName="gbx-pygx-custom-parity"`). Register light then heavy under the SAME `gbx_custom_*` SQL names: collect ALL light results first (`from databricks.labs.gbx.pygx import functions as gx; gx.register(spark)`), then heavy overwrites them (`from databricks.labs.gbx.gridx.custom import functions as hx; hx.register(spark)` — confirm the heavy registration entry point; if heavy registers via the package `functions.register(spark)` use that), then collect heavy. Build the SAME grid spec in both tiers (`gbx_custom_grid(0,1000000,0,1000000,2,1000,1000,27700)`). Assert: - - **Exact cell-ID / set**: `pointascell` (same BIGINT), `polyfill` (sorted cell-set equality), `kring` (sorted set equality). Include edge cells (origin cell at `(100,100)`, a max-corner cell near `(999900,999900)`), a multi-resolution grid (`cell_splits` 2 and 4, res 0 and a deeper res), and a grid **with** (`srid=27700`) AND **without** (`srid=-1`) a CRS — the cell ids must be identical regardless of srid (srid is metadata only). - - **Geometry WKB within 1e-6**: decode both tiers' WKB (`shapely.from_wkb`), assert `get_srid == 0` in BOTH (custom carries no SRID), and `equals_exact(lg.normalize(), hg.normalize(), 1e-6)` for `cellaswkb`, `centroid`. `cellaswkt` decoded via `from_wkt`, compared the same way. - - **All-4-encodings geom input** (Resolved decision 2): for `pointascell` and `polyfill`, feed the SAME geometry as WKB, EWKB (SRID-stamped bytes via `shapely.to_wkb(g, include_srid=True)` on a `set_srid`'d geom), WKT, and EWKT (`"SRID=27700;POINT(...)"`); assert ALL four produce the identical cell id / cell set in BOTH tiers (and that light == heavy for each encoding). - - **Y-NaN lock-in** (Resolved decision 3): build a geometry whose coordinate has a NaN Y is not directly expressible via WKT, so assert at the `_custom`/heavy boundary: in LIGHT, `_custom.point_to_cell_id(conf, 530000.0, float("nan"), 0)` raises `ValueError`; in HEAVY, a `gbx_custom_pointascell` over a point built with NaN Y raises (or, if NaN cannot round-trip through WKB, assert the light-tier guard directly and comment that the heavy fix is covered by the heavy Scala suite in Task 8). Document the chosen approach in the test docstring. - - **Contingency**: if any cell set diverges, fix the `_custom.py` port until exact — EXACT parity is the bar (no tolerance on cell IDs). Likely divergence points: the `_trunc_long` (truncate-toward-zero vs floor), the `+1` polyfill over-scan range bounds, and the `pointascell` first-coordinate-vs-centroid choice. -- [ ] **Step 3: run in Docker** `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pygx/test_parity_custom.py --with-integration --log parity-custom.log` → green (skips only without JAR). -- [ ] **Step 4: commit** (`test(pygx): light-vs-heavy custom-grid exact parity (cells + WKB + 4-encodings + NaN-Y)`). - ---- - -## Task 10: function-info + binding parity (7 already in registered_functions.txt) - -**Files:** verify `docs/tests/python/api/gridx_functions_sql.py` (the 7 `custom_*_sql_example()` exist — confirmed present, lines 434–479) + confirm all 7 `custom_*` Column wrappers exist in `functions.py`. - -- [ ] **Step 1:** confirm the 7 `gbx_custom_*` are in `docs/tests-function-info/registered_functions.txt` (they are — lines 144–150) and each has a `*_sql_example()` in `gridx_functions_sql.py` (they do — `custom_grid`, `custom_pointascell`, `custom_cellaswkb`, `custom_cellaswkt`, `custom_centroid`, `custom_polyfill`, `custom_kring`). No new entries — this task verifies parity, not adds. -- [ ] **Step 2:** run `bash scripts/commands/gbx-test-bindings.sh --log bindings-custom.log` → PASS (every `gbx_custom_*` present in Scala `override def name` + Python `functions.py` binding + `function-info.json` key). Fix upstream if it fails (a missing Python wrapper is the likely gap — all 7 `custom_*` wrappers from Task 7 must be importable). -- [ ] **Step 3:** regen function-info if any example changed (`bash scripts/commands/gbx-docs-function-info.sh`); the examples are unchanged so this should be a no-op. Commit if anything changed (`docs(pygx): custom-grid binding/function-info parity`); otherwise note "no change — parity already holds" and skip. - ---- - -## Task 11: bench harness — custom-grid legs + `--grid-custom-only` - -**Files:** modify `python/geobrix/src/databricks/labs/gbx/bench/` (`corpus_vector.py` add custom generators; `readers.py` add `run_custom_*` + `_register_custom`; `cluster.py` add `_CELL_GRID_CUSTOM` + `benchmark_grid_custom`/`grid_custom_only` config); `notebooks/tests/push_and_run_bench_on_cluster.py` (`--benchmark-grid-custom` / `--grid-custom-only` flags). - -- [ ] **Step 1:** add custom-grid corpus generators to `corpus_vector.py` mirroring the BNG block (`generate_custom_points` → points within the grid extent for `pointascell`; `generate_custom_polygons` → polygons within the extent for `polyfill`; `generate_custom_cells` → single BIGINT cell ids for `cellaswkb`/`cellaswkt`/`centroid`/`kring`, computed via pure-Python `_custom` so both tiers consume identical inputs). Use a fixed grid spec (`0,1000000,0,1000000,2,1000,1000,27700`) shared by both tiers. -- [ ] **Step 2:** add `_register_custom(spark, api)` (paralleling `_register_bng`) and representative legs to `readers.py`: `run_custom_pointascell` (scalar encode), `run_custom_polyfill` (geom→array), `run_custom_kring` (cell-in→array), `run_custom_cellaswkb` (cell→WKB, the UDF-boundary leg). Same `(spark, run_id, warmup, measured, *, api, n_rows, res, where)` shape, `category="grid"`, light-vs-heavy timing + **exact cell-set / decoded-geom parity** verdicts. Each leg builds the grid via `gbx_custom_grid(...)` so both tiers share the struct. -- [ ] **Step 3:** add `_CELL_GRID_CUSTOM` to `cluster.py` (mirror `_CELL_GRID_BNG`: light leg collected before heavy registration), wire `benchmark_grid_custom` / `grid_custom_only` config keys (parallel to `benchmark_grid_bng` / `grid_bng_only` at the `cfg.get(...)` sites + the `BENCHMARK_GRID_CUSTOM`/`GRID_CUSTOM_ONLY` template constants + the `cells.append(_cell(_CELL_GRID_CUSTOM))` branch), and the launcher `--benchmark-grid-custom` / `--grid-custom-only` flags in `push_and_run_bench_on_cluster.py` (mirror the `--benchmark-grid-bng` / `--grid-bng-only` wiring at lines ~280–283, 346–347, 504–507, 536, 612–613 + the `run_id` suffix `-grid-custom` + the skip-fn-benchmarks branch). -- [ ] **Step 4: local smoke** at tiny scale in the `geobrix-dev` container (resolve SQL, both tiers run, parity verdicts PASS). Do NOT run the cluster here. -- [ ] **Step 5: commit** (`feat(bench): custom-grid light-vs-heavy bench legs (--grid-custom-only)`). - ---- - -## Task 12: cluster bench run - -- [ ] **Step 1:** controller-orchestrated (not a subagent): build+stage JAR+wheel to the sample-data Volume (the JAR already carries the Task 8 fix); restart the standing 0519 bench cluster (restart, don't auto-terminate mid-iteration); poll libs INSTALLED; run `gbx:bench:cluster --grid-custom-only` ONCE (pass `--row-counts 1000`; verify exactly one geobrix-bench run on the cluster's cluster_id); verify rows; fetch `summary.md` (give the user the `bench-out//summary.md` link, unprompted). -- [ ] **Step 2:** record the light-vs-heavy medians + exact-parity verdicts (for the docs in Task 13). No commit (bench writes to the Volume/table). Note any custom function materially slower than heavy (the WKB-UDF-boundary tax applies to `cellaswkb`/`centroid`; pure cell-id ops `pointascell`/`polyfill`/`kring` should be competitive-to-faster — no JVM/JTS). Terminate the cluster only if started FRESH; if it was the standing 0519 cluster, suggest termination, don't auto-kill. - ---- - -## Task 13: docs — flip custom grids heavy→both on every surface (+ supersede the spec note) - -**Files:** `docs/docs/api/gridx-functions.mdx`, `execution-tiers.mdx`, `performance.mdx`, `benchmarking.mdx`, `README.md`, `docs/src/pages/index.js`, `docs/docs/intro.mdx`; the pygx light-tier spec out-of-scope note; `function-info`. - -- [ ] **Step 1: `gridx-functions.mdx`** — flip the **Custom Grid Functions** section badges heavy→both: the section header (line 1027, ` Custom-grid functions ... are heavyweight-only`) becomes `` with a lightweight note ("Powered by a pure-Python port of the custom-grid system + shapely; identical `gbx_custom_*` SQL names — a drop-in swap. Cell ids are BIGINT; geometry outputs are plain WKB, no SRID — the grid's `srid` is metadata only."), and EVERY per-function `### gbx_custom_*` badge → `` (all 7: grid, pointascell, cellaswkb, cellaswkt, centroid, polyfill, kring). Update the top-of-page summary line (line 15) so GridX is **fully** lightweight (quadbin + BNG + custom all both-tier — remove the "while the custom-grid functions remain heavyweight-only" clause). Document that geom inputs accept WKB/EWKB/WKT/EWKT in both tiers (Resolved decision 2) and the `pointascell` first-coordinate semantics. -- [ ] **Step 2: `execution-tiers.mdx`** — remove custom grids from the heavyweight-only reasons (line 45: drop "You need the heavy-only GridX custom grids" — keep the `conforming` triangulation clause; line 47: drop "GridX's custom-grid APIs" from the remaining-heavyweight-only sentence). Update the GridX framing: "GridX is now FULLY lightweight — quadbin, BNG, AND custom grids run in both tiers." The remaining heavyweight-only surfaces become: the vector OGR readers, the `conforming` triangulation mode, and the heavy `pmtiles` DataSource writer. -- [ ] **Step 3: `performance.mdx`** — extend the "GridX (pygx)" subsection with the custom cell ops: the execution-shapes (scalar pandas-UDF `pointascell` + cell geometry `cellaswkb`/`cellaswkt`/`centroid`, the array `@udf` `polyfill`/`kring`, the validating `@udf` `gbx_custom_grid`), the `pygx/_custom.py` module row in the modules table (pure-Python custom-grid system port + shapely), and the perf narrative from the Task 12 numbers (STRING/BIGINT cell-id ops competitive; WKB-geometry ops carry the UDF-boundary tax). -- [ ] **Step 4: `benchmarking.mdx`** — extend the **Grid tab** with a custom-grid subsection: light (pygx) vs heavy (gridx.custom) with exact-output parity across the representative shapes (scalar encode `gbx_custom_pointascell`, geom→cell-array `gbx_custom_polyfill`, cell-in→array `gbx_custom_kring`, cell→WKB `gbx_custom_cellaswkb`), with the Task 12 medians + parity verdicts, and the `gbx:bench:cluster --grid-custom-only` invocation (per the bench-changes-update-docs rule, the numbers go here in the same stroke as Task 12). -- [ ] **Step 5: README / `index.js` / `intro.mdx`** — flip custom to lightweight-available: - - `README.md`: GridX bullet → "GridX — BNG, Quadbin, AND custom grids all in both tiers (lightweight `pygx` + heavyweight Scala)." - - `docs/src/pages/index.js`: GridX card + the heavyweight-only line — custom no longer heavy-only; GridX fully lightweight. - - `docs/docs/intro.mdx`: lightweight tier now covers RasterX, VectorX, and the **full GridX** (BNG + quadbin + custom); GridX no longer appears in any heavyweight-only enumeration. -- [ ] **Step 6: supersede the pygx light-tier spec note** — in `docs/superpowers/specs/2026-06-14-pygx-light-tier-design.md`, update the "Out of scope" custom-gridding bullet to note it is superseded by `2026-06-14-pygx-custom-gridding-light-tier-design.md` (custom now in both tiers). -- [ ] **Step 7:** `function-info` regen if any example changed (`bash scripts/commands/gbx-docs-function-info.sh`); `cd docs && npm run build` → SUCCESS; `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/` → empty (QC internals-leak gate). -- [ ] **Step 8: commit** (`docs(pygx): custom-grid lightweight tier across all surfaces`). - ---- - -## Self-Review - -**Spec coverage:** all 7 custom functions (Tasks 1–7, enumerated against `registered_functions.txt` lines 144–150); pure-Python port of `CustomGridSystem.scala`/`GridConf.scala`, no PyPI lib (Tasks 1–5); WKB no-SRID for custom (Tasks 3, 7, 9); cell-ID bit-packing `res << 56 | (posY * totalCellsX + posX)` ported verbatim (Task 1, locked in Task 9); `maxResolution = min(20, floor(56/bitsPerResolution))` cell_splits-dependent (Task 1, Resolved decision 4); `pointToCellID` four guards in order (Task 2); Y-NaN typo FIXED in BOTH tiers (Task 8 heavy + Task 2 light + Task 9 lock-in, Resolved decision 3); `polyfill` `+1` over-scan ported as INTENTIONAL (Task 5); `kring` Chebyshev clamp to `totalCells` (Task 4); `gbx_custom_grid` = validating `@udf` with eager `require(...)` parity (Task 6, Resolved decision 1); geom inputs all-4-encodings BOTH tiers, heavy already supports it (no decoder extension), light via `parse_geom` (Tasks 7, 9, Resolved decision 2); EXACT cell-set parity (Task 9); no `*_agg`/`*explode` (none added — heavy has none); Serverless-safe udf-only (Task 7 guard); no new deps (shapely only); bench legs + `--grid-custom-only` (Tasks 11–12); all doc surfaces incl. performance.mdx + the spec out-of-scope supersede (Task 13); function-info/binding parity (Task 10); TDD ordering: Spark-free core FIRST (Tasks 1–5), registered-fn (Tasks 6–7), JAR-gated parity (Task 9). - -**Heavy findings (authored into the plan):** the `pointToCellID` Y-NaN typo is at `CustomGridSystem.scala:250` (`require(!x.isNaN && !x.isNaN, ...)` — second clause repeats `x`). Heavy ALREADY accepts all 4 geom encodings via `Custom_PointAsCell.decodeGeom` (`Custom_PointAsCell.scala:60-66`, `fromWKB`/`fromWKT` strip EWKB/EWKT SRID), reused by `Custom_Polyfill` — so NO geom-decoder extension is required; the only heavy change is the one-character Y-NaN fix (Task 8). `JTS.toWKB` (`JTS.scala:159`) is the 2D no-SRID variant. - -**Placeholder scan:** the soft spots are intentional faithful-port markers with the Task-9 parity test as the exact definition of done: (a) the `gbx_custom_grid` 7-vs-8-arg arity has an explicit verify-in-Step-4 note with a concrete robust solution (Python default `srid=-1` bound by position) and a documented fallback; (b) the `pointascell` first-coordinate-vs-centroid choice carries an explicit note to port `geom.getCoordinate` (first vertex) and verify in Task 9; (c) `_trunc_long` truncate-toward-zero (vs `floor`) and the `polyfill` `range(first, last+2)` bounds are called out as the likely divergence points in the Task-9 contingency. All 7 registrations are listed verbatim in the `register` block; nothing is left unnamed. - -**Name / type / formula consistency:** `_custom` fn names (`CustomGridConf`, `conf_from_row`, `point_to_cell_id`, `cell_id_to_polygon`/`cell_id_to_centroid`, `cell_aswkb`/`cell_aswkt`/`cell_centroid`, `polyfill`, `k_ring`, `total_cells_x/y`, `cell_width/height`, `get_cell_id`/`get_cell_resolution`/`get_cell_position`/`get_cell_position_x/y`, `get_cell_position_from_positions`/`from_coordinates`, `get_cell_center_x/y`), SQL names (`gbx_custom_*`, exactly the 7), schema (`CUSTOM_GRID_SCHEMA`, fields matching `Custom_GridSpec.gridStructType`), and the bit-packing formula (`cellPos | (res << 56)`, `posY * totalCellsX + posX`) + `maxResolution` formula are identical across Tasks 1–13. diff --git a/docs/superpowers/plans/2026-06-14-pygx-quadbin.md b/docs/superpowers/plans/2026-06-14-pygx-quadbin.md deleted file mode 100644 index 3cc58a898..000000000 --- a/docs/superpowers/plans/2026-06-14-pygx-quadbin.md +++ /dev/null @@ -1,536 +0,0 @@ -# pygx Phase 1 — quadbin Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Ship the 10 lightweight quadbin GridX functions in a new pure-Python `databricks.labs.gbx.pygx` package, at exact cell-set parity with the heavy Scala tier. - -**Architecture:** Pure-Python/PySpark, Serverless/Connect-safe (only `spark.udf.register`/`spark.udtf.register` + Column exprs — never `_jvm`/`spark.conf.set`/`.rdd`). Cell math via the `quadbin` PyPI lib (already a dep) + ported `Quadbin.scala` logic where the lib lacks a primitive; geometry via shapely → **EWKB** (SRID 4326). Mirrors the just-completed `pyvx` package patterns. - -**Tech Stack:** Python 3.12, `quadbin` (0.2.x), `shapely` 2.x, PySpark `@udf`/`pandas_udf`. **No new dependencies** (`quadbin` + `shapely` already in the `[light]` extra). - -**Spec:** `docs/superpowers/specs/2026-06-14-pygx-light-tier-design.md` (Phase 1). **Branch:** `pygx-light`. Out of scope: BNG (separate plan), `gbx_custom_*` (stays heavy-only). - ---- - -## The 10 functions (parity targets, all `gbx_quadbin_*`) - -| Function | Shape | Output | Light source | -|---|---|---|---| -| `pointascell(lon,lat,res)` | scalar | BIGINT | `quadbin.point_to_cell(lon,lat,res)`; res ∈ [0,26] | -| `resolution(cell)` | scalar | INT | `quadbin.get_resolution(cell)` | -| `kring(cell,k)` | scalar | ARRAY\ | `quadbin.k_ring(cell,k)`; k ≥ 0 | -| `distance(a,b)` | scalar | INT | **custom**: same-res-or-error, Chebyshev on `cell_to_tile` coords | -| `polyfill(geom,res)` | scalar | ARRAY\ | **custom**: bbox-cell enumeration matching `Quadbin.scala`; res ∈ [0,20] | -| `aswkb(cell)` | scalar | BINARY (EWKB polygon, SRID 4326) | `cell_to_bounding_box` → shapely box → `to_wkb(include_srid=True)` | -| `centroid(cell)` | scalar | BINARY (EWKB point) | `cell_to_point` → shapely Point → EWKB | -| `cellunion(cells)` | scalar | BINARY (EWKB MultiPolygon) | per-cell polygon → shapely `unary_union` → EWKB | -| `tessellate(geom,res)` | scalar | ARRAY\\> | polyfill bbox → per-cell shapely intersection → EWKB | -| `cellunion_agg(cell)` | grouped-agg | BINARY (EWKB MultiPolygon) | grouped-agg `pandas_udf` returning BINARY directly | - -Heavy reference: `src/main/scala/com/databricks/labs/gbx/gridx/quadbin/` + `gridx/grid/Quadbin.scala`. Key heavy facts: `resolution = ((cell >>> 52) & 0x1f).toInt`; geometry outputs use `JTS.toEWKB` after `setSRID(4326)`; `distance` requires equal resolution; `polyfill`/`tessellate` use the geometry **envelope** (bbox), res ≤ 20. - -## Conventions (every task) -- Spark-free core tests (`_quadbin.py`) run on host: `.venv-pyrx/bin/python -m pytest -v`. Registered-fn + parity tests run in the `geobrix-dev` container: `bash scripts/commands/gbx-test-python.sh --path --log .log`. -- Commit (no push unless a task says so): `chmod -R u+rwX .git/objects`; subject ≤72 + WHY body; trailer exactly `Co-authored-by: Isaac`. -- Serverless guard: never add `_jvm`/`sparkContext`/`.rdd`/`spark.conf.set` to `pygx`. - -## File Structure -| File | Responsibility | -|---|---| -| `python/geobrix/src/databricks/labs/gbx/pygx/__init__.py` | package marker (docstring) | -| `pygx/_env.py` | `assert_quadbin_available()` (quadbin + shapely guard) | -| `pygx/_geom.py` | `parse_geom` (WKB/EWKB/WKT/EWKT) — copy of pyvx's | -| `pygx/_serde.py` | `QUADBIN_CELL_SCHEMA` (tessellate struct) | -| `pygx/_quadbin.py` | cell math (lib + ported logic) + shapely EWKB geometry build | -| `pygx/functions.py` | `register(spark)` + UDFs/agg + Column wrappers | -| `python/geobrix/test/pygx/conftest.py` | spark fixture (copy of pyvx's) | -| `python/geobrix/test/pygx/test_quadbin_core.py` | Spark-free `_quadbin` unit tests | -| `python/geobrix/test/pygx/test_quadbin_udf.py` | registered-fn tests | -| `python/geobrix/test/pygx/test_parity_quadbin.py` | JAR-gated cross-tier parity | -| `python/geobrix/test/pygx/test_geom.py` | `parse_geom` unit tests | - ---- - -## Task 1: Package skeleton + env guard + geom + serde - -**Files:** create `pygx/__init__.py`, `pygx/_env.py`, `pygx/_geom.py`, `pygx/_serde.py`, `test/pygx/__init__.py`, `test/pygx/conftest.py`, `test/pygx/test_geom.py`. - -- [ ] **Step 1: failing test** `test/pygx/test_geom.py` -```python -import pytest -shapely = pytest.importorskip("shapely") -from shapely import to_wkb, set_srid, get_srid # noqa: E402 -from shapely.geometry import Point # noqa: E402 -from databricks.labs.gbx.pygx import _geom - - -def test_parse_wkb_wkt_ewkt_ewkb_none(): - assert _geom.parse_geom(None) is None - assert _geom.parse_geom("") is None - assert _geom.parse_geom(to_wkb(Point(1, 2))).equals(Point(1, 2)) - assert _geom.parse_geom("POINT (1 2)").equals(Point(1, 2)) - g = _geom.parse_geom("SRID=4326;POINT (1 2)") - assert g.equals(Point(1, 2)) and get_srid(g) == 4326 - e = _geom.parse_geom(to_wkb(set_srid(Point(1, 2), 4326), include_srid=True)) - assert get_srid(e) == 4326 -``` - -- [ ] **Step 2: run → FAIL** `.venv-pyrx/bin/python -m pytest python/geobrix/test/pygx/test_geom.py -v` (no `pygx`). - -- [ ] **Step 3: implement the package files** - -`pygx/__init__.py`: -```python -"""pygx — pure-Python/PySpark light GridX tier (Serverless-safe). - -Mirrors the heavyweight ``gridx`` functions (``gbx_quadbin_*``, ``gbx_bng_*``) -with no JVM, no JAR, no native GDAL. See databricks.labs.gbx.pygx.functions. -""" -``` - -`pygx/_geom.py` — copy `python/geobrix/src/databricks/labs/gbx/pyvx/_geom.py` verbatim (the `parse_geom(x)` handling WKB/EWKB bytes + WKT/EWKT text, empty→None, `SRID=` prefix → `set_srid`). - -`pygx/_env.py`: -```python -"""Environment checks for the pygx light tier.""" - - -def assert_quadbin_available() -> None: - """Raise a clear ImportError if the quadbin light deps are missing.""" - missing = [] - try: - import quadbin # noqa: F401 - except Exception: # noqa: BLE001 - missing.append("quadbin") - try: - import shapely # noqa: F401 - except Exception: # noqa: BLE001 - missing.append("shapely") - if missing: - raise ImportError( - "pygx quadbin requires the [light] extra; missing: " - + ", ".join(missing) - + ". Install with: pip install 'geobrix[light]'" - ) -``` - -`pygx/_serde.py`: -```python -from pyspark.sql.types import BinaryType, LongType, StructField, StructType - -QUADBIN_CELL_SCHEMA = StructType( - [ - StructField("cell", LongType(), False), - StructField("geom", BinaryType(), True), - ] -) -``` - -`test/pygx/__init__.py` — empty. `test/pygx/conftest.py` — copy `test/pyvx/conftest.py` verbatim (change `appName` to `"pygx-tests"`). - -- [ ] **Step 4: run → PASS** (6 assertions). - -- [ ] **Step 5: commit** -```bash -git add python/geobrix/src/databricks/labs/gbx/pygx/ python/geobrix/test/pygx/ -git commit -m "feat(pygx): package skeleton + geom/env/serde for quadbin - -New pure-Python GridX package; parse_geom (WKB/EWKB/WKT/EWKT) mirroring pyvx, -quadbin/shapely env guard, tessellate cell struct schema. Spark-free, TDD." -``` - ---- - -## Task 2: `_quadbin.py` cell-ID math — pointascell, resolution, kring, distance - -**Files:** create `pygx/_quadbin.py`, `test/pygx/test_quadbin_core.py`. - -The `quadbin` 0.2.x public API (verified): `point_to_cell`, `get_resolution`, `k_ring`, `cell_to_point`, `cell_to_boundary`, `cell_to_bounding_box`, `cell_to_tile`, `tile_to_cell`, `geometry_to_cells`, `cell_area`. There is **no** `cell_distance`/`polyfill_bbox`. - -- [ ] **Step 1: failing tests** `test/pygx/test_quadbin_core.py` -```python -import pytest -pytest.importorskip("quadbin") -import quadbin # noqa: E402 -from databricks.labs.gbx.pygx import _quadbin - - -def test_pointascell_matches_lib(): - cell = _quadbin.point_as_cell(-122.4194, 37.7749, 10) - assert cell == quadbin.point_to_cell(-122.4194, 37.7749, 10) - assert _quadbin.resolution(cell) == 10 - - -def test_resolution_bitformula(): - cell = quadbin.point_to_cell(0.0, 0.0, 14) - assert _quadbin.resolution(cell) == ((cell >> 52) & 0x1F) # heavy formula - - -def test_kring_matches_lib_and_includes_center(): - cell = quadbin.point_to_cell(0.0, 0.0, 10) - ring = _quadbin.k_ring(cell, 1) - assert cell in ring and len(ring) == 9 and sorted(ring) == sorted(quadbin.k_ring(cell, 1)) - - -def test_distance_same_resolution_chebyshev(): - a = quadbin.point_to_cell(0.0, 0.0, 10) - b = quadbin.point_to_cell(0.5, 0.5, 10) - ta, tb = quadbin.cell_to_tile(a), quadbin.cell_to_tile(b) - expected = max(abs(ta[0] - tb[0]), abs(ta[1] - tb[1])) - assert _quadbin.distance(a, b) == expected - - -def test_distance_mismatched_resolution_raises(): - a = quadbin.point_to_cell(0.0, 0.0, 10) - b = quadbin.point_to_cell(0.0, 0.0, 11) - with pytest.raises(ValueError, match="same resolution"): - _quadbin.distance(a, b) - - -def test_pointascell_resolution_validation(): - with pytest.raises(ValueError): - _quadbin.point_as_cell(0.0, 0.0, 27) # > 26 -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** the four functions in `pygx/_quadbin.py`. `cell_to_tile` returns `(x, y, z)`; use indices 0/1 for the Chebyshev distance. Mirror `Quadbin.scala` validation (`point_as_cell` res ∈ [0,26]; `distance` equal-resolution). -```python -"""Pure-Python quadbin GridX core for the pygx light tier. - -Cell math via the `quadbin` package; logic the package lacks (distance, bbox -polyfill) is ported to match the heavy `gridx/grid/Quadbin.scala` exactly. -Geometry outputs are EWKB (SRID 4326), matching heavy's JTS.toEWKB. -""" -import quadbin - -_MAX_RES = 26 -_MAX_POLYFILL_RES = 20 - - -def point_as_cell(lon: float, lat: float, resolution: int) -> int: - z = int(resolution) - if z < 0 or z > _MAX_RES: - raise ValueError(f"quadbin resolution must be in [0,{_MAX_RES}]; got {z}") - return quadbin.point_to_cell(float(lon), float(lat), z) - - -def resolution(cell: int) -> int: - return quadbin.get_resolution(int(cell)) - - -def k_ring(cell: int, k: int) -> list: - if int(k) < 0: - raise ValueError(f"k must be >= 0; got {k}") - return list(quadbin.k_ring(int(cell), int(k))) - - -def distance(cell_a: int, cell_b: int) -> int: - if resolution(cell_a) != resolution(cell_b): - raise ValueError("quadbin_distance: cells must be at same resolution") - ax, ay = quadbin.cell_to_tile(int(cell_a))[:2] - bx, by = quadbin.cell_to_tile(int(cell_b))[:2] - return int(max(abs(ax - bx), abs(ay - by))) -``` - -- [ ] **Step 4: run → PASS** (6 tests). - -- [ ] **Step 5: commit** (`feat(pygx): quadbin cell-id math (pointascell/resolution/kring/distance)`). - ---- - -## Task 3: `_quadbin.py` geometry — aswkb, centroid, cellunion (EWKB) - -**Files:** modify `pygx/_quadbin.py`, `test/pygx/test_quadbin_core.py`. - -Heavy emits **EWKB** (`JTS.toEWKB`, SRID 4326). Light: shapely `to_wkb(geom, include_srid=True)` after `set_srid(geom, 4326)`. Use `quadbin.cell_to_bounding_box(cell)` → `(west, south, east, north)` for the polygon, `quadbin.cell_to_point(cell)` for the centroid (returns a GeoJSON-ish point or `(lon,lat)` — verify the return shape in Step 3 and adapt). - -- [ ] **Step 1: failing tests** (append) -```python -from shapely import from_wkb, get_srid # noqa: E402 - -def test_aswkb_is_ewkb_polygon_srid4326(): - cell = quadbin.point_to_cell(0.0, 0.0, 10) - g = from_wkb(_quadbin.as_wkb(cell)) - assert g.geom_type == "Polygon" and get_srid(g) == 4326 - w, s, e, n = quadbin.cell_to_bounding_box(cell) - assert abs(g.bounds[0] - w) < 1e-9 and abs(g.bounds[2] - e) < 1e-9 - - -def test_centroid_is_ewkb_point_srid4326(): - cell = quadbin.point_to_cell(0.0, 0.0, 10) - g = from_wkb(_quadbin.centroid(cell)) - assert g.geom_type == "Point" and get_srid(g) == 4326 - - -def test_cellunion_is_ewkb_and_covers_cells(): - cells = quadbin.k_ring(quadbin.point_to_cell(0.0, 0.0, 8), 1) - g = from_wkb(_quadbin.cell_union(list(cells))) - assert g.geom_type in ("Polygon", "MultiPolygon") and get_srid(g) == 4326 - - -def test_cellunion_empty_or_none_is_none(): - assert _quadbin.cell_union([]) is None - assert _quadbin.cell_union(None) is None -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** (append to `_quadbin.py`). First confirm the `cell_to_point` / `cell_to_bounding_box` return shapes via `.venv-pyrx/bin/python -c "import quadbin; print(quadbin.cell_to_bounding_box(quadbin.point_to_cell(0,0,10))); print(quadbin.cell_to_point(quadbin.point_to_cell(0,0,10)))"` and adapt the unpacking. -```python -from shapely import set_srid, to_wkb, union_all -from shapely.geometry import Point, box - - -def _ewkb(geom) -> bytes: - return to_wkb(set_srid(geom, 4326), include_srid=True) - - -def as_wkb(cell: int) -> bytes: - w, s, e, n = quadbin.cell_to_bounding_box(int(cell)) - return _ewkb(box(w, s, e, n)) - - -def centroid(cell: int) -> bytes: - pt = quadbin.cell_to_point(int(cell)) # confirm shape in Step 3; expect [lon, lat] or geojson - lon, lat = (pt["coordinates"] if isinstance(pt, dict) else pt) - return _ewkb(Point(lon, lat)) - - -def cell_union(cells) -> "bytes | None": - if not cells: - return None - polys = [box(*quadbin.cell_to_bounding_box(int(c))) for c in cells if c is not None] - if not polys: - return None - return _ewkb(union_all(polys)) -``` - -- [ ] **Step 4: run → PASS.** - -- [ ] **Step 5: commit** (`feat(pygx): quadbin geometry — aswkb/centroid/cellunion (EWKB SRID 4326)`). - ---- - -## Task 4: `_quadbin.py` polyfill + tessellate - -**Files:** modify `pygx/_quadbin.py`, `test/pygx/test_quadbin_core.py`. - -Heavy `polyfill`/`tessellate` use the geometry **envelope** (bbox) at the resolution. Port `Quadbin.scala`'s bbox-cell enumeration: compute the bbox `(w,s,e,n)`, then enumerate every cell whose tile lies within the bbox tile range at `res`. Reference `gridx/grid/Quadbin.scala` for the exact enumeration (tile range from the corner cells via `cell_to_tile`/`tile_to_cell`); the parity test (Task 7) is the exact-cell-set definition of done. Tessellate = polyfill the bbox, then per cell intersect its polygon with the input geom and emit `(cell, EWKB(intersection))`, dropping empty intersections, matching heavy. - -- [ ] **Step 1: failing tests** (append; use `_geom.parse_geom` for input) -```python -from shapely.geometry import box as _box # noqa: E402 -from shapely import to_wkb as _to_wkb # noqa: E402 - -def test_polyfill_bbox_cells_resolution(): - geom = _to_wkb(_box(-0.1, -0.1, 0.1, 0.1)) - cells = _quadbin.polyfill(geom, 12) - assert len(cells) > 0 - assert all(_quadbin.resolution(c) == 12 for c in cells) - - -def test_polyfill_resolution_validation(): - import pytest - with pytest.raises(ValueError): - _quadbin.polyfill(_to_wkb(_box(0, 0, 1, 1)), 21) # > 20 - - -def test_tessellate_returns_cell_geom_pairs(): - geom = _to_wkb(_box(-0.05, -0.05, 0.05, 0.05)) - chips = _quadbin.tessellate(geom, 12) - assert len(chips) > 0 - cell0, gwkb0 = chips[0] - assert isinstance(cell0, int) - from shapely import from_wkb, get_srid - g0 = from_wkb(gwkb0) - assert get_srid(g0) == 4326 and not g0.is_empty -``` - -- [ ] **Step 2: run → FAIL.** - -- [ ] **Step 3: implement** `polyfill(geom, res)` and `tessellate(geom, res)` in `_quadbin.py`. Use `_geom.parse_geom` on the input; compute `.bounds`; enumerate bbox cells matching `Quadbin.scala` (validate res ∈ [0,20]); tessellate intersects each cell's `box(*cell_to_bounding_box)` with the parsed geom. Return `polyfill` → `list[int]`; `tessellate` → `list[(int, bytes)]`. (Import `parse_geom` from `._geom`.) **The exact enumeration must match `Quadbin.scala`; if `geometry_to_cells` on the bbox polygon yields the same set, prefer it; otherwise port the tile-range loop.** Verify against heavy in Task 7. - -- [ ] **Step 4: run → PASS.** - -- [ ] **Step 5: commit** (`feat(pygx): quadbin polyfill + tessellate (bbox cells, EWKB chips)`). - ---- - -## Task 5: `functions.py` — register + UDFs + agg + wrappers - -**Files:** create `pygx/functions.py`, `test/pygx/test_quadbin_udf.py`. - -Mirror `pyvx/functions.py`: `ColLike`, `_col`, scalar `spark.udf.register` with explicit return types, a grouped-agg `pandas_udf` returning BINARY for `cellunion_agg`, and Column wrappers. `tessellate` returns `ARRAY>` via a plain `@f.udf(ArrayType(QUADBIN_CELL_SCHEMA))` (heavy returns an array, not exploded — no UDTF needed). `polyfill`/`kring`/`cellunion` return `ArrayType(LongType())` or `BinaryType()`. - -- [ ] **Step 1: failing test** `test/pygx/test_quadbin_udf.py` -```python -import pytest -pytest.importorskip("quadbin") -shapely = pytest.importorskip("shapely") -from shapely import from_wkb, get_srid, to_wkb # noqa: E402 -from shapely.geometry import box # noqa: E402 -from databricks.labs.gbx.pygx import functions as gx - - -def test_pointascell_and_resolution(spark): - gx.register(spark) - row = spark.sql("SELECT gbx_quadbin_pointascell(-122.4194, 37.7749, 10) AS c").collect()[0] - import quadbin - assert row["c"] == quadbin.point_to_cell(-122.4194, 37.7749, 10) - r = spark.sql(f"SELECT gbx_quadbin_resolution({row['c']}) AS r").collect()[0] - assert r["r"] == 10 - - -def test_aswkb_ewkb(spark): - gx.register(spark) - import quadbin - c = quadbin.point_to_cell(0.0, 0.0, 10) - out = spark.sql(f"SELECT gbx_quadbin_aswkb({c}) AS w").collect()[0] - g = from_wkb(bytes(out["w"])) - assert g.geom_type == "Polygon" and get_srid(g) == 4326 - - -def test_tessellate_struct_array(spark): - gx.register(spark) - df = spark.createDataFrame([(bytearray(to_wkb(box(-0.05, -0.05, 0.05, 0.05))),)], "g binary") - df.createOrReplaceTempView("v") - rows = spark.sql("SELECT t.cell, t.geom FROM v LATERAL VIEW explode(gbx_quadbin_tessellate(g, 12)) AS t").collect() - assert len(rows) > 0 and isinstance(rows[0]["cell"], int) - - -def test_cellunion_agg(spark): - gx.register(spark) - import quadbin - cells = list(quadbin.k_ring(quadbin.point_to_cell(0.0, 0.0, 8), 1)) - df = spark.createDataFrame([(c,) for c in cells], "cell long") - out = df.agg(gx.quadbin_cellunion_agg("cell").alias("u")).collect()[0] - g = from_wkb(bytes(out["u"])) - assert g.geom_type in ("Polygon", "MultiPolygon") and get_srid(g) == 4326 -``` - -- [ ] **Step 2: run → FAIL** (`bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pygx/test_quadbin_udf.py --log quadbin-udf.log`). - -- [ ] **Step 3: implement `functions.py`.** Imports + `ColLike`/`_col` copied from `pyvx/functions.py`. Scalar UDFs wrap `_quadbin` functions; register with explicit return types. `cellunion_agg` is a grouped-agg `pandas_udf(BinaryType())` taking one `pd.Series` of cells → `_quadbin.cell_union(list-of-cells)`. Example skeleton: -```python -from typing import List, Union -import pandas as pd -from pyspark.sql import Column, SparkSession -from pyspark.sql import functions as f -from pyspark.sql.functions import pandas_udf -from pyspark.sql.types import ArrayType, BinaryType, IntegerType, LongType - -from . import _env, _geom, _quadbin -from ._serde import QUADBIN_CELL_SCHEMA - -ColLike = Union[Column, str, bool, int, float, bytes] - -def _col(x: ColLike): - return x if isinstance(x, (Column, str)) else f.lit(x) - -def _pointascell(lon, lat, res): return _quadbin.point_as_cell(lon, lat, res) -def _resolution(cell): return _quadbin.resolution(cell) -def _kring(cell, k): return _quadbin.k_ring(cell, k) -def _distance(a, b): return _quadbin.distance(a, b) -def _polyfill(geom, res): return _quadbin.polyfill(geom, res) -def _aswkb(cell): return _quadbin.as_wkb(cell) -def _centroid(cell): return _quadbin.centroid(cell) -def _cellunion(cells): return _quadbin.cell_union(list(cells) if cells else cells) -def _tessellate(geom, res): - return [(int(c), g) for (c, g) in _quadbin.tessellate(geom, res)] - -@pandas_udf(BinaryType()) -def _cellunion_agg_udf(cell: pd.Series) -> bytes: - return _quadbin.cell_union([int(c) for c in cell if c is not None]) - -def register(spark: SparkSession = None) -> None: - """Register the pygx quadbin SQL functions (Serverless-safe: udf only).""" - _env.assert_quadbin_available() - if spark is None: - spark = SparkSession.builder.getOrCreate() - spark.udf.register("gbx_quadbin_pointascell", _pointascell, LongType()) - spark.udf.register("gbx_quadbin_resolution", _resolution, IntegerType()) - spark.udf.register("gbx_quadbin_kring", _kring, ArrayType(LongType())) - spark.udf.register("gbx_quadbin_distance", _distance, IntegerType()) - spark.udf.register("gbx_quadbin_polyfill", _polyfill, ArrayType(LongType())) - spark.udf.register("gbx_quadbin_aswkb", _aswkb, BinaryType()) - spark.udf.register("gbx_quadbin_centroid", _centroid, BinaryType()) - spark.udf.register("gbx_quadbin_cellunion", _cellunion, BinaryType()) - spark.udf.register("gbx_quadbin_tessellate", _tessellate, ArrayType(QUADBIN_CELL_SCHEMA)) - spark.udf.register("gbx_quadbin_cellunion_agg", _cellunion_agg_udf) -``` -Then add Column wrappers mirroring `pyvx` (`quadbin_pointascell(lon,lat,res)` → `f.call_function("gbx_quadbin_pointascell", _col(lon),_col(lat),_col(res))`, etc., and `quadbin_cellunion_agg(cell)` → `_cellunion_agg_udf(_col(cell))`). - -- [ ] **Step 4: run → PASS** (4 tests). Also run the Serverless guard (`test/pyrx/test_serverless_no_spark_config.py` if it scans pygx, else confirm no `_jvm`/`conf` in pygx). - -- [ ] **Step 5: commit** (`feat(pygx): register 10 quadbin functions (udf + grouped-agg)`). - ---- - -## Task 6: function-info + bindings - -**Files:** modify `docs/tests/python/api/gridx_functions_sql.py` (only if quadbin examples need the `mode`/arg refresh — they exist already; verify), `pygx/functions.py` (ensure all 10 Column wrappers exist for import-parity). - -- [ ] **Step 1:** confirm the 10 `gbx_quadbin_*` are in `docs/tests-function-info/registered_functions.txt` (they are) and have `*_sql_example()` in `docs/tests/python/api/gridx_functions_sql.py` (they do). No new entries needed; this task verifies parity, not adds. -- [ ] **Step 2:** run `bash scripts/commands/gbx-test-bindings.sh --log bindings-quadbin.log` → PASS (every registered quadbin fn present in Scala + Python + function-info). Fix upstream if it fails. -- [ ] **Step 3: commit** if any wrapper/example changed (`docs(pygx): quadbin binding/function-info parity`); otherwise note "no change — parity already holds" and skip. - ---- - -## Task 7: cross-tier parity (JAR-gated) - -**Files:** create `test/pygx/test_parity_quadbin.py`. - -- [ ] **Step 1: rebuild + stage the JAR** (heavy unchanged so far, but the parity test needs a JAR present): `bash scripts/commands/gbx-data-push-jar.sh` then copy `target/geobrix-0.4.0-jar-with-dependencies.jar` → `python/geobrix/lib/`. (If unchanged from a prior stage, a present JAR suffices.) - -- [ ] **Step 2: write the JAR-gated parity test** — copy the gating block from `test/pyvx/test_parity_mvt.py` (the `_JARS` glob on `parents[2]/"lib"` + `spark_with_jar` fixture + active-session skip). Register light then heavy (`from databricks.labs.gbx.pygx import functions as gx; gx.register(spark)`; heavy via `from databricks.labs.gbx.gridx.quadbin import functions as hx; hx.register(spark)` — collect light result first, then heavy overwrites the SQL name, then collect heavy). Assert per function: - - **Exact**: `pointascell`, `resolution`, `kring` (sorted set equality), `distance`, `polyfill` (sorted cell-set equality), `tessellate` (cell-set equality), `cellunion`/`cellunion_agg` (decoded-geometry equality), over a deterministic point/geometry/cell-list fixture. - - **Geometry within 1e-6**: decode both tiers' EWKB (`shapely.from_wkb`), assert `get_srid==4326` both, and coordinates equal within 1e-6 (use `g_light.equals_exact(g_heavy, 1e-6)` or `.normalize()` compare). - - **Contingency:** if `pointascell`/`polyfill`/`tessellate` cell sets diverge (lib vs `Quadbin.scala`), port the exact `Quadbin.scala` encode/enumeration into `_quadbin.py` until exact — exact parity is the bar (no tolerance on cell IDs). - -- [ ] **Step 3: run in Docker** `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pygx/test_parity_quadbin.py --with-integration --log parity-quadbin.log` → green (skips only without JAR). - -- [ ] **Step 4: commit** (`test(pygx): light-vs-heavy quadbin exact parity (cells + EWKB geom)`). - ---- - -## Task 8: bench harness — quadbin legs - -**Files:** modify `python/geobrix/src/databricks/labs/gbx/bench/` (`corpus_grid.py` new or extend `corpus_vector.py`; `readers.py` `run_quadbin_*`; `cluster.py` `_CELL_GRID_QUADBIN`); `notebooks/tests/push_and_run_bench_on_cluster.py` (`--grid-quadbin-only`). - -- [ ] **Step 1:** add a corpus generator (points for pointascell; geometries for polyfill/tessellate; cell-id arrays for cellunion) mirroring `bench/corpus_vector.py` style. -- [ ] **Step 2:** add `run_quadbin_pointascell` / `run_quadbin_polyfill` / `run_quadbin_tessellate` / `run_quadbin_cellunion_agg` (representative coverage: a scalar, a geom→array, a struct-array, an agg) to `readers.py`, mirroring `run_legacy_aswkb`/`run_triangulate` signatures, with light-vs-heavy timing + **exact cell-set / decoded-geom parity** assertions. -- [ ] **Step 3:** add `_CELL_GRID_QUADBIN` to `cluster.py` (mirror `_CELL_VECTOR_TIN`: light leg collected before heavy registration) + the launcher `--grid-quadbin-only` flag (mirror `--vector-tin-only`). -- [ ] **Step 4: local smoke** at tiny scale in the `geobrix-dev` container (resolve SQL, both tiers run, parity verdicts PASS). Do NOT run the cluster here. -- [ ] **Step 5: commit** (`feat(bench): quadbin light-vs-heavy bench legs (--grid-quadbin-only)`). - ---- - -## Task 9: cluster bench run - -- [ ] **Step 1:** controller-orchestrated (not a subagent): build+stage JAR+wheel to the sample-data Volume; restart the bench cluster; poll libs INSTALLED; run `gbx:bench:cluster --grid-quadbin-only` once; verify one run + rows; fetch `summary.md`; terminate the cluster after capture. -- [ ] **Step 2:** record the light-vs-heavy medians + exact-parity verdicts (for the docs in Task 10). No commit (bench writes to the Volume/table). - ---- - -## Task 10: docs — all surfaces - -**Files:** `docs/docs/api/gridx-functions.mdx`, `execution-tiers.mdx`, `performance.mdx`, `benchmarking.mdx`, `README.md`, `docs/src/pages/index.js`, `docs/docs/intro.mdx`; `function-info`. - -- [ ] **Step 1: `gridx-functions.mdx`** — the page currently has a single page-level ``. Restructure so the **quadbin section** carries `` with a lightweight note ("Powered by the **quadbin** package + shapely; the `quadbin_distance`/`quadbin_polyfill` cell math mirrors the heavy implementation"), while the **custom-grid** and **BNG** sections keep ``. Document the EWKB SRID-4326 geometry outputs. -- [ ] **Step 2: `execution-tiers.mdx`** — in the "heavyweight-only" reasons, move quadbin out (it's now both-tier) while KEEPING `gbx_custom_*` (custom grids) **and** BNG (`gbx_bng_*`) heavyweight-only. Update the GridX framing accordingly. -- [ ] **Step 3: `performance.mdx`** — add a "GridX (pygx)" subsection: quadbin execution shapes (scalar UDFs + the `cellunion_agg` grouped-agg + the `tessellate` array-UDF), the `pygx/_quadbin.py` module (quadbin lib + shapely), and the perf narrative from the Task 9 numbers. -- [ ] **Step 4: `benchmarking.mdx`** — fill the **Grid tab** (currently "Grid (soon)") with the quadbin light-vs-heavy timing + exact-parity verdicts from Task 9. -- [ ] **Step 5: README / `index.js` / `intro.mdx`** — reflect quadbin lightweight availability. README GridX bullet: note quadbin is now lightweight (`pygx`), BNG still heavyweight (planned). `index.js` GridX card + the heavyweight-only line. `intro.mdx` note pygx alongside pyrx/pyvx. -- [ ] **Step 6:** `function-info` regen if examples changed; `cd docs && npm run build` → SUCCESS; `grep -rn -iE "wave [0-9]+" docs/docs/` → empty. -- [ ] **Step 7: commit** (`docs(pygx): quadbin lightweight tier across all surfaces`). - ---- - -## Self-Review - -**Spec coverage:** ✅ all 10 functions (Tasks 2–5); ✅ EWKB SRID-4326 (Tasks 3–4); ✅ distance/polyfill ported-not-lib (Tasks 2,4, corrected); ✅ exact cell-set parity (Task 7); ✅ Serverless-safe (Task 5 guard); ✅ no new deps (uses existing quadbin/shapely); ✅ bench (Tasks 8–9); ✅ all doc surfaces incl. performance.mdx + keeping custom-grid/BNG heavy-only (Task 10); ✅ function-info/binding parity (Task 6). - -**Placeholder scan:** the two soft spots are intentional: `_quadbin.centroid` unpacking is verified-in-step-3 (lib return shape), and `polyfill`/`tessellate` enumeration references `Quadbin.scala` with the parity test (Task 7) as the exact definition of done — a faithful-port pattern (same as pyvx's Sloan), not a placeholder. - -**Type consistency:** `_quadbin` function names (`point_as_cell`, `resolution`, `k_ring`, `distance`, `as_wkb`, `centroid`, `cell_union`, `polyfill`, `tessellate`), SQL names (`gbx_quadbin_*`), schema (`QUADBIN_CELL_SCHEMA`), and the `register`/wrapper shapes are consistent across tasks. diff --git a/docs/superpowers/plans/2026-06-20-eo-series-stac-refactor.md b/docs/superpowers/plans/2026-06-20-eo-series-stac-refactor.md deleted file mode 100644 index 1aef0c6df..000000000 --- a/docs/superpowers/plans/2026-06-20-eo-series-stac-refactor.md +++ /dev/null @@ -1,224 +0,0 @@ -# EO-Series → StacClient Refactor Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. - -**Goal:** Refactor the EO-series example notebooks to perform STAC search/download/repair via the new `databricks.labs.gbx.stac.StacClient`, removing the redundant per-notebook STAC + resilient-download helpers, while preserving the `band_` table contract that nb03/nb04 consume. - -**Architecture:** nb01 search → `client.search()`; nb02 download+repair → `client.download()` + `client.repair()`, with each per-band table rebuilt as a notebook-level join (download results ⋈ per-item metadata). nb03/nb04 are NOT modified — they read the same downstream columns. config_nb installs `geobrix[light,stac]`, instantiates the client, and keeps only the non-STAC utilities. - -**Tech Stack:** Databricks Serverless (environment version 5, Python 3.12), PySpark, `databricks.labs.gbx.stac`, `databricks.labs.gbx.pyrx` (rx), Delta. - -## Global Constraints - -- **Serverless-safe:** NO `spark.conf.set` (use `set_conf_safe`), NO `.cache()`/`.persist()`, parallelism via `DataFrame.repartition(N)` only. (eo-series already follows this.) -- **`band_` table contract (nb02 → nb03):** MUST contain at least `item_id`, `band_name`, `date`, `out_file_path`. MUST also carry `is_out_file_valid` + `out_file_sz` (nb02's own repair reads them). All other legacy columns (`timestamp`, `h3_set`, `item_collection`, `stac_version`, `item_bbox`, `item_properties`, `asset`, `out_dir_fuse`, `out_filename`, `last_update`) may be DROPPED — `finalize_tiled_band_tbl` discards them and nb03/nb04 never read them. -- **`band_*_h3` contract (nb03 → nb04):** unchanged (`cellid`, `date`, `band_name`, `tile`); nb03/nb04 untouched. -- **`StacClient` API (verbatim):** - - `StacClient(catalog=PLANETARY_COMPUTER, sign="planetary_computer")` - - `search(df, geojson_col, collections: List[str], datetime: str, partitions=512) -> DataFrame` → carried-input-cols + `item_id, date, item_bbox, asset_name, href, item_properties`. One row per (input-row, item, asset). - - `download(df, out_dir, asset_names=None, name="{asset_name}_{item_id}.tif", validate=True, max_tries=5, partitions=None) -> DataFrame` → `item_id, asset_name, out_file_path, out_file_sz, is_out_file_valid, last_update`. Input df must have `item_id` + `asset_name`; href is RE-SIGNED per attempt from item_id+asset_name (a stale search href is not required). Dedups to unique `(item_id, asset_name)` internally. - - `repair(table_or_df, where="is_out_file_valid = false") -> DataFrame` → re-downloads invalid rows, Delta MERGE back, returns repaired subset. -- **No restartPython in config_nb** (it is `%run`-ed; restart wipes the caller's context). The `%pip install` lives in config_nb's own cells which run before the `%run`-based imports — keep the existing placement. -- **Install line:** config_nb installs `geobrix[light,stac]` from the staged wheel (`file:///Volumes/geospatial_docs/geobrix/sample-data/geobrix-0.4.0-py3-none-any.whl`). -- **Notebook edits:** edit `.ipynb` via the NotebookEdit tool (these are Jupyter JSON). Keep existing `displayHTML` screenshot cells intact (they are the committed visual output). -- **Collection/datetime:** the series uses collection `"sentinel-2-l2a"` and a datetime range string like `"2022-06-01/2022-06-01"`. The geojson column is named `"geojson"`. - ---- - -### Task 1: config_nb — install `[light,stac]`, instantiate StacClient, strip redundant helpers - -**Files:** -- Modify: `notebooks/examples/eo-series/config_nb.ipynb` -- Modify: `notebooks/examples/eo-series/library.py` - -**Interfaces:** -- Produces: a module/global `stac_client = StacClient(...)` available to nb01/nb02 after `%run ./config_nb`; retains `set_conf_safe`, `file_size`, `timestamp_filename`, `get_now_formatted`, `finalize_tiled_band_tbl`, `gen_tessellate_tiled_band`, `FORCE_REBUILD`, and all viz helpers (`plot_raster`, `plot_file`, `to_numpy_arr`, `rasterio_lambda`, `_decimated_read`, `_percentile_stretch`). - -- [ ] **Step 1: Update the `%pip install` cell** so the GeoBrix line installs the stac extra. Change the `geobrix[light]` line to: - ``` - %pip install --quiet "geobrix[light,stac] @ file:///Volumes/geospatial_docs/geobrix/sample-data/geobrix-0.4.0-py3-none-any.whl" - ``` - Remove `pystac pystac_client planetary_computer tenacity` from the SECOND `%pip` line (they now come via `[stac]`); KEEP `folium mapclassify geopandas rich` (viz deps not in `[stac]`). - -- [ ] **Step 2: Add a StacClient instantiation cell** (after the imports cell that does `from databricks.labs.gbx.stac import StacClient`). Add the import to the existing import block and a cell: - ```python - from databricks.labs.gbx.stac import StacClient - stac_client = StacClient() # default catalog = Planetary Computer, sign = planetary_computer - ``` - -- [ ] **Step 3: Remove the redundant STAC + download helpers from `library.py`:** delete `ps_client`, `get_items`, `get_assets`, `get_assets_for_cells`, `download_asset`, `download_asset_v2`. KEEP the viz helpers (`plot_raster`, `plot_file`, `to_numpy_arr`, `rasterio_lambda`, `_decimated_read`, `_percentile_stretch`, `_needs_percentile_stretch`, `_render`) and any non-STAC imports they need. Remove now-unused imports (`pystac_client`, `planetary_computer`, `tenacity`) from library.py if nothing else uses them. - -- [ ] **Step 4: Remove the orchestration helpers from config_nb** that StacClient replaces: `download_band`, `update_assets`, `download_missing_assets`. KEEP `set_conf_safe`, `file_size`, `timestamp_filename`, `get_now_formatted`, `finalize_tiled_band_tbl`, `gen_tessellate_tiled_band`. (These last two are GeoBrix tiling, not STAC.) - -- [ ] **Step 5: Verify config_nb has no remaining references** to the removed names. Grep the eo-series dir: - Run: `grep -rn "get_assets_for_cells\|download_asset_v2\|download_asset(\|download_band\|download_missing_assets\|update_assets\|ps_client\|library.get_items\|library.get_assets" notebooks/examples/eo-series/` - Expected: only references inside nb01/nb02 that Tasks 2–3 will fix (config_nb + library.py clean). - -- [ ] **Step 6: Commit.** - ```bash - git add notebooks/examples/eo-series/config_nb.ipynb notebooks/examples/eo-series/library.py - git commit -m "refactor(eo-series): config_nb installs [light,stac] + StacClient; drop redundant STAC/download helpers" - ``` - ---- - -### Task 2: nb01 — search via `client.search()` - -**Files:** -- Modify: `notebooks/examples/eo-series/01. Search STACs.ipynb` - -**Interfaces:** -- Consumes: `stac_client` (Task 1), `df_cell_json` (H3 cell rows with a `"geojson"` column), `time_range` (e.g. `"2022-06-01/2022-06-01"`). -- Produces: the `cell_assets_<...>` Delta table with `client.search()` columns: carried `cellid` (+ any other carried input cols), `item_id`, `date`, `item_bbox`, `asset_name`, `href`, `item_properties`. One row per (cell, item, asset). - -- [ ] **Step 1: Replace the main search call.** Find the cell calling `library.get_assets_for_cells(df_cell_json.repartition(512), time_range, "sentinel-2-l2a", spark)` and replace with: - ```python - cell_assets_df = stac_client.search( - df_cell_json, - geojson_col="geojson", - collections=["sentinel-2-l2a"], - datetime=time_range, - partitions=512, - ) - ``` - Keep the surrounding `if LAST_UPDATED is None`/`FORCE_REBUILD` guard and the table write. If a `last_update` provenance column is desired, add `.withColumn("last_update", F.current_timestamp())` (lazy, Serverless-safe) before the write. - -- [ ] **Step 2: Fix the demo-only search cell** (the manual `library.ps_client.search(...)` viz cell). Replace `library.ps_client` with a local client opened for the demo, OR drop the demo cell if redundant. Minimal replacement: - ```python - import pystac_client, planetary_computer - _demo_cat = pystac_client.Client.open( - "https://planetarycomputer.microsoft.com/api/stac/v1", - modifier=planetary_computer.sign_inplace, - ) - _demo_items = _demo_cat.search(collections=["sentinel-2-l2a"], intersects=region, datetime=time_range).item_collection() - ``` - -- [ ] **Step 3: Adapt any downstream column references in nb01** that used the old schema. The old output had `asset` (map with `.name`/`.href`); the new output has flat `asset_name`/`href`. Update any `asset.name`→`asset_name`, `asset.href`→`href`. The old `timestamp`/`item_collection`/`stac_version` columns no longer exist — remove references (nb02 no longer needs them per the contract). - -- [ ] **Step 4: Update the nb01 markdown** that describes the search step to reference `stac_client.search(...)` and the one-row-per-(cell,item,asset) output (keep the Serverless-strategy notes). Do not leak internal vocabulary. - -- [ ] **Step 5: Commit.** - ```bash - git add "notebooks/examples/eo-series/01. Search STACs.ipynb" - git commit -m "refactor(eo-series): nb01 search via StacClient.search" - ``` - ---- - -### Task 3: nb02 — download + repair via `client.download()` / `client.repair()`, rebuild band tables - -**Files:** -- Modify: `notebooks/examples/eo-series/02. Download STACs.ipynb` - -**Interfaces:** -- Consumes: `stac_client`, the `cell_assets_<...>` table from nb01 (`eod_item_df`), `FORCE_REBUILD`, `EO_DIR`. -- Produces: per-band `band_` Delta tables with columns `item_id, band_name, date, out_file_path, out_file_sz, is_out_file_valid` (+ harmless `last_update`). Satisfies the nb03 contract (`item_id, band_name, date, out_file_path`). - -- [ ] **Step 1: Replace the per-band download.** For each band, replace the `download_band(...)` call with a notebook block that (a) filters search rows to the band, (b) downloads, (c) joins back per-item `date`, (d) writes the band table. Reference implementation (factor into a small local helper `build_band_table(band)` defined in a nb02 cell — it is example-local orchestration, intentionally NOT in config_nb): - ```python - from pyspark.sql import functions as F - - def build_band_table(band: str, eod_item_df, force_rebuild: bool): - band_tbl = f"band_{band.lower()}" - if not force_rebuild and spark.catalog.tableExists(band_tbl): - return spark.read.table(band_tbl) - # one (item, asset) per band; download dedups to unique (item_id, asset_name) - band_rows = eod_item_df.filter(F.col("asset_name") == band) - # per-item metadata to rejoin (date) — distinct so it is one row per item - item_meta = band_rows.select("item_id", "date").distinct() - out_dir = f"{EO_DIR}/{band}" - files = stac_client.download( - band_rows.select("item_id", "asset_name"), - out_dir, - asset_names=[band], - name="{asset_name}_{item_id}.tif", - validate=True, - max_tries=5, - ) - band_df = ( - files.join(item_meta, on="item_id", how="left") - .withColumn("band_name", F.lit(band)) - .select("item_id", "band_name", "date", - "out_file_path", "out_file_sz", "is_out_file_valid", "last_update") - ) - band_df.write.mode("overwrite").saveAsTable(band_tbl) - return spark.read.table(band_tbl) - ``` - -- [ ] **Step 2: Replace the band loop / single-band example.** Where nb02 currently loops `download_band(...)` over the band list, call `build_band_table(band, eod_item_df, FORCE_REBUILD)`. Preserve the example's single-band (`B02`) demonstration cell, now calling `build_band_table("B02", ...)`. - -- [ ] **Step 3: Replace `download_missing_assets(...)` with `client.repair(...)`.** Where nb02 retries invalid rows: - ```python - repaired = stac_client.repair(f"band_{band.lower()}", where="is_out_file_valid = false") - ``` - Keep the dry-run/demo framing as markdown if useful (note `repair` itself has no dry-run; describe it, then run the live call). Remove the `do_dry_run` plumbing. - -- [ ] **Step 4: Update nb02 markdown** to describe `client.download()` (resilient: re-sign per attempt, read-validate, retry/backoff, local-stage→publish; dedups (item_id, asset_name)) and `client.repair()` (Delta MERGE of invalid rows). Keep the Serverless-strategy notes (repartition/Arrow-cap). No internal vocabulary. - -- [ ] **Step 5: Verify no removed-helper references remain.** Run: - `grep -rn "download_band\|download_missing_assets\|update_assets\|download_asset" "notebooks/examples/eo-series/02. Download STACs.ipynb"` - Expected: nothing. - -- [ ] **Step 6: Commit.** - ```bash - git add "notebooks/examples/eo-series/02. Download STACs.ipynb" - git commit -m "refactor(eo-series): nb02 download+repair via StacClient; rebuild band tables as join" - ``` - ---- - -### Task 4: Docs — new STAC API page + eo-series page + README parity - -**Files:** -- Create: `docs/docs/api/stac.mdx` -- Modify: `docs/sidebars.js` -- Modify: `docs/docs/notebooks/eo-series.mdx` -- Modify: `notebooks/examples/eo-series/README.md` - -**Interfaces:** -- Produces: a user-facing STAC API page registered in the API sidebar category. - -- [ ] **Step 1: Create `docs/docs/api/stac.mdx`** — a dedicated STAC API page. Frontmatter: `--- sidebar_position: title: STAC ---` (match the existing api pages' frontmatter style; set a real `title` so the browser tab isn't the logo JSX, per the existing api-page convention). Cover, in user-facing voice (NO internal/wave vocabulary): - - What it is: a lightweight, Serverless-safe STAC client (`databricks.labs.gbx.stac.StacClient`) for **distributed** search + **resilient** download + **repair** against any STAC catalog (default Planetary Computer). Frame the distributed-parallelism advantage over single-node STAC scripts (factual, not marketing) — consistent with the other light-tier overview pages. - - Install: `pip install geobrix[light,stac]` (opt-in extra: pystac-client, planetary-computer, tenacity, requests). Note Serverless environment version 5 (Python 3.12). - - API reference for the three methods with their exact signatures + the output columns (copy from the Global Constraints `StacClient API` block): `search(df, geojson_col, collections, datetime, partitions=512)`, `download(df, out_dir, asset_names=None, name=..., validate=True, max_tries=5, partitions=None)`, `repair(table_or_df, where=...)`. - - A short end-to-end example (search AOI rows → download assets → repair invalid), illustrative code blocks. - - Resilience behavior: re-sign per attempt, read-validation (rejects throttled/truncated), retry/backoff, local-stage→publish-only-when-valid, idempotent skip of already-valid files; dedup to unique (item_id, asset_name). - - Serverless notes: parallelism via `partitions=`/repartition (no spark.conf), no caching (materialize to a Delta table / Volume), one task per asset. - - A link to the EO-series notebooks as the full worked example. - - (Doc-test backing is NOT required for this page — StacClient is an integration/network client; the doc-tests-as-source convention targets the Docker+sample-data SQL/raster examples. Use clear illustrative code blocks and point to the eo-series notebooks for executed end-to-end usage.) - -- [ ] **Step 2: Register the page in `docs/sidebars.js`** in the API category (the `items:` array around lines 76–99), e.g. add `{ type: 'doc', id: 'api/stac', label: 'STAC' }` after the PMTiles entry (`api/pmtiles-functions`) so it sits with the other light-tier API pages. - -- [ ] **Step 3:** Update the eo-series docs page (`docs/docs/notebooks/eo-series.mdx`) + the eo-series `README.md` to describe the STAC step as `StacClient` (search/download/repair) rather than the old per-notebook helpers; note the `geobrix[light,stac]` install; link to the new `api/stac` page. Keep voice user-facing (no wave numbers / internal vocabulary). Preserve the existing lightweight-tier / Serverless framing. - -- [ ] **Step 4: Quick internal-vocab check.** Run: `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/api/stac.mdx docs/docs/notebooks/eo-series.mdx` → must print nothing. - -- [ ] **Step 5: Commit.** - ```bash - git add docs/docs/api/stac.mdx docs/sidebars.js docs/docs/notebooks/eo-series.mdx notebooks/examples/eo-series/README.md - git commit -m "docs(stac): add STAC API page + eo-series page/README describe StacClient + [light,stac]" - ``` - ---- - -### Task 5: Serverless re-validation (nb01 → nb04 chain) - -**Files:** none (validation only). - -This is the acceptance gate. The refactor touches nb01/nb02 schemas; nb03/nb04 must still run green against the rebuilt band tables. Re-stage the wheel (built from current source incl. the stac module) and run the chain on Serverless env v5 with `FORCE_REBUILD=True`. - -- [ ] **Step 1:** Build + stage the `[light,stac]` wheel to `dbfs:/Volumes/geospatial_docs/geobrix/sample-data/geobrix-0.4.0-py3-none-any.whl` (see `/tmp/stac-smoke-task.md` for the exact build+stage commands: `GBX_BUNDLE_SKIP_JAR_UPLOAD=1` build via `.venv-pyrx`, then `databricks fs cp --overwrite` to the sample-data volume; profile `oauth-fe`, `DATABRICKS_AUTH_STORAGE=plaintext`). NOTE: nb03/nb04 use heavy `rx.rst_*`? No — they use the light `rx` (pyrx) tier; the wheel's `[light]` covers them, but if any cell needs the JAR, build WITH the JAR instead. -- [ ] **Step 2:** Run nb01 on Serverless env v5 with `FORCE_REBUILD=True`; confirm SUCCESS and the `cell_assets_<...>` table is written with the new flat schema (`item_id, asset_name, href, date, item_bbox, item_properties, cellid`). -- [ ] **Step 3:** Run nb02 with `FORCE_REBUILD=True`; confirm each `band_` table has `item_id, band_name, date, out_file_path, out_file_sz, is_out_file_valid` and `is_out_file_valid` is mostly true (PC throttling may flag a few — run the `client.repair` cell and confirm it recovers them). -- [ ] **Step 4:** Run nb03 then nb04 with `FORCE_REBUILD=True`; confirm SUCCESS and the final outputs match the previously-validated run (band_*_h3 row counts in the same ballpark, band_stack produced, stacked tifs written). Investigate any schema/contract break. -- [ ] **Step 5:** Report the run URLs + row counts. No commit (validation only); the executed-notebook EXPORT is a separate follow-on phase (not in this plan). - ---- - -## Self-Review notes - -- Spec coverage: config_nb install/client + helper removal (T1), nb01 search (T2), nb02 download+repair+band rebuild (T3), docs — new STAC API page + eo-series page + README (T4), validation (T5). ✔ -- Contract: band_ keeps `item_id/band_name/date/out_file_path` (nb03) + `is_out_file_valid/out_file_sz` (repair); nb03/nb04 untouched. ✔ -- Type consistency: `client.search` output `asset_name`/`href` flat (not `asset` map) — T2/T3 adapt refs. `client.download` needs `item_id`+`asset_name` input — T3 passes exactly those. ✔ -- Serverless: no conf/cache; repartition handled inside StacClient; band write via saveAsTable (no persist). ✔ diff --git a/docs/superpowers/plans/2026-06-20-stac-light-api.md b/docs/superpowers/plans/2026-06-20-stac-light-api.md deleted file mode 100644 index c35f2ca4f..000000000 --- a/docs/superpowers/plans/2026-06-20-stac-light-api.md +++ /dev/null @@ -1,929 +0,0 @@ -# STAC Lightweight API Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build `databricks.labs.gbx.stac` — a catalog-agnostic, Serverless-safe `StacClient` (distributed search + resilient download + repair) consolidating the EO-series STAC helpers. - -**Architecture:** A `StacClient` holds catalog URL + signing config and exposes `search` / `download` / `repair`. Pure parsing/validation helpers are unit-tested without Spark or network; thin pandas-UDF / DataFrame wrappers fan out via `repartition` (no `spark.conf`). Signing is pluggable; the catalog opener is injectable so unit tests use a fake catalog. - -**Tech Stack:** Python 3.12, PySpark (Spark Connect), pandas-UDFs, `pystac-client`, `planetary-computer`, `rasterio` (read-validation), `tenacity` (retry), `requests`, Delta (`delta-spark`) for repair. - -## Global Constraints - -- Serverless-safe: NO `spark.conf.set(...)`, NO `.cache()`/`.persist()`. Parallelism via `DataFrame.repartition(N)` only (a user repartition is not AQE-coalesced). -- Lightweight tier only — pure Python, no JAR, runs on Serverless environment version 5+ (Python 3.12) and classic. -- New deps live in a NEW optional extra `geobrix[stac]` (`pystac-client`, `planetary-computer`); do NOT add them to `[light]`. -- Volume I/O is sequential-only (FUSE can't seek): download to worker-local disk, validate there, publish to the Volume with a sequential copy. -- A downloaded asset is valid IFF it opens AND decodes a window — never size-only. -- Catalog-agnostic: catalog URL + signing are config, default Planetary Computer + `sign_inplace`. -- Test markers: network tests use `@pytest.mark.integration` (excluded from CI), matching `python/geobrix/pyproject.toml`. - -## File Structure - -- Create `python/geobrix/src/databricks/labs/gbx/stac/__init__.py` — exports `StacClient`. -- Create `.../stac/_sign.py` — `resolve_signer(sign) -> Callable[[str], str]`. -- Create `.../stac/_search.py` — pure parsers (`parse_item`, `extract_assets`) + `search_items_udf` builder. -- Create `.../stac/_download.py` — `download_href`, `fetch_validate_publish` (resilient fetch + read-validation). -- Create `.../stac/client.py` — `StacClient` (`search` / `download` / `repair`). -- Create tests under `python/geobrix/test/stac/`: `test_sign.py`, `test_search.py`, `test_download.py`, `test_client.py`, `test_serverless_no_spark_config.py`. -- Modify `python/geobrix/pyproject.toml` — add the `[stac]` optional-dependencies extra. - ---- - -### Task 1: `[stac]` extra + package skeleton - -**Files:** -- Modify: `python/geobrix/pyproject.toml` (`[project.optional-dependencies]`) -- Create: `python/geobrix/src/databricks/labs/gbx/stac/__init__.py` -- Test: `python/geobrix/test/stac/test_package.py` - -**Interfaces:** -- Produces: the importable package `databricks.labs.gbx.stac` exporting `StacClient` (defined in Task 5; until then `__init__` imports lazily so the package imports before the class exists is NOT allowed — so this task creates `__init__` re-exporting from `client`, and a minimal `client.StacClient` stub is created here and fleshed out in Task 5). - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/stac/test_package.py -def test_stac_exports_client(): - from databricks.labs.gbx.stac import StacClient - assert StacClient is not None -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd python/geobrix && python -m pytest test/stac/test_package.py -v` -Expected: FAIL — `ModuleNotFoundError: No module named 'databricks.labs.gbx.stac'` - -- [ ] **Step 3: Add the `[stac]` extra** - -In `python/geobrix/pyproject.toml`, under `[project.optional-dependencies]`, add: - -```toml -# STAC catalog client (lightweight): distributed search + resilient download + repair. -# Optional extra so [light] users who don't need STAC don't pull pystac/planetary-computer. -stac = [ - "pystac-client>=0.7,<1", - "planetary-computer>=1.0,<2", -] -``` - -- [ ] **Step 4: Create the package + a minimal client stub** - -```python -# python/geobrix/src/databricks/labs/gbx/stac/__init__.py -"""Lightweight, Serverless-safe STAC client: search + resilient download + repair.""" -from databricks.labs.gbx.stac.client import StacClient - -__all__ = ["StacClient"] -``` - -```python -# python/geobrix/src/databricks/labs/gbx/stac/client.py -"""StacClient — catalog-agnostic STAC search/download/repair (fleshed out in later tasks).""" - -PLANETARY_COMPUTER = "https://planetarycomputer.microsoft.com/api/stac/v1" - - -class StacClient: - """Holds catalog URL + signing config; exposes search/download/repair.""" - - def __init__(self, catalog=PLANETARY_COMPUTER, sign="planetary_computer", _catalog_opener=None): - self.catalog = catalog - self.sign = sign - self._catalog_opener = _catalog_opener -``` - -- [ ] **Step 5: Run test to verify it passes** - -Run: `cd python/geobrix && python -m pytest test/stac/test_package.py -v` -Expected: PASS - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/pyproject.toml python/geobrix/src/databricks/labs/gbx/stac/ python/geobrix/test/stac/test_package.py -git commit -m "feat(stac): package skeleton + [stac] extra" -``` - ---- - -### Task 2: Signing strategies (`_sign.py`) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/stac/_sign.py` -- Test: `python/geobrix/test/stac/test_sign.py` - -**Interfaces:** -- Produces: `resolve_signer(sign) -> Callable[[str], str]` — maps `"planetary_computer"` to `planetary_computer.sign`, `None` to identity, a callable to itself; raises `ValueError` otherwise. Also `resolve_modifier(sign)` returning a pystac-client `modifier` (for `Client.open`) or `None`. - -- [ ] **Step 1: Write the failing tests** - -```python -# python/geobrix/test/stac/test_sign.py -import pytest -from databricks.labs.gbx.stac._sign import resolve_signer - - -def test_none_is_identity(): - s = resolve_signer(None) - assert s("http://x/y.tif?token=abc") == "http://x/y.tif?token=abc" - - -def test_callable_passthrough(): - s = resolve_signer(lambda h: h + "?signed") - assert s("http://x") == "http://x?signed" - - -def test_unknown_raises(): - with pytest.raises(ValueError): - resolve_signer("not-a-strategy") -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `cd python/geobrix && python -m pytest test/stac/test_sign.py -v` -Expected: FAIL — `ModuleNotFoundError: ... stac._sign` - -- [ ] **Step 3: Implement `_sign.py`** - -```python -# python/geobrix/src/databricks/labs/gbx/stac/_sign.py -"""Signing strategies for STAC asset hrefs. - -A *signer* is ``Callable[[str], str]`` applied to an asset href. A *modifier* is the -pystac-client ``modifier=`` callback applied to each item on search (Planetary -Computer's ``sign_inplace`` mutates item asset hrefs in place). -""" -from typing import Callable, Optional - - -def _identity(href: str) -> str: - return href - - -def resolve_signer(sign) -> Callable[[str], str]: - """Resolve a signer: 'planetary_computer' | None | callable -> Callable[[str],str].""" - if sign is None: - return _identity - if callable(sign): - return sign - if sign == "planetary_computer": - import planetary_computer - - return planetary_computer.sign - raise ValueError( - f"sign must be 'planetary_computer', None, or a callable; got {sign!r}" - ) - - -def resolve_modifier(sign) -> Optional[Callable]: - """Resolve the pystac-client Client.open(modifier=...) for search-time signing.""" - if sign == "planetary_computer": - import planetary_computer - - return planetary_computer.sign_inplace - if sign is None or callable(sign): - # A bare callable signs per-asset at download time, not via the search modifier. - return None - raise ValueError( - f"sign must be 'planetary_computer', None, or a callable; got {sign!r}" - ) -``` - -- [ ] **Step 4: Run to verify it passes** - -Run: `cd python/geobrix && python -m pytest test/stac/test_sign.py -v` -Expected: PASS (3 passed) - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/stac/_sign.py python/geobrix/test/stac/test_sign.py -git commit -m "feat(stac): pluggable signing strategies" -``` - ---- - -### Task 3: Search parsers + UDF (`_search.py`) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/stac/_search.py` -- Test: `python/geobrix/test/stac/test_search.py` - -**Interfaces:** -- Consumes: nothing from other tasks. -- Produces: - - `parse_item(item_json: str) -> dict` — keys `item_id, date (str|None), item_bbox (list|None), item_properties (dict)`. - - `extract_assets(item_json: str) -> list[dict]` — each `{"asset_name": str, "href": str}` (plus passthrough asset fields). - - `search_one(catalog, collections: list[str], datetime: str, geojson: str) -> list[str]` — item JSON strings for one AOI, with tenacity retry; returns `[]` on permanent failure. - -- [ ] **Step 1: Write the failing tests** - -```python -# python/geobrix/test/stac/test_search.py -import json -from databricks.labs.gbx.stac._search import parse_item, extract_assets, search_one - -_ITEM = json.dumps({ - "id": "S2_X", - "collection": "sentinel-2-l2a", - "bbox": [1.0, 2.0, 3.0, 4.0], - "properties": {"datetime": "2022-06-01T19:49:11Z", "eo:cloud_cover": 5}, - "assets": { - "B02": {"href": "http://x/B02.tif", "type": "image/tiff"}, - "B03": {"href": "http://x/B03.tif", "type": "image/tiff"}, - }, -}) - - -def test_parse_item_fields(): - p = parse_item(_ITEM) - assert p["item_id"] == "S2_X" - assert p["date"] == "2022-06-01" - assert p["item_bbox"] == [1.0, 2.0, 3.0, 4.0] - assert p["item_properties"]["eo:cloud_cover"] == 5 - - -def test_extract_assets(): - a = extract_assets(_ITEM) - names = sorted(x["asset_name"] for x in a) - assert names == ["B02", "B03"] - b02 = next(x for x in a if x["asset_name"] == "B02") - assert b02["href"] == "http://x/B02.tif" - - -def test_search_one_uses_catalog_and_retries(monkeypatch): - calls = {"n": 0} - - class FakeItem: - def __init__(self, d): self._d = d - def to_dict(self): return self._d - - class FakeSearch: - def item_collection(self): return [FakeItem(json.loads(_ITEM))] - - class FakeCatalog: - def search(self, collections, intersects, datetime): - calls["n"] += 1 - assert collections == ["sentinel-2-l2a"] - return FakeSearch() - - out = search_one(FakeCatalog(), ["sentinel-2-l2a"], "2022-06-01", '{"type":"Point","coordinates":[1,2]}') - assert calls["n"] == 1 - assert json.loads(out[0])["id"] == "S2_X" -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `cd python/geobrix && python -m pytest test/stac/test_search.py -v` -Expected: FAIL — `ModuleNotFoundError: ... stac._search` - -- [ ] **Step 3: Implement `_search.py`** - -```python -# python/geobrix/src/databricks/labs/gbx/stac/_search.py -"""STAC search internals: pure parsers + a per-AOI search with retry. - -The Spark fan-out (a pandas-UDF over AOI rows) lives in client.py; these helpers are -pure/injectable so they unit-test without Spark or the network. -""" -import json -from typing import Dict, List - - -def parse_item(item_json: str) -> Dict: - """Extract the stable item fields from a STAC item JSON string.""" - d = json.loads(item_json) - props = d.get("properties") or {} - dt = props.get("datetime") - return { - "item_id": d.get("id"), - "date": dt[:10] if isinstance(dt, str) else None, - "item_bbox": d.get("bbox"), - "item_properties": props, - } - - -def extract_assets(item_json: str) -> List[Dict]: - """One dict per asset: {'asset_name', 'href', ...passthrough fields...}.""" - d = json.loads(item_json) - out = [] - for name, asset in (d.get("assets") or {}).items(): - row = {"asset_name": name, "href": asset.get("href")} - for k, v in asset.items(): - if k != "href": - row[k] = v - out.append(row) - return out - - -def search_one(catalog, collections: List[str], datetime: str, geojson: str) -> List[str]: - """Search one AOI; return item JSON strings. Retries transient failures; on a - permanent failure returns [] (so one bad AOI does not fail the whole job).""" - from tenacity import retry, stop_after_attempt, wait_exponential - - @retry(wait=wait_exponential(multiplier=2, min=4, max=60), stop=stop_after_attempt(5), reraise=True) - def _do(): - search = catalog.search( - collections=collections, intersects=json.loads(geojson), datetime=datetime - ) - return [json.dumps(item.to_dict()) for item in search.item_collection()] - - try: - return _do() - except Exception: - return [] -``` - -- [ ] **Step 4: Run to verify it passes** - -Run: `cd python/geobrix && python -m pytest test/stac/test_search.py -v` -Expected: PASS (3 passed) - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/stac/_search.py python/geobrix/test/stac/test_search.py -git commit -m "feat(stac): search parsers + per-AOI search with retry" -``` - ---- - -### Task 4: Resilient download (`_download.py`) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/stac/_download.py` -- Test: `python/geobrix/test/stac/test_download.py` - -**Interfaces:** -- Consumes: nothing. -- Produces: - - `download_href(href, outpath, get=requests.get)` — streams to `outpath`; `raise_for_status()` so HTTP errors raise (retried by caller). - - `read_validate(path) -> bool` — True iff the file opens AND decodes a window (rasterio). - - `fetch_validate_publish(href_fn, out_dir, filename, get=..., max_tries=5, sleep=time.sleep) -> str|None` — download to local temp, read-validate, publish (sequential copy) to `out_dir`; re-fetch with backoff via `href_fn()` (re-signs) up to `max_tries`; returns the published path or None. - -- [ ] **Step 1: Write the failing tests** (use a real tiny GTiff fixture + a garbage file) - -```python -# python/geobrix/test/stac/test_download.py -import os -import numpy as np -import rasterio -from rasterio.transform import from_origin -from databricks.labs.gbx.stac._download import read_validate, fetch_validate_publish - - -def _write_gtiff(path): - with rasterio.open( - path, "w", driver="GTiff", height=8, width=8, count=1, dtype="uint8", - crs="EPSG:4326", transform=from_origin(0, 8, 1, 1), - ) as dst: - dst.write((np.arange(64, dtype="uint8")).reshape(1, 8, 8)) - - -def test_read_validate_true_for_real_gtiff(tmp_path): - p = tmp_path / "ok.tif"; _write_gtiff(str(p)) - assert read_validate(str(p)) is True - - -def test_read_validate_false_for_garbage(tmp_path): - p = tmp_path / "bad.tif"; p.write_bytes(b"throttled" * 100) - assert read_validate(str(p)) is False - - -def test_fetch_publishes_only_valid(tmp_path): - src = tmp_path / "src.tif"; _write_gtiff(str(src)) - out_dir = tmp_path / "out" - - def get(href, timeout=None, stream=None): - class R: - def raise_for_status(self): pass - def iter_content(self, n): yield open(str(src), "rb").read() - return R() - - res = fetch_validate_publish(lambda: "http://x/ok.tif", str(out_dir), "ok.tif", get=get) - assert res == os.path.join(str(out_dir), "ok.tif") - assert os.path.exists(res) - - -def test_fetch_retries_then_gives_up_on_bad(tmp_path): - out_dir = tmp_path / "out" - tries = {"n": 0} - - def get(href, timeout=None, stream=None): - tries["n"] += 1 - - class R: - def raise_for_status(self): pass - def iter_content(self, n): yield b"throttled-not-a-raster" - return R() - - res = fetch_validate_publish( - lambda: "http://x/bad.tif", str(out_dir), "bad.tif", get=get, max_tries=3, sleep=lambda s: None - ) - assert res is None - assert tries["n"] == 3 - assert not os.path.exists(os.path.join(str(out_dir), "bad.tif")) -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `cd python/geobrix && python -m pytest test/stac/test_download.py -v` -Expected: FAIL — `ModuleNotFoundError: ... stac._download` - -- [ ] **Step 3: Implement `_download.py`** - -```python -# python/geobrix/src/databricks/labs/gbx/stac/_download.py -"""Resilient STAC asset download: HTTP-error-aware fetch + read-validation + retry. - -A faithful fetch (no transformation). Validity = the file OPENS and DECODES a window -(rejects throttled error bodies and truncated files a size check would accept). -Volume I/O is sequential-only, so we download to local disk, validate locally, then -publish with a sequential copy. -""" -import os -import shutil -import tempfile -import time -from typing import Callable, Optional - -import requests - - -def download_href(href: str, outpath: str, get: Callable = requests.get) -> str: - """Stream an href to outpath. raise_for_status() so HTTP throttle/expiry (429/403) - raises -> the caller's retry backs off instead of writing the error body as data.""" - resp = get(href, timeout=100, stream=True) - resp.raise_for_status() - with open(outpath, "wb") as fh: - for chunk in resp.iter_content(1024 * 1024): - if chunk: - fh.write(chunk) - return outpath - - -def read_validate(path: str) -> bool: - """True iff the file opens AND decodes a window (a genuine readable raster).""" - import rasterio - from rasterio.windows import Window - - try: - with rasterio.open(path) as ds: - ds.read(1, window=Window(0, 0, min(512, ds.width), min(512, ds.height))) - return True - except Exception: - return False - - -def fetch_validate_publish( - href_fn: Callable[[], str], - out_dir: str, - filename: str, - get: Callable = requests.get, - max_tries: int = 5, - sleep: Callable = time.sleep, -) -> Optional[str]: - """Download -> read-validate -> publish to out_dir (sequential copy), with retries. - - href_fn() is called each attempt so the href is (re-)signed (signed URLs expire). - On any failure (HTTP error, throttled body, truncation, decode failure) back off and - re-fetch up to max_tries; then return None (caller flags is_out_file_valid=False). - """ - outpath = os.path.join(out_dir, filename) - os.makedirs(out_dir, exist_ok=True) - for attempt in range(max_tries): - tmpd = tempfile.mkdtemp(prefix="gbx_stac_dl_") - try: - local = os.path.join(tmpd, filename) - download_href(href_fn(), local, get=get) - if read_validate(local): - shutil.copyfile(local, outpath) # publish only validated files - return outpath - except Exception: - pass - finally: - shutil.rmtree(tmpd, ignore_errors=True) - if attempt < max_tries - 1: - sleep(min(60, 4 * (2 ** attempt))) - try: - if os.path.exists(outpath): - os.remove(outpath) - except OSError: - pass - return None -``` - -- [ ] **Step 4: Run to verify it passes** - -Run: `cd python/geobrix && python -m pytest test/stac/test_download.py -v` -Expected: PASS (4 passed) - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/stac/_download.py python/geobrix/test/stac/test_download.py -git commit -m "feat(stac): resilient download (HTTP-aware fetch + read-validate + retry)" -``` - ---- - -### Task 5: `StacClient` orchestration (`client.py`) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/stac/client.py` -- Test: `python/geobrix/test/stac/test_client.py` - -**Interfaces:** -- Consumes: `_sign.resolve_signer/resolve_modifier`, `_search.parse_item/extract_assets/search_one`, `_download.fetch_validate_publish`. -- Produces: - - `StacClient(catalog=PLANETARY_COMPUTER, sign="planetary_computer", _catalog_opener=None)`. - - `StacClient._open_catalog()` — returns a catalog via `_catalog_opener()` if set, else `pystac_client.Client.open(self.catalog, modifier=resolve_modifier(self.sign))`. - - `StacClient.search(df, geojson_col, collections, datetime, partitions=512) -> DataFrame` — columns: carried input cols (except the items/assets scratch), `item_id, date, item_bbox, item_properties, asset_name, href`. - - `StacClient.download(df, out_dir, asset_names=None, name="{asset_name}_{item_id}.tif", validate=True, max_tries=5, partitions=None) -> DataFrame` — columns: `item_id, asset_name, out_file_path, out_file_sz, is_out_file_valid`. Dedups to unique `(item_id, asset_name)`. - - `StacClient.repair(target, where="is_out_file_valid = false", spark=None) -> DataFrame` — `target` is a table name (str) or DataFrame; re-downloads invalid rows and (for a table) Delta-MERGEs them back; returns the repaired subset. - -- [ ] **Step 1: Write the failing tests** (local SparkSession + injected fake catalog) - -```python -# python/geobrix/test/stac/test_client.py -import json -import pytest -from pyspark.sql import SparkSession -from databricks.labs.gbx.stac import StacClient - -_ITEM = { - "id": "S2_X", "collection": "sentinel-2-l2a", "bbox": [1.0, 2.0, 3.0, 4.0], - "properties": {"datetime": "2022-06-01T19:49:11Z"}, - "assets": {"B02": {"href": "http://x/B02.tif"}, "B03": {"href": "http://x/B03.tif"}}, -} - - -@pytest.fixture(scope="module") -def spark(): - s = SparkSession.builder.master("local[2]").appName("stac-test").getOrCreate() - yield s - s.stop() - - -class _FakeItem: - def __init__(self, d): self._d = d - def to_dict(self): return self._d - - -class _FakeSearch: - def item_collection(self): return [_FakeItem(_ITEM)] - - -class _FakeCatalog: - def search(self, collections, intersects, datetime): return _FakeSearch() - - -def test_search_explodes_items_to_asset_rows(spark): - client = StacClient(_catalog_opener=lambda: _FakeCatalog()) - df = spark.createDataFrame([("cellA", '{"type":"Point","coordinates":[1,2]}')], ["cellid", "geojson"]) - out = client.search(df, geojson_col="geojson", collections=["sentinel-2-l2a"], datetime="2022-06-01", partitions=2) - rows = {(r["cellid"], r["item_id"], r["asset_name"]) for r in out.collect()} - assert rows == {("cellA", "S2_X", "B02"), ("cellA", "S2_X", "B03")} - one = out.filter("asset_name = 'B02'").first() - assert one["date"] == "2022-06-01" and one["href"] == "http://x/B02.tif" -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `cd python/geobrix && python -m pytest test/stac/test_client.py -v` -Expected: FAIL — `AttributeError: 'StacClient' object has no attribute 'search'` - -- [ ] **Step 3: Implement `client.py`** (replace the stub from Task 1) - -```python -# python/geobrix/src/databricks/labs/gbx/stac/client.py -"""StacClient — catalog-agnostic, Serverless-safe STAC search/download/repair. - -Parallelism is via DataFrame.repartition(N) (NOT spark.conf, which is a no-op on -Serverless). No .cache()/.persist(). Asset download is resilient (read-validated + -retried). The catalog opener is injectable (_catalog_opener) for unit tests. -""" -from typing import Callable, List, Optional - -from pyspark.sql import DataFrame, functions as F -from pyspark.sql.types import ( - ArrayType, DoubleType, MapType, StringType, StructField, StructType, -) - -from databricks.labs.gbx.stac import _download, _search -from databricks.labs.gbx.stac._sign import resolve_modifier, resolve_signer - -PLANETARY_COMPUTER = "https://planetarycomputer.microsoft.com/api/stac/v1" - -_ASSET_SCHEMA = ArrayType(StructType([ - StructField("asset_name", StringType()), - StructField("href", StringType()), -])) -_ITEM_SCHEMA = StructType([ - StructField("item_id", StringType()), - StructField("date", StringType()), - StructField("item_bbox", ArrayType(DoubleType())), - StructField("item_properties", MapType(StringType(), StringType())), -]) - - -class StacClient: - def __init__(self, catalog=PLANETARY_COMPUTER, sign="planetary_computer", _catalog_opener=None): - self.catalog = catalog - self.sign = sign - self._catalog_opener = _catalog_opener - - def _open_catalog(self): - if self._catalog_opener is not None: - return self._catalog_opener() - import pystac_client - - return pystac_client.Client.open(self.catalog, modifier=resolve_modifier(self.sign)) - - def search(self, df: DataFrame, geojson_col: str, collections: List[str], - datetime: str, partitions: int = 512) -> DataFrame: - opener = self._catalog_opener - catalog_url, sign = self.catalog, self.sign - - @F.udf(ArrayType(StringType())) - def _items(geojson): - if opener is not None: - cat = opener() - else: - import pystac_client - cat = pystac_client.Client.open(catalog_url, modifier=resolve_modifier(sign)) - return _search.search_one(cat, list(collections), datetime, geojson) - - @F.udf(_ASSET_SCHEMA) - def _assets(item_json): - return [(a["asset_name"], a["href"]) for a in _search.extract_assets(item_json)] - - @F.udf(_ITEM_SCHEMA) - def _item_fields(item_json): - p = _search.parse_item(item_json) - props = {k: str(v) for k, v in (p["item_properties"] or {}).items()} - return (p["item_id"], p["date"], p["item_bbox"], props) - - carried = [c for c in df.columns if c != geojson_col] - return ( - df.repartition(partitions) - .withColumn("_item", F.explode(_items(F.col(geojson_col)))) - .withColumn("_f", _item_fields("_item")) - .withColumn("_a", F.explode(_assets("_item"))) - .select( - *carried, - F.col("_f.item_id").alias("item_id"), - F.col("_f.date").alias("date"), - F.col("_f.item_bbox").alias("item_bbox"), - F.col("_f.item_properties").alias("item_properties"), - F.col("_a.asset_name").alias("asset_name"), - F.col("_a.href").alias("href"), - ) - ) - - def download(self, df: DataFrame, out_dir: str, asset_names: Optional[List[str]] = None, - name: str = "{asset_name}_{item_id}.tif", validate: bool = True, - max_tries: int = 5, partitions: Optional[int] = None) -> DataFrame: - if asset_names: - df = df.filter(F.col("asset_name").isin(list(asset_names))) - targets = df.select("item_id", "asset_name").distinct() - n = partitions if partitions is not None else max(1, targets.count()) - catalog_url, sign, opener = self.catalog, self.sign, self._catalog_opener - - @F.udf(StringType()) - def _fetch(item_id, asset_name): - if opener is not None: - cat = opener() - else: - import pystac_client - cat = pystac_client.Client.open(catalog_url, modifier=resolve_modifier(sign)) - signer = resolve_signer(sign) - - def href_fn(): - item = cat.get_item(item_id) - return signer(item.assets[asset_name].href) - - filename = name.format(asset_name=asset_name, item_id=item_id) - if not validate: - # still download to local + publish, just skip the decode check - from databricks.labs.gbx.stac._download import fetch_validate_publish - return fetch_validate_publish(href_fn, out_dir, filename, max_tries=max_tries) - return _download.fetch_validate_publish(href_fn, out_dir, filename, max_tries=max_tries) - - @F.udf("long") - def _size(path): - import os - return os.path.getsize(path) if path and os.path.exists(path) else None - - return ( - targets.repartition(n) - .withColumn("out_file_path", _fetch("item_id", "asset_name")) - .withColumn("out_file_sz", _size("out_file_path")) - .withColumn("is_out_file_valid", F.col("out_file_path").isNotNull()) - ) - - def repair(self, target, where: str = "is_out_file_valid = false", - spark=None, out_dir: Optional[str] = None) -> DataFrame: - from pyspark.sql import SparkSession - - spark = spark or SparkSession.getActiveSession() - is_table = isinstance(target, str) - df = spark.table(target) if is_table else target - invalid = df.filter(where) - repaired = self.download( - invalid.select("item_id", "asset_name"), - out_dir or _common_dir(invalid), - ) - if is_table: - from delta.tables import DeltaTable - - dt = DeltaTable.forName(spark, target) - (dt.alias("t").merge( - repaired.alias("u"), - "t.item_id = u.item_id AND t.asset_name = u.asset_name") - .whenMatchedUpdate(set={ - "out_file_path": "u.out_file_path", - "out_file_sz": "u.out_file_sz", - "is_out_file_valid": "u.is_out_file_valid", - }).execute()) - return repaired - - -def _common_dir(df: DataFrame) -> str: - """Infer the output dir from existing out_file_path values (repair convenience).""" - import os - - row = df.filter(F.col("out_file_path").isNotNull()).select("out_file_path").first() - if row is None: - raise ValueError("repair: cannot infer out_dir; pass out_dir=...") - return os.path.dirname(row["out_file_path"]) -``` - -- [ ] **Step 4: Run to verify it passes** - -Run: `cd python/geobrix && python -m pytest test/stac/test_client.py -v` -Expected: PASS (1 passed) - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/stac/client.py python/geobrix/test/stac/test_client.py -git commit -m "feat(stac): StacClient search/download/repair orchestration" -``` - ---- - -### Task 6: Serverless-safety guard test + integration test - -**Files:** -- Create: `python/geobrix/test/stac/test_serverless_no_spark_config.py` -- Modify: `python/geobrix/test/stac/test_client.py` (append the marked integration test) - -**Interfaces:** -- Consumes: the whole `stac` package source. -- Produces: a static guard test asserting no forbidden Serverless patterns in `gbx/stac/*.py`; one `@pytest.mark.integration` end-to-end test against real Planetary Computer. - -- [ ] **Step 1: Write the failing guard test** - -```python -# python/geobrix/test/stac/test_serverless_no_spark_config.py -import pathlib - -_FORBIDDEN = ["spark.conf.set", ".cache()", ".persist("] - - -def test_stac_module_has_no_serverless_forbidden_calls(): - root = pathlib.Path(__file__).resolve().parents[2] / "src/databricks/labs/gbx/stac" - offenders = [] - for py in root.glob("*.py"): - text = py.read_text() - for pat in _FORBIDDEN: - if pat in text: - offenders.append(f"{py.name}: {pat}") - assert not offenders, f"Serverless-forbidden calls in stac module: {offenders}" -``` - -- [ ] **Step 2: Run to verify it passes immediately** (the module was written clean) - -Run: `cd python/geobrix && python -m pytest test/stac/test_serverless_no_spark_config.py -v` -Expected: PASS (this guards against regressions; it should pass now) - -- [ ] **Step 3: Add the marked integration test** - -Append to `python/geobrix/test/stac/test_client.py`: - -```python -@pytest.mark.integration -def test_pc_search_and_download_one_asset(spark, tmp_path): - client = StacClient() # real Planetary Computer + sign_inplace - df = spark.createDataFrame( - [('{"type":"Point","coordinates":[-131.6,55.3]}',)], ["geojson"] - ) - assets = client.search(df, geojson_col="geojson", collections=["sentinel-2-l2a"], - datetime="2022-06-01/2022-06-05", partitions=1) - assets = assets.filter("asset_name = 'B02'").limit(1) - assert assets.count() == 1 - files = client.download(assets, str(tmp_path), asset_names=["B02"], max_tries=3) - row = files.first() - assert row["is_out_file_valid"] is True -``` - -- [ ] **Step 4: Run the full stac suite (integration excluded, matching CI)** - -Run: `cd python/geobrix && python -m pytest test/stac/ -v -m "not integration"` -Expected: PASS (all unit tests; integration deselected) - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/test/stac/ -git commit -m "test(stac): serverless-safety guard + marked PC integration test" -``` - ---- - -### Task 7: Serverless environment-v5 compatibility check - -**Files:** -- Create: `python/geobrix/test/stac/test_env_v5_compat.py` (host-runnable pin sanity) -- Create: `notebooks/tests/stac_env_v5_smoke.py` (Serverless env-v5 smoke source) - -**Why:** `[light]` taught us a dep can resolve yet fail to IMPORT on env v5 / Python 3.12 (rio-tiler 9.3.0, PEP 728 TypedDict). The `[stac]` deps must be confirmed to install + import on env v5, not just locally. - -**Interfaces:** Consumes the `[stac]` extra (T1) + `StacClient` (T5). Produces a host pin-sanity test + a Serverless env-v5 smoke + a recorded run. - -- [ ] **Step 1: Host pin-sanity test** - -```python -# python/geobrix/test/stac/test_env_v5_compat.py -"""Env-v5 (Python 3.12) compatibility guard for the [stac] extra (pin sanity only; -the live import is exercised by notebooks/tests/stac_env_v5_smoke.py on Serverless).""" -import pathlib -import re - - -def _stac_deps(): - txt = (pathlib.Path(__file__).resolve().parents[2] / "pyproject.toml").read_text() - block = re.search(r"\nstac = \[(.*?)\]", txt, re.S) - assert block, "[stac] extra not found in pyproject.toml" - return block.group(1) - - -def test_stac_extra_declares_pystac_and_pc(): - deps = _stac_deps() - assert "pystac-client" in deps and "planetary-computer" in deps - - -def test_stac_pins_support_py312(): - deps = _stac_deps() - assert re.search(r"pystac-client>=0\.7", deps) - assert re.search(r"planetary-computer>=1\.0", deps) -``` - -- [ ] **Step 2: Run host test** — `cd python/geobrix && python -m pytest test/stac/test_env_v5_compat.py -v` → PASS (2). - -- [ ] **Step 3: Serverless env-v5 smoke source** - -```python -# notebooks/tests/stac_env_v5_smoke.py -# Databricks notebook source -# Run as a one-time job on Serverless ENVIRONMENT VERSION 5 (Python 3.12): asserts the -# [stac] extra installs + imports on v5 (catches rio-tiler-9.3.0-style breakage early). -import json -res = {"py": __import__("sys").version.split()[0]} -try: - import importlib.metadata as md - import pystac_client, planetary_computer # noqa: F401 - res["pystac_client"] = md.version("pystac-client") - res["planetary_computer"] = md.version("planetary-computer") - from databricks.labs.gbx.stac import StacClient - res["client_catalog"] = StacClient().catalog # construct only, no network - res["stac_import"] = "ok" -except Exception as e: - import traceback - res["error"] = repr(e); res["tb"] = traceback.format_exc()[-1200:] -dbutils.notebook.exit(json.dumps(res)) -``` - -- [ ] **Step 4: Run smoke on env v5 + record** — stage a wheel built with `[stac]`, upload the smoke, submit a one-time job with `environments:[{spec:{environment_version:"5"}}]` whose first cell `%pip install "geobrix[light,stac] @ "`. Confirm JSON: `py`=3.12.x, both dep versions present, `stac_import=="ok"`, no `error`. Record in the report. - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/test/stac/test_env_v5_compat.py notebooks/tests/stac_env_v5_smoke.py -git commit -m "test(stac): env-v5 (Python 3.12) compatibility check + smoke" -``` - ---- - -## Self-Review - -- **Spec coverage:** `StacClient` class ✔ (T5); catalog-agnostic + signing ✔ (T2, `_open_catalog`); `[stac]` extra, not `[light]` ✔ (T1); search→items→assets typed cols ✔ (T3, T5); resilient download (raise-on-HTTP-error, read-validate, re-sign+retry/backoff, local-stage→publish) ✔ (T4); dedup `(item_id, asset_name)` ✔ (T5 `download`); repair via Delta MERGE ✔ (T5); Serverless-safe (repartition, no conf/cache) ✔ (T5, guarded T6); unit tests w/ injectable catalog + marked integration ✔ (T3–T6); drop `generate_cells` ✔ (not ported); **env-v5 compatibility check** ✔ (T7: host pin-sanity + Serverless env-v5 import smoke). -- **Placeholder scan:** none — every step has full code/commands. -- **Type consistency:** `fetch_validate_publish(href_fn, out_dir, filename, get, max_tries, sleep)` used consistently (T4 def, T5 call); `search`/`download`/`repair` signatures match the Interfaces blocks and the spec's API surface; column names (`item_id, asset_name, href, date, item_bbox, item_properties, out_file_path, out_file_sz, is_out_file_valid`) consistent across T3/T5. - -## Follow-on (separate, after this plan is green) — per the agreed sequence - -1. **Refactor the EO-series** to use `StacClient` (nb01 `search`, nb02 `download` + `repair`); remove the redundant `library.py`/`config_nb` STAC helpers (keep viz/plot). Update `config_nb` install to `geobrix[light,stac]`. -2. **Executed-notebook export + commit** (interactive Run-all → `--format JUPYTER`, screenshots ride along) for eo-series + xView re-validation. diff --git a/docs/superpowers/plans/2026-06-22-register-only-light-tier.md b/docs/superpowers/plans/2026-06-22-register-only-light-tier.md deleted file mode 100644 index 062f34de1..000000000 --- a/docs/superpowers/plans/2026-06-22-register-only-light-tier.md +++ /dev/null @@ -1,940 +0,0 @@ -# register(only=[...]) — Selective SQL Registration (Light Tiers) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add an optional `only` parameter to the lightweight `register()` functions (`pyrx`, `pygx`, `pyvx`) so a session can register a subset of a tier's SQL functions instead of the whole set. - -**Architecture:** A shared `_register.py` helper normalizes/validates requested names (case-insensitive, both `gbx_`/short forms) and runs a per-package "grouped registrar map" — an ordered list of `(guard_thunk, {sql_name: register_fn})` groups. `register(only=None)` registers everything in today's order; `register(only=[...])` registers exactly the requested names and runs a group's availability guard only when ≥1 of its functions is selected. - -**Tech Stack:** Python 3.12, PySpark (`spark.udf.register` / `spark.udtf.register`), pytest. Pure-Python — no JAR, no Docker. Tests run via `gbx:test:python --path python/geobrix/test//`. - -## Global Constraints - -- Spec: `docs/superpowers/specs/2026-06-22-register-only-light-tier-design.md`. -- Scope is **light tiers only** (`pyrx`, `pygx`, `pyvx`). Do NOT touch heavy (`rasterx`/Scala/JAR) registration — heavy `only=` is a deferred follow-up. -- `only=None` (default) MUST be behavior-identical to today: every function registered, in the same order, with all availability guards run. -- `only=[]` registers nothing (no-op, no error). -- Name handling: strip + `.lower()`, then prepend `gbx_` if absent. Accept both SQL (`gbx_rst_slope`) and short (`rst_slope`, `RST_Slope`) forms. -- Unknown name (after normalization) → raise `ValueError` listing the offending name(s) with up to 3 `difflib` close matches. Never silently skip. -- A group's availability guard (`_env.assert_*_available()`) runs only if ≥1 of its functions is selected. -- New public param signature: `register(spark: SparkSession = None, only: Optional[List[str]] = None) -> None`. -- One canonical name per function; no aliases. No emojis. Match surrounding code style (4-space indent, existing import ordering). - ---- - -### Task 1: Shared `_register.py` helper (normalize + validate + run groups) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/_register.py` -- Test: `python/geobrix/test/test_register_helper.py` - -**Interfaces:** -- Produces: - - `normalize_name(name: str) -> str` — strip+lower, prepend `gbx_` if absent (SQL functions). - - `normalize_datasource_name(name: str) -> str` — strip+lower, append `_gbx` if absent (DataSource format names: `raster` → `raster_gbx`). - - `resolve_only(only: Iterable[str], valid: Iterable[str], normalizer: Callable[[str], str] = normalize_name) -> Set[str]` — normalize all (via `normalizer`) + validate against `valid`; raise `ValueError` on unknown. - - `run_groups(groups: List[Tuple[Callable[[], None], Dict[str, Callable[[Any], None]]]], spark, only: Optional[Iterable[str]]) -> None` — register selected functions; run each group's guard only if it has a selected function. Validation uses the union of all group names (via the default `normalize_name`). - -- [ ] **Step 1: Write the failing tests** - -```python -# python/geobrix/test/test_register_helper.py -"""Unit tests for the shared selective-registration helper.""" -import pytest - -from databricks.labs.gbx import _register - - -def test_normalize_name_short_and_full_and_case(): - assert _register.normalize_name("rst_slope") == "gbx_rst_slope" - assert _register.normalize_name("gbx_rst_slope") == "gbx_rst_slope" - assert _register.normalize_name("RST_Slope") == "gbx_rst_slope" - assert _register.normalize_name("GBX_RST_Slope") == "gbx_rst_slope" - assert _register.normalize_name(" BNG_Polyfill ") == "gbx_bng_polyfill" - - -def test_normalize_datasource_name_suffix_and_case(): - assert _register.normalize_datasource_name("raster") == "raster_gbx" - assert _register.normalize_datasource_name("raster_gbx") == "raster_gbx" - assert _register.normalize_datasource_name("RASTER_GBX") == "raster_gbx" - assert _register.normalize_datasource_name(" Shapefile ") == "shapefile_gbx" - - -def test_resolve_only_with_datasource_normalizer(): - valid = {"raster_gbx", "gtiff_gbx", "shapefile_gbx"} - assert _register.resolve_only( - ["raster", "GTIFF_GBX"], valid, normalizer=_register.normalize_datasource_name - ) == {"raster_gbx", "gtiff_gbx"} - - -def test_resolve_only_returns_canonical_subset(): - valid = {"gbx_rst_slope", "gbx_rst_clip", "gbx_rst_width"} - assert _register.resolve_only(["rst_slope", "GBX_RST_Clip"], valid) == { - "gbx_rst_slope", - "gbx_rst_clip", - } - - -def test_resolve_only_empty_returns_empty_set(): - assert _register.resolve_only([], {"gbx_rst_slope"}) == set() - - -def test_resolve_only_unknown_raises_with_name_and_suggestion(): - with pytest.raises(ValueError) as ei: - _register.resolve_only(["rst_slpe"], {"gbx_rst_slope", "gbx_rst_clip"}) - msg = str(ei.value) - assert "rst_slpe" in msg - assert "gbx_rst_slope" in msg # close-match suggestion - - -def test_run_groups_only_registers_selected_and_runs_only_their_guards(): - calls = {"guardA": 0, "guardB": 0} - registered = [] - - def guardA(): - calls["guardA"] += 1 - - def guardB(): - calls["guardB"] += 1 - - groups = [ - (guardA, {"gbx_a_one": lambda s: registered.append("a_one"), - "gbx_a_two": lambda s: registered.append("a_two")}), - (guardB, {"gbx_b_one": lambda s: registered.append("b_one")}), - ] - _register.run_groups(groups, spark=None, only=["a_one"]) - assert registered == ["a_one"] - assert calls == {"guardA": 1, "guardB": 0} # guardB not run — no b fn selected - - -def test_run_groups_none_registers_all_and_runs_all_guards(): - calls = [] - registered = [] - groups = [ - (lambda: calls.append("gA"), {"gbx_a_one": lambda s: registered.append("a_one")}), - (lambda: calls.append("gB"), {"gbx_b_one": lambda s: registered.append("b_one")}), - ] - _register.run_groups(groups, spark=None, only=None) - assert registered == ["a_one", "b_one"] - assert calls == ["gA", "gB"] - - -def test_run_groups_validates_against_union_of_groups(): - groups = [ - (lambda: None, {"gbx_a_one": lambda s: None}), - (lambda: None, {"gbx_b_one": lambda s: None}), - ] - # b_one is valid (other group); typo is not - with pytest.raises(ValueError): - _register.run_groups(groups, spark=None, only=["a_one", "nope_x"]) - - -def test_run_groups_empty_only_registers_nothing_and_no_guards(): - calls = [] - registered = [] - groups = [ - (lambda: calls.append("gA"), {"gbx_a_one": lambda s: registered.append("a_one")}), - ] - _register.run_groups(groups, spark=None, only=[]) - assert registered == [] - assert calls == [] # no function selected => guard not run -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/test_register_helper.py` -Expected: FAIL — `ModuleNotFoundError: ... _register` (module doesn't exist yet). - -- [ ] **Step 3: Write the helper** - -```python -# python/geobrix/src/databricks/labs/gbx/_register.py -"""Shared helpers for selective SQL registration: register(spark, only=[...]). - -Used by the lightweight register() functions (pyrx, pygx, pyvx) so each can -register a subset of its gbx_* SQL functions. Names are case-insensitive and -accept either the short form (rst_slope) or the full SQL name (gbx_rst_slope). -""" -from __future__ import annotations - -import difflib -from typing import Any, Callable, Dict, Iterable, List, Optional, Set, Tuple - -# A group = (availability-guard thunk, {canonical_sql_name: register_fn(spark)}). -Group = Tuple[Callable[[], None], Dict[str, Callable[[Any], None]]] - - -def normalize_name(name: str) -> str: - """Normalize one requested name to its canonical gbx_ SQL name. - - Strips whitespace, lowercases (SQL names are lowercase; Scala classes are - CamelCase, so users may type RST_Slope / BNG_Polyfill), and prepends gbx_ - if absent. 'rst_slope', 'RST_Slope', 'gbx_rst_slope' -> 'gbx_rst_slope'. - """ - n = name.strip().lower() - return n if n.startswith("gbx_") else f"gbx_{n}" - - -def normalize_datasource_name(name: str) -> str: - """Normalize one DataSource format name to its canonical form. - - DataSource formats use a `_gbx` suffix (not a `gbx_` prefix). Strips + - lowercases, then appends `_gbx` if absent. 'raster', 'RASTER', - 'raster_gbx' -> 'raster_gbx'. - """ - n = name.strip().lower() - return n if n.endswith("_gbx") else f"{n}_gbx" - - -def resolve_only( - only: Iterable[str], - valid: Iterable[str], - normalizer: Callable[[str], str] = normalize_name, -) -> Set[str]: - """Normalize requested names (via `normalizer`) and validate against `valid`. - - Returns the set of canonical names to register. Raises ValueError that lists - any name not matching a registerable target (after normalization), with up - to 3 difflib close matches each. - """ - valid_set = set(valid) - requested = [(orig, normalizer(orig)) for orig in only] - unknown = [(orig, norm) for orig, norm in requested if norm not in valid_set] - if unknown: - lines = [] - for orig, norm in unknown: - matches = difflib.get_close_matches(norm, valid_set, n=3) - hint = f" -> did you mean: {', '.join(matches)}?" if matches else "" - lines.append(f" {orig!r}{hint}") - raise ValueError( - "register(only=...) got unrecognized name(s):\n" - + "\n".join(lines) - + "\nPass a registerable name (or its short form) for this tier." - ) - return {norm for _, norm in requested} - - -def run_groups(groups: List[Group], spark: Any, only: Optional[Iterable[str]]) -> None: - """Register the selected functions across `groups`. - - only=None registers every function in every group (guards all run, in order). - only=[...] registers exactly the named functions; a group's guard runs only - when >=1 of its functions is selected. Validation is against the union of all - group names. - """ - all_names: Set[str] = set() - for _guard, entries in groups: - all_names |= set(entries) - wanted = None if only is None else resolve_only(only, all_names) - for guard, entries in groups: - selected = [fn for name, fn in entries.items() if wanted is None or name in wanted] - if not selected: - continue - guard() - for fn in selected: - fn(spark) -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/test_register_helper.py` -Expected: PASS (9 tests). - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/_register.py python/geobrix/test/test_register_helper.py -git commit -m "feat(register): shared only= normalize/validate/run-groups helper" -``` - ---- - -### Task 2: pyrx `register(only=[...])` - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` (the `register` function, ~lines 50-105) -- Test: `python/geobrix/test/pyrx/test_register_only.py` - -**Interfaces:** -- Consumes: `_register.run_groups`. -- Produces: `pyrx.functions.register(spark=None, only=None)`; a module-level builder `_registrar_groups() -> List[Group]`. - -- [ ] **Step 1: Write the failing tests** - -```python -# python/geobrix/test/pyrx/test_register_only.py -"""register(only=[...]) selective registration for the pyrx tier.""" -import pytest - -from databricks.labs.gbx.pyrx import functions as prx - - -def _exists(spark, name): - return spark.catalog.functionExists(name) - - -def test_only_subset_registers_just_those(spark): - for n in ("gbx_rst_slope", "gbx_rst_clip"): - spark.sql(f"DROP TEMPORARY FUNCTION IF EXISTS {n}") - prx.register(spark, only=["rst_slope"]) - assert _exists(spark, "gbx_rst_slope") - assert not _exists(spark, "gbx_rst_clip") - - -def test_only_accepts_both_name_forms(spark): - for n in ("gbx_rst_width", "gbx_rst_height"): - spark.sql(f"DROP TEMPORARY FUNCTION IF EXISTS {n}") - prx.register(spark, only=["gbx_rst_width", "RST_Height"]) - assert _exists(spark, "gbx_rst_width") - assert _exists(spark, "gbx_rst_height") - - -def test_only_selects_udtf_and_pmtiles_agg(spark): - for n in ("gbx_rst_retile", "gbx_pmtiles_agg"): - spark.sql(f"DROP TEMPORARY FUNCTION IF EXISTS {n}") - prx.register(spark, only=["gbx_rst_retile", "gbx_pmtiles_agg"]) - assert _exists(spark, "gbx_rst_retile") - assert _exists(spark, "gbx_pmtiles_agg") - - -def test_only_unknown_name_raises(spark): - with pytest.raises(ValueError) as ei: - prx.register(spark, only=["rst_slpe"]) - assert "rst_slpe" in str(ei.value) - - -def test_only_none_registers_full_set(spark): - prx.register(spark) - for n in ("gbx_rst_width", "gbx_rst_slope", "gbx_rst_retile", "gbx_pmtiles_agg"): - assert _exists(spark, n) -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_register_only.py` -Expected: FAIL — `register()` rejects the `only` keyword (`TypeError: register() got an unexpected keyword argument 'only'`). - -- [ ] **Step 3: Refactor `register` to the grouped registrar map** - -Replace the current `register` (the `for name, udf_obj in SQL_REGISTRY.items(): ...` loop, the explicit `spark.udtf.register(...)` calls, and the trailing `register_pmtiles_agg(spark)`) with a builder + `run_groups`. The builder transcribes exactly the names registered today: every `SQL_REGISTRY` entry, the 14 UDTFs (verbatim names below), and `gbx_pmtiles_agg`. - -```python -from typing import List, Optional - -from databricks.labs.gbx import _register - - -def _registrar_groups() -> List[_register.Group]: - """One group for pyrx (rasterio guard): scalar/agg UDFs (from SQL_REGISTRY), - UDTFs, and the format-agnostic pmtiles aggregate. Insertion order matches the - pre-only register() ordering so only=None is behavior-identical.""" - entries = {} - for name, udf_obj in SQL_REGISTRY.items(): - entries[name] = lambda s, n=name, u=udf_obj: s.udf.register(n, u) - - udtfs = [ - ("gbx_rst_polygonize", _RstPolygonizeUDTF), - ("gbx_rst_h3_rastertogridavg", _RstH3RasterToGridAvgUDTF), - ("gbx_rst_h3_rastertogridcount", _RstH3RasterToGridCountUDTF), - ("gbx_rst_h3_rastertogridmax", _RstH3RasterToGridMaxUDTF), - ("gbx_rst_h3_rastertogridmin", _RstH3RasterToGridMinUDTF), - ("gbx_rst_h3_rastertogridmedian", _RstH3RasterToGridMedianUDTF), - ("gbx_rst_quadbin_rastertogridavg", _RstQuadbinRasterToGridAvgUDTF), - ("gbx_rst_quadbin_rastertogridcount", _RstQuadbinRasterToGridCountUDTF), - ("gbx_rst_quadbin_rastertogridmax", _RstQuadbinRasterToGridMaxUDTF), - ("gbx_rst_quadbin_rastertogridmin", _RstQuadbinRasterToGridMinUDTF), - ("gbx_rst_quadbin_rastertogridmedian", _RstQuadbinRasterToGridMedianUDTF), - ("gbx_rst_separatebands", _RstSeparateBandsUDTF), - ("gbx_rst_retile", _RstRetileUDTF), - ("gbx_rst_tooverlappingtiles", _RstToOverlappingTilesUDTF), - ("gbx_rst_maketiles", _RstMakeTilesUDTF), - ("gbx_rst_h3_tessellate", _RstH3TessellateUDTF), - ("gbx_rst_xyzpyramid", _RstXyzPyramidUDTF), - ] - for name, cls in udtfs: - entries[name] = lambda s, n=name, c=cls: s.udtf.register(n, c) - - def _reg_pmtiles(s): - from databricks.labs.gbx.pmtiles import register_pmtiles_agg - - register_pmtiles_agg(s) - - entries["gbx_pmtiles_agg"] = _reg_pmtiles - return [(lambda: _env.assert_rasterio_available(), entries)] - - -def register(spark: SparkSession = None, only: Optional[List[str]] = None) -> None: - """Explicitly register the pyrx functions as Spark SQL functions. - - Installs the same ``gbx_rst_*`` SQL names the heavyweight rasterx package - uses, but powered by the pyspark/rasterio implementation (no JAR). Call this - once when you want the functions from SQL. The Python Column API - (``prx.rst_width(col)``) works WITHOUT this call. - - You register the lightweight OR the heavyweight package in a given session; - they share the ``gbx_rst_*`` names, so the last registration wins. - - Args: - spark: Spark session (uses the active session if not provided). - only: Optional list of function names to register (instead of all). - Accepts SQL names (``gbx_rst_slope``) or short names (``rst_slope``), - case-insensitively. ``None`` registers everything; ``[]`` registers - nothing. An unrecognized name raises ``ValueError``. - """ - if spark is None: - spark = SparkSession.builder.getOrCreate() - _register.run_groups(_registrar_groups(), spark, only) -``` - -Note: the count check that's audited in Task 2 Step 5 relies on the UDTF list above being complete — verify it matches the `spark.udtf.register(...)` calls present in `register` before the refactor (17 UDTFs total across the two comment blocks; transcribe all, do not drop any). - -**Important — there is no separate `_fromfile_udf` registration in pyrx `register` to preserve** (the fromfile UDF is registered by the *heavy* `rasterx.register`, not here). Do not add it. - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_register_only.py` -Expected: PASS (5 tests). - -- [ ] **Step 5: Run the full pyrx registration suite (no regression)** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_sql_registration.py` -Expected: PASS — `only=None` path is unchanged, all existing SQL registration tests still pass. - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/pyrx/functions.py python/geobrix/test/pyrx/test_register_only.py -git commit -m "feat(pyrx): register(only=[...]) selective SQL registration" -``` - ---- - -### Task 3: pygx `register(only=[...])` (multi-guard: quadbin / bng / custom) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pygx/functions.py` (the `register` function, ~lines 621-680) -- Test: `python/geobrix/test/pygx/test_register_only.py` - -**Interfaces:** -- Consumes: `_register.run_groups`, `pygx._env` guards. -- Produces: `pygx.functions.register(spark=None, only=None)`; `_registrar_groups() -> List[Group]` with three guarded groups. - -- [ ] **Step 1: Write the failing tests** - -```python -# python/geobrix/test/pygx/test_register_only.py -"""register(only=[...]) selective registration for the pygx tier.""" -import pytest - -from databricks.labs.gbx.pygx import functions as pgx - - -def _exists(spark, name): - return spark.catalog.functionExists(name) - - -def test_only_subset_quadbin(spark): - for n in ("gbx_quadbin_polyfill", "gbx_bng_polyfill"): - spark.sql(f"DROP TEMPORARY FUNCTION IF EXISTS {n}") - pgx.register(spark, only=["quadbin_polyfill"]) - assert _exists(spark, "gbx_quadbin_polyfill") - assert not _exists(spark, "gbx_bng_polyfill") - - -def test_only_accepts_camelcase(spark): - spark.sql("DROP TEMPORARY FUNCTION IF EXISTS gbx_bng_polyfill") - pgx.register(spark, only=["BNG_Polyfill"]) - assert _exists(spark, "gbx_bng_polyfill") - - -def test_only_unknown_raises(spark): - with pytest.raises(ValueError) as ei: - pgx.register(spark, only=["quadbin_polifyll"]) - assert "quadbin_polifyll" in str(ei.value) - - -def test_only_does_not_trip_unselected_subgroup_guard(spark, monkeypatch): - # Selecting only a quadbin fn must NOT call the bng/custom availability guards. - from databricks.labs.gbx.pygx import _env - - def _boom(): - raise RuntimeError("guard should not be called") - - monkeypatch.setattr(_env, "assert_bng_available", _boom) - monkeypatch.setattr(_env, "assert_custom_available", _boom) - pgx.register(spark, only=["gbx_quadbin_resolution"]) # must not raise - assert _exists(spark, "gbx_quadbin_resolution") - - -def test_only_none_registers_all_subgroups(spark): - pgx.register(spark) - for n in ("gbx_quadbin_polyfill", "gbx_bng_polyfill", "gbx_custom_polyfill"): - assert _exists(spark, n) -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pygx/test_register_only.py` -Expected: FAIL — `TypeError: register() got an unexpected keyword argument 'only'`. - -- [ ] **Step 3: Refactor `register` to three guarded groups** - -Replace the body of `register` with a builder that transcribes each existing `spark.udf.register(...)` / `spark.udtf.register(...)` call into a `name: thunk` entry, **preserving the return-type argument** where present (e.g. `ArrayType(LongType())`, `ArrayType(StringType())`, `ArrayType(QUADBIN_CELL_SCHEMA)`, `ArrayType(BNG_CHIP_SCHEMA)`). The guard for the group must close over `_env` so it resolves at call time (monkeypatchable). Each group's thunk is `lambda: _env.assert__available()`. - -```python -from typing import List, Optional - -from databricks.labs.gbx import _register -from databricks.labs.gbx.pygx import _env - - -def _registrar_groups() -> List[_register.Group]: - quadbin = { - "gbx_quadbin_pointascell": lambda s: s.udf.register("gbx_quadbin_pointascell", _pointascell_udf), - "gbx_quadbin_resolution": lambda s: s.udf.register("gbx_quadbin_resolution", _resolution_udf), - "gbx_quadbin_distance": lambda s: s.udf.register("gbx_quadbin_distance", _distance_udf), - "gbx_quadbin_aswkb": lambda s: s.udf.register("gbx_quadbin_aswkb", _aswkb_udf), - "gbx_quadbin_centroid": lambda s: s.udf.register("gbx_quadbin_centroid", _centroid_udf), - "gbx_quadbin_cellunion": lambda s: s.udf.register("gbx_quadbin_cellunion", _cellunion_udf), - "gbx_quadbin_kring": lambda s: s.udf.register("gbx_quadbin_kring", _kring, ArrayType(LongType())), - "gbx_quadbin_polyfill": lambda s: s.udf.register("gbx_quadbin_polyfill", _polyfill, ArrayType(LongType())), - "gbx_quadbin_tessellate": lambda s: s.udf.register("gbx_quadbin_tessellate", _tessellate, ArrayType(QUADBIN_CELL_SCHEMA)), - "gbx_quadbin_cellunion_agg": lambda s: s.udf.register("gbx_quadbin_cellunion_agg", _cellunion_agg_udf), - } - bng = { - "gbx_bng_pointascell": lambda s: s.udf.register("gbx_bng_pointascell", _bng_pointascell_udf), - "gbx_bng_eastnorthasbng": lambda s: s.udf.register("gbx_bng_eastnorthasbng", _bng_eastnorthasbng_udf), - "gbx_bng_cellarea": lambda s: s.udf.register("gbx_bng_cellarea", _bng_cellarea_udf), - "gbx_bng_distance": lambda s: s.udf.register("gbx_bng_distance", _bng_distance_udf), - "gbx_bng_euclideandistance": lambda s: s.udf.register("gbx_bng_euclideandistance", _bng_euclideandistance_udf), - "gbx_bng_aswkb": lambda s: s.udf.register("gbx_bng_aswkb", _bng_aswkb_udf), - "gbx_bng_aswkt": lambda s: s.udf.register("gbx_bng_aswkt", _bng_aswkt_udf), - "gbx_bng_centroid": lambda s: s.udf.register("gbx_bng_centroid", _bng_centroid_udf), - "gbx_bng_cellintersection": lambda s: s.udf.register("gbx_bng_cellintersection", _bng_cellintersection_udf), - "gbx_bng_cellunion": lambda s: s.udf.register("gbx_bng_cellunion", _bng_cellunion_udf), - "gbx_bng_kring": lambda s: s.udf.register("gbx_bng_kring", _bng_kring, ArrayType(StringType())), - "gbx_bng_kloop": lambda s: s.udf.register("gbx_bng_kloop", _bng_kloop, ArrayType(StringType())), - "gbx_bng_polyfill": lambda s: s.udf.register("gbx_bng_polyfill", _bng_polyfill, ArrayType(StringType())), - "gbx_bng_geomkring": lambda s: s.udf.register("gbx_bng_geomkring", _bng_geomkring, ArrayType(StringType())), - "gbx_bng_geomkloop": lambda s: s.udf.register("gbx_bng_geomkloop", _bng_geomkloop, ArrayType(StringType())), - "gbx_bng_tessellate": lambda s: s.udf.register("gbx_bng_tessellate", _bng_tessellate, ArrayType(BNG_CHIP_SCHEMA)), - "gbx_bng_kringexplode": lambda s: s.udtf.register("gbx_bng_kringexplode", _BngKRingExplode), - "gbx_bng_kloopexplode": lambda s: s.udtf.register("gbx_bng_kloopexplode", _BngKLoopExplode), - "gbx_bng_geomkringexplode": lambda s: s.udtf.register("gbx_bng_geomkringexplode", _BngGeomKRingExplode), - "gbx_bng_geomkloopexplode": lambda s: s.udtf.register("gbx_bng_geomkloopexplode", _BngGeomKLoopExplode), - "gbx_bng_tessellateexplode": lambda s: s.udtf.register("gbx_bng_tessellateexplode", _BngTessellateExplode), - "gbx_bng_cellunion_agg": lambda s: s.udf.register("gbx_bng_cellunion_agg", _bng_cellunion_agg_udf), - "gbx_bng_cellintersection_agg": lambda s: s.udf.register("gbx_bng_cellintersection_agg", _bng_cellintersection_agg_udf), - } - custom = { - "gbx_custom_grid": lambda s: s.udf.register("gbx_custom_grid", _custom_grid_udf), - "gbx_custom_pointascell": lambda s: s.udf.register("gbx_custom_pointascell", _custom_pointascell_udf), - "gbx_custom_cellaswkb": lambda s: s.udf.register("gbx_custom_cellaswkb", _custom_cellaswkb_udf), - "gbx_custom_cellaswkt": lambda s: s.udf.register("gbx_custom_cellaswkt", _custom_cellaswkt_udf), - "gbx_custom_centroid": lambda s: s.udf.register("gbx_custom_centroid", _custom_centroid_udf), - "gbx_custom_polyfill": lambda s: s.udf.register("gbx_custom_polyfill", _custom_polyfill, ArrayType(LongType())), - "gbx_custom_kring": lambda s: s.udf.register("gbx_custom_kring", _custom_kring, ArrayType(LongType())), - } - return [ - (lambda: _env.assert_quadbin_available(), quadbin), - (lambda: _env.assert_bng_available(), bng), - (lambda: _env.assert_custom_available(), custom), - ] - - -def register(spark: SparkSession = None, only: Optional[List[str]] = None) -> None: - """Register the pygx grid SQL functions (Serverless-safe: udf/udtf only). - - Args: - spark: Spark session (uses the active session if not provided). - only: Optional list of function names to register (instead of all). - Accepts SQL names (``gbx_bng_polyfill``) or short names - (``bng_polyfill``), case-insensitively. ``None`` registers everything; - ``[]`` registers nothing. An unrecognized name raises ``ValueError``. - A sub-module's availability guard runs only when >=1 of its functions - is selected. - """ - if spark is None: - spark = SparkSession.builder.getOrCreate() - _register.run_groups(_registrar_groups(), spark, only) -``` - -Verify every name/return-type matches the pre-refactor `register` body exactly (10 quadbin + 23 bng + 7 custom = 40 entries). Keep the existing `from pyspark.sql.types import ArrayType, LongType, StringType` (and `QUADBIN_CELL_SCHEMA`, `BNG_CHIP_SCHEMA`) imports the module already has. - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pygx/test_register_only.py` -Expected: PASS (5 tests). - -- [ ] **Step 5: Run the existing pygx UDF suites (no regression)** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pygx/` -Expected: PASS — existing quadbin/bng/custom tests unaffected by the `only=None` path. - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/pygx/functions.py python/geobrix/test/pygx/test_register_only.py -git commit -m "feat(pygx): register(only=[...]) selective SQL registration" -``` - ---- - -### Task 4: pyvx `register(only=[...])` (guards: mvt / legacy / tin + pmtiles) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py` (the `register` function, ~lines 267-283) -- Test: `python/geobrix/test/pyvx/test_register_only.py` - -**Interfaces:** -- Consumes: `_register.run_groups`, `pyvx._env` guards. -- Produces: `pyvx.functions.register(spark=None, only=None)`; `_registrar_groups() -> List[Group]`. - -- [ ] **Step 1: Write the failing tests** - -```python -# python/geobrix/test/pyvx/test_register_only.py -"""register(only=[...]) selective registration for the pyvx tier.""" -import pytest - -from databricks.labs.gbx.pyvx import functions as pvx - - -def _exists(spark, name): - return spark.catalog.functionExists(name) - - -def test_only_subset_mvt(spark): - for n in ("gbx_st_asmvt", "gbx_st_legacyaswkb"): - spark.sql(f"DROP TEMPORARY FUNCTION IF EXISTS {n}") - pvx.register(spark, only=["st_asmvt"]) - assert _exists(spark, "gbx_st_asmvt") - assert not _exists(spark, "gbx_st_legacyaswkb") - - -def test_only_selects_udtf_and_pmtiles(spark): - for n in ("gbx_st_asmvt_pyramid", "gbx_pmtiles_agg"): - spark.sql(f"DROP TEMPORARY FUNCTION IF EXISTS {n}") - pvx.register(spark, only=["gbx_st_asmvt_pyramid", "gbx_pmtiles_agg"]) - assert _exists(spark, "gbx_st_asmvt_pyramid") - assert _exists(spark, "gbx_pmtiles_agg") - - -def test_only_does_not_trip_unselected_guard(spark, monkeypatch): - from databricks.labs.gbx.pyvx import _env - - def _boom(): - raise RuntimeError("guard should not be called") - - monkeypatch.setattr(_env, "assert_tin_available", _boom) - monkeypatch.setattr(_env, "assert_legacy_available", _boom) - pvx.register(spark, only=["gbx_st_asmvt"]) # must not raise - assert _exists(spark, "gbx_st_asmvt") - - -def test_only_unknown_raises(spark): - with pytest.raises(ValueError) as ei: - pvx.register(spark, only=["st_asmtv"]) - assert "st_asmtv" in str(ei.value) - - -def test_only_none_registers_all(spark): - pvx.register(spark) - for n in ("gbx_st_asmvt", "gbx_st_legacyaswkb", "gbx_st_triangulate", "gbx_pmtiles_agg"): - assert _exists(spark, n) -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/test_register_only.py` -Expected: FAIL — `TypeError: register() got an unexpected keyword argument 'only'`. - -- [ ] **Step 3: Refactor `register` to guarded groups** - -Preserve the `BinaryType()` return-type arg on `gbx_st_legacyaswkb`. The pmtiles group has no `_env` guard, so use a no-op guard (`lambda: None`). - -```python -from typing import List, Optional - -from databricks.labs.gbx import _register -from databricks.labs.gbx.pyvx import _env - - -def _registrar_groups() -> List[_register.Group]: - mvt = { - "gbx_st_asmvt": lambda s: s.udf.register("gbx_st_asmvt", _asmvt_udf), - "gbx_st_asmvt_pyramid": lambda s: s.udtf.register("gbx_st_asmvt_pyramid", _AsMvtPyramidUDTF), - } - legacy = { - "gbx_st_legacyaswkb": lambda s: s.udf.register("gbx_st_legacyaswkb", _legacyaswkb_impl, BinaryType()), - } - tin = { - "gbx_st_triangulate": lambda s: s.udtf.register("gbx_st_triangulate", _TriangulateUDTF), - "gbx_st_interpolateelevationbbox": lambda s: s.udtf.register("gbx_st_interpolateelevationbbox", _InterpElevBBoxUDTF), - "gbx_st_interpolateelevationgeom": lambda s: s.udtf.register("gbx_st_interpolateelevationgeom", _InterpElevGeomUDTF), - } - - def _reg_pmtiles(s): - from databricks.labs.gbx.pmtiles import register_pmtiles_agg - - register_pmtiles_agg(s) - - pmtiles = {"gbx_pmtiles_agg": _reg_pmtiles} - return [ - (lambda: _env.assert_mvt_available(), mvt), - (lambda: _env.assert_legacy_available(), legacy), - (lambda: _env.assert_tin_available(), tin), - (lambda: None, pmtiles), - ] - - -def register(spark: SparkSession = None, only: Optional[List[str]] = None) -> None: - """Register the pyvx VectorX SQL functions (Serverless-safe: udf/udtf only). - - Args: - spark: Spark session (uses the active session if not provided). - only: Optional list of function names to register (instead of all). - Accepts SQL names (``gbx_st_asmvt``) or short names (``st_asmvt``), - case-insensitively. ``None`` registers everything; ``[]`` registers - nothing. An unrecognized name raises ``ValueError``. A sub-module's - availability guard runs only when >=1 of its functions is selected. - """ - if spark is None: - spark = SparkSession.builder.getOrCreate() - _register.run_groups(_registrar_groups(), spark, only) -``` - -Keep the module's existing `from pyspark.sql.types import BinaryType` import. - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/test_register_only.py` -Expected: PASS (5 tests). - -- [ ] **Step 5: Run existing pyvx suites (no regression)** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyvx/` -Expected: PASS. - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/pyvx/functions.py python/geobrix/test/pyvx/test_register_only.py -git commit -m "feat(pyvx): register(only=[...]) selective SQL registration" -``` - ---- - -### Task 5: `ds.register.register(spark, only=[...])` — readers/writers by format name - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/register.py` (the `register` function) -- Test: `python/geobrix/test/ds/test_register_only.py` - -**Interfaces:** -- Consumes: `_register.resolve_only`, `_register.normalize_datasource_name`. -- Produces: `ds.register.register(spark=None, only=None)`. - -The 9 light DataSources select by **format name** (`name()` classmethod): `raster_gbx`, `gtiff_gbx`, `pmtiles_gbx`, `vector_gbx`, `shapefile_gbx`, `geojson_gbx`, `geojsonl_gbx`, `gpkg_gbx`, `file_gdb_gbx`. - -- [ ] **Step 1: Write the failing tests** - -```python -# python/geobrix/test/ds/test_register_only.py -"""register(only=[...]) selective registration for the light DataSources.""" -import pytest - -from databricks.labs.gbx.ds import register as ds_register - - -def _format_ok(spark, fmt): - """A format is registered if .format(fmt) builds a reader without an - 'unsupported data source' error. Loading an empty path raises a DIFFERENT - (path/IO) error, so we treat only the unsupported-format error as 'absent'.""" - try: - spark.read.format(fmt).load("/tmp/__nonexistent_gbx_probe__") - return True - except Exception as e: # noqa: BLE001 - msg = str(e).lower() - if "unable to find" in msg or "data source" in msg and "not" in msg: - return False - return True # some other error => the format WAS resolved - - -def test_only_subset_registers_just_those(spark): - ds_register.register(spark, only=["raster_gbx", "gtiff_gbx"]) - assert _format_ok(spark, "raster_gbx") - assert _format_ok(spark, "gtiff_gbx") - - -def test_only_accepts_bare_name_without_suffix(spark): - ds_register.register(spark, only=["raster"]) # -> raster_gbx - assert _format_ok(spark, "raster_gbx") - - -def test_only_unknown_format_raises(spark): - with pytest.raises(ValueError) as ei: - ds_register.register(spark, only=["raster_gpx"]) - assert "raster_gpx" in str(ei.value) - - -def test_only_none_registers_all(spark): - ds_register.register(spark) - for fmt in ("raster_gbx", "gtiff_gbx", "shapefile_gbx", "geojson_gbx"): - assert _format_ok(spark, fmt) -``` - -Note: a `spark` fixture must exist for `python/geobrix/test/ds/`. If `python/geobrix/test/ds/conftest.py` has no `spark` fixture, add a module-scoped one mirroring `python/geobrix/test/pyrx/conftest.py` (a plain `SparkSession.builder.master("local[2]")` session, no JARs). - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_register_only.py` -Expected: FAIL — `TypeError: register() got an unexpected keyword argument 'only'`. - -- [ ] **Step 3: Add `only` to `ds.register.register`** - -```python -# python/geobrix/src/databricks/labs/gbx/ds/register.py (replace the register function) -from typing import List, Optional - -from databricks.labs.gbx import _register - - -def register(spark: Optional[SparkSession] = None, only: Optional[List[str]] = None) -> None: - """Register the light DataSources (raster_gbx, gtiff_gbx, pmtiles_gbx, and the - vector readers/writers). Uses the active session if not given. - - Args: - spark: Spark session (active session if not provided). - only: Optional list of format names to register (instead of all 9). - Accepts the format name with or without the ``_gbx`` suffix - (``raster`` or ``raster_gbx``), case-insensitively. ``None`` registers - everything; ``[]`` registers nothing. An unrecognized format raises - ``ValueError``. - """ - if spark is None: - spark = SparkSession.builder.getOrCreate() - by_name = {src.name(): src for src in _SOURCES} - if only is None: - selected = list(_SOURCES) - else: - wanted = _register.resolve_only( - only, by_name.keys(), normalizer=_register.normalize_datasource_name - ) - selected = [by_name[n] for n in wanted] - for source in selected: - spark.dataSource.register(source) -``` - -Keep `_SOURCES`, the existing imports, and `_try_register_on_import()` unchanged. `_SOURCES` order is preserved for `only=None` (iterate `_SOURCES`, not the resolved set). - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_register_only.py` -Expected: PASS (4 tests). - -- [ ] **Step 5: Run existing ds suites (no regression)** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/` -Expected: PASS — `only=None` path unchanged. - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/register.py python/geobrix/test/ds/test_register_only.py -git commit -m "feat(ds): register(only=[...]) selective reader/writer registration" -``` - ---- - -### Task 6: Docs — "Registering a subset" (execution-tiers + readers/writers overview) - -**Files:** -- Modify: `docs/docs/api/execution-tiers.mdx` (add a subsection after "The one-line swap", ~line 25) -- Modify: `docs/docs/readers/overview.mdx` (extend the "Register first" note, ~line 26-35) -- Modify: `docs/docs/writers/overview.mdx` (extend the "Register first" note, ~line 27-36) - -**Interfaces:** none (docs only). - -- [ ] **Step 1: Add the subsection** - -Insert after the `:::warning Heavyweight needs more than the wheel` admonition (before `## Tradeoffs`): - -````markdown -## Registering a subset (`only=`) - -`register()` installs every `gbx_*` SQL name for the tier. To register just the functions a session uses, pass `only=` (lightweight tiers — `pyrx`, `pygx`, `pyvx`): - -```python -from databricks.labs.gbx.pyrx import functions as rx - -rx.register(spark, only=["rst_slope", "rst_clip"]) # just these two -rx.register(spark) # all (default) -``` - -Names are case-insensitive and accept either the SQL name (`gbx_rst_slope`) or the short form (`rst_slope`). An unrecognized name raises `ValueError` (typo guard). `only=[]` registers nothing. - -**Readers and writers** register through a separate entry point and take `only=` too — selected by **format name** (with or without the `_gbx` suffix): - -```python -from databricks.labs.gbx.ds import register as ds_register - -ds_register.register(spark, only=["raster_gbx", "gtiff_gbx"]) # just these formats -ds_register.register(spark, only=["shapefile"]) # 'shapefile' -> 'shapefile_gbx' -ds_register.register(spark) # all readers/writers (default) -``` - -**Mixing tiers per function.** Because both tiers share the `gbx_*` names (last registration wins), you can register the heavyweight set and then override individual functions with the lightweight implementation: - -```python -from databricks.labs.gbx.rasterx import functions as heavy -from databricks.labs.gbx.pyrx import functions as light - -heavy.register(spark) # all heavy gbx_rst_* -light.register(spark, only=["rst_slope"]) # gbx_rst_slope now lightweight -``` - -The reverse — re-registering a few **heavy** functions over a lightweight session — is not yet available; `only=` is currently a lightweight-tier feature (heavy registers its full set). Mixing works because both tiers use the same tile struct and GTiff payload, so a tile produced by one tier flows into a function from the other. -```` - -- [ ] **Step 2: Extend the "Register first" note in readers/writers overview** - -In BOTH `docs/docs/readers/overview.mdx` and `docs/docs/writers/overview.mdx`, the `:::note Register first` admonition ends with: - -```python -from databricks.labs.gbx.ds.register import register -register(spark) -``` - -Add a sentence immediately after that code block (inside the `:::note`), in each file: - -```markdown -To register only the formats this session uses, pass `only=` (by format name, with or without the `_gbx` suffix): - -```python -register(spark, only=["raster_gbx", "gtiff_gbx"]) -``` -``` - -(In `writers/overview.mdx` use writer-appropriate example formats, e.g. `register(spark, only=["raster_gbx", "geojson_gbx"])`.) An unrecognized format raises `ValueError`. - -- [ ] **Step 3: Verify the docs edits landed** - -Run: `grep -n "Registering a subset" docs/docs/api/execution-tiers.mdx && grep -n "only=" docs/docs/readers/overview.mdx docs/docs/writers/overview.mdx` -Expected: the new heading in execution-tiers and the `only=` note in both overview pages. (No doc-test executes this MDX prose; the code blocks are illustrative, consistent with each page's existing register block.) - -- [ ] **Step 4: Commit** - -```bash -git add docs/docs/api/execution-tiers.mdx docs/docs/readers/overview.mdx docs/docs/writers/overview.mdx -git commit -m "docs: document register(only=[...]) for functions and readers/writers" -``` - ---- - -## Notes for the implementer - -- The `spark` fixture in each light package's `conftest.py` is **module-scoped** and shared, so SQL temp functions accumulate within a test module. The `only=` tests therefore `DROP TEMPORARY FUNCTION IF EXISTS` the names they assert on (both present- and absent-checks) so they're order-independent. This `DROP TEMPORARY FUNCTION` pattern is already used in `python/geobrix/test/pmtiles_light/test_agg_light_udf.py`. -- If `spark.catalog.functionExists("gbx_...")` does not report a temp UDTF in this Spark build, fall back to `"" in [f.name for f in spark.catalog.listFunctions()]` — but try `functionExists` first. -- Late-binding in `lambda`: the pyrx scalar/UDTF closures bind `name`/`udf`/`cls` as default args (`lambda s, n=name, u=udf_obj: ...`) to avoid the classic loop-variable capture bug. The pygx/pyvx maps hard-code each name literal in the lambda body, so they don't need default-arg binding. Keep it that way. -- Run `gbx:lint:python --check` (isort/black/flake8) before the final commit of each task; reformat in-container if host black differs. diff --git a/docs/superpowers/plans/2026-06-23-gbx-viz-escape-hatches.md b/docs/superpowers/plans/2026-06-23-gbx-viz-escape-hatches.md deleted file mode 100644 index 8ee972d95..000000000 --- a/docs/superpowers/plans/2026-06-23-gbx-viz-escape-hatches.md +++ /dev/null @@ -1,894 +0,0 @@ -# gbx.viz + pyrx escape-hatches Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Promote the EO-series notebook helpers into the package as a tier-agnostic `databricks.labs.gbx.viz` module (new `[viz]` extra) plus two Python-only pyrx escape-hatches (`tile_to_numpy`, `rst_apply`). - -**Architecture:** `gbx.viz` is tier-agnostic (operates on raster bytes / Spark DataFrames) and lives at the package top level; its heavy deps (matplotlib, geopandas) are lazy-imported behind a `viz/_env.py` guard mirroring `pyrx/_env.py`. The escape-hatches operate on the pyrx tile struct via `pyrx/_serde.open_tile` and live in `pyrx/core/escape.py`, re-exported on `pyrx.functions`. They are Python-API-only (never SQL-registered). - -**Tech Stack:** Python 3.12, PySpark 4.0, rasterio (light), matplotlib + geopandas + folium + mapclassify (new `[viz]`), h3 (light), pytest. - -## Global Constraints - -- Target branch `beta/0.4.0`. Spec: `docs/superpowers/specs/2026-06-22-gbx-viz-escape-hatches-design.md`. -- Serverless-safe: NO `spark.conf.set`, `.cache()/.persist()/.localCheckpoint()`, `_jvm`/`sparkContext`/`.rdd` anywhere in package code. -- `[viz]` deps are pinned+hash-locked in `requirements-pyrx-ci.{in,txt}` (regen with `--generate-hashes`); versions match `requirements-dev-container.in` where present. -- Package code imports only matplotlib + geopandas (+ already-light shapely/h3) — NOT folium/mapclassify (those are user-side `.explore()` deps shipped in the extra). The dep guard checks only what our code imports. -- Escape-hatches are Python-API-only: NOT added to the SQL registry or `registered_functions.txt`; binding-parity / `function-info.json` stay unchanged. -- Heavy deps lazy-imported inside functions, guarded by `assert_viz_available()` raising `pip install 'geobrix[viz]'`. Matplotlib forced to `Agg` when no display. -- Tests use real assertions on real synthesized rasters (reuse `make_geotiff_bytes` from `test/pyrx/conftest.py`); matplotlib `Agg`; no pixel comparison. No mocking of geopandas/rasterio. -- `gbx:lint:python --check` (isort/black/flake8) must pass; run black/isort IN the dev container (host black may differ). -- No internal/wave vocabulary in any `docs/docs/` page (QC `internals-leak` gate). -- All test runs happen in the `geobrix-dev` Docker container (`bash scripts/commands/gbx-docker-start.sh`; run via `scripts/commands/gbx-test-python.sh --path

` or `docker exec geobrix-dev bash -lc '...pytest...'`). - ---- - -## File Structure - -- Create `python/geobrix/src/databricks/labs/gbx/viz/__init__.py` — public exports. -- Create `.../viz/_env.py` — `assert_viz_available()` lazy-dep guard. -- Create `.../viz/_raster.py` — `plot_raster`, `plot_file` + private render pipeline. -- Create `.../viz/_vector.py` — `as_gdf`, `cells_as_gdf`. -- Create `.../pyrx/core/escape.py` — `tile_to_numpy`, `rst_apply`. -- Modify `.../pyrx/functions.py` — re-export `tile_to_numpy`, `rst_apply`. -- Modify `python/geobrix/pyproject.toml` — add `[viz]` extra. -- Modify `python/geobrix/requirements-pyrx-ci.in` + regenerate `.txt`. -- Modify `python/geobrix/test/conftest.py` — add `"viz"` to `_LIGHT_TEST_DIRS`. -- Modify `.github/actions/pyrx_build/action.yml` — add `test/viz` to the light dir list. -- Create `python/geobrix/test/viz/__init__.py`, `.../test/viz/test_raster.py`, `.../test/viz/test_vector.py`. -- Create `python/geobrix/test/pyrx/test_escape.py`. -- Create `docs/docs/api/viz.mdx`; modify `docs/docs/api/raster-functions.mdx`. - ---- - -### Task 1: `[viz]` extra + lightweight CI lock + viz package skeleton + dep guard - -**Files:** -- Modify: `python/geobrix/pyproject.toml` (after the `stac = [...]` block, ~line 129) -- Modify: `python/geobrix/requirements-pyrx-ci.in` (regen `.txt`) -- Create: `python/geobrix/src/databricks/labs/gbx/viz/__init__.py` -- Create: `python/geobrix/src/databricks/labs/gbx/viz/_env.py` -- Test: `python/geobrix/test/viz/__init__.py`, `python/geobrix/test/viz/test_env.py` - -**Interfaces:** -- Produces: `databricks.labs.gbx.viz._env.assert_viz_available() -> None` (raises `ImportError` with `[viz]` guidance if matplotlib or geopandas missing). `viz/__init__.py` exports `plot_raster, plot_file, as_gdf, cells_as_gdf` (added in later tasks; in this task `__init__.py` is created empty-but-importable). - -- [ ] **Step 1: Add the `[viz]` extra to pyproject.toml** - -Insert after the `stac = [...]` block: - -```toml -# Visualization helpers (gbx.viz): matplotlib raster rendering + geopandas/folium -# map adapters. Optional so [light] users who don't visualize don't pull the GUI -# stack. Package code imports only matplotlib + geopandas; folium + mapclassify are -# for the user's GeoDataFrame.explore() maps. -viz = [ - "matplotlib>=3.7,<4", - "geopandas>=1.0,<2", - "folium>=0.16,<1", - "mapclassify>=2.6,<3", -] -``` - -- [ ] **Step 2: Create the viz package skeleton** - -`viz/__init__.py`: -```python -"""gbx.viz — tier-agnostic visualization helpers (requires the [viz] extra). - -Raster rendering (plot_raster / plot_file) and Spark DataFrame -> GeoDataFrame -adapters (as_gdf / cells_as_gdf) for interactive maps. Install with -``pip install 'geobrix[viz]'``. -""" -``` -(Exports are added by later tasks; keep importable now.) - -`viz/_env.py`: -```python -"""Lazy-dependency guard for gbx.viz (the [viz] extra). - -Visualization deps are heavy and optional. Package code imports them only inside -functions, after calling assert_viz_available(), which raises a clear install -hint when they are absent — mirroring pyrx/_env.py::assert_rasterio_available(). -""" - - -def assert_viz_available() -> None: - """Raise ImportError with [viz] guidance if matplotlib or geopandas is missing. - - Only the deps gbx.viz code actually imports are checked (matplotlib for raster - rendering, geopandas for the GeoDataFrame adapters). folium / mapclassify are - user-side GeoDataFrame.explore() deps and are not imported by this package. - """ - missing = [] - for mod in ("matplotlib", "geopandas"): - try: - __import__(mod) - except ImportError: - missing.append(mod) - if missing: - raise ImportError( - "gbx.viz requires the [viz] extra (missing: " - + ", ".join(missing) - + "). Install with: pip install 'geobrix[viz]'" - ) -``` - -- [ ] **Step 3: Write the failing test** - -`test/viz/__init__.py`: empty file. - -`test/viz/test_env.py`: -```python -import builtins - -import pytest - -from databricks.labs.gbx.viz._env import assert_viz_available - - -def test_assert_viz_available_passes_when_present(): - # matplotlib + geopandas are installed in the light/dev/CI env. - assert assert_viz_available() is None - - -def test_assert_viz_available_raises_actionable_error(monkeypatch): - real_import = builtins.__import__ - - def fake(name, *a, **k): - if name == "geopandas": - raise ImportError("No module named 'geopandas'") - return real_import(name, *a, **k) - - monkeypatch.setattr(builtins, "__import__", fake) - with pytest.raises(ImportError) as ei: - assert_viz_available() - msg = str(ei.value) - assert "geopandas" in msg and "geobrix[viz]" in msg -``` - -- [ ] **Step 4: Add `[viz]` deps to the light CI lock** - -Edit `requirements-pyrx-ci.in` — insert before the `# --- test runner ---` block: -``` -# --- visualization ([viz] extra): gbx.viz raster rendering + GeoDataFrame map -# adapters. Package code imports matplotlib + geopandas; folium + mapclassify are -# for user GeoDataFrame.explore() maps. Matches requirements-dev-container.in. --- -matplotlib==3.10.9 -geopandas==1.1.1 -folium==0.21.0 -mapclassify==2.11.0 -``` -(If `requirements-dev-container.in` pins different versions, match those exact versions instead — check with `grep -iE "^matplotlib==|^geopandas==|^folium==|^mapclassify==" requirements-dev-container.txt`.) - -- [ ] **Step 5: Regenerate the hash-pinned lock (in the container, via the proxy)** - -Run: -``` -docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && UV_INDEX_URL=https://pypi-proxy.dev.databricks.com/simple uv pip compile --generate-hashes --python-version 3.12 -o requirements-pyrx-ci.txt requirements-pyrx-ci.in' -``` -Expected: `requirements-pyrx-ci.txt` gains matplotlib/geopandas/folium/mapclassify + transitives, each with `--hash=sha256:` lines; no pre-existing top-level pin changes version. - -- [ ] **Step 6: Run the tests** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/viz/test_env.py` -Expected: 2 passed. (If matplotlib/geopandas aren't yet in the dev container, `pip install matplotlib geopandas folium mapclassify` in the container first; the lock guarantees CI has them.) - -- [ ] **Step 7: Lint + commit** - -```bash -docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && black src/databricks/labs/gbx/viz test/viz && isort src/databricks/labs/gbx/viz test/viz' -git add python/geobrix/pyproject.toml python/geobrix/requirements-pyrx-ci.in python/geobrix/requirements-pyrx-ci.txt python/geobrix/src/databricks/labs/gbx/viz/ python/geobrix/test/viz/ -git commit -m "feat(viz): [viz] extra + package skeleton + dep guard" -``` - ---- - -### Task 2: `viz._raster` percentile-stretch + decimation pipeline (pure functions) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/viz/_raster.py` -- Test: `python/geobrix/test/viz/test_raster.py` - -**Interfaces:** -- Produces: `_needs_percentile_stretch(data) -> bool`; `_percentile_stretch(data, lo_pct=2, hi_pct=98) -> np.ndarray`; `_decimated_read(src, max_pixels) -> (data, transform, scale)`. Consumed by `plot_raster`/`plot_file` (Task 3). - -- [ ] **Step 1: Write the failing tests** - -`test/viz/test_raster.py`: -```python -import numpy as np -import pytest - -from databricks.labs.gbx.viz import _raster - - -def test_needs_stretch_true_for_uint16_over_255(): - data = np.array([[0, 300], [1000, 65535]], dtype="uint16") - assert _raster._needs_percentile_stretch(data) is True - - -def test_needs_stretch_false_for_float_and_small_int(): - assert _raster._needs_percentile_stretch(np.array([[0.1, 0.9]], dtype="float32")) is False - assert _raster._needs_percentile_stretch(np.array([[0, 200]], dtype="uint8")) is False - - -def test_percentile_stretch_scales_to_unit_range_ignoring_mask(): - band = np.arange(100, dtype="uint16").reshape(1, 10, 10) * 10 # 0..9900 - masked = np.ma.MaskedArray(band, mask=np.zeros_like(band, dtype=bool)) - masked.mask[0, 0, 0] = True # exclude an outlier-free pixel - out = _raster._percentile_stretch(masked) - assert out.dtype == np.float32 - assert float(out.min()) >= 0.0 and float(out.max()) <= 1.0 - assert isinstance(out, np.ma.MaskedArray) - assert out.mask[0, 0, 0] # mask preserved -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/viz/test_raster.py` -Expected: FAIL (module/functions not defined). - -- [ ] **Step 3: Implement the pipeline (ported verbatim from library.py)** - -`viz/_raster.py`: -```python -"""Raster rendering pipeline for gbx.viz (decimation + percentile stretch). - -Ported from notebooks/examples/eo-series/library.py. matplotlib/rasterio are -lazy-imported inside the public plotters (Task 3); the numeric helpers here use -only numpy and the rasterio dataset passed in. -""" - -import numpy as np - - -def _decimated_read(src, max_pixels): - """Read `src` (rasterio DatasetReader) decimated so max(width,height)<=max_pixels. - - Returns (data, transform, scale). masked=True so nodata is honored downstream. - """ - import rasterio - - scale = max(src.width, src.height) / max_pixels - if scale > 1: - out_shape = (src.count, int(src.height // scale), int(src.width // scale)) - data = src.read( - out_shape=out_shape, - resampling=rasterio.enums.Resampling.bilinear, - masked=True, - ) - transform = src.transform * src.transform.scale( - src.width / data.shape[-1], - src.height / data.shape[-2], - ) - else: - data = src.read(masked=True) - transform = src.transform - return data, transform, scale - - -def _needs_percentile_stretch(data): - """True when data is integer-typed with a max above matplotlib's RGB int 255.""" - if not np.issubdtype(data.dtype, np.integer): - return False - mx = np.ma.max(data) if isinstance(data, np.ma.MaskedArray) else data.max() - if mx is np.ma.masked: - return False - return int(mx) > 255 - - -def _percentile_stretch(data, lo_pct=2, hi_pct=98): - """Per-band 2-98th percentile stretch to [0,1] float32; masked pixels excluded.""" - if data.ndim == 2: - data = data[np.newaxis, ...] - is_masked = isinstance(data, np.ma.MaskedArray) - out = np.empty(data.shape, dtype=np.float32) - for b in range(data.shape[0]): - band = data[b] - valid = band.compressed() if is_masked else np.asarray(band).ravel() - if valid.size == 0: - out[b] = 0.0 - continue - lo, hi = np.percentile(valid, (lo_pct, hi_pct)) - rng = max(float(hi - lo), 1e-9) - out[b] = np.clip((np.asarray(band, dtype=np.float32) - lo) / rng, 0.0, 1.0) - return np.ma.MaskedArray(out, mask=data.mask) if is_masked else out -``` - -- [ ] **Step 4: Run to verify pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/viz/test_raster.py` -Expected: 3 passed. - -- [ ] **Step 5: Lint + commit** - -```bash -docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && black src/databricks/labs/gbx/viz/_raster.py test/viz/test_raster.py && isort src/databricks/labs/gbx/viz/_raster.py test/viz/test_raster.py' -git add python/geobrix/src/databricks/labs/gbx/viz/_raster.py python/geobrix/test/viz/test_raster.py -git commit -m "feat(viz): raster decimation + percentile-stretch pipeline" -``` - ---- - -### Task 3: `viz._raster` plot_raster / plot_file + `_render` - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/viz/_raster.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/viz/__init__.py` -- Test: `python/geobrix/test/viz/test_raster.py` - -**Interfaces:** -- Consumes: `_decimated_read`, `_needs_percentile_stretch`, `_percentile_stretch` (Task 2); `assert_viz_available` (Task 1); `make_geotiff_bytes` (test fixture, `test/pyrx/conftest.py`). -- Produces: `plot_raster(raster_bytes, *, fig_w=10, fig_h=10, max_pixels=2000) -> None`; `plot_file(path, *, fig_w=10, fig_h=10, max_pixels=2000) -> None`. - -- [ ] **Step 1: Write the failing tests** - -Append to `test/viz/test_raster.py`: -```python -import matplotlib - -matplotlib.use("Agg") # headless: no display needed -import matplotlib.pyplot as plt # noqa: E402 - -from databricks.labs.gbx.viz import plot_file, plot_raster # noqa: E402 -from test.pyrx.conftest import make_geotiff_bytes # noqa: E402 - - -def test_plot_raster_produces_a_figure(): - plt.close("all") - plot_raster(make_geotiff_bytes(width=8, height=8, count=1)) - assert len(plt.get_fignums()) == 1 - plt.close("all") - - -def test_plot_file_produces_a_figure(tmp_path): - p = tmp_path / "t.tif" - p.write_bytes(make_geotiff_bytes(width=8, height=8, count=3)) - plt.close("all") - plot_file(str(p)) - assert len(plt.get_fignums()) == 1 - plt.close("all") -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/viz/test_raster.py` -Expected: FAIL (`plot_raster`/`plot_file` not importable from `gbx.viz`). - -- [ ] **Step 3: Implement `_render` + the public plotters** - -Append to `viz/_raster.py`: -```python -def _render(data, transform, *, title, fig_w, fig_h, scale): - """Stretch when needed, then plot via rasterio.plot.show (Agg-safe).""" - import matplotlib - - if matplotlib.get_backend().lower() != "agg": - try: - matplotlib.get_current_fig_manager() - except Exception: - matplotlib.use("Agg") - from matplotlib import pyplot - from rasterio.plot import show - - if _needs_percentile_stretch(data): - data = _percentile_stretch(data) - fig, ax = pyplot.subplots(1, figsize=(fig_w, fig_h)) - if data.shape[0] == 1: - show(data, ax=ax, transform=transform, cmap="viridis") - else: - show(data, ax=ax, transform=transform) - full_title = f"{title} (scale 1/{round(scale, 1)}x)" if scale > 1 else title - ax.set_title(full_title) - pyplot.show() - - -def plot_raster(raster_bytes, *, fig_w=10, fig_h=10, max_pixels=2000): - """Render a raster from in-memory bytes (e.g. a tile's `raster` field). - - Auto-decimates above max_pixels; integer rasters whose values exceed 255 - (typical EO UInt16) get a per-band 2-98% percentile stretch. Single-band -> - viridis; multi-band -> RGB. Requires the [viz] extra. - """ - from databricks.labs.gbx.viz._env import assert_viz_available - - assert_viz_available() - from rasterio.io import MemoryFile - - with MemoryFile(bytes(raster_bytes)) as mf: - with mf.open() as src: - data, transform, scale = _decimated_read(src, max_pixels) - _render(data, transform, title="tile.raster", fig_w=fig_w, fig_h=fig_h, scale=scale) - - -def plot_file(path, *, fig_w=10, fig_h=10, max_pixels=2000): - """Render a raster from disk (TIF, VRT, ...) with the plot_raster pipeline.""" - from databricks.labs.gbx.viz._env import assert_viz_available - - assert_viz_available() - import rasterio - - with rasterio.open(path) as src: - data, transform, scale = _decimated_read(src, max_pixels) - _render( - data, - transform, - title=f"File: {str(path).split('/')[-1]}", - fig_w=fig_w, - fig_h=fig_h, - scale=scale, - ) -``` - -Update `viz/__init__.py` to export them: -```python -from databricks.labs.gbx.viz._raster import plot_file, plot_raster - -__all__ = ["plot_raster", "plot_file"] -``` - -- [ ] **Step 4: Run to verify pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/viz/test_raster.py` -Expected: 5 passed. - -- [ ] **Step 5: Lint + commit** - -```bash -docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && black src/databricks/labs/gbx/viz test/viz && isort src/databricks/labs/gbx/viz test/viz' -git add python/geobrix/src/databricks/labs/gbx/viz/_raster.py python/geobrix/src/databricks/labs/gbx/viz/__init__.py python/geobrix/test/viz/test_raster.py -git commit -m "feat(viz): plot_raster + plot_file" -``` - ---- - -### Task 4: `viz._vector` as_gdf / cells_as_gdf - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/viz/_vector.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/viz/__init__.py` -- Test: `python/geobrix/test/viz/test_vector.py` - -**Interfaces:** -- Consumes: `assert_viz_available` (Task 1); a Spark session (test fixture pattern from `test/pyrx/conftest.py`). -- Produces: `as_gdf(df, wkt_col="wkt", *, max_rows=10_000) -> geopandas.GeoDataFrame`; `cells_as_gdf(df, cell_col="cellid", extra_cols=(), *, max_rows=10_000) -> geopandas.GeoDataFrame`. - -- [ ] **Step 1: Write the failing tests** - -`test/viz/test_vector.py`: -```python -import logging -import warnings - -import pytest - - -@pytest.fixture(scope="module") -def spark(): - logging.getLogger("py4j").setLevel(logging.ERROR) - from pyspark.sql import SparkSession - - s = ( - SparkSession.builder.master("local[2]") - .appName("viz-vector-tests") - .getOrCreate() - ) - yield s - - -def test_as_gdf_crs_geometry_and_columns(spark): - from databricks.labs.gbx.viz import as_gdf - - df = spark.createDataFrame( - [("a", "POINT (1 2)"), ("b", "POINT (3 4)")], ["name", "wkt"] - ) - gdf = as_gdf(df) - assert gdf.crs.to_epsg() == 4326 - assert list(gdf["name"]) == ["a", "b"] - assert "wkt" not in gdf.columns - assert all(gdf.geometry.is_valid) - - -def test_as_gdf_truncates_and_warns_over_max_rows(spark): - from databricks.labs.gbx.viz import as_gdf - - df = spark.range(5).selectExpr("id", "concat('POINT (', id, ' 0)') AS wkt") - with warnings.catch_warnings(record=True) as caught: - warnings.simplefilter("always") - gdf = as_gdf(df, max_rows=2) - assert len(gdf) == 2 - assert any("truncated" in str(w.message).lower() for w in caught) - - -def test_cells_as_gdf_boundary_from_h3_lib(spark): - import h3 - - from databricks.labs.gbx.viz import cells_as_gdf - - cell_int = h3.str_to_int(h3.latlng_to_cell(0.0, 0.0, 5)) - df = spark.createDataFrame([(cell_int, 7)], ["cellid", "count"]) - gdf = cells_as_gdf(df, extra_cols=["count"]) - assert gdf.crs.to_epsg() == 4326 - assert list(gdf["count"]) == [7] - assert gdf.geometry.iloc[0].geom_type == "Polygon" -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/viz/test_vector.py` -Expected: FAIL (`as_gdf`/`cells_as_gdf` not importable). - -- [ ] **Step 3: Implement `_vector.py`** - -```python -"""Spark DataFrame -> GeoDataFrame adapters for gbx.viz interactive maps. - -Collect to the driver (single-node viz); guarded by max_rows so a large frame -does not OOM the driver. Boundaries for H3 cells use the h3 lib (portable), not -the Databricks-native h3_boundaryaswkt. -""" - -import warnings - - -def as_gdf(df, wkt_col="wkt", *, max_rows=10_000): - """Spark DataFrame with a WKT column -> geopandas.GeoDataFrame (EPSG:4326). - - Collects to the driver. With max_rows set (default 10_000) the frame is - truncated to max_rows and a warning is emitted; pass max_rows=None to opt out. - """ - from databricks.labs.gbx.viz._env import assert_viz_available - - assert_viz_available() - import geopandas as gpd - - if wkt_col not in df.columns: - raise ValueError( - f"as_gdf: column {wkt_col!r} not in DataFrame columns {df.columns}" - ) - if max_rows is None: - pdf = df.toPandas() - else: - pdf = df.limit(max_rows + 1).toPandas() - if len(pdf) > max_rows: - pdf = pdf.iloc[:max_rows] - warnings.warn( - f"as_gdf: output truncated to max_rows={max_rows} for driver-side " - "viz; pass max_rows=None to collect all rows.", - stacklevel=2, - ) - geometry = gpd.GeoSeries.from_wkt(pdf[wkt_col], crs=4326) - pdf = pdf.drop(columns=[wkt_col]) - pdf["geometry"] = geometry.values - return gpd.GeoDataFrame(pdf, geometry="geometry", crs=4326) - - -def cells_as_gdf(df, cell_col="cellid", extra_cols=(), *, max_rows=10_000): - """H3 cell ids (bigint) -> boundary polygons as a GeoDataFrame (EPSG:4326). - - Boundaries come from the h3 lib (h3 v4 takes a string index, so each bigint - cellid is converted via h3.int_to_str). extra_cols are carried through. - """ - from databricks.labs.gbx.viz._env import assert_viz_available - - assert_viz_available() - import h3 - from shapely.geometry import Polygon - - cols = [cell_col, *extra_cols] - if max_rows is None: - pdf = df.select(*cols).toPandas() - else: - pdf = df.select(*cols).limit(max_rows + 1).toPandas() - if len(pdf) > max_rows: - pdf = pdf.iloc[:max_rows] - warnings.warn( - f"cells_as_gdf: output truncated to max_rows={max_rows} for " - "driver-side viz; pass max_rows=None to collect all rows.", - stacklevel=2, - ) - - def _boundary(cell_int): - ring = h3.cell_to_boundary(h3.int_to_str(int(cell_int))) - # h3 v4 returns (lat, lng) pairs; shapely wants (lng, lat). - return Polygon([(lng, lat) for lat, lng in ring]) - - import geopandas as gpd - - geometry = [_boundary(c) for c in pdf[cell_col]] - return gpd.GeoDataFrame(pdf, geometry=geometry, crs=4326) -``` - -Update `viz/__init__.py`: -```python -from databricks.labs.gbx.viz._raster import plot_file, plot_raster -from databricks.labs.gbx.viz._vector import as_gdf, cells_as_gdf - -__all__ = ["plot_raster", "plot_file", "as_gdf", "cells_as_gdf"] -``` - -- [ ] **Step 4: Run to verify pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/viz/test_vector.py` -Expected: 3 passed. (If the h3 v4 boundary ring order differs, adjust `_boundary`; verify with `h3.cell_to_boundary(h3.int_to_str(cell_int))` shape in the container.) - -- [ ] **Step 5: Lint + commit** - -```bash -docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && black src/databricks/labs/gbx/viz test/viz && isort src/databricks/labs/gbx/viz test/viz' -git add python/geobrix/src/databricks/labs/gbx/viz/_vector.py python/geobrix/src/databricks/labs/gbx/viz/__init__.py python/geobrix/test/viz/test_vector.py -git commit -m "feat(viz): as_gdf + cells_as_gdf" -``` - ---- - -### Task 5: pyrx escape-hatches `tile_to_numpy` + `rst_apply` - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyrx/core/escape.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` (add the two imports/re-exports near the other `core` imports, ~line 38-50) -- Test: `python/geobrix/test/pyrx/test_escape.py` - -**Interfaces:** -- Consumes: `pyrx._serde.open_tile(raster_bytes)` (context manager → rasterio DatasetReader); `pyrx._udf._col` (str/Column normalizer); `make_geotiff_bytes` + `spark` (test/pyrx/conftest.py). -- Produces: `tile_to_numpy(tile_or_bytes) -> np.ndarray`; `rst_apply(tile_col, fn, returnType=DoubleType()) -> Column`. Both re-exported on `databricks.labs.gbx.pyrx.functions`. - -- [ ] **Step 1: Write the failing tests** - -`test/pyrx/test_escape.py`: -```python -import numpy as np -from pyspark.sql import functions as f -from pyspark.sql.types import IntegerType - -from databricks.labs.gbx.pyrx import _serde -from databricks.labs.gbx.pyrx.functions import rst_apply, tile_to_numpy -from test.pyrx.conftest import make_geotiff_bytes - - -def test_tile_to_numpy_bytes_and_struct_agree(): - raw = make_geotiff_bytes(width=4, height=3, count=2) - arr_bytes = tile_to_numpy(raw) - assert isinstance(arr_bytes, np.ndarray) - assert arr_bytes.shape == (2, 3, 4) - tile = _serde.build_tile(raw, "GTiff", cellid=0) - arr_struct = tile_to_numpy(tile) - assert np.array_equal(arr_bytes, arr_struct) - - -def test_rst_apply_scalar_with_nondefault_returntype(spark): - raw = make_geotiff_bytes(width=4, height=3, count=1) - tile = _serde.build_tile(raw, "GTiff", cellid=0) - df = spark.createDataFrame([(tile,)], ["tile"]) - out = df.select( - rst_apply("tile", lambda ds: ds.count, returnType=IntegerType()).alias("nbands") - ).collect() - assert out[0]["nbands"] == 1 # ds.count == band count == 1 - - -def test_rst_apply_null_tile_returns_null(spark): - df = spark.createDataFrame([(None,)], "tile struct>") - out = df.select(rst_apply("tile", lambda ds: 1.0).alias("v")).collect() - assert out[0]["v"] is None -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_escape.py` -Expected: FAIL (`rst_apply`/`tile_to_numpy` not importable). - -- [ ] **Step 3: Implement `core/escape.py`** - -```python -"""Python-only escape-hatches for users whose needs fall outside the rst_* surface. - -NOT SQL-registered (tile_to_numpy returns a host object; rst_apply takes a Python -callable), so neither appears in registered_functions.txt / function-info.json. -""" - -from pyspark.sql import Column -from pyspark.sql.functions import udf -from pyspark.sql.types import DataType, DoubleType - -from databricks.labs.gbx.pyrx import _serde -from databricks.labs.gbx.pyrx._udf import _col - - -def tile_to_numpy(tile_or_bytes): - """Read a tile's raster into a numpy ndarray (all bands). - - Accepts a tile struct (a Row/dict with a 'raster' field) or raw bytes. The - "drop to numpy" hatch: call on a collected tile, or inside your own UDF. - """ - if isinstance(tile_or_bytes, (bytes, bytearray)): - raw = bytes(tile_or_bytes) - else: - raw = bytes(tile_or_bytes["raster"]) - with _serde.open_tile(raw) as ds: - return ds.read() - - -def rst_apply(tile_col, fn, returnType: DataType = DoubleType()) -> Column: - """Apply an arbitrary rasterio function to each tile, returning one scalar/row. - - fn receives an open rasterio DatasetReader and returns a value of returnType - (default DoubleType; any Spark DataType). The escape-hatch for "GeoBrix lacks - function X — run your own rasterio per tile". Scalar return only. Null/empty - tile -> null. - """ - - @udf(returnType=returnType) - def _apply(tile): - if tile is None or tile["raster"] is None: - return None - with _serde.open_tile(bytes(tile["raster"])) as ds: - return fn(ds) - - return _apply(_col(tile_col)) -``` - -Modify `pyrx/functions.py` — add near the other `from ...pyrx.core import` lines: -```python -from databricks.labs.gbx.pyrx.core.escape import rst_apply, tile_to_numpy -``` -(Confirm `_udf._col` exists and accepts a str/Column; it is used at `_udf.py:22`.) - -- [ ] **Step 4: Run to verify pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_escape.py` -Expected: 3 passed. - -- [ ] **Step 5: Confirm binding-parity is unaffected** - -Run: `bash scripts/commands/gbx-test-bindings.sh` (or `docs/scripts/check-binding-parity.py`). -Expected: PASS — `tile_to_numpy`/`rst_apply` are NOT in `registered_functions.txt`, so the counts are unchanged. - -- [ ] **Step 6: Lint + commit** - -```bash -docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && black src/databricks/labs/gbx/pyrx/core/escape.py src/databricks/labs/gbx/pyrx/functions.py test/pyrx/test_escape.py && isort src/databricks/labs/gbx/pyrx/core/escape.py src/databricks/labs/gbx/pyrx/functions.py test/pyrx/test_escape.py' -git add python/geobrix/src/databricks/labs/gbx/pyrx/core/escape.py python/geobrix/src/databricks/labs/gbx/pyrx/functions.py python/geobrix/test/pyrx/test_escape.py -git commit -m "feat(pyrx): tile_to_numpy + rst_apply escape-hatches" -``` - ---- - -### Task 6: Wire `test/viz` into the lightweight CI tier + clean-venv verify - -**Files:** -- Modify: `python/geobrix/test/conftest.py` (line 40, `_LIGHT_TEST_DIRS`) -- Modify: `.github/actions/pyrx_build/action.yml` (the `pytest test/pyrx ...` dir list, ~line 67) - -**Interfaces:** none (CI config only). - -- [ ] **Step 1: Add `"viz"` to `_LIGHT_TEST_DIRS`** - -In `test/conftest.py`, change: -```python -_LIGHT_TEST_DIRS = ["bench", "ds", "pyrx", "pyvx", "pygx", "pmtiles_light", "stac"] -``` -to: -```python -_LIGHT_TEST_DIRS = ["bench", "ds", "pyrx", "pyvx", "pygx", "pmtiles_light", "stac", "viz"] -``` -And update the docstring's "Light test dirs so far:" line to append `, viz`. - -- [ ] **Step 2: Add `test/viz` to the light CI pytest dir list** - -In `.github/actions/pyrx_build/action.yml`, change the pytest line to include `test/viz`: -``` -pytest test/pyrx test/ds test/pyvx test/pygx test/pmtiles_light test/stac test/viz -m "not integration" -v -``` -Update the preceding comment to list `viz (gbx.viz, [viz] extra)`. - -- [ ] **Step 3: Verify heavy phase skips viz (no rasterio/geopandas)** - -Run (simulates heavy env — block the light deps): -``` -docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && python3 - < **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a DGGS-cell rasterizer to RasterX — `rst_h3_rasterize_agg` (H3 cellids → raster tile, pixel-centroid burn) plus the `rst_h3_gridspec` shared-grid helper and a scalar `gbx_h3_cell_bbox` — in both the lightweight (pyrx) and heavyweight (rasterx) tiers. - -**Architecture:** Pixel-centroid burn is the exact inverse of `rst_h3_rastertogrid` (each output pixel takes the value of the cell containing its center). The light tier is built first as pure functions + a grouped `pandas_udf` (SQL returns BINARY, Python wrapper composes the tile struct — the established light-agg convention); the heavy tier mirrors `RST_RasterizeAgg` as a Scala UDAF, validated against the light tier by a JAR-gated parity test. The grid helper computes a snapped global-lattice grid so per-threshold bands stack aligned and per-cell tiles merge losslessly. - -**Tech Stack:** Python 3.12, PySpark 4.0, rasterio + h3 + numpy + pyproj (all already light deps), Scala 2.13 / Spark 4 / GDAL (heavy). Spec: `docs/superpowers/specs/2026-06-23-h3-cell-rasterizer-design.md`. - -## Global Constraints - -- Target branch `beta/0.4.0`. Names are fixed: `rst_h3_rasterize_agg`, `rst_h3_gridspec`, scalar `gbx_h3_cell_bbox`. -- **Algorithm:** pixel-centroid burn — pixel center → `h3.latlng_to_cell(lat, lon, res)` → value if cell in set else NoData (-9999.0). Resolution inferred from the cells (`h3.get_resolution`); error on mixed resolutions in a group. -- **Default `value`** when omitted/null = `1.0` (presence mask). -- **CRS:** default EPSG:4326; optional projected `srid` (pixel centers unprojected to lon/lat for the H3 lookup). No new deps. -- **Grid:** default auto extent + pixel size from H3 resolution; `mode='centroids'` (default) | `'spatial_envelope'`; `kring_pad` (int, default 1) expands the cell set by N rings (NoData margin) before bounds; origin snapped to a `pixel_size` multiple (global lattice). -- **Light SQL agg returns `BINARY`**; the Python wrapper composes the tile struct via `_as_tile_udf` (per [[light-agg-struct-return-convention]]). Document the deviation as an orange `:::warning` like the other `rst_*_agg` light functions. -- **Int handling:** PySpark passes H3 ids as signed `Long`; h3 ids are unsigned 64-bit. Normalize every cellid through `_h3_str(cellid)` = `h3.int_to_str(int(cellid) & 0xFFFFFFFFFFFFFFFF)` and key all maps by the h3 string (per [[jts_towkb_strips_z]] int-tolerance note). -- **Serverless-safe:** no `spark.conf.set`, `.cache()/.persist()`, `_jvm`/`sparkContext`/`.rdd` in package code. -- **Binding parity:** add every new SQL function to Scala `register`, Python `functions.py`, `function-info.json`, and `docs/tests-function-info/registered_functions.txt` (the QC `binding-parity` gate runs on push). -- All tests/lint run in the `geobrix-dev` Docker container (`bash scripts/commands/gbx-docker-start.sh`; `bash scripts/commands/gbx-test-python.sh --path

`; lint via in-container black/isort). Scala tests via `bash scripts/commands/gbx-test-scala.sh --suite ''`. -- No internal/"wave" vocabulary in any `docs/docs/` page. - ---- - -## File Structure - -- Create `python/geobrix/src/databricks/labs/gbx/pyrx/core/cellraster.py` — pure burn + gridspec math (`cells_to_raster`, `compute_gridspec`, `cell_bbox`, `_h3_str`). -- Modify `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` — scalar `gbx_h3_cell_bbox` UDF, `rst_h3_gridspec` DataFrame helper, grouped `_rst_h3_rasterize_agg_udf` + `rst_h3_rasterize_agg` Python API, `SQL_REGISTRY` entries. -- Create `python/geobrix/test/pyrx/test_core_cellraster.py`, `.../test_h3_gridspec.py`, `.../test_h3_rasterize_agg.py`, `.../test_h3_rasterize_validate.py`, `.../test_h3_rasterize_fcc.py`. -- Create fixture `python/geobrix/test/pyrx/data/fcc_uflw_miamidade_subset.csv`. -- Create Scala `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/agg/RST_H3_RasterizeAgg.scala`; modify `rasterx/functions.scala`. Scalar `gbx_h3_cell_bbox` heavy expr under `gridx` expressions; Scala tests under `src/test/scala/.../rasterx/`. -- Modify `docs/tests-function-info/registered_functions.txt`, `docs/tests/python/api/rasterx_functions_sql.py`, `docs/docs/api/raster-functions.mdx`. -- Create `notebooks/examples/h3-rasterize/` (notebook + README) — DEM isoband demo. - ---- - -### Task 1: Light burn + gridspec core (pure functions) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pyrx/core/cellraster.py` -- Test: `python/geobrix/test/pyrx/test_core_cellraster.py` - -**Interfaces:** -- Produces: `_h3_str(cellid:int)->str`; `cell_bbox(cellid:int, srid:int=4326, mode:str="centroids")->tuple[float,float,float,float]`; `compute_gridspec(cellids, srid=4326, pixel_size=None, mode="centroids", kring_pad=1)->tuple[xmin,ymin,xmax,ymax,pixel_size,width,height,srid]`; `cells_to_raster(cell_values:dict[int,float], xmin,ymin,xmax,ymax,pixel_size,width,height,srid,resolution)->bytes` (arg order matches the `compute_gridspec` 8-tuple). Consumed by Tasks 2 (gridspec/bbox) and 3 (agg). - -- [ ] **Step 1: Write the failing tests** - -`python/geobrix/test/pyrx/test_core_cellraster.py`: -```python -import h3 -import numpy as np - -from databricks.labs.gbx.pyrx.core import cellraster as cr -from databricks.labs.gbx.pyrx import _serde - - -def _cell(lat, lon, res=9): - return h3.str_to_int(h3.latlng_to_cell(lat, lon, res)) - - -def test_h3_str_normalizes_signed_long(): - cid_unsigned = h3.str_to_int(h3.latlng_to_cell(0.0, 0.0, 9)) - signed = cid_unsigned - (1 << 64) if cid_unsigned >= (1 << 63) else cid_unsigned - assert cr._h3_str(signed) == cr._h3_str(cid_unsigned) == h3.int_to_str(cid_unsigned) - - -def test_compute_gridspec_single_cell_centroids_uses_kring1(): - cid = _cell(0.0, 0.0, 9) - # kring_pad=1 (default): a single cell is non-degenerate (neighbor centroids) - xmin, ymin, xmax, ymax, px, w, h, srid = cr.compute_gridspec([cid]) - assert w >= 3 and h >= 3 and srid == 4326 - assert xmax > xmin and ymax > ymin - # kring_pad=0: degenerate -> 1x1 (centroid only) - g0 = cr.compute_gridspec([cid], kring_pad=0) - assert g0[5] == 1 and g0[6] == 1 - - -def test_compute_gridspec_origin_snapped_to_lattice(): - cid = _cell(10.0, 20.0, 9) - xmin, ymin, xmax, ymax, px, w, h, srid = cr.compute_gridspec([cid], pixel_size=0.01) - # origin is an integer multiple of pixel_size -> independently-built grids align - assert abs((xmin / 0.01) - round(xmin / 0.01)) < 1e-9 - assert abs((ymax / 0.01) - round(ymax / 0.01)) < 1e-9 - - -def test_compute_gridspec_rejects_mixed_resolution(): - import pytest - a = _cell(0.0, 0.0, 9) - b = _cell(0.0, 0.0, 8) - with pytest.raises(ValueError, match="resolution"): - cr.compute_gridspec([a, b]) - - -def test_cells_to_raster_partition_property(): - # polyfill a small area -> cells; rasterize as presence mask; every burned - # pixel centroid must re-index to a cell IN the set, every NoData pixel must not. - res = 9 - poly = h3.LatLngPoly([(0.0, 0.0), (0.0, 0.02), (0.02, 0.02), (0.02, 0.0)]) - cells = {h3.str_to_int(c) for c in h3.polygon_to_cells(poly, res)} - cell_values = {c: 1.0 for c in cells} - g = cr.compute_gridspec(list(cells), kring_pad=1) - raster = cr.cells_to_raster(cell_values, *g, resolution=res) - with _serde.open_tile(raster) as ds: - arr = ds.read(1) - t = ds.transform - nod = ds.nodata - cellset = {cr._h3_str(c) for c in cells} - for row in range(ds.height): - for col in range(ds.width): - lon, lat = (t * (col + 0.5, row + 0.5)) - idx = h3.latlng_to_cell(lat, lon, res) - burned = arr[row, col] != nod - assert burned == (idx in cellset) -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_core_cellraster.py` -Expected: FAIL (`No module named ...core.cellraster`). - -- [ ] **Step 3: Implement `cellraster.py`** - -```python -"""Rasterize a set of H3 cells onto a regular grid (pixel-centroid burn). - -The inverse of core.gridagg.raster_to_grid: there each pixel centroid is indexed -to an H3 cell; here each output pixel takes the value of the cell containing its -centroid. Pure functions (no Spark); rasterio + h3 + numpy + pyproj only. -""" -import math - -import h3 -import numpy as np -from rasterio.io import MemoryFile -from rasterio.transform import Affine - -_NODATA = -9999.0 -_U64 = 0xFFFFFFFFFFFFFFFF - - -def _h3_str(cellid) -> str: - """Canonical h3 string for a (possibly signed) Spark Long cell id.""" - return h3.int_to_str(int(cellid) & _U64) - - -def _resolution(cell_strs) -> int: - res = h3.get_resolution(next(iter(cell_strs))) - for c in cell_strs: - if h3.get_resolution(c) != res: - raise ValueError("H3 cell set has mixed resolutions") - return res - - -def _reproject(xs, ys, src, dst): - if src == dst: - return np.asarray(xs, dtype="float64"), np.asarray(ys, dtype="float64") - from pyproj import Transformer - - tr = Transformer.from_crs(src, dst, always_xy=True) - x2, y2 = tr.transform(np.asarray(xs), np.asarray(ys)) - return np.asarray(x2, dtype="float64"), np.asarray(y2, dtype="float64") - - -def cell_bbox(cellid, srid=4326, mode="centroids"): - """(xmin, ymin, xmax, ymax) for one cell in `srid`. - - mode='centroids' -> the centroid point (degenerate bbox); 'spatial_envelope' - -> the hexagon boundary envelope. - """ - c = _h3_str(cellid) - if mode == "centroids": - lat, lon = h3.cell_to_latlng(c) - lons, lats = [lon], [lat] - elif mode == "spatial_envelope": - b = h3.cell_to_boundary(c) # [(lat, lon), ...] - lats = [p[0] for p in b] - lons = [p[1] for p in b] - else: - raise ValueError(f"unknown mode {mode!r}") - xs, ys = _reproject(lons, lats, 4326, srid) - return float(xs.min()), float(ys.min()), float(xs.max()), float(ys.max()) - - -def compute_gridspec(cellids, srid=4326, pixel_size=None, mode="centroids", kring_pad=1): - """Snapped, lattice-aligned grid spec for a cell set. - - Returns (xmin, ymin, xmax, ymax, pixel_size, width, height, srid). - """ - cells = {_h3_str(c) for c in cellids} - if not cells: - raise ValueError("empty cell set") - res = _resolution(cells) - if kring_pad and kring_pad > 0: - padded = set() - for c in cells: - padded.update(h3.grid_disk(c, kring_pad)) - cells = padded - - if mode == "centroids": - pts = [h3.cell_to_latlng(c) for c in cells] # (lat, lon) - lons = [p[1] for p in pts] - lats = [p[0] for p in pts] - elif mode == "spatial_envelope": - lons, lats = [], [] - for c in cells: - for (la, lo) in h3.cell_to_boundary(c): - lons.append(lo) - lats.append(la) - else: - raise ValueError(f"unknown mode {mode!r}") - - xs, ys = _reproject(lons, lats, 4326, srid) - bxmin, bxmax = float(xs.min()), float(xs.max()) - bymin, bymax = float(ys.min()), float(ys.max()) - - if pixel_size is None: - edge_m = h3.average_hexagon_edge_length(res, unit="m") - if srid == 4326: - midlat = (bymin + bymax) / 2.0 - pixel_size = edge_m / (111320.0 * max(math.cos(math.radians(midlat)), 1e-6)) - else: - pixel_size = edge_m - - if mode == "centroids": - half = pixel_size / 2.0 - bxmin -= half; bxmax += half - bymin -= half; bymax += half - - xmin = math.floor(bxmin / pixel_size) * pixel_size - ymax = math.ceil(bymax / pixel_size) * pixel_size - width = max(1, int(math.ceil((bxmax - xmin) / pixel_size))) - height = max(1, int(math.ceil((ymax - bymin) / pixel_size))) - xmax = xmin + width * pixel_size - ymin = ymax - height * pixel_size - return (xmin, ymin, xmax, ymax, pixel_size, width, height, srid) - - -def cells_to_raster(cell_values, xmin, ymin, xmax, ymax, pixel_size, width, height, - srid, resolution): - """Burn {cellid:int -> value:float} onto a width x height grid (centroid burn). - - Arg order matches the `compute_gridspec` 8-tuple (so callers splat it: - `cells_to_raster(cell_values, *gridspec, resolution=res)`). The snapped grid has - square pixels of `pixel_size`. Returns single-band float64 GTiff bytes; NoData - where no cell covers a pixel. - """ - lut = {_h3_str(c): float(v) for c, v in cell_values.items()} - transform = Affine(pixel_size, 0.0, xmin, 0.0, -pixel_size, ymax) - - cols = (np.arange(width) + 0.5) - rows = (np.arange(height) + 0.5) - gx, gy = np.meshgrid(xmin + cols * pixel_size, ymax - rows * pixel_size) # (h, w) - lon, lat = _reproject(gx.ravel(), gy.ravel(), srid, 4326) - - out = np.full(lon.size, _NODATA, dtype="float64") - # Scalar h3 index per pixel (no array API). The grid is bounded to the cells' - # padded bbox, so this is O(pixels-in-footprint). PERF FOLLOW-UP: restrict to - # pixels within each cell's local window instead of the whole grid. - for i in range(lon.size): - v = lut.get(h3.int_to_str(h3.str_to_int(h3.latlng_to_cell(float(lat[i]), float(lon[i]), resolution)))) - if v is not None: - out[i] = v - - data = out.reshape(height, width) - profile = dict(driver="GTiff", width=width, height=height, count=1, - dtype="float64", crs=f"EPSG:{srid}", transform=transform, nodata=_NODATA) - with MemoryFile() as mf: - with mf.open(**profile) as ds: - ds.write(data, 1) - return mf.read() -``` - -- [ ] **Step 4: Run to verify pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_core_cellraster.py` -Expected: 5 passed. (If `h3.polygon_to_cells` differs in 4.4.2, use `h3.polygon_to_cells_experimental(poly, res, contain="overlap")` as in `tessellate.py:91`.) - -- [ ] **Step 5: Lint + commit** - -```bash -docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && black src/databricks/labs/gbx/pyrx/core/cellraster.py test/pyrx/test_core_cellraster.py && isort src/databricks/labs/gbx/pyrx/core/cellraster.py test/pyrx/test_core_cellraster.py' -git add python/geobrix/src/databricks/labs/gbx/pyrx/core/cellraster.py python/geobrix/test/pyrx/test_core_cellraster.py -git commit -m "feat(pyrx): H3 cell rasterize core (centroid burn + gridspec)" -``` - ---- - -### Task 2: Light scalar `gbx_h3_cell_bbox` + `rst_h3_gridspec` helper - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` -- Test: `python/geobrix/test/pyrx/test_h3_gridspec.py` - -**Interfaces:** -- Consumes: `cellraster.cell_bbox`, `cellraster.compute_gridspec` (Task 1); `_col`, `_serde`, the `spark` fixture (`test/pyrx/conftest.py`). -- Produces: scalar UDF `_h3_cell_bbox_udf` registered as `gbx_h3_cell_bbox`; Python `rst_h3_gridspec(df, cell_col="cellid", *group_cols, srid=4326, pixel_size=None, mode="centroids", kring_pad=1) -> DataFrame` (adds a `grid STRUCT` column). Consumed by the customer workflow + Task 8 docs. - -- [ ] **Step 1: Write the failing test** - -`python/geobrix/test/pyrx/test_h3_gridspec.py`: -```python -import h3 - -from databricks.labs.gbx.pyrx import functions as rx - - -def test_rst_h3_gridspec_matches_core(spark): - res = 9 - poly = h3.LatLngPoly([(0.0, 0.0), (0.0, 0.02), (0.02, 0.02), (0.02, 0.0)]) - cells = [h3.str_to_int(c) for c in h3.polygon_to_cells(poly, res)] - df = spark.createDataFrame([(int(c), "TX1") for c in cells], ["cellid", "tx"]) - out = rx.rst_h3_gridspec(df, "cellid", "tx", pixel_size=0.005).collect() - assert len(out) == 1 - g = out[0]["grid"] - from databricks.labs.gbx.pyrx.core import cellraster as cr - exp = cr.compute_gridspec(cells, pixel_size=0.005) - assert g["width"] == exp[5] and g["height"] == exp[6] - assert abs(g["xmin"] - exp[0]) < 1e-9 -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_h3_gridspec.py` -Expected: FAIL (`module 'functions' has no attribute 'rst_h3_gridspec'`). - -- [ ] **Step 3: Implement in `functions.py`** - -Add the scalar bbox UDF + helper (near the other tile/scalar UDFs; import `cellraster` at the top with the other `from ...pyrx.core import` lines): -```python -from databricks.labs.gbx.pyrx.core import cellraster as cellraster_core - -_GRID_SCHEMA = StructType([ - StructField("xmin", DoubleType()), StructField("ymin", DoubleType()), - StructField("xmax", DoubleType()), StructField("ymax", DoubleType()), - StructField("pixel_size", DoubleType()), - StructField("width", IntegerType()), StructField("height", IntegerType()), - StructField("srid", IntegerType()), -]) - -_BBOX_SCHEMA = StructType([ - StructField("xmin", DoubleType()), StructField("ymin", DoubleType()), - StructField("xmax", DoubleType()), StructField("ymax", DoubleType()), -]) - - -@f.udf(_BBOX_SCHEMA) -def _h3_cell_bbox_udf(cellid, srid, mode): - if cellid is None: - return None - xmin, ymin, xmax, ymax = cellraster_core.cell_bbox( - int(cellid), int(srid) if srid is not None else 4326, mode or "centroids" - ) - return (xmin, ymin, xmax, ymax) - - -def gbx_h3_cell_bbox(cellid: ColLike, srid: ColLike = None, mode: ColLike = None) -> Column: - """Bounding box of one H3 cell in `srid` (centroid point or hexagon envelope).""" - return _h3_cell_bbox_udf(_col(cellid), _col(srid) if srid is not None else f.lit(4326), - _col(mode) if mode is not None else f.lit("centroids")) - - -def rst_h3_gridspec(df, cell_col="cellid", *group_cols, srid=4326, pixel_size=None, - mode="centroids", kring_pad=1): - """Add a `grid` struct (snapped shared canvas) per group of H3 cells. - - Implemented as scalar per-cell bbox + native min/max + the snap arithmetic, so - it works identically in both tiers and avoids the grouped-pandas_udf struct limit. - """ - @f.udf(_GRID_SCHEMA) - def _snap_udf(xmin, ymin, xmax, ymax, mid_lat, res): - return cellraster_core.snap_bounds( - float(xmin), float(ymin), float(xmax), float(ymax), - srid, pixel_size, mode, float(mid_lat), int(res), - ) - - b = _h3_cell_bbox_udf(_col(cell_col), f.lit(srid), f.lit(mode)) - # NOTE: kring_pad expansion for the bounds is applied inside the bbox via the - # padded boundary; for centroids mode the half-pixel pad is in snap_bounds. - gcols = list(group_cols) - enriched = df.withColumn("_bb", b) - agg = (enriched.groupBy(*gcols) if gcols else enriched.groupBy()) - bounds = agg.agg( - f.min("_bb.xmin").alias("xmin"), f.min("_bb.ymin").alias("ymin"), - f.max("_bb.xmax").alias("xmax"), f.max("_bb.ymax").alias("ymax"), - ) - mid = ((f.col("ymin") + f.col("ymax")) / 2.0) - res_col = f.lit(0) # resolution carried via snap default if unused; see note - return bounds.withColumn( - "grid", - _snap_udf(f.col("xmin"), f.col("ymin"), f.col("xmax"), f.col("ymax"), mid, res_col), - ) -``` -And add a `snap_bounds(...)` helper to `cellraster.py` extracting the snap arithmetic from `compute_gridspec` (DRY — `compute_gridspec` should call it too): -```python -def snap_bounds(bxmin, bymin, bxmax, bymax, srid, pixel_size, mode, mid_lat, res): - if pixel_size is None: - edge_m = h3.average_hexagon_edge_length(res, unit="m") if res else 1.0 - pixel_size = (edge_m / (111320.0 * max(math.cos(math.radians(mid_lat)), 1e-6)) - if srid == 4326 else edge_m) - if mode == "centroids": - half = pixel_size / 2.0 - bxmin -= half; bxmax += half; bymin -= half; bymax += half - xmin = math.floor(bxmin / pixel_size) * pixel_size - ymax = math.ceil(bymax / pixel_size) * pixel_size - width = max(1, int(math.ceil((bxmax - xmin) / pixel_size))) - height = max(1, int(math.ceil((ymax - bymin) / pixel_size))) - return (xmin, ymax - height * pixel_size, xmin + width * pixel_size, ymax, - pixel_size, width, height, srid) -``` -Register the scalar in `SQL_REGISTRY` (the `_sql_accessors`/scalar map): add `"gbx_h3_cell_bbox": _h3_cell_bbox_udf`. - -**Implementation note for the implementer:** the `kring_pad` and per-cell resolution need to reach `_snap_udf`. Resolve by computing the H3 resolution from a sampled `cell_col` value at helper-call time (driver-side `df.select(cell_col).first()`), and apply `kring_pad` by expanding each cell's bbox via `cellraster.cell_bbox` over `h3.grid_disk(cell, kring_pad)` inside `_h3_cell_bbox_udf` (pass `kring_pad` as a 4th lit arg). Keep `compute_gridspec` (Task 1) as the single-call reference; the helper reproduces its result group-wise. The Task-2 test asserts the helper equals `compute_gridspec` — make them agree. - -- [ ] **Step 4: Run to verify pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_h3_gridspec.py` -Expected: 1 passed. - -- [ ] **Step 5: Lint + commit** - -```bash -docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && black src/databricks/labs/gbx/pyrx/functions.py src/databricks/labs/gbx/pyrx/core/cellraster.py test/pyrx/test_h3_gridspec.py && isort ' -git add python/geobrix/src/databricks/labs/gbx/pyrx/functions.py python/geobrix/src/databricks/labs/gbx/pyrx/core/cellraster.py python/geobrix/test/pyrx/test_h3_gridspec.py -git commit -m "feat(pyrx): gbx_h3_cell_bbox scalar + rst_h3_gridspec helper" -``` - ---- - -### Task 3: Light `rst_h3_rasterize_agg` (grouped pandas_udf + tile wrapper) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` -- Test: `python/geobrix/test/pyrx/test_h3_rasterize_agg.py` - -**Interfaces:** -- Consumes: `cellraster.cells_to_raster`, `cellraster.compute_gridspec` (Task 1); `_as_tile_udf` (functions.py:2870), `_col`, `_serde`. -- Produces: `_rst_h3_rasterize_agg_udf` (`@pandas_udf(BinaryType())`), `rst_h3_rasterize_agg(cellid, value=None, srid=None, pixel_size=None, xmin=None, ymin=None, xmax=None, ymax=None, width=None, height=None, mode="centroids", kring_pad=1) -> Column`, and `"gbx_rst_h3_rasterize_agg"` in `_sql_aggregators`. - -- [ ] **Step 1: Write the failing test** - -`python/geobrix/test/pyrx/test_h3_rasterize_agg.py`: -```python -import h3 - -from databricks.labs.gbx.pyrx import functions as rx -from databricks.labs.gbx.pyrx import _serde - - -def test_rst_h3_rasterize_agg_presence_mask(spark): - res = 9 - poly = h3.LatLngPoly([(0.0, 0.0), (0.0, 0.02), (0.02, 0.02), (0.02, 0.0)]) - cells = [h3.str_to_int(c) for c in h3.polygon_to_cells(poly, res)] - df = spark.createDataFrame([(int(c), "TX1") for c in cells], ["cellid", "tx"]) - out = ( - df.groupBy("tx") - .agg(rx.rst_h3_rasterize_agg("cellid").alias("tile")) - .collect() - ) - tile = out[0]["tile"] - assert tile is not None and tile["raster"] is not None - with _serde.open_tile(bytes(tile["raster"])) as ds: - arr = ds.read(1) - # presence mask -> covered pixels are 1.0, count matches >=1 per cell - assert (arr == 1.0).sum() >= len(cells) - assert ds.nodata == -9999.0 - - -def test_rst_h3_rasterize_agg_burns_value(spark): - res = 9 - c = h3.str_to_int(h3.latlng_to_cell(0.0, 0.0, res)) - df = spark.createDataFrame([(int(c), 42.0, "TX1")], ["cellid", "val", "tx"]) - out = ( - df.groupBy("tx") - .agg(rx.rst_h3_rasterize_agg("cellid", "val").alias("tile")) - .collect() - ) - with _serde.open_tile(bytes(out[0]["tile"]["raster"])) as ds: - arr = ds.read(1) - assert (arr == 42.0).sum() >= 1 -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_h3_rasterize_agg.py` -Expected: FAIL (no `rst_h3_rasterize_agg`). - -- [ ] **Step 3: Implement in `functions.py`** - -```python -@pandas_udf(BinaryType()) -def _rst_h3_rasterize_agg_udf( - cellid: pd.Series, value: pd.Series, srid: pd.Series, pixel_size: pd.Series, - xmin: pd.Series, ymin: pd.Series, xmax: pd.Series, ymax: pd.Series, - width: pd.Series, height: pd.Series, mode: pd.Series, kring_pad: pd.Series, -) -> bytes: - from databricks.labs.gbx.pyrx import _env - from databricks.labs.gbx.pyrx.core import cellraster as cr - - _env.configure_gdal_env() - cells = [int(c) for c in cellid if c is not None] - if not cells: - return None - vals = [float(v) if v is not None else 1.0 for v in value] if value is not None else [1.0] * len(cells) - cell_values = {} - for c, v in zip(cells, vals): - cell_values[c] = v # last-wins (cells of one res don't overlap) - - res = cr._resolution([cr._h3_str(c) for c in cells]) - _srid = int(srid.iloc[0]) if srid is not None and srid.iloc[0] is not None else 4326 - _mode = mode.iloc[0] if mode is not None and mode.iloc[0] is not None else "centroids" - _kp = int(kring_pad.iloc[0]) if kring_pad is not None and kring_pad.iloc[0] is not None else 1 - - def _has(s): - return s is not None and s.iloc[0] is not None - - if _has(xmin) and _has(width): - grid = (float(xmin.iloc[0]), float(ymin.iloc[0]), float(xmax.iloc[0]), - float(ymax.iloc[0]), (xmax.iloc[0] - xmin.iloc[0]) / int(width.iloc[0]), - int(width.iloc[0]), int(height.iloc[0]), _srid) - else: - _ps = float(pixel_size.iloc[0]) if _has(pixel_size) else None - grid = cr.compute_gridspec(cells, srid=_srid, pixel_size=_ps, mode=_mode, kring_pad=_kp) - return cr.cells_to_raster(cell_values, *grid, resolution=res) - - -def rst_h3_rasterize_agg(cellid: ColLike, value: ColLike = None, srid: ColLike = None, - pixel_size: ColLike = None, xmin: ColLike = None, ymin: ColLike = None, - xmax: ColLike = None, ymax: ColLike = None, width: ColLike = None, - height: ColLike = None, mode: ColLike = None, kring_pad: ColLike = None) -> Column: - """Rasterize a group's H3 cells into ONE tile (pixel-centroid burn). - - value omitted -> presence mask (1.0/NoData). Supply an explicit extent - (xmin..height, e.g. from rst_h3_gridspec) for aligned band stacking; else the - grid is auto-derived per mode/kring_pad. - """ - def _c(x, default): - return _col(x) if x is not None else f.lit(default) - return _as_tile_udf( - _rst_h3_rasterize_agg_udf( - _col(cellid), _c(value, None), _c(srid, 4326), _c(pixel_size, None), - _c(xmin, None), _c(ymin, None), _c(xmax, None), _c(ymax, None), - _c(width, None), _c(height, None), _c(mode, "centroids"), _c(kring_pad, 1), - ) - ) -``` -Add to `_sql_aggregators` (functions.py:3421): `"gbx_rst_h3_rasterize_agg": _rst_h3_rasterize_agg_udf,`. - -- [ ] **Step 4: Run to verify pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_h3_rasterize_agg.py` -Expected: 2 passed. - -- [ ] **Step 5: Lint + commit** - -```bash -docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && black src/databricks/labs/gbx/pyrx/functions.py test/pyrx/test_h3_rasterize_agg.py && isort ' -git add python/geobrix/src/databricks/labs/gbx/pyrx/functions.py python/geobrix/test/pyrx/test_h3_rasterize_agg.py -git commit -m "feat(pyrx): rst_h3_rasterize_agg grouped aggregator" -``` - ---- - -### Task 4: Light registration — binding parity + function-info + doc SQL example - -**Files:** -- Modify: `docs/tests-function-info/registered_functions.txt`, `docs/tests/python/api/rasterx_functions_sql.py`, `docs/tests/python/api/test_rasterx_functions_sql.py` -- Run: `gbx:docs:function-info` - -**Interfaces:** none (registration/metadata). - -- [ ] **Step 1: Add the function names to the canonical list** - -Append to `docs/tests-function-info/registered_functions.txt` (alphabetical position): `gbx_rst_h3_rasterize_agg` and `gbx_h3_cell_bbox`. (Do NOT add `rst_h3_gridspec` — it is a Python/DataFrame helper, not a registered SQL UDF; the scalar `gbx_h3_cell_bbox` is the SQL surface.) - -- [ ] **Step 2: Add doc-test SQL examples** - -In `docs/tests/python/api/rasterx_functions_sql.py`, add `rst_h3_rasterize_agg_sql_example()` and `h3_cell_bbox_sql_example()` returning runnable SQL strings (use a small `VALUES` cell set; group-by; assert a non-null tile / non-null bbox). Follow the existing `*_sql_example()` shape in that file. Add matching tests in `test_rasterx_functions_sql.py`. - -- [ ] **Step 3: Regenerate function-info + run the binding-parity check** - -Run (subagent / Docker — may take minutes): -``` -bash scripts/commands/gbx-docs-function-info.sh -bash scripts/commands/gbx-test-bindings.sh -``` -Expected: function-info.json gains both functions; binding-parity PASSES (counts consistent: the SQL functions exist as Python `functions.py` entries + in `registered_functions.txt` + `function-info.json`; the Scala side is added in Task 7 — until then run binding-parity in light-only mode or expect the Scala-missing note for the two new names, which Task 7 resolves). - -- [ ] **Step 4: Commit** - -```bash -git add docs/tests-function-info/registered_functions.txt docs/tests/python/api/rasterx_functions_sql.py docs/tests/python/api/test_rasterx_functions_sql.py src/main/resources/com/databricks/labs/gbx/function-info.json -git commit -m "docs(function-info): register gbx_rst_h3_rasterize_agg + gbx_h3_cell_bbox" -``` - ---- - -### Task 5: Validation tests — round-trip vs rastertogrid + partition (CI, light) - -**Files:** -- Test: `python/geobrix/test/pyrx/test_h3_rasterize_validate.py` - -**Interfaces:** Consumes `rx.rst_h3_rasterize_agg`, `rx.rst_h3_rastertogridavg` (existing), `cellraster`, sample DEM. - -- [ ] **Step 1: Write the round-trip + partition tests** - -```python -import os -import h3 -import numpy as np - -from databricks.labs.gbx.pyrx import functions as rx -from databricks.labs.gbx.pyrx import _serde -from databricks.labs.gbx.pyrx.core import cellraster as cr - -DEM = os.path.join( - os.environ.get("GBX_SAMPLE_DATA_ROOT", - os.path.join(os.path.dirname(__file__), "../../../../sample-data/Volumes/main/default/geobrix_samples/geobrix-examples")), - "nyc/elevation/srtm_n40w073.tif", -) - - -def test_roundtrip_rastertogrid_then_rasterize(spark): - # DEM -> (cellid, measure) via rastertogridavg -> rasterize back; covered - # pixels' values match the per-cell measures (centroid inverse). - res = 7 - with open(DEM, "rb") as fh: - content = fh.read() - df = spark.createDataFrame([(content,)], ["raster"]).selectExpr( - "gbx_rst_fromcontent(raster, 'GTiff') AS tile" - ) - rx.register(spark) - # rastertogridavg returns array>> (one per band) - cells = df.selectExpr( - "explode(gbx_rst_h3_rastertogridavg(tile, %d)[0]) AS c" % res - ).selectExpr("c.cellID AS cellid", "c.measure AS measure") - cellrows = cells.collect() - assert len(cellrows) > 0 - cv = {int(r["cellid"]): float(r["measure"]) for r in cellrows} - g = cr.compute_gridspec(list(cv.keys()), kring_pad=0) - raster = cr.cells_to_raster(cv, *g, resolution=res) - with _serde.open_tile(raster) as ds: - arr = ds.read(1) - covered = arr[arr != ds.nodata] - # every burned value equals some cell measure (within float tolerance) - measures = np.array(sorted(cv.values())) - assert covered.size > 0 - assert np.isclose(covered, measures[np.searchsorted(measures, covered).clip(0, len(measures) - 1)], atol=1e-6).mean() > 0.99 - - -def test_partition_property_via_agg(spark): - res = 9 - poly = h3.LatLngPoly([(0.0, 0.0), (0.0, 0.03), (0.03, 0.03), (0.03, 0.0)]) - cells = [h3.str_to_int(c) for c in h3.polygon_to_cells(poly, res)] - df = spark.createDataFrame([(int(c), "TX1") for c in cells], ["cellid", "tx"]) - tile = df.groupBy("tx").agg(rx.rst_h3_rasterize_agg("cellid").alias("t")).collect()[0]["t"] - cellset = {cr._h3_str(c) for c in cells} - with _serde.open_tile(bytes(tile["raster"])) as ds: - arr = ds.read(1); t = ds.transform - for row in range(ds.height): - for col in range(ds.width): - lon, lat = t * (col + 0.5, row + 0.5) - assert (arr[row, col] != ds.nodata) == (h3.latlng_to_cell(lat, lon, res) in cellset) -``` - -- [ ] **Step 2: Run (Docker, sample data mounted)** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_h3_rasterize_validate.py` -Expected: 2 passed. (Round-trip needs the registered SQL UDFs; gridagg assumes EPSG:4326 — the SRTM DEM is 4326, so no reprojection.) - -- [ ] **Step 3: Commit** - -```bash -git add python/geobrix/test/pyrx/test_h3_rasterize_validate.py -git commit -m "test(pyrx): H3 rasterize round-trip + partition validation" -``` - ---- - -### Task 6: FCC realistic fixture + test - -**Files:** -- Create: `python/geobrix/test/pyrx/data/fcc_uflw_miamidade_subset.csv` -- Test: `python/geobrix/test/pyrx/test_h3_rasterize_fcc.py` - -**Interfaces:** Consumes `rx.rst_h3_rasterize_agg`, `rx.rst_h3_gridspec`, `rst_frombands_agg`. - -- [ ] **Step 1: Curate the committed subset (one-time, document the command)** - -From the (gitignored) source `input/broadband_wireless/bdc_12_UnlicensedFixedWireless_fixed_broadband_D25_09jun2026.csv`, take one provider, Miami-Dade (`block_geoid` prefix `12086`), a few speed tiers, capped to a few hundred rows: -```bash -docker exec geobrix-dev bash -lc 'cd /root/geobrix && python3 - <=400: break -with open(out,"w",newline="") as f: - w=csv.DictWriter(f, fieldnames=["provider_id","max_advertised_download_speed","h3_res8_id"]); w.writeheader(); w.writerows(rows) -print("wrote", len(rows), "rows") -PY' -``` -(FCC BDC data is public/open — committing a small subset is fine.) - -- [ ] **Step 2: Write the test (rasterize per speed tier, stack, assert coverage)** - -```python -import csv, os -import h3 -from databricks.labs.gbx.pyrx import functions as rx -from databricks.labs.gbx.pyrx import _serde - -CSV = os.path.join(os.path.dirname(__file__), "data/fcc_uflw_miamidade_subset.csv") - - -def test_fcc_rasterize_per_speed_tier(spark): - rows = list(csv.DictReader(open(CSV))) - data = [(h3.str_to_int(r["h3_res8_id"]), int(r["max_advertised_download_speed"]), - r["provider_id"]) for r in rows] - df = spark.createDataFrame(data, ["cellid", "speed", "provider"]) - # one raster per (provider, speed tier); res-8 cells, presence mask - tiles = (df.groupBy("provider", "speed") - .agg(rx.rst_h3_rasterize_agg("cellid").alias("tile")).collect()) - assert len(tiles) >= 1 - for t in tiles: - with _serde.open_tile(bytes(t["tile"]["raster"])) as ds: - assert (ds.read(1) == 1.0).sum() >= 1 # cells burned -``` - -- [ ] **Step 3: Run** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_h3_rasterize_fcc.py` -Expected: 1 passed. - -- [ ] **Step 4: Commit** - -```bash -git add python/geobrix/test/pyrx/data/fcc_uflw_miamidade_subset.csv python/geobrix/test/pyrx/test_h3_rasterize_fcc.py -git commit -m "test(pyrx): FCC fixed-wireless H3 rasterize fixture + test" -``` - ---- - -### Task 7: Heavy `RST_H3_RasterizeAgg` UDAF + `gbx_h3_cell_bbox` + registration - -**Files:** -- Create: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/agg/RST_H3_RasterizeAgg.scala` -- Create: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_H3_CellBBox.scala` (scalar) -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/functions.scala` -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/RST_H3_RasterizeAggTest.scala` - -**Interfaces:** Consumes `H3.cellIdToGeometry`, `H3.pointToCellID`, `H3.kRing` (gridx/grid/H3.scala), `VectorRasterBridge.buildEmptyRaster` / `toGTiffBytes`. Produces SQL `gbx_rst_h3_rasterize_agg` (tile struct) + `gbx_h3_cell_bbox` (bbox struct). - -- [ ] **Step 1: Write the Scala test** - -`RST_H3_RasterizeAggTest.scala` — register functions, build a DataFrame of res-9 cellids (use `H3.pointToCellID` for a few points), `groupBy(...).agg(rst_h3_rasterize_agg($"cellid"))`, assert the returned tile reads as a raster whose covered pixels (centroid → cell) all map into the input set. Mirror `RST_RasterizeAggTest` structure if present, else a fresh suite with a local SparkSession + JAR. - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.RST_H3_RasterizeAggTest'` -Expected: compile failure (class not defined). - -- [ ] **Step 3: Implement the UDAF** (model on `RST_RasterizeAgg.scala:22-208`) - -Accumulate `(cellId: Long, value: Double)` pairs (not WKB) with the same `MAX_BUFFER_BYTES` guard (8 + 8 bytes/row). In `eval`: derive resolution from the first cell (`H3.resolution(cellId)`; error on mixed); when an explicit extent is absent, compute the snapped grid from the cell centroids/envelope + kring_pad (port `cellraster.compute_gridspec`'s snap arithmetic); build the empty raster via `VectorRasterBridge.buildEmptyRaster`; **burn by pixel-centroid** — for each pixel center compute its geo coord (the `RST_H3_RasterToGrid` affine, file `grid/RST_H3_RasterToGrid.scala`), `H3.pointToCellID(lon, lat, res)`, look up the value, write to the band array; return the tile `InternalRow.fromSeq(Seq(0L, bytes, mapData))`. Signature/params mirror the light `rst_h3_rasterize_agg` (cellid, value, srid, pixel_size, xmin..height, mode, kring_pad). - -Implement `RST_H3_CellBBox` scalar returning `struct` from `H3.cellIdToGeometry(cellId).getEnvelopeInternal` (envelope mode) or the cell centroid (centroids mode), reprojected to `srid` via `OSRTransformGeometry`. - -- [ ] **Step 4: Register** in `functions.scala` (after line 83 `rd.register(RST_RasterizeAgg)`): -```scala -rd.register(RST_H3_RasterizeAgg) -rd.register(RST_H3_CellBBox) -``` -plus the imports and the `functions` object `def`s (mirror `rst_rasterize_agg`). Add the two names to `registered_functions.txt` Scala expectations if separate. - -- [ ] **Step 5: Run to verify pass** - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.RST_H3_RasterizeAggTest'` -Expected: PASS. - -- [ ] **Step 6: Lint (scalastyle) + commit** - -```bash -bash scripts/commands/gbx-lint-scalastyle.sh -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/agg/RST_H3_RasterizeAgg.scala src/main/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_H3_CellBBox.scala src/main/scala/com/databricks/labs/gbx/rasterx/functions.scala src/test/scala/com/databricks/labs/gbx/rasterx/RST_H3_RasterizeAggTest.scala -git commit -m "feat(rasterx): RST_H3_RasterizeAgg UDAF + gbx_h3_cell_bbox (heavy tier)" -``` - ---- - -### Task 8: JAR-gated heavy↔light parity test - -**Files:** -- Test: `python/geobrix/test/rasterx/test_h3_rasterize_parity.py` - -**Interfaces:** Consumes heavy `rasterx.functions` (JAR) + light `pyrx` core for the expected cell set. - -- [ ] **Step 1: Write the parity test** - -Register the heavy tier; build a res-9 cell set; rasterize via heavy `gbx_rst_h3_rasterize_agg` on an explicit grid (from light `cellraster.compute_gridspec` so both use the identical canvas); assert the heavy raster's covered-pixel set == the light raster's covered-pixel set (centroid partition is deterministic, so the masks must match exactly). Gate on JAR presence per the `test/rasterx` convention (the suite imports `from databricks.labs.gbx.rasterx import functions as rx` and is skipped without the JAR). - -- [ ] **Step 2: Run (Docker with JAR)** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/rasterx/test_h3_rasterize_parity.py` -Expected: PASS (covered-pixel masks identical between tiers). - -- [ ] **Step 3: Commit** - -```bash -git add python/geobrix/test/rasterx/test_h3_rasterize_parity.py -git commit -m "test(rasterx): heavy<->light H3 rasterize parity (JAR-gated)" -``` - ---- - -### Task 9: Docs — raster-functions.mdx entries - -**Files:** -- Modify: `docs/docs/api/raster-functions.mdx` - -**Interfaces:** none. - -- [ ] **Step 1: Add the Aggregator entry** `rst_h3_rasterize_agg` in the Aggregator Functions section (with ` `, the light-tier BINARY `:::warning` mirroring the other `rst_*_agg` entries, and the `*_sql_example` `` from Task 4). Cross-reference `rst_h3_rastertogrid*` as the inverse and `rst_frombands_agg` for stacking. - -- [ ] **Step 2: Add `gbx_h3_cell_bbox` + a short `rst_h3_gridspec` usage note** (the helper is Python/DataFrame, documented as the way to compute the shared grid for aligned stacking + per-cell `rst_merge_agg` merge). Keep the centroid-vs-spatial_envelope `mode` and `kring_pad` explained. - -- [ ] **Step 3: Verify internals-leak gate + commit** - -Run: `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/api/raster-functions.mdx` (expect none). -```bash -git add docs/docs/api/raster-functions.mdx -git commit -m "docs(rasterx): rst_h3_rasterize_agg + gbx_h3_cell_bbox + rst_h3_gridspec" -``` - ---- - -### Task 10: DEM-isoband notebook example - -**Files:** -- Create: `notebooks/examples/h3-rasterize/h3_rasterize_demo.ipynb`, `notebooks/examples/h3-rasterize/README.md` - -**Interfaces:** none (example). - -- [ ] **Step 1: Build the notebook** — full flow on the sample DEM `srtm_n40w073.tif`: - 1. Quantize the DEM into N filled elevation isobands (`rasterio.features.shapes` on a banded array) → multipolygons over a range of thresholds (elevation bands stand in for signal thresholds). - 2. `h3_polyfill` each band → `(band_level, cellid)`. - 3. `rx.rst_h3_gridspec` over the union of all bands → one shared grid. - 4. `rx.rst_h3_rasterize_agg` per band on the shared grid → aligned band tiles. - 5. `rst_frombands_agg` to stack → multi-band raster; render with `gbx.viz.plot_raster`; show the stack reconstructs the terrain. - Include the telco mapping in prose (band level ↔ signal threshold; this is the same flow as the customer pipeline). - -- [ ] **Step 2: README** — describe the demo + that it validates the rasterize→stack flow with no external data; note it must be run on a cluster/Serverless to refresh outputs. - -- [ ] **Step 3: Commit** (source-level; outputs refreshed on a cluster run) - -```bash -git add notebooks/examples/h3-rasterize/ -git commit -m "docs(notebooks): H3 cell rasterize + stacking demo (DEM isobands)" -``` - ---- - -## Final verification (after all tasks) - -- [ ] Full light suite incl. the new tests in a clean venv from the lock — all pass (`pip install --require-hashes -r requirements-pyrx-ci.txt && pip install --no-deps . && pytest test/pyrx ...`). (No new deps expected — rasterio/h3/numpy/pyproj are already in `[light]`; confirm `pyproj` is in the lock, it is via pyogrio.) -- [ ] `bash scripts/commands/gbx-test-bindings.sh` — binding-parity PASS across Scala + Python + function-info for `gbx_rst_h3_rasterize_agg` and `gbx_h3_cell_bbox`. -- [ ] `bash scripts/commands/gbx-lint-python.sh --check` and `gbx-lint-scalastyle.sh` — clean. -- [ ] Heavy Scala suite + JAR-gated parity green (Docker). -- [ ] Push to `beta/0.4.0`; confirm CI `build main` green on both tiers. diff --git a/docs/superpowers/plans/2026-06-24-vizx-static-map.md b/docs/superpowers/plans/2026-06-24-vizx-static-map.md deleted file mode 100644 index 2e6c23ec1..000000000 --- a/docs/superpowers/plans/2026-06-24-vizx-static-map.md +++ /dev/null @@ -1,914 +0,0 @@ -# VizX static-map helper (`plot_static`) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add `databricks.labs.gbx.vizx.plot_static`, a static (non-interactive) map renderer that draws Spark- or GeoPandas-derived geometries / H3 cells over a contextily basemap, baked into a GitHub-renderable PNG. - -**Architecture:** One new file `vizx/_static_map.py` exposing `plot_static`, built on three private helpers — a pure `_geom_strategy(dtype)` decode-strategy chooser, `_resolve_gdf(...)` (Spark/GeoPandas → EPSG:4326 GeoDataFrame, reusing the shared `parse_geom`), and a `grid_system` dispatch table (h3 implemented; quadbin/bng/custom forward-declared). The renderer reprojects to Web Mercator, plots with GeoPandas, and overlays a contextily basemap inside a try/except that degrades to no-basemap on any failure. - -**Tech Stack:** Python 3.12, geopandas, shapely, matplotlib, contextily, h3, pyspark (local mode in tests). Lock files are uv-compiled with `--generate-hashes`. - -## Global Constraints - -- Module is `databricks.labs.gbx.vizx`; public function name is exactly `plot_static`. No aliases (beta no-aliases policy). -- Requires the `[vizx]` extra; package code calls `assert_viz_available()` before importing matplotlib/geopandas (lazy imports inside functions only). -- Geometry inputs must accept the same encodings as every other `gbx_st_*` function — reuse `databricks.labs.gbx._geom.parse_geom` for the decode (WKB/EWKB/WKT/EWKT). Do not write a second decoder. -- `grid_system` vocabulary is exactly `None | 'h3' | 'quadbin' | 'bng' | 'custom'`. v1 implements `'h3'`; `'quadbin'`, `'bng'`, `'custom'` raise `NotImplementedError` ("planned fast-follow"). -- Basemap uses contextily with default provider `CartoDB.Positron`; any basemap failure (no egress, HTTP error, missing dep) degrades to a `warnings.warn` + render-without-basemap. Never a hard error. -- Supply-chain pinning: the published `[vizx]` extra uses a range pin (`contextily>=1.5,<2`); the execution-env lock files (`requirements-pyrx-ci.txt`, `requirements-dev-container.txt`) pin `contextily` and all transitive deps exact-version + `--hash=sha256` (regenerated via `uv pip compile --generate-hashes`, never hand-edited). -- All Maven/test/lint work runs inside the `geobrix-dev` Docker container. Tests run via `bash scripts/commands/gbx-test-python.sh --path `; lint via `bash scripts/commands/gbx-lint-python.sh --check`. -- Lands on PR #45 (`refactor/vizx-rebrand`). Commit locally; **do not push** (each push triggers CI) — the controller pushes only on the user's explicit go. -- Black/isort/flake8 must pass (CI gate). Use in-container `black` (host black may differ). - ---- - -### Task 1: Add `contextily` dependency + regenerate hash-pinned locks - -**Files:** -- Modify: `python/geobrix/pyproject.toml` (the `vizx = [...]` extra, ~line 133) -- Modify: `python/geobrix/requirements-pyrx-ci.in` (the `[vizx]` visualization block, ~line with `mapclassify==2.10.0`) -- Modify: `python/geobrix/requirements-dev-container.in` (the geospatial dev stack block) -- Regenerate: `python/geobrix/requirements-pyrx-ci.txt`, `python/geobrix/requirements-dev-container.txt` - -**Interfaces:** -- Consumes: nothing. -- Produces: `contextily` importable in the dev container and pinned in both locks. - -- [ ] **Step 1: Add the range pin to the published extra** - -In `python/geobrix/pyproject.toml`, the `vizx` extra becomes: - -```toml -vizx = [ - "matplotlib>=3.7,<4", - "geopandas>=1.0,<2", - "folium>=0.16,<1", - "mapclassify>=2.6,<3", - "contextily>=1.5,<2", -] -``` - -- [ ] **Step 2: Add the exact pin to both `.in` source files** - -In `python/geobrix/requirements-pyrx-ci.in`, inside the `# --- visualization ([vizx] extra) ...` block, append after `mapclassify==2.10.0`: - -``` -contextily==1.6.2 -``` - -In `python/geobrix/requirements-dev-container.in`, inside the `# --- geospatial dev stack (not in DBR) ---` block, append after `pyproj==3.7.2`: - -``` -contextily==1.6.2 -``` - -(If the corp PyPI proxy does not have `1.6.2`, use the latest available `1.x` and keep the `>=1.5,<2` range in the extra consistent.) - -- [ ] **Step 3: Regenerate both hash-pinned locks (in the container, proxy available)** - -Run: - -```bash -bash scripts/commands/gbx-docker-exec.sh "cd /root/geobrix/python/geobrix && uv pip compile --generate-hashes --python-version 3.12 --output-file requirements-pyrx-ci.txt requirements-pyrx-ci.in && uv pip compile --generate-hashes --python-version 3.12 --output-file requirements-dev-container.txt requirements-dev-container.in" -``` - -Expected: both `.txt` files now contain a hash-pinned `contextily==1.6.2 \` block plus any new transitive deps (e.g. `xyzservices`, `mercantile`, `geographiclib`/`geopy`, `joblib`), each with `--hash=sha256:` lines. - -- [ ] **Step 4: Verify `contextily` is pinned + transitive deps captured** - -Run: - -```bash -grep -n "^contextily==" python/geobrix/requirements-pyrx-ci.txt python/geobrix/requirements-dev-container.txt -``` - -Expected: one match in each file, immediately followed by `--hash=sha256:` lines. - -- [ ] **Step 5: Install into the running dev container so test tasks can import it** - -Run: - -```bash -bash scripts/commands/gbx-docker-exec.sh "pip install --require-hashes -r /root/geobrix/python/geobrix/requirements-dev-container.txt && python -c 'import contextily; print(contextily.__version__)'" -``` - -Expected: the install succeeds (clean-resolve from the hashed lock) and prints `1.6.2`. - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/pyproject.toml python/geobrix/requirements-pyrx-ci.in python/geobrix/requirements-dev-container.in python/geobrix/requirements-pyrx-ci.txt python/geobrix/requirements-dev-container.txt -git commit -m "build(vizx): add contextily dep, hash-pin in CI + dev-container locks" -``` - ---- - -### Task 2: `_static_map.py` — geometry-path resolution (`grid_system=None`) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py` -- Test: `python/geobrix/test/vizx/test_static_map.py` - -**Interfaces:** -- Consumes: `databricks.labs.gbx._geom.parse_geom(x) -> shapely geometry | None`. -- Produces: - - `_geom_strategy(dtype) -> 'native' | 'binary' | 'string'` (raises `ValueError` otherwise); `dtype` is a `pyspark.sql.types.DataType`. - - `_detect_geom_col(df, grid_system) -> str`. - - `_collect_limited(df, max_rows) -> pandas.DataFrame` (truncate-and-warn). - - `_resolve_gdf(data, geom_col, grid_system, max_rows, srid) -> geopandas.GeoDataFrame` (EPSG:4326-or-`srid`). For this task only the `grid_system is None` and GeoDataFrame-passthrough paths are live; the cell path is added in Task 3. - -- [ ] **Step 1: Write the failing tests** - -Create `python/geobrix/test/vizx/test_static_map.py`: - -```python -import logging -import warnings - -import pytest -from pyspark.sql.types import BinaryType, LongType, StringType - - -@pytest.fixture(scope="module") -def spark(): - logging.getLogger("py4j").setLevel(logging.ERROR) - from pyspark.sql import SparkSession - - s = ( - SparkSession.builder.master("local[2]") - .appName("viz-static-map-tests") - .getOrCreate() - ) - yield s - - -# --- _geom_strategy (pure, no Spark) --- - - -def test_geom_strategy_string_binary_native_and_error(): - from databricks.labs.gbx.vizx import _static_map as sm - - assert sm._geom_strategy(StringType()) == "string" - assert sm._geom_strategy(BinaryType()) == "binary" - assert sm._geom_strategy(LongType()) is None or True # placeholder; replaced below - - -def test_geom_strategy_rejects_unsupported(): - from databricks.labs.gbx.vizx import _static_map as sm - - with pytest.raises(ValueError): - sm._geom_strategy(LongType()) - - -class _FakeGeoType: - # mimics a Databricks GEOMETRY/GEOGRAPHY dataType for routing tests - def __init__(self, name): - self._name = name - - def typeName(self): - return self._name - - def simpleString(self): - return self._name - - -def test_geom_strategy_native_for_geometry_and_geography(): - from databricks.labs.gbx.vizx import _static_map as sm - - assert sm._geom_strategy(_FakeGeoType("geometry")) == "native" - assert sm._geom_strategy(_FakeGeoType("geography")) == "native" - - -# --- _resolve_gdf geometry path --- - - -def test_resolve_gdf_wkt_string(spark): - from databricks.labs.gbx.vizx import _static_map as sm - - df = spark.createDataFrame( - [("a", "POINT (1 2)"), ("b", "POINT (3 4)")], ["name", "wkt"] - ) - gdf = sm._resolve_gdf(df, None, None, 10_000, None) - assert gdf.crs.to_epsg() == 4326 - assert list(gdf["name"]) == ["a", "b"] - assert "wkt" not in gdf.columns - assert [g.x for g in gdf.geometry] == [1.0, 3.0] - - -def test_resolve_gdf_wkb_matches_wkt(spark): - import shapely - - from databricks.labs.gbx.vizx import _static_map as sm - - wkb = bytearray(shapely.to_wkb(shapely.from_wkt("POINT (5 6)"))) - df = spark.createDataFrame([(wkb,)], ["geometry"]) - gdf = sm._resolve_gdf(df, None, None, 10_000, None) - assert (gdf.geometry.iloc[0].x, gdf.geometry.iloc[0].y) == (5.0, 6.0) - - -def test_resolve_gdf_passes_through_geodataframe(): - import geopandas as gpd - from shapely.geometry import Point - - from databricks.labs.gbx.vizx import _static_map as sm - - g = gpd.GeoDataFrame({"v": [1]}, geometry=[Point(0, 0)], crs=4326) - assert sm._resolve_gdf(g, None, None, 10_000, None) is g - - -def test_resolve_gdf_unknown_column_type_raises(spark): - from databricks.labs.gbx.vizx import _static_map as sm - - df = spark.createDataFrame([(1,)], ["geometry"]) # LongType, no grid_system - with pytest.raises(ValueError): - sm._resolve_gdf(df, None, None, 10_000, None) - - -def test_resolve_gdf_truncates_and_warns(spark): - from databricks.labs.gbx.vizx import _static_map as sm - - df = spark.range(5).selectExpr("concat('POINT (', id, ' 0)') AS wkt") - with warnings.catch_warnings(record=True) as caught: - warnings.simplefilter("always") - gdf = sm._resolve_gdf(df, None, None, 2, None) - assert len(gdf) == 2 - assert any("max_rows" in str(w.message) for w in caught) -``` - -Delete the placeholder line in `test_geom_strategy_string_binary_native_and_error` (the `LongType() is None or True`); keep only the `StringType`/`BinaryType` asserts there: - -```python -def test_geom_strategy_string_binary_native_and_error(): - from databricks.labs.gbx.vizx import _static_map as sm - - assert sm._geom_strategy(StringType()) == "string" - assert sm._geom_strategy(BinaryType()) == "binary" -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_static_map.py` -Expected: FAIL — `ModuleNotFoundError: ... _static_map` / `AttributeError: _geom_strategy`. - -- [ ] **Step 3: Implement the geometry path** - -Create `python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py`: - -```python -"""Static (non-interactive) map rendering for gbx.vizx. - -plot_static renders Spark- or GeoPandas-derived geometries / DGGS cells over a -contextily basemap as a static matplotlib figure -- the GitHub-renderable -counterpart to GeoDataFrame.explore(). Requires the [vizx] extra. -""" - -import warnings - -_GEOM_COL_CANDIDATES = ("wkt", "geometry", "geom", "ewkt", "wkb", "ewkb") -_CELL_COL_CANDIDATES = ("cellid", "cell", "cell_id", "h3", "quadbin", "bng", "index") - - -def _geom_strategy(dtype): - """Decode strategy for a Spark geometry column's dataType. - - Returns 'native' (Databricks GEOMETRY/GEOGRAPHY -> st_asbinary in Spark), - 'binary' (WKB/EWKB), or 'string' (WKT/EWKT). Raises ValueError otherwise. - """ - name = dtype.typeName().lower() - simple = dtype.simpleString().lower() - if "geometry" in name or "geography" in name or "geometry" in simple or "geography" in simple: - return "native" - if name == "binary": - return "binary" - if name == "string": - return "string" - raise ValueError( - f"plot_static: geometry column has unsupported type {dtype.simpleString()!r}; " - "coerce it to WKB/WKT first (e.g. st_asbinary(col) / st_astext(col)), or " - "pass grid_system= for DGGS cell ids." - ) - - -def _detect_geom_col(df, grid_system): - """Auto-detect the geometry/cell column name. Raise ValueError if ambiguous.""" - cols = df.columns - lower = {c.lower(): c for c in cols} - if grid_system is not None: - for cand in _CELL_COL_CANDIDATES: - if cand in lower: - return lower[cand] - if len(cols) == 1: - return cols[0] - raise ValueError( - "plot_static: could not auto-detect the cell-id column; pass " - f"geom_col= explicitly (columns: {cols})." - ) - for f in df.schema.fields: - s = f.dataType.simpleString().lower() - if "geometry" in s or "geography" in s: - return f.name - for cand in _GEOM_COL_CANDIDATES: - if cand in lower: - return lower[cand] - raise ValueError( - "plot_static: could not auto-detect the geometry column; pass geom_col= " - f"explicitly (columns: {cols})." - ) - - -def _collect_limited(df, max_rows): - """Collect a Spark DataFrame to pandas with a truncate-and-warn row guard.""" - if max_rows is None: - return df.toPandas() - pdf = df.limit(max_rows + 1).toPandas() - if len(pdf) > max_rows: - pdf = pdf.iloc[:max_rows] - warnings.warn( - f"plot_static: output truncated to max_rows={max_rows} for driver-side " - "viz; pass max_rows=None to collect all rows.", - stacklevel=2, - ) - return pdf - - -def _resolve_gdf(data, geom_col, grid_system, max_rows, srid): - """Spark DataFrame or GeoDataFrame -> geopandas.GeoDataFrame (EPSG:4326 or srid).""" - import geopandas as gpd - - if isinstance(data, gpd.GeoDataFrame): - return data - - col = geom_col or _detect_geom_col(data, grid_system) - - if grid_system is not None: - return _resolve_cells(data, col, grid_system, max_rows) # added in Task 3 - - from databricks.labs.gbx._geom import parse_geom - - field = data.schema[col] - strategy = _geom_strategy(field.dataType) - work = data - if strategy == "native": - from pyspark.sql.functions import expr - - work = data.withColumn(col, expr(f"st_asbinary(`{col}`)")) - if srid is None and "geography" in field.dataType.simpleString().lower(): - srid = 4326 - - pdf = _collect_limited(work, max_rows) - geoms = [parse_geom(v) for v in pdf[col]] - pdf = pdf.drop(columns=[col]) - return gpd.GeoDataFrame(pdf, geometry=geoms, crs=(srid or 4326)) -``` - -Note: `_resolve_cells` is referenced but not yet defined — Task 2's tests never set `grid_system`, so that branch is not exercised here. Task 3 adds `_resolve_cells`. - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_static_map.py` -Expected: PASS (all Task-2 tests green). - -- [ ] **Step 5: Lint** - -Run: `bash scripts/commands/gbx-docker-exec.sh "cd /root/geobrix && black python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py python/geobrix/test/vizx/test_static_map.py && isort python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py python/geobrix/test/vizx/test_static_map.py"` -Then: `bash scripts/commands/gbx-lint-python.sh --check` -Expected: lint passes. - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py python/geobrix/test/vizx/test_static_map.py -git commit -m "feat(vizx): _static_map geometry-path resolution (parse_geom reuse)" -``` - ---- - -### Task 3: `grid_system` cell dispatch (h3 implemented; quadbin/bng/custom NYI) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py` -- Test: `python/geobrix/test/vizx/test_static_map.py` - -**Interfaces:** -- Consumes: `_collect_limited`, `_detect_geom_col` (Task 2). -- Produces: `_resolve_cells(data, col, grid_system, max_rows) -> geopandas.GeoDataFrame` and module-level `_GRID_DISPATCH` dict; the `'h3'` resolver accepts string h3 indices and long bigints. - -- [ ] **Step 1: Write the failing tests** - -Append to `python/geobrix/test/vizx/test_static_map.py`: - -```python -def _ny_hex_string(): - import h3 - - return h3.latlng_to_cell(40.7, -74.0, 9) # string h3 index - - -def test_resolve_cells_h3_string_and_long_match(spark): - import h3 - - from databricks.labs.gbx.vizx import _static_map as sm - - s = _ny_hex_string() - as_long = h3.str_to_int(s) - - df_str = spark.createDataFrame([(s,)], ["cellid"]) - df_long = spark.createDataFrame([(as_long,)], ["cellid"]) - - g_str = sm._resolve_gdf(df_str, None, "h3", 10_000, None) - g_long = sm._resolve_gdf(df_long, None, "h3", 10_000, None) - - assert g_str.crs.to_epsg() == 4326 - # identical boundary polygon from either id form - assert g_str.geometry.iloc[0].equals(g_long.geometry.iloc[0]) - - -def test_resolve_cells_carries_attribute_columns(spark): - from databricks.labs.gbx.vizx import _static_map as sm - - s = _ny_hex_string() - df = spark.createDataFrame([(s, 7)], ["cellid", "count"]) - gdf = sm._resolve_gdf(df, "cellid", "h3", 10_000, None) - assert list(gdf["count"]) == [7] - assert "cellid" not in gdf.columns - - -@pytest.mark.parametrize("gs", ["quadbin", "bng", "custom"]) -def test_resolve_cells_fast_follow_not_implemented(spark, gs): - from databricks.labs.gbx.vizx import _static_map as sm - - df = spark.createDataFrame([(1,)], ["cellid"]) - with pytest.raises(NotImplementedError): - sm._resolve_gdf(df, "cellid", gs, 10_000, None) - - -def test_resolve_cells_unknown_grid_system_raises(spark): - from databricks.labs.gbx.vizx import _static_map as sm - - df = spark.createDataFrame([(1,)], ["cellid"]) - with pytest.raises(ValueError): - sm._resolve_gdf(df, "cellid", "geohash", 10_000, None) -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_static_map.py` -Expected: FAIL — `NameError: _resolve_cells` / `AttributeError`. - -- [ ] **Step 3: Implement the cell dispatch** - -Append to `python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py` (after `_collect_limited`, before `_resolve_gdf`): - -```python -def _h3_boundary(cell): - import h3 - from shapely.geometry import Polygon - - idx = cell if isinstance(cell, str) else h3.int_to_str(int(cell)) - ring = h3.cell_to_boundary(idx) # (lat, lng) pairs in h3 v4 - return Polygon([(lng, lat) for lat, lng in ring]) - - -def _h3_boundaries(values): - return [_h3_boundary(c) for c in values] - - -def _nyi(name): - def _raise(_values): - raise NotImplementedError( - f"plot_static: grid_system={name!r} is a planned fast-follow; " - "not supported yet." - ) - - return _raise - - -_GRID_DISPATCH = { - "h3": _h3_boundaries, - "quadbin": _nyi("quadbin"), - "bng": _nyi("bng"), - "custom": _nyi("custom"), -} - - -def _resolve_cells(data, col, grid_system, max_rows): - """DGGS cell-id column -> boundary-polygon GeoDataFrame (EPSG:4326).""" - import geopandas as gpd - - if grid_system not in _GRID_DISPATCH: - raise ValueError( - f"plot_static: grid_system={grid_system!r} is not one of " - f"{sorted(_GRID_DISPATCH)} or None." - ) - pdf = _collect_limited(data, max_rows) - geometry = _GRID_DISPATCH[grid_system](pdf[col].tolist()) - return gpd.GeoDataFrame(pdf.drop(columns=[col]), geometry=geometry, crs=4326) -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_static_map.py` -Expected: PASS (all Task-2 and Task-3 tests green). - -- [ ] **Step 5: Lint** - -Run: `bash scripts/commands/gbx-docker-exec.sh "cd /root/geobrix && black python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py python/geobrix/test/vizx/test_static_map.py && isort python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py python/geobrix/test/vizx/test_static_map.py"` -Then: `bash scripts/commands/gbx-lint-python.sh --check` -Expected: lint passes. - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py python/geobrix/test/vizx/test_static_map.py -git commit -m "feat(vizx): plot_static h3 cell dispatch; quadbin/bng/custom forward-declared" -``` - ---- - -### Task 4: `plot_static` public renderer (basemap + fallback + overlay) and export - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/__init__.py` -- Test: `python/geobrix/test/vizx/test_static_map.py` - -**Interfaces:** -- Consumes: `_resolve_gdf` (Tasks 2-3), `assert_viz_available` from `vizx/_env.py`. -- Produces: `plot_static(data, *, geom_col=None, grid_system=None, column=None, cmap="viridis", legend=True, basemap=True, basemap_source=None, alpha=0.8, edgecolor="face", markersize=None, title=None, fig_w=10, fig_h=10, max_rows=10_000, srid=None, ax=None) -> matplotlib.axes.Axes`. Exported from `databricks.labs.gbx.vizx`. - -- [ ] **Step 1: Write the failing tests** - -Append to `python/geobrix/test/vizx/test_static_map.py` (and add the headless backend lines at the very top of the file, mirroring `test_raster.py`): - -At the **top** of the file, immediately after `import warnings`, add: - -```python -import matplotlib - -matplotlib.use("Agg") # headless: no display needed -import matplotlib.pyplot as plt # noqa: E402 -``` - -New tests appended at the end: - -```python -def test_plot_static_returns_axes_and_one_figure(spark): - from databricks.labs.gbx.vizx import plot_static - - plt.close("all") - df = spark.createDataFrame([("POINT (1 2)",)], ["wkt"]) - ax = plot_static(df, basemap=False) - assert ax is not None - assert len(plt.get_fignums()) == 1 - plt.close("all") - - -def test_plot_static_choropleth_column_with_legend(spark): - from databricks.labs.gbx.vizx import plot_static - - plt.close("all") - df = spark.createDataFrame( - [("POLYGON ((0 0, 1 0, 1 1, 0 1, 0 0))", 3)], ["wkt", "v"] - ) - ax = plot_static(df, column="v", basemap=False) - assert ax.get_figure() is not None - plt.close("all") - - -def test_plot_static_overlay_reuses_axes(spark): - from databricks.labs.gbx.vizx import plot_static - - plt.close("all") - df1 = spark.createDataFrame([("POINT (1 1)",)], ["wkt"]) - df2 = spark.createDataFrame([("POINT (2 2)",)], ["wkt"]) - ax = plot_static(df1, basemap=False) - ax2 = plot_static(df2, basemap=False, ax=ax) - assert ax2 is ax - assert len(plt.get_fignums()) == 1 # no new figure created for the overlay - plt.close("all") - - -def test_plot_static_basemap_fallback_warns(spark, monkeypatch): - import contextily - - from databricks.labs.gbx.vizx import plot_static - - def _boom(*a, **k): - raise RuntimeError("no egress") - - monkeypatch.setattr(contextily, "add_basemap", _boom) - plt.close("all") - df = spark.createDataFrame([("POINT (1 2)",)], ["wkt"]) - with warnings.catch_warnings(record=True) as caught: - warnings.simplefilter("always") - ax = plot_static(df, basemap=True) - assert ax is not None - assert len(plt.get_fignums()) == 1 # figure still produced - assert any("basemap unavailable" in str(w.message) for w in caught) - plt.close("all") -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_static_map.py` -Expected: FAIL — `ImportError: cannot import name 'plot_static'`. - -- [ ] **Step 3: Implement `plot_static`** - -Append to `python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py` (at the end): - -```python -def plot_static( - data, - *, - geom_col=None, - grid_system=None, - column=None, - cmap="viridis", - legend=True, - basemap=True, - basemap_source=None, - alpha=0.8, - edgecolor="face", - markersize=None, - title=None, - fig_w=10, - fig_h=10, - max_rows=10_000, - srid=None, - ax=None, -): - """Render geometries / DGGS cells over a basemap as a static figure. - - ``data`` is a Spark DataFrame or a geopandas.GeoDataFrame. Geometry columns - accept WKT/EWKT/WKB/EWKB and native GEOMETRY/GEOGRAPHY (decoded via the - shared parse_geom); set ``grid_system`` ('h3' in v1) to treat the column as - DGGS cell ids (string or long). The contextily basemap is rendered when - ``basemap=True``; any failure (no egress / missing dep) degrades to a - warning and a basemap-less render. Returns the matplotlib Axes; pass it back - via ``ax=`` to overlay layers. Requires the [vizx] extra. - """ - from databricks.labs.gbx.vizx._env import assert_viz_available - - assert_viz_available() - - import matplotlib.pyplot as plt - - gdf = _resolve_gdf(data, geom_col, grid_system, max_rows, srid) - - created = ax is None - if created: - _, ax = plt.subplots(1, figsize=(fig_w, fig_h)) - - plot_gdf = gdf.to_crs(3857) if basemap else gdf - - kwargs = {"ax": ax, "alpha": alpha, "edgecolor": edgecolor, "cmap": cmap} - if column is not None: - kwargs["column"] = column - kwargs["legend"] = legend - if markersize is not None: - kwargs["markersize"] = markersize - plot_gdf.plot(**kwargs) - - if basemap: - try: - import contextily as cx - - source = basemap_source or cx.providers.CartoDB.Positron - cx.add_basemap(ax, source=source, crs=plot_gdf.crs) - except Exception as exc: # noqa: BLE001 — offline/no-egress/missing -> fallback - warnings.warn( - f"plot_static: basemap unavailable ({type(exc).__name__}: {exc}); " - "rendering without basemap. Ensure network egress to the tile " - "server at execution time for the basemap to bake into the output.", - stacklevel=2, - ) - - if title: - ax.set_title(title) - ax.set_axis_off() - - if created: - plt.show() - return ax -``` - -- [ ] **Step 4: Export from the package** - -Edit `python/geobrix/src/databricks/labs/gbx/vizx/__init__.py`: - -```python -from databricks.labs.gbx.vizx._raster import plot_file, plot_mask_layers, plot_raster -from databricks.labs.gbx.vizx._static_map import plot_static -from databricks.labs.gbx.vizx._vector import as_gdf, cells_as_gdf, grid_as_gdf - -__all__ = [ - "plot_raster", - "plot_file", - "plot_mask_layers", - "plot_static", - "as_gdf", - "cells_as_gdf", - "grid_as_gdf", -] -``` - -- [ ] **Step 5: Run tests to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_static_map.py` -Expected: PASS (full file green). - -- [ ] **Step 6: Run the whole vizx suite (no regressions)** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/` -Expected: PASS. - -- [ ] **Step 7: Lint** - -Run: `bash scripts/commands/gbx-docker-exec.sh "cd /root/geobrix && black python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py python/geobrix/src/databricks/labs/gbx/vizx/__init__.py python/geobrix/test/vizx/test_static_map.py && isort python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py python/geobrix/src/databricks/labs/gbx/vizx/__init__.py python/geobrix/test/vizx/test_static_map.py"` -Then: `bash scripts/commands/gbx-lint-python.sh --check` -Expected: lint passes. - -- [ ] **Step 8: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py python/geobrix/src/databricks/labs/gbx/vizx/__init__.py python/geobrix/test/vizx/test_static_map.py -git commit -m "feat(vizx): plot_static renderer with contextily basemap + fallback" -``` - ---- - -### Task 5: Document `plot_static` in `vizx.mdx` - -**Files:** -- Modify: `docs/docs/api/vizx.mdx` - -**Interfaces:** -- Consumes: the `plot_static` signature from Task 4. Produces: no code. - -- [ ] **Step 1: Add the `plot_static` section** - -In `docs/docs/api/vizx.mdx`, after the `### grid_as_gdf` block (ends near the worked-example note before `## Escape hatches`), insert a new top-level section: - -````markdown -## Static maps - -`plot_static` renders Spark- or GeoPandas-derived geometries (or H3 cells) over -a basemap as a **static** matplotlib figure — the GitHub-renderable counterpart -to `GeoDataFrame.explore()` (whose Leaflet/folium output renders a blank -*"Make this Notebook Trusted"* placeholder on GitHub and the docs site). - -The basemap is fetched from a web tile server (via `contextily`) **at execution -time** and rasterized into the figure, so it bakes into the committed notebook -output PNG — GitHub then displays it with no network. If the executing -environment has no egress, the map renders without a basemap and a warning is -emitted (never a hard error). - -### `plot_static` - -```python -plot_static( - data, *, geom_col=None, grid_system=None, column=None, cmap="viridis", - legend=True, basemap=True, basemap_source=None, alpha=0.8, edgecolor="face", - markersize=None, title=None, fig_w=10, fig_h=10, max_rows=10_000, - srid=None, ax=None, -) -``` - -`data` is a Spark DataFrame **or** a `geopandas.GeoDataFrame`. Returns the -matplotlib `Axes`; pass it back via `ax=` to overlay layers on one map. - -**Geometry columns** accept the same encodings as every other `gbx_st_*` -function — WKT, EWKT, WKB, EWKB, and native `GEOMETRY` / `GEOGRAPHY` (coerced -in-Spark via `st_asbinary`). Set **`grid_system`** to treat the column as DGGS -cell ids instead: - -| `grid_system` | Behaviour | -|---|---| -| `None` (default) | Column is a geometry encoding (WKT/EWKT/WKB/EWKB/`GEOMETRY`/`GEOGRAPHY`). | -| `'h3'` | Column holds H3 cell ids (string index **or** bigint); rendered as cell-boundary polygons. | -| `'quadbin'`, `'bng'`, `'custom'` | Planned — currently raise `NotImplementedError`. | - -```python -from databricks.labs.gbx.vizx import plot_static - -# H3 choropleth over a basemap, then overlay the shared-canvas boundary: -ax = plot_static(cells_df, grid_system="h3", column="count", title="Coverage") -plot_static(grid_boundary_df, basemap=False, ax=ax, edgecolor="red") -``` - -`basemap_source` overrides the default `contextily.providers.CartoDB.Positron`; -`basemap=False` skips tiles entirely (deterministic, no network). -```` - -- [ ] **Step 2: Verify no internal-vocabulary leak (QC gate)** - -Run: `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/api/vizx.mdx` -Expected: no output. - -- [ ] **Step 3: Commit** - -```bash -git add docs/docs/api/vizx.mdx -git commit -m "docs(vizx): document plot_static static-map helper" -``` - ---- - -### Task 6: Adopt `plot_static` in the h3-rasterize notebook - -**Files:** -- Modify: `notebooks/examples/h3-rasterize/h3_rasterize_isobands.ipynb` - -**Interfaces:** -- Consumes: `plot_static` (Task 4). Produces: no code (notebook edit only). - -**Note:** the deliverable is the edited **code cells**. The committed output PNGs (with a baked basemap) are refreshed when the user re-executes the notebook on a cluster with egress — not in this task. Do not attempt to execute the notebook here. - -- [ ] **Step 1: Locate the static-plot cell** - -Run: `python3 -c "import json; nb=json.load(open('notebooks/examples/h3-rasterize/h3_rasterize_isobands.ipynb')); [print(i, repr(''.join(c['source'])[:120])) for i,c in enumerate(nb['cells']) if c['cell_type']=='code' and 'bands_gdf.plot' in ''.join(c['source'])]"` -Expected: prints the index of the cell containing `bands_gdf.plot(column="band_level" ...)` + `grid_gdf.boundary.plot(...)`. - -- [ ] **Step 2: Replace the plot cell body with `plot_static`** - -Using a small Python script (so JSON stays valid), set that code cell's source to: - -```python -# Static map: H3 isobands as a choropleth over a basemap, with the shared -# canvas boundary overlaid. Renders on GitHub (static PNG). For an interactive -# pan/zoom version in Databricks, use bands_gdf.explore(...) / grid_gdf.explore(m=...). -from databricks.labs.gbx.vizx import plot_static - -ax = plot_static( - bands_gdf, - column="band_level", - cmap="viridis", - title="DEM isobands (H3)", -) -plot_static(grid_gdf, basemap=False, ax=ax, edgecolor="red", alpha=1.0) -``` - -Here `bands_gdf` / `grid_gdf` are the existing GeoDataFrames already built earlier in the notebook (from `cells_as_gdf` / `grid_as_gdf`). If they are still Spark DataFrames at that point, pass the Spark frames with `grid_system="h3"` instead — verify which by reading the cell that defines `bands_gdf`. - -Use this script form to edit (preserves notebook JSON): - -```bash -python3 - <<'PY' -import json -f = "notebooks/examples/h3-rasterize/h3_rasterize_isobands.ipynb" -nb = json.load(open(f)) -NEW = '''# Static map: H3 isobands as a choropleth over a basemap, with the shared -# canvas boundary overlaid. Renders on GitHub (static PNG). For an interactive -# pan/zoom version in Databricks, use bands_gdf.explore(...) / grid_gdf.explore(m=...). -from databricks.labs.gbx.vizx import plot_static - -ax = plot_static( - bands_gdf, - column="band_level", - cmap="viridis", - title="DEM isobands (H3)", -) -plot_static(grid_gdf, basemap=False, ax=ax, edgecolor="red", alpha=1.0) -''' -IDX = None # set to the cell index found in Step 1 -assert IDX is not None, "set IDX from Step 1" -nb["cells"][IDX]["source"] = NEW.splitlines(keepends=True) -nb["cells"][IDX]["outputs"] = [] -nb["cells"][IDX]["execution_count"] = None -json.dump(nb, open(f, "w"), indent=1) -open(f, "a").write("\n") -print("updated cell", IDX) -PY -``` - -- [ ] **Step 3: Verify the notebook still parses + the static-render note still holds** - -Run: `python3 -c "import json; json.load(open('notebooks/examples/h3-rasterize/h3_rasterize_isobands.ipynb')); print('ok')"` -Expected: `ok`. - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/` (sanity: code import path unchanged) -Expected: PASS. - -- [ ] **Step 4: Commit** - -```bash -git add "notebooks/examples/h3-rasterize/h3_rasterize_isobands.ipynb" -git commit -m "docs(notebooks): adopt vizx.plot_static in h3-rasterize map cell" -``` - ---- - -## Out of scope / fast-follow - -- `grid_system` `'quadbin'`, `'bng'`, `'custom'` cell→boundary resolvers (the dispatch seam is in place; each is one entry in `_GRID_DISPATCH` plus a resolver + tests). -- eo-series 01/03 cell-map adoption (optional; mirror Task 6 once h3-rasterize is validated by a real re-run). -- An offline/committed tile cache for no-egress executors (explicitly out of v1 per the spec). - -## Self-Review - -**Spec coverage:** plot_static signature + return-Axes-overlay (Task 4) ✓; Spark+GeoPandas input (Tasks 2,4) ✓; geometry encodings via parse_geom incl. native GEOMETRY/GEOGRAPHY (Task 2) ✓; grid_system h3 + forward-declared NYI (Task 3) ✓; contextily basemap + graceful fallback + 3857 reproject + CartoDB.Positron default (Task 4) ✓; contextily in [vizx] range pin + exact+hash locks both files (Task 1) ✓; tests enumerated in the spec all map to Task 2/3/4 tests ✓; docs (Task 5) ✓; notebook adoption (Task 6) ✓; PR #45 + no-push (Global Constraints) ✓. - -**Placeholder scan:** none — every code step has complete code; the only deliberate IDX placeholder in Task 6 Step 2 is guarded by an `assert IDX is not None` and instructed in Step 1. - -**Type consistency:** `_resolve_gdf(data, geom_col, grid_system, max_rows, srid)` signature identical across Tasks 2-4; `_resolve_cells` / `_GRID_DISPATCH` names consistent Task 3↔used nowhere else; `plot_static` keyword names identical between Task 4 impl, tests, docs (Task 5), and notebook usage (Task 6). diff --git a/docs/superpowers/plans/2026-06-25-vector-writer-column-options.md b/docs/superpowers/plans/2026-06-25-vector-writer-column-options.md deleted file mode 100644 index 7365d7619..000000000 --- a/docs/superpowers/plans/2026-06-25-vector-writer-column-options.md +++ /dev/null @@ -1,620 +0,0 @@ -# Vector Writer Column Options (`geomCol`/`sridCol`/`projCol`) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Let users point the vector writers at their existing geometry/SRID/proj columns by name (`geomCol`/`sridCol`/`projCol`) instead of renaming columns to the `X`/`X_srid`/`X_srid_proj` convention — in both the light tier (all five `*_gbx` writers) and the heavy tier (`geojsonl` only). - -**Architecture:** Generalize the single role-derivation function in each tier to accept optional column-name overrides, then thread the three options from each writer's option map into it. Add a per-format output geometry name for the light writers so an arbitrary input column name doesn't leak into the output file. - -**Tech Stack:** Python (PySpark DataSource V2, pyogrio), Scala (Spark DataSource V2, GDAL/OGR JNI). Heavy work builds + tests in the `geobrix-dev` Docker container via Maven. - -## Global Constraints - -- Option names are exactly `geomCol`, `sridCol`, `projCol`; parsed case-insensitively (light writers already lowercase the options dict; heavy `GeoJSONL_RowWriter` already builds a lowercased `ciOptions`). -- Resolution rule: each option, if given, must name an **existing** column (clear error otherwise); if omitted it falls back to its convention name. **geom** required; **srid** required (option or `_srid`; clear error if unresolvable); **proj** optional. -- No options passed → behavior identical to today (backward compatible). -- Identical option names + semantics across light and heavy; the existing heavy↔light `geojsonl` round-trip must keep holding. -- Light tier: all five writers (`geojson_gbx`, `geojsonl_gbx`, `gpkg_gbx`, `shapefile_gbx`, `file_gdb_gbx`) via the shared `_writer_col_roles`. Heavy tier: `geojsonl` only (the other heavy OGR formats are read-only). -- Output geometry name per format (light): `GPKG` → `geom`, `OpenFileGDB` → `SHAPE`; `GeoJSON`/`GeoJSONSeq`/`ESRI Shapefile` geometry is structural (the name is inert). -- All Python test/lint runs in the `geobrix-dev` container: tests `bash scripts/commands/gbx-test-python.sh --path

/x.shp")` (bare `.shp`) reads N rows (currently throws `Unable to open x.shx`). -- [ ] **Step 2: Run → FAIL.** -- [ ] **Step 3: Implement** — when the resolved data path is a single `.shp` (sidecar-bundle primary), discover stem-siblings from the `.shp`'s **parent directory** (Hadoop FS list of the parent, filter by stem) and stage them alongside in `stageHeadForSchemaSpark`; the `OGR_Batch` read path already co-copies stem-siblings via `copyToPath`, so confirm it covers the bare-`.shp` partition too. -- [ ] **Step 4: Run → PASS** (in Docker w/ Volumes or local fixture). -- [ ] **Step 5: Commit.** - -### Task B4: Heavy — schema-divergence error (shared message) - -**Files:** Modify `OGR_Batch.scala` / `OGR_DataSource.scala` + `HadoopUtils.scala` (shared message constant). Test: `OgrReaderContractTest.scala`. - -- [ ] **Step 1: Failing test** — a dir with two differing-schema shapefiles raises the shared message (same wording as light). -- [ ] **Step 2: Run → FAIL.** -- [ ] **Step 3: Implement** — during planning, when multiple `.shp` stems are present, compare each `.shp`'s inferred schema to the head's; on divergence throw `IllegalArgumentException` with the shared message. (Single stem / single file unchanged.) -- [ ] **Step 4: Run → PASS.** -- [ ] **Step 5: Commit.** - -### Task B5: Reader contract docs + cross-tier parity note - -**Files:** Modify `docs/docs/readers/*.mdx` (shapefile reader page). Test: doc-test if applicable. - -- [ ] Document the `.load()` contract (single `.shp`; recursive same-schema dir; `.shp.zip`), identical in both tiers. Render-check. Commit. - ---- - -## Workstream C — Heavy read-only OGR clear-error - -### Task C1: Reject writes to read-only OGR formats with a clear message - -**Files:** Modify `src/main/scala/.../vectorx/ds/ogr/OGR_DataSource.scala` (and `OGR_Table.scala` if the guard belongs at the table/capability layer). Test: `OgrReaderContractTest.scala`. - -- [ ] **Step 1: Failing test** — `df.write.format("shapefile_ogr").save(path)` raises a clear error containing `"read-only"` and `"shapefile_gbx"` (currently throws `NoSuchFileException` from `stageHeadForSchemaSpark`). -- [ ] **Step 2: Run → FAIL.** -- [ ] **Step 3: Implement** — ensure a write attempt on `ogr`/`shapefile_ogr`/`gpkg_ogr`/`file_gdb_ogr`/`geojson_ogr` fails fast with an actionable message *before* `inferSchema` reads the target. Options to evaluate in implementation: drop `supportsExternalMetadata` for the write path, or have `getTable` return a table whose `capabilities()` excludes `BATCH_WRITE` and let Spark's "table does not support writes" surface — but prefer an explicit GeoBrix message naming the `_gbx` alternative. Reads must be unaffected (verify an existing read test still passes). -- [ ] **Step 4: Run → PASS** (write-guard test + a read regression test). -- [ ] **Step 5: Commit.** - ---- - -## Self-Review - -**Spec coverage:** §3 contract → A1/A2; §4 EXT → A1; §5 applicability (light now) → A3/A4; §6 heavy read-only error → C1; §9 tests → A4/B5 round-trips + unit tests. #71 reader contract → B1–B5. PMTiles → helper is reusable (A2), application out of scope (noted). ✅ -**Placeholder scan:** B2/B3/B4 implementation steps describe the mechanism with the exact files + message but defer some Scala specifics to the implementer — acceptable as they hinge on `read_info`/Hadoop-FS list calls already used in the file; the shared message string is fixed in Global Constraints. No "TBD"/"handle edge cases". -**Type consistency:** `_resolve_single_file_output(path, file_name, ext)`, `_canonical_ext(driver, zip_enabled)`, `_complete_ext(name, ext)` used consistently A1→A3. ✅ - ---- - -## Workstream D — PMTiles (single-unit naming, both tiers) - -Apply the `fileName` + adaptive-naming contract to PMTiles. Single-archive (light `shardZoom=0` + heavy `pmtiles`) = full contract with `EXT=".pmtiles"`. Light **sharded** (default `shardZoom=6`, a directory of shards + overview + STAC) = the SAME 3-case resolution for the output DIRECTORY with `EXT=""` (no extension; directory unit), creating parents. Decision recorded 2026-06-27: "single-archive full; sharded dir adaptive too". - -### Task D1: light `pmtiles_gbx` fileName + adaptive naming (both modes) -- In `python/.../ds/vector.py`: make `_resolve_single_file_output(path, file_name, ext)` accept `ext=""` (directory unit) → 3-case resolution with NO extension completion and NO wrong-ext rejection (just resolve the name + mkdir parents); `_complete_ext(name, "")` returns `name` unchanged. Add `".pmtiles"` to `_RECOGNIZED_EXTS`. -- In `python/.../ds/pmtiles.py` `__init__`: read `opts.get("filename")`; set `self.path = _resolve_single_file_output(to_local_path(path), file_name, ".pmtiles" if self.shard_zoom==0 else "")`. Compute `scratch_dir`/`_target_exists`/`_clear_target` AFTER resolution (unchanged logic). -- Tests (`python/.../test/ds/`): single-archive (stem→`.pmtiles`; `fileName`; existing-dir→`/.pmtiles`) and sharded (stem→dir; `fileName`→`//` dir; existing-dir→`//`) resolution + a round-trip each; cross-writer wrong-ext (a `.gpkg` fileName on single → clear error). - -### Task D2: heavy `pmtiles` fileName + adaptive naming (single-archive) -- Add `HadoopUtils.resolveSingleFileOutput(path, fileName, ext, hConf)` (Scala) mirroring the Python 3-case using Hadoop FS (`exists`/`isDirectory`/`mkdirs` on `cleanPath`'d BARE `/Volumes` paths — never `file:`). -- In `PMTiles_WriteBuilder.build()`: read `options.get("fileName")`; resolve `path` (EXT=`.pmtiles`) before constructing `PMTiles_BatchWrite`. -- Tests (`src/test/.../pmtiles/`): write to a stem path → `.pmtiles`; `fileName`; existing-dir. - -### Task D3: docs -- The pmtiles writer doc page: document `fileName` + the 3-case naming for single-archive (`.pmtiles`) and sharded (directory) modes; note both tiers. diff --git a/docs/superpowers/plans/2026-06-27-helios-sp1-overture-data-source.md b/docs/superpowers/plans/2026-06-27-helios-sp1-overture-data-source.md deleted file mode 100644 index f55407c6f..000000000 --- a/docs/superpowers/plans/2026-06-27-helios-sp1-overture-data-source.md +++ /dev/null @@ -1,1686 +0,0 @@ -# Overture Data Source (SP1) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to execute this plan — dispatch each task to a fresh subagent with the full task text, gate at the review checkpoint between tasks, and never carry implementation context forward beyond what a task's **Interfaces** block declares. - -**Goal:** Build `gbx.sample.overture` — an API-level, distributed, AOI-driven Overture Maps GeoParquet data source (all themes/types), mirroring `gbx.stac.StacClient`'s shape and test-injection seams. The performant default is a distributed Spark read of Overture GeoParquet over the cloud path with `bbox`-struct predicate pushdown, written distributed to a UC Volume plus an optional metadata Delta table; whole-file STAC HTTP-href download is the fallback. Discovery traverses Overture's static STAC `catalog.json`, filters items client-side by bbox, and uses the `overturemaps` CLI as a fast-path when present. - -**Architecture:** Two modules sit beside `sample/_bundle.py` in the WHL: -- `sample/_overture_discover.py` — pure, driver-side, network-free-when-injected helpers: bbox-intersection math, STAC catalog traversal (via injectable opener), release resolution, and an optional `overturemaps` CLI fast-path. Unit-testable in isolation with no Spark and no network. -- `sample/overture.py` — public `OvertureClient` (`discover` / `download` / `read`) + the `download_overture_aoi` one-shot convenience. Holds the Spark distribution logic: Serverless-safe `repartition(N, col)` distributed read + AOI rewrite (default), asset-level HTTP-href download (fallback), parquet-open validation, idempotent skip, and the metadata Delta `MERGE`. - -`OvertureClient` carries two injection seams identical to `StacClient`: `_catalog_opener` (returns a traversable catalog object; when set, discovery runs on the driver with no network) and `_get_fn` (an HTTP fetcher passed through to the fallback downloader). Both default to `None` (production paths construct the real opener/fetcher). - -**Tech Stack:** Python 3.12+, PySpark 4.0.0 (local mode for unit tests), `pystac` (static catalog traversal — distinct from `pystac-client`), `geopandas`/`pyarrow` (parquet read, already present), Delta Lake (`delta.tables.DeltaTable` for the MERGE). `overturemaps` CLI optional (fast-path only). Tests run offline via injected opener + injected fetcher + a local `SparkSession`. - -## Global Constraints - -These project-wide constraints (from the spec's cross-cutting sections and `CLAUDE.md`) apply to every task below. Copying them verbatim so a subagent executing one task in isolation does not violate them: - -- **Version floor:** Python 3.12+. New deps must be pinned in lockstep with DBR 17.3 LTS where a DBR-installed version exists; otherwise pinned independently. -- **Serverless is the first-class target (hard constraints):** - - Parallelism is **only** via `DataFrame.repartition(N, column)` — hash by a column. A number-only `repartition(N)` is AQE-coalesced back toward 1 partition (serial) on Serverless and is forbidden. - - **No** `spark.conf.set` (no-op on Serverless), **no** `.cache()` / `.persist()` / `.checkpoint()`, **no** `.rdd`, **no** `sparkContext` / `_jvm`. Only `udf.register` + Column expressions and DataFrame ops. - - When iterating a plan locally, **verify partitions are not coalesced** with `df.rdd.getNumPartitions()` — note: `.rdd` is allowed *only* in local-test assertions, never in product code paths. (The local SparkSession in tests is Classic, so `.rdd` works there for the assertion; production code never touches it.) - - `CREATE TEMP TABLE` materialization (to pin a distributed result) is Serverless / DBR 18.1+ only — do not rely on it for the default path's correctness. -- **Light-CI-lock checklist (do BOTH halves, see Task 11):** - - (a) Add new light runtime deps to `python/geobrix/requirements-pyrx-ci.in` **and** `python/geobrix/requirements-dev-container.in`, then recompile the hash-pinned `.txt` locks (`uv pip compile --generate-hashes --python-version 3.12 ...`). - - (b) Register the new `test/sample/` directory in **both** `test/conftest.py`'s `_LIGHT_TEST_DIRS` (so the heavyweight CI phase skips it) **and** the explicit pytest dir list in `.github/actions/pyrx_build/action.yml` (so the light phase RUNS it). -- **Unity Catalog Volumes rules:** `/Volumes/...` is FUSE-mounted — use `pathlib`/`os`, not the Files SDK. The Volume root must pre-exist; only paths under it can be created (`os.makedirs(volume_root, exist_ok=True)` is a no-op). Avoid `seek`; sequential I/O only. For writes prefer temp-file-then-`shutil.copy`. Sanitize env-derived strings before building volume paths. -- **TDD by default:** every task writes a failing test first, runs it to confirm the failure, then writes minimal code to pass, then re-runs green, then commits. The test is the definition of done. -- **Commit hygiene:** subject ≤72 chars; a WHY body for any non-trivial/multi-purpose commit. End commit messages with the `Co-authored-by: Isaac` trailer. -- **No aliases / canonical names:** one canonical name per public symbol; signatures below are pinned because SP2/SP3 depend on them. -- **Docs voice:** no internal planning vocabulary (no wave numbers / dispatch references) in any user-facing text. - ---- - -### Task 1: bbox-intersect util + discovery parsing (`_overture_discover.py`) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/sample/_overture_discover.py` -- Test: `python/geobrix/test/sample/test_overture_discover.py` -- Create: `python/geobrix/test/sample/__init__.py` (empty package marker, mirrors `test/stac/__init__.py` so cloudpickle resolves fakes by module name) - -**Interfaces:** -- Produces: - - `bbox_intersects(a, b) -> bool` where `a`, `b` are `(minx, miny, maxx, maxy)` tuples; axis-aligned overlap test (touching edges count as intersecting). - - `normalize_bbox(bbox) -> tuple[float, float, float, float]` — accepts a 4-tuple/list, validates `minx<=maxx` and `miny<=maxy`, returns a float tuple; raises `ValueError` on malformed input. - - `OVERTURE_THEMES: dict[str, list[str]]` — the canonical theme→types map: `addresses:["address"]`, `base:["infrastructure","land","land_cover","land_use","water","bathymetry"]`, `buildings:["building","building_part"]`, `divisions:["division","division_area","division_boundary"]`, `places:["place"]`, `transportation:["connector","segment"]`. - - `expand_themes(themes) -> list[tuple[str, str]]` — `None` → every `(theme, type)` pair from `OVERTURE_THEMES`; a list of theme names → that subset's pairs; raises `ValueError` on an unknown theme. - -- [ ] **Step 1: Write the failing test for `bbox_intersects` + `normalize_bbox`.** -```python -# python/geobrix/test/sample/test_overture_discover.py -import pytest - -from databricks.labs.gbx.sample._overture_discover import ( - bbox_intersects, - normalize_bbox, -) - - -def test_bbox_intersects_overlap(): - assert bbox_intersects((0, 0, 10, 10), (5, 5, 15, 15)) is True - - -def test_bbox_intersects_touching_edge(): - # touching edges count as intersecting (inclusive) - assert bbox_intersects((0, 0, 10, 10), (10, 0, 20, 10)) is True - - -def test_bbox_intersects_disjoint(): - assert bbox_intersects((0, 0, 1, 1), (5, 5, 6, 6)) is False - - -def test_normalize_bbox_returns_floats(): - assert normalize_bbox([1, 2, 3, 4]) == (1.0, 2.0, 3.0, 4.0) - - -def test_normalize_bbox_rejects_inverted(): - with pytest.raises(ValueError): - normalize_bbox((10, 0, 0, 10)) -``` - -- [ ] **Step 2: Run it; confirm it fails on the missing module.** - `gbx:test:python --path test/sample/test_overture_discover.py` - Expected: `ModuleNotFoundError: No module named 'databricks.labs.gbx.sample._overture_discover'` (collection error). - -- [ ] **Step 3: Write minimal `bbox_intersects` + `normalize_bbox` (+ module docstring) in `_overture_discover.py`.** -```python -"""Overture static-STAC discovery helpers (driver-side, network-free when injected). - -Kept separate from overture.py so the catalog traversal / bbox-intersect / CLI -fast-path logic is unit-testable in isolation, with no Spark and no network. The -catalog opener is injected by OvertureClient (_catalog_opener) for offline tests, -exactly like StacClient's seam. -""" - -from __future__ import annotations - -from typing import List, Optional, Tuple - -Bbox = Tuple[float, float, float, float] - - -def normalize_bbox(bbox) -> Bbox: - """Validate and float-cast a (minx, miny, maxx, maxy) bbox.""" - if bbox is None or len(bbox) != 4: - raise ValueError(f"bbox must be (minx, miny, maxx, maxy); got {bbox!r}") - minx, miny, maxx, maxy = (float(v) for v in bbox) - if minx > maxx or miny > maxy: - raise ValueError(f"bbox is inverted (min > max): {bbox!r}") - return (minx, miny, maxx, maxy) - - -def bbox_intersects(a, b) -> bool: - """Axis-aligned overlap test; touching edges count as intersecting.""" - ax0, ay0, ax1, ay1 = a - bx0, by0, bx1, by1 = b - return ax0 <= bx1 and bx0 <= ax1 and ay0 <= by1 and by0 <= ay1 -``` - -- [ ] **Step 4: Re-run; confirm the bbox tests pass.** - `gbx:test:python --path test/sample/test_overture_discover.py` - Expected: 5 passed. - -- [ ] **Step 5: Add the failing test for `OVERTURE_THEMES` + `expand_themes`.** -```python -# append to test_overture_discover.py -from databricks.labs.gbx.sample._overture_discover import ( - OVERTURE_THEMES, - expand_themes, -) - - -def test_overture_themes_complete(): - assert set(OVERTURE_THEMES) == { - "addresses", - "base", - "buildings", - "divisions", - "places", - "transportation", - } - assert OVERTURE_THEMES["buildings"] == ["building", "building_part"] - - -def test_expand_themes_none_is_all_pairs(): - pairs = expand_themes(None) - assert ("buildings", "building") in pairs - assert ("transportation", "segment") in pairs - # one pair per (theme, type) - assert len(pairs) == sum(len(v) for v in OVERTURE_THEMES.values()) - - -def test_expand_themes_subset(): - assert expand_themes(["places"]) == [("places", "place")] - - -def test_expand_themes_unknown_raises(): - with pytest.raises(ValueError): - expand_themes(["weather"]) -``` - -- [ ] **Step 6: Run it; confirm ImportError on the new symbols.** - `gbx:test:python --path test/sample/test_overture_discover.py` - Expected: `ImportError: cannot import name 'OVERTURE_THEMES'`. - -- [ ] **Step 7: Add `OVERTURE_THEMES` + `expand_themes`.** -```python -OVERTURE_THEMES = { - "addresses": ["address"], - "base": ["infrastructure", "land", "land_cover", "land_use", "water", "bathymetry"], - "buildings": ["building", "building_part"], - "divisions": ["division", "division_area", "division_boundary"], - "places": ["place"], - "transportation": ["connector", "segment"], -} - - -def expand_themes(themes: Optional[List[str]]) -> List[Tuple[str, str]]: - """themes=None -> every (theme, type) pair; a list -> that subset's pairs.""" - names = list(OVERTURE_THEMES) if themes is None else list(themes) - pairs: List[Tuple[str, str]] = [] - for name in names: - if name not in OVERTURE_THEMES: - raise ValueError( - f"unknown Overture theme {name!r}; valid: {sorted(OVERTURE_THEMES)}" - ) - pairs.extend((name, t) for t in OVERTURE_THEMES[name]) - return pairs -``` - -- [ ] **Step 8: Re-run; confirm all pass.** - `gbx:test:python --path test/sample/test_overture_discover.py` - Expected: 9 passed. - -- [ ] **Step 9: Commit.** - `git add python/geobrix/src/databricks/labs/gbx/sample/_overture_discover.py python/geobrix/test/sample/__init__.py python/geobrix/test/sample/test_overture_discover.py` - `git commit -m "feat(sample): overture bbox-intersect + theme expansion helpers" -m "Foundation for the Overture discovery module: axis-aligned bbox intersection, bbox normalization, and the canonical theme->type map + expansion (None => all themes). Pure driver-side helpers, no Spark/network, unit-tested in isolation." -m "Co-authored-by: Isaac"` - ---- - -### Task 2: static catalog traversal with an injected opener - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/_overture_discover.py` -- Test: `python/geobrix/test/sample/test_overture_discover.py` -- Create: `python/geobrix/test/sample/_fake_overture_catalog.py` (top-level importable fakes, mirrors `test/stac/_fake_catalog.py`) - -**Interfaces:** -- Consumes: `bbox_intersects`, `normalize_bbox`, `expand_themes` (Task 1). -- Produces: - - `traverse_catalog(opener, bbox, theme_pairs) -> list[dict]` — walks a static STAC catalog (root `catalog.json` → child collections → items), filters items whose `bbox` intersects the AOI, restricts to the requested `(theme, type)` pairs, and returns one dict per intersecting GeoParquet asset with keys: `theme`, `type`, `href`, `asset_bbox` (a 4-float list). `opener()` returns a catalog object exposing `.get_children()` (collections) and `.get_items()` (items); each item exposes `.bbox`, `.properties` (carrying `theme`/`type`), and `.assets` (mapping name → object with `.href`). This is the `pystac.Catalog` shape, faked in tests. - -- [ ] **Step 1: Write the importable fake catalog.** -```python -# python/geobrix/test/sample/_fake_overture_catalog.py -"""Top-level importable fake Overture static STAC catalog for offline tests. - -Mirrors the pystac.Catalog surface that traverse_catalog walks: a root with -get_children() -> collections, each with get_items() -> items, each item with -.bbox, .properties (theme/type), and .assets (name -> obj with .href). -Importable (not a closure) so it can be injected as _catalog_opener and, if ever -used on a worker, resolved by cloudpickle via the module name. -""" - -from __future__ import annotations - - -class _Asset: - def __init__(self, href): - self.href = href - - -class _Item: - def __init__(self, bbox, theme, type_, href): - self.bbox = bbox - self.properties = {"theme": theme, "type": type_} - self.assets = {"data": _Asset(href)} - - -class _Collection: - def __init__(self, items): - self._items = items - - def get_items(self): - return list(self._items) - - -class FakeOvertureCatalog: - """Two collections: SF buildings (intersects) + a faraway places item (disjoint).""" - - def get_children(self): - sf = _Collection( - [ - _Item( - [-122.52, 37.70, -122.36, 37.83], - "buildings", - "building", - "s3://overturemaps-us-west-2/release/buildings/building/sf.parquet", - ) - ] - ) - faraway = _Collection( - [ - _Item( - [10.0, 50.0, 11.0, 51.0], - "places", - "place", - "s3://overturemaps-us-west-2/release/places/place/eu.parquet", - ) - ] - ) - return [sf, faraway] - - -def open_fake_overture(): - return FakeOvertureCatalog() -``` - -- [ ] **Step 2: Write the failing test for `traverse_catalog`.** -```python -# append to test_overture_discover.py -from databricks.labs.gbx.sample._overture_discover import traverse_catalog -from test.sample._fake_overture_catalog import open_fake_overture - - -def test_traverse_catalog_bbox_filters_disjoint(): - sf_bbox = (-122.45, 37.74, -122.40, 37.78) - rows = traverse_catalog(open_fake_overture, sf_bbox, [("buildings", "building")]) - assert len(rows) == 1 - r = rows[0] - assert r["theme"] == "buildings" - assert r["type"] == "building" - assert r["href"].endswith("sf.parquet") - assert r["asset_bbox"] == [-122.52, 37.70, -122.36, 37.83] - - -def test_traverse_catalog_skips_unrequested_pairs(): - # AOI covers the whole world, but we only ask for places -> the SF building drops out - rows = traverse_catalog(open_fake_overture, (-180, -90, 180, 90), [("places", "place")]) - assert [r["type"] for r in rows] == ["place"] -``` - -- [ ] **Step 3: Run it; confirm ImportError / failure.** - `gbx:test:python --path test/sample/test_overture_discover.py` - Expected: `ImportError: cannot import name 'traverse_catalog'`. - -- [ ] **Step 4: Implement `traverse_catalog`.** -```python -def traverse_catalog(opener, bbox, theme_pairs): - """Walk a static STAC catalog and return one dict per intersecting GeoParquet asset. - - opener() returns a pystac.Catalog-shaped object. Items are filtered by AOI - bbox intersection and restricted to the requested (theme, type) pairs. - """ - aoi = normalize_bbox(bbox) - wanted = set(theme_pairs) - rows = [] - catalog = opener() - for collection in catalog.get_children(): - for item in collection.get_items(): - props = item.properties or {} - pair = (props.get("theme"), props.get("type")) - if pair not in wanted: - continue - item_bbox = list(item.bbox) - if not bbox_intersects(aoi, tuple(item_bbox)): - continue - for asset in item.assets.values(): - rows.append( - { - "theme": pair[0], - "type": pair[1], - "href": asset.href, - "asset_bbox": [float(v) for v in item_bbox], - } - ) - return rows -``` - -- [ ] **Step 5: Re-run; confirm green.** - `gbx:test:python --path test/sample/test_overture_discover.py` - Expected: 11 passed. - -- [ ] **Step 6: Commit.** - `git add python/geobrix/src/databricks/labs/gbx/sample/_overture_discover.py python/geobrix/test/sample/_fake_overture_catalog.py python/geobrix/test/sample/test_overture_discover.py` - `git commit -m "feat(sample): traverse Overture static STAC with injected opener" -m "traverse_catalog walks a pystac.Catalog-shaped root (children -> items), filters items by AOI bbox intersection, restricts to requested (theme,type) pairs, and emits one row per GeoParquet asset (theme/type/href/asset_bbox). Tested offline with an importable fake catalog (no network)." -m "Co-authored-by: Isaac"` - ---- - -### Task 3: release resolution + optional `overturemaps` CLI fast-path - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/_overture_discover.py` -- Test: `python/geobrix/test/sample/test_overture_discover.py` - -**Interfaces:** -- Consumes: `traverse_catalog`, `expand_themes` (Tasks 1–2). -- Produces: - - `resolve_release(opener, release=None) -> str` — `release=None` → the latest release id from the catalog (a catalog exposing `.extra_fields["overture:releases"]` as a sorted list, or `.id`); an explicit string returns unchanged. Raises `ValueError` if `release=None` and no release metadata is discoverable. - - `cli_discover(bbox, theme_pairs, release, runner=subprocess.run) -> Optional[list[dict]]` — when the `overturemaps` CLI is importable/on PATH, shell out per `(theme, type)` to list intersecting parquet paths and return rows in the same shape as `traverse_catalog`; returns `None` when the CLI is unavailable (signals the caller to use the traversal fallback). `runner` is injectable for tests. - -- [ ] **Step 1: Failing test for `resolve_release`.** -```python -# append to test_overture_discover.py -from databricks.labs.gbx.sample._overture_discover import resolve_release - - -class _RelCatalog: - extra_fields = {"overture:releases": ["2024-01-01", "2024-07-01"]} - - -class _NoRelCatalog: - extra_fields = {} - id = None - - -def test_resolve_release_explicit_passthrough(): - assert resolve_release(lambda: _RelCatalog(), "2023-12-12") == "2023-12-12" - - -def test_resolve_release_latest(): - assert resolve_release(lambda: _RelCatalog(), None) == "2024-07-01" - - -def test_resolve_release_missing_raises(): - with pytest.raises(ValueError): - resolve_release(lambda: _NoRelCatalog(), None) -``` - -- [ ] **Step 2: Run; confirm ImportError.** - `gbx:test:python --path test/sample/test_overture_discover.py` - Expected: `ImportError: cannot import name 'resolve_release'`. - -- [ ] **Step 3: Implement `resolve_release` + import `subprocess`.** -```python -import subprocess # add to the module imports - - -def resolve_release(opener, release: Optional[str] = None) -> str: - """release=None -> latest release id from the catalog; an explicit string passes through.""" - if release is not None: - return release - catalog = opener() - releases = getattr(catalog, "extra_fields", {}).get("overture:releases") - if releases: - return sorted(releases)[-1] - cat_id = getattr(catalog, "id", None) - if cat_id: - return cat_id - raise ValueError( - "could not resolve latest Overture release from the catalog; pass release=... explicitly" - ) -``` - -- [ ] **Step 4: Re-run; confirm green.** - `gbx:test:python --path test/sample/test_overture_discover.py` - Expected: 14 passed. - -- [ ] **Step 5: Failing test for `cli_discover` (injected runner; absent-CLI path).** -```python -# append to test_overture_discover.py -from databricks.labs.gbx.sample._overture_discover import cli_discover - - -def test_cli_discover_absent_returns_none(monkeypatch): - # No overturemaps on PATH -> None so the caller falls back to traversal. - monkeypatch.setattr( - "databricks.labs.gbx.sample._overture_discover.shutil.which", - lambda name: None, - ) - assert cli_discover((-122.5, 37.7, -122.3, 37.8), [("buildings", "building")], "2024-07-01") is None - - -def test_cli_discover_present_parses_runner(monkeypatch): - monkeypatch.setattr( - "databricks.labs.gbx.sample._overture_discover.shutil.which", - lambda name: "/usr/bin/overturemaps", - ) - - class _Completed: - returncode = 0 - stdout = "s3://overturemaps-us-west-2/2024-07-01/buildings/building/part-0.parquet\n" - - rows = cli_discover( - (-122.5, 37.7, -122.3, 37.8), - [("buildings", "building")], - "2024-07-01", - runner=lambda *a, **k: _Completed(), - ) - assert len(rows) == 1 - assert rows[0]["theme"] == "buildings" - assert rows[0]["type"] == "building" - assert rows[0]["href"].endswith("part-0.parquet") - assert rows[0]["asset_bbox"] == [-122.5, 37.7, -122.3, 37.8] -``` - -- [ ] **Step 6: Run; confirm ImportError.** - `gbx:test:python --path test/sample/test_overture_discover.py` - Expected: `ImportError: cannot import name 'cli_discover'`. - -- [ ] **Step 7: Implement `cli_discover` + import `shutil`.** -```python -import shutil # add to the module imports - - -def cli_discover(bbox, theme_pairs, release, runner=subprocess.run): - """Fast-path via the `overturemaps` CLI when present; None otherwise. - - Returns rows shaped like traverse_catalog (theme/type/href/asset_bbox). The - asset_bbox is the AOI bbox (the CLI lists paths intersecting the bbox, not - per-file extents), which is sufficient for downstream pushdown bookkeeping. - """ - if shutil.which("overturemaps") is None: - return None - aoi = normalize_bbox(bbox) - bbox_arg = ",".join(str(v) for v in aoi) - rows = [] - for theme, type_ in theme_pairs: - completed = runner( - [ - "overturemaps", - "download", - "--bbox", - bbox_arg, - "--release", - release, - "--type", - type_, - "--list-paths", - ], - capture_output=True, - text=True, - ) - if getattr(completed, "returncode", 1) != 0: - continue - for line in (completed.stdout or "").splitlines(): - href = line.strip() - if href: - rows.append( - { - "theme": theme, - "type": type_, - "href": href, - "asset_bbox": list(aoi), - } - ) - return rows -``` - -- [ ] **Step 8: Re-run; confirm green.** - `gbx:test:python --path test/sample/test_overture_discover.py` - Expected: 16 passed. - -- [ ] **Step 9: Commit.** - `git add python/geobrix/src/databricks/labs/gbx/sample/_overture_discover.py python/geobrix/test/sample/test_overture_discover.py` - `git commit -m "feat(sample): Overture release resolution + CLI fast-path" -m "resolve_release picks the latest release from catalog metadata (or passes an explicit pin through); cli_discover shells out to the optional overturemaps CLI per (theme,type) and returns None when absent so callers fall back to static traversal. Runner injectable; tested offline." -m "Co-authored-by: Isaac"` - ---- - -### Task 4: `OvertureClient.discover` → DataFrame - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/sample/overture.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/__init__.py` -- Test: `python/geobrix/test/sample/test_overture.py` - -**Interfaces:** -- Consumes: `_overture_discover.expand_themes`, `traverse_catalog`, `cli_discover`, `resolve_release`. -- Produces: - - `class OvertureClient(catalog="https://stac.overturemaps.org/catalog.json", release=None, _catalog_opener=None, _get_fn=None)`. - - `OvertureClient.discover(bbox, themes=None, release=None) -> DataFrame` — columns (in order): `theme: string`, `type: string`, `href: string`, `asset_bbox: array`, `release: string`. `themes=None` → all themes/types. CLI fast-path is tried first (skipped when `_catalog_opener` is injected — offline tests force traversal); otherwise static traversal via the opener. - -- [ ] **Step 1: Failing test for `discover`.** -```python -# python/geobrix/test/sample/test_overture.py -import pytest - -pyspark = pytest.importorskip("pyspark") - -from databricks.labs.gbx.sample.overture import OvertureClient -from test.sample._fake_overture_catalog import open_fake_overture - - -@pytest.fixture(scope="module") -def spark(): - from pyspark.sql import SparkSession - - s = ( - SparkSession.builder.master("local[2]") - .appName("overture-sp1-test") - .config("spark.sql.shuffle.partitions", "8") - .getOrCreate() - ) - yield s - s.stop() - - -def test_discover_columns_and_filter(spark): - client = OvertureClient( - release="2024-07-01", _catalog_opener=open_fake_overture - ) - df = client.discover((-122.45, 37.74, -122.40, 37.78), themes=["buildings"]) - assert df.columns == ["theme", "type", "href", "asset_bbox", "release"] - rows = df.collect() - assert len(rows) == 1 - assert rows[0]["theme"] == "buildings" - assert rows[0]["release"] == "2024-07-01" - assert rows[0]["asset_bbox"] == [-122.52, 37.70, -122.36, 37.83] - - -def test_discover_all_themes_when_none(spark): - client = OvertureClient(release="2024-07-01", _catalog_opener=open_fake_overture) - df = client.discover((-180, -90, 180, 90), themes=None) - # fake catalog has a building + a place; both fall inside the world bbox - assert {r["type"] for r in df.collect()} == {"building", "place"} -``` - -- [ ] **Step 2: Run; confirm collection/import failure.** - `gbx:test:python --path test/sample/test_overture.py` - Expected: `ModuleNotFoundError: No module named 'databricks.labs.gbx.sample.overture'`. - -- [ ] **Step 3: Implement `OvertureClient.__init__` + `_open_catalog` + `discover`.** -```python -"""OvertureClient — distributed, AOI-driven Overture Maps GeoParquet data source. - -Mirrors gbx.stac.StacClient: a static-catalog discovery step (driver-side, -metadata-only), then DISTRIBUTED asset I/O. Default I/O path is a distributed -Spark read of Overture GeoParquet over the cloud path with bbox-struct predicate -pushdown (AOI rows only), written to a UC Volume + an optional metadata Delta -table; an HTTP-href whole-file download is the fallback. Serverless-safe: -parallelism only via repartition(N, col); no spark.conf/cache/persist/.rdd. - -Injection seams (offline tests): _catalog_opener (returns a pystac.Catalog-shaped -object) and _get_fn (an HTTP fetcher passed to the fallback downloader). -""" - -from __future__ import annotations - -from typing import TYPE_CHECKING, List, Optional - -if TYPE_CHECKING: - from pyspark.sql import DataFrame - -OVERTURE_CATALOG = "https://stac.overturemaps.org/catalog.json" - -# Spark schema for discover() output — pinned; SP2/SP3 depend on it. -_DISCOVER_COLS = ["theme", "type", "href", "asset_bbox", "release"] - - -def _discover_schema(): - from pyspark.sql.types import ( - ArrayType, - DoubleType, - StringType, - StructField, - StructType, - ) - - return StructType( - [ - StructField("theme", StringType()), - StructField("type", StringType()), - StructField("href", StringType()), - StructField("asset_bbox", ArrayType(DoubleType())), - StructField("release", StringType()), - ] - ) - - -class OvertureClient: - def __init__( - self, - catalog: str = OVERTURE_CATALOG, - release: Optional[str] = None, - _catalog_opener=None, - _get_fn=None, - ): - self.catalog = catalog - self.release = release - self._catalog_opener = _catalog_opener - self._get_fn = _get_fn - - def _open_catalog(self): - if self._catalog_opener is not None: - return self._catalog_opener() - import pystac - - return pystac.Catalog.from_file(self.catalog) - - def _opener(self): - return self._catalog_opener if self._catalog_opener is not None else self._open_catalog - - def discover(self, bbox, themes=None, release=None) -> "DataFrame": - """One row per intersecting GeoParquet asset for the AOI. - - Columns: theme, type, href, asset_bbox, release. themes=None => ALL. - Driver-side + metadata-only (lightweight); asset I/O happens in download(). - """ - from pyspark.sql import SparkSession - - from databricks.labs.gbx.sample._overture_discover import ( - cli_discover, - expand_themes, - resolve_release, - traverse_catalog, - ) - - opener = self._opener() - rel = resolve_release(opener, release or self.release) - pairs = expand_themes(themes) - - rows = None - # CLI fast-path only in production (no injected opener); offline tests force traversal. - if self._catalog_opener is None: - rows = cli_discover(bbox, pairs, rel) - if rows is None: - rows = traverse_catalog(opener, bbox, pairs) - - for r in rows: - r["release"] = rel - - spark = SparkSession.getActiveSession() - schema = _discover_schema() - if not rows: - return spark.createDataFrame([], schema) - ordered = [tuple(r[c] for c in _DISCOVER_COLS) for r in rows] - return spark.createDataFrame(ordered, schema) -``` - -- [ ] **Step 4: Re-export `OvertureClient` from `sample/__init__.py`.** - Add `OvertureClient` (and, after Task 10, `download_overture_aoi`) to the import block and `__all__`: -```python -from databricks.labs.gbx.sample.overture import OvertureClient - -# ... extend __all__ with "OvertureClient" -``` - -- [ ] **Step 5: Re-run; confirm green.** - `gbx:test:python --path test/sample/test_overture.py` - Expected: 2 passed. - -- [ ] **Step 6: Commit.** - `git add python/geobrix/src/databricks/labs/gbx/sample/overture.py python/geobrix/src/databricks/labs/gbx/sample/__init__.py python/geobrix/test/sample/test_overture.py` - `git commit -m "feat(sample): OvertureClient.discover over static STAC" -m "OvertureClient mirrors StacClient's shape and injection seams. discover() resolves the release, expands themes (None=>all), tries the CLI fast-path (production only) then static traversal, and returns a typed DataFrame (theme/type/href/asset_bbox/release). Re-exported from sample/__init__." -m "Co-authored-by: Isaac"` - ---- - -### Task 5: distributed read + AOI rewrite (the performant default download path) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/overture.py` -- Test: `python/geobrix/test/sample/test_overture.py` - -**Interfaces:** -- Consumes: `OvertureClient.discover` output DataFrame. -- Produces: - - private `OvertureClient._download_distributed(assets_df, out_dir, *, bbox, validate, partitions) -> DataFrame` — for each discovered asset, distributed-read the GeoParquet over its cloud `href` with a `bbox`-struct predicate pushdown (`F.col("bbox.xmin") <= maxx`, etc., when a `bbox` struct column is present; else read whole then no-op filter), repartition by `(theme, type, href)` (column hash, never number-only), write the AOI subset to `out_dir///` as parquet on the Volume, and emit the metadata rows (cols per the `download` contract). Must not coalesce: tested via `getNumPartitions`. - -- [ ] **Step 1: Failing test asserting distributed plan is not coalesced + AOI subset written.** -```python -# append to test_overture.py -import os - - -def _write_fake_overture_parquet(spark, path, bbox_struct=True): - """Write a tiny GeoParquet-ish parquet with a bbox struct so pushdown can fire.""" - from pyspark.sql import Row - - if bbox_struct: - rows = [ - Row(id=1, bbox=Row(xmin=-122.42, ymin=37.75, xmax=-122.41, ymax=37.76)), - Row(id=2, bbox=Row(xmin=10.0, ymin=50.0, xmax=10.1, ymax=50.1)), # outside SF - ] - else: - rows = [Row(id=1), Row(id=2)] - spark.createDataFrame(rows).write.mode("overwrite").parquet(path) - - -def test_download_distributed_writes_aoi_subset(spark, tmp_path): - src = str(tmp_path / "src.parquet") - _write_fake_overture_parquet(spark, src) - out_dir = str(tmp_path / "out") - - client = OvertureClient(release="2024-07-01", _catalog_opener=open_fake_overture) - from pyspark.sql.types import ( - ArrayType, - DoubleType, - StringType, - StructField, - StructType, - ) - - schema = StructType( - [ - StructField("theme", StringType()), - StructField("type", StringType()), - StructField("href", StringType()), - StructField("asset_bbox", ArrayType(DoubleType())), - StructField("release", StringType()), - ] - ) - assets = spark.createDataFrame( - [("buildings", "building", src, [-122.52, 37.70, -122.36, 37.83], "2024-07-01")], - schema, - ) - meta = client._download_distributed( - assets, out_dir, bbox=(-122.45, 37.74, -122.40, 37.78), validate=True, partitions=4 - ) - # Serverless-safety: hash-by-column repartition is NOT AQE-coalesced to 1. - assert meta.rdd.getNumPartitions() > 1 - mrows = meta.collect() - assert len(mrows) == 1 - written = mrows[0]["source"] - assert written == mrows[0]["path"] # source aliased as path - assert os.path.isdir(written) or os.path.exists(written) - # only the in-AOI row survived the bbox-struct pushdown - subset = spark.read.parquet(written) - assert subset.count() == 1 - assert subset.collect()[0]["id"] == 1 -``` - -- [ ] **Step 2: Run; confirm AttributeError on the missing method.** - `gbx:test:python --path test/sample/test_overture.py` - Expected: `AttributeError: 'OvertureClient' object has no attribute '_download_distributed'`. - -- [ ] **Step 3: Implement `_download_distributed`.** -```python - def _download_distributed( - self, assets_df, out_dir, *, bbox, validate, partitions - ) -> "DataFrame": - """Performant default: distributed read of each asset's GeoParquet with a - bbox-struct predicate pushdown, AOI subset written to the Volume per - (theme, type). Returns the metadata rows. Serverless-safe: repartition by - column only; no spark.conf/cache/persist.""" - import os - - from pyspark.sql import SparkSession - from pyspark.sql import functions as F - - from databricks.labs.gbx.sample._overture_discover import normalize_bbox - - spark = SparkSession.getActiveSession() - minx, miny, maxx, maxy = normalize_bbox(bbox) - assets = assets_df.select(*_DISCOVER_COLS).collect() - - meta_rows = [] - for a in assets: - df = spark.read.parquet(a["href"]) - # bbox-struct predicate pushdown when the Overture `bbox` struct is present. - if "bbox" in df.columns: - df = df.filter( - (F.col("bbox.xmin") <= F.lit(maxx)) - & (F.col("bbox.xmax") >= F.lit(minx)) - & (F.col("bbox.ymin") <= F.lit(maxy)) - & (F.col("bbox.ymax") >= F.lit(miny)) - ) - target = os.path.join(out_dir, a["theme"], a["type"]) - # Hash-by-column repartition (NOT number-only): on Serverless a - # round-robin repartition(N) is AQE-coalesced to 1 (serial). Hash by a - # real source column so the per-asset row groups spread across cores. - # Prefer the Overture `id` column; else hash by the first column. - key = "id" if "id" in df.columns else df.columns[0] - ( - df.repartition(partitions, F.col(key)) - .write.mode("overwrite") - .parquet(target) - ) - valid = True - if validate: - try: - spark.read.parquet(target).limit(1).count() - except Exception: - valid = False - try: - sz = sum( - os.path.getsize(os.path.join(target, f)) - for f in os.listdir(target) - if f.endswith(".parquet") - ) - except OSError: - sz = None - meta_rows.append( - { - "theme": a["theme"], - "type": a["type"], - "source": target, - "out_file_sz": sz, - "is_out_file_valid": valid, - "asset_bbox": a["asset_bbox"], - "release": a["release"], - "href": a["href"], - } - ) - - return _meta_dataframe(spark, meta_rows, partitions) -``` - Add a module-level helper that builds the metadata DataFrame (reused by the fallback path in Task 6), repartitions by a column so the result stays distributed and is not AQE-coalesced, and aliases `source` as `path`: -```python -_META_COLS = [ - "theme", - "type", - "source", - "path", - "out_file_sz", - "is_out_file_valid", - "last_update", - "asset_bbox", - "release", - "href", -] - - -def _meta_schema(): - from pyspark.sql.types import ( - ArrayType, - BooleanType, - DoubleType, - LongType, - StringType, - StructField, - StructType, - TimestampType, - ) - - return StructType( - [ - StructField("theme", StringType()), - StructField("type", StringType()), - StructField("source", StringType()), - StructField("out_file_sz", LongType()), - StructField("is_out_file_valid", BooleanType()), - StructField("asset_bbox", ArrayType(DoubleType())), - StructField("release", StringType()), - StructField("href", StringType()), - ] - ) - - -def _meta_dataframe(spark, meta_rows, partitions): - from pyspark.sql import functions as F - - cols = ["theme", "type", "source", "out_file_sz", "is_out_file_valid", "asset_bbox", "release", "href"] - if not meta_rows: - df = spark.createDataFrame([], _meta_schema()) - else: - df = spark.createDataFrame( - [tuple(r[c] for c in cols) for r in meta_rows], _meta_schema() - ) - n = max(1, partitions or 1) - return ( - # repartition by (theme, type, source) keeps the result distributed - # (column-hash, not AQE-coalesced) per the Serverless rule. - df.repartition(n, F.col("theme"), F.col("type"), F.col("source")) - .withColumn("path", F.col("source")) - .withColumn("last_update", F.current_timestamp()) - .select( - "theme", - "type", - "source", - "path", - "out_file_sz", - "is_out_file_valid", - "last_update", - "asset_bbox", - "release", - "href", - ) - ) -``` - Implementer note: the repartition hashes by a real source column (`id` when present, else the first column) — never a number-only `repartition(partitions)`, which AQE-coalesces to serial on Serverless. The Step 1 test asserts `getNumPartitions() > 1` on the resulting metadata DataFrame. - -- [ ] **Step 4: Re-run; confirm green and the not-coalesced assertion holds.** - `gbx:test:python --path test/sample/test_overture.py` - Expected: 3 passed (the new test included). - -- [ ] **Step 5: Commit.** - `git add python/geobrix/src/databricks/labs/gbx/sample/overture.py python/geobrix/test/sample/test_overture.py` - `git commit -m "feat(sample): distributed Overture read + AOI rewrite default" -m "Performant default download path: distributed-read each asset's GeoParquet with a bbox-struct predicate pushdown (AOI rows only), write the subset per (theme,type) to the Volume, emit metadata. Repartition by column (never number-only) so the plan stays distributed on Serverless; verified via getNumPartitions > 1." -m "Co-authored-by: Isaac"` - ---- - -### Task 6: whole-file HTTP-href download fallback (injected `_get_fn`) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/overture.py` -- Test: `python/geobrix/test/sample/test_overture.py` - -**Interfaces:** -- Consumes: `OvertureClient.discover` output; the `_get_fn` seam. -- Produces: - - private `OvertureClient._download_fallback(assets_df, out_dir, *, validate, max_tries, partitions) -> DataFrame` — fan whole-file downloads out with `repartition(N, F.col("href"))`, stream each `href` to `out_dir///` (temp-file then sequential copy, Volume-safe), validate by opening the parquet, emit the same metadata schema as the distributed path. Uses `self._get_fn` when injected (offline tests), `requests.get` otherwise. - -- [ ] **Step 1: Failing test with an injected fetcher (no network).** -```python -# append to test_overture.py -def test_download_fallback_injected_fetcher(spark, tmp_path): - # build a real source parquet, then serve its bytes through a fake _get_fn - src = str(tmp_path / "asset.parquet") - from pyspark.sql import Row - - spark.createDataFrame([Row(id=1), Row(id=2)]).coalesce(1).write.mode( - "overwrite" - ).parquet(src) - # the "asset" is a single part file inside src - part = [f for f in os.listdir(src) if f.endswith(".parquet")][0] - src_file = os.path.join(src, part) - - def fake_get(href, timeout=None, stream=False): - class _Resp: - status_code = 200 - - def raise_for_status(self): - pass - - def iter_content(self, n): - with open(src_file, "rb") as fh: - while True: - chunk = fh.read(n) - if not chunk: - break - yield chunk - - return _Resp() - - client = OvertureClient( - release="2024-07-01", _catalog_opener=open_fake_overture, _get_fn=fake_get - ) - from pyspark.sql.types import ( - ArrayType, - DoubleType, - StringType, - StructField, - StructType, - ) - - schema = StructType( - [ - StructField("theme", StringType()), - StructField("type", StringType()), - StructField("href", StringType()), - StructField("asset_bbox", ArrayType(DoubleType())), - StructField("release", StringType()), - ] - ) - out_dir = str(tmp_path / "out_fb") - assets = spark.createDataFrame( - [("places", "place", "http://fake/place.parquet", [0.0, 0.0, 1.0, 1.0], "2024-07-01")], - schema, - ) - meta = client._download_fallback( - assets, out_dir, validate=True, max_tries=2, partitions=4 - ) - assert meta.rdd.getNumPartitions() > 1 - row = meta.collect()[0] - assert row["is_out_file_valid"] is True - assert os.path.exists(row["source"]) - assert row["source"] == row["path"] - assert spark.read.parquet(row["source"]).count() == 2 -``` - -- [ ] **Step 2: Run; confirm AttributeError.** - `gbx:test:python --path test/sample/test_overture.py` - Expected: `AttributeError: ... '_download_fallback'`. - -- [ ] **Step 3: Implement `_download_fallback`.** -```python - def _download_fallback( - self, assets_df, out_dir, *, validate, max_tries, partitions - ) -> "DataFrame": - """Fallback: whole-file HTTP download fanned out by href (column-hash - repartition, Serverless-safe). Temp-file then sequential copy (Volume-safe). - Returns the same metadata schema as the distributed path.""" - import os - import shutil - import tempfile - - from pyspark.sql import SparkSession - from pyspark.sql import functions as F - from pyspark.sql.types import ( - ArrayType, - BooleanType, - DoubleType, - LongType, - StringType, - StructField, - StructType, - ) - - spark = SparkSession.getActiveSession() - get_fn = self._get_fn # None in production; injectable for tests - _validate = validate - _max_tries = max_tries - - row_schema = StructType( - [ - StructField("theme", StringType()), - StructField("type", StringType()), - StructField("source", StringType()), - StructField("out_file_sz", LongType()), - StructField("is_out_file_valid", BooleanType()), - StructField("asset_bbox", ArrayType(DoubleType())), - StructField("release", StringType()), - StructField("href", StringType()), - ] - ) - - @F.udf(row_schema) - def _fetch(theme, type_, href, asset_bbox, release): - getter = get_fn - if getter is None: - import requests - - getter = requests.get - target_dir = os.path.join(out_dir, theme, type_) - os.makedirs(target_dir, exist_ok=True) - basename = os.path.basename(href.split("?")[0]) or "asset.parquet" - outpath = os.path.join(target_dir, basename) - # idempotent skip: a present, openable file is left as-is - if os.path.exists(outpath) and _is_valid_parquet(outpath): - sz = os.path.getsize(outpath) - return (theme, type_, outpath, sz, True, asset_bbox, release, href) - last_exc = None - for _ in range(max(1, _max_tries)): - tmpd = tempfile.mkdtemp(prefix="gbx_overture_") - try: - local = os.path.join(tmpd, basename) - resp = getter(href, timeout=100, stream=True) - resp.raise_for_status() - with open(local, "wb") as fh: - for chunk in resp.iter_content(1024 * 1024): - if chunk: - fh.write(chunk) - ok = (not _validate) or _is_valid_parquet(local) - if ok: - shutil.copyfile(local, outpath) - sz = os.path.getsize(outpath) - return (theme, type_, outpath, sz, True, asset_bbox, release, href) - except Exception as exc: # noqa: BLE001 - last_exc = exc - finally: - shutil.rmtree(tmpd, ignore_errors=True) - return (theme, type_, None, None, False, asset_bbox, release, href) - - n = max(1, partitions or 1) - fetched = ( - assets_df.select(*_DISCOVER_COLS) - # column-hash repartition by href (NOT number-only) for Serverless. - .repartition(n, F.col("href")) - .withColumn( - "_m", - _fetch("theme", "type", "href", "asset_bbox", "release"), - ) - .select("_m.*") - .withColumn("path", F.col("source")) - .withColumn("last_update", F.current_timestamp()) - .select( - "theme", - "type", - "source", - "path", - "out_file_sz", - "is_out_file_valid", - "last_update", - "asset_bbox", - "release", - "href", - ) - ) - return fetched -``` - Add the module-level parquet validity helper (importable inside the UDF): -```python -def _is_valid_parquet(path: str) -> bool: - """True iff the parquet opens (pyarrow). Validity = opens, not raster-decodable.""" - try: - import pyarrow.parquet as pq - - pq.ParquetFile(path).metadata # touch metadata to force a read - return True - except Exception: # noqa: BLE001 - return False -``` - -- [ ] **Step 4: Re-run; confirm green.** - `gbx:test:python --path test/sample/test_overture.py` - Expected: 4 passed. - -- [ ] **Step 5: Commit.** - `git add python/geobrix/src/databricks/labs/gbx/sample/overture.py python/geobrix/test/sample/test_overture.py` - `git commit -m "feat(sample): whole-file Overture download fallback path" -m "Asset-level HTTP-href download for when no cloud read is available: fan out by href (column-hash repartition, Serverless-safe), temp-file then sequential copy (Volume-safe), retry + parquet-open validate + idempotent skip. _get_fn injectable; tested offline with a fake fetcher serving real parquet bytes." -m "Co-authored-by: Isaac"` - ---- - -### Task 7: public `download` — path selection, validation, idempotency - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/overture.py` -- Test: `python/geobrix/test/sample/test_overture.py` - -**Interfaces:** -- Consumes: `_download_distributed`, `_download_fallback`. -- Produces: - - `OvertureClient.download(assets_df, out_dir, *, table=None, validate=True, max_tries=5, partitions=None) -> DataFrame` — columns (order pinned): `theme, type, source, path, out_file_sz, is_out_file_valid, last_update, asset_bbox, release, href`. Chooses the distributed read path when an asset `href` is a cloud path (`s3://`/`abfs://`/`abfss://`/`gs://`/`wasbs://`); otherwise the HTTP-href fallback. (The `table=` MERGE is added in Task 8.) `partitions=None` defaults to `max(1, asset_count)`. - -- [ ] **Step 1: Failing test for path selection + idempotent skip.** -```python -# append to test_overture.py -def test_download_routes_cloud_to_distributed(spark, tmp_path): - src = str(tmp_path / "cloudish.parquet") - from pyspark.sql import Row - - # simulate a cloud asset with a local path that LOOKS like a cloud read target - spark.createDataFrame([Row(id=1, bbox=Row(xmin=-122.42, ymin=37.75, xmax=-122.41, ymax=37.76))]).write.mode( - "overwrite" - ).parquet(src) - client = OvertureClient(release="2024-07-01", _catalog_opener=open_fake_overture) - from pyspark.sql.types import ( - ArrayType, - DoubleType, - StringType, - StructField, - StructType, - ) - - schema = StructType( - [ - StructField("theme", StringType()), - StructField("type", StringType()), - StructField("href", StringType()), - StructField("asset_bbox", ArrayType(DoubleType())), - StructField("release", StringType()), - ] - ) - # local file path -> not a cloud scheme, so it routes to distributed-read anyway - # only when the caller forces it; here use a real local path and assert columns + idempotency - assets = spark.createDataFrame( - [("buildings", "building", src, [-122.52, 37.70, -122.36, 37.83], "2024-07-01")], - schema, - ) - out_dir = str(tmp_path / "dl") - meta = client.download( - assets, out_dir, bbox=(-122.45, 37.74, -122.40, 37.78), partitions=4 - ) - assert meta.columns == [ - "theme", - "type", - "source", - "path", - "out_file_sz", - "is_out_file_valid", - "last_update", - "asset_bbox", - "release", - "href", - ] - first = meta.collect() - assert len(first) == 1 and first[0]["is_out_file_valid"] is True - # idempotent re-run: same target, still valid, no error - meta2 = client.download( - assets, out_dir, bbox=(-122.45, 37.74, -122.40, 37.78), partitions=4 - ) - assert meta2.collect()[0]["is_out_file_valid"] is True -``` - -- [ ] **Step 2: Run; confirm TypeError (download takes no `bbox`) / AttributeError.** - `gbx:test:python --path test/sample/test_overture.py` - Expected: `TypeError: download() got an unexpected keyword argument 'bbox'` (or the method is absent). - -- [ ] **Step 3: Implement `download` with the routing.** Add `bbox` to the public signature (needed for the distributed pushdown; default `None` → read whole asset). Pin the kwargs from the spec contract plus `bbox`: -```python - _CLOUD_SCHEMES = ("s3://", "s3a://", "abfs://", "abfss://", "gs://", "wasbs://") - - def download( - self, - assets_df, - out_dir, - *, - bbox=None, - table=None, - validate=True, - max_tries=5, - partitions=None, - ) -> "DataFrame": - """Distributed download of discovered assets to out_dir (a Volume). - - Default path: distributed read + AOI rewrite with bbox-struct pushdown - (when assets are cloud-readable). Fallback: whole-file HTTP-href download. - table= UPSERTs the metadata to a Delta table keyed by - (theme, type, source). Serverless-safe; idempotent skip on valid targets. - """ - from pyspark.sql import functions as F - - assets = assets_df.select(*_DISCOVER_COLS) - n = partitions if partitions is not None else max(1, assets.count()) - - hrefs = [r["href"] for r in assets.select("href").distinct().collect()] - is_cloud = bool(hrefs) and all( - any(h.startswith(s) for s in self._CLOUD_SCHEMES) - or h.startswith("/") # local Volume / FUSE path is Spark-readable - for h in hrefs - ) - - if is_cloud: - meta = self._download_distributed( - assets, out_dir, bbox=bbox, validate=validate, partitions=n - ) - else: - meta = self._download_fallback( - assets, out_dir, validate=validate, max_tries=max_tries, partitions=n - ) - - if table is not None: - meta = self._merge_metadata(meta, table) # added in Task 8 - return meta -``` - Note for the implementer: until Task 8 lands `_merge_metadata`, keep the `table` branch out (the spec contract has `table=None` default, so this test passes without it). Add the `_merge_metadata` call in Task 8. - -- [ ] **Step 4: Re-run; confirm green.** - `gbx:test:python --path test/sample/test_overture.py` - Expected: 5 passed. - -- [ ] **Step 5: Commit.** - `git add python/geobrix/src/databricks/labs/gbx/sample/overture.py python/geobrix/test/sample/test_overture.py` - `git commit -m "feat(sample): OvertureClient.download path selection" -m "Public download() routes cloud-readable assets to the distributed read+AOI-rewrite path and http hrefs to the whole-file fallback, returns the pinned metadata schema (source aliased as path), and is idempotent on re-run. Partitions default to asset count." -m "Co-authored-by: Isaac"` - ---- - -### Task 8: metadata Delta MERGE table output - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/overture.py` -- Test: `python/geobrix/test/sample/test_overture.py` - -**Interfaces:** -- Consumes: `download` metadata DataFrame. -- Produces: - - private `OvertureClient._merge_metadata(meta_df, table) -> DataFrame` — first run creates `table` (Delta) from `meta_df`; subsequent runs UPSERT via `DeltaTable.merge` keyed by `(theme, type, source)` (whenMatchedUpdate the volatile cols `path, out_file_sz, is_out_file_valid, last_update, asset_bbox, release, href`; whenNotMatchedInsertAll). Returns `meta_df` unchanged. Mirrors `StacClient.repair`'s MERGE. - -- [ ] **Step 1: Failing test for create-then-upsert idempotency.** Requires Delta; gate with `importorskip`. -```python -# append to test_overture.py -def test_download_table_merge_idempotent(spark, tmp_path): - pytest.importorskip("delta") - # a SparkSession with Delta configured is required; skip if not available - try: - spark.sql("SELECT 1") # sanity - spark.range(1).write.format("delta").mode("overwrite").save( - str(tmp_path / "_delta_probe") - ) - except Exception: - pytest.skip("Delta not enabled on this local SparkSession") - - src = str(tmp_path / "m.parquet") - from pyspark.sql import Row - - spark.createDataFrame([Row(id=1, bbox=Row(xmin=-122.42, ymin=37.75, xmax=-122.41, ymax=37.76))]).write.mode( - "overwrite" - ).parquet(src) - client = OvertureClient(release="2024-07-01", _catalog_opener=open_fake_overture) - from pyspark.sql.types import ( - ArrayType, - DoubleType, - StringType, - StructField, - StructType, - ) - - schema = StructType( - [ - StructField("theme", StringType()), - StructField("type", StringType()), - StructField("href", StringType()), - StructField("asset_bbox", ArrayType(DoubleType())), - StructField("release", StringType()), - ] - ) - assets = spark.createDataFrame( - [("buildings", "building", src, [-122.52, 37.70, -122.36, 37.83], "2024-07-01")], - schema, - ) - table = "overture_meta_test" - out_dir = str(tmp_path / "dl2") - client.download(assets, out_dir, bbox=(-122.45, 37.74, -122.40, 37.78), table=table, partitions=2) - client.download(assets, out_dir, bbox=(-122.45, 37.74, -122.40, 37.78), table=table, partitions=2) - # MERGE keyed by (theme, type, source) -> still exactly one row, not two - assert spark.table(table).count() == 1 - spark.sql(f"DROP TABLE IF EXISTS {table}") -``` - Note: if the module-scoped `spark` fixture lacks Delta, add the Delta extension/catalog config to the fixture builder (`spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension`, `spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog`) guarded by an `importorskip("delta")`, OR keep this test skipped locally and exercise the MERGE in the optional Volume/cluster smoke (note below). Prefer configuring the fixture so the test runs in Docker where `delta-spark` is present. - -- [ ] **Step 2: Run; confirm failure (no `_merge_metadata` wired / table not created).** - `gbx:test:python --path test/sample/test_overture.py::test_download_table_merge_idempotent` - Expected: failure or skip if Delta unavailable; failure (`AnalysisException`/`AttributeError`) when Delta is present. - -- [ ] **Step 3: Implement `_merge_metadata` and wire it into `download`.** -```python - def _merge_metadata(self, meta_df, table): - """Create or UPSERT the metadata Delta table keyed by (theme, type, source).""" - from pyspark.sql import SparkSession - - spark = SparkSession.getActiveSession() - if not spark.catalog.tableExists(table): - meta_df.write.format("delta").mode("overwrite").saveAsTable(table) - return meta_df - - from delta.tables import DeltaTable - - dt = DeltaTable.forName(spark, table) - ( - dt.alias("t") - .merge( - meta_df.alias("u"), - "t.theme = u.theme AND t.type = u.type AND t.source = u.source", - ) - .whenMatchedUpdate( - set={ - "path": "u.path", - "out_file_sz": "u.out_file_sz", - "is_out_file_valid": "u.is_out_file_valid", - "last_update": "u.last_update", - "asset_bbox": "u.asset_bbox", - "release": "u.release", - "href": "u.href", - } - ) - .whenNotMatchedInsertAll() - .execute() - ) - return meta_df -``` - In `download`, keep the `if table is not None: meta = self._merge_metadata(meta, table)` branch (now real). - -- [ ] **Step 4: Re-run; confirm green (or skipped on a Delta-less local session, green in Docker).** - `gbx:test:python --path test/sample/test_overture.py` - Expected: all pass in Docker; locally the Delta test may skip. - -- [ ] **Step 5: Commit.** - `git add python/geobrix/src/databricks/labs/gbx/sample/overture.py python/geobrix/test/sample/test_overture.py` - `git commit -m "feat(sample): metadata Delta MERGE for Overture downloads" -m "table= persists/UPSERTs the per-asset metadata to a Delta table, idempotent MERGE keyed by (theme,type,source) (mirrors StacClient.repair). First run creates the table; re-runs update volatile cols so the catalog stays queryable and re-runnable." -m "Co-authored-by: Isaac"` - ---- - -### Task 9: `read` — from a Volume dir and from a metadata table/DataFrame - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/overture.py` -- Test: `python/geobrix/test/sample/test_overture.py` - -**Interfaces:** -- Consumes: downloaded parquet on a Volume; the metadata Delta table/DataFrame (`source`/`path` column). -- Produces: - - `OvertureClient.read(source, theme=None, type=None, bbox=None) -> DataFrame` — `source` may be (a) a Volume directory (read parquet recursively, filter by `theme`/`type` sub-path or columns), (b) a metadata Delta **table name** (string resolving to a table with a `source`/`path` column → union-read each per-asset path), or (c) a metadata **DataFrame** (same). `bbox` applies the `bbox`-struct AOI filter when a `bbox` struct column is present. - -- [ ] **Step 1: Failing test for both read modes.** -```python -# append to test_overture.py -def test_read_from_volume_dir(spark, tmp_path): - client = OvertureClient(release="2024-07-01", _catalog_opener=open_fake_overture) - out_dir = str(tmp_path / "rd") - target = os.path.join(out_dir, "buildings", "building") - from pyspark.sql import Row - - spark.createDataFrame( - [Row(id=1, bbox=Row(xmin=-122.42, ymin=37.75, xmax=-122.41, ymax=37.76))] - ).write.mode("overwrite").parquet(target) - df = client.read(out_dir) - assert df.count() == 1 - # bbox filter retains the in-AOI row - df2 = client.read(out_dir, bbox=(-122.45, 37.74, -122.40, 37.78)) - assert df2.count() == 1 - df3 = client.read(out_dir, bbox=(0, 0, 1, 1)) # disjoint - assert df3.count() == 0 - - -def test_read_from_metadata_dataframe(spark, tmp_path): - client = OvertureClient(release="2024-07-01", _catalog_opener=open_fake_overture) - target = str(tmp_path / "assetdir") - from pyspark.sql import Row - - spark.createDataFrame([Row(id=7)]).write.mode("overwrite").parquet(target) - from pyspark.sql import functions as F - - meta = spark.createDataFrame([(target,)], ["source"]).withColumn( - "path", F.col("source") - ) - df = client.read(meta) - assert df.collect()[0]["id"] == 7 -``` - -- [ ] **Step 2: Run; confirm AttributeError.** - `gbx:test:python --path test/sample/test_overture.py` - Expected: `AttributeError: ... 'read'`. - -- [ ] **Step 3: Implement `read`.** -```python - def read(self, source, theme=None, type=None, bbox=None) -> "DataFrame": - """Load downloaded GeoParquet back into Spark with an optional bbox AOI filter. - - source may be a Volume directory, a metadata Delta table NAME, or a - metadata DataFrame carrying a source/path column pointing at per-asset paths. - """ - from pyspark.sql import DataFrame as _DF - from pyspark.sql import SparkSession - from pyspark.sql import functions as F - - spark = SparkSession.getActiveSession() - - def _read_paths(paths): - # union-read each per-asset path (recursive parquet under each) - dfs = [spark.read.parquet(p) for p in paths] - out = dfs[0] - for d in dfs[1:]: - out = out.unionByName(d, allowMissingColumns=True) - return out - - if isinstance(source, _DF): - col = "source" if "source" in source.columns else "path" - paths = [r[col] for r in source.select(col).distinct().collect()] - df = _read_paths(paths) - elif isinstance(source, str) and spark.catalog.tableExists(source): - meta = spark.table(source) - col = "source" if "source" in meta.columns else "path" - paths = [r[col] for r in meta.select(col).distinct().collect()] - df = _read_paths(paths) - else: - # a Volume directory: read parquet recursively (per theme/type subdirs) - base = source - if theme is not None and type is not None: - import os - - base = os.path.join(source, theme, type) - df = spark.read.option("recursiveFileLookup", "true").parquet(base) - - if bbox is not None and "bbox" in df.columns: - from databricks.labs.gbx.sample._overture_discover import normalize_bbox - - minx, miny, maxx, maxy = normalize_bbox(bbox) - df = df.filter( - (F.col("bbox.xmin") <= F.lit(maxx)) - & (F.col("bbox.xmax") >= F.lit(minx)) - & (F.col("bbox.ymin") <= F.lit(maxy)) - & (F.col("bbox.ymax") >= F.lit(miny)) - ) - return df -``` - -- [ ] **Step 4: Re-run; confirm green.** - `gbx:test:python --path test/sample/test_overture.py` - Expected: 7 (+ Delta-gated) passed. - -- [ ] **Step 5: Commit.** - `git add python/geobrix/src/databricks/labs/gbx/sample/overture.py python/geobrix/test/sample/test_overture.py` - `git commit -m "feat(sample): OvertureClient.read from dir or metadata table" -m "read() loads downloaded GeoParquet back into Spark from a Volume directory, a metadata Delta table name, or a metadata DataFrame (source/path column), with an optional bbox-struct AOI filter. The metadata table can directly drive distributed reads." -m "Co-authored-by: Isaac"` - ---- - -### Task 10: `download_overture_aoi` convenience one-shot - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/overture.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/__init__.py` -- Test: `python/geobrix/test/sample/test_overture.py` - -**Interfaces:** -- Consumes: `OvertureClient.discover` + `download`. -- Produces: - - module fn `download_overture_aoi(bbox, out_dir, themes=None, release=None, table=None) -> DataFrame` — constructs a default `OvertureClient`, `discover`s the AOI, then `download`s (passing `bbox` for the AOI pushdown and `table`). Returns the metadata DataFrame. - -- [ ] **Step 1: Failing test (offline via injected opener through a default-client monkeypatch).** -```python -# append to test_overture.py -def test_download_overture_aoi_one_shot(spark, tmp_path, monkeypatch): - # Force the convenience fn's default client to use the fake opener (offline). - import databricks.labs.gbx.sample.overture as ov - - src = str(tmp_path / "aoi.parquet") - from pyspark.sql import Row - - spark.createDataFrame( - [Row(id=1, bbox=Row(xmin=-122.42, ymin=37.75, xmax=-122.41, ymax=37.76))] - ).write.mode("overwrite").parquet(src) - - orig_init = ov.OvertureClient.__init__ - - def patched_init(self, *a, **k): - k["_catalog_opener"] = open_fake_overture - orig_init(self, *a, **k) - - monkeypatch.setattr(ov.OvertureClient, "__init__", patched_init) - - # The fake catalog's SF building href points at s3://...; rewrite discover to our local src - orig_discover = ov.OvertureClient.discover - - def patched_discover(self, bbox, themes=None, release=None): - df = orig_discover(self, bbox, themes=themes, release=release) - from pyspark.sql import functions as F - - return df.withColumn("href", F.lit(src)) - - monkeypatch.setattr(ov.OvertureClient, "discover", patched_discover) - - out_dir = str(tmp_path / "oneshot") - meta = ov.download_overture_aoi( - (-122.45, 37.74, -122.40, 37.78), out_dir, themes=["buildings"], release="2024-07-01" - ) - rows = meta.collect() - assert len(rows) == 1 - assert rows[0]["theme"] == "buildings" - assert rows[0]["is_out_file_valid"] is True -``` - -- [ ] **Step 2: Run; confirm AttributeError (no `download_overture_aoi`).** - `gbx:test:python --path test/sample/test_overture.py` - Expected: `AttributeError: module ... has no attribute 'download_overture_aoi'`. - -- [ ] **Step 3: Implement `download_overture_aoi` and re-export it.** -```python -def download_overture_aoi(bbox, out_dir, themes=None, release=None, table=None) -> "DataFrame": - """One-shot: discover the AOI's Overture assets and download them to out_dir. - - Constructs a default OvertureClient, discovers (themes=None => all), then - downloads with the AOI bbox pushdown and an optional metadata Delta table. - """ - client = OvertureClient(release=release) - assets = client.discover(bbox, themes=themes, release=release) - return client.download(assets, out_dir, bbox=bbox, table=table) -``` - In `sample/__init__.py` add `download_overture_aoi` to the `overture` import and to `__all__`. - -- [ ] **Step 4: Re-run; confirm green.** - `gbx:test:python --path test/sample/test_overture.py` - Expected: all (non-Delta-gated) pass. - -- [ ] **Step 5: Commit.** - `git add python/geobrix/src/databricks/labs/gbx/sample/overture.py python/geobrix/src/databricks/labs/gbx/sample/__init__.py python/geobrix/test/sample/test_overture.py` - `git commit -m "feat(sample): download_overture_aoi one-shot convenience" -m "Module-level convenience that discovers an AOI's Overture assets and downloads them in one call (themes=None => all), passing the AOI bbox pushdown and optional metadata Delta table through. Re-exported from sample/__init__." -m "Co-authored-by: Isaac"` - ---- - -### Task 11: light-CI-lock wiring (deps + test dir registration) + final green run - -**Files:** -- Modify: `python/geobrix/requirements-pyrx-ci.in` (+ regenerate `requirements-pyrx-ci.txt`) -- Modify: `python/geobrix/requirements-dev-container.in` (+ regenerate `requirements-dev-container.txt`) -- Modify: `python/geobrix/pyproject.toml` (add `pystac` to a new/extended `overture` extra or the `stac` extra) -- Modify: `python/geobrix/test/conftest.py` (`_LIGHT_TEST_DIRS`) -- Modify: `.github/actions/pyrx_build/action.yml` (pytest dir list) -- Test: full light-suite run - -**Interfaces:** -- Consumes: the `pystac` runtime dependency introduced by `_overture_discover._open_catalog`. -- Produces: a CI environment that installs `pystac` and a test phase that RUNS `test/sample/`. - -- [ ] **Step 1: Add `pystac` to both `.in` files.** In `requirements-pyrx-ci.in`, add under a new comment block (mirroring the STAC block): -``` -# --- Overture static-STAC catalog traversal (gbx.sample.overture). pystac is the -# static-catalog reader (distinct from pystac-client, which is the search-API client -# the stac/ suite stubs). geopandas/pyarrow for parquet read are already pinned above. --- -pystac==1.11.0 -``` - Add the same `pystac==1.11.0` pin to `requirements-dev-container.in` under the "geospatial dev stack" block. (Confirm `1.11.0` resolves on the corp PyPI proxy; bump to the latest <2 that does.) - -- [ ] **Step 2: Add the `overture` extra to `pyproject.toml`.** Mirror the `stac` extra: -```toml -# Overture Maps data source (gbx.sample.overture): static-STAC catalog traversal. -# pystac is the static-catalog reader; geopandas/pyarrow (in [light]) do parquet read. -overture = [ - "pystac>=1.9,<2", -] -``` - -- [ ] **Step 3: Register `test/sample` in `conftest.py`.** Add `"sample"` to `_LIGHT_TEST_DIRS` and extend the "Light test dirs so far:" docstring line to include `sample`. - -- [ ] **Step 4: Register `test/sample` in the light CI phase.** In `.github/actions/pyrx_build/action.yml`, add `test/sample` to the pytest dir list and extend the inline comment listing the light dirs: -``` -pytest test/pyrx test/ds test/pyvx test/pygx test/pmtiles_light test/stac test/vizx test/sample -m "not integration" -v -``` - -- [ ] **Step 5: Regenerate the hash-pinned locks (in Docker, per the .in header instructions).** Dispatch via a Task subagent (long-running, touches the container): - - pyrx-ci: `uv pip compile --generate-hashes --python-version 3.12 --output-file requirements-pyrx-ci.txt requirements-pyrx-ci.in` (cwd `python/geobrix`). - - dev-container: `docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && uv pip compile --generate-hashes --python-version 3.12 --output-file requirements-dev-container.txt requirements-dev-container.in'`. - Expected: both `.txt` files gain hashed `pystac==...` (and any new transitive pins). - -- [ ] **Step 6: Run the full sample suite green (in Docker, all tests incl. Delta).** Dispatch via a Task subagent: - `gbx:test:python --path test/sample/ --log overture-sp1.log` - Expected: all of `test_overture_discover.py` + `test_overture.py` pass, including the Delta MERGE test (Docker has `delta-spark`). - -- [ ] **Step 7: Lint before commit.** - `gbx:lint:python --check` (verify isort/black/flake8 against the Docker formatter; reformat in-container if the host black differs). - Expected: clean. - -- [ ] **Step 8: Commit.** - `git add python/geobrix/requirements-pyrx-ci.in python/geobrix/requirements-pyrx-ci.txt python/geobrix/requirements-dev-container.in python/geobrix/requirements-dev-container.txt python/geobrix/pyproject.toml python/geobrix/test/conftest.py .github/actions/pyrx_build/action.yml` - `git commit -m "build(sample): light-CI-lock wiring for gbx.sample.overture" -m "Add pystac (static-STAC traversal) to both CI .in locks + regenerate hashed .txt; add the [overture] extra; register test/sample in both the heavy-skip _LIGHT_TEST_DIRS and the light-phase pytest dir list so the suite is collected in Docker/light CI and skipped in the heavy env. Full sample suite green." -m "Co-authored-by: Isaac"` - ---- - -### Task 12: capture validated performance gains (standing practice) - -**Files:** -- Create (if a gain is validated): `docs/superpowers/performance/README.md` (index, first time) + `docs/superpowers/performance/.md` (the pattern) -- Add: a thin pointer memory under the user's geobrix memory dir (slug + one-line `[[link]]`) - -**Interfaces:** -- Consumes: any measured distribution/throughput gain surfaced while building SP1 (e.g. bbox-struct pushdown reducing rows read; column-hash repartition keeping the AOI rewrite distributed vs a number-only repartition coalescing to serial). -- Produces: a recorded pattern with an applicability matrix, per the spec's performance methodology. - -- [ ] **Step 1: Decide whether SP1 produced a validated gain.** The likely candidate is "column-hash repartition keeps the Overture AOI rewrite distributed where number-only repartition coalesces to serial on Serverless" — already a known rule, so assess whether SP1 adds a *new* validated data point (e.g. measured row-reduction from bbox-struct pushdown on real Overture parquet during the optional Volume/cluster smoke). If no NEW gain is measured offline, record the assessment verdict ("no new gain beyond the existing repartition-by-column rule; pushdown row-reduction to be measured in the optional cluster smoke") and skip the corpus file. - -- [ ] **Step 2: If a gain is validated, write the corpus pattern file.** Create `docs/superpowers/performance/README.md` (if absent) as a one-line index, then `docs/superpowers/performance/overture-bbox-pushdown.md` with sections: problem → symptom/signature → the fix → applicability matrix (light-similar: other `sample`/`ds` distributed readers; heavy-same+similar: N/A — no heavy Overture path) → evidence/bench numbers → canonical code refs (`overture.py::_download_distributed`). - -- [ ] **Step 3: Add the paired thin pointer memory.** One line: slug + one-line summary that `[[links]]` to the corpus file (do not bloat MEMORY.md; keep under ~200 chars). - -- [ ] **Step 4: Commit (only the docs/memory, if created).** - `git add docs/superpowers/performance/` - `git commit -m "docs(perf): capture Overture distribution gain assessment" -m "Per the tiling performance methodology: record the SP1 distribution finding (bbox-struct pushdown + column-hash repartition keeping the AOI rewrite distributed) with its applicability matrix (light-similar readers; no heavy Overture path), or the not-applicable verdict when no new gain beyond the existing repartition-by-column rule was measured." -m "Co-authored-by: Isaac"` - ---- - -## Optional Volume / cluster smoke (not required for SP1 unit tests) - -All SP1 unit tests above run **offline** (injected `_catalog_opener` + injected `_get_fn` + a local `SparkSession`); Docker is needed only for the Delta MERGE test (Task 8) and the lock regeneration (Task 11). The following are **optional** validations, deferred but noted: - -- A real-network discover against `https://stac.overturemaps.org/catalog.json` (confirms the live catalog shape matches `traverse_catalog`'s assumptions; spec open item). -- A cluster/Serverless smoke confirming Spark can directly read Overture's public cloud paths (`s3://overturemaps-us-west-2` / `abfs://...`) for the distributed default — the spec flags this as a "validate during SP1" open item. If direct cloud read is unavailable/requester-pays-blocked, the HTTP-href fallback becomes the primary path (the code already supports both); record the verdict in the perf corpus (Task 12). diff --git a/docs/superpowers/plans/2026-06-27-helios-sp2-vizx-viewers.md b/docs/superpowers/plans/2026-06-27-helios-sp2-vizx-viewers.md deleted file mode 100644 index 0edff427a..000000000 --- a/docs/superpowers/plans/2026-06-27-helios-sp2-vizx-viewers.md +++ /dev/null @@ -1,1403 +0,0 @@ -# VizX Viewers (SP2) Implementation Plan - -**REQUIRED SUB-SKILL:** Use the `superpowers:test-driven-development` skill for every task — write the failing test first, watch it fail for the right reason, then write the minimal implementation to make it pass. Use `superpowers:verification-before-completion` before claiming any task done. - -**Goal:** Ship net-new public `gbx.vizx` viewers — `plot_pmtiles` (interactive MapLibre GL JS + pmtiles.js, base64-embedded archive, with a Python-side static fallback) and `plot_cog` (rasterio overview read over a contextily basemap) — plus a reusable driver-side PMTiles inspector `gbx.pmtiles.pmtiles_info`. These let the Helios notebook series *show* its PMTiles/COG output inline in a Databricks notebook. - -**Architecture:** -- **Inspector (`pmtiles/_inspect.py`):** `pmtiles_info(path) -> dict` reads a `.pmtiles` header via the existing `pmtiles` PyPI dep (`Reader.header()` / `Reader.metadata()`), normalizing `tile_type` (enum → string), min/max zoom, bounds (e7 → degrees), tile count, and tilejson-ish metadata. Consumed by both viewers and the static fallback. -- **Interactive path (`vizx/_pmtiles.py`):** build a self-contained HTML page that CDN-loads pinned `maplibre-gl` + `pmtiles` JS, registers the `pmtiles://` protocol, and feeds the archive bytes as a base64 in-browser `pmtiles.FileSource` — no HTTP server, no remote range requests. Vector (`tile_type == "mvt"`) → a MapLibre vector layer; raster (png/jpeg/webp) → a raster layer. Rendered through the existing `_interactive._notebook_display_html()` displayHTML channel with its IPython fallback chain. -- **Static fallback (`vizx/_pmtiles.py`):** the interactive map is the default; when the base64-embedded archive would exceed `max_embed_mb` (base64 bloats ~33%) and `fallback=True` (default), degrade to a static render — decode tiles with the Python `pmtiles` reader and composite: raster → reuse `vizx.plot_raster`; vector → decode MVT (`mapbox_vector_tile.decode`, already a pyvx dep) to shapely geometries → reuse `vizx.plot_static` over a contextily basemap. `fallback=False` raises instead of degrading; `max_embed_mb=0` deliberately forces the static render (GitHub-renderable committed notebooks). No new dep. -- **COG (`vizx/_cog.py`):** `plot_cog(path, *, band=None, **kw)` → `rasterio` overview/decimated read → `plot_raster` over a contextily basemap; the interactive-map raster-source injection is **optional** (decided in Task 7 — default static-only). - -**Tech Stack:** Python 3.12, PySpark/Spark Connect (driver-side only here — no executors), `pmtiles` (present), `rasterio` (present), `mapbox_vector_tile` (present, a pyvx dep), `matplotlib`/`geopandas`/`contextily` ([vizx] extra, present). MapLibre GL JS + pmtiles.js are CDN-loaded at **pinned versions** (no Python dep). - ---- - -## Global Constraints - -- **Python 3.12+.** Pure driver-side code; no Spark executors, no Serverless config knobs. -- **`[vizx]` extra, import-guarded.** Every public function calls `assert_viz_available()` (from `vizx/_env.py`) before importing matplotlib/geopandas/contextily. The inspector lives in `gbx.pmtiles` and uses only the `pmtiles` dep (already present, light tier) — it does NOT require `[vizx]`. -- **NO new Python deps.** The interactive path is CDN JS. The static fallback reuses `pmtiles` + `mapbox_vector_tile` + `rasterio` + the `[vizx]` stack, all present. `plot_cog` uses `rasterio` only. (If, and only if, `plot_cog` later adopts `rio-tiler` for nicer overview selection, run the full light-CI-lock checklist: add to `requirements-pyrx-ci.in` AND `requirements-dev-container.in`, recompile the hashed `.txt`, then re-pin. This plan does NOT adopt rio-tiler — `band=`/decimated read is sufficient.) -- **Pin CDN JS versions** for reproducibility: `maplibre-gl@4.7.1` (`https://unpkg.com/maplibre-gl@4.7.1/dist/maplibre-gl.js` + `maplibre-gl.css`) and `pmtiles@3.2.1` (`https://unpkg.com/pmtiles@3.2.1/dist/pmtiles.js`). Defined as module constants so tests assert the exact pinned URLs. -- **TDD** — failing test first, minimal impl, per-task green run + commit. -- **Commit hygiene** — subject ≤72 chars; a WHY body for non-trivial commits. End commit messages with the `Co-authored-by: Isaac` trailer. -- **Docs voice** — no internal planning vocabulary (no wave numbers) in any docstring or docs page. -- **Docker:** none of these unit tests need Docker — they are offline/driver-only. They DO need the `[vizx]` extra deps (`matplotlib`, `geopandas`, `contextily`) plus `pmtiles`, `mapbox_vector_tile`, `rasterio` present in the env. The dev container has all of these; `gbx:test:python --path python/geobrix/test/vizx/...` runs them. If a bare host venv lacks them, `pip install 'geobrix[vizx]'` then ensure `pmtiles`/`mapbox-vector-tile` are installed (they ship with `[light]`). - ---- - -### Task 1: PMTiles inspector — `pmtiles_info` - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/pmtiles/_inspect.py` -- Test: `python/geobrix/test/pmtiles_light/test_inspect.py` - -**Interfaces:** -- Produces: `pmtiles_info(path: str | bytes | bytearray) -> dict` with keys `tile_type` (str: `"mvt"|"png"|"jpeg"|"webp"|"avif"|"unknown"`), `min_zoom` (int), `max_zoom` (int), `bounds` (tuple `(min_lon, min_lat, max_lon, max_lat)` in degrees), `center` (tuple `(lon, lat, zoom)`), `tile_count` (int), `metadata` (dict), `tile_compression` (str). -- Consumes: the `pmtiles` PyPI dep (`pmtiles.reader.Reader`, `MemorySource`, `MmapSource`, `all_tiles`; `pmtiles.tile.TileType`, `Compression`). - -**Steps:** - -- [ ] **Step 1** — Write the failing test. The fixture builds a tiny in-memory PMTiles archive (raster PNG + vector MVT) with the SAME `Writer` path the light agg uses, so the test is self-contained and needs no committed binary. - -```python -# python/geobrix/test/pmtiles_light/test_inspect.py -"""Offline tests for the driver-side PMTiles inspector (pmtiles_info).""" - -import io - -import pytest -from pmtiles.tile import Compression, TileType, zxy_to_tileid -from pmtiles.writer import Writer - -_PNG = b"\x89PNG\r\n\x1a\n" + b"\x00" * 16 # sniffs as PNG -_MVT = b"mvt-payload\x00\x01\x02" # non-magic bytes => MVT - - -def _build_archive(tiles, tile_type, *, name="demo"): - """tiles: list of (z, x, y, payload). Returns PMTiles v3 bytes over SF.""" - buf = io.BytesIO() - w = Writer(buf) - zs = [z for z, _, _, _ in tiles] - header = { - "tile_type": tile_type, - "tile_compression": Compression.NONE, - "internal_compression": Compression.GZIP, - "min_zoom": min(zs), - "max_zoom": max(zs), - "min_lon_e7": int(-122.52 * 1e7), - "min_lat_e7": int(37.70 * 1e7), - "max_lon_e7": int(-122.35 * 1e7), - "max_lat_e7": int(37.83 * 1e7), - "center_zoom": min(zs), - "center_lon_e7": int(-122.44 * 1e7), - "center_lat_e7": int(37.76 * 1e7), - } - for z, x, y, payload in sorted(tiles, key=lambda t: zxy_to_tileid(t[0], t[1], t[2])): - w.write_tile(zxy_to_tileid(z, x, y), payload) - w.finalize(header, {"name": name, "vector_layers": [{"id": "demo"}]}) - return buf.getvalue() - - -@pytest.fixture -def raster_pmtiles(): - return _build_archive([(0, 0, 0, _PNG), (1, 0, 0, _PNG)], TileType.PNG) - - -@pytest.fixture -def vector_pmtiles(): - return _build_archive([(10, 163, 395, _MVT)], TileType.MVT, name="bldgs") - - -def test_info_from_bytes_raster(raster_pmtiles): - from databricks.labs.gbx.pmtiles import pmtiles_info - - info = pmtiles_info(raster_pmtiles) - assert info["tile_type"] == "png" - assert info["min_zoom"] == 0 - assert info["max_zoom"] == 1 - assert info["tile_count"] == 2 - minlon, minlat, maxlon, maxlat = info["bounds"] - assert -122.6 < minlon < maxlon < -122.3 - assert 37.6 < minlat < maxlat < 37.9 - assert info["metadata"].get("name") == "demo" - assert info["tile_compression"] == "none" - - -def test_info_from_bytes_vector(vector_pmtiles): - from databricks.labs.gbx.pmtiles import pmtiles_info - - info = pmtiles_info(vector_pmtiles) - assert info["tile_type"] == "mvt" - assert info["min_zoom"] == info["max_zoom"] == 10 - assert info["tile_count"] == 1 - assert info["metadata"].get("name") == "bldgs" - - -def test_info_from_path(raster_pmtiles, tmp_path): - from databricks.labs.gbx.pmtiles import pmtiles_info - - p = tmp_path / "r.pmtiles" - p.write_bytes(raster_pmtiles) - info = pmtiles_info(str(p)) - assert info["tile_type"] == "png" - assert info["tile_count"] == 2 - - -def test_info_strips_dbfs_scheme(raster_pmtiles, tmp_path): - # Databricks Volume paths often arrive scheme-qualified; the bare FUSE path - # is what the reader opens. Strip dbfs:/file: like plot_file does. - from databricks.labs.gbx.pmtiles import pmtiles_info - - p = tmp_path / "r.pmtiles" - p.write_bytes(raster_pmtiles) - info = pmtiles_info("dbfs:" + str(p)) - assert info["tile_count"] == 2 - - -def test_center_tuple(vector_pmtiles): - from databricks.labs.gbx.pmtiles import pmtiles_info - - lon, lat, zoom = pmtiles_info(vector_pmtiles)["center"] - assert -122.6 < lon < -122.3 and 37.6 < lat < 37.9 and zoom == 10 -``` - -- [ ] **Step 2** — Run, expect failure (import error / no `pmtiles_info`): - -``` -gbx:test:python --path python/geobrix/test/pmtiles_light/test_inspect.py -``` -Expected: `ImportError: cannot import name 'pmtiles_info'` (collection error) on all 5 tests. - -- [ ] **Step 3** — Minimal implementation: - -```python -# python/geobrix/src/databricks/labs/gbx/pmtiles/_inspect.py -"""Driver-side PMTiles inspector. Spark-side PMTiles read is unsupported, so a -local-driver header reader is broadly useful and is consumed by the gbx.vizx -viewers (type detection + static fallback). Uses the existing `pmtiles` dep.""" - -from __future__ import annotations - -from typing import Union - -from pmtiles.reader import MemorySource, Reader, all_tiles -from pmtiles.tile import Compression, TileType - -# TileType / Compression enum -> the lowercase string keys the viewers branch on. -_TILE_TYPE_NAME = { - TileType.MVT: "mvt", - TileType.PNG: "png", - TileType.JPEG: "jpeg", - TileType.WEBP: "webp", - TileType.AVIF: "avif", - TileType.UNKNOWN: "unknown", -} -_COMPRESSION_NAME = { - Compression.UNKNOWN: "unknown", - Compression.NONE: "none", - Compression.GZIP: "gzip", - Compression.BROTLI: "brotli", - Compression.ZSTD: "zstd", -} - - -def _strip_scheme(path: str) -> str: - for scheme in ("dbfs:", "file:"): - if path.startswith(scheme): - path = path[len(scheme) :] - break - if path.startswith("//"): - path = "/" + path.lstrip("/") - return path - - -def _read_bytes(path_or_bytes: Union[str, bytes, bytearray]) -> bytes: - if isinstance(path_or_bytes, (bytes, bytearray)): - return bytes(path_or_bytes) - with open(_strip_scheme(str(path_or_bytes)), "rb") as f: - return f.read() - - -def pmtiles_info(path: Union[str, bytes, bytearray]) -> dict: - """Parse a .pmtiles archive header into a plain dict. - - ``path`` is a filesystem path (Volume/DBFS scheme prefixes are stripped) or - the archive bytes. Returns ``tile_type`` (lowercase string), ``min_zoom`` / - ``max_zoom`` (int), ``bounds`` (min_lon, min_lat, max_lon, max_lat degrees), - ``center`` (lon, lat, zoom), ``tile_count`` (int), ``metadata`` (dict), and - ``tile_compression`` (lowercase string). Driver-side only. - """ - data = _read_bytes(path_or_bytes) - source = MemorySource(data) - reader = Reader(source) - h = reader.header() - metadata = reader.metadata() - tile_count = sum(1 for _ in all_tiles(MemorySource(data))) - return { - "tile_type": _TILE_TYPE_NAME.get(h["tile_type"], "unknown"), - "tile_compression": _COMPRESSION_NAME.get(h["tile_compression"], "unknown"), - "min_zoom": int(h["min_zoom"]), - "max_zoom": int(h["max_zoom"]), - "bounds": ( - h["min_lon_e7"] / 1e7, - h["min_lat_e7"] / 1e7, - h["max_lon_e7"] / 1e7, - h["max_lat_e7"] / 1e7, - ), - "center": ( - h["center_lon_e7"] / 1e7, - h["center_lat_e7"] / 1e7, - int(h["center_zoom"]), - ), - "tile_count": int(tile_count), - "metadata": dict(metadata) if metadata else {}, - } -``` - -Wire the lazy re-export in `pmtiles/__init__.py` — extend `__all__` and `__getattr__` so heavy-tier imports still don't pull `pandas`/`_agg_light`, but `pmtiles_info` (which only needs `pmtiles`, present in both tiers) imports cleanly: - -```python -# edit python/geobrix/src/databricks/labs/gbx/pmtiles/__init__.py -__all__ = ["register_pmtiles_agg", "pmtiles_info"] - - -def __getattr__(name): - if name == "register_pmtiles_agg": - from databricks.labs.gbx.pmtiles._agg_light import register_pmtiles_agg - - return register_pmtiles_agg - if name == "pmtiles_info": - from databricks.labs.gbx.pmtiles._inspect import pmtiles_info - - return pmtiles_info - raise AttributeError(f"module {__name__!r} has no attribute {name!r}") -``` - -- [ ] **Step 4** — Run, expect green: - -``` -gbx:test:python --path python/geobrix/test/pmtiles_light/test_inspect.py -``` -Expected: `5 passed`. - -- [ ] **Step 5** — Commit: - -``` -git add python/geobrix/src/databricks/labs/gbx/pmtiles/_inspect.py \ - python/geobrix/src/databricks/labs/gbx/pmtiles/__init__.py \ - python/geobrix/test/pmtiles_light/test_inspect.py -git commit -m "feat(pmtiles): driver-side pmtiles_info header inspector - -Spark-side PMTiles read is unsupported, so a local-driver header reader -is needed by the VizX viewers for vector/raster type detection and the -static fallback. Lazy re-export keeps heavy-tier imports pandas-free. - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: Vector/raster type classification helper - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py` (helpers only this task) -- Test: `python/geobrix/test/vizx/test_pmtiles.py` (helper tests this task) - -**Interfaces:** -- Produces: `_is_raster_type(tile_type: str) -> bool` and `_archive_bytes(path_or_bytes) -> bytes` (mirrors the inspector's path/scheme handling) in `vizx/_pmtiles.py`. -- Consumes: `pmtiles_info` from Task 1. - -**Steps:** - -- [ ] **Step 1** — Write the failing test: - -```python -# python/geobrix/test/vizx/test_pmtiles.py -"""Offline tests for plot_pmtiles (interactive HTML + static fallback).""" - -import io - -import pytest -from pmtiles.tile import Compression, TileType, zxy_to_tileid -from pmtiles.writer import Writer - -_PNG = b"\x89PNG\r\n\x1a\n" + b"\x00" * 16 - - -def _build_archive(tiles, tile_type, *, name="demo"): - buf = io.BytesIO() - w = Writer(buf) - zs = [z for z, _, _, _ in tiles] - header = { - "tile_type": tile_type, - "tile_compression": Compression.NONE, - "internal_compression": Compression.GZIP, - "min_zoom": min(zs), - "max_zoom": max(zs), - "min_lon_e7": int(-122.52 * 1e7), - "min_lat_e7": int(37.70 * 1e7), - "max_lon_e7": int(-122.35 * 1e7), - "max_lat_e7": int(37.83 * 1e7), - "center_zoom": min(zs), - "center_lon_e7": int(-122.44 * 1e7), - "center_lat_e7": int(37.76 * 1e7), - } - for z, x, y, payload in sorted(tiles, key=lambda t: zxy_to_tileid(t[0], t[1], t[2])): - w.write_tile(zxy_to_tileid(z, x, y), payload) - w.finalize(header, {"name": name, "vector_layers": [{"id": "demo"}]}) - return buf.getvalue() - - -def test_is_raster_type(): - from databricks.labs.gbx.vizx import _pmtiles as p - - assert p._is_raster_type("png") is True - assert p._is_raster_type("jpeg") is True - assert p._is_raster_type("webp") is True - assert p._is_raster_type("avif") is True - assert p._is_raster_type("mvt") is False - assert p._is_raster_type("unknown") is False - - -def test_archive_bytes_passthrough_and_path(tmp_path): - from databricks.labs.gbx.vizx import _pmtiles as p - - raw = _build_archive([(0, 0, 0, _PNG)], TileType.PNG) - assert p._archive_bytes(raw) == raw - f = tmp_path / "a.pmtiles" - f.write_bytes(raw) - assert p._archive_bytes(str(f)) == raw - assert p._archive_bytes("dbfs:" + str(f)) == raw -``` - -- [ ] **Step 2** — Run, expect failure: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_pmtiles.py -``` -Expected: `ModuleNotFoundError` / `ImportError: cannot import name '_pmtiles'` on collection. - -- [ ] **Step 3** — Minimal implementation (create `vizx/_pmtiles.py` with helpers + the pinned CDN constants used in later tasks): - -```python -# python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py -"""Inline PMTiles viewer for gbx.vizx. - -Interactive path: a self-contained MapLibre GL JS + pmtiles.js HTML page -(CDN-loaded at pinned versions) with the archive base64-embedded as an -in-browser FileSource — no tile server, no remote range requests. Interactive -by default; when the embedded archive would exceed ``max_embed_mb`` and -``fallback`` is set (the default), decode tiles on the driver and reuse -plot_raster (raster) / plot_static (vector) over a contextily basemap -(``max_embed_mb=0`` forces this static path). Requires the [vizx] extra for the -static fallback. Driver-side only. -""" - -from __future__ import annotations - -import base64 -from typing import Union - -# Pinned CDN versions for reproducibility (asserted by tests). -_MAPLIBRE_JS = "https://unpkg.com/maplibre-gl@4.7.1/dist/maplibre-gl.js" -_MAPLIBRE_CSS = "https://unpkg.com/maplibre-gl@4.7.1/dist/maplibre-gl.css" -_PMTILES_JS = "https://unpkg.com/pmtiles@3.2.1/dist/pmtiles.js" - -_RASTER_TYPES = frozenset({"png", "jpeg", "webp", "avif"}) - - -def _is_raster_type(tile_type: str) -> bool: - """True for image tile types (raster layer); False for mvt/unknown (vector).""" - return tile_type in _RASTER_TYPES - - -def _strip_scheme(path: str) -> str: - for scheme in ("dbfs:", "file:"): - if path.startswith(scheme): - path = path[len(scheme) :] - break - if path.startswith("//"): - path = "/" + path.lstrip("/") - return path - - -def _archive_bytes(path_or_bytes: Union[str, bytes, bytearray]) -> bytes: - """Read a .pmtiles path (Volume/DBFS scheme stripped) or pass bytes through.""" - if isinstance(path_or_bytes, (bytes, bytearray)): - return bytes(path_or_bytes) - with open(_strip_scheme(str(path_or_bytes)), "rb") as f: - return f.read() -``` - -- [ ] **Step 4** — Run, expect green: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_pmtiles.py -``` -Expected: `2 passed`. - -- [ ] **Step 5** — Commit: - -``` -git add python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py \ - python/geobrix/test/vizx/test_pmtiles.py -git commit -m "feat(vizx): pmtiles type-detect + archive-bytes helpers - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: Interactive HTML builder (`_build_pmtiles_html`) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py` -- Test: `python/geobrix/test/vizx/test_pmtiles.py` - -**Interfaces:** -- Produces: `_build_pmtiles_html(archive_b64: str, info: dict, *, style=None) -> str` in `vizx/_pmtiles.py`. -- Consumes: the pinned CDN constants; `info` dict shape from `pmtiles_info`. - -**Steps:** - -- [ ] **Step 1** — Write the failing test (assert HTML structure: pinned CDN script tags, base64 source embed, protocol registration, vector-vs-raster layer): - -```python -# append to python/geobrix/test/vizx/test_pmtiles.py -def _info(tile_type, *, min_zoom=0, max_zoom=2): - return { - "tile_type": tile_type, - "tile_compression": "none", - "min_zoom": min_zoom, - "max_zoom": max_zoom, - "bounds": (-122.52, 37.70, -122.35, 37.83), - "center": (-122.44, 37.76, min_zoom), - "tile_count": 3, - "metadata": {"vector_layers": [{"id": "demo"}]}, - } - - -def test_build_html_pins_cdn_versions(): - from databricks.labs.gbx.vizx import _pmtiles as p - - html = p._build_pmtiles_html("QUJD", _info("png")) - assert "maplibre-gl@4.7.1/dist/maplibre-gl.js" in html - assert "maplibre-gl@4.7.1/dist/maplibre-gl.css" in html - assert "pmtiles@3.2.1/dist/pmtiles.js" in html - - -def test_build_html_embeds_base64_and_registers_protocol(): - from databricks.labs.gbx.vizx import _pmtiles as p - - html = p._build_pmtiles_html("QUJDREVG", _info("png")) - assert "QUJDREVG" in html # the base64 archive is embedded inline - assert "new pmtiles.Protocol" in html - assert "addProtocol" in html - assert "pmtiles.FileSource" in html or "FileSource" in html - assert "pmtiles://" in html - - -def test_build_html_raster_layer_for_png(): - from databricks.labs.gbx.vizx import _pmtiles as p - - html = p._build_pmtiles_html("QUJD", _info("png")) - assert '"type": "raster"' in html or "'type': 'raster'" in html or "type: \"raster\"" in html - - -def test_build_html_vector_layer_for_mvt(): - from databricks.labs.gbx.vizx import _pmtiles as p - - html = p._build_pmtiles_html("QUJD", _info("mvt")) - assert '"type": "vector"' in html or "type: \"vector\"" in html - # the source-layer id from the metadata vector_layers drives the fill layer - assert "demo" in html - - -def test_build_html_honors_custom_style(): - from databricks.labs.gbx.vizx import _pmtiles as p - - html = p._build_pmtiles_html("QUJD", _info("mvt"), style={"version": 8, "layers": []}) - assert '"version": 8' in html -``` - -- [ ] **Step 2** — Run, expect failure: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_pmtiles.py -``` -Expected: 5 new tests fail with `AttributeError: module ... has no attribute '_build_pmtiles_html'`. - -- [ ] **Step 3** — Minimal implementation (append to `_pmtiles.py`): - -```python -# append to python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py -import json - - -def _default_style(info: dict, source_name: str) -> dict: - """A minimal MapLibre style: a pmtiles:// source + one raster or vector layer.""" - is_raster = _is_raster_type(info["tile_type"]) - source = { - "type": "raster" if is_raster else "vector", - "url": f"pmtiles://{source_name}", - } - if is_raster: - source["tileSize"] = 256 - layers = [{"id": "tiles", "type": "raster", "source": source_name}] - else: - # Vector: one fill + one line layer per declared source-layer (MVT layer - # name). The pmtiles metadata's vector_layers carries those ids; fall - # back to a single "layer0" when absent. - vlayers = info.get("metadata", {}).get("vector_layers") or [{"id": "layer0"}] - layers = [] - for vl in vlayers: - sl = vl.get("id", "layer0") - layers.append( - { - "id": f"{sl}-fill", - "type": "fill", - "source": source_name, - "source-layer": sl, - "paint": {"fill-color": "#3388ff", "fill-opacity": 0.4}, - } - ) - layers.append( - { - "id": f"{sl}-line", - "type": "line", - "source": source_name, - "source-layer": sl, - "paint": {"line-color": "#1144aa", "line-width": 0.5}, - } - ) - return {"version": 8, "sources": {source_name: source}, "layers": layers} - - -def _build_pmtiles_html(archive_b64: str, info: dict, *, style=None) -> str: - """Build a self-contained MapLibre GL JS + pmtiles.js page (CDN-pinned). - - The archive bytes ride inline as ``archive_b64`` and are wrapped in an - in-browser ``pmtiles.FileSource`` (decoded from base64) registered under the - ``pmtiles://`` protocol, so the map streams entirely client-side — no tile - server, no remote range requests. - """ - source_name = "gbx" - map_style = style if style is not None else _default_style(info, source_name) - style_json = json.dumps(map_style) - minlon, minlat, maxlon, maxlat = info["bounds"] - clon, clat, czoom = info["center"] - return f""" - - - - - - - -
- -""" -``` - -- [ ] **Step 4** — Run, expect green: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_pmtiles.py -``` -Expected: all tests pass (7 total in the file so far). - -- [ ] **Step 5** — Commit: - -``` -git add python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py \ - python/geobrix/test/vizx/test_pmtiles.py -git commit -m "feat(vizx): MapLibre+pmtiles.js HTML builder (base64 FileSource) - -CDN versions pinned (maplibre-gl 4.7.1, pmtiles 3.2.1); archive rides -inline as a base64 in-browser FileSource so the map streams client-side -with no tile server. Vector vs raster layer chosen from the header type. - -Co-authored-by: Isaac" -``` - ---- - -### Task 4: Raster static fallback (`_static_raster_fallback`) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py` -- Test: `python/geobrix/test/vizx/test_pmtiles.py` - -**Interfaces:** -- Produces: `_static_raster_fallback(data: bytes, info: dict, **plot_kw) -> None` in `vizx/_pmtiles.py` — picks the lowest-zoom tile, decodes the image payload, hands it to `vizx.plot_raster`. -- Consumes: `pmtiles.reader.all_tiles`; `vizx.plot_raster`. - -**Steps:** - -- [ ] **Step 1** — Write the failing test. To make a *real* decodable raster tile, build a tiny PNG with rasterio/matplotlib in the fixture so `plot_raster` actually decodes it: - -```python -# append to python/geobrix/test/vizx/test_pmtiles.py -def _real_png_tile(): - # A real 8x8 RGB PNG so plot_raster's rasterio MemoryFile can decode it. - import io as _io - - import matplotlib - matplotlib.use("Agg") - import numpy as np - from matplotlib.image import imsave - - buf = _io.BytesIO() - imsave(buf, (np.random.rand(8, 8, 3)), format="png") - return buf.getvalue() - - -def test_static_raster_fallback_calls_plot_raster(monkeypatch): - from databricks.labs.gbx.vizx import _pmtiles as p - - png = _real_png_tile() - archive = _build_archive([(0, 0, 0, png)], TileType.PNG) - captured = {} - monkeypatch.setattr( - "databricks.labs.gbx.vizx.plot_raster", - lambda raster_bytes, **kw: captured.update(n=len(raster_bytes)), - ) - info = p_info = __import__( - "databricks.labs.gbx.pmtiles", fromlist=["pmtiles_info"] - ).pmtiles_info(archive) - p._static_raster_fallback(archive, info) - assert captured["n"] == len(png) # the decoded lowest-zoom tile bytes -``` - -- [ ] **Step 2** — Run, expect failure: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_pmtiles.py -``` -Expected: `AttributeError: ... '_static_raster_fallback'`. - -- [ ] **Step 3** — Minimal implementation (append to `_pmtiles.py`): - -```python -# append to python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py -from pmtiles.reader import MemorySource, all_tiles # noqa: E402 -from pmtiles.tile import tileid_to_zxy # noqa: E402 - - -def _lowest_zoom_tile(data: bytes): - """Return (z, x, y, payload) for the lowest-zoom tile (the coarsest overview).""" - best = None - for tileid, payload in all_tiles(MemorySource(data)): - z, x, y = tileid_to_zxy(tileid) - if best is None or z < best[0]: - best = (z, x, y, payload) - return best - - -def _static_raster_fallback(data: bytes, info: dict, **plot_kw) -> None: - """Decode the coarsest raster tile and render it via plot_raster.""" - from databricks.labs.gbx.vizx import plot_raster - - tile = _lowest_zoom_tile(data) - if tile is None: - raise ValueError("plot_pmtiles: archive has no tiles to render") - plot_raster(tile[3], **plot_kw) -``` - -- [ ] **Step 4** — Run, expect green: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_pmtiles.py -``` -Expected: all pass. - -- [ ] **Step 5** — Commit: - -``` -git add python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py \ - python/geobrix/test/vizx/test_pmtiles.py -git commit -m "feat(vizx): raster pmtiles static fallback via plot_raster - -Co-authored-by: Isaac" -``` - ---- - -### Task 5: Vector static fallback (`_static_vector_fallback`) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py` -- Test: `python/geobrix/test/vizx/test_pmtiles.py` - -**Interfaces:** -- Produces: `_static_vector_fallback(data: bytes, info: dict, **plot_kw) -> object` — decode each MVT tile to geometries (tile-local → WGS84 via `pyvx._mvt._tile_bounds`), build a GeoDataFrame, render via `vizx.plot_static` over a contextily basemap. -- Consumes: `mapbox_vector_tile.decode`; `pyvx._mvt` (`_tile_bounds`); `geopandas`; `vizx.plot_static`. - -**Steps:** - -- [ ] **Step 1** — Write the failing test. Build a vector PMTiles whose MVT tile holds a real geometry, assert the fallback yields a non-empty GeoDataFrame in EPSG:4326 and calls plot_static: - -```python -# append to python/geobrix/test/vizx/test_pmtiles.py -def _real_mvt_tile(z, x, y): - # Encode a polygon in tile-local pixel space for tile (z,x,y) (origin NW), - # the same convention pyvx writes, so the fallback reprojects it back to 4326. - import mapbox_vector_tile as mvt - from shapely.geometry import box - - return mvt.encode( - {"name": "demo", "features": [ - {"geometry": box(1000, 1000, 3000, 3000), "properties": {"v": 1}}]}, - default_options={"extents": 4096, "y_coord_down": True}, - ) - - -def test_static_vector_fallback_builds_gdf_and_plots(monkeypatch): - import geopandas as gpd - - from databricks.labs.gbx.vizx import _pmtiles as p - - z, x, y = 10, 163, 395 # an SF-area tile - blob = _real_mvt_tile(z, x, y) - archive = _build_archive([(z, x, y, blob)], TileType.MVT) - info = __import__( - "databricks.labs.gbx.pmtiles", fromlist=["pmtiles_info"] - ).pmtiles_info(archive) - - captured = {} - - def _fake_plot_static(gdf, **kw): - captured["gdf"] = gdf - captured["kw"] = kw - return "AX" - - monkeypatch.setattr( - "databricks.labs.gbx.vizx.plot_static", _fake_plot_static - ) - out = p._static_vector_fallback(archive, info, basemap=False) - assert out == "AX" - gdf = captured["gdf"] - assert isinstance(gdf, gpd.GeoDataFrame) - assert len(gdf) >= 1 - assert gdf.crs.to_epsg() == 4326 - # geometry reprojected into the SF tile's lon/lat extent - minx, miny, maxx, maxy = gdf.total_bounds - assert -123 < minx < maxx < -121 and 37 < miny < maxy < 39 -``` - -- [ ] **Step 2** — Run, expect failure: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_pmtiles.py -``` -Expected: `AttributeError: ... '_static_vector_fallback'`. - -- [ ] **Step 3** — Minimal implementation (append to `_pmtiles.py`): - -```python -# append to python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py -def _decode_mvt_to_geoms(payload: bytes, z: int, x: int, y: int): - """Decode one MVT tile to (shapely_geom, props) pairs in WGS-84 (EPSG:4326). - - MVT features are tile-local pixel coords [0, extent] with the NW origin - (y down), matching what pyvx writes; invert that transform back to lon/lat - using the same tile-bounds math. - """ - import mapbox_vector_tile as mvt - from shapely.geometry import shape - from shapely.ops import transform - - from databricks.labs.gbx.pyvx._mvt import _tile_bounds - - decoded = mvt.decode(payload) - out = [] - for layer in decoded.values(): - extent = layer.get("extent", 4096) - minx, miny, maxx, maxy = _tile_bounds(z, x, y) - sx = (maxx - minx) / extent - sy = (maxy - miny) / extent - - def _to_lonlat(px, py, zc=None, _minx=minx, _maxy=maxy, _sx=sx, _sy=sy): - return (_minx + px * _sx, _maxy - py * _sy) - - for feat in layer.get("features", []): - geom = shape(feat["geometry"]) - if geom.is_empty: - continue - out.append((transform(_to_lonlat, geom), feat.get("properties", {}))) - return out - - -def _static_vector_fallback(data: bytes, info: dict, **plot_kw): - """Decode MVT tiles to geometries and render via plot_static (contextily).""" - import geopandas as gpd - - from databricks.labs.gbx.vizx import plot_static - - geoms, rows = [], [] - for tileid, payload in all_tiles(MemorySource(data)): - z, x, y = tileid_to_zxy(tileid) - for geom, props in _decode_mvt_to_geoms(payload, z, x, y): - geoms.append(geom) - rows.append(props) - if not geoms: - raise ValueError("plot_pmtiles: vector archive decoded to no geometries") - gdf = gpd.GeoDataFrame(rows, geometry=geoms, crs=4326) - return plot_static(gdf, **plot_kw) -``` - -- [ ] **Step 4** — Run, expect green: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_pmtiles.py -``` -Expected: all pass. - -- [ ] **Step 5** — Commit: - -``` -git add python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py \ - python/geobrix/test/vizx/test_pmtiles.py -git commit -m "feat(vizx): vector pmtiles static fallback (MVT decode -> gdf) - -Decodes tile-local MVT features back to WGS-84 with the same tile-bounds -math pyvx writes, then renders via plot_static over a contextily basemap. - -Co-authored-by: Isaac" -``` - ---- - -### Task 6: `plot_pmtiles` dispatch + size guard - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py` -- Test: `python/geobrix/test/vizx/test_pmtiles.py` - -**Interfaces:** -- Produces: `plot_pmtiles(path_or_bytes, *, max_embed_mb=64, fallback=True, style=None, **map_kwargs)` — **PINNED public signature**. -- Consumes: `_archive_bytes`, `pmtiles_info`, `_build_pmtiles_html`, `_is_raster_type`, `_static_raster_fallback`, `_static_vector_fallback`, `_interactive._notebook_display_html`. - -**Steps:** - -- [ ] **Step 1** — Write the failing test. Cover: interactive HTML routed through displayHTML; size-guard → raster fallback; explicit `fallback=True` → vector fallback; bad type. Patch `_notebook_display_html` to capture the HTML and assert structure: - -```python -# append to python/geobrix/test/vizx/test_pmtiles.py -def test_plot_pmtiles_interactive_routes_through_displayhtml(monkeypatch): - from databricks.labs.gbx.vizx import _pmtiles as p - - archive = _build_archive([(0, 0, 0, _PNG)], TileType.PNG) - captured = {} - monkeypatch.setattr( - "databricks.labs.gbx.vizx._interactive._notebook_display_html", - lambda: (lambda html: captured.update(html=html)), - ) - out = p.plot_pmtiles(archive) # small -> interactive - assert out is None # displayHTML render returns None - html = captured["html"] - assert "maplibre-gl@4.7.1" in html and "pmtiles@3.2.1" in html - assert "pmtiles://" in html - - -def test_plot_pmtiles_size_guard_uses_raster_fallback(monkeypatch): - from databricks.labs.gbx.vizx import _pmtiles as p - - png = _real_png_tile() - archive = _build_archive([(0, 0, 0, png)], TileType.PNG) - called = {} - monkeypatch.setattr(p, "_static_raster_fallback", - lambda data, info, **kw: called.update(raster=True)) - # max_embed_mb tiny -> archive exceeds it -> static path - p.plot_pmtiles(archive, max_embed_mb=1e-9) - assert called.get("raster") is True - - -def test_plot_pmtiles_size_guard_uses_vector_fallback(monkeypatch): - from databricks.labs.gbx.vizx import _pmtiles as p - - blob = _real_mvt_tile(10, 163, 395) - archive = _build_archive([(10, 163, 395, blob)], TileType.MVT) - called = {} - monkeypatch.setattr(p, "_static_vector_fallback", - lambda data, info, **kw: called.update(vector=True) or "AX") - # tiny budget -> archive exceeds it -> static vector path (fallback default True) - p.plot_pmtiles(archive, max_embed_mb=1e-9) - assert called.get("vector") is True - - -def test_plot_pmtiles_oversized_without_fallback_raises(): - from databricks.labs.gbx.vizx import _pmtiles as p - - archive = _build_archive([(0, 0, 0, _PNG)], TileType.PNG) - with pytest.raises(ValueError, match="exceeds max_embed_mb"): - p.plot_pmtiles(archive, max_embed_mb=1e-9, fallback=False) -``` - -- [ ] **Step 2** — Run, expect failure: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_pmtiles.py -``` -Expected: `AttributeError: ... 'plot_pmtiles'`. - -- [ ] **Step 3** — Minimal implementation (append to `_pmtiles.py`): - -```python -# append to python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py -def plot_pmtiles(path_or_bytes, *, max_embed_mb=64, fallback=True, style=None, - **map_kwargs): - """Render a .pmtiles archive inline in a Databricks/Jupyter notebook. - - Interactive path (default, when the archive fits): a MapLibre GL JS + - pmtiles.js page (CDN-pinned) with the archive base64-embedded as an - in-browser FileSource, rendered via displayHTML — no tile server, no remote - range requests. Vector (MVT) -> a vector layer; raster (PNG/JPEG/WebP/AVIF) - -> a raster layer, auto-detected from the archive header. - - Static fallback (when the base64-embedded archive would exceed - ``max_embed_mb`` — base64 bloats ~33% — and ``fallback=True``, the default): - decode tiles on the driver and composite. Raster -> plot_raster; vector -> - decode MVT to geometries and plot_static over a contextily basemap. - ``fallback=False`` raises instead of degrading; ``max_embed_mb=0`` - deliberately forces the static render (for GitHub-renderable notebooks). - ``map_kwargs`` flow to the chosen static plotter. ``style`` overrides the - auto MapLibre style on the interactive path. Requires the [vizx] extra for - the static fallback. - """ - from databricks.labs.gbx.pmtiles import pmtiles_info - from databricks.labs.gbx.vizx._interactive import _notebook_display_html - - data = _archive_bytes(path_or_bytes) - info = pmtiles_info(data) - - # Interactive by default. base64 inflates ~33%; compare the *embedded* size - # against the budget and only then degrade to the static render. - embed_mb = (len(data) * 4 / 3) / (1024 * 1024) - if embed_mb > max_embed_mb: - if not fallback: - raise ValueError( - f"plot_pmtiles: archive embeds to ~{embed_mb:.1f} MB which " - f"exceeds max_embed_mb={max_embed_mb}; pass fallback=True for a " - "static render or raise max_embed_mb (max_embed_mb=0 forces static)." - ) - if _is_raster_type(info["tile_type"]): - return _static_raster_fallback(data, info, **map_kwargs) - return _static_vector_fallback(data, info, **map_kwargs) - - archive_b64 = base64.b64encode(data).decode("ascii") - html = _build_pmtiles_html(archive_b64, info, style=style) - dh = _notebook_display_html() - if dh is not None: - dh(html) - return None - try: - from IPython.display import HTML, display - - display(HTML(html)) - return None - except Exception: # noqa: BLE001 — no IPython: return the HTML string - return html -``` - -> **Note on `fallback` semantics:** per the spec, the **interactive** MapLibre map is the default. `fallback` governs what happens when the base64-embedded archive would exceed `max_embed_mb`: `fallback=True` (default) degrades gracefully to the static render; `fallback=False` raises so the caller knows the archive is too large to embed. For a deliberately static, GitHub-renderable render (committed notebooks where an interactive map wouldn't persist), pass `max_embed_mb=0`. Tests above lock this contract (`test_plot_pmtiles_interactive_routes_through_displayhtml` proves interactive-by-default; the size-guard tests prove auto-degrade; `test_plot_pmtiles_oversized_without_fallback_raises` proves the raise). - -- [ ] **Step 4** — Run, expect green: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_pmtiles.py -``` -Expected: all pass. - -- [ ] **Step 5** — Commit: - -``` -git add python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py \ - python/geobrix/test/vizx/test_pmtiles.py -git commit -m "feat(vizx): plot_pmtiles dispatch + base64 size guard - -Interactive MapLibre map by default; when the ~33%-inflated base64 embed -would exceed max_embed_mb, fallback=True (default) degrades to the static -render and fallback=False raises. max_embed_mb=0 forces static. - -Co-authored-by: Isaac" -``` - ---- - -### Task 7: `plot_cog` - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/vizx/_cog.py` -- Test: `python/geobrix/test/vizx/test_cog.py` - -**Interfaces:** -- Produces: `plot_cog(path, *, band=None, **kw)` — **PINNED public signature**. -- Consumes: `rasterio`; `vizx._raster.plot_file` / `plot_raster` machinery; contextily basemap (via the decimated read + a static render). - -**Decision (spec open item):** `plot_cog` is **static-only** (rasterio overview/decimated read over a contextily basemap). The interactive raster-source injection is left out — a COG is not a PMTiles archive, so embedding it as an in-browser FileSource is not applicable, and a remote raster source would need range requests against storage (the very thing the PMTiles base64 embed avoids). If a future need arises, convert the COG to raster PMTiles (`gbx_rst_cog_convert` → pyramid → `gbx_pmtiles_agg`) and use `plot_pmtiles`. This keeps `plot_cog` a single, predictable static path. - -**Steps:** - -- [ ] **Step 1** — Write the failing test. Build a real single-band + multi-band COG-ish GeoTIFF in memory with rasterio, assert `plot_cog` produces a figure and honors `band=`: - -```python -# python/geobrix/test/vizx/test_cog.py -"""Offline tests for plot_cog (rasterio overview read over a contextily basemap).""" - -import matplotlib -import pytest - -matplotlib.use("Agg") -import matplotlib.pyplot as plt # noqa: E402 -import numpy as np # noqa: E402 - - -def _write_tif(tmp_path, bands=3, size=32, crs="EPSG:3857"): - import rasterio - from rasterio.transform import from_bounds - - path = tmp_path / "cog.tif" - data = (np.random.rand(bands, size, size) * 1000).astype("uint16") - transform = from_bounds(-1.36e7, 4.5e6, -1.35e7, 4.51e6, size, size) - with rasterio.open( - path, "w", driver="GTiff", height=size, width=size, count=bands, - dtype="uint16", crs=crs, transform=transform, - ) as dst: - dst.write(data) - return str(path) - - -def test_plot_cog_renders_figure(tmp_path): - from databricks.labs.gbx.vizx import plot_cog - - plt.close("all") - path = _write_tif(tmp_path, bands=3) - plot_cog(path) - assert len(plt.get_fignums()) >= 1 - plt.close("all") - - -def test_plot_cog_band_select(tmp_path, monkeypatch): - from databricks.labs.gbx.vizx import _cog - - path = _write_tif(tmp_path, bands=3) - captured = {} - # capture the array handed to the renderer to confirm a single band was read - monkeypatch.setattr( - _cog, "_render_cog", - lambda data, transform, **kw: captured.update(shape=data.shape), - ) - _cog.plot_cog(path, band=2) - assert captured["shape"][0] == 1 # one band selected - - -def test_plot_cog_strips_dbfs_scheme(tmp_path): - from databricks.labs.gbx.vizx import plot_cog - - plt.close("all") - path = _write_tif(tmp_path, bands=1) - plot_cog("dbfs:" + path) # must not raise on the scheme prefix - assert len(plt.get_fignums()) >= 1 - plt.close("all") -``` - -- [ ] **Step 2** — Run, expect failure: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_cog.py -``` -Expected: `ImportError: cannot import name 'plot_cog'` / `_cog` missing. - -- [ ] **Step 3** — Minimal implementation: - -```python -# python/geobrix/src/databricks/labs/gbx/vizx/_cog.py -"""Cloud-Optimized GeoTIFF viewer for gbx.vizx. - -Reads a COG decimated (an overview-equivalent read) and renders it over a -contextily basemap as a static matplotlib figure. Driver-side; requires the -[vizx] extra plus rasterio. -""" - -from __future__ import annotations - -import warnings - - -def _strip_scheme(path: str) -> str: - for scheme in ("dbfs:", "file:"): - if path.startswith(scheme): - path = path[len(scheme) :] - break - if path.startswith("//"): - path = "/" + path.lstrip("/") - return path - - -def _render_cog(data, transform, *, crs, fig_w, fig_h, title, basemap, basemap_source): - """Render a decimated COG array (bands, h, w) over a contextily basemap.""" - import matplotlib.pyplot as plt - import numpy as np - from rasterio.plot import plotting_extent, show - - from databricks.labs.gbx.vizx._raster import ( - _needs_percentile_stretch, - _percentile_stretch, - ) - - if _needs_percentile_stretch(data): - data = _percentile_stretch(data) - _, ax = plt.subplots(1, figsize=(fig_w, fig_h)) - if data.shape[0] == 1: - band = data[0] - ax.imshow(band, extent=plotting_extent(band, transform), cmap="viridis") - else: - show(data, ax=ax, transform=transform) - if basemap and crs is not None: - try: - import contextily as cx - - source = basemap_source or cx.providers.CartoDB.Positron - cx.add_basemap(ax, source=source, crs=crs) - except Exception as exc: # noqa: BLE001 — offline/no-egress -> warn + skip - warnings.warn( - f"plot_cog: basemap unavailable ({type(exc).__name__}: {exc}); " - "rendering without basemap.", - stacklevel=2, - ) - if title: - ax.set_title(title) - ax.set_axis_off() - - -def plot_cog(path, *, band=None, max_pixels=2000, fig_w=10, fig_h=10, - basemap=True, basemap_source=None, title=None, **kw): - """Render a Cloud-Optimized GeoTIFF inline over a contextily basemap. - - Reads ``path`` decimated so the longest edge is <= ``max_pixels`` (uses the - COG's overviews when present). ``band`` (1-based) selects a single band; - otherwise all bands render (1 -> viridis, 3+ -> RGB). Volume/DBFS scheme - prefixes are stripped. Requires the [vizx] extra plus rasterio. - """ - from databricks.labs.gbx.vizx._env import assert_viz_available - - assert_viz_available() - import rasterio - - from databricks.labs.gbx.vizx._raster import _decimated_read - - p = _strip_scheme(str(path)) - with rasterio.open(p) as src: - if band is not None: - scale = max(src.width, src.height) / max_pixels - out_h = max(1, int(src.height // scale)) if scale > 1 else src.height - out_w = max(1, int(src.width // scale)) if scale > 1 else src.width - data = src.read( - indexes=[band], - out_shape=(1, out_h, out_w), - resampling=rasterio.enums.Resampling.bilinear, - masked=True, - ) - transform = src.transform * src.transform.scale( - src.width / out_w, src.height / out_h - ) - else: - data, transform, _ = _decimated_read(src, max_pixels) - crs = src.crs - _render_cog( - data, transform, crs=crs, fig_w=fig_w, fig_h=fig_h, - title=title or "COG", basemap=basemap, basemap_source=basemap_source, - ) -``` - -- [ ] **Step 4** — Run, expect green: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_cog.py -``` -Expected: all pass. (The basemap fetch fails offline → warns + renders without it; the test only asserts a figure is produced.) - -- [ ] **Step 5** — Commit: - -``` -git add python/geobrix/src/databricks/labs/gbx/vizx/_cog.py \ - python/geobrix/test/vizx/test_cog.py -git commit -m "feat(vizx): plot_cog static COG viewer over contextily basemap - -Decimated/overview rasterio read; band= selects one band. Static-only -(interactive raster-source injection deferred — COGs aren't PMTiles; -convert to raster PMTiles + plot_pmtiles for an interactive map). - -Co-authored-by: Isaac" -``` - ---- - -### Task 8: Public exports - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/__init__.py` -- Test: `python/geobrix/test/vizx/test_pmtiles.py` (export assertion) - -**Interfaces:** -- Produces: `from databricks.labs.gbx.vizx import plot_pmtiles, plot_cog`; both in `__all__`. - -**Steps:** - -- [ ] **Step 1** — Write the failing test: - -```python -# append to python/geobrix/test/vizx/test_pmtiles.py -def test_public_exports(): - import databricks.labs.gbx.vizx as vizx - - assert hasattr(vizx, "plot_pmtiles") - assert hasattr(vizx, "plot_cog") - assert "plot_pmtiles" in vizx.__all__ - assert "plot_cog" in vizx.__all__ -``` - -- [ ] **Step 2** — Run, expect failure: - -``` -gbx:test:python --path python/geobrix/test/vizx/test_pmtiles.py::test_public_exports -``` -Expected: `AttributeError: module 'databricks.labs.gbx.vizx' has no attribute 'plot_pmtiles'`. - -- [ ] **Step 3** — Minimal implementation (edit `vizx/__init__.py`): - -```python -from databricks.labs.gbx.vizx._cog import plot_cog -from databricks.labs.gbx.vizx._interactive import plot_interactive -from databricks.labs.gbx.vizx._pmtiles import plot_pmtiles -from databricks.labs.gbx.vizx._raster import plot_file, plot_mask_layers, plot_raster -from databricks.labs.gbx.vizx._static_map import plot_static -from databricks.labs.gbx.vizx._vector import as_gdf, cells_as_gdf, grid_as_gdf - -__all__ = [ - "plot_raster", - "plot_file", - "plot_mask_layers", - "plot_static", - "plot_interactive", - "plot_pmtiles", - "plot_cog", - "as_gdf", - "cells_as_gdf", - "grid_as_gdf", -] -``` - -> Verify `_pmtiles`/`_cog` are import-safe at module level: `_pmtiles.py` top-level imports `base64`, `json`, and `pmtiles.*` (present in both light tiers); `_cog.py` top-level imports only `warnings`. Heavy viz deps stay lazy inside functions. So importing `vizx` does not pull matplotlib/geopandas — consistent with the rest of the package. - -- [ ] **Step 4** — Run the full vizx + inspector suites: - -``` -gbx:test:python --path python/geobrix/test/vizx/ -gbx:test:python --path python/geobrix/test/pmtiles_light/test_inspect.py -``` -Expected: all green. - -- [ ] **Step 5** — Commit: - -``` -git add python/geobrix/src/databricks/labs/gbx/vizx/__init__.py \ - python/geobrix/test/vizx/test_pmtiles.py -git commit -m "feat(vizx): export plot_pmtiles + plot_cog - -Co-authored-by: Isaac" -``` - ---- - -### Task 9: Docs note + doc-test wiring + final green run - -**Files:** -- Modify: `docs/docs/api/` viz/vizx page (the existing VizX reference page) — add `plot_pmtiles` / `plot_cog` / `pmtiles_info` with a short usage snippet sourced from a doc-test. -- Create (if a vizx doc-test module exists, extend it; else add): `docs/tests/python/.../vizx_*` doc-test functions exercising `pmtiles_info` on a fixture archive and the static fallback (offline, real assertions). - -**Steps:** - -- [ ] **Step 1** — Locate the VizX docs page and any existing vizx doc-test module: - -``` -grep -rln "plot_static\|plot_raster\|gbx.vizx" docs/docs/ docs/tests/python/ -``` - -- [ ] **Step 2** — Add a doc-test that builds a tiny fixture PMTiles archive (the same `_build_archive` helper as the unit tests — copy it inline; doc-tests must be self-contained and execute real assertions), calls `pmtiles_info`, asserts `tile_type`/`tile_count`, and runs `plot_pmtiles(..., max_embed_mb=0, basemap=False)` (forces the static path) to confirm it produces output without network. Import the snippet into the MDX via raw-loader per repo convention. Keep the docs voice clean (no internal vocabulary). - -- [ ] **Step 3** — Run the vizx doc tests in Docker (doc tests only run in Docker). Dispatch a Task subagent for the long-running container run: - -``` -gbx:test:python-docs --log helios-sp2-vizx-docs.log -``` -Expected: the new vizx doc-test nodes pass. Narrow to the failing node IDs and rerun only those until green; do not retest passing packages. - -- [ ] **Step 4** — Run binding-parity-adjacent sanity (these are pure-Python public API additions, not registered SQL functions, so `registered_functions.txt` is unchanged — confirm no parity check expects them): - -``` -grep -rn "plot_pmtiles\|plot_cog\|pmtiles_info" python/geobrix/src/databricks/labs/gbx/bench/registered_functions.txt docs/tests-function-info/registered_functions.txt -``` -Expected: no matches (these are module functions, not SQL UDFs — nothing to register). - -- [ ] **Step 5** — Run python lint (CI gate) before committing docs: - -``` -gbx:lint:python --check -``` -Fix isort/black/flake8 findings in-container if the host black differs. - -- [ ] **Step 6** — Commit: - -``` -git add docs/ -git commit -m "docs(vizx): document plot_pmtiles, plot_cog, pmtiles_info - -Co-authored-by: Isaac" -``` - ---- - -### Task 10: Capture validated performance gains - -**Files:** -- Create (only if a gain is validated): `docs/superpowers/performance/.md` + `docs/superpowers/performance/README.md` (index, if first) + a thin pointer memory entry. - -**Steps:** - -- [ ] **Step 1** — Assess whether SP2 surfaced any reusable rendering/decoding gain worth capturing. Candidate: the **base64-embed-vs-tile-server PMTiles rendering** pattern (no HTTP server / no remote range requests; entire archive streams in-browser) and the **driver-side overview/decimated read** for `plot_cog` (avoids full-resolution reads). These are correctness/UX patterns more than a measured speedup; record only what is genuinely a *gain over an alternative* with evidence. - -- [ ] **Step 2** — If a gain qualifies, write one corpus file: problem → symptom/signature → the fix → applicability matrix (light-similar: other VizX inline renderers; heavy-same+similar: N/A — these are driver-side light-tier-only viewers, so record "heavy not applicable" and why) → evidence (the offline test asserting no network calls + the embed-size math) → canonical code refs (`vizx/_pmtiles.py`, `vizx/_cog.py`). Create `docs/superpowers/performance/README.md` as the index if this is the first corpus entry. - -- [ ] **Step 3** — Add the paired thin pointer memory (slug + one-line) that `[[links]]` to the corpus file. Keep it one line under ~200 chars (MEMORY.md is already near its size limit). - -- [ ] **Step 4** — If no gain qualifies, record the assessment verdict ("no measurable rendering gain; the base64-embed pattern is a correctness/portability choice, not a speedup") in the SP2 plan-completion note and skip corpus/memory creation. Either way, the assessment is performed, not assumed. - -- [ ] **Step 5** — Final full green run across both touched suites + commit any corpus/memory: - -``` -gbx:test:python --path python/geobrix/test/vizx/ -gbx:test:python --path python/geobrix/test/pmtiles_light/ -``` -Expected: all green. Commit: - -``` -git add docs/superpowers/performance/ 2>/dev/null; \ -git commit -m "docs(perf): capture PMTiles base64-embed rendering pattern - -Co-authored-by: Isaac" || echo "no perf corpus entry (assessed: not a measurable gain)" -``` - ---- - -## Self-review against the spec (SP2 + cross-cutting) - -- **Coverage of SP2 surface:** `plot_pmtiles` (Tasks 2-6, pinned signature), `plot_cog` (Task 7, pinned signature), `pmtiles_info` (Task 1, pinned `-> dict`) — all present. Files match the spec exactly: `vizx/_pmtiles.py`, `vizx/_cog.py`, `pmtiles/_inspect.py`; tests `test/vizx/test_pmtiles.py`, `test/vizx/test_cog.py`, plus the inspector test placed in the already-registered `test/pmtiles_light/` dir (matching the existing `pmtiles_light`/`pmtiles_bindings` layout) rather than a brand-new `test/pmtiles/` dir — avoids a CI-lock dir registration the spec's "vizx test dir already registered" note implies is the goal. -- **Interactive path** uses `_notebook_display_html()` + the IPython fallback chain (Task 6), pinned CDN versions (Task 3 constants), base64 in-browser `FileSource` + `pmtiles://` protocol registration (Task 3, asserted by tests). -- **Type detection** from header `tile_type` (Task 1 inspector → Task 2 `_is_raster_type`). -- **Size guard** compares the ~33%-inflated base64 size against `max_embed_mb` (Task 6). -- **Static fallbacks** reuse `plot_raster` (Task 4) and `plot_static` over contextily (Task 5); no new dep — `mapbox_vector_tile` is an existing pyvx dep, `contextily` an existing `[vizx]` dep. -- **No new deps** (Global Constraints); rio-tiler explicitly NOT adopted — the CI-lock checklist is documented as conditional only. -- **Placeholder scan:** no TODOs; no dead branches. `fallback` semantics corrected to interactive-by-default with auto-degrade (the size-guard tests + the no-fallback-raise test lock the contract). -- **Type consistency:** these are Python module functions, not registered SQL UDFs, so cross-language naming / `registered_functions.txt` parity does not apply (Task 9 Step 4 confirms no parity hook expects them). -- **TDD + per-task commits + commit hygiene** (≤72-char subjects, WHY bodies, `Co-authored-by: Isaac`) throughout. -- **Performance capture** step present (Task 10), assessment-not-assumed. -- **Docs voice** clean (Task 9). diff --git a/docs/superpowers/plans/2026-06-27-helios-sp3-notebook-series.md b/docs/superpowers/plans/2026-06-27-helios-sp3-notebook-series.md deleted file mode 100644 index 75f837550..000000000 --- a/docs/superpowers/plans/2026-06-27-helios-sp3-notebook-series.md +++ /dev/null @@ -1,1303 +0,0 @@ -# Helios Notebook Series (SP3) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. Each task is one independently-reviewable deliverable (a notebook, README, docs page, or sidebar entry). Notebooks are NOT TDD unit code, so each task ends in a concrete **VALIDATION** step (Docker cell-by-cell execution + asserted artifact) instead of a pytest assert — same rigor, different evidence. - -**Goal:** Build the `notebooks/examples/helios/` series — a `config_nb.ipynb` spine + three numbered notebooks (NB01 Vector Engine / MVT, NB02 Visual Basemap / XYZ, NB03 Analytical Core / COG+STAC) + `README.md`, plus a `docs/docs/notebooks/helios.mdx` page and `docs/sidebars.js` entry. The series tells one **solar site-selection** meta-narrative over a single **San Francisco** AOI, end-to-end: ingest → tile (MVT / XYZ / COG) → package as PMTiles → inspect/visualize, Serverless lightweight tier by default, heavyweight switchable. - -**Architecture:** Mirror the eo-series spine exactly. `config_nb.ipynb` holds all shared state (%pip light tier, imports, `OvertureClient` + `StacClient` setup, the SF AOI bbox constant, `ETL_DIR` Volume config, the `set_conf_safe()` Serverless guard, the tier switch, `FORCE_REBUILD`, and **series-only** helpers: solar slope/aspect scoring, demo plot wrappers, Delta table finalizers). Each numbered notebook begins with `%run ./config_nb` and produces named Volume paths / Delta tables / PMTiles archives that the next notebook (and the docs page) reference by exact name. Anything generally reusable was already promoted to SP1 (`gbx.sample.overture`) / SP2 (`gbx.vizx.plot_pmtiles`/`plot_cog`, `gbx.pmtiles.pmtiles_info`) — this plan **consumes** those signatures and re-implements nothing. - -**Tech Stack:** Databricks notebooks (`.ipynb`), Python 3.12 / Spark 4 / Serverless (env v5+); GeoBrix `[light,stac,vizx]` wheel; `databricks.labs.gbx.pyrx`/`pyvx` (lightweight) with a commented `rasterx`/`vectorx` heavyweight option; existing registered SQL `gbx_st_asmvt`, `gbx_st_asmvt_pyramid`, `gbx_rst_to_webmercator`, `gbx_rst_xyzpyramid`, `gbx_rst_cog_convert`, `gbx_pmtiles_agg`; Docusaurus MDX docs; validated in Docker via `gbx:test:notebooks`. - -**Spec:** `docs/superpowers/specs/2026-06-27-helios-tiling-series-overture-design.md` (Sub-project 3 + cross-cutting). -**Branch:** the SP3 feature branch off `beta/0.4.0` (created for the Helios work; SP1/SP2 land first). PR into `beta/0.4.0`. - ---- - -## Global Constraints - -These apply to every task; do not restate them per step, but honor them everywhere. - -- **Serverless lightweight tier is the DEFAULT, heavyweight switchable.** `config_nb` selects `pyrx`/`pyvx` (option-1, default); a commented option-2 imports `rasterx`/`vectorx` for a classic x86 cluster + JAR + GDAL init script. Heavy/light call the **same SQL names** (`gbx_*`), so the notebook body is tier-agnostic after the import switch. -- **Serverless hard rules** (from the spec + repo memory): parallelism only via `DataFrame.repartition(N, "col")` — never number-only `repartition(N)` (AQE-coalesced to serial); **no** `spark.conf.set` outside the `set_conf_safe()` guard, **no** `.cache()`/`.persist()`/`.checkpoint()`/`.rdd`/`sparkContext`. Where you'd cache, write a managed Delta table and read it back. `CREATE TEMP TABLE` is Serverless / DBR 18.1+ only. -- **One San Francisco AOI** for all three notebooks: the N37W123 quad reused from h3-rasterize. `SF_AOI_BBOX = (-123.0, 37.0, -122.0, 38.0)` (minx, miny, maxx, maxy, EPSG:4326). A tighter demo sub-bbox `SF_CITY_BBOX = (-122.52, 37.70, -122.35, 37.83)` (the SF peninsula proper) keeps Overture buildings + DEM volumes demo-friendly; both live in `config_nb`. -- **User-facing docs voice** — README.md and helios.mdx are read by end users: NO internal planning vocabulary (no wave numbers, no dispatch/subagent references, no "SP1/SP2/SP3"). The QC `internals-leak` check enforces this. Frame GeoBrix as an on-ramp to Databricks-native spatial where it fits; factual, not marketing. -- **Notebooks validated in Docker** via `gbx:test:notebooks --path examples/helios/.ipynb` (cell-by-cell, no kernel; the container must be started with `start_docker_with_volumes.sh` so `/Volumes` is mounted). Doc tests are the documentation source convention applies to `docs/tests/` code, not these example notebooks — but every notebook must execute green cell-by-cell in Docker against real sample data, with real asserted artifacts (a PMTiles archive exists on the Volume and `pmtiles_info` parses it; a plot returns a figure). -- **Commit hygiene** — subject ≤72 chars, a WHY body for non-trivial commits, end with the `Co-authored-by: Isaac` trailer. One commit per task. Hold pushes (commit locally; push on the user's go). -- **No placeholders / TODOs** in committed notebooks. Every cell has real narrative or real runnable code. -- **Shipped default `INTERACTIVE_PLOTS = False`** (GitHub-renderable static images); `True` for live folium/MapLibre. All notebook plot calls route through the toggle-aware `config_nb` helpers (`show_pmtiles`, `show_cog`, and any demo plot wrappers), so the committed `.ipynb` renders fast static images on GitHub by default and flips to the interactive experience with one variable. -- **Compose with Databricks-native spatial where natural.** GeoBrix tiling is an on-ramp to Databricks-native `ST_*` / H3; NB01 and NB03 weave native functions into the solar narrative where it reads naturally (roof geometry / H3 roof-density and slope-per-cell aggregation), NB02 deliberately does not (a pure raster basemap step). Factual framing, no marketing, no internal vocabulary. - ---- - -## Task 0: Series scaffold + diagram generator - -**Files:** -- Create dir: `notebooks/examples/helios/` -- Create: `resources/images/generators/helios.py` (the per-notebook diagram generator, mirroring `resources/images/generators/eo-series.py`) -- Produces (committed): `resources/images/diagrams/helios/helios-01.svg`/`.png`, `helios-02.svg`/`.png`, `helios-03.svg`/`.png` - -**Interfaces:** Produces the three `helios-0N.png` images that NB01–NB03 and `helios.mdx` reference by exact path. No code dependency on SP1/SP2. - -- [ ] **Step 1: Copy and adapt the diagram generator.** Copy `resources/images/generators/eo-series.py` to `resources/images/generators/helios.py`. Keep its palette, glyph, chip, and four-stage layout machinery verbatim. Replace the four notebook diagram definitions with **three** Helios diagrams. Each is a catchy **data → tile → PMTiles → view** flow with a hero glyph per stage and a footer of the GeoBrix/Databricks function chips that notebook actually uses: - - `helios-01` (Vector Engine / MVT): stages `Overture buildings (SF)` → `gbx_st_asmvt + st_asmvt_pyramid` → `gbx_pmtiles_agg → vector .pmtiles` → `plot_pmtiles`. Glyphs: building footprints → vector-tile grid → stacked-archive → map pin. Chips: `OvertureClient.discover/download/read`, `gbx_st_asmvt`, `gbx_st_asmvt_pyramid`, `gbx_pmtiles_agg`, `plot_pmtiles`. - - `helios-02` (Visual Basemap / XYZ): stages `NAIP aerial (SF)` → `gbx_rst_to_webmercator` → `gbx_rst_xyzpyramid → gbx_pmtiles_agg → raster .pmtiles` → `plot_pmtiles`. Glyphs: aerial swatch → web-mercator globe → XYZ pyramid → map pin. Chips: `gbx_rst_to_webmercator`, `gbx_rst_xyzpyramid`, `gbx_pmtiles_agg`, `pmtiles_info`, `plot_pmtiles`. - - `helios-03` (Analytical Core / COG+STAC): stages `USGS 3DEP DEM (SF)` → `gbx_rst_cog_convert → COGs + STAC Delta` → `slope/hillshade → gbx_rst_xyzpyramid → .pmtiles` → `plot_cog + plot_pmtiles`. Glyphs: contour DEM → catalog/COG → hillshade relief → map pin. Chips: `gbx_rst_cog_convert`, `StacClient`/STAC Delta, `rst_terrainslope`/`rst_hillshade`, `gbx_rst_xyzpyramid`, `plot_cog`, `plot_pmtiles`. -- [ ] **Step 2: Update the module docstring** re-render block to loop `01 02 03` (not `01 02 03 04`) and to point at `helios-$n.svg`/`.png`. Keep the Chrome-headless screenshot + PIL bbox-trim recipe identical. -- [ ] **Step 3 (VALIDATION):** Run `python3 resources/images/generators/helios.py` then the Chrome-headless + PIL crop recipe from the docstring on the host (Chrome is host-only, not in Docker). Assert the three `.svg` and three trimmed `.png` files exist and open (`python3 -c "from PIL import Image; [Image.open(f'resources/images/helios-{n}.png').load() for n in ('01','02','03')]"` exits 0). Expected: three non-empty PNGs, each a four-stage horizontal flow with a chip footer. -- [ ] **Step 4 (commit):** `git add notebooks/examples/helios resources/images/generators/helios.py resources/images/helios-0*.svg resources/images/helios-0*.png && git commit` — subject `feat(helios): add notebook-series diagram generator + images`; body: WHY (per-notebook data→tile→PMTiles flow diagrams for the SF solar series, mirrors eo-series.py). Trailer `Co-authored-by: Isaac`. - ---- - -## Task 1: `config_nb.ipynb` — the shared spine - -**Files:** -- Create: `notebooks/examples/helios/config_nb.ipynb` - -**Interfaces:** -- *Consumes:* `from databricks.labs.gbx.sample.overture import OvertureClient` (`OvertureClient()`); `from databricks.labs.gbx.stac import StacClient`; `from databricks.labs.gbx.vizx import plot_pmtiles, plot_cog` + existing `plot_raster, plot_file, plot_static, cells_as_gdf`; `from databricks.labs.gbx.pmtiles import pmtiles_info`; `from databricks.labs.gbx.pyrx import functions as rx`, `from databricks.labs.gbx.pyvx import functions as vx`, `from databricks.labs.gbx.ds.register import register`. -- *Produces (notebook globals every later NB relies on, by exact name):* `overture = OvertureClient()`, `stac_client = StacClient()`, `rx`, `vx`, `register`, `plot_pmtiles`, `plot_cog`, `pmtiles_info`, `plot_static`, `plot_interactive`, `set_conf_safe`, `FORCE_REBUILD`, `INTERACTIVE_PLOTS`, `catalog_name="geospatial_docs"`, `schema_name="helios"`, `ETL_DIR=/Volumes///data`, `HELIOS_DIR=${ETL_DIR}/sf`, `SF_AOI_BBOX`, `SF_CITY_BBOX`, and series helpers `solar_score(...)`, `finalize_delta(...)`, `show_pmtiles(...)`, `show_cog(...)`, `show_raster(...)`. - -Build the notebook cell-by-cell. Use real cell content (markdown text + python code) exactly as written below. - -- [ ] **Step 1 — markdown cell (title):** - ```markdown - # Helios — Shared Configuration Notebook - - This notebook sets up the **Helios** tiling series: a San Francisco **solar - site-selection** walkthrough that takes three data layers — building footprints, - aerial imagery, and terrain — all the way to **PMTiles** map archives you can view - inline. Every numbered notebook (`01.`, `02.`, `03.`) runs `%run ./config_nb` to - establish catalog / schema / Volume paths, install GeoBrix, instantiate the - `OvertureClient` and `StacClient`, and register shared helpers. - ``` - -- [ ] **Step 2 — markdown cell (libraries / UC):** - ```markdown - __Libraries__ - - * GeoBrix is installed below (the lightweight `[light,stac,vizx]` wheel) — nothing is assumed pre-staged. - * Default tier is **lightweight** (`databricks.labs.gbx.pyrx` / `pyvx`), which runs on **Serverless**. Flip to option-2 (`rasterx` / `vectorx`) for the heavyweight tier on a classic x86 cluster (JAR + GDAL init script). - - __Unity Catalog__ - - * Replace `catalog_name` and `schema_name` with your preferred locations. - * A Volume named `data` must exist under `catalog_name`/`schema_name`. - ``` - -- [ ] **Step 3 — code cell (%pip install, light tier):** - ```python - # -- GeoBrix: lightweight tier (option-1, default). Installed here so nothing is - # assumed pre-staged. For the heavyweight tier (option-2 below) attach the - # GeoBrix JAR + GDAL init script to a classic x86 cluster. - # - # Resilient two-step install (for WHL refreshes in the SAME session) — keep BOTH - # lines; do not collapse to one. Step 1 force-reinstalls WITHOUT deps so a freshly - # rebuilt wheel's bytes always replace the cached install even when the version - # string is unchanged (pip would otherwise treat same-version as "already satisfied" - # and skip it). Step 2 installs the EXTRAS without --no-deps so light/stac/vizx/ - # overture dependencies resolve — a bare --no-deps install drops the extras and - # surfaces as ModuleNotFound at import. A `%restart_python` (next cell) then loads - # the fresh bytes. Same pattern as the eo-series / h3-rasterize examples. - %pip install --quiet --disable-pip-version-check --force-reinstall --no-deps "geobrix @ file:///Volumes/geospatial_docs/geobrix/sample-data/geobrix-0.4.0-py3-none-any.whl" - %pip install --quiet "geobrix[light,stac,vizx,overture] @ file:///Volumes/geospatial_docs/geobrix/sample-data/geobrix-0.4.0-py3-none-any.whl" - %pip install --quiet rich - ``` - -- [ ] **Step 4 — code cell (`%restart_python`):** - ```python - %restart_python - ``` - -- [ ] **Step 5 — code cell (Spark/Delta imports):** - ```python - # -- databricks + delta + spark functions - from delta.tables import * - from pyspark.databricks.sql import functions as DBF - from pyspark.sql import functions as F - from pyspark.sql.functions import col, udf, pandas_udf - from pyspark.sql.types import * - from pyspark.sql.window import Window - ``` - -- [ ] **Step 6 — code cell (other imports):** - ```python - # -- other imports - from datetime import datetime - from databricks.labs.gbx.stac import StacClient - from databricks.labs.gbx.sample.overture import OvertureClient - - import os - import pandas as pd - import pathlib - import warnings - - warnings.simplefilter("ignore") - ``` - -- [ ] **Step 7 — markdown cell (client setup):** - ```markdown - ## Data-source clients - - - `OvertureClient` discovers and downloads Overture Maps GeoParquet (buildings, transportation, places, …) for an AOI into a Volume, with an optional metadata Delta catalog. Used in notebook 01. - - `StacClient` catalogs the COGs we produce in notebook 03 into a queryable Delta table (the same client used by the EO series for Planetary Computer). See [STAC API](https://databrickslabs.github.io/geobrix/docs/api/stac). - ``` - -- [ ] **Step 8 — code cell (clients):** - ```python - overture = OvertureClient() # Overture Maps static STAC catalog; latest release resolved by default - stac_client = StacClient() # used in NB03 to STAC-catalog the COGs we generate - ``` - -- [ ] **Step 9 — code cell (`set_conf_safe`, Serverless guard):** - ```python - # -- Spark conf tuning, guarded for Serverless -- - # Serverless forbids runtime spark.conf.set; set_conf_safe() no-ops there (AQE - # handles partitioning). On classic clusters it applies high-parallelism tuning. - def set_conf_safe(key, value): - try: - spark.conf.set(key, value) - return True - except Exception as e: - print(f"... skipping spark.conf.set({key}) [Serverless?]: {type(e).__name__}") - return False - - set_conf_safe("spark.sql.adaptive.coalescePartitions.enabled", "false") - set_conf_safe("spark.sql.shuffle.partitions", 512) - ``` - -- [ ] **Step 10 — code cell (tier switch + registration):** - ```python - # -- GeoBrix tier selection (default: lightweight) -- - # option-1: lightweight tier (pure Python / PySpark, runs on Serverless) -- DEFAULT - from databricks.labs.gbx.pyrx import functions as rx # rx.rst_* (raster) - from databricks.labs.gbx.pyvx import functions as vx # vx.st_* (vector) - - # option-2: heavyweight tier (Scala JAR + GDAL init script on a classic x86 cluster) - # from databricks.labs.gbx.rasterx import functions as rx - # from databricks.labs.gbx.vectorx import functions as vx - - rx.register(spark) # registers gbx_rst_*, gbx_rst_xyzpyramid, gbx_rst_cog_convert, gbx_pmtiles_agg, ... - vx.register(spark) # registers gbx_st_asmvt, gbx_st_asmvt_pyramid, ... - - # -- light readers/writers (gtiff_gbx, binaryFile patterns, pmtiles writer) -- - from databricks.labs.gbx.ds.register import register - register(spark) - ``` - -- [ ] **Step 11 — code cell (viz + inspector imports):** - ```python - # -- visualization: PMTiles + COG viewers (net-new), plus the raster + vector helpers. - # plot_static / plot_interactive back the INTERACTIVE_PLOTS toggle (static images for - # GitHub by default; folium pan/zoom maps when True). See the helper cells below. - from databricks.labs.gbx.vizx import ( - plot_pmtiles, plot_cog, plot_raster, plot_file, plot_static, plot_interactive, - cells_as_gdf, - ) - from databricks.labs.gbx.pmtiles import pmtiles_info # driver-side PMTiles header inspector - ``` - -- [ ] **Step 12 — code cell (catalog/schema/rebuild):** - ```python - # -- rebuild control: when True, force re-create of tables / re-download by feeding - # the do_overwrite / skip-guards below. Set per-notebook after %run ./config_nb. - FORCE_REBUILD = False - - # Interactive folium maps are slow to render and heavy in result size. - # False (default; jobs + docs) -> fast static images. True -> interactive folium maps. - INTERACTIVE_PLOTS = False - - catalog_name = "geospatial_docs" - schema_name = "helios" - - sql(f"USE CATALOG {catalog_name}") - sql(f"CREATE DATABASE IF NOT EXISTS {schema_name}") - sql(f"USE DATABASE {schema_name}") - print(f"... catalog: '{catalog_name}' (USE)") - print(f"... schema: '{schema_name}' (CREATE / USE)") - ``` - -- [ ] **Step 13 — code cell (ETL dirs + SF AOI):** - ```python - ETL_DIR = f"/Volumes/{catalog_name}/{schema_name}/data" # <- Volume ('data') must exist - HELIOS_DIR = f"{ETL_DIR}/sf" - dbutils.fs.mkdirs(HELIOS_DIR) - os.environ["ETL_DIR"] = ETL_DIR - os.environ["HELIOS_DIR"] = HELIOS_DIR - - # -- San Francisco AOI (reuses the h3-rasterize N37W123 quad) ------------------- - # Full 1x1-deg quad (minx, miny, maxx, maxy, EPSG:4326): SF peninsula, Marin, East Bay. - SF_AOI_BBOX = (-123.0, 37.0, -122.0, 38.0) - # Tighter city sub-bbox keeps Overture/DEM volumes demo-friendly (the SF peninsula proper). - SF_CITY_BBOX = (-122.52, 37.70, -122.35, 37.83) - - print(f"... ETL_DIR: '{ETL_DIR}'") - print(f"... HELIOS_DIR: '{HELIOS_DIR}' (MKDIRS)") - print(f"... SF_CITY_BBOX: {SF_CITY_BBOX}") - ``` - -- [ ] **Step 14 — markdown cell (Helper Functions header):** - ```markdown - ## Helper Functions - - Series-only helpers (not promoted to the GeoBrix API because they are specific to - this solar-site walkthrough): - - - `solar_score(slope_col, aspect_col)` — a simple south-facing-gentle-slope solar - suitability score for the terrain step (notebook 03). - - `finalize_delta(df, tbl_name, ...)` — idempotent managed-Delta materializer - (write-once unless `FORCE_REBUILD`), the Serverless-safe alternative to `.cache()`. - - `show_pmtiles(path)` / `show_cog(path)` / `show_raster(...)` — thin demo wrappers - that print `pmtiles_info(...)` header metadata then render through the - `INTERACTIVE_PLOTS` toggle: **False** (default; GitHub-renderable static images) calls - `plot_pmtiles(path, max_embed_mb=0, ...)` / `plot_cog(...)` / `plot_static(...)`; - **True** calls the interactive `plot_pmtiles(path, ...)` (MapLibre) and - `plot_interactive(...)` (folium) paths. Set `INTERACTIVE_PLOTS = True` in a notebook - (after `%run ./config_nb`) for live pan/zoom maps. - ``` - -- [ ] **Step 15 — code cell (`solar_score`):** - ```python - def solar_score(slope_col="slope_deg", aspect_col="aspect_deg"): - """Column expression: solar-suitability score in [0, 1] for a roof/terrain facet. - - Favors gentle slopes (best near a target tilt for SF latitude ~30 deg) and - south-facing aspect (180 deg). Aspect is GDAL convention (0=N, 90=E, 180=S, - 270=W). This is a didactic score, not a production PV yield model. - """ - target_tilt = 30.0 - slope_term = 1.0 - (F.abs(F.col(slope_col) - target_tilt) / 90.0) - # cos of (aspect - south) folds 0..360 into a south-preference in [0, 1] - aspect_term = (F.cos(F.radians(F.col(aspect_col) - 180.0)) + 1.0) / 2.0 - return F.greatest(F.lit(0.0), slope_term) * aspect_term - ``` - -- [ ] **Step 16 — code cell (`finalize_delta`):** - ```python - def finalize_delta(df, tbl_name, do_display=True): - """Idempotent managed-Delta materializer (Serverless-safe stand-in for cache()). - - Writes `df` to managed table `tbl_name` once; re-reads on subsequent runs unless - FORCE_REBUILD is True. Returns the table DataFrame. - """ - if FORCE_REBUILD: - sql(f"DROP TABLE IF EXISTS {tbl_name}") - if not spark.catalog.tableExists(tbl_name): - df.write.mode("overwrite").saveAsTable(tbl_name) - print(f"... wrote table {tbl_name} ({spark.table(tbl_name).count():,} rows)") - else: - print(f"... table {tbl_name} exists (skip; FORCE_REBUILD=False)") - out = spark.table(tbl_name) - if do_display: - out.printSchema() - return out - ``` - -- [ ] **Step 17 — code cell (toggle-aware viewers: `show_pmtiles` / `show_cog` / `show_raster`):** - ```python - def show_pmtiles(path, **kw): - """Print the PMTiles header, then render through the INTERACTIVE_PLOTS toggle. - - INTERACTIVE_PLOTS=False (default): plot_pmtiles(path, max_embed_mb=0, ...) forces - the GitHub-renderable static image. True: the interactive MapLibre map (default - plot_pmtiles path, base64-embedded in-browser FileSource). - """ - info = pmtiles_info(path) - print(f"... pmtiles: type={info.get('tile_type')} " - f"zoom={info.get('min_zoom')}-{info.get('max_zoom')} " - f"bounds={info.get('bounds')}") - if INTERACTIVE_PLOTS: - return plot_pmtiles(path, **kw) # interactive MapLibre (default) - return plot_pmtiles(path, max_embed_mb=0, **kw) # static render (max_embed_mb=0) - - def show_cog(path, **kw): - """Render a COG. plot_cog is static (rasterio overview read over a contextily - basemap) in both modes; the toggle is honored for API symmetry with show_pmtiles.""" - return plot_cog(path, **kw) - - def show_raster(df_or_path, **kw): - """Render a raster/vector overlay through the toggle: plot_static (False, default) - or plot_interactive (True, folium). Accepts a Spark DataFrame of cells/geoms or a - file path, matching the underlying vizx helpers.""" - if INTERACTIVE_PLOTS: - return plot_interactive(df_or_path, **kw) - return plot_static(df_or_path, **kw) - ``` - > NOTE: `show_cog` honors `INTERACTIVE_PLOTS` only for call-site symmetry — per SP2, `plot_cog` is static-only by design (a COG is not a PMTiles archive; an interactive remote-raster source would need range requests, the very thing PMTiles base64 embedding avoids). Keep `show_cog` static in both modes; do not invent an interactive COG path. - -- [ ] **Step 18 (VALIDATION):** `gbx:test:notebooks --path examples/helios/config_nb.ipynb --log helios-config.log` in the Docker container (started with `start_docker_with_volumes.sh`). Note the `%pip`/`%restart_python` cells run in the runner's venv; the cell-by-cell runner pre-installs the light deps. Expected: every cell reports OK; the final cell defines `solar_score`, `finalize_delta`, `show_pmtiles`, `show_cog`, `show_raster` without error; `INTERACTIVE_PLOTS` defaults to `False` and `SF_CITY_BBOX` is printed. If `%pip` from the Volume wheel can't resolve in the runner venv (the runner pre-installs light deps), confirm the runner picks up `OvertureClient`/`plot_pmtiles` from the installed `geobrix` — if not, the env-lock work in SP1/SP2 must register the new modules; record the gap and proceed (config_nb is %run-only, exercised end-to-end in Task 4's full-series validation). -- [ ] **Step 19 (commit):** `git add notebooks/examples/helios/config_nb.ipynb && git commit` — subject `feat(helios): add config_nb spine for SF solar tiling series`; body: WHY (shared %run setup: light tier, Overture+STAC clients, SF AOI, ETL dirs, series-only solar/Delta/viewer helpers). Trailer. - ---- - -## Task 2: NB01 — Vector Engine (MVT) - -**Files:** -- Create: `notebooks/examples/helios/01. Vector Engine (MVT).ipynb` - -**Interfaces:** -- *Consumes (from `%run ./config_nb`):* `overture` (`OvertureClient`), `vx`, `register`, `plot_pmtiles`/`show_pmtiles`, `pmtiles_info`, `HELIOS_DIR`, `SF_CITY_BBOX`, `finalize_delta`, `FORCE_REBUILD`. Overture API: `overture.discover(SF_CITY_BBOX, themes=["buildings"])` → DataFrame `theme,type,href,asset_bbox,release`; `overture.download(assets_df, out_dir, *, table="overture_buildings_meta", validate=True, partitions=...)` → `theme,type,source,path,out_file_sz,is_out_file_valid,last_update,asset_bbox,release,href`; `overture.read(source, theme="buildings", type="building", bbox=SF_CITY_BBOX)` → GeoParquet rows with a geometry column. GeoBrix SQL: `gbx_st_asmvt`, `gbx_st_asmvt_pyramid`, `gbx_pmtiles_agg`. Databricks-native SQL (the on-ramp composition): `st_geomfromwkb`, `st_area`, `st_centroid`, `st_x`, `st_y`, `h3_longlatash3`. -- *Produces:* Volume dir `${HELIOS_DIR}/overture/` (downloaded GeoParquet), Delta table `overture_buildings_meta` (asset catalog), Delta table `sf_buildings_mvt_tiles` (per `(z,x,y)` MVT bytes), and the vector PMTiles archive `${HELIOS_DIR}/tiles/sf_buildings.pmtiles`. - -- [ ] **Step 1 — markdown cell (title + meta-narrative):** - ```markdown - # Helios 01 — Vector Engine: building footprints to vector PMTiles - - ![Overture buildings to vector PMTiles](https://raw.githubusercontent.com/databrickslabs/geobrix/main/resources/images/diagrams/helios/helios-01.png) - - **Solar site-selection, step 1: the candidate surfaces.** Every rooftop in San - Francisco is a potential solar surface. This notebook pulls **Overture Maps building - footprints** for the city, encodes them into **Mapbox Vector Tiles** with - `gbx_st_asmvt`, pyramids them across zoom levels with `gbx_st_asmvt_pyramid`, folds - the whole pyramid into one **PMTiles** archive with `gbx_pmtiles_agg`, and views it - inline with `plot_pmtiles`. The result is a single self-contained vector basemap of - every candidate roof — the geometry layer the later notebooks score for solar yield. - - Along the way we compose with **Databricks-native** `ST_*` and `H3` functions — GeoBrix - tiling is an on-ramp into the native spatial engine, not a replacement for it — to - quantify roof area and bin roofs into an H3 roof-density surface. - - > Runs on the **lightweight tier (Serverless)** by default. See `config_nb` for the - > heavyweight switch. - ``` - -- [ ] **Step 2 — code cell (`%run`):** - ```python - %run ./config_nb - ``` - -- [ ] **Step 3 — code cell (optional rebuild flag):** - ```python - # Flip to True to fully rebuild this notebook's tables / re-download / re-tile. - FORCE_REBUILD = False - ``` - -- [ ] **Step 4 — markdown cell (discover):** - ```markdown - ## 1. Discover Overture building assets for the SF AOI - - `OvertureClient.discover` traverses Overture's static STAC catalog and returns one - row per GeoParquet asset that intersects our bbox — metadata only, on the driver. - We narrow to the `buildings` theme; `themes=None` would select every theme. - ``` - -- [ ] **Step 5 — code cell (discover):** - ```python - assets = overture.discover(SF_CITY_BBOX, themes=["buildings"]) - display(assets) # theme, type, href, asset_bbox, release - print(f"... {assets.count()} intersecting building assets") - ``` - -- [ ] **Step 6 — markdown cell (download):** - ```markdown - ## 2. Download the AOI subset to a Volume (+ catalog it in Delta) - - `OvertureClient.download` reads only the AOI rows (bbox predicate pushdown) and - writes them to the Volume **distributed across workers**, returning a metadata - DataFrame and UPSERTing it into the `overture_buildings_meta` Delta table (idempotent - MERGE keyed by `theme, type, source`). On Serverless the download fans out via - `repartition(N, col)` — no cluster knobs to tune. - ``` - -- [ ] **Step 7 — code cell (download):** - ```python - OVERTURE_DIR = f"{HELIOS_DIR}/overture" - meta = overture.download( - assets, OVERTURE_DIR, - table="overture_buildings_meta", - validate=True, - partitions=64, # hash fan-out; Serverless-safe parallelism - ) - display(meta.select("theme", "type", "source", "out_file_sz", "is_out_file_valid")) - ``` - -- [ ] **Step 8 — markdown cell (read):** - ```markdown - ## 3. Load the building geometries - - `OvertureClient.read` loads the downloaded GeoParquet back into Spark, re-applying - the bbox AOI filter. We keep the building polygon geometry and a stable id. - ``` - -- [ ] **Step 9 — code cell (read + project to tile inputs):** - ```python - buildings = ( - overture.read("overture_buildings_meta", theme="buildings", type="building", bbox=SF_CITY_BBOX) - .select(F.col("id").alias("feature_id"), F.col("geometry")) - ) - print(f"... {buildings.count():,} building footprints in the AOI") - display(buildings.limit(5)) - ``` - -- [ ] **Step 9b — markdown cell (native ST/H3: roof area + H3 roof density):** - ```markdown - ## 3b. Quantify candidate roof space with Databricks-native ST + H3 - - Before tiling, we use **Databricks-native** spatial functions on the same footprints — - GeoBrix tiling composes directly with the native engine. Native `st_area` / - `st_centroid` (over `st_geomfromwkb`) give each roof's **available area** and a point - to index; native `h3_longlatash3` bins those centroids into H3 cells so we can - aggregate **roof density and total roof area per cell** — a coarse "where are the - candidate solar surfaces concentrated?" view that complements the per-building tiles. - ``` - -- [ ] **Step 9c — code cell (native ST roof metrics):** - ```python - # Native ST: parse WKB -> GEOMETRY, then area (m^2, planar in the layer CRS) + centroid. - # GeoBrix readers hand us a WKB geometry column; st_geomfromwkb bridges to native ST. - roofs = buildings.selectExpr( - "feature_id", - "geometry", - "st_area(st_geomfromwkb(geometry)) AS roof_area_m2", - "st_x(st_centroid(st_geomfromwkb(geometry))) AS lon", - "st_y(st_centroid(st_geomfromwkb(geometry))) AS lat", - ) - display(roofs.orderBy(F.col("roof_area_m2").desc()).limit(5)) - ``` - > NOTE: `st_geomfromwkb`, `st_area`, `st_centroid`, `st_x`, `st_y` are Databricks-native ST built-ins (already used across GeoBrix docs, e.g. `docs/tests/python/api/sql_api.py`). Confirm the GeoBrix reader's geometry column is WKB (not EWKB/WKT); if it is already a native `GEOMETRY`, drop the `st_geomfromwkb(...)` wrapper. ST area is planar in the column CRS — for true m² either work in a projected CRS or use `GEOGRAPHY`/`st_area` semantics per the Databricks ST reference; this didactic step reports relative roof size. No placeholder — wire the real geometry encoding during implementation. - -- [ ] **Step 9d — code cell (native H3 roof-density aggregation):** - ```python - # Native H3: index each roof centroid to an H3 cell (res 11 ~ city-block scale) and - # aggregate roof count + total area per cell -> a roof-density surface. - H3_RES = 11 - roof_density = ( - roofs.selectExpr("*", f"h3_longlatash3(lon, lat, {H3_RES}) AS h3_cell") - .groupBy("h3_cell") - .agg(F.count("*").alias("n_roofs"), - F.sum("roof_area_m2").alias("total_roof_area_m2")) - ) - display(roof_density.orderBy(F.col("total_roof_area_m2").desc()).limit(10)) - ``` - > NOTE: `h3_longlatash3(lng, lat, res)` is the Databricks-native point->H3 built-in (arg order is longitude, latitude; see the Databricks H3 reference / `docs/docs/databricks-spatial.mdx` H3_POINT_INDEX). Confirm the exact name + arg order against the H3 functions reference during implementation (`h3_longlatash3` is the standard; some surfaces expose `h3_pointash3` over a geometry). Native H3 requires Photon or Databricks SQL (Pro/Serverless). No placeholder. - -- [ ] **Step 10 — markdown cell (MVT pyramid):** - ```markdown - ## 4. Encode + pyramid to vector tiles - - `gbx_st_asmvt_pyramid` is a table-valued function (UDTF): for each feature it emits - one `(z, x, y, mvt_bytes)` row per zoom level in the requested range, binning the - geometry into the web-mercator tile grid and encoding tile-local MVT. We pick a - city-scale zoom range (z12–z16). Attributes ride along natively (here just - `feature_id`). - ``` - -- [ ] **Step 11 — code cell (asmvt_pyramid via SQL):** - ```python - buildings.createOrReplaceTempView("sf_buildings") - mvt = spark.sql(""" - SELECT t.zoom AS z, t.tile_x AS x, t.tile_y AS y, t.mvt AS mvt - FROM sf_buildings, - LATERAL gbx_st_asmvt_pyramid(geometry, 12, 16, named_struct('feature_id', feature_id)) AS t - """) - sf_mvt = finalize_delta(mvt, "sf_buildings_mvt_tiles") - print(f"... {sf_mvt.count():,} (z,x,y) MVT tiles across z12-z16") - ``` - > NOTE: the exact `gbx_st_asmvt_pyramid` LATERAL output column names (`zoom`/`tile_x`/`tile_y`/`mvt`) must be confirmed against `pyvx` registration during implementation; if `gbx_st_asmvt` (single-tile, grouped) is the registered pyramid entry point instead of a UDTF, group buildings by an `gbx_st_tile_id`-style binning and aggregate per `(z,x,y)`. Confirm against `docs/docs/api/vectorx-functions.mdx` and adjust this cell to the real signature before finalizing — no placeholder. - -- [ ] **Step 12 — markdown cell (PMTiles agg):** - ```markdown - ## 5. Fold the tile pyramid into one PMTiles archive - - `gbx_pmtiles_agg` is a grouped aggregate that folds a set of `(mvt, z, x, y)` tiles - into a single PMTiles v3 archive (BINARY). We aggregate the whole pyramid into one - archive and write it to the Volume. - ``` - -- [ ] **Step 13 — code cell (pmtiles_agg + write):** - ```python - archive_row = ( - sf_mvt.groupBy(F.lit(1).alias("_g")) - .agg(F.expr("gbx_pmtiles_agg(mvt, z, x, y)").alias("archive")) - .select("archive") - .collect()[0] - ) - TILES_DIR = f"{HELIOS_DIR}/tiles" - dbutils.fs.mkdirs(TILES_DIR) - PMTILES_PATH = f"{TILES_DIR}/sf_buildings.pmtiles" - # FUSE-safe sequential write from the driver (single archive, bytes already in memory) - with open(PMTILES_PATH, "wb") as f: - f.write(archive_row["archive"]) - print(f"... wrote {PMTILES_PATH} ({os.path.getsize(PMTILES_PATH):,} bytes)") - ``` - -- [ ] **Step 14 — markdown cell (view):** - ```markdown - ## 6. View the vector PMTiles inline - - `show_pmtiles` prints the `pmtiles_info` header, then renders through the - `INTERACTIVE_PLOTS` toggle (set in `config_nb`): the default **False** produces a - fast static image that renders on GitHub and the docs site; set `INTERACTIVE_PLOTS = - True` for an interactive MapLibre layer (streamed in-browser, no tile server). - `plot_pmtiles` auto-detects the vector archive in either mode. - ``` - -- [ ] **Step 15 — code cell (view):** - ```python - show_pmtiles(PMTILES_PATH) - ``` - -- [ ] **Step 16 — markdown cell (recap):** - ```markdown - ## What we built - - - `overture_buildings_meta` (Delta) — the queryable asset catalog of downloaded GeoParquet. - - `sf_buildings_mvt_tiles` (Delta) — one row per `(z, x, y)` vector tile. - - `sf_buildings.pmtiles` (Volume) — a self-contained vector basemap of every candidate roof. - - A native **ST roof-area** table and an **H3 roof-density** aggregation — showing the - tiles compose directly with Databricks-native spatial. - - Next: **notebook 02** drapes NAIP aerial imagery behind these footprints as a visual basemap. - ``` - -- [ ] **Step 17 (VALIDATION):** `gbx:test:notebooks --path "examples/helios/01. Vector Engine (MVT).ipynb" --log helios-01.log` in Docker (with `/Volumes`). Because the runner remaps absolute Volume paths under a temp workdir by default, run with `--allow-absolute-reads --allow-absolute-writes` only if the Overture/sample data must come from the real Volume; otherwise rely on the remap + sample data. Expected: cells execute green; `sf_buildings.pmtiles` exists and `pmtiles_info(PMTILES_PATH)["tile_type"]` is the MVT type; `show_pmtiles` returns a rendered object (static image by default since `INTERACTIVE_PLOTS=False`); the native-ST roof-metrics + H3 roof-density cells produce non-empty results. If the network Overture catalog is unreachable in Docker, gate the discover/download cells behind an env check and fall back to a small committed sample GeoParquet under sample-data; record the assumption and keep the tiling+PMTiles+view cells live. NOTE: native `st_*`/`h3_longlatash3` need Databricks ST/H3 (Photon / Databricks SQL); the cell-by-cell Docker runner may lack them — if so, gate the native-ST/H3 cells behind a capability check (try the expr; skip with a printed note on failure) so tiling stays green, and record the gap. Resolve the exact native names in Task 7 Step 3. -- [ ] **Step 18 (commit):** `git add "notebooks/examples/helios/01. Vector Engine (MVT).ipynb" && git commit` — subject `feat(helios): add NB01 Overture buildings to vector PMTiles`; body WHY. Trailer. - ---- - -## Task 3: NB02 — Visual Basemap (XYZ raster) - -**Files:** -- Create: `notebooks/examples/helios/02. Visual Basemap (XYZ).ipynb` - -**Interfaces:** -- *Consumes (from `%run ./config_nb`):* `rx`, `register`, `plot_pmtiles`/`show_pmtiles`, `pmtiles_info`, `plot_file`, `HELIOS_DIR`, `SF_CITY_BBOX`, `finalize_delta`, `FORCE_REBUILD`. SQL: `gbx_rst_to_webmercator`, `gbx_rst_xyzpyramid`, `gbx_pmtiles_agg`. Reader: `binaryFile` → `rst_fromcontent` and/or `gtiff_gbx`. -- *Produces:* Volume dir `${HELIOS_DIR}/naip/` (staged NAIP GeoTIFF), Delta table `sf_naip_xyz_tiles` (per `(z,x,y)` PNG tile bytes), raster PMTiles `${HELIOS_DIR}/tiles/sf_naip.pmtiles`. - -- [ ] **Step 1 — markdown cell (title + meta-narrative):** - ```markdown - # Helios 02 — Visual Basemap: NAIP aerial imagery to raster PMTiles - - ![NAIP aerial to raster PMTiles](https://raw.githubusercontent.com/databrickslabs/geobrix/main/resources/images/diagrams/helios/helios-02.png) - - **Solar site-selection, step 2: the visual context.** Before scoring roofs, we want - to *see* them. This notebook stages **NAIP** (National Agriculture Imagery Program) - aerial imagery for San Francisco, reprojects it to web mercator with - `gbx_rst_to_webmercator`, slices it into an **XYZ tile pyramid** with - `gbx_rst_xyzpyramid`, packages the pyramid as raster **PMTiles** via - `gbx_pmtiles_agg`, and views it with `plot_pmtiles`. This aerial basemap sits behind - the building footprints from notebook 01. - - > Runs on the **lightweight tier (Serverless)** by default. - ``` - -- [ ] **Step 2 — code cell (`%run`):** - ```python - %run ./config_nb - ``` - -- [ ] **Step 3 — code cell (rebuild flag):** - ```python - FORCE_REBUILD = False - ``` - -- [ ] **Step 4 — markdown cell (NAIP staging — notebook helper, NOT a module):** - ```markdown - ## 1. Stage NAIP aerial imagery (notebook helper) - - NAIP is hosted as Cloud-Optimized GeoTIFFs in the public AWS Open Data registry - (`s3://naip-analytic`, public/requester-pays) and is also discoverable via STAC on - Planetary Computer (collection `naip`). NAIP does **not** get a module-level API in - GeoBrix — this is a **notebook-local helper** that fetches one SF tile and stages it - to the Volume (idempotent, FUSE-safe sequential copy). On Serverless it uses - rasterio's bundled GDAL (no `gdal_translate` CLI). - ``` - -- [ ] **Step 5 — code cell (NAIP staging helper):** - ```python - import shutil - from databricks.labs.gbx.sample import get_temp_dir # node-local scratch helper - - NAIP_DIR = f"{HELIOS_DIR}/naip" - dbutils.fs.mkdirs(NAIP_DIR) - NAIP_PATH = f"{NAIP_DIR}/sf_naip.tif" - - def stage_naip(dest=NAIP_PATH, bbox=SF_CITY_BBOX): - """Stage one SF NAIP COG to the Volume via Planetary Computer STAC (idempotent).""" - if os.path.exists(dest) and not FORCE_REBUILD: - print(f"... NAIP already staged at {dest}") - return dest - import planetary_computer as pc - import pystac_client, rasterio - cat = pystac_client.Client.open( - "https://planetarycomputer.microsoft.com/api/stac/v1", - modifier=pc.sign_inplace, - ) - minx, miny, maxx, maxy = bbox - item = next(cat.search(collections=["naip"], bbox=[minx, miny, maxx, maxy], - limit=1).items()) - href = item.assets["image"].href - tmp = get_temp_dir() - local = tmp / "sf_naip.tif" - with rasterio.open(href) as src: - profile = {**src.profile, "driver": "GTiff"} - win = src.window(minx, miny, maxx, maxy) # crop to AOI to keep volume demo-friendly - data = src.read(window=win) - profile.update(width=data.shape[2], height=data.shape[1], - transform=src.window_transform(win)) - with rasterio.open(local, "w", **profile) as dst: - dst.write(data) - shutil.copy(str(local), dest) # FUSE-safe sequential copy - print(f"... staged NAIP -> {dest} ({os.path.getsize(dest):,} bytes)") - return dest - - stage_naip() - ``` - -- [ ] **Step 6 — markdown cell (preview the source):** - ```markdown - ## 2. Preview the source imagery - - `plot_file` renders the staged GeoTIFF straight from the Volume (auto-decimation, - per-band percentile stretch). This is a static source preview (a raw source raster has - no tiled interactive form); the tiled PMTiles **product** is what the `INTERACTIVE_PLOTS` - toggle governs at the view step below. - ``` - -- [ ] **Step 7 — code cell (preview):** - ```python - plot_file(NAIP_PATH, fig_w=8, fig_h=6) - ``` - -- [ ] **Step 8 — markdown cell (load as tile):** - ```markdown - ## 3. Load the imagery into a typed tile - - We read the GeoTIFF bytes with the `binaryFile` reader and build a typed `tile` - struct via `rst_fromcontent` — the temp-file-free path that avoids executor races. - ``` - -- [ ] **Step 9 — code cell (binaryFile → rst_fromcontent):** - ```python - naip = ( - spark.read.format("binaryFile").load(NAIP_PATH) - .select(rx.rst_fromcontent(F.col("content")).alias("tile")) - ) - print(f"... loaded {naip.count()} source tile(s)") - ``` - -- [ ] **Step 10 — markdown cell (reproject):** - ```markdown - ## 4. Reproject to web mercator - - XYZ / PMTiles tiles live in web mercator (EPSG:3857). `gbx_rst_to_webmercator` - reprojects the tile so the pyramid aligns to the slippy-map grid. - ``` - -- [ ] **Step 11 — code cell (to_webmercator):** - ```python - naip_3857 = naip.select(rx.rst_to_webmercator("tile").alias("tile")) - ``` - -- [ ] **Step 12 — markdown cell (xyzpyramid):** - ```markdown - ## 5. Build the XYZ tile pyramid - - `gbx_rst_xyzpyramid` slices the reprojected raster into a pyramid of slippy-map PNG - tiles across a zoom range, emitting `(z, x, y, tile_bytes)` rows. We pick a - city-scale zoom range (z12–z16) to match notebook 01. - ``` - -- [ ] **Step 13 — code cell (xyzpyramid):** - ```python - naip_3857.createOrReplaceTempView("sf_naip_tile") - xyz = spark.sql(""" - SELECT p.zoom AS z, p.tile_x AS x, p.tile_y AS y, p.tile AS png - FROM sf_naip_tile, - LATERAL gbx_rst_xyzpyramid(tile, 12, 16) AS p - """) - sf_xyz = finalize_delta(xyz, "sf_naip_xyz_tiles") - print(f"... {sf_xyz.count():,} (z,x,y) raster tiles across z12-z16") - ``` - > NOTE: confirm the `gbx_rst_xyzpyramid` output column names + arg order (tile, min_zoom, max_zoom) against `docs/docs/api/raster-functions.mdx` / pyrx registration during implementation; adjust to the real signature — no placeholder. - -- [ ] **Step 14 — markdown cell (pmtiles agg):** - ```markdown - ## 6. Package as raster PMTiles - - Same `gbx_pmtiles_agg` aggregate as notebook 01 — it auto-detects PNG tiles and - writes a **raster** PMTiles archive. - ``` - -- [ ] **Step 15 — code cell (pmtiles_agg + write):** - ```python - archive_row = ( - sf_xyz.groupBy(F.lit(1).alias("_g")) - .agg(F.expr("gbx_pmtiles_agg(png, z, x, y)").alias("archive")) - .select("archive").collect()[0] - ) - TILES_DIR = f"{HELIOS_DIR}/tiles" - dbutils.fs.mkdirs(TILES_DIR) - NAIP_PMTILES = f"{TILES_DIR}/sf_naip.pmtiles" - with open(NAIP_PMTILES, "wb") as f: - f.write(archive_row["archive"]) - print(f"... wrote {NAIP_PMTILES} ({os.path.getsize(NAIP_PMTILES):,} bytes)") - ``` - -- [ ] **Step 16 — markdown cell (view):** - ```markdown - ## 7. View the raster PMTiles inline - - `show_pmtiles` renders through the `INTERACTIVE_PLOTS` toggle — a static image by - default (GitHub/docs-renderable), or an interactive MapLibre raster layer when - `INTERACTIVE_PLOTS = True`. - ``` - -- [ ] **Step 17 — code cell (view):** - ```python - show_pmtiles(NAIP_PMTILES) - ``` - -- [ ] **Step 18 — markdown cell (recap):** - ```markdown - ## What we built - - - `sf_naip_xyz_tiles` (Delta) — one row per `(z, x, y)` PNG tile. - - `sf_naip.pmtiles` (Volume) — a self-contained aerial basemap. - - Next: **notebook 03** adds the analytical layer — terrain slope and aspect from a - USGS 3DEP DEM, the inputs to a solar suitability score. - ``` - -- [ ] **Step 19 (VALIDATION):** `gbx:test:notebooks --path "examples/helios/02. Visual Basemap (XYZ).ipynb" --log helios-02.log` in Docker. Expected: green cells; `sf_naip.pmtiles` exists; `pmtiles_info` reports a raster `tile_type` (PNG/JPEG/WebP); `show_pmtiles` renders. If NAIP STAC is unreachable in Docker, gate `stage_naip` behind a reachability check and fall back to the committed `srtm_n37w123.tif` (or a small RGB sample) so the reproject→pyramid→PMTiles→view chain still runs; record the assumption. -- [ ] **Step 20 (commit):** `git add "notebooks/examples/helios/02. Visual Basemap (XYZ).ipynb" && git commit` — subject `feat(helios): add NB02 NAIP imagery to raster PMTiles`; body WHY. Trailer. - ---- - -## Task 4: NB03 — Analytical Core (COG + STAC) - -**Files:** -- Create: `notebooks/examples/helios/03. Analytical Core (COG + STAC).ipynb` - -**Interfaces:** -- *Consumes (from `%run ./config_nb`):* `rx`, `register`, `stac_client`, `plot_cog`/`show_cog`, `plot_pmtiles`/`show_pmtiles`, `pmtiles_info`, `solar_score`, `finalize_delta`, `HELIOS_DIR`, `SF_CITY_BBOX`, `FORCE_REBUILD`. GeoBrix SQL: `gbx_rst_cog_convert`, terrain `rst_terrainslope`/`rst_terrainaspect`/`rst_hillshade` (confirm registered names), `gbx_rst_h3_rastertogridavg`, `gbx_rst_to_webmercator`, `gbx_rst_xyzpyramid`, `gbx_pmtiles_agg`. Databricks-native SQL (the on-ramp composition): `h3_centeraswkb` (companions `h3_boundaryaswkb`/`h3_hexring`). -- *Produces:* Volume dir `${HELIOS_DIR}/dem/` (staged 3DEP DEM), Volume dir `${HELIOS_DIR}/cog/` (COGs), Delta table `sf_cog_catalog` (STAC catalog of the COGs), Delta table `sf_terrain` (slope/aspect/hillshade tiles), Delta table `sf_solar_cells` (per-H3-cell avg slope/aspect + `solar_score` + native H3 cell geometry), raster PMTiles `${HELIOS_DIR}/tiles/sf_hillshade.pmtiles`. - -- [ ] **Step 1 — markdown cell (title + meta-narrative):** - ```markdown - # Helios 03 — Analytical Core: terrain, COGs, STAC, and solar scoring - - ![3DEP DEM to COG + STAC + hillshade PMTiles](https://raw.githubusercontent.com/databrickslabs/geobrix/main/resources/images/diagrams/helios/helios-03.png) - - **Solar site-selection, step 3: the analytical layer.** Roof solar yield depends on - **slope** and **aspect** (south-facing, gently sloped wins). This notebook stages a - **USGS 3DEP** DEM for San Francisco, converts it to **Cloud-Optimized GeoTIFFs** with - `gbx_rst_cog_convert`, **catalogs the COGs into a queryable STAC Delta table** with - `StacClient`, derives slope/aspect/hillshade, aggregates them into a per-**H3-cell** - `solar_score` index (composing with Databricks-native H3), and renders the hillshade as - PMTiles. The COG + STAC catalog is the analysis-ready, time-travel-friendly artifact; - the H3 solar-suitability cells are the grid-indexed analytical layer; the hillshade - PMTiles is the human-readable relief view. - - > GeoBrix tiling and raster→grid aggregation are an on-ramp into Databricks-native - > spatial: the H3 `cellid` we produce is a standard native id you can join and render - > with native H3 functions. - - > Runs on the **lightweight tier (Serverless)** by default. - ``` - -- [ ] **Step 2 — code cell (`%run`):** - ```python - %run ./config_nb - ``` - -- [ ] **Step 3 — code cell (rebuild flag):** - ```python - FORCE_REBUILD = False - ``` - -- [ ] **Step 4 — markdown cell (DEM staging helper):** - ```markdown - ## 1. Stage a USGS 3DEP DEM (notebook helper) - - USGS 3DEP elevation is on the public AWS Open Data registry and discoverable via - Planetary Computer STAC (collection `3dep-seamless`). Like NAIP, 3DEP stays a - **notebook-local helper** — no module API. We stage one SF DEM tile to the Volume - (idempotent). For offline runs we fall back to the h3-rasterize SRTM tile already - staged at `geobrix-examples/sf/elevation/srtm_n37w123.tif`. - ``` - -- [ ] **Step 5 — code cell (DEM staging helper):** - ```python - import shutil - from databricks.labs.gbx.sample import get_temp_dir - - DEM_DIR = f"{HELIOS_DIR}/dem" - dbutils.fs.mkdirs(DEM_DIR) - DEM_PATH = f"{DEM_DIR}/sf_3dep.tif" - SRTM_FALLBACK = "/Volumes/geospatial_docs/geobrix/sample-data/geobrix-examples/sf/elevation/srtm_n37w123.tif" - - def stage_dem(dest=DEM_PATH, bbox=SF_CITY_BBOX): - """Stage one SF 3DEP DEM tile via Planetary Computer STAC; fall back to the - already-staged SRTM tile when offline (idempotent).""" - if os.path.exists(dest) and not FORCE_REBUILD: - print(f"... DEM already staged at {dest}") - return dest - try: - import planetary_computer as pc - import pystac_client, rasterio - cat = pystac_client.Client.open( - "https://planetarycomputer.microsoft.com/api/stac/v1", - modifier=pc.sign_inplace, - ) - minx, miny, maxx, maxy = bbox - item = next(cat.search(collections=["3dep-seamless"], - bbox=[minx, miny, maxx, maxy], limit=1).items()) - href = item.assets["data"].href - tmp = get_temp_dir(); local = tmp / "sf_3dep.tif" - with rasterio.open(href) as src: - win = src.window(minx, miny, maxx, maxy) - data = src.read(window=win) - profile = {**src.profile, "driver": "GTiff", - "width": data.shape[2], "height": data.shape[1], - "transform": src.window_transform(win)} - with rasterio.open(local, "w", **profile) as dst: - dst.write(data) - shutil.copy(str(local), dest) - print(f"... staged 3DEP -> {dest}") - except Exception as e: - print(f"... 3DEP STAC unavailable ({type(e).__name__}); falling back to SRTM") - shutil.copy(SRTM_FALLBACK, dest) - return dest - - stage_dem() - plot_file(DEM_PATH, fig_w=8, fig_h=6) - ``` - -- [ ] **Step 6 — markdown cell (COG convert):** - ```markdown - ## 2. Convert the DEM to Cloud-Optimized GeoTIFF - - `gbx_rst_cog_convert` rewrites the raster as a COG (internal tiling + overviews) so - downstream tools can do fast windowed/overview reads. We load the DEM as a typed - tile, convert, and write the COG bytes to the Volume. - ``` - -- [ ] **Step 7 — code cell (cog_convert + write):** - ```python - dem = ( - spark.read.format("binaryFile").load(DEM_PATH) - .select(rx.rst_fromcontent(F.col("content")).alias("tile")) - ) - cog = dem.select(rx.rst_cog_convert("tile").alias("tile")) # SQL: gbx_rst_cog_convert - COG_DIR = f"{HELIOS_DIR}/cog"; dbutils.fs.mkdirs(COG_DIR) - COG_PATH = f"{COG_DIR}/sf_dem_cog.tif" - cog_bytes = cog.select(rx.rst_asbinary("tile").alias("b")).collect()[0]["b"] - with open(COG_PATH, "wb") as f: - f.write(cog_bytes) - print(f"... wrote COG {COG_PATH} ({os.path.getsize(COG_PATH):,} bytes)") - ``` - > NOTE: confirm the bytes-extraction accessor (`rst_asbinary` vs `rst_tobytes` vs a `.tile.raster` struct field) and `rst_cog_convert` arg signature against pyrx registration during implementation; use the `gtiff_gbx`/`pmtiles`-style writer if that is the canonical COG write path. No placeholder — wire the real accessor. - -- [ ] **Step 8 — markdown cell (view the COG):** - ```markdown - ## 3. View the COG - - `plot_cog` does a rasterio overview read and renders the elevation surface. - ``` - -- [ ] **Step 9 — code cell (plot_cog):** - ```python - show_cog(COG_PATH) - ``` - -- [ ] **Step 10 — markdown cell (STAC catalog the COGs):** - ```markdown - ## 4. Catalog the COGs into a STAC Delta table - - We register the COG(s) as STAC items in a queryable Delta table — a re-runnable, - time-travel-friendly catalog of the analysis-ready elevation assets, keyed by their - Volume path. This is the same cataloging shape the EO series uses for downloaded assets. - ``` - -- [ ] **Step 11 — code cell (build STAC catalog rows):** - ```python - import rasterio - with rasterio.open(COG_PATH) as src: - b = src.bounds - cog_meta = [( - "sf_dem_cog", COG_PATH, str(src.crs), - float(b.left), float(b.bottom), float(b.right), float(b.top), - "3dep-seamless", - )] - cog_df = spark.createDataFrame( - cog_meta, - "item_id string, source string, crs string, minx double, miny double, " - "maxx double, maxy double, collection string", - ) - sf_cog_catalog = finalize_delta(cog_df, "sf_cog_catalog") - display(sf_cog_catalog) - ``` - > NOTE: if `StacClient` exposes a `catalog(...)`/`register_items(...)` helper for local COGs, prefer it over hand-building rows; confirm the StacClient surface during implementation and use the real method if present — otherwise the hand-built Delta catalog above is the documented fallback. - -- [ ] **Step 12 — markdown cell (slope/aspect/hillshade + solar score):** - ```markdown - ## 5. Derive slope, aspect, hillshade, and a solar score - - Slope and aspect come straight from the DEM tile (`rst_terrainslope` / `rst_terrainaspect`, - auto-scaled from the CRS). `solar_score` (defined in `config_nb`) favors gently sloped, - south-facing terrain. `rst_hillshade` produces the relief shading we tile for the view. - ``` - -- [ ] **Step 13 — code cell (terrain + score):** - ```python - terrain = cog.select( - rx.rst_terrainslope("tile").alias("slope_tile"), - rx.rst_terrainaspect("tile").alias("aspect_tile"), - rx.rst_hillshade("tile").alias("hillshade_tile"), - ) - sf_terrain = finalize_delta(terrain, "sf_terrain", do_display=True) - print("... slope / aspect / hillshade tiles materialized") - # solar_score(slope_col, aspect_col) is applied per H3 cell in the next step (5b), - # where slope+aspect are aggregated onto an H3 grid via gbx_rst_h3_rastertogridavg. - ``` - > NOTE: confirm the registered terrain function names (`rst_terrainslope`/`rst_slope`, `rst_terrainaspect`/`rst_aspect`, `rst_hillshade`) and whether they take a CRS-scale arg, against `docs/docs/api/raster-functions.mdx`. The repo memory "Terrain CRS-scale GDAL-normal" notes slope/hillshade auto-scale from CRS — no manual `-s`. Wire the real names; no placeholder. - -- [ ] **Step 13b — markdown cell (native H3 solar-suitability index):** - ```markdown - ## 5b. Aggregate slope + aspect into a per-H3-cell solar-suitability index - - GeoBrix raster→grid aggregation bins the slope and aspect rasters onto **H3 cells** - (`rst_h3_rastertogridavg`), emitting a standard H3 integer `cellid` per cell. Because - that `cellid` is a native H3 id, the result joins and renders directly with - **Databricks-native H3** functions — `h3_centeraswkb` / `h3_boundaryaswkb` for cell - geometry, `h3_hexring` for neighborhoods. We apply `solar_score(slope, aspect)` (from - `config_nb`) per cell to get a coarse south-facing-gentle-slope suitability surface. - This is the analytical payoff: a queryable, grid-indexed solar-suitability layer that - composes with native H3 and the NB01 roof-density cells on the same index. - ``` - -- [ ] **Step 13c — code cell (raster→H3 grid + solar score + native H3 geometry):** - ```python - H3_RES = 11 - # GeoBrix raster->H3 grid aggregation: mean slope + mean aspect per H3 cell. - cog.createOrReplaceTempView("sf_cog_tile") - cells = spark.sql(f""" - SELECT s.cellID AS cellid, s.measure AS avg_slope, a.measure AS avg_aspect - FROM (SELECT t.cellID, t.measure - FROM sf_cog_tile, - LATERAL gbx_rst_h3_rastertogridavg(rst_terrainslope(tile), {H3_RES}) t) s - JOIN (SELECT t.cellID, t.measure - FROM sf_cog_tile, - LATERAL gbx_rst_h3_rastertogridavg(rst_terrainaspect(tile), {H3_RES}) t) a - ON s.cellID = a.cellID - """) - # solar_score expects degree columns; alias to its defaults. - scored = cells.select( - "cellid", "avg_slope", "avg_aspect", - solar_score(slope_col="avg_slope", aspect_col="avg_aspect").alias("solar_score"), - ) - # Native H3: cellid is a standard H3 id -> native cell geometry for joins/rendering. - scored = scored.selectExpr( - "*", "h3_centeraswkb(cellid) AS cell_center_wkb" - ) - sf_solar_cells = finalize_delta(scored, "sf_solar_cells") - display(sf_solar_cells.orderBy(F.col("solar_score").desc()).limit(10)) - ``` - > NOTE: confirm the GeoBrix raster→H3 grid function name + LATERAL output columns (`gbx_rst_h3_rastertogridavg` and its `cellID`/`measure` fields) against `docs/docs/api/raster-functions.mdx` (the `rst_h3_rastertogrid*` family) and the registered terrain names (`rst_terrainslope`/`rst_terrainaspect`). `h3_centeraswkb(cellid)` is the Databricks-native H3 cell-center built-in (companion: `h3_boundaryaswkb`, `h3_hexring`); confirm the exact native name + that it accepts an integer cell id against the Databricks H3 functions reference. Native H3 requires Photon or Databricks SQL. No placeholder — wire the real signatures. - -- [ ] **Step 14 — markdown cell (hillshade → PMTiles):** - ```markdown - ## 6. Tile the hillshade to raster PMTiles - - Reproject the hillshade to web mercator, pyramid it, and fold into PMTiles — the same - `to_webmercator → xyzpyramid → pmtiles_agg` chain as notebook 02, now over the relief. - ``` - -- [ ] **Step 15 — code cell (hillshade pmtiles):** - ```python - hs_3857 = sf_terrain.select(rx.rst_to_webmercator("hillshade_tile").alias("tile")) - hs_3857.createOrReplaceTempView("sf_hillshade_tile") - hs_xyz = spark.sql(""" - SELECT p.zoom AS z, p.tile_x AS x, p.tile_y AS y, p.tile AS png - FROM sf_hillshade_tile, - LATERAL gbx_rst_xyzpyramid(tile, 11, 14) AS p - """) - archive_row = ( - hs_xyz.groupBy(F.lit(1).alias("_g")) - .agg(F.expr("gbx_pmtiles_agg(png, z, x, y)").alias("archive")) - .select("archive").collect()[0] - ) - TILES_DIR = f"{HELIOS_DIR}/tiles"; dbutils.fs.mkdirs(TILES_DIR) - HS_PMTILES = f"{TILES_DIR}/sf_hillshade.pmtiles" - with open(HS_PMTILES, "wb") as f: - f.write(archive_row["archive"]) - print(f"... wrote {HS_PMTILES} ({os.path.getsize(HS_PMTILES):,} bytes)") - ``` - -- [ ] **Step 16 — code cell (view):** - ```python - show_pmtiles(HS_PMTILES) - ``` - -- [ ] **Step 17 — markdown cell (recap + series close):** - ```markdown - ## What we built — and the full picture - - - `sf_cog_catalog` (Delta) — the queryable STAC catalog of analysis-ready COGs. - - `sf_terrain` (Delta) — slope / aspect / hillshade tiles. - - `sf_solar_cells` (Delta) — per-H3-cell avg slope/aspect + `solar_score`, with native - H3 cell geometry — joins directly with the NB01 roof-density cells on the same index. - - `sf_hillshade.pmtiles` (Volume) — the relief view. - - Across the series we built three PMTiles layers over one SF AOI — **buildings** - (vector, NB01), **NAIP aerial** (raster, NB02), and **hillshade** (raster, NB03) — - plus a COG + STAC catalog of the elevation. Stack the building footprints over the - aerial basemap, score each roof by the terrain `solar_score`, and you have an - end-to-end distributed solar site-selection pipeline — ingest → tile → PMTiles → view, - all on Databricks. - ``` - -- [ ] **Step 18 (VALIDATION):** `gbx:test:notebooks --path "examples/helios/03. Analytical Core (COG + STAC).ipynb" --log helios-03.log` in Docker. Expected: green cells; `sf_dem_cog.tif` exists and `plot_cog` renders; `sf_cog_catalog`/`sf_terrain`/`sf_solar_cells` tables exist (`sf_solar_cells` has a `solar_score` column and a native-H3 `cell_center_wkb`); `sf_hillshade.pmtiles` exists and `pmtiles_info` reports a raster type. The DEM staging helper already falls back to the staged SRTM tile offline, so this notebook should validate fully in Docker against the committed sample DEM. NOTE: native H3 (`h3_centeraswkb`) needs Photon / Databricks SQL; the cell-by-cell Docker runner may not have native H3 registered — if so, gate the native-H3 columns behind a capability check (try the expr, fall back to skipping the `cell_center_wkb` column) so the rest of the notebook stays green, and record the gap. The GeoBrix raster→grid aggregation + `solar_score` run in either environment. -- [ ] **Step 19 (commit):** `git add "notebooks/examples/helios/03. Analytical Core (COG + STAC).ipynb" && git commit` — subject `feat(helios): add NB03 DEM to COG+STAC + hillshade PMTiles`; body WHY. Trailer. - ---- - -## Task 5: README.md - -**Files:** -- Create: `notebooks/examples/helios/README.md` - -**Interfaces:** Documents the artifacts NB01–NB03 produce by their exact table/Volume names; references the three `helios-0N.png` images by `../../../resources/images/...` relative path (matching eo-series/h3-rasterize). No code. - -- [ ] **Step 1:** Write README.md mirroring the eo-series/h3-rasterize structure, sections in order: - - **Title + one-paragraph intro** — "Helios — Distributed Tiling to PMTiles" framing: one SF AOI, three layers (buildings/MVT, NAIP/XYZ, terrain/COG+STAC), all to PMTiles, solar site-selection narrative. Link to GeoBrix docs. - - **Lightweight-tier blockquote** — light `[light,stac,vizx]` default on Serverless; heavyweight switch via option-2; link Execution Tiers. (Copy the phrasing pattern from h3-rasterize/eo-series.) - - **Data-source blockquote** — Overture Maps (buildings) via `OvertureClient`; NAIP + USGS 3DEP via notebook helpers from Planetary Computer / AWS Open Data, with the SRTM offline fallback; all staged to the Volume idempotently. - - **Notebooks at a glance** — three `###` subsections (01/02/03), each with `![...](../../../resources/images/helios-0N.png)` and 3 bullet highlights (distributed fan-out, the key functions, the produced PMTiles artifact). Stress the distributed parallelism advantage over single-node tile generation (factual). For NB01 and NB03, one highlight notes the **Databricks-native** composition (NB01: ST roof area + H3 roof density; NB03: H3 per-cell `solar_score`) — the on-ramp framing, factual, not marketing. - - **Files** table — `config_nb.ipynb` + the three numbered notebooks + their one-line purpose (mirror the eo-series Files table wording). - - **Prerequisites** — DBR 17.3/18 LTS or Serverless (env v5+); GeoBrix 0.4.0 `[light,stac,vizx]` wheel; UC `catalog_name`/`schema_name` + a `data` Volume; heavyweight x86 + JAR + GDAL note. - - **Run order** — open config_nb, set catalog/schema, run 01→02→03; each starts `%run ./config_nb`; `FORCE_REBUILD=True` to rebuild. One line: the notebooks ship with `INTERACTIVE_PLOTS = False` so the committed `.ipynb` renders fast static maps on GitHub — set `INTERACTIVE_PLOTS = True` (in `config_nb` or after `%run`) for interactive folium/MapLibre maps. - - **Data flow** — an ASCII flow (the catchy data→tile→PMTiles spine), see Step 2. - - **Serverless execution strategy** — copy the eo-series section's substance (no `spark.conf` tuning outside `set_conf_safe`, `repartition(N, col)` not number-only, no `.cache()` → write Delta, sequential Volume I/O), trimmed to this series. - - **Key GeoBrix / Databricks functions shown** — `OvertureClient.discover/download/read`; `gbx_st_asmvt`, `gbx_st_asmvt_pyramid`; `gbx_rst_to_webmercator`, `gbx_rst_xyzpyramid`, `gbx_rst_cog_convert`, `gbx_rst_h3_rastertogridavg`, terrain; `gbx_pmtiles_agg`; `plot_pmtiles`, `plot_cog`, `pmtiles_info`; `StacClient`. Composed with **Databricks-native** spatial (the on-ramp): `st_geomfromwkb`/`st_area`/`st_centroid` for roof metrics (NB01), `h3_longlatash3` for roof density (NB01), `h3_centeraswkb` for H3 solar-suitability cell geometry (NB03). - - **Gotchas** — PMTiles read is driver-side only (no Spark read); Overture cloud-path read vs HTTP-href fallback; NAIP/3DEP network reachability + SRTM fallback; base64 embed size guard on `plot_pmtiles` (>64 MB → static); Serverless repartition-by-column. - - **Related resources** — links to the [Helios docs page], [EO-Series], [H3 Rasterize], RasterX/VectorX/VizX/PMTiles API pages. -- [ ] **Step 2:** Embed this ASCII data flow: - ```text - San Francisco AOI (one bbox, reused across all three notebooks) - │ - ┌─────┴───────────────┬─────────────────────────────┐ - ▼ ▼ ▼ - Overture buildings NAIP aerial (helper) USGS 3DEP DEM (helper) - (OvertureClient) │ │ - │ ▼ gbx_rst_to_webmercator ▼ gbx_rst_cog_convert - ▼ gbx_st_asmvt │ │ → COGs + STAC Delta - + st_asmvt_pyramid ▼ gbx_rst_xyzpyramid ▼ slope/aspect/hillshade - │ │ ▼ gbx_rst_xyzpyramid - ▼ gbx_pmtiles_agg ▼ gbx_pmtiles_agg ▼ gbx_pmtiles_agg - sf_buildings.pmtiles sf_naip.pmtiles sf_hillshade.pmtiles - │ │ │ - └─────────────────────┴──────────────┬──────────────┘ - ▼ - plot_pmtiles / plot_cog (inline) - → solar site-selection view - ``` -- [ ] **Step 3 (VALIDATION):** `grep -rniE "wave [0-9]+|wave-[0-9]+|subagent|dispatch|SP[0-9]|sub-project" notebooks/examples/helios/README.md` must print nothing (doc-voice). Confirm the three image paths resolve (`ls resources/images/helios-0{1,2,3}.png`). Render-preview the markdown to eyeball tables/flow. -- [ ] **Step 4 (commit):** `git add notebooks/examples/helios/README.md && git commit` — subject `docs(helios): add notebook-series README`; body WHY. Trailer. - ---- - -## Task 6: docs page + sidebar entry - -**Files:** -- Create: `docs/docs/notebooks/helios.mdx` -- Modify: `docs/sidebars.js` (add `'notebooks/helios'` to the Notebooks category) - -**Interfaces:** The MDX mirrors `docs/docs/notebooks/eo-series.mdx` structure; image refs use `../../../resources/images/helios-0N.png` (same relative depth as eo-series.mdx). Sidebar adds one entry. - -- [ ] **Step 1:** Write `docs/docs/notebooks/helios.mdx` mirroring `eo-series.mdx`: - - Frontmatter: `--- \n sidebar_position: 4 \n title: Helios — Tiling to PMTiles \n ---` (eo-series is position 1; h3-rasterize/xview follow; place Helios after them). - - `# Helios — Distributed Tiling to PMTiles` + the one-paragraph intro (SF AOI, three layers, solar narrative, on-ramp-to-Databricks-native framing where natural). - - `:::tip View on GitHub` admonition → `https://github.com/databrickslabs/geobrix/tree/main/notebooks/examples/helios`. - - `:::info Runs on the lightweight tier (Serverless) by default` admonition (copy the eo-series wording; light `[light,stac,vizx]`, heavyweight option-2 switch, `set_conf_safe`, Execution Tiers link). - - `:::note` on the PMTiles viewer (driver-side render, base64 in-browser FileSource, >64 MB static fallback) — link the [VizX](../api/vizx) and [PMTiles](../api/pmtiles-functions) pages. Add a one-line note that the notebooks ship with `INTERACTIVE_PLOTS = False` for GitHub-renderable static maps and that readers can set `INTERACTIVE_PLOTS = True` for interactive folium/MapLibre maps. - - `:::tip` (or a sentence in `## Run order`) — GeoBrix tiling composes with Databricks-native `ST_*`/H3 (the on-ramp): NB01 uses native ST roof area + H3 roof density, NB03 builds a per-H3-cell `solar_score`. Factual, no internal vocabulary. - - `## Notebooks at a glance` — three `###` subsections with `![...](../../../resources/images/helios-0N.png)` and 3 bullets each (same content as the README highlights). - - `## Files` table (config_nb + 3 notebooks). - - `## Prerequisites`, `## Run order`, `## Data flow` (the same ASCII flow as the README), `## Key GeoBrix / Databricks functions shown`, `## Gotchas` — all mirroring eo-series.mdx, trimmed to Helios. -- [ ] **Step 2:** Edit `docs/sidebars.js` — in the `Notebooks` category `items` array (currently `'notebooks/eo-series'`, `'notebooks/xview'`, `'notebooks/h3-rasterize'`), append `'notebooks/helios'`. -- [ ] **Step 3 (reciprocal cross-link sweep):** The series touches many functions/writers — add a link FROM each docs page/listing that showcases a Helios-touched function/writer TO the new series page (`../notebooks/helios`), mirroring the EXISTING convention (a one-line "worked example" sentence near the function + a Related-links list entry, exactly like the `[EO Series](../notebooks/eo-series)` / `[H3 rasterize notebook](../notebooks/h3-rasterize)` links already in `api/stac.mdx`, `api/vizx.mdx`, `api/raster-functions.mdx`). Link target is `../notebooks/helios` from every `docs/docs//*.mdx` (api/, writers/, sample-data/ are all one level under docs/docs). **Inventory of pages/places to touch** (add the link near the named function/section; confirm the exact anchor at edit time): - | Page | Function(s)/feature the series showcases | NB | - |---|---|---| - | `api/vectorx-functions.mdx` | `gbx_st_asmvt`, `gbx_st_asmvt_pyramid` | NB01 | - | `api/raster-functions.mdx` | `gbx_rst_to_webmercator`, `gbx_rst_xyzpyramid`, `gbx_rst_cog_convert`, terrain (slope/hillshade), `gbx_pmtiles_agg` (raster PMTiles) | NB02, NB03 | - | `api/pmtiles-functions.mdx` | `gbx_pmtiles_agg` (vector + raster) | NB01, NB02, NB03 | - | `writers/pmtiles.mdx` (+ `writers/overview.mdx`) | `.write.format("pmtiles")` writer | NB02/NB03 (large pyramids) | - | `api/vizx.mdx` | `plot_pmtiles`, `plot_cog`, `pmtiles_info` | all three | - | `api/stac.mdx` | `StacClient` (catalog COGs) — ADD helios ALONGSIDE the existing eo-series link | NB03 | - | `api/h3-raster-tessellation.mdx` | `gbx_rst_h3_rastertogridavg` (per-H3-cell solar score) | NB03 | - | `sample-data/overview.mdx` | Overture data source (`gbx.sample.overture`) acquisition | NB01 | - Only add the link on the CANONICAL function-listing page per function — do NOT spam the aggregate pages (`performance.mdx`, `execution-tiers.mdx`, `api/overview.mdx`) that merely mention the functions in passing (an `api/overview.mdx` one-line mention in the tiling/PMTiles narrative is optional). Keep docs voice clean. -- [ ] **Step 4 (VALIDATION):** Doc-voice grep **must be empty**: `grep -rniE "wave [0-9]+|wave-[0-9]+" docs/docs/notebooks/helios.mdx` prints nothing; also `grep -rniE "subagent|dispatch|\bSP[0-9]\b|sub-project" docs/docs/notebooks/helios.mdx` prints nothing. Confirm the reciprocal links resolve: `grep -rl "notebooks/helios" docs/docs` should list the inventory pages above (and `notebooks/helios.mdx` itself). Build-check the docs locally if practical (`gbx:docs:dev` or the docs build) to confirm the MDX parses and the sidebar entry resolves; at minimum confirm `notebooks/helios` matches the new file id and the three image paths exist. -- [ ] **Step 5 (commit):** `git add docs/docs/notebooks/helios.mdx docs/sidebars.js docs/docs/api/ docs/docs/writers/ docs/docs/sample-data/ && git commit` — subject `docs(helios): Helios docs page + sidebar + reciprocal function-page links`; body WHY (incl. the cross-link sweep). Trailer. - ---- - -## Task 7: Full-series Docker validation + doc-voice sweep + gains capture - -**Files:** none new (validation + possible fixups to Tasks 1–6 artifacts; performance corpus files only if a gain is found). - -**Interfaces:** End-to-end exercise of the whole series through `%run ./config_nb`. - -- [ ] **Step 1 (full-series run):** In the Docker container (started via `start_docker_with_volumes.sh`), run all four notebooks in order so `%run ./config_nb` state threads through: - ``` - gbx:test:notebooks --path "examples/helios/config_nb.ipynb" --log helios-all-00.log - gbx:test:notebooks --path "examples/helios/01. Vector Engine (MVT).ipynb" --log helios-all-01.log - gbx:test:notebooks --path "examples/helios/02. Visual Basemap (XYZ).ipynb" --log helios-all-02.log - gbx:test:notebooks --path "examples/helios/03. Analytical Core (COG + STAC).ipynb" --log helios-all-03.log - ``` - Tail each log; assert every cell reports OK and the three `*.pmtiles` archives + the COG exist. NOTE: the cell-by-cell runner does not chain `%run` across separate invocations — if a numbered notebook depends on config_nb globals, the runner must `exec` config_nb first (it handles `%run` by inlining). Confirm the runner inlines `%run ./config_nb`; if it does not, add a `notebooks/tests/examples/` thin pytest harness that execs config_nb then the notebook in one interpreter (mirror an existing harness) — fix the harness, do not work around in the notebook. -- [ ] **Step 2 (doc-voice final sweep):** `grep -rniE "wave [0-9]+|wave-[0-9]+" docs/docs/notebooks/helios.mdx` **must be empty**. Also sweep the notebooks + README: `grep -rniE "wave [0-9]+|subagent|dispatch|\bSP[0-9]\b|sub-project|orchestrator" notebooks/examples/helios/ docs/docs/notebooks/helios.mdx` — empty. Fix any leak inline and amend the owning task's commit (or a small `docs(helios): scrub internal vocabulary` commit). -- [ ] **Step 3 (binding/registered-name confirmation):** Confirm every **GeoBrix** SQL name used in the notebooks exists in `docs/tests-function-info/registered_functions.txt` (`gbx_st_asmvt`, `gbx_st_asmvt_pyramid`, `gbx_rst_to_webmercator`, `gbx_rst_xyzpyramid`, `gbx_rst_cog_convert`, `gbx_rst_h3_rastertogridavg`, `gbx_pmtiles_agg`, and the terrain names) and that the pyrx/pyvx Python wrappers used (`rx.rst_fromcontent`, `rx.rst_to_webmercator`, `rx.rst_cog_convert`, `rx.rst_terrainslope`/etc., `vx.*`) match real bindings. Separately confirm the **Databricks-native** names the on-ramp cells call (`st_geomfromwkb`, `st_area`, `st_centroid`, `st_x`, `st_y`, `h3_longlatash3`, `h3_centeraswkb`) against the Databricks ST / H3 SQL functions reference and `docs/docs/databricks-spatial.mdx` — these are Databricks built-ins, NOT in `registered_functions.txt`, so the binding-parity check does not cover them; verify name + arg order (esp. `h3_longlatash3(lng, lat, res)`) and the geometry encoding the GeoBrix reader emits (WKB vs already-native GEOMETRY). Where a NOTE in Tasks 2–4 flagged an unconfirmed signature (GeoBrix or native), resolve it now and patch the notebook cell to the real signature. No placeholder ships. -- [ ] **Step 4 (capture validated gains — standing practice):** If building/validating any notebook surfaced a tiling-path improvement (e.g. an XYZ/MVT pyramid or `pmtiles_agg` speedup, a Serverless repartition fix, a COG-convert windowing win), capture it per the spec's performance methodology: - - Create (if absent) `docs/superpowers/performance/README.md` (index) and one pattern file `docs/superpowers/performance/.md` with: problem → symptom/signature → the fix → applicability matrix (light-similar fns / heavy same+similar fns, verdict recorded even when "not applicable") → evidence/bench numbers → canonical code refs. - - Add a paired thin pointer memory (slug + one line) `[[linking]]` to that corpus file (keep `MEMORY.md` index entries one line; the file is over the size limit, so do not bloat it). - - If the gain touches a function classified by execution shape, reflect it in user-facing `docs/docs/api/performance.mdx` and `benchmarking.mdx` per the "bench changes → update docs" rule — kept distinct from the internal corpus. - - If **no** gain was found, record that verdict in the task commit body (one line) and skip the corpus files. -- [ ] **Step 5 (commit):** If Steps 1–4 produced fixups or corpus files: `git add -A && git commit` — subject `test(helios): validate full series + capture tiling gains` (or `docs(helios): scrub vocabulary + confirm signatures` if no perf file); body: what was validated, any signature corrections, the gains verdict (captured-as `` / none-found). Trailer. - ---- - -## Task 8: Execute the series end-to-end on Serverless (sequential run chain) - -Serverless is the series' **target** environment; this task runs the whole series there -as a real job — the real-environment counterpart to Task 7's Docker cell-by-cell validation. -The chain is `nb01 → nb02 → nb03`, each numbered notebook running with `config_nb` (which it -`%run`s as its first cell — separate job tasks do NOT share Python state, so `config_nb` must -execute inside each run; it is not a standalone predecessor task). The sequential `depends_on` -ordering exists to thread the **Volume/Delta artifacts** (nb01 writes `sf_buildings.pmtiles`; -nb02/nb03 read/extend the shared Volume + Delta tables), not Python state. - -**Files:** -- Create: `notebooks/examples/helios/run_series_serverless.py` (the runner). -- Create: `scripts/commands/gbx-run-helios-series.md` + `scripts/commands/gbx-run-helios-series.sh` (new `run` category, per the "Adding a `gbx:*` command" procedure). - -**Interfaces:** -- Consumes: the four committed notebooks (`config_nb`, `01. Vector Engine (MVT)`, `02. Visual Basemap (XYZ)`, `03. Analytical Core (COG + STAC)`); the staged wheel on the sample-data Volume (config_nb `%pip`-installs it per Task 1 Step 3); a Serverless-enabled workspace with the `geospatial_docs.helios` catalog/schema + `data` Volume pre-created. -- Produces: a Databricks multi-task **Serverless** job run; prints the `run_page_url` (always emit it). - -- [ ] **Step 1 — write the runner.** Uses the default Databricks SDK auth resolution (OAuth profile / `DATABRICKS_CONFIG_PROFILE`); uploads all four notebooks to one workspace dir so `%run ./config_nb` resolves; submits sequential Serverless notebook tasks. - -```python -# notebooks/examples/helios/run_series_serverless.py -"""Run the Helios series end-to-end on Databricks Serverless as a sequential -job-run chain (01 -> 02 -> 03). Each notebook %run-s config_nb as its first cell, -so config_nb executes inside every task (job tasks don't share Python state). The -depends_on chain threads the shared Volume/Delta artifacts, not Python state. - -Auth: default SDK resolution (OAuth profile / DATABRICKS_CONFIG_PROFILE). Serverless -is selected by submitting notebook tasks with NO cluster spec. -""" -from __future__ import annotations - -import argparse -import sys - -from databricks.sdk import WorkspaceClient -from databricks.sdk.service import jobs, workspace - -# config_nb is %run-ed inside each numbered notebook; it is NOT its own task. -ALL_NOTEBOOKS = ["config_nb", "01. Vector Engine (MVT)", - "02. Visual Basemap (XYZ)", "03. Analytical Core (COG + STAC)"] -ORDERED = ALL_NOTEBOOKS[1:] - - -def _upload(w: WorkspaceClient, local_dir: str, ws_dir: str) -> None: - w.workspace.mkdirs(ws_dir) - for name in ALL_NOTEBOOKS: - with open(f"{local_dir}/{name}.ipynb", "rb") as f: - content = f.read() - w.workspace.upload(f"{ws_dir}/{name}", content, - format=workspace.ImportFormat.JUPYTER, overwrite=True) - - -def _tasks(ws_dir: str) -> list: - tasks, prev = [], None - for i, name in enumerate(ORDERED): - key = f"helios_{i + 1:02d}" - tasks.append(jobs.SubmitTask( - task_key=key, - notebook_task=jobs.NotebookTask(notebook_path=f"{ws_dir}/{name}"), - depends_on=[jobs.TaskDependency(task_key=prev)] if prev else None, - )) # no cluster spec -> Serverless - prev = key - return tasks - - -def main(argv=None) -> int: - ap = argparse.ArgumentParser() - ap.add_argument("--local-dir", default="notebooks/examples/helios") - ap.add_argument("--workspace-dir", required=True, help="e.g. /Users//helios") - ap.add_argument("--profile", default=None) - ap.add_argument("--no-wait", action="store_true") - args = ap.parse_args(argv) - - w = WorkspaceClient(profile=args.profile) if args.profile else WorkspaceClient() - _upload(w, args.local_dir, args.workspace_dir) - run = w.jobs.submit(run_name="helios-series", tasks=_tasks(args.workspace_dir)) - url = w.jobs.get_run(run.run_id).run_page_url - print(f"helios-series submitted: {url}") # ALWAYS emit the run URL - if args.no_wait: - return 0 - final = w.jobs.wait_get_run_job_terminated_or_skipped(run_id=run.run_id) - state = final.state.result_state - print(f"helios-series finished: {state} — {url}") - return 0 if str(state).endswith("SUCCESS") else 1 - - -if __name__ == "__main__": - sys.exit(main()) -``` - -> NOTE: confirm the workspace allows Serverless jobs. Some SDK/job versions require an -> `environment_key` + `environments=[jobs.JobEnvironment(...)]` on serverless notebook tasks; -> config_nb does its own `%pip` install, so a bare serverless task should suffice — if the -> submit rejects it for a missing environment, add one minimal `environments` entry and set -> `environment_key` on each `SubmitTask`. Resolve against the SDK version pinned in the repo. - -- [ ] **Step 2 — add the `gbx:run:helios-series` command** (`.md` + `.sh`) per the repo procedure: source `common.sh`, support `--help`/`--log`, pass through `--workspace-dir` (required), `--profile`, `--no-wait`. The `.sh` resolves `PROJECT_ROOT` and invokes the runner: - -```bash -# scripts/commands/gbx-run-helios-series.sh (core) -python "$PROJECT_ROOT/notebooks/examples/helios/run_series_serverless.py" \ - --workspace-dir "$WORKSPACE_DIR" ${PROFILE:+--profile "$PROFILE"} ${NO_WAIT:+--no-wait} -``` - -- [ ] **Step 3 — `chmod +x scripts/commands/gbx-run-helios-series.sh`.** - -- [ ] **Step 4 (VALIDATION — real Serverless run, consumes compute):** authenticate (`databricks-authentication`), then: - ``` - gbx:run:helios-series --workspace-dir /Users//helios --log helios-serverless.log - ``` - Expected: the runner prints the `run_page_url`; the run reaches `SUCCESS`; in the UI the three tasks ran in order `helios_01 → helios_02 → helios_03` on Serverless; and on the Volume the three `*.pmtiles` archives + the COG + the `sf_solar_cells` Delta table exist (the same artifacts Task 7 asserts in Docker). Emit the `run_page_url` to the user. If a task fails, open its task run, fix the owning notebook (Task 2–4), re-stage the wheel if needed, and re-run. - -- [ ] **Step 5 — commit:** -```bash -git add notebooks/examples/helios/run_series_serverless.py \ - scripts/commands/gbx-run-helios-series.md scripts/commands/gbx-run-helios-series.sh -git commit -m "feat(helios): Serverless end-to-end series run chain - -Run the series on its target environment (Serverless) as a sequential -job: nb01 -> nb02 -> nb03, each %run-ing config_nb, depends_on threading -the shared Volume/Delta artifacts. gbx:run:helios-series wraps the SDK -jobs.submit runner and emits the run_page_url. - -Co-authored-by: Isaac" -``` - ---- - -## Self-review against the spec (SP3 + cross-cutting) - -- **Coverage:** config_nb spine ✓ (Task 1), NB01 MVT ✓ (Task 2), NB02 XYZ ✓ (Task 3), NB03 COG+STAC ✓ (Task 4), README ✓ (Task 5), helios.mdx + sidebar ✓ (Task 6), full-series Docker validation + doc-voice grep + gains capture ✓ (Task 7), diagrams ✓ (Task 0), Serverless end-to-end sequential run chain ✓ (Task 8 — the target-environment run). config_nb runs inside each task via `%run` (job tasks don't share Python state); `depends_on` threads the Volume/Delta artifacts. One SF AOI ✓; solar narrative ✓; per-notebook data→tile→PMTiles diagram ✓ (Task 0 generates the committed PNGs); ample plotting ✓ (`plot_file`/`plot_cog`/`show_pmtiles` in each NB); series-only helpers in config_nb ✓ (`solar_score`, `finalize_delta`, `show_pmtiles`/`show_cog`/`show_raster`); no SP1/SP2 re-implementation ✓ (consumed only). -- **Type consistency with pinned SP1/SP2 signatures:** `OvertureClient().discover(bbox, themes=...)`, `.download(assets_df, out_dir, *, table=..., validate=..., partitions=...)` returning `theme,type,source,path,...`, `.read(source, theme=, type=, bbox=)` — used verbatim in NB01. `plot_pmtiles(path, ...)`, `plot_cog(path, ...)`, `pmtiles_info(path)` — used verbatim; the toggle helpers use the pinned `plot_pmtiles(path, max_embed_mb=0, ...)` static form from SP2 (where `max_embed_mb=0` forces the static render) and `plot_cog` stays static-only per the SP2 decision. `plot_static`/`plot_interactive` imported from vizx for the `show_raster` toggle. -- **`INTERACTIVE_PLOTS` toggle (Refinement 1):** added to `config_nb` Step 12 (rebuild-control cell, next to `FORCE_REBUILD`), default `False` with the exact required comment. Viz imports (Step 11) add `plot_interactive`. Toggle-aware helpers (`show_pmtiles`/`show_cog`/`show_raster`, Step 17) branch on it — static (`plot_pmtiles(..., max_embed_mb=0)` / `plot_static` / `plot_cog`) by default, interactive (`plot_pmtiles(...)` MapLibre / `plot_interactive` folium) when `True`. NB01 view (Step 14 md + Step 15 `show_pmtiles`) and NB02 view (Step 16 md + Step 17 `show_pmtiles`) and NB03 views (`show_cog`/`show_pmtiles`) all route through the helpers; `plot_file` source-previews are intentionally left static (a raw source raster has no tiled interactive form, noted in NB02 Step 6). Global Constraint + README Run-order line + helios.mdx `:::note` carry the one-line reader instruction. -- **Native ST/H3 injection (Refinement 2) — where, and where NOT:** - - **NB01 (injected):** native `st_geomfromwkb` + `st_area`/`st_centroid`/`st_x`/`st_y` for roof area + centroid (Step 9c, "available roof space"), and native `h3_longlatash3` to bin roof centroids into an H3 **roof-density** aggregation (Step 9d). On-ramp framing in the intro + recap. - - **NB03 (injected):** GeoBrix `gbx_rst_h3_rastertogridavg` aggregates slope+aspect onto H3 cells, `solar_score` applied per cell, and native `h3_centeraswkb` gives the native H3 cell geometry (Step 13c → `sf_solar_cells`); the `cellid` is a standard native H3 id that joins NB01's roof-density cells on the same index. On-ramp framing in intro + recap. - - **NB02 (deliberately NOT):** a pure NAIP raster basemap step — no natural ST/H3 fit, so none was forced (per the "don't force it" instruction). - - **config_nb AOI (deliberately NOT):** the SF AOI stays a plain bbox tuple — Overture/NAIP/3DEP staging helpers consume `(minx,miny,maxx,maxy)` directly; expressing it as a native ST geometry / H3 cell set would not simplify any downstream cell, so it was left as-is. -- **Native function names flagged for confirmation (Task 7 Step 3):** `st_geomfromwkb`/`st_area`/`st_centroid`/`st_x`/`st_y` are confirmed in-repo (used in `docs/tests/python/api/sql_api.py` etc.); `h3_longlatash3` (arg order lng,lat,res) and `h3_centeraswkb` (+ companions `h3_boundaryaswkb`/`h3_hexring`) are flagged with NOTE blocks to confirm exact name + arg order against the Databricks H3 SQL reference / `databricks-spatial.mdx`. The reader-geometry encoding (WKB vs native GEOMETRY) is flagged too. All native cells carry a "gate behind a capability check, skip with a printed note if native ST/H3 unavailable in the Docker runner" instruction so tiling stays green; binding-parity does NOT cover native built-ins (not in `registered_functions.txt`). -- **Placeholder scan:** every cell has real narrative + real code; the few unconfirmed SQL signatures (GeoBrix and native) are flagged with an explicit "confirm + wire the real signature, no placeholder" NOTE and are resolved in Task 7 Step 3 before ship. -- **Doc voice:** README + mdx avoid internal vocabulary; the native-spatial framing is factual on-ramp language (no marketing); Task 5/6/7 grep gates enforce it (QC `internals-leak`). -- **Validation rigor:** each notebook task ends in a concrete Docker `gbx:test:notebooks` run with asserted artifacts (PMTiles archive exists + `pmtiles_info` parses + plot renders + native-derived tables non-empty / capability-gated), not just "should work." diff --git a/docs/superpowers/plans/2026-06-28-pmtiles-vector-merge.md b/docs/superpowers/plans/2026-06-28-pmtiles-vector-merge.md deleted file mode 100644 index ed461295c..000000000 --- a/docs/superpowers/plans/2026-06-28-pmtiles-vector-merge.md +++ /dev/null @@ -1,1222 +0,0 @@ -# gbx_pmtiles_agg Vector-Merge Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make `gbx_pmtiles_agg` produce correct multi-feature vector tiles so the `st_asmvt_pyramid → gbx_pmtiles_agg` pipeline preserves all features for real data. When multiple rows share the same `(z, x, y)` and their payloads sniff as MVT, decode each blob, union features per layer name, and re-encode one merged MVT at the same extent. Raster first-wins is unchanged. - -**Architecture:** The fix is contained to two files: `_agg_light.py` (light) and `PMTilesAcc.scala`/`PMTiles_Agg.scala` (heavy), plus a new `MvtMerger` object for the JVM decode path. The `_assemble_archive` function in the light tier changes from a `seen`-set deduplication loop to a group-by-tileid dict that accumulates payloads, then resolves each tileid through a vector-vs-raster branch. The heavy tier performs the same grouping inside `eval` on `PMTilesAcc.tiles`, using GDAL's OGR MVT driver which can both CREATE and OPEN `.pbf` datasources — the decode path is `gdal.OpenEx(vsimemPath, OF_VECTOR)` with the same `/vsimem/` scratch pattern `MvtWriter.encode` already uses for the write side. - -**Tech Stack:** Python 3.12, `mapbox_vector_tile` (already a `[light]` dep — `decode` + `encode`), `pmtiles` PyPI package (already present). Scala 2.13, GDAL OGR Java bindings (`gdal.OpenEx` + `GetLayer` + `GetNextFeature` for decode; `MvtWriter.encode` for re-encode), JTS for geometry WKB handling. - -**Reference spec:** `docs/superpowers/specs/2026-06-28-pmtiles-vector-merge-design.md` - ---- - -## Global Constraints - -- **TDD.** Each task: write the failing test first, confirm it fails with the expected message/assertion, implement the minimal fix, confirm green. -- **Both tiers at parity.** Light and heavy must produce equivalent merged tiles (same feature count, same layer names, same geometry type, same attribute values). -- **POLYGON parity test.** The mandatory cross-tier parity test MUST include a POLYGON feature (not just points) — points-only gives a false pass per the MVT tile-local contract. -- **Raster first-wins unchanged.** Every existing raster test in `test_agg_light_core.py` and `PMTiles_AggTest.scala` must stay green with zero modification. -- **Serverless-safe light tier.** `_agg_light.py` must NEVER use `._jvm`, `._jsc`, `.sparkContext`, `.rdd`, `.conf.set(`. No new imports beyond packages already in `[light]`. -- **No new Python dependencies.** `mapbox_vector_tile` is already present; `decode` is all that's needed on the light side. Do not add any new entries to `pyproject.toml`. -- **OOM/partition cap preserved.** The 100 MiB byte-count guard runs during accumulation (unchanged). Merged blobs may be larger than any single input blob; this is acceptable — the cap is on the raw accumulated bytes, not the merged output. -- **Scalastyle clean.** Heavy Scala additions must pass `gbx:lint:scalastyle` (matches CI) before committing. -- **Docker for heavy tests.** All Scala tests and the cross-tier parity test require the `geobrix-dev` Docker container. -- **Commit hygiene.** Subject ≤72 chars + a WHY body on every commit + `Co-authored-by: Isaac` trailer. Run `chmod -R u+rwX .git/objects` before each commit (env permission gotcha). - ---- - -## Heavy-tier MVT decode: critical pre-work finding - -**GDAL OGR MVT driver supports both CREATE and OPEN.** There is no separate JVM MVT decoder library required. The same driver used by `MvtWriter.encode` for creation can read `.pbf` files via `gdal.OpenEx(vsimemPath, OF_VECTOR)`. The pattern is: - -1. Write the blob bytes into a `/vsimem//0/0/0.pbf` path using `gdal.FileFromMemBuffer`. -2. Open the datasource with `gdal.OpenEx(vsimemPath, OF_VECTOR)`. -3. Iterate `ds.GetLayer(i)` → `layer.GetNextFeature()` to read `(layerName, geom_wkb, attrs)`. -4. Clean up with `gdal.Unlink` + `gdal.RmdirRecursive`. - -This same `/vsimem/` + `ReadDirRecursive` + `GetMemFileBuffer` idiom is already proven by `MvtWriter.encode` for the write side and by `RST_GridFromPoints.scala` for `gdal.OpenEx(..., OF_VECTOR)`. No new JVM dependency is needed. A new helper object `MvtDecoder` (placed in `com.databricks.labs.gbx.vectorx.mvt`) encapsulates this read path, mirroring the existing `MvtWriter` structure. Task 2, Step 1 adds and unit-tests `MvtDecoder` before wiring it into the UDAF. - ---- - -## Task 1: Light-tier vector merge in `_assemble_archive` - -**Goal:** `_assemble_archive` accumulates all payloads per tileid, detects MVT vs raster from the first non-null payload per group, merges MVT blobs via `mapbox_vector_tile.decode` + `encode`, and keeps raster first-wins. - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pmtiles/_agg_light.py` -- Modify: `python/geobrix/test/pmtiles_light/test_agg_light_core.py` - -**Interfaces:** -- Consumes: unchanged public signature `_assemble_archive(data, zs, xs, ys, metadata)`. -- Produces: same `Optional[bytes]` PMTiles archive. For a tileid with 2 MVT blobs the packed tile must decode to both features. For a tileid with 2 PNG blobs only the first is stored. - ---- - -- [ ] **Step 1: Write failing tests for vector merge + raster-unchanged regression** - -Add to `python/geobrix/test/pmtiles_light/test_agg_light_core.py`: - -```python -import mapbox_vector_tile as mvt -from shapely.geometry import Polygon, box -from shapely import to_wkb - - -def _real_mvt_blob(poly_coords, prop_id: int, layer: str = "bldg") -> bytes: - """Encode one POLYGON feature as a real MVT blob at extent=4096.""" - poly = Polygon(poly_coords) - return mvt.encode( - { - "name": layer, - "features": [ - { - "geometry": poly, - "properties": {"id": prop_id}, - } - ], - }, - default_options={"extents": 4096, "y_coord_down": True}, - ) - - -_POLY_A = _real_mvt_blob( - [(100, 100), (200, 100), (200, 200), (100, 200), (100, 100)], prop_id=1 -) -_POLY_B = _real_mvt_blob( - [(300, 300), (400, 300), (400, 400), (300, 400), (300, 300)], prop_id=2 -) -_POLY_C_OTHER_LAYER = _real_mvt_blob( - [(500, 500), (600, 500), (600, 600), (500, 600), (500, 500)], - prop_id=3, - layer="roads", -) - - -def _decode_mvt_from_archive(blob, z, x, y, tmp_path): - """Extract and decode the MVT blob for (z, x, y) from a PMTiles archive.""" - from pmtiles.reader import MmapSource, Reader - - p = tmp_path / "merge.pmtiles" - p.write_bytes(blob) - with open(p, "rb") as f: - r = Reader(MmapSource(f)) - raw = r.get(z, x, y) - assert raw is not None, f"tile ({z},{x},{y}) missing from archive" - return mvt.decode(raw) - - -def test_vector_merge_two_features_same_tileid(tmp_path): - """Two MVT blobs for the same (z,x,y) must merge into one tile with 2 features.""" - blob = _assemble_archive([_POLY_A, _POLY_B], [3, 3], [2, 2], [4, 4], {}) - assert blob is not None - decoded = _decode_mvt_from_archive(blob, 3, 2, 4, tmp_path) - assert "bldg" in decoded, f"layer 'bldg' missing; got layers: {list(decoded.keys())}" - feat_ids = {f["properties"]["id"] for f in decoded["bldg"]["features"]} - assert feat_ids == {1, 2}, f"expected both feature ids; got {feat_ids}" - - -def test_vector_merge_geometry_type_preserved(tmp_path): - """Merged features must retain POLYGON geometry type (not downgraded to Point).""" - blob = _assemble_archive([_POLY_A, _POLY_B], [3, 3], [2, 2], [4, 4], {}) - decoded = _decode_mvt_from_archive(blob, 3, 2, 4, tmp_path) - for feat in decoded["bldg"]["features"]: - assert feat["geometry"]["type"] == "Polygon", ( - f"feature id={feat['properties']['id']} geometry not Polygon: " - f"{feat['geometry']['type']}" - ) - - -def test_vector_merge_multi_layer(tmp_path): - """Blobs from different layers for the same tileid are both preserved.""" - blob = _assemble_archive( - [_POLY_A, _POLY_C_OTHER_LAYER], [3, 3], [2, 2], [4, 4], {} - ) - decoded = _decode_mvt_from_archive(blob, 3, 2, 4, tmp_path) - assert "bldg" in decoded and "roads" in decoded, ( - f"expected both layers; got {list(decoded.keys())}" - ) - - -def test_vector_merge_distinct_tileids_unchanged(tmp_path): - """Blobs for distinct tileids are stored separately — no cross-tile bleed.""" - blob = _assemble_archive([_POLY_A, _POLY_B], [3, 3], [2, 4], [4, 6], {}) - decoded_a = _decode_mvt_from_archive(blob, 3, 2, 4, tmp_path) - decoded_b = _decode_mvt_from_archive(blob, 3, 4, 6, tmp_path) - assert {f["properties"]["id"] for f in decoded_a["bldg"]["features"]} == {1} - assert {f["properties"]["id"] for f in decoded_b["bldg"]["features"]} == {2} - - -def test_raster_first_wins_unchanged(tmp_path): - """PNG tiles for the same (z,x,y) still keep first-wins (no change to raster path).""" - _PNG2 = b"\x89PNG\r\n\x1a\n" + b"\x01" * 16 - blob = _assemble_archive([_PNG, _PNG2], [1, 1], [0, 0], [0, 0], {}) - tiles = _decode(blob, tmp_path) - assert tiles[(1, 0, 0)] == _PNG, "raster first-wins violated after vector-merge change" -``` - -Run — expect all 5 new tests to fail (currently all same-tileid blobs drop to first-wins so `feat_ids == {1}` not `{1, 2}`): - -```bash -python/geobrix/.venv-pyrx/bin/python -m pytest \ - python/geobrix/test/pmtiles_light/test_agg_light_core.py \ - -k "vector_merge or raster_first_wins_unchanged" -v 2>&1 | tail -20 -``` - -Expected: 5 FAILED, specifically the merge tests show `AssertionError: expected both feature ids; got {1}`. - ---- - -- [ ] **Step 2: Add `_merge_mvt_blobs` helper to `_agg_light.py`** - -First, add `mapbox_vector_tile` to the existing top-of-file imports in `_agg_light.py` (it is already a `[light]` dependency — used by `pyvx/_mvt.py`; this is the first use in `pmtiles`): - -```python -import mapbox_vector_tile as mvt -``` - -Place this import in the stdlib/third-party block after the existing `pmtiles` imports. Then add the following private function **before** `_assemble_archive`: - -```python -def _merge_mvt_blobs(blobs: list[bytes], extent: int = 4096) -> bytes: - """Decode multiple single-feature MVT blobs and union features per layer name. - - Geometry stays in tile-local [0, extent] integer space — no reprojection - (each blob is already tile-local for the same (z,x,y); decode/encode round-trips - the local coords). Attributes are preserved per feature. - - Returns one merged MVT blob. If the list has a single blob, returns it directly - to avoid a decode/encode round-trip for the common single-feature case. - """ - if len(blobs) == 1: - return blobs[0] - layers: dict[str, list] = {} - for blob in blobs: - try: - decoded = mvt.decode(blob) - except Exception: - # Malformed blob: skip rather than crashing the whole group. - continue - for layer_name, layer_data in decoded.items(): - layers.setdefault(layer_name, []).extend(layer_data.get("features", [])) - if not layers: - return blobs[0] # nothing decoded cleanly; fall back to first - tile_spec = { - name: {"features": feats} - for name, feats in layers.items() - } - return mvt.encode(tile_spec, default_options={"extents": extent, "y_coord_down": True}) -``` - ---- - -- [ ] **Step 3: Rewrite `_assemble_archive` to group payloads by tileid** - -First, update the top-of-file import to add `TileType`: -```python -# Before: -from pmtiles.tile import Compression, zxy_to_tileid -# After: -from pmtiles.tile import Compression, TileType, zxy_to_tileid -``` - -Then replace the `seen`/`if tileid in seen: continue` logic with a per-tileid accumulator. The new logic: - -1. First pass: accumulate `tileid → [bytes]` (preserving arrival order for raster first-wins). Also track `first_payload` for tile-type sniff and running byte total. -2. Detect tile type once from `first_payload` (unchanged). -3. Second pass: for each tileid in sorted order, call `_merge_mvt_blobs(payloads)` if MVT, else take `payloads[0]` if raster. - -Replace the body of `_assemble_archive` in `_agg_light.py` with: - -```python -def _assemble_archive( - data: Sequence, - zs: Sequence, - xs: Sequence, - ys: Sequence, - metadata: Optional[dict] = None, -) -> Optional[bytes]: - """Fold a group's (bytes, z, x, y) tiles into one PMTiles v3 archive (bytes). - - Null payloads are skipped; an all-null/empty group returns None. For vector - (MVT) tiles, multiple blobs for the same (z,x,y) are merged into one - multi-feature tile (decode each, union features per layer, re-encode). - For raster (PNG/JPEG/WebP), first-write-wins is preserved. Tiles are - written in ascending Hilbert TileID order. - """ - # Phase 1: accumulate all non-null payloads per tileid. - tileid_payloads: dict[int, list[bytes]] = {} - tileid_coords: dict[int, tuple[int, int, int]] = {} - total = 0 - first_payload = None - for d, z, x, y in zip(data, zs, xs, ys): - if d is None: - continue - b = bytes(d) - total += len(b) - if total > _MAX_ARCHIVE_BYTES: - raise ValueError( - f"pmtiles_agg group payload exceeds {_MAX_ARCHIVE_BYTES} bytes; " - "split into more groups or fewer tiles per archive" - ) - tileid = zxy_to_tileid(int(z), int(x), int(y)) - if first_payload is None: - first_payload = b - tileid_payloads.setdefault(tileid, []).append(b) - tileid_coords[tileid] = (int(z), int(x), int(y)) - - if not tileid_payloads: - return None - - tile_type = sniff_tile_type(first_payload) - # sniff_tile_type returns pmtiles.tile.TileType; MVT is the fallback for - # non-PNG/JPEG/WebP/AVIF payloads. Add TileType to the existing top-of-file - # import: `from pmtiles.tile import Compression, TileType, zxy_to_tileid` - is_vector = (tile_type == TileType.MVT) - - # Phase 2: resolve each tileid to one output blob. - tiles = [] - for tileid in sorted(tileid_payloads.keys()): - z, x, y = tileid_coords[tileid] - payloads = tileid_payloads[tileid] - if is_vector and len(payloads) > 1: - resolved = _merge_mvt_blobs(payloads) - else: - resolved = payloads[0] - tiles.append((z, x, y, tileid, resolved)) - - info = build_header_info( - [(z, x, y) for (z, x, y, _, _) in tiles], - SlippyGrid(), - tile_type, - Compression.NONE, - metadata or {}, - ) - buf = io.BytesIO() - writer = Writer(buf) - for _, _, _, tileid, b in tiles: # already sorted - writer.write_tile(tileid, b) - writer.finalize(info.header_dict(), info.metadata) - return buf.getvalue() -``` - -**Note on tile-type detection:** `sniff_tile_type` (from `ds.tiles._header`) returns a `pmtiles.tile.TileType` enum value — `TileType.MVT` for any non-PNG/JPEG/WebP/AVIF payload (confirmed from `_header.py`). The `is_vector` check `tile_type == TileType.MVT` is the correct expression. - ---- - -- [ ] **Step 4: Run failing tests — now expect green** - -```bash -python/geobrix/.venv-pyrx/bin/python -m pytest \ - python/geobrix/test/pmtiles_light/test_agg_light_core.py -v 2>&1 | tail -30 -``` - -Expected: all tests pass including the 5 new ones plus all pre-existing tests (single_tile_roundtrip, multi_zoom_roundtrip, png_payload_roundtrip, metadata_roundtrip, null_payloads_skipped, empty_group_returns_none, duplicate_tileid_dropped, cap_exceeded_raises). - ---- - -- [ ] **Step 5: Run the full UDF test suite** - -```bash -python/geobrix/.venv-pyrx/bin/python -m pytest \ - python/geobrix/test/pmtiles_light/ -v 2>&1 | tail -30 -``` - -Expected: all tests pass. - ---- - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/src/databricks/labs/gbx/pmtiles/_agg_light.py \ - python/geobrix/test/pmtiles_light/test_agg_light_core.py -git commit -m "$(cat <<'EOF' -fix(pmtiles): merge multi-feature MVT blobs per (z,x,y) in light tier - -gbx_pmtiles_agg was dropping all but the first feature per tile for -vector (MVT) data, making the st_asmvt_pyramid→gbx_pmtiles_agg -pipeline produce single-feature tiles for real datasets. This fix -groups payloads by tileid, decodes and unions features per layer, -and re-encodes one merged MVT blob. Raster first-wins is unchanged. - -Co-authored-by: Isaac -EOF -)" -``` - ---- - -## Task 2: Heavy-tier MVT decode helper + vector merge in `PMTilesAcc` - -**Goal:** Add `MvtDecoder` (OGR MVT read via `/vsimem/`) to the heavy-tier `vectorx.mvt` package. Wire it into `PMTiles_Agg.eval` so that MVT tileids with >1 payload are decoded, unioned per layer, and re-encoded via the existing `MvtWriter`. Raster tileids keep first-wins. - -**Files:** -- New: `src/main/scala/com/databricks/labs/gbx/vectorx/mvt/MvtDecoder.scala` -- Modify: `src/main/scala/com/databricks/labs/gbx/pmtiles/PMTiles_Agg.scala` -- Modify: `src/test/scala/com/databricks/labs/gbx/pmtiles/PMTiles_AggTest.scala` - -**Interfaces:** -- `MvtDecoder.decode(blob: Array[Byte]): Seq[(String, Array[Byte], Map[String, Any])]` — returns `(layerName, geom_wkb, attrs)` tuples for all features across all layers in the blob. -- `PMTiles_Agg.eval` groups `buffer.tiles` by `zxy_to_tileid`, branches on `tileType == TILE_TYPE_MVT`, calls `MvtDecoder.decode` + accumulates features per layer + calls `MvtWriter.encode` per layer + assembles a merged blob. Raster tileids use `payloads.head`. - ---- - -- [ ] **Step 1: Write failing unit test for `MvtDecoder`** - -Add to `src/test/scala/com/databricks/labs/gbx/pmtiles/PMTiles_AggTest.scala` (or a new `MvtDecoderTest.scala` in `src/test/scala/com/databricks/labs/gbx/vectorx/mvt/`): - -```scala -import com.databricks.labs.gbx.vectorx.mvt.{MvtDecoder, MvtWriter} -import org.apache.spark.sql.catalyst.plans.PlanTest -import org.apache.spark.sql.test.SilentSparkSession - -class MvtDecoderTest extends PlanTest with SilentSparkSession { - - // Build a real MVT blob via MvtWriter (tile-local polygon + attrs). - private def encodePolygon(id: Int, x0: Int, y0: Int): Array[Byte] = { - import com.databricks.labs.gbx.vectorx.jts.JTS - import org.locationtech.jts.geom.{Coordinate, GeometryFactory} - val gf = new GeometryFactory() - val ring = gf.createLinearRing(Array( - new Coordinate(x0, y0), new Coordinate(x0 + 100, y0), - new Coordinate(x0 + 100, y0 + 100), new Coordinate(x0, y0 + 100), - new Coordinate(x0, y0) - )) - val poly = gf.createPolygon(ring) - val wkb = JTS.toWKB(poly) - MvtWriter.encode("bldg", 4096, Seq((wkb, Map("id" -> id)))) - } - - test("MvtDecoder round-trips a real polygon MVT blob") { - val blob = encodePolygon(42, 100, 100) - assert(blob.nonEmpty, "MvtWriter produced empty blob") - val features = MvtDecoder.decode(blob) - assert(features.nonEmpty, "MvtDecoder returned no features") - val (layerName, geomWkb, attrs) = features.head - assert(layerName == "bldg", s"expected layer 'bldg'; got '$layerName'") - assert(attrs.get("id").contains(42) || attrs.get("id").exists(_.toString == "42"), - s"expected id=42; got attrs=$attrs") - assert(geomWkb != null && geomWkb.nonEmpty, "geomWkb is empty") - } - - test("MvtDecoder returns empty Seq for empty byte array") { - assert(MvtDecoder.decode(Array.emptyByteArray).isEmpty) - } -} -``` - -Run in Docker — expect compilation failure (MvtDecoder does not exist yet): - -```bash -bash scripts/commands/gbx-docker-exec.sh \ - "mvn test -pl . -Dtest=MvtDecoderTest -DfailIfNoTests=false -P skipScoverage -q" \ - 2>&1 | tail -20 -``` - -Expected: `error: object MvtDecoder is not a member of package ...` or equivalent compile error. - ---- - -- [ ] **Step 2: Implement `MvtDecoder.scala`** - -Create `src/main/scala/com/databricks/labs/gbx/vectorx/mvt/MvtDecoder.scala`: - -```scala -package com.databricks.labs.gbx.vectorx.mvt - -import com.databricks.labs.gbx.rasterx.gdal.GDALManager -import com.databricks.labs.gbx.vectorx.jts.JTS -import org.gdal.gdal.gdal -import org.gdal.ogr.ogr.{GetDriverByName => OGRGetDriverByName} -import org.gdal.ogrConstants._ - -import scala.collection.mutable.ArrayBuffer -import scala.util.Try - -/** - * Decode a Mapbox Vector Tile (MVT) protobuf blob into features. - * - * Uses GDAL's OGR MVT driver (the same driver `MvtWriter.encode` uses for creation) - * opened in read mode via a `/vsimem/` scratch path. Mirrors `MvtWriter`'s resource - * management: every Dataset / Feature is `.delete()`'d and the `/vsimem/` tree is - * cleaned up before returning. - * - * GDAL thread-safety: OGR drivers are registered via `GDALManager.initOgr()` (the - * synchronized guard), matching the requirement in CLAUDE.md. The `/vsimem/` paths - * are UUID-namespaced to avoid collisions across concurrent Spark tasks. - * - * Returns `Seq[(layerName, geom_wkb, attrs)]`. Geometry WKB is in the tile-local - * pixel space of the input blob (no coordinate transformation). Features with null - * or empty geometries are skipped. If the blob is empty or unparseable, returns - * an empty Seq (never throws). - */ -object MvtDecoder { - - /** - * Decode `blob` into a flat sequence of `(layerName, geomWkb, attrs)` tuples. - * - * @param blob MVT protobuf bytes (tile-local coordinates). - * @return All features across all layers; empty Seq if the blob is empty or - * cannot be decoded. - */ - def decode(blob: Array[Byte]): Seq[(String, Array[Byte], Map[String, Any])] = { - if (blob == null || blob.isEmpty) return Seq.empty - MvtWriter.ensureNativeLoadedPublic() - GDALManager.initOgr() - - val uuid = java.util.UUID.randomUUID().toString.replace("-", "_") - val rootPath = s"/vsimem/gbx_mvtdec_$uuid" - // The OGR MVT reader expects a directory datasource with the tile at 0/0/0.pbf. - val pbfPath = s"$rootPath/0/0/0.pbf" - - Try(gdal.Mkdir(rootPath, 0)).toOption - Try(gdal.Mkdir(s"$rootPath/0", 0)).toOption - Try(gdal.Mkdir(s"$rootPath/0/0", 0)).toOption - gdal.FileFromMemBuffer(pbfPath, blob) - - val result = ArrayBuffer.empty[(String, Array[Byte], Map[String, Any])] - val driver = OGRGetDriverByName("MVT") - if (driver == null) return Seq.empty - - val ds = Try(driver.Open(rootPath, 0)).toOption.orNull - if (ds == null) { - gdal.RmdirRecursive(rootPath) - return Seq.empty - } - - try { - val layerCount = ds.GetLayerCount() - var li = 0 - while (li < layerCount) { - val layer = ds.GetLayer(li) - if (layer != null) { - val layerName = layer.GetName() - layer.ResetReading() - var feat = layer.GetNextFeature() - while (feat != null) { - try { - val geom = feat.GetGeometryRef() - if (geom != null) { - val wkb = geom.ExportToWkb() - if (wkb != null && wkb.nonEmpty) { - val attrs = readAttrs(feat) - result += ((layerName, wkb, attrs)) - } - } - } finally { - feat.delete() - } - feat = layer.GetNextFeature() - } - } - li += 1 - } - } finally { - ds.delete() - gdal.RmdirRecursive(rootPath) - } - result.toSeq - } - - /** Extract all field values from a feature as a Map[String, Any] with native types. */ - private def readAttrs(feat: org.gdal.ogr.Feature): Map[String, Any] = { - val defn = feat.GetDefnRef() - val count = defn.GetFieldCount() - val m = scala.collection.mutable.Map.empty[String, Any] - var i = 0 - while (i < count) { - val fieldDefn = defn.GetFieldDefn(i) - val name = fieldDefn.GetNameRef() - val fieldType = fieldDefn.GetFieldType() - val value: Any = fieldType match { - case OFTInteger => feat.GetFieldAsInteger(i) - case OFTInteger64 => feat.GetFieldAsInteger64(i) - case OFTReal => feat.GetFieldAsDouble(i) - case _ => feat.GetFieldAsString(i) - } - m(name) = value - i += 1 - } - m.toMap - } -} -``` - -**IMPORTANT implementation note:** `MvtWriter.ensureNativeLoaded()` is currently `private`. Before implementing `MvtDecoder`, either: -- Make `ensureNativeLoaded` `private[mvt]` in `MvtWriter.scala` so `MvtDecoder` (same package) can call it, **or** -- Duplicate the one-liner guard inside `MvtDecoder` (less preferred — inconsistent). - -Prefer `private[mvt]` visibility change in `MvtWriter.scala`. The method body is a single `System.load` check — this is a minimal safe change. - ---- - -- [ ] **Step 3: Run the MvtDecoder unit test** - -```bash -bash scripts/commands/gbx-docker-exec.sh \ - "mvn test -pl . -Dtest=MvtDecoderTest -DfailIfNoTests=false -P skipScoverage -q" \ - 2>&1 | tail -20 -``` - -Expected: `Tests run: 2, Failures: 0, Errors: 0`. - ---- - -- [ ] **Step 4: Write failing tests for heavy-tier vector merge** - -Add new tests to `src/test/scala/com/databricks/labs/gbx/pmtiles/PMTiles_AggTest.scala`: - -```scala -import com.databricks.labs.gbx.vectorx.mvt.{MvtDecoder, MvtWriter} -import com.databricks.labs.gbx.vectorx.jts.JTS -import org.locationtech.jts.geom.{Coordinate, GeometryFactory} - -// Helper to build a real polygon WKB in tile-local coords. -private def polygonWkb(x0: Int, y0: Int): Array[Byte] = { - val gf = new GeometryFactory() - val ring = gf.createLinearRing(Array( - new Coordinate(x0, y0), new Coordinate(x0 + 100, y0), - new Coordinate(x0 + 100, y0 + 100), new Coordinate(x0, y0 + 100), - new Coordinate(x0, y0) - )) - JTS.toWKB(gf.createPolygon(ring)) -} - -private def realMvtBlob(id: Int, x0: Int, y0: Int, layer: String = "bldg"): Array[Byte] = - MvtWriter.encode(layer, 4096, Seq((polygonWkb(x0, y0), Map("id" -> id)))) - -test("pmtiles_agg merges two MVT blobs for the same (z,x,y) into one tile with 2 features") { - spark.sparkContext.setLogLevel("ERROR") - functions.register(spark) - import functions._ - - val blobA = realMvtBlob(id = 1, x0 = 100, y0 = 100) - val blobB = realMvtBlob(id = 2, x0 = 300, y0 = 300) - // Two rows for same (z=3, x=2, y=4) → must merge. - val df = spark.createDataFrame(Seq( - (3, 2, 4, blobA), - (3, 2, 4, blobB) - )).toDF("z", "x", "y", "bytes") - - val archive = df.agg(pmtiles_agg(col("bytes"), col("z"), col("x"), col("y")).as("pmt")) - .collect().head.getAs[Array[Byte]]("pmt") - - // Extract the tile from the archive. - val tileBytes = PMTilesTestHelper.readTile(archive, z = 3, x = 2, y = 4) - assert(tileBytes.nonEmpty, "merged tile must be present in archive") - val features = MvtDecoder.decode(tileBytes) - val ids = features.map(_._3.get("id").map(_.toString.toInt).getOrElse(-1)).toSet - assert(ids == Set(1, 2), s"expected both feature ids; got $ids") -} - -test("pmtiles_agg preserves POLYGON geometry type in merged MVT tile") { - spark.sparkContext.setLogLevel("ERROR") - functions.register(spark) - import functions._ - - val blobA = realMvtBlob(id = 1, x0 = 100, y0 = 100) - val blobB = realMvtBlob(id = 2, x0 = 300, y0 = 300) - val df = spark.createDataFrame(Seq((3, 2, 4, blobA), (3, 2, 4, blobB))) - .toDF("z", "x", "y", "bytes") - val archive = df.agg(pmtiles_agg(col("bytes"), col("z"), col("x"), col("y")).as("pmt")) - .collect().head.getAs[Array[Byte]]("pmt") - val tileBytes = PMTilesTestHelper.readTile(archive, z = 3, x = 2, y = 4) - val features = MvtDecoder.decode(tileBytes) - // All decoded geometries must parse as polygons (not degenerated). - features.foreach { case (_, wkb, _) => - val geom = JTS.fromWKB(wkb) - assert(geom != null && !geom.isEmpty, s"decoded geometry is null/empty") - assert(geom.getGeometryType == "Polygon" || geom.getGeometryType == "MultiPolygon", - s"expected Polygon; got ${geom.getGeometryType}") - } -} - -test("pmtiles_agg raster first-wins unchanged after vector-merge change") { - spark.sparkContext.setLogLevel("ERROR") - functions.register(spark) - import functions._ - - val pngA = Array[Byte](0x89.toByte, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 0x01, 0x00) - val pngB = Array[Byte](0x89.toByte, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 0x02, 0x00) - val df = spark.createDataFrame(Seq((1, 0, 0, pngA), (1, 0, 0, pngB))) - .toDF("z", "x", "y", "bytes") - val archive = df.agg(pmtiles_agg(col("bytes"), col("z"), col("x"), col("y")).as("pmt")) - .collect().head.getAs[Array[Byte]]("pmt") - val tileBytes = PMTilesTestHelper.readTile(archive, z = 1, x = 0, y = 0) - assert(tileBytes.sameElements(pngA), "raster first-wins violated") -} -``` - -**Note:** These tests reference `PMTilesTestHelper.readTile` — a small helper that reads a PMTiles archive (byte array → temp file → `PMTilesV3Reader.readTile` or direct binary parse). Add this helper object in the test source tree at `src/test/scala/com/databricks/labs/gbx/pmtiles/PMTilesTestHelper.scala`. It reads the PMTiles v3 index and returns the raw tile bytes for a given `(z, x, y)`, or throws if not found. Use the existing binary parse pattern from `PMTilesV3EncoderTest` as a reference. - -Run in Docker — expect compilation success but test failures (merge not yet implemented): - -```bash -bash scripts/commands/gbx-docker-exec.sh \ - "mvn test -pl . -Dtest=PMTiles_AggTest -DfailIfNoTests=false -P skipScoverage -q" \ - 2>&1 | tail -30 -``` - -Expected: the 3 new tests fail on the feature-count / geometry-type assertions; the 5 existing tests remain green. - ---- - -- [ ] **Step 5: Add vector-merge logic to `PMTiles_Agg.eval`** - -Modify `PMTiles_Agg.scala`. The `eval` method currently iterates `buffer.tiles` in insertion order and hands them all to `PMTilesV3Encoder.encode`. Replace `eval` with a version that groups by tileid, branches on `tileType`, and merges MVT payloads: - -```scala -override def eval(buffer: PMTilesAcc): Any = { - if (buffer.tiles.isEmpty) { - return PMTilesV3Encoder.encode(Iterator.empty, buffer.metadataJson) - } - val firstNonNull = buffer.tiles.iterator.map(_._4).find(b => b != null && b.nonEmpty) - val tileType = firstNonNull.map(PMTiles_Agg.detectTileType).getOrElse(PMTilesV3Encoder.TILE_TYPE_MVT) - val isVector = tileType == PMTilesV3Encoder.TILE_TYPE_MVT - - // Group payloads by tileid (Hilbert order key), preserving insertion order within each group. - import com.databricks.labs.gbx.pmtiles.{PMTilesHilbert => Hilbert} - val grouped = scala.collection.mutable.LinkedHashMap.empty[Long, scala.collection.mutable.ArrayBuffer[(Int, Int, Int, Array[Byte])]] - buffer.tiles.foreach { case row @ (z, x, y, _) => - val tid = Hilbert.zxyToTileid(z, x, y) - grouped.getOrElseUpdate(tid, scala.collection.mutable.ArrayBuffer.empty) += row - } - - // Resolve each tileid to one blob: merge for MVT, first for raster. - val resolved: Iterator[(Int, Int, Int, Array[Byte])] = grouped.iterator.map { - case (_, rows) => - val (z, x, y, _) = rows.head - val payloads = rows.map(_._4).filter(b => b != null && b.nonEmpty) - val blob = - if (isVector && payloads.length > 1) - PMTiles_Agg.mergeMvtPayloads(payloads.toSeq) - else - payloads.headOption.getOrElse(Array.emptyByteArray) - (z, x, y, blob) - } - - PMTilesV3Encoder.encode(resolved, buffer.metadataJson, tileType) -} -``` - -Add a companion helper `mergeMvtPayloads` to `PMTiles_Agg` object: - -```scala -private[pmtiles] def mergeMvtPayloads(payloads: Seq[Array[Byte]]): Array[Byte] = { - if (payloads.length == 1) return payloads.head - import com.databricks.labs.gbx.vectorx.mvt.{MvtDecoder, MvtWriter} - // Decode all blobs, union features per layer name. - val layerFeatures = scala.collection.mutable.LinkedHashMap.empty[String, scala.collection.mutable.ArrayBuffer[(Array[Byte], Map[String, Any])]] - payloads.foreach { blob => - MvtDecoder.decode(blob).foreach { case (layerName, wkb, attrs) => - layerFeatures.getOrElseUpdate(layerName, scala.collection.mutable.ArrayBuffer.empty) += ((wkb, attrs)) - } - } - if (layerFeatures.isEmpty) return payloads.head - // Re-encode: one MvtWriter.encode call per layer, concatenate into a single-layer archive. - // MVT spec allows multiple layers in one tile via concatenated protobuf messages. - // MvtWriter produces a single-layer .pbf; for multi-layer we concatenate the raw protobuf - // bytes (valid per MVT spec — each layer is a top-level repeated field). - val layerBlobs = layerFeatures.map { case (layerName, feats) => - MvtWriter.encode(layerName, MvtWriter.DefaultExtent, feats.toSeq) - } - // Concatenate layer protobufs — valid because MVT is a repeated `layers` field at tag 3. - val out = new java.io.ByteArrayOutputStream() - layerBlobs.foreach(out.write) - out.toByteArray -} -``` - -**IMPORTANT:** `PMTilesHilbert.zxyToTileid` — confirm the exact utility available in the heavy tier for computing Hilbert tile IDs. Search `PMTilesV3Encoder.scala` or `PMTilesEntry.scala` for the existing `zxyToTileid` / `tileid` conversion. Use whatever is already present rather than reimplementing. If the conversion is inlined in `PMTilesV3Encoder`, extract a package-private `def zxyToTileid(z: Int, x: Int, y: Int): Long` method and call it here. - ---- - -- [ ] **Step 6: Run all heavy PMTiles tests in Docker** - -```bash -bash scripts/commands/gbx-docker-exec.sh \ - "mvn test -pl . -Dtest=PMTiles_AggTest,MvtDecoderTest -DfailIfNoTests=false -P skipScoverage -q" \ - 2>&1 | tail -30 -``` - -Expected: all tests green, including the 3 new merge tests and all 5 pre-existing tests. - ---- - -- [ ] **Step 7: Scalastyle check** - -```bash -bash scripts/commands/gbx-lint-scalastyle.sh 2>&1 | tail -20 -``` - -Expected: no scalastyle violations. - ---- - -- [ ] **Step 8: Commit** - -```bash -chmod -R u+rwX .git/objects -git add \ - src/main/scala/com/databricks/labs/gbx/vectorx/mvt/MvtDecoder.scala \ - src/main/scala/com/databricks/labs/gbx/vectorx/mvt/MvtWriter.scala \ - src/main/scala/com/databricks/labs/gbx/pmtiles/PMTiles_Agg.scala \ - src/test/scala/com/databricks/labs/gbx/pmtiles/PMTiles_AggTest.scala \ - src/test/scala/com/databricks/labs/gbx/pmtiles/PMTilesTestHelper.scala \ - src/test/scala/com/databricks/labs/gbx/vectorx/mvt/MvtDecoderTest.scala -git commit -m "$(cat <<'EOF' -fix(pmtiles): merge multi-feature MVT blobs per (z,x,y) in heavy tier - -Same correctness fix as the light tier: gbx_pmtiles_agg was discarding -all but the first feature for vector tiles with multiple rows at the -same (z,x,y). Add MvtDecoder (OGR MVT read via /vsimem/) and group/merge -in PMTiles_Agg.eval. Raster first-wins is preserved. - -Co-authored-by: Isaac -EOF -)" -``` - ---- - -## Task 3: Light-vs-heavy parity test (POLYGON multi-feature) - -**Goal:** A cross-tier parity test packs two POLYGON single-feature MVT blobs for the same `(z,x,y)` through both light and heavy `gbx_pmtiles_agg`, reads the merged tile from each archive, and asserts both features are present with their properties. This is the mandatory POLYGON parity gate per the spec and the MVT tile-local contract. - -**Files:** -- New: `python/geobrix/test/pmtiles_light/test_parity_pmtiles_merge.py` - -**Interfaces:** -- Consumes: light `_assemble_archive` (pure Python, no Spark needed) and heavy `gbx_pmtiles_agg` Spark UDAF (via JAR). The heavy tier produces a PMTiles archive in the same format; light and heavy merged tiles must decode to the same feature set. -- The test follows the same JAR-skip pattern as `test_parity_mvt.py`: auto-skip if no JAR staged. - ---- - -- [ ] **Step 1: Write the parity test file** - -Create `python/geobrix/test/pmtiles_light/test_parity_pmtiles_merge.py`: - -```python -"""Light vs heavy gbx_pmtiles_agg vector-merge parity. - -Packs two POLYGON single-feature MVT blobs for the same (z,x,y) through both -tiers. Decodes the packed tile from each archive and asserts both features are -present with their geometry type and property values intact. - -POLYGON is mandatory — points-only gives a false pass per the MVT tile-local -contract (see CLAUDE.md). - -Heavy requires the geobrix JAR staged under python/geobrix/lib/ and GDAL/OGR -native libraries. Auto-skips when absent. Run in geobrix-dev Docker: - bash scripts/commands/gbx-test-python.sh \\ - --path python/geobrix/test/pmtiles_light/test_parity_pmtiles_merge.py \\ - --with-integration --log pmtiles-merge-parity.log -""" - -from pathlib import Path - -import mapbox_vector_tile as mvt -import pytest -from pmtiles.reader import MmapSource, Reader -from pmtiles.tile import zxy_to_tileid -from shapely.geometry import Polygon -from shapely import to_wkb - -from databricks.labs.gbx.pmtiles._agg_light import _assemble_archive - -pytestmark = pytest.mark.integration - -_HERE = Path(__file__).resolve() -_JARS = sorted((_HERE.parents[2] / "lib").glob("geobrix-*-jar-with-dependencies.jar")) - -# Two distinct tile-local polygons (in [0, 4096] pixel space), different ids. -_EXTENT = 4096 -_POLY_A = Polygon([(100, 100), (200, 100), (200, 200), (100, 200), (100, 100)]) -_POLY_B = Polygon([(300, 300), (400, 300), (400, 400), (300, 400), (300, 300)]) - - -def _make_mvt_blob(poly: Polygon, prop_id: int, layer: str = "bldg") -> bytes: - return mvt.encode( - {"name": layer, "features": [{"geometry": poly, "properties": {"id": prop_id}}]}, - default_options={"extents": _EXTENT, "y_coord_down": True}, - ) - - -_BLOB_A = _make_mvt_blob(_POLY_A, prop_id=1) -_BLOB_B = _make_mvt_blob(_POLY_B, prop_id=2) - -_Z, _X, _Y = 3, 2, 4 - - -def _read_tile_from_archive(archive: bytes, z: int, x: int, y: int) -> bytes: - import tempfile, os - with tempfile.NamedTemporaryFile(suffix=".pmtiles", delete=False) as f: - f.write(archive) - name = f.name - try: - with open(name, "rb") as f: - r = Reader(MmapSource(f)) - tile = r.get(z, x, y) - finally: - os.unlink(name) - assert tile is not None, f"tile ({z},{x},{y}) missing from archive" - return tile - - -def _decode_features(tile: bytes) -> dict: - """Return {id: geometry_type} for all features in the 'bldg' layer.""" - decoded = mvt.decode(tile) - assert "bldg" in decoded, f"layer 'bldg' missing; layers: {list(decoded.keys())}" - return { - f["properties"]["id"]: f["geometry"]["type"] - for f in decoded["bldg"]["features"] - } - - -# ── Light tier (Spark-free) ─────────────────────────────────────────────────── - -def test_light_vector_merge_parity_polygon(tmp_path): - """Light tier: two POLYGON blobs for same (z,x,y) → both features in merged tile.""" - archive = _assemble_archive([_BLOB_A, _BLOB_B], [_Z, _Z], [_X, _X], [_Y, _Y], {}) - assert archive is not None - tile = _read_tile_from_archive(archive, _Z, _X, _Y) - feats = _decode_features(tile) - assert set(feats.keys()) == {1, 2}, f"light: expected ids {{1,2}}; got {set(feats.keys())}" - assert feats[1] == "Polygon", f"light: id=1 not Polygon: {feats[1]}" - assert feats[2] == "Polygon", f"light: id=2 not Polygon: {feats[2]}" - - -# ── Heavy tier (JAR + GDAL) ────────────────────────────────────────────────── - -@pytest.fixture(scope="module") -def spark_with_jar(): - if not _JARS: - pytest.skip( - "no geobrix JAR staged under python/geobrix/lib/ — run in geobrix-dev Docker" - ) - import logging - from pyspark.sql import SparkSession - - logging.getLogger("py4j").setLevel(logging.ERROR) - active = SparkSession.getActiveSession() - if active is not None: - active_jars = active.conf.get("spark.jars", "") - if str(_JARS[-1]) not in active_jars: - pytest.skip( - "A JAR-free Spark session is already live; run in isolation: " - "gbx:test:python --path python/geobrix/test/pmtiles_light/" - "test_parity_pmtiles_merge.py --with-integration" - ) - session = ( - SparkSession.builder.master("local[2]") - .appName("gbx-pmtiles-merge-parity") - .config("spark.sql.shuffle.partitions", "2") - .config( - "spark.driver.extraJavaOptions", - "-Djava.library.path=/usr/local/lib:/usr/lib:/usr/java/packages/lib:" - "/usr/lib64:/lib64:/lib:/usr/local/hadoop/lib/native", - ) - .config("spark.jars", str(_JARS[-1])) - .getOrCreate() - ) - yield session - - -def test_heavy_vector_merge_parity_polygon(spark_with_jar): - """Heavy tier: two POLYGON blobs for same (z,x,y) → both features in merged tile.""" - from databricks.labs.gbx.pmtiles import functions as pt - from databricks.labs.gbx.pmtiles._agg_light import register_pmtiles_agg - - # Use the heavy UDAF registered from the JAR (not the light UDF). - from databricks.labs.gbx.vectorx import functions as hx - hx.register(spark_with_jar) - - df = spark_with_jar.createDataFrame( - [ - ("g", bytearray(_BLOB_A), _Z, _X, _Y), - ("g", bytearray(_BLOB_B), _Z, _X, _Y), - ], - ["grp", "tile", "z", "x", "y"], - ) - from pyspark.sql import functions as f - archive = bytes( - df.groupBy("grp") - .agg(f.expr("gbx_pmtiles_agg(tile, z, x, y)").alias("arc")) - .collect()[0]["arc"] - ) - tile = _read_tile_from_archive(archive, _Z, _X, _Y) - feats = _decode_features(tile) - assert set(feats.keys()) == {1, 2}, f"heavy: expected ids {{1,2}}; got {set(feats.keys())}" - assert feats[1] == "Polygon", f"heavy: id=1 not Polygon: {feats[1]}" - assert feats[2] == "Polygon", f"heavy: id=2 not Polygon: {feats[2]}" - - -def test_light_vs_heavy_merged_tile_equivalent(spark_with_jar): - """Light and heavy merged tiles must decode to equivalent feature sets. - - Geometry coordinate precision may differ by ±1 (integer quantization in - OGR MVT round-trip vs mapbox_vector_tile native encoding), so we compare - feature counts, geometry types, and attribute values — not raw bytes. - """ - from databricks.labs.gbx.vectorx import functions as hx - hx.register(spark_with_jar) - - # Light merged tile. - light_archive = _assemble_archive( - [_BLOB_A, _BLOB_B], [_Z, _Z], [_X, _X], [_Y, _Y], {} - ) - light_tile = _read_tile_from_archive(light_archive, _Z, _X, _Y) - light_feats = _decode_features(light_tile) - - # Heavy merged tile. - df = spark_with_jar.createDataFrame( - [ - ("g", bytearray(_BLOB_A), _Z, _X, _Y), - ("g", bytearray(_BLOB_B), _Z, _X, _Y), - ], - ["grp", "tile", "z", "x", "y"], - ) - from pyspark.sql import functions as f - heavy_archive = bytes( - df.groupBy("grp") - .agg(f.expr("gbx_pmtiles_agg(tile, z, x, y)").alias("arc")) - .collect()[0]["arc"] - ) - heavy_tile = _read_tile_from_archive(heavy_archive, _Z, _X, _Y) - heavy_feats = _decode_features(heavy_tile) - - assert light_feats.keys() == heavy_feats.keys(), ( - f"feature id mismatch: light={set(light_feats.keys())} heavy={set(heavy_feats.keys())}" - ) - for fid in light_feats: - assert light_feats[fid] == heavy_feats[fid], ( - f"geometry type mismatch for id={fid}: light={light_feats[fid]} heavy={heavy_feats[fid]}" - ) -``` - ---- - -- [ ] **Step 2: Run light-only parity tests (no JAR required)** - -```bash -python/geobrix/.venv-pyrx/bin/python -m pytest \ - python/geobrix/test/pmtiles_light/test_parity_pmtiles_merge.py \ - -k "light" -v 2>&1 | tail -20 -``` - -Expected: `test_light_vector_merge_parity_polygon` passes (Task 1 already landed). The `heavy` and `light_vs_heavy` tests skip (no JAR outside Docker). - ---- - -- [ ] **Step 3: Run full parity test suite in Docker** - -```bash -bash scripts/commands/gbx-test-python.sh \ - --path python/geobrix/test/pmtiles_light/test_parity_pmtiles_merge.py \ - --with-integration --log pmtiles-merge-parity.log -``` - -Expected: all 3 tests pass including `test_light_vs_heavy_merged_tile_equivalent`. - ---- - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX .git/objects -git add python/geobrix/test/pmtiles_light/test_parity_pmtiles_merge.py -git commit -m "$(cat <<'EOF' -test(pmtiles): light-vs-heavy vector-merge parity (POLYGON multi-feature) - -Cross-tier parity gate: two POLYGON blobs for the same tile must produce -equivalent merged tiles in both light and heavy gbx_pmtiles_agg. Uses -polygons (not points) per the MVT tile-local contract. - -Co-authored-by: Isaac -EOF -)" -``` - ---- - -## Task 4: Documentation update - -**Goal:** Update `docs/docs/api/pmtiles-functions.mdx` to document the merge-vs-first-wins behaviour. Per the spec: vector tiles are merged; raster tiles keep first-wins. User-facing voice — no internal vocabulary. - -**Files:** -- Modify: `docs/docs/api/pmtiles-functions.mdx` - ---- - -- [ ] **Step 1: Locate the `gbx_pmtiles_agg` section in the MDX file** - -Read `docs/docs/api/pmtiles-functions.mdx` and find the `gbx_pmtiles_agg` description block. Identify the exact lines describing how duplicate `(z,x,y)` tiles are handled. - ---- - -- [ ] **Step 2: Add a merge-vs-first-wins note** - -In the `gbx_pmtiles_agg` description, add a short paragraph after the current description of tile deduplication. The note must: -- Explain that **vector (MVT) tiles** with the same `(z, x, y)` are **merged** — features from each blob are combined into one multi-feature tile, preserving all attributes and layer names. -- Explain that **raster tiles** (PNG, JPEG, WebP) keep **first-write-wins** — images cannot be meaningfully merged. -- Not use any internal vocabulary (no "wave", no "subagent", no "first-wins" jargon — use "the first tile" instead). - -Example phrasing: - -> When multiple rows share the same tile coordinates `(z, x, y)`: -> - **Vector (MVT) tiles** — features from all matching rows are combined into one -> multi-feature tile. Features from different layers are kept in their respective -> layers; attributes are preserved per feature. -> - **Raster tiles (PNG, JPEG, WebP)** — the first non-null tile is used; subsequent -> tiles for the same coordinates are ignored (raster images cannot be merged). -> -> Tile type is detected automatically from the content of the first non-null payload. - ---- - -- [ ] **Step 3: Pre-commit voice check** - -```bash -grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/api/pmtiles-functions.mdx -``` - -Expected: no output (clean). - ---- - -- [ ] **Step 4: Commit** - -```bash -chmod -R u+rwX .git/objects -git add docs/docs/api/pmtiles-functions.mdx -git commit -m "$(cat <<'EOF' -docs(pmtiles): document MVT merge vs raster first-wins in gbx_pmtiles_agg - -User-facing note: vector tiles at the same (z,x,y) are merged into one -multi-feature tile; raster tiles keep the first. This makes the behaviour -of the st_asmvt_pyramid→gbx_pmtiles_agg pipeline explicit in the reference. - -Co-authored-by: Isaac -EOF -)" -``` - ---- - -## Task 5: Perf / bench verdict - -**Goal:** Assess whether the vector-merge change shifts timings materially enough to warrant a `benchmarking.mdx` update and/or a `docs/superpowers/performance/` corpus entry. - -**Files:** -- Read (no modification unless warranted): `docs/docs/api/benchmarking.mdx` -- Possibly modify: `docs/docs/api/benchmarking.mdx` (bench-doc rule: any benchmarking change reflected in the same stroke) -- Possibly create: `docs/superpowers/performance/2026-06-28-pmtiles-merge-perf.md` - ---- - -- [ ] **Step 1: Assess the merge overhead locally** - -The merge adds a `mapbox_vector_tile.decode` + re-encode round-trip per duplicate tileid. For the common pipeline (`st_asmvt_pyramid → groupBy → gbx_pmtiles_agg`), each tile typically has 5–100 features at zoom levels 0–14. The merge is a correctness fix, not a hot path change for typical single-feature-per-tile datasets (where it takes the fast `if len(blobs) == 1: return blobs[0]` path with zero decode overhead). - -Run a quick local timing check: - -```python -import timeit, mapbox_vector_tile as mvt -from shapely.geometry import Polygon - -poly = Polygon([(100, 100), (200, 100), (200, 200), (100, 200), (100, 100)]) -blob = mvt.encode( - {"name": "bldg", "features": [{"geometry": poly, "properties": {"id": 1}}]}, - default_options={"extents": 4096, "y_coord_down": True}, -) - -# Single-blob path (no decode — should be ~0 overhead) -t1 = timeit.timeit(lambda: blob if len([blob]) == 1 else None, number=10000) - -# Two-blob merge path (decode+encode — baseline cost per tile) -blobs = [blob, blob] -t2 = timeit.timeit( - lambda: mvt.encode( - {k: {"features": v["features"] + v["features"]} - for k, v in mvt.decode(blob).items()}, - default_options={"extents": 4096, "y_coord_down": True}, - ), - number=1000, -) -print(f"single-blob path: {t1*1000/10000:.3f} ms/tile") -print(f"two-blob merge path: {t2*1000/1000:.3f} ms/tile") -``` - ---- - -- [ ] **Step 2: Record verdict** - -If the merge path overhead is < 1 ms per tile (expected for small features): **this is a correctness fix, not a measurable perf regression**. No `benchmarking.mdx` change is needed. Record the verdict in a performance corpus note: - -Create `docs/superpowers/performance/2026-06-28-pmtiles-merge-perf.md`: - -```markdown -# PMTiles agg vector-merge: perf verdict (2026-06-28) - -## Change -`gbx_pmtiles_agg` now merges multi-feature MVT blobs per (z,x,y) instead of -keeping only the first. Raster first-wins is unchanged. - -## Cost model -- **Single-blob path** (one row per tileid): `if len(blobs) == 1: return blobs[0]` - — zero decode/encode overhead. The common case for sparse datasets. -- **Multi-blob merge path** (N rows per tileid): N `mapbox_vector_tile.decode` calls - + 1 `encode` call. For typical tile sizes (< 50 KB, < 100 features per tile) the - measured per-tile overhead is < 0.5 ms on the driver/executor CPU. - -## Verdict -This is a **correctness fix**. Perf overhead is negligible for the typical -`st_asmvt_pyramid → gbx_pmtiles_agg` pipeline. No change to `benchmarking.mdx` -is warranted until a measured regression is observed at real scale (the 1000-tile -cluster bench is the appropriate check if needed). - -## Follow-up -If dense-data pipelines (>1000 features per tile) show measurable overhead, the -merge path can be optimised with a direct protobuf concatenation (valid per MVT -spec, fields 3 = repeated layers) that avoids the full decode/encode round-trip. -Track as a future enhancement if cluster bench flags it. -``` - -If the measured overhead is > 2 ms per tile (unexpected — would indicate an issue in the implementation): file a follow-up task and note it in `benchmarking.mdx` per the bench-doc rule. - ---- - -- [ ] **Step 3: Commit corpus note (if created)** - -```bash -chmod -R u+rwX .git/objects -git add docs/superpowers/performance/2026-06-28-pmtiles-merge-perf.md -git commit -m "$(cat <<'EOF' -perf(pmtiles): record vector-merge overhead verdict (correctness fix) - -Merge path overhead is < 0.5 ms per tile for typical payloads; single-blob -path has zero overhead. No benchmarking.mdx change warranted. - -Co-authored-by: Isaac -EOF -)" -``` - ---- - -## Completion checklist - -Before declaring this plan done, verify: - -- [ ] `python/geobrix/test/pmtiles_light/test_agg_light_core.py` — all tests green, including the 5 new vector-merge tests. -- [ ] `python/geobrix/test/pmtiles_light/test_agg_light_udf.py` — all tests green (regression: no change needed there but confirm). -- [ ] `python/geobrix/test/pmtiles_light/test_parity_pmtiles_merge.py` — all 3 tests green in Docker with JAR. -- [ ] `PMTiles_AggTest.scala` — all 8 tests green (5 original + 3 new). -- [ ] `MvtDecoderTest.scala` — all 2 tests green. -- [ ] `gbx:lint:scalastyle` — clean. -- [ ] `docs/docs/api/pmtiles-functions.mdx` — merge-vs-first-wins note present, no internal vocab leak. -- [ ] `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/` — no output. -- [ ] Performance verdict recorded. diff --git a/docs/superpowers/plans/2026-06-28-vizx-multilayer-viewer.md b/docs/superpowers/plans/2026-06-28-vizx-multilayer-viewer.md deleted file mode 100644 index 3a74d250e..000000000 --- a/docs/superpowers/plans/2026-06-28-vizx-multilayer-viewer.md +++ /dev/null @@ -1,1124 +0,0 @@ -# VizX Multi-Layer Viewer Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Give VizX a unified, simple-for-users way to render any combination of vector, raster, and grid layers in one notebook map — a halo capability for "I have more data than a notebook cell can hold." - -**Architecture:** A small `Layer` abstraction (vector/raster/grid/pmtiles) consumed by two renderers — `plot_static` (matplotlib, multi-layer on one `Axes`) and `plot_interactive` (one self-contained **MapLibre GL** page; folium retired). A `>64 MB` ladder (URL → embed → simplify → static) keeps it honest at scale, with `simplify_tiles_from_source` / `simplify_tiles_from_archive` (tippecanoe / GeoBrix-distributed / rasterio) producing budget-bounded overviews. A Phase-1.5 follow-on adds a dynamic zoom cut-over (embed z0–10, stream z11+ on pan/zoom via an AnyWidget JS↔kernel channel — proven on Serverless by Spike B). - -**Tech Stack:** Python 3.12, `databricks.labs.gbx.vizx` (Python-only, no Scala/heavy changes), MapLibre GL JS + pmtiles.js (SRI-pinned), matplotlib + contextily (static), tippecanoe (PyPI manylinux wheel, vector simplify), rasterio (raster overviews), anywidget (Phase-1.5), pytest + Docker doc-tests. - -## Global Constraints - -- **One canonical name per concept** (beta = no aliases): `geom_col` (vector geometry), `cellid_col` (DGGS cell id; auto-detect via `_CELL_COL_CANDIDATES = ("cellid","cell","cell_id","h3","quadbin","bng","index")`), `column` (value to color/symbolize by). Do NOT rename `plot_static`'s existing `column`. -- **Interactive engine is MapLibre GL only.** folium is **retired** and removed from the `[vizx]` extra. `plot_interactive` no longer uses folium. -- **Python-only.** No Scala/heavy-tier or new-Spark-function changes. VizX runs driver-side. -- **Indefinite single-archive in a notebook is OUT (Phase-2/App).** Spike A: a MANAGED volume yields no presigned URL; Files-API `Range` is CORS-blocked. The `>64 MB` ladder's URL rung applies only to a user-supplied CORS-reachable `http(s)` URL (external-volume presign is parked). -- **No silent degradation.** Every reduction/simplification/fallback emits a loud, actionable warning. -- **Supply chain:** every execution-env dependency exact-version + hash-pinned (`--require-hashes`) in the `[vizx]` extra and the CI lock; the injected MapLibre/pmtiles.js use **Subresource Integrity** (`integrity="sha384-…" crossorigin="anonymous"`), not bare CDN tags. -- **Docs are executable doc-tests** (single-source rule): code lives in `docs/tests/python/`, runs in Docker, is imported by `.mdx`. -- **simplify engine policy:** tippecanoe (driver, moderate vector) / GeoBrix distributed tiling (large vector) / rasterio overviews (raster). tippecanoe is VizX viz plumbing — NOT a product tiler; GeoBrix's own tiling remains the product story. -- **Default basemap:** CARTO Positron (hosted), configurable, `none` option; must work under normal Serverless conditions. contextily retained for the static path. - ---- - -## File Structure - -**New:** -- `python/geobrix/src/databricks/labs/gbx/vizx/_layers.py` — `Layer` dataclass + `vector_layer`/`raster_layer`/`grid_layer`/`pmtiles_layer` constructors + `as_layers()` coercion (bare input → one Layer). -- `python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py` — per-layer → MapLibre sources/layers adapters; self-contained HTML builder (SRI-pinned JS, CARTO basemap); the `>64 MB` ladder. -- `python/geobrix/src/databricks/labs/gbx/vizx/_simplify.py` — `simplify_tiles_spec` schema/validation; `simplify_tiles_from_source`; `simplify_tiles_from_archive`; engine policy. -- `python/geobrix/src/databricks/labs/gbx/vizx/_dynamic.py` — Phase-1.5 AnyWidget cut-over viewer. -- Tests: `python/geobrix/test/vizx/test_layers.py`, `test_maplibre.py`, `test_ladder.py`, `test_simplify.py`, `test_dynamic.py`. -- Docs: `docs/docs/api/vizx-layers.mdx`; doc-test code `docs/tests/python/api/vizx_layers.py`; diagram `resources/images/diagrams/vizx/vizx-layers.{svg,png}` + generator `resources/images/generators/vizx-layers.py`. - -**Modified:** -- `vizx/__init__.py` — export new symbols; route `plot_static`/`plot_interactive` to accept layers. -- `vizx/_static_map.py` — accept a `Layer` list; reproject + draw in order. -- `vizx/_cog.py` — add `ax=` parameter. -- `vizx/_interactive.py` — re-implemented on MapLibre (folium removed) or reduced to a thin delegator to `_maplibre`. -- `vizx/_pmtiles.py` — `plot_pmtiles` delegates to `_maplibre` (single `pmtiles_layer`). -- `python/geobrix/pyproject.toml` (or setup) `[vizx]` extra + CI lock `requirements-*.in/.txt`. -- `notebooks/examples/helios/02. Visual Basemap (XYZ).ipynb`, `03. Analytical Core (COG + STAC).ipynb` — real overlays. -- `notebooks/examples/helios/README.md`, `docs/docs/notebooks/helios.mdx` — prose fix. -- `docs/docs/api/vizx.mdx` — multi-layer + ladder narrative (or link new page). -- `notebooks/examples/{eo-series,h3-rasterize,xview}/*` — audited/migrated to the new surface. - -**Prerequisite (ops, not a code task):** refresh the staged light wheel at `/Volumes/geospatial_docs/geobrix/sample-data/geobrix-0.4.0-py3-none-any.whl` (current one predates `pmtiles_info` in `gbx.pmtiles`) before doc-test/notebook runs on that workspace. Track via `gbx:data:push-wheel`. - ---- - -## Task 1: `Layer` model + constructors - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/vizx/_layers.py` -- Test: `python/geobrix/test/vizx/test_layers.py` - -**Interfaces:** -- Produces: `@dataclass Layer(kind: str, data, *, geom_col=None, cellid_col=None, column=None, grid_system=None, grid_conf=None, cmap="viridis", opacity=None, color=None, width=None, fill=True, label=None, style=None, simplify=None, band=None)`; `kind ∈ {"vector","raster","grid","pmtiles"}`. Constructors `vector_layer`, `raster_layer`, `grid_layer`, `pmtiles_layer` return a `Layer`. `as_layers(obj) -> list[Layer]` coerces a `Layer`, a list of `Layer`, or a bare input (DataFrame/path/bytes/array) into `list[Layer]` (bare → one vector or raster layer inferred by type; a `str`/`bytes` ending `.pmtiles` or PMTiles magic → `pmtiles`). - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/vizx/test_layers.py -import pytest -from databricks.labs.gbx.vizx._layers import ( - Layer, vector_layer, raster_layer, grid_layer, pmtiles_layer, as_layers, -) - -def test_constructors_set_kind_and_params(): - v = vector_layer("df", geom_col="geom", column="pop", label="cities") - assert v.kind == "vector" and v.geom_col == "geom" and v.column == "pop" and v.label == "cities" - g = grid_layer("df", grid_system="h3", cellid_col="h3", column="score") - assert g.kind == "grid" and g.grid_system == "h3" and g.cellid_col == "h3" and g.column == "score" - r = raster_layer("/x.tif", band=1, cmap="terrain") - assert r.kind == "raster" and r.band == 1 and r.cmap == "terrain" - p = pmtiles_layer("/x.pmtiles") - assert p.kind == "pmtiles" - -def test_grid_layer_requires_grid_system(): - with pytest.raises(TypeError): - grid_layer("df") # grid_system is keyword-required - -def test_as_layers_coerces_single_and_list(): - v = vector_layer("df") - assert as_layers(v) == [v] - assert as_layers([v, v]) == [v, v] - -def test_as_layers_bare_pmtiles_path(): - [lyr] = as_layers("/data/x.pmtiles") - assert lyr.kind == "pmtiles" -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_layers.py` -Expected: FAIL (ImportError: cannot import name 'Layer'). - -- [ ] **Step 3: Write minimal implementation** - -```python -# python/geobrix/src/databricks/labs/gbx/vizx/_layers.py -"""Layer model for the unified VizX viewers (vector / raster / grid / pmtiles).""" -from dataclasses import dataclass, field -from typing import Any, Optional - -_VALID = {"vector", "raster", "grid", "pmtiles"} - - -@dataclass -class Layer: - kind: str - data: Any - geom_col: Optional[str] = None - cellid_col: Optional[str] = None - column: Optional[str] = None - grid_system: Optional[str] = None - grid_conf: Optional[dict] = None - cmap: str = "viridis" - opacity: Optional[float] = None - color: Optional[str] = None - width: Optional[float] = None - fill: bool = True - band: Optional[int] = None - style: Optional[dict] = None - simplify: Optional[dict] = None - label: Optional[str] = None - - def __post_init__(self): - if self.kind not in _VALID: - raise ValueError(f"Layer.kind must be one of {_VALID}, got {self.kind!r}") - - -def vector_layer(data, *, geom_col=None, column=None, cmap="viridis", fill=True, - color=None, width=None, opacity=0.8, simplify=None, label=None): - return Layer("vector", data, geom_col=geom_col, column=column, cmap=cmap, fill=fill, - color=color, width=width, opacity=opacity, simplify=simplify, label=label) - - -def raster_layer(data, *, band=None, cmap="viridis", opacity=1.0, label=None): - return Layer("raster", data, band=band, cmap=cmap, opacity=opacity, label=label) - - -def grid_layer(data, *, grid_system, cellid_col=None, column=None, cmap="viridis", - opacity=0.7, grid_conf=None, label=None): - return Layer("grid", data, grid_system=grid_system, cellid_col=cellid_col, column=column, - cmap=cmap, opacity=opacity, grid_conf=grid_conf, label=label) - - -def pmtiles_layer(data, *, style=None, simplify=None, label=None): - return Layer("pmtiles", data, style=style, simplify=simplify, label=label) - - -def _looks_pmtiles(obj) -> bool: - if isinstance(obj, (bytes, bytearray)): - return obj[:7] == b"PMTiles" - if isinstance(obj, str): - return obj.endswith(".pmtiles") - return False - - -def as_layers(obj) -> list: - """Coerce a Layer / list[Layer] / bare input into list[Layer].""" - if isinstance(obj, Layer): - return [obj] - if isinstance(obj, (list, tuple)) and obj and all(isinstance(x, Layer) for x in obj): - return list(obj) - if _looks_pmtiles(obj): - return [pmtiles_layer(obj)] - # bare raster: a path to a known raster ext, ndarray, or tile struct -> raster; else vector. - if isinstance(obj, str) and obj.lower().endswith((".tif", ".tiff", ".cog")): - return [raster_layer(obj)] - try: - import numpy as np - if isinstance(obj, np.ndarray): - return [raster_layer(obj)] - except ImportError: - pass - return [vector_layer(obj)] -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_layers.py` -Expected: PASS (4 tests). - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_layers.py python/geobrix/test/vizx/test_layers.py -git commit -m "feat(vizx): Layer model + vector/raster/grid/pmtiles constructors" -``` - ---- - -## Task 2: `plot_cog` gains `ax=` (static composition) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_cog.py` (`plot_cog`, `_render_cog`) -- Test: `python/geobrix/test/vizx/test_cog_ax.py` - -**Interfaces:** -- Consumes: existing `plot_cog(path, *, band=None, max_pixels=2000, fig_w=10, fig_h=10, basemap=True, basemap_source=None, title=None, **kw)`. -- Produces: `plot_cog(..., ax=None)` — when `ax` is provided, draws onto it and returns it (no new figure, no `plt.show`); when `None`, behaves as today. - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/vizx/test_cog_ax.py -import matplotlib -matplotlib.use("Agg") -import matplotlib.pyplot as plt -from databricks.labs.gbx.vizx._cog import plot_cog - -def test_plot_cog_draws_on_provided_axes(tmp_path): - # build a tiny 1-band GeoTIFF - import numpy as np, rasterio - from rasterio.transform import from_origin - p = tmp_path / "x.tif" - data = (np.arange(64, dtype="float32").reshape(8, 8)) - with rasterio.open(p, "w", driver="GTiff", height=8, width=8, count=1, dtype="float32", - crs="EPSG:3857", transform=from_origin(0, 8, 1, 1)) as ds: - ds.write(data, 1) - fig, ax = plt.subplots() - n_before = len(ax.images) - out = plot_cog(str(p), basemap=False, ax=ax) - assert out is ax - assert len(ax.images) > n_before # drew onto the SAME axes -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_cog_ax.py` -Expected: FAIL (`plot_cog() got an unexpected keyword argument 'ax'`). - -- [ ] **Step 3: Write minimal implementation** - -In `_cog.py`, thread `ax` through `plot_cog` and `_render_cog`: - -```python -def plot_cog(path, *, band=None, max_pixels=2000, fig_w=10, fig_h=10, - basemap=True, basemap_source=None, title=None, ax=None, **kw): - data, transform, crs = _decode_cog(path, band=band, max_pixels=max_pixels) # existing decode - return _render_cog(data, transform, crs=crs, fig_w=fig_w, fig_h=fig_h, title=title, - basemap=basemap, basemap_source=basemap_source, ax=ax) - -def _render_cog(data, transform, *, crs, fig_w, fig_h, title, basemap, basemap_source, ax=None): - import matplotlib.pyplot as plt - from rasterio.plot import plotting_extent, show - owns_fig = ax is None - if owns_fig: - _, ax = plt.subplots(1, figsize=(fig_w, fig_h)) - if data.shape[0] == 1: - ax.imshow(data[0], extent=plotting_extent(data[0], transform), cmap="viridis") - else: - show(data, ax=ax, transform=transform) - if basemap and crs is not None: - try: - import contextily as cx - cx.add_basemap(ax, source=basemap_source, crs=crs) - except Exception: - pass - if title: - ax.set_title(title) - return ax -``` - -Keep the existing decode helper name; only the rendering path adds `ax`. - -- [ ] **Step 4: Run test to verify it passes** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_cog_ax.py` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_cog.py python/geobrix/test/vizx/test_cog_ax.py -git commit -m "feat(vizx): plot_cog accepts ax= for static multi-layer overlay" -``` - ---- - -## Task 3: `plot_static(layers)` — matplotlib multi-layer compositor - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py` (`plot_static`) -- Test: `python/geobrix/test/vizx/test_static_layers.py` - -**Interfaces:** -- Consumes: `as_layers` (Task 1); `Layer`; existing single-layer `plot_static` body (vector/grid via geopandas, reproject to 3857); `plot_cog(..., ax=)` (Task 2); existing `plot_raster`. -- Produces: `plot_static(layers, *, basemap=True, basemap_source=None, title=None, fig_w=10, fig_h=10, ax=None, **single_layer_kwargs)` — accepts a `Layer`/list/bare input; draws each layer in order on one `Axes` (reprojected to EPSG:3857); returns the `Axes`. The legacy keyword call `plot_static(df, column=..., geom_col=..., grid_system=...)` still works (coerced to one layer; the per-layer kwargs override the layer fields). - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/vizx/test_static_layers.py -import matplotlib; matplotlib.use("Agg") -import geopandas as gpd -from shapely.geometry import Point, Polygon -from databricks.labs.gbx.vizx._static_map import plot_static -from databricks.labs.gbx.vizx._layers import vector_layer - -def _gdf(geoms): - return gpd.GeoDataFrame({"v": range(len(geoms))}, geometry=geoms, crs="EPSG:4326") - -def test_two_vector_layers_one_axes(): - pts = _gdf([Point(-122.4, 37.7), Point(-122.41, 37.72)]) - polys = _gdf([Polygon([(-122.5, 37.7), (-122.4, 37.7), (-122.4, 37.8), (-122.5, 37.8)])]) - ax = plot_static([vector_layer(polys, column="v"), vector_layer(pts, color="red")], - basemap=False) - # both layers drew: at least one collection from polys + one from pts - assert len(ax.collections) >= 2 - -def test_legacy_single_dataframe_call_still_works(): - pts = _gdf([Point(-122.4, 37.7)]) - ax = plot_static(pts, column="v", basemap=False) - assert ax is not None -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_static_layers.py` -Expected: FAIL (`test_two_vector_layers_one_axes` — a list of Layers isn't handled). - -- [ ] **Step 3: Write minimal implementation** - -Refactor `plot_static` so its current body becomes `_draw_one_layer(layer, ax, ...)` and the public function loops: - -```python -def plot_static(layers, *, basemap=True, basemap_source=None, title=None, - fig_w=10, fig_h=10, ax=None, **legacy): - import matplotlib.pyplot as plt - from databricks.labs.gbx.vizx._layers import as_layers, Layer - lyrs = as_layers(layers) - # legacy keyword overrides apply to a single coerced layer - if legacy and len(lyrs) == 1: - for k, v in legacy.items(): - if hasattr(lyrs[0], k): - setattr(lyrs[0], k, v) - owns = ax is None - if owns: - _, ax = plt.subplots(figsize=(fig_w, fig_h)) - for lyr in lyrs: - _draw_one_layer(lyr, ax) # reprojects to 3857, draws (existing logic per kind) - if basemap: - try: - import contextily as cx - cx.add_basemap(ax, source=basemap_source, crs="EPSG:3857") - except Exception: - pass - if title: - ax.set_title(title) - return ax -``` - -`_draw_one_layer` dispatches by `lyr.kind`: `vector`/`grid` reuse the existing geopandas reproject+plot path (grid via the existing `_resolve_cells` + `_GRID_DISPATCH`, using `lyr.cellid_col`/`grid_system`/`column`); `raster` calls `plot_cog(lyr.data, band=lyr.band, basemap=False, ax=ax)` (Task 2) or `plot_raster(..., ax=ax)`. Preserve the existing 3857 reprojection so layers align. - -- [ ] **Step 4: Run test to verify it passes** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_static_layers.py` -Expected: PASS (2 tests). Also re-run the existing `test_static_map.py` to confirm no regression: -`bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_static_map.py` → PASS. - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py python/geobrix/test/vizx/test_static_layers.py -git commit -m "feat(vizx): plot_static accepts a Layer list (matplotlib compositor)" -``` - ---- - -## Task 4: MapLibre per-layer adapters - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py` -- Test: `python/geobrix/test/vizx/test_maplibre.py` - -**Interfaces:** -- Consumes: `Layer` (Task 1); `as_gdf`/`grid_as_gdf`/`cells_as_gdf` from `vizx._vector`; `pmtiles_info` from `databricks.labs.gbx.pmtiles`. -- Produces: `layer_to_sources_layers(layer, idx) -> (sources: dict, layers: list[dict], embed_bytes: int)` — converts one `Layer` to MapLibre `sources` entries + `layers` entries and reports embed cost. Vector/grid → an inline `geojson` source + fill/line/circle layers keyed `f"gbx{idx}"`; raster → an `image` source with 4-corner `coordinates`; pmtiles → a source `{"type": "", "url": "pmtiles://gbx{idx}"}` plus a sidecar dict recording the archive bytes/URL for the HTML builder. - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/vizx/test_maplibre.py -import geopandas as gpd -from shapely.geometry import Polygon -from databricks.labs.gbx.vizx._maplibre import layer_to_sources_layers -from databricks.labs.gbx.vizx._layers import vector_layer - -def test_vector_layer_becomes_geojson_source_and_fill_layer(): - gdf = gpd.GeoDataFrame( - {"v": [1]}, - geometry=[Polygon([(-122.5, 37.7), (-122.4, 37.7), (-122.4, 37.8), (-122.5, 37.8)])], - crs="EPSG:4326", - ) - sources, layers, embed = layer_to_sources_layers(vector_layer(gdf, column="v"), 0) - assert "gbx0" in sources and sources["gbx0"]["type"] == "geojson" - assert any(l["type"] in ("fill", "line", "circle") for l in layers) - assert embed > 0 -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_maplibre.py` -Expected: FAIL (ImportError). - -- [ ] **Step 3: Write minimal implementation** - -```python -# python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py -"""MapLibre GL compositor for plot_interactive — per-layer adapters + HTML builder.""" -import json - - -def _gdf_for(layer): - from databricks.labs.gbx.vizx import _vector - if layer.kind == "grid": - return _vector.cells_as_gdf(layer.data, cell_col=layer.cellid_col or "cellid") - return _vector.as_gdf(layer.data) if not hasattr(layer.data, "geometry") else layer.data - - -def layer_to_sources_layers(layer, idx): - sid = f"gbx{idx}" - if layer.kind in ("vector", "grid"): - gdf = _gdf_for(layer).to_crs(4326) - gj = json.loads(gdf.to_json()) - src = {sid: {"type": "geojson", "data": gj}} - geomtypes = {f["geometry"]["type"] for f in gj["features"]} - layers = [] - if geomtypes & {"Polygon", "MultiPolygon"}: - layers.append({"id": f"{sid}-fill", "type": "fill", "source": sid, - "paint": {"fill-color": layer.color or "#3388ff", - "fill-opacity": layer.opacity or 0.5}}) - if geomtypes & {"LineString", "MultiLineString", "Polygon", "MultiPolygon"}: - layers.append({"id": f"{sid}-line", "type": "line", "source": sid, - "paint": {"line-color": layer.color or "#1f6fb5", - "line-width": layer.width or 1.0}}) - if geomtypes & {"Point", "MultiPoint"}: - layers.append({"id": f"{sid}-circle", "type": "circle", "source": sid, - "paint": {"circle-color": layer.color or "#e04e2a", - "circle-radius": 4}}) - return src, layers, len(json.dumps(gj).encode()) - if layer.kind == "raster": - png_b64, corners = _raster_to_image(layer) # helper below - src = {sid: {"type": "image", "url": f"data:image/png;base64,{png_b64}", - "coordinates": corners}} - return src, [{"id": f"{sid}-raster", "type": "raster", "source": sid, - "paint": {"raster-opacity": layer.opacity or 1.0}}], len(png_b64) - if layer.kind == "pmtiles": - # the HTML builder embeds/streams; record bytes/url via a sidecar attribute - from databricks.labs.gbx.pmtiles import pmtiles_info - info = _resolve_pmtiles_bytes_or_url(layer) # {"mode","bytes"|"url","tile_type"} - is_raster = "raster" in str(info["tile_type"]).lower() or "png" in str(info["tile_type"]).lower() - src = {sid: {"type": "raster" if is_raster else "vector", "url": f"pmtiles://{sid}"}} - layers = ([{"id": f"{sid}-raster", "type": "raster", "source": sid}] if is_raster - else [{"id": f"{sid}-fill", "type": "fill", "source": sid, - "source-layer": "buildings", - "paint": {"fill-color": layer.color or "#c33", "fill-opacity": 0.5}}]) - src[sid]["_gbx_pmtiles"] = info # sidecar consumed by build_html - return src, layers, (len(info["bytes"]) if info["mode"] == "embed" else 0) - raise ValueError(layer.kind) -``` - -Add `_raster_to_image(layer)` (decimate via rasterio to ≤ `raster_max_px`, render to RGBA PNG, base64, return `(b64, [[ulx,uly],[urx,ury],[lrx,lry],[llx,lly]])` in lon/lat) and `_resolve_pmtiles_bytes_or_url(layer)` (path/bytes → read bytes + detect tile type via `pmtiles_info`; an `http(s)` URL → `{"mode":"url","url":...}`). - -- [ ] **Step 4: Run test to verify it passes** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/vizx/test_maplibre.py` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py python/geobrix/test/vizx/test_maplibre.py -git commit -m "feat(vizx): MapLibre per-layer adapters (geojson/image/pmtiles)" -``` - ---- - -## Task 5: MapLibre self-contained HTML builder (multi-source, SRI-pinned, CARTO basemap) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py` (`build_html`) -- Test: `python/geobrix/test/vizx/test_maplibre.py` (add cases) - -**Interfaces:** -- Consumes: `layer_to_sources_layers` (Task 4). -- Produces: `build_html(prepared: list[tuple[sources, layers, info]], *, basemap="carto-positron", center=None, zoom=None) -> str` — one HTML page that registers the `pmtiles://` protocol once, embeds each pmtiles archive as a base64 `FileSource` (or a `FetchSource(url)` for url-mode), merges all sources/layers into one MapLibre style on top of the basemap, and loads MapLibre/pmtiles.js via **SRI-pinned** ` - -""" - -def _pmtiles_register_js(sid, info): - if info["mode"] == "url": - return f"proto.add(new pmtiles.PMTiles({json.dumps(info['url'])}));\n" - import base64 - b64 = base64.b64encode(info["bytes"]).decode() - return (f"const _b{sid}=Uint8Array.from(atob({json.dumps(b64)}),c=>c.charCodeAt(0));\n" - f"proto.add(new pmtiles.PMTiles(new pmtiles.FileSource(" - f"new File([_b{sid}.buffer],'{sid}.pmtiles'))));\n") -``` - -(The two SRI placeholders are computed and pinned in Task 12 against the locked unpkg versions; until then the test asserts the *attribute shape* `sha384-`, which the literal satisfies.) - -- [ ] **Step 4: Run test to verify it passes** → PASS. -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py python/geobrix/test/vizx/test_maplibre.py -git commit -m "feat(vizx): self-contained multi-source MapLibre HTML builder" -``` - ---- - -## Task 6: The `>64 MB` ladder - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py` (`prepare_layers`) -- Test: `python/geobrix/test/vizx/test_ladder.py` - -**Interfaces:** -- Consumes: `layer_to_sources_layers` (Task 4); `simplify_tiles_from_*`/spec (Tasks 8-10 — import lazily; Task 6 handles only rungs 1,2,4 and calls the simplify hook if present). -- Produces: `prepare_layers(layers, *, max_embed_mb=64, simplify_tiles_spec=None, fallback=True) -> dict` returning `{"mode": "interactive"|"static", "prepared": [...], "warnings": [...]}`. Order per layer: (1) pmtiles with explicit `http(s)` URL → url-stream (0 embed); (2) prepared bytes ≤ budget → embed; (3) `simplify_tiles_spec`/`layer.simplify` present → simplify to ≤ budget, embed; (4) else → `mode="static"`. **Budget authority:** the per-layer `embed_bytes` from Task 4 is a *heuristic* used only to choose which layer to shed/simplify (note: Task-4 raster reports base64 length, vector reports raw JSON, pmtiles reports raw archive bytes — not directly comparable). The actual budget gate is the **size of the assembled HTML** (`len(build_html(prepared).encode())`) vs `max_embed_mb` — measure the real payload, don't sum mixed per-layer figures. A finished pmtiles archive is never shrunk (only url or static get it past budget). Every reduction/fallback appends a loud warning. **When the static fallback runs, a pmtiles layer is decoded to an image/geometry via the existing `_static_raster_fallback` / `_static_vector_fallback` helpers in `_pmtiles.py` and handed to `plot_static` as a `raster_layer`/`vector_layer` — pmtiles layers are never silently dropped** (`plot_static` itself only handles vector/grid/raster and must warn, not skip, on a direct pmtiles layer). - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/vizx/test_ladder.py -import geopandas as gpd -from shapely.geometry import Point -from databricks.labs.gbx.vizx._maplibre import prepare_layers -from databricks.labs.gbx.vizx._layers import vector_layer, pmtiles_layer - -def _small_gdf(): - return gpd.GeoDataFrame({"v": [1]}, geometry=[Point(-122.4, 37.7)], crs="EPSG:4326") - -def test_under_budget_is_interactive(): - out = prepare_layers([vector_layer(_small_gdf())], max_embed_mb=64) - assert out["mode"] == "interactive" - -def test_oversize_pmtiles_without_url_or_spec_falls_back_to_static(): - big = pmtiles_layer(b"PMTiles" + b"\x03" + b"\x00" * (5 * 1024 * 1024)) - out = prepare_layers([big], max_embed_mb=1, fallback=True) - assert out["mode"] == "static" - assert any("static" in w.lower() for w in out["warnings"]) -``` - -- [ ] **Step 2: Run test to verify it fails** → FAIL (no `prepare_layers`). -- [ ] **Step 3: Write minimal implementation** — implement the four-rung loop; for rung 3 call `_simplify_layer(layer, spec)` (defined in Task 9, imported lazily inside the function with a clear error if `[vizx]` simplify deps are missing); accumulate `embed_total`; if it exceeds budget and no mitigation succeeds and `fallback`, return `mode="static"` with a warning naming the offending layer + the three remedies (stage a URL / pre-tile / reduce AOI); if `fallback=False`, raise `ValueError`. -- [ ] **Step 4: Run test to verify it passes** → PASS. -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py python/geobrix/test/vizx/test_ladder.py -git commit -m "feat(vizx): >64MB ladder (url->embed->simplify->static) with warnings" -``` - ---- - -## Task 7: `plot_interactive(layers)` on MapLibre — folium retired - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_interactive.py` (rewrite `plot_interactive`), `python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py` (`plot_pmtiles` delegates) -- Test: `python/geobrix/test/vizx/test_interactive_maplibre.py` - -**Interfaces:** -- Consumes: `as_layers` (Task 1), `prepare_layers` + `build_html` (Tasks 5,6). -- Produces: `plot_interactive(layers, *, basemap="carto-positron", simplify_tiles_spec=None, max_embed_mb=64, fallback=True, center=None, zoom=None) -> str|None` — coerces input, runs the ladder; `mode="interactive"` → `build_html` + `displayHTML` (return the HTML string when not in a notebook); `mode="static"` → delegates to `plot_static(layers)`. `plot_pmtiles(path_or_bytes, **kw)` becomes `plot_interactive([pmtiles_layer(path_or_bytes)], **kw)`. **No `folium` import anywhere in vizx.** - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/vizx/test_interactive_maplibre.py -import geopandas as gpd -from shapely.geometry import Point -from databricks.labs.gbx.vizx._interactive import plot_interactive -from databricks.labs.gbx.vizx._layers import vector_layer - -def test_interactive_returns_maplibre_html_for_layers(): - gdf = gpd.GeoDataFrame({"v": [1]}, geometry=[Point(-122.4, 37.7)], crs="EPSG:4326") - html = plot_interactive([vector_layer(gdf)]) - assert "maplibregl.Map" in html - -def test_no_folium_import_in_vizx(): - import importlib, pkgutil, databricks.labs.gbx.vizx as v - for m in pkgutil.iter_modules(v.__path__): - src = importlib.import_module(f"databricks.labs.gbx.vizx.{m.name}") - assert "folium" not in (getattr(src, "__file__", "") or "") # sanity; real check below - # grep-style: no module imports folium - import subprocess, os - root = os.path.dirname(v.__file__) - out = subprocess.run(["grep", "-rl", "import folium", root], capture_output=True, text=True) - assert out.stdout.strip() == "", f"folium still imported in: {out.stdout}" -``` - -- [ ] **Step 2: Run test to verify it fails** → FAIL (current `plot_interactive` is folium; signature mismatch). -- [ ] **Step 3: Write minimal implementation** — rewrite `plot_interactive` per the interface; delete folium code paths; update `plot_pmtiles` to delegate. When `displayHTML` is available (notebook) call it and return `None`; else return the HTML (so tests can assert). -- [ ] **Step 4: Run test to verify it passes** → PASS. Re-run existing `test_pmtiles*` and confirm `plot_pmtiles` single-archive still renders. -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_interactive.py python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py python/geobrix/test/vizx/test_interactive_maplibre.py -git commit -m "feat(vizx): plot_interactive on MapLibre (layers); retire folium" -``` - ---- - -## Task 8: `simplify_tiles_spec` schema + validation - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/vizx/_simplify.py` (schema part) -- Test: `python/geobrix/test/vizx/test_simplify.py` - -**Interfaces:** -- Produces: `normalize_spec(spec: dict|None) -> dict` applying defaults `{"budget_mb":64,"min_z":0,"max_z":10,"tolerance":"auto","drop_densest":True,"cluster_distance":None,"keep_attrs":None,"raster_max_px":1024,"effort":"fast"}` and validating types/ranges (`min_z<=max_z`, `budget_mb>0`, `effort∈{"fast","full"}`); raises `ValueError` on bad input. - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/vizx/test_simplify.py -import pytest -from databricks.labs.gbx.vizx._simplify import normalize_spec - -def test_defaults_applied(): - s = normalize_spec(None) - assert s["budget_mb"] == 64 and s["min_z"] == 0 and s["max_z"] == 10 and s["effort"] == "fast" - -def test_override_and_validation(): - assert normalize_spec({"max_z": 12})["max_z"] == 12 - with pytest.raises(ValueError): - normalize_spec({"min_z": 8, "max_z": 4}) - with pytest.raises(ValueError): - normalize_spec({"effort": "turbo"}) -``` - -- [ ] **Step 2: Run test to verify it fails** → FAIL (ImportError). -- [ ] **Step 3: Write minimal implementation** — `normalize_spec` merges defaults, validates, returns the dict. -- [ ] **Step 4: Run test to verify it passes** → PASS. -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_simplify.py python/geobrix/test/vizx/test_simplify.py -git commit -m "feat(vizx): simplify_tiles_spec schema + validation" -``` - ---- - -## Task 9: `simplify_tiles_from_source` - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_simplify.py` -- Test: `python/geobrix/test/vizx/test_simplify.py` (add) - -**Interfaces:** -- Consumes: `normalize_spec` (Task 8); tippecanoe binary (PyPI wheel); rasterio (raster); GeoBrix pyvx tiling (large-vector branch — import lazily). -- Produces: `simplify_tiles_from_source(source, *, spec=None, out_path=None) -> bytes|str` — vector source (GeoDataFrame/GeoJSON/path): write GeoJSON to a temp file, run `tippecanoe -z{max_z} -Z{min_z} --maximum-tile-bytes {budget} [--drop-densest-as-needed] [--cluster-distance N] -o out.pmtiles in.geojson`, return bytes (or write to `out_path`). Raster source: rasterio overview downsample to `raster_max_px` → raster PMTiles. The very-large-vector distributed branch is gated behind an explicit `engine="distributed"` in `spec` (default driver/tippecanoe); log which engine ran. - -- [ ] **Step 1: Write the failing test** (skips cleanly if tippecanoe absent so the suite is portable): - -```python -import shutil, pytest -from databricks.labs.gbx.vizx._simplify import simplify_tiles_from_source - -@pytest.mark.skipif(shutil.which("tippecanoe") is None, reason="tippecanoe not installed") -def test_simplify_from_geojson_under_budget(tmp_path): - import geopandas as gpd - from shapely.geometry import Polygon - gdf = gpd.GeoDataFrame( - {"v": [1, 2]}, - geometry=[Polygon([(0, 0), (1, 0), (1, 1), (0, 1)]), - Polygon([(2, 2), (3, 2), (3, 3), (2, 3)])], - crs="EPSG:4326", - ) - out = tmp_path / "o.pmtiles" - p = simplify_tiles_from_source(gdf, spec={"max_z": 6, "budget_mb": 8}, out_path=str(out)) - assert out.exists() and out.read_bytes()[:7] == b"PMTiles" -``` - -- [ ] **Step 2: Run test to verify it fails** → FAIL (ImportError) or skip if tippecanoe absent (then implement + verify in Docker where the wheel installs). -- [ ] **Step 3: Write minimal implementation** — GeoDataFrame → `to_file(tmp.geojson, driver="GeoJSON")`; build the tippecanoe argv from the normalized spec; `subprocess.run(check=True)`; read bytes; raster branch via rasterio. Raise a clear error if tippecanoe is missing and `engine != "distributed"`. -- [ ] **Step 4: Run test to verify it passes** — run in Docker (`gbx:test:python`) where `[vizx]` (incl. tippecanoe) is installed: PASS. -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_simplify.py python/geobrix/test/vizx/test_simplify.py -git commit -m "feat(vizx): simplify_tiles_from_source (tippecanoe / rasterio, budget-bounded)" -``` - ---- - -## Task 10: `simplify_tiles_from_archive` - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_simplify.py` -- Test: `python/geobrix/test/vizx/test_simplify.py` (add) - -**Interfaces:** -- Consumes: `normalize_spec` (Task 8); `tile-join` (ships with tippecanoe). -- Produces: `simplify_tiles_from_archive(pmtiles_path, *, spec=None, out_path=None) -> bytes|str` — `tile-join --maximum-zoom={max_z} --maximum-tile-bytes={budget} -o out.pmtiles in.pmtiles` (down-zoom + budget-trim an existing archive without re-tiling from source). Returns bytes or writes `out_path`. - -- [ ] **Step 1: Write the failing test** (skip if `tile-join` absent; in Docker it's present): - -```python -import shutil, pytest -from databricks.labs.gbx.vizx._simplify import simplify_tiles_from_source, simplify_tiles_from_archive - -@pytest.mark.skipif(shutil.which("tile-join") is None, reason="tile-join not installed") -def test_archive_downzoom_trims(tmp_path): - import geopandas as gpd - from shapely.geometry import Polygon - gdf = gpd.GeoDataFrame({"v": [1]}, geometry=[Polygon([(0,0),(1,0),(1,1),(0,1)])], crs="EPSG:4326") - src = tmp_path / "full.pmtiles" - simplify_tiles_from_source(gdf, spec={"max_z": 8}, out_path=str(src)) - out = tmp_path / "ov.pmtiles" - simplify_tiles_from_archive(str(src), spec={"max_z": 4, "budget_mb": 4}, out_path=str(out)) - assert out.exists() and out.read_bytes()[:7] == b"PMTiles" -``` - -- [ ] **Step 2: Run test to verify it fails** → FAIL/skip. -- [ ] **Step 3: Write minimal implementation** — build the `tile-join` argv from the normalized spec; `subprocess.run(check=True)`. -- [ ] **Step 4: Run test to verify it passes** (Docker) → PASS. -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_simplify.py python/geobrix/test/vizx/test_simplify.py -git commit -m "feat(vizx): simplify_tiles_from_archive (tile-join down-zoom/trim)" -``` - ---- - -## Task 11: Wire simplify into the ladder + budget-escalate the archive path - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py` (`_simplify_layer` rung-3 hook), `python/geobrix/src/databricks/labs/gbx/vizx/_simplify.py` (`simplify_tiles_from_archive` budget escalation) -- Test: `python/geobrix/test/vizx/test_ladder.py` (add), `python/geobrix/test/vizx/test_simplify.py` (escalation) - -**Interfaces:** -- Consumes: `simplify_tiles_from_source`/`simplify_tiles_from_archive` (Tasks 9-10), `normalize_spec` (Task 8), `_decode_mvt_to_geoms`/`all_tiles` from `_pmtiles.py`. -- Produces: `_simplify_layer(layer, spec) -> Layer` — routes by layer input (source data → `from_source`; existing archive path → `from_archive`), returns a `pmtiles_layer` of the simplified bytes; used as ladder rung 3. - -**Budget escalation for `simplify_tiles_from_archive` (NEW — makes the archive budget contract real):** -`tile-join` only trims by zoom, so a zoom-trimmed archive may still have tiles over `budget_mb`. When `budget_mb` is requested and the trim is insufficient, **escalate to a source re-tile**: (1) run the cheap `tile-join` zoom-trim; (2) inspect the trimmed archive's max tile byte size; (3) if a tile still exceeds `budget_mb * 1 MiB`, **decode the archive's highest-zoom (max_z) tiles to a GeoDataFrame** via `_decode_mvt_to_geoms` over `all_tiles` (geographic geoms from z/x/y), then call `simplify_tiles_from_source(gdf, spec=...)` (tippecanoe `--drop-densest-as-needed`) which enforces the byte budget. Replace the prior `warnings.warn("budget_mb ignored")` with: within-budget-after-trim → no warning; escalated → `warnings.warn("budget enforced by re-tiling decoded features — slower; overview-grade precision", UserWarning)`; raster/undecodable archive → keep zoom-trim + the existing ignored-budget warning (no features to re-tile). Escalation is on-overflow only (the cheap trim is the common path). - -- [ ] **Step 1: Failing tests** — (a) ladder: an oversize vector layer + `simplify_tiles_spec` → `prepare_layers` returns `mode=="interactive"` with a "simplified" warning (skip if tippecanoe absent). (b) escalation: build a source archive whose tiles exceed a tiny `budget_mb`, `simplify_tiles_from_archive(archive, spec={"budget_mb":, "max_z":...})` → assert the result's max tile size is now ≤ budget (i.e. it re-tiled from source, not just zoom-trimmed) and a UserWarning about re-tiling fired. -- [ ] **Step 2: Run → FAIL.** -- [ ] **Step 3: Implement** — the `from_archive` escalation in `_simplify.py` (decode→`from_source` on overflow; reuse `_decode_mvt_to_geoms`/`all_tiles`), and `_simplify_layer` in `_maplibre.py` (route source→`from_source`, archive→`from_archive`; return a `pmtiles_layer` of the bytes), called from `prepare_layers` rung 3. -- [ ] **Step 4: Run → PASS** (Docker, tippecanoe present). Full vizx suite. -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py python/geobrix/src/databricks/labs/gbx/vizx/_simplify.py python/geobrix/test/vizx/test_ladder.py python/geobrix/test/vizx/test_simplify.py -git commit -m "feat(vizx): wire simplify into ladder; archive budget-escalates to source re-tile" -``` - ---- - -## Task 11b: Proactive embed-size audit + report (no surprises) - -The 64 MB embed budget should never be a *surprise* fallback — the per-layer + total size and the -chosen path must be auditable up front, before (and as part of) rendering. - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py` (`prepare_layers` returns an `audit`; add `audit_layers`), `_interactive.py` (`plot_interactive` reports the audit + a `dry_run` arg) -- Test: `python/geobrix/test/vizx/test_ladder.py` (add audit tests) - -**Interfaces:** -- Consumes: `layer_to_sources_layers`, `build_html`, `prepare_layers` (Tasks 4-6,11). -- Produces: - - `audit_layers(layers, *, max_embed_mb=64, simplify_tiles_spec=None) -> dict` — a DRY pre-flight (no `displayHTML`, no render): `{"layers":[{"label","kind","embed_bytes","max_tile_bytes"(archives only, else None)}], "total_embed_bytes", "max_embed_bytes", "fits": bool, "verdict": "embed"|"simplify"|"url"|"static"}`. - - `prepare_layers(...)` adds the same `audit` dict to its return. - - `plot_interactive(..., dry_run=False)`: always **prints a concise audit line** before rendering (e.g. `"[vizx] buildings 12.0MB + naip 40.0MB = 52.0MB ≤ 64MB → embedding inline"` or `"... 80MB > 64MB → simplifying"` / `"→ static fallback"`); `dry_run=True` returns the audit dict WITHOUT rendering. - -**Budget clarity (document + enforce):** the audited **total assembled-HTML size** is the embed-budget authority (the thing that can surprise); `simplify_tiles_spec.budget_mb` is the tippecanoe **per-tile** cap (rename its schema doc-comment from "total archive ceiling" to "per-tile byte cap"), and the ladder's post-simplify HTML-total re-check (Task 6) is what guarantees the embed fits. The audit reports BOTH the total (vs `max_embed_mb`) and per-archive max tile size so neither is a surprise. - -- [ ] **Step 1: Failing tests** — `audit_layers([small vector])` returns `fits=True`, `verdict="embed"`, with a `total_embed_bytes` and a per-layer entry; `plot_interactive([...], dry_run=True)` returns the audit dict and does NOT render (no HTML string with `maplibregl.Map`). An oversize case → `fits=False`, `verdict in {"simplify","static"}`. -- [ ] **Step 2: Run → FAIL.** -- [ ] **Step 3: Implement** — factor the size computation in `prepare_layers` into an `audit` dict (it already measures the assembled HTML size); expose `audit_layers`; have `plot_interactive` print the one-line summary and honor `dry_run`. Fix the `budget_mb` doc-comment in `_simplify.py` `normalize_spec` (per-tile, not total). -- [ ] **Step 4: Run → PASS.** Full vizx suite. -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py python/geobrix/src/databricks/labs/gbx/vizx/_interactive.py python/geobrix/src/databricks/labs/gbx/vizx/_simplify.py python/geobrix/test/vizx/test_ladder.py -git commit -m "feat(vizx): proactive embed-size audit + report (audit_layers, dry_run)" -``` - ---- - -## Task 12: Exports, extras, SRI pinning, back-compat - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/__init__.py`; `python/geobrix/pyproject.toml` (`[vizx]` extra); CI lock `requirements-pyrx-ci.in` + recompiled hashed `.txt`; `_maplibre.py` (real SRI hashes); **`_pmtiles.py` + `test_pmtiles.py` — remove the now-dead `_build_pmtiles_html` + its duplicate/divergent `_MAPLIBRE_JS`/`_PMTILES_JS` constants (pinned `pmtiles@3.2.1` vs `_maplibre.py`'s `3.2.0`) since `plot_pmtiles` now delegates; consolidate ONE CDN pin + SRI in `_maplibre.py` and drop the ~8 `test_pmtiles` tests that exercised the old standalone builder (the delegation path is covered by the new tests).** -- Test: `python/geobrix/test/vizx/test_exports.py` - -**Interfaces:** -- Produces: `__init__` exports `vector_layer, raster_layer, grid_layer, pmtiles_layer, simplify_tiles_from_source, simplify_tiles_from_archive` (plus the existing `plot_*`, `as_gdf`, etc.); `[vizx]` extra **adds** `tippecanoe`, `anywidget` and **removes** `folium`; the two SRI hashes in `_maplibre.py` are the real `sha384` of the locked MapLibre/pmtiles.js versions. - -- [ ] **Step 1: Write the failing test** - -```python -# python/geobrix/test/vizx/test_exports.py -import databricks.labs.gbx.vizx as v - -def test_new_public_symbols_exported(): - for name in ("vector_layer", "raster_layer", "grid_layer", "pmtiles_layer", - "simplify_tiles_from_source", "simplify_tiles_from_archive"): - assert hasattr(v, name), name - assert name in v.__all__ if hasattr(v, "__all__") else True - -def test_sri_hashes_are_real(): - from databricks.labs.gbx.vizx import _maplibre as m - assert m._MAPLIBRE_JS_SRI.startswith("sha384-") and "REPLACE" not in m._MAPLIBRE_JS_SRI - assert m._PMTILES_JS_SRI.startswith("sha384-") and "REPLACE" not in m._PMTILES_JS_SRI -``` - -- [ ] **Step 2: Run test to verify it fails** → FAIL. -- [ ] **Step 3: Write minimal implementation** — add imports + `__all__` entries; edit `[vizx]` extra (add tippecanoe/anywidget, drop folium); compute SRI: `curl -s | openssl dgst -sha384 -binary | openssl base64 -A` for each pinned version, paste as `sha384-`; update the CI `.in` and recompile the hashed `.txt` per the light-CI-lock procedure. -- [ ] **Step 4: Run test to verify it passes** → PASS. Also run `gbx:test:bindings` is N/A (vizx not in registered_functions), but run `gbx:lint:python --check`. -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/__init__.py python/geobrix/pyproject.toml python/geobrix/requirements-pyrx-ci.in python/geobrix/requirements-pyrx-ci.txt python/geobrix/src/databricks/labs/gbx/vizx/_maplibre.py python/geobrix/test/vizx/test_exports.py -git commit -m "feat(vizx): export layer/simplify API; [vizx] +tippecanoe +anywidget -folium; pin SRI" -``` - ---- - -## Task 13: Phase-1.5 dynamic zoom cut-over (AnyWidget) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/vizx/_dynamic.py` -- Test: `python/geobrix/test/vizx/test_dynamic.py` - -**Interfaces:** -- Consumes: `simplify_tiles_from_source`/`_archive` (overview build), `anywidget`, `normalize_spec`. -- Produces: `plot_interactive_dynamic(layers, *, simplify_tiles_spec=None, on_viewport=None, **kw) -> anywidget.AnyWidget` — embeds the simplified `min_z..max_z` overview as the base MapLibre source; the widget's `_esm` registers a `moveend` handler that, at `zoom > max_z`, calls `model.send({bbox, zoom})`; a Python `on_msg` handler invokes `on_viewport(bbox, zoom)` (default: tile the current viewport from the source via `simplify_tiles_from_source` with `min_z=max_z+1`), base64s the result into a synced trait, and the JS adds/updates a detail source on `change`. Comm contract proven by Spike B (`model.send` → `on_msg` → trait → `change`). - -- [ ] **Step 1: Write the failing test** (logic-level; the browser comm is covered by Spike B, not unit-testable headlessly): - -```python -# python/geobrix/test/vizx/test_dynamic.py -import pytest -anywidget = pytest.importorskip("anywidget") -from databricks.labs.gbx.vizx._dynamic import plot_interactive_dynamic, _viewport_payload -from databricks.labs.gbx.vizx._layers import vector_layer -import geopandas as gpd -from shapely.geometry import Point - -def test_builds_widget_with_overview_and_esm(): - gdf = gpd.GeoDataFrame({"v": [1]}, geometry=[Point(-122.4, 37.7)], crs="EPSG:4326") - w = plot_interactive_dynamic([vector_layer(gdf)], simplify_tiles_spec={"max_z": 8}) - assert isinstance(w, anywidget.AnyWidget) - assert "moveend" in w._esm and "model.send" in w._esm - -def test_viewport_payload_only_fires_above_seam(): - assert _viewport_payload(bbox=[-122.5,37.7,-122.4,37.8], zoom=12, max_z=10) is not None - assert _viewport_payload(bbox=[-122.5,37.7,-122.4,37.8], zoom=9, max_z=10) is None -``` - -- [ ] **Step 2: Run test to verify it fails** → FAIL. -- [ ] **Step 3: Write minimal implementation** — the AnyWidget subclass with `_esm` (overview embedded via the Task-5 builder output; `moveend` → `model.send` at `zoom>max_z`; `change:detail` → add/replace detail source) + the Python `on_msg` handler + `_viewport_payload` gate. Default `on_viewport` tiles the viewport via `simplify_tiles_from_source`. -- [ ] **Step 4: Run test to verify it passes** → PASS. (End-to-end comm verified manually per Spike B; note that in the report.) -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_dynamic.py python/geobrix/test/vizx/test_dynamic.py -git commit -m "feat(vizx): Phase-1.5 dynamic zoom cut-over (AnyWidget overview+stream)" -``` - -> Task 13 lands the **reactive** loop only (moveend → prepare current viewport → refresh). Predictive prefetch is the separate Task 13b below — build it on top once the reactive comm is proven. - ---- - -## Task 13b: Predictive tile prefetch for the dynamic viewer - -A polish layer on Task 13: render the initial viewport, then a background thread pre-prepares -adjacent tiles before the user pans/zooms to them, served from cache for instant response. - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/vizx/_dynamic.py` (cache + prefetch thread) -- Test: `python/geobrix/test/vizx/test_dynamic.py` (add) - -**Interfaces:** -- Consumes: the Task-13 AnyWidget viewport callback + the viewport tiler (`simplify_tiles_from_source`/the per-viewport prepare). -- Produces: a bounded driver-side **LRU tile cache** keyed by `(z, x, y)`; a `threading`-based prefetch worker that, after each viewport request, prepares the **ring of adjacent parent tiles** at the current zoom (and optionally `z+1`) into the cache; cache-hit serving so a pan to a prepared neighbor returns with no re-tile. The on-demand path checks the cache first (hit → instant), miss → prepare + return + trigger neighbor prefetch. - -**Constraints / guardrails:** -- **Bounded cache with an explicit evict policy** (configurable max entry count / total bytes) — prefetched tiles must not grow driver memory unbounded: - - *Baseline:* **LRU by last access** — a tile the user panned away from stops being accessed, ages to least-recently-used, and is evicted when new tiles need room (so "move away → removed" is automatic and memory is bounded). - - *Map-aware refinement (note, build after baseline):* when full, evict the tile **farthest from the current viewport center** rather than purely oldest, keeping the active neighborhood warm. - - *Prefetch guard:* a speculatively-prefetched tile that was **never viewed** must be evictable **before** a viewed tile (viewed > prefetched-unviewed), so eager prefetch can't push out real history. -- Prefetch runs on a **background `threading` worker** (daemon), never blocking the comm callback; heavy tiling (tippecanoe subprocess) is fine off-thread on the Serverless driver. -- **Coalesce / cancel**: a rapid sequence of viewport changes should not pile up stale prefetch work — cancel or skip prefetch for viewports the user has already left. -- Start with **neighbor-ring** prefetch; pan-velocity direction prediction is a later refinement (note, don't build). -- No `spark.conf`/`.rdd` (Serverless-safe); cache is pure driver-side Python. - -- [ ] **Step 1: Failing tests** — (a) a `_TileCache` LRU: insert beyond capacity evicts the oldest; get returns a hit/miss. (b) after a viewport request, the prefetch worker populates the neighbor-ring cache entries (test with a stub tiler + a join/flush on the worker so the test is deterministic — no real sleeps). (c) a second request for a prefetched neighbor is served from cache (the tiler is NOT called again). -- [ ] **Step 2: Run → FAIL.** -- [ ] **Step 3: Implement** the `_TileCache` (LRU, bounded), the prefetch worker (daemon thread, coalescing), and cache-first serving in the viewport callback. Make the worker test-injectable (pass the tiler + allow a synchronous flush) so tests are deterministic. -- [ ] **Step 4: Run → PASS.** Full vizx suite. (End-to-end pan-prefetch UX is verified manually like the rest of the dynamic tier.) -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/vizx/_dynamic.py python/geobrix/test/vizx/test_dynamic.py -git commit -m "feat(vizx): predictive neighbor-ring tile prefetch for the dynamic viewer" -``` - ---- - -## Task 14: Red-carpet docs (`vizx-layers.mdx`) + executable doc-tests + diagram - -**Files:** -- Create: `docs/docs/api/vizx-layers.mdx`; `docs/tests/python/api/vizx_layers.py`; `resources/images/generators/vizx-layers.py` + `resources/images/diagrams/vizx/vizx-layers.{svg,png}` -- Modify: `docs/sidebars.js` (add the page); `docs/docs/api/vizx.mdx` (link to the new page) -- Test: doc-tests via `gbx:test:python-docs` - -**Interfaces:** -- Consumes: the full public API (Tasks 1-13). -- Produces: a narrative page teaching the problem + the ladder, with multi-layer static + interactive examples, the ephemeral-vs-durable story, honest scale guidance (link Helios NB04 sharding + note the App is the indefinite-single-archive path), and a decision-tree diagram. Doc code is real + asserted in `vizx_layers.py` and imported by the `.mdx` via raw-loader. - -- [ ] **Step 1: Write the failing doc-test** - -```python -# docs/tests/python/api/vizx_layers.py -def multilayer_static_example(): - import geopandas as gpd - from shapely.geometry import Point, Polygon - from databricks.labs.gbx.vizx import vector_layer, grid_layer, plot_static - pts = gpd.GeoDataFrame({"v":[1]}, geometry=[Point(-122.4,37.7)], crs="EPSG:4326") - ax = plot_static([vector_layer(pts, color="red")], basemap=False) - assert ax is not None - return ax - -def simplify_durable_example(tmp_path): - import shutil - if shutil.which("tippecanoe") is None: - return None - import geopandas as gpd - from shapely.geometry import Polygon - from databricks.labs.gbx.vizx import simplify_tiles_from_source - gdf = gpd.GeoDataFrame({"v":[1]}, geometry=[Polygon([(0,0),(1,0),(1,1),(0,1)])], crs="EPSG:4326") - out = f"{tmp_path}/overview.pmtiles" - simplify_tiles_from_source(gdf, spec={"max_z":6}, out_path=out) - return out -``` - -- [ ] **Step 2: Run to verify it fails** → FAIL (until imports exist; here they do after Tasks 1-9, so this gates the docs land after code). -- [ ] **Step 3: Write the `.mdx`** importing those functions via `!!raw-loader!`, add the page to `sidebars.js`, link from `vizx.mdx`; generate the diagram (extend the diagram-generator pattern; render via the documented Chrome+PIL recipe into `resources/images/diagrams/vizx/`). -- [ ] **Step 4: Run doc-tests in Docker** - -Run (via Task subagent): `bash scripts/commands/gbx-test-python-docs.sh --path docs/tests/python/api/vizx_layers.py --log vizx-layers-docs.log` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add docs/docs/api/vizx-layers.mdx docs/tests/python/api/vizx_layers.py docs/sidebars.js docs/docs/api/vizx.mdx resources/images/generators/vizx-layers.py resources/images/diagrams/vizx/ -git commit -m "docs(vizx): red-carpet multi-layer + ladder page with executable examples" -``` - ---- - -## Task 14b: Capture real interactive screenshots + embed in docs/README/notebooks - -Real screenshots of the interactive MapLibre viewer in action — so `vizx-layers.mdx`, the -`README.md`, and the notebooks (which render static by default) all *preview* the interactive -experience. **Runs AFTER Task 14 (owns `vizx-layers.mdx`) and Task 16 (finalizes the Helios -notebook cells)** so screenshots embed into finished files, not race them. - -**Files:** -- Create: `resources/images/diagrams/vizx/screenshots/*.png` (the captures) + a small capture script (host-only, git-ignored or under `resources/images/generators/`). -- Modify: `docs/docs/api/vizx-layers.mdx`, `notebooks/examples/helios/README.md` (+ `docs/docs/notebooks/helios.mdx`), and the Helios notebooks NB01–04 (add a ~50%-width screenshot in a markdown cell directly ABOVE each conditional `plot_interactive`/`show_pmtiles` cell). - -**Interfaces:** `plot_interactive([...])` returns the self-contained HTML string (Task 7); the chrome-devtools MCP renders + screenshots. - -**Capture pipeline:** -- [ ] **Step 1 (de-risk PROTOTYPE):** build ONE small multi-layer interactive HTML via `plot_interactive([vector + raster + grid], …)` (return the HTML string; write to a temp `.html` under the repo so it's host-reachable). Via the chrome-devtools MCP: `new_page` → `navigate_page` to the `file://` URL → **`wait_for` the MapLibre canvas to actually paint** (wait on the rendered `.maplibregl-canvas` / a network-idle, NOT a fixed sleep) → `take_screenshot`. Read the PNG back and CONFIRM it shows a rendered map (not blank/loading). If headless can't paint the WebGL map even after waiting: fall back to (a) a real-browser capture, or (b) screenshot the static composite (`plot_static`) and label it as the static preview — and report the limitation. **Do not proceed to embedding until a non-blank capture is confirmed.** -- [ ] **Step 2:** produce the final captures — a full-size view for `vizx-layers.mdx`/README (the multi-layer overlay) and the per-notebook captures (one per notebook's interactive cell, content matching that notebook: NB01 buildings, NB02 buildings+NAIP, NB03 hillshade+buildings(+solar grid), NB04 a shard). Save under `resources/images/diagrams/vizx/screenshots/`. -- [ ] **Step 3:** embed — `vizx-layers.mdx` (a hero screenshot near the interactive section) + `README.md` + each Helios notebook (a markdown cell with the ~50%-width image immediately above the conditional interactive cell, captioned "Interactive view (INTERACTIVE_PLOTS=True)"). Use the repo-relative image path convention the other docs use. -- [ ] **Step 4:** verify the notebooks still parse (nbformat) + the cell-by-cell harness reaches its config ceiling unchanged; the mdx/README render references resolve. `grep` for internal vocab in any touched user-facing doc → none. -- [ ] **Step 5: Commit** the screenshots + the doc/notebook edits: `docs(vizx): real interactive viewer screenshots in docs, README, notebooks` (≤72; trailer `Co-authored-by: Isaac`). - -**Note:** the capture script + headless render are HOST-only (Chrome). Screenshots need CDN + CARTO-basemap internet to paint; the host has it. - ---- - -## Task 15: Audit, migrate, AND showcase the new viewer in the example notebooks - -This task does two things per notebook: (a) migrate any usage the folium-retirement or -`plot_interactive` signature change would break, and (b) **actively showcase** the new multi-layer -interactive viewer on each notebook's *final* artifact — not just keep it from breaking. The -showcase is the point: these notebooks are where users learn the capability exists. - -**Files:** -- Modify: `notebooks/examples/eo-series/03. Gridded EO Data.ipynb`, `04. Band Stacking + Clipping.ipynb`, `notebooks/examples/xview/Clipping - xView.ipynb`, `notebooks/examples/h3-rasterize/*.ipynb`, and their READMEs / `docs/docs/notebooks/*.mdx` where they call `plot_*`/`show_*`/folium. -- Test: per-notebook cell-by-cell harness up to the config ceiling. - -**Interfaces:** -- Consumes: `vector_layer`/`raster_layer`/`grid_layer`/`pmtiles_layer` + `plot_interactive([...])` (Tasks 1,7); `cells_as_gdf` (existing). Produces: every VizX usage reads consistently against the new surface; no folium references remain; each example ends with a real multi-layer interactive showcase honoring `INTERACTIVE_PLOTS`. - -- [ ] **Step 1: Audit (work-list).** Run `grep -rn -E "plot_interactive|plot_static|plot_pmtiles|plot_cog|plot_raster|show_(pmtiles|cog|raster)|folium" notebooks/ docs/docs` and record every hit with its call shape. No code change yet. -- [ ] **Step 2: Migrate** each affected call to the new signatures (a multi-arg `plot_interactive(df, column=...)` still works via coercion; remove folium-specific kwargs; fix prose referencing folium). -- [ ] **Step 3: Showcase — eo-series.** The series currently renders its final raster with a single-layer `plot_raster`. Add an interactive multi-layer cell on the merged/stacked result: - - **NB03 "Gridded EO Data"** — after the `rst_merge_agg` cell that produces `kring_df` (the merged raster per H3 kring; the existing `plot_raster(kring_df.select("tile.raster").first()[0])`), add: - ```python - # Showcase: the merged raster + its H3 tessellation in one interactive map. - from databricks.labs.gbx.vizx import raster_layer, grid_layer, plot_interactive - plot_interactive([ - raster_layer(kring_df.select("tile.raster").first()[0]), - grid_layer(kring_df, grid_system="h3", cellid_col="kring", opacity=0.25), - ]) - ``` - - **NB04 "Band Stacking + Clipping"** — after `stacked_df` (the band-stacked rasters) and the clip cell (`to_plot[0]["clip_tile"]["raster"]`), add a showcase overlaying the stacked RGB tile with the clip cutline / H3 cells via `plot_interactive([raster_layer(), grid_layer(stacked_df, grid_system="h3", cellid_col="cellid", opacity=0.25)])`. - Both go through `show_*`/the `INTERACTIVE_PLOTS` toggle so the committed `.ipynb` stays static for GitHub. -- [ ] **Step 4: Showcase — xview.** After the clip result (`clip_raster = clip_row['tile_clip']['raster']`, currently `plot_raster`/`plot_file`), add the headline "clip to vector" view interactively — the clipped aerial tile with the labeled detected-object boundaries on top: - ```python - from databricks.labs.gbx.vizx import raster_layer, vector_layer, plot_interactive - plot_interactive([ - raster_layer(clip_raster), - vector_layer(objects_gdf, color="#ff3", width=2), # the xView object boundaries (from the objects table) - ]) - ``` - (`objects_gdf` = the detected-objects geometries built in section [3]; convert via `as_gdf` if it's a Spark DataFrame.) -- [ ] **Step 5: Showcase — h3-rasterize (opportunistic).** If it produces a raster + an H3 grid, add `plot_interactive([raster_layer(), grid_layer(, grid_system="h3")])` on the final result; skip if no natural multi-layer pairing. -- [ ] **Step 6: Verify** each touched notebook with `bash scripts/commands/gbx-test-notebooks.sh --path ""` to its config ceiling (no NEW early-cell break); `grep -rn folium notebooks/ docs/docs` prints nothing. -- [ ] **Step 7: Commit** - -```bash -git add notebooks/ docs/docs -git commit -m "docs: migrate + showcase the multi-layer viewer in eo-series/xview/h3-rasterize" -``` - ---- - -## Task 16: Helios NB02/NB03 real overlays + prose fix - -**Files:** -- Modify: `notebooks/examples/helios/02. Visual Basemap (XYZ).ipynb`, `03. Analytical Core (COG + STAC).ipynb`; `notebooks/examples/helios/README.md`; `docs/docs/notebooks/helios.mdx` -- Test: cell-by-cell harness to config ceiling - -**Interfaces:** -- Consumes: `plot_interactive([...])` multi-layer. -- Produces: NB02 actually overlays the buildings (NB01) over the NAIP basemap in one `plot_interactive([pmtiles_layer(naip), pmtiles_layer(buildings)])` call; NB03 overlays hillshade + buildings (+ the solar-score grid where natural). Prose in README/helios.mdx no longer implies an overlay the old single-archive calls didn't do. - -- [ ] **Step 1: Edit NB02** to add a real multi-layer overlay cell using `plot_interactive([...])` (after the single-archive cells, with an honest comment). -- [ ] **Step 2: Edit NB03** similarly (hillshade + buildings + optional `grid_layer` of `solar_score`). -- [ ] **Step 3: Fix prose** in README + helios.mdx (the "overlays it with the buildings layer" lines now describe a real overlay; keep the static `INTERACTIVE_PLOTS` default honest). -- [ ] **Step 4: Verify** with `gbx:test:notebooks --path` for NB02/NB03 to the config ceiling; `grep -rn -iE "wave [0-9]" docs/docs/notebooks/helios.mdx` prints nothing (QC voice). -- [ ] **Step 5: Commit** - -```bash -git add "notebooks/examples/helios/02. Visual Basemap (XYZ).ipynb" "notebooks/examples/helios/03. Analytical Core (COG + STAC).ipynb" notebooks/examples/helios/README.md docs/docs/notebooks/helios.mdx -git commit -m "docs(helios): real multi-layer overlays in NB02/NB03 + honest prose" -``` - ---- - -## Self-Review - -**Spec coverage:** Layer model (T1); plot_static multi-layer + plot_cog ax= (T2,T3); MapLibre adapters/builder/folium-retire (T4,T5,T7); >64MB ladder (T6,T11); simplify two-flavor + spec + tippecanoe/tile-join/rasterio engine policy (T8,T9,T10); exports/extras/SRI/supply-chain (T12); Phase-1.5 dynamic cut-over (T13); red-carpet docs + diagram + doc-tests (T14); notebook/doc audit + migrate + **showcase** the new viewer in eo-series/xview/h3-rasterize (T15); Helios rewiring + prose (T16). Indefinite single-archive correctly OUT (App). contextily retained on the static path (T2/T3). Covered. - -**Placeholder scan:** The only intentional deferred literal is the SRI hash, explicitly computed and asserted real in Task 12 (test guards against `REPLACE`). No "TBD/handle edge cases" steps. - -**Type consistency:** `Layer` fields and constructor params (`geom_col`/`cellid_col`/`column`/`grid_system`/`simplify`) are used identically in T3/T4/T11; `prepare_layers`/`build_html`/`layer_to_sources_layers`/`normalize_spec`/`simplify_tiles_from_source`/`simplify_tiles_from_archive` names match across tasks; `plot_interactive`/`plot_static` signatures consistent T3/T7/T11. - ---- - -## Notes for the executor - -- **Docker-only tests:** Tasks 9-11, 14 need tippecanoe/`tile-join` + sample data → run via `gbx:test:python` / `gbx:test:python-docs` in the dev container. Pure-logic tests (T1,T2,T3,T4,T5,T6,T8,T13) run host or Docker. -- **Prerequisite:** refresh the staged light wheel on `geospatial_docs` before the Helios/doc-test runs (current one predates `pmtiles_info` in `gbx.pmtiles`). -- **Lint before push:** `gbx:lint:python --check` (host black may differ from Docker — verify in-container). diff --git a/docs/superpowers/plans/2026-06-29-xyz-tile-data-aware-rescale-heavy.md b/docs/superpowers/plans/2026-06-29-xyz-tile-data-aware-rescale-heavy.md deleted file mode 100644 index f38c5b54b..000000000 --- a/docs/superpowers/plans/2026-06-29-xyz-tile-data-aware-rescale-heavy.md +++ /dev/null @@ -1,958 +0,0 @@ -# XYZ Tile Data-Aware Rescale — Heavy/Classic Tier Implementation Plan (Phase 2) - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. ALL Scala build/test/lint/bindings work runs **in the `geobrix-dev` Docker container** — dispatch the long-running Docker commands via a Task subagent (CLAUDE.md orchestrator pattern); never run them inline. - -**Goal:** Mirror the light-tier `rescale` parameter (Phase 1, complete) in the heavy/classic Scala tier (`RST_TileXYZ` / `RST_XYZPyramid`) so non-8-bit imagery recovers contrast at PNG/JPEG/WEBP encode time, computed from **whole-dataset per-band min/max once per source**, fed to `gdal_translate -scale`. Then add the cross-tier pixel/value-distribution parity gate that proves both tiers feed the SAME per-band `(min,max)`. - -**Architecture:** Add a trailing `rescale` expression to `RST_TileXYZ` (InvokedExpression) and `RST_XYZPyramid` (CollectionGenerator). A new helper resolves the user's `rescale` arg + the open source `Dataset` into a per-band `Seq[(Double,Double)]` (or `None` for pass-through), exactly mirroring light's `_resolve_in_range`: - -- `"auto"` (default): uint8 source → `None` (pass-through); non-8-bit → per-band whole-dataset `(min,max)` via `BandAccessors.getMinMax` (`ComputeRasterMinMax(_, 0)` = exact). -- `"none"`: `None` (today's full-dtype-range crush). -- explicit `(min,max)` pair: that pair repeated for every band. - -The resolved per-band ranges are formatted into GDAL `-scale_ min max 0 255` flags (per-band repeatable) and threaded through the existing `RST_TileXYZ.execute` → `GDALTranslate.executeTranslate` → `OperatorOptions.appendOptions` PNG/JPEG/WEBP branch. `RST_XYZPyramid` resolves the ranges **once** before the tile loop (mirroring light's resolve-once / no-seams contract) and passes them to every `RST_TileXYZ.execute`. - -**Tech Stack:** Scala 2.13.16, Spark 4.0.0, Java 17, GDAL Java bindings (`org.gdal.gdal`), ScalaTest (`AnyFunSuite`). All tests run in the `geobrix-dev` Docker container via `gbx:*` commands. - -## Global Constraints - -> **LOCKED DECISIONS (user-approved 2026-06-29) — these override any hedges/"if unsure"/"DECISION NEEDED" notes elsewhere in this plan:** -> 1. **PNG-ONLY scope.** Apply `-scale` to the PNG byte-output branch ONLY. Do NOT add a JPEG/WEBP `-scale` branch in this work — leave JPEG/WEBP exactly as they are today (default branch, no `-ot Byte`/`-scale` change). Where a task shows an optional `case "JPEG" | "WEBP"` branch, OMIT it. Add a one-line note in the function docstring/scaladoc that `rescale` currently affects PNG output (the Helios/spec path); JPEG/WEBP rescale is a documented future follow-up. -> 2. **Accept the NoData divergence + document it.** `ComputeRasterMinMax` ignores NoData unless band NoData is set, whereas light's `ds.statistics(approx=False)` honors the mask. Accept this for now (fixtures are NoData-free). Add a scaladoc note on `resolveScale` that on NoData rasters heavy's min/max may include masked values and can diverge slightly from light — a known limitation, revisit if needed. -> 3. **Explicit pair = single `"min,max"` STRING** (e.g. `'8000,12000'`), as already specified. No numeric-arg arity change. -> 4. **Task 7 is REQUIRED, not optional.** The cross-language live parity check in the bench/doc-test container is part of the definition of done (the in-Scala Task 6 is the algebraic gate; Task 7 is the real cross-tier proof). Treat Task 7 as a normal required task. -> 5. **Reflection contract CONFIRMED** (controller verified): `RST_TileXYZ` is an `InvokedExpression` with four eval overloads `evalBinary/evalPath(row, z, x, y, format, size, resampling, conf)` (Int and Long variants) and builder `case 4..7`. Inserting `rescale` means adding a `UTF8String rescale` param to all four overloads IMMEDIATELY BEFORE `conf`, and extending the builder to `case 4..8` with `Literal("auto")` defaults. The SQL-level exercise in Tasks 2/4/6 is the catch-net for any positional mismatch. - -- **Parity is pixel/value-distribution-level, NOT byte-level.** Heavy re-encodes a GTiff per tile and GDAL's PNG encoder differs from rio-tiler's; exact-byte cross-tier equality is NOT guaranteed (established "light-readers" convention). The cross-tier parity test asserts equivalent per-band value **distribution** within a tolerance for `"auto"`. The ONE byte-level assertion is uint8 pass-through being identical *within each tier* (auto == none for uint8, no `-scale` emitted). -- **GDALManager-guarded GDAL registration only.** All GDAL driver registration goes through the synchronized `GDALManager.init` (already reached via `RST_ExpressionUtil.init`) / `GDALManager.initOgr`. Stats computation (`ComputeRasterMinMax`) operates on an already-open `Dataset`/`Band` and does NOT register drivers or mutate process-global config — it is safe under concurrency. NEVER add a raw `gdal.AllRegister()` / `gdal.GetDriverByName` / `gdal.SetConfigOption` on the new code path. -- **Parameter named exactly `rescale`** — one canonical name, no aliases (beta = break to stabilize). -- **Cross-language naming consistency** (from CLAUDE.md): - - Scala class: `RST_TileXYZ` / `RST_XYZPyramid` (unchanged). - - SQL (registered): `gbx_rst_tilexyz` / `gbx_rst_xyzpyramid` (unchanged). - - The new arg is `rescale` everywhere (Python `rescale` already shipped in Phase 1). -- **Heavy tests + lint + bindings run in Docker.** Commands: - - `gbx:test:scala --suite ''` (single suite) / `--suites 'A,B'` - - `gbx:lint:scalastyle` (matches CI — run before push) - - `gbx:test:bindings` (binding-parity gate, also run by the QC judge on push) - - `gbx:docs:function-info` (regenerate `function-info.json`) -- **Rescale semantics (copied verbatim from the spec):** - - **`"auto"` (default):** - - **uint8 source → pass through unchanged.** Already display-ready (RGB / NAIP byte imagery); never touched. Protects the cases that are correct today. - - **non-8-bit source → rescale to Byte using whole-dataset per-band min/max,** computed **once per source** (not per tile). Recovers contrast AND guarantees every tile shares one mapping → **no tile-to-tile seams.** - - **`"none"`:** today's raw full-dtype-range behavior. Explicit escape hatch for anyone depending on current output. - - **`(min, max)` explicit pair:** use exactly these bounds, skip the stats read. (Per-band uniform; a single pair applied to all bands.) -- **`rescale` accepted values:** the string `"auto"` (default), the string `"none"`, or a 2-element `(min, max)` numeric pair. The Scala SQL surface accepts `rescale` as the string modes via a trailing string arg; the explicit numeric pair is expressed as a string literal `"min,max"` (e.g. `'8000,12000'`) parsed by the resolver, keeping the SQL signature a single extra `STRING` arg (consistent with `format`/`resampling` being strings). Anything else throws `IllegalArgumentException` (fail-fast, mirroring light's `ValueError`). -- **Stats read cost:** one `ComputeRasterMinMax` per band per source (not per tile); skipped entirely for uint8 and the explicit-pair path. `RST_XYZPyramid` resolves once for the whole pyramid. - -## File Structure - -``` -src/main/scala/com/databricks/labs/gbx/rasterx/ - expressions/web/RST_TileXYZ.scala (modify: +rescaleExpr child, +resolve helper, +scale threading, +builder arity) - expressions/web/RST_XYZPyramid.scala (modify: +rescaleExpr child, resolve-once, +builder arity) - operator/OperatorOptions.scala (modify: PNG/JPEG/WEBP branches accept an optional -scale string) -src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/ - WebMercatorTileTest.scala (extend: heavy rescale unit assertions) - XYZRescaleParityTest.scala (NEW: cross-tier pixel/value-distribution parity gate) -docs/tests-function-info/registered_functions.txt (no NEW function; arg-only change — confirm no row needed) -docs/tests/python/api/rasterx_functions_sql.py (modify: add rescale to *_sql_example()) -src/main/resources/com/databricks/labs/gbx/function-info.json (regenerate via gbx:docs:function-info) -``` - -> **Note:** `RST_AsFormat` and `RST_FromContent` callers of `OperatorOptions.appendOptions` MUST be unaffected — the `-scale` injection is opt-in (only when a non-empty scale string is supplied in `writeOptions`), default unchanged. Task 1 enforces this with a regression assertion. - ---- - -### Task 1: `OperatorOptions` PNG/JPEG/WEBP branches accept an optional `-scale` string (default unchanged) - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/operator/OperatorOptions.scala` -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/WebMercatorTileTest.scala` (extend) — exercised indirectly; a focused string-level assertion lives here to keep the suite single. - -**Current state (real):** `OperatorOptions.appendOptions` (line 18) at line 62 emits for PNG: -```scala -case "PNG" => s"$command $ofFlag $format -ot Byte -a_nodata none" // PNG Byte format, strip NoData to avoid tRNS issues -``` -There is NO JPEG/WEBP-specific branch today — both fall through to the `case f =>` default (line 64) `s"$command $ofFlag $f $cos"`. (Confirm during implementation: `RST_TileXYZ.execute` line 132-137 maps webp/jpeg extensions and passes `format -> "JPEG"/"WEBP"`; those currently hit the default branch with compression `-co` flags, which is the pre-existing behavior. We add `-scale` to the byte-output path without changing the format-flag stamping.) - -**Design decision:** Thread the resolved per-band scale as a single pre-formatted string in `writeOptions` under key `"scale"` (e.g. `"-scale_1 8000 12000 0 255 -scale_2 ..."`). When present and non-empty, append it to the byte-output (PNG/JPEG/WEBP) branches. Keeping it a formatted string (not a structured type) means `OperatorOptions` stays a pure string assembler — its existing role — and the per-band formatting lives in `RST_TileXYZ` (Task 2) where the `Dataset` band count is known. - -GDAL `-scale_ src_min src_max dst_min dst_max` is the per-band repeatable form (`-scale_1 ... -scale_2 ...`); it co-exists with `-ot Byte`. `-a_nodata none` is preserved (stripping NoData avoids PNG tRNS issues, unrelated to scaling). - -- [ ] **Step 1: Write the failing test** - -Add to `WebMercatorTileTest.scala` (it already sets up GDAL via `GDALManager` in `beforeAll`). Add a string-assembly assertion against `OperatorOptions.appendOptions`. We need a `Dataset` for the signature — reuse `srcDs`: - -```scala - test("OperatorOptions PNG branch injects -scale when scale option supplied") { - val withScale = com.databricks.labs.gbx.rasterx.operator.OperatorOptions.appendOptions( - "gdal_translate", - Map("format" -> "PNG", "scale" -> "-scale_1 8000 12000 0 255"), - srcDs - ) - withScale should include("-ot Byte") - withScale should include("-a_nodata none") - withScale should include("-scale_1 8000 12000 0 255") - } - - test("OperatorOptions PNG branch unchanged when no scale option") { - val noScale = com.databricks.labs.gbx.rasterx.operator.OperatorOptions.appendOptions( - "gdal_translate", Map("format" -> "PNG"), srcDs - ) - noScale shouldBe "gdal_translate -of PNG -ot Byte -a_nodata none" - noScale should not include "-scale" - } -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Dispatch a Task subagent to run (Docker): -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.web.WebMercatorTileTest' --log heavy-rescale-op-options.log -``` -Expected: the two new tests FAIL — `-scale` is not injected (first test fails on `include("-scale_1...")`). - -- [ ] **Step 3: Implement — append the optional `-scale` string in the byte-output branches** - -In `OperatorOptions.appendOptions`, after the `val cos = ...` line (line 55) add: -```scala - // Optional per-band rescale string (e.g. "-scale_1 min max 0 255 -scale_2 ..."), - // supplied by the XYZ tilers for data-aware 8-bit encoding. Empty/absent => no -scale - // (today's full-dtype-range behavior). See RST_TileXYZ rescale resolution. - val scaleFlags = writeOptions.getOrElse("scale", "").trim - val scaleSuffix = if (scaleFlags.isEmpty) "" else s" $scaleFlags" -``` -Then change the PNG branch (line 62) to: -```scala - case "PNG" => s"$command $ofFlag $format -ot Byte -a_nodata none$scaleSuffix" // PNG Byte format, strip NoData to avoid tRNS issues -``` -JPEG/WEBP currently hit the `case f =>` default. To carry `-scale` onto those byte outputs too, add explicit branches BEFORE the `case f =>` (preserving `cos` compression flags so existing JPEG/WEBP output is otherwise unchanged): -```scala - case "JPEG" | "WEBP" => s"$command $ofFlag $format -ot Byte$cos$scaleSuffix" -``` - -> **Implementation caution (verify in Step 4):** confirm the pre-change JPEG/WEBP output by reading the default-branch result for those formats BEFORE adding the explicit branch (assert the new branch reproduces the old `-of ` and only adds `-ot Byte` + optional scale). If JPEG/WEBP did NOT previously get `-ot Byte`, adding it is the intended behavior (they are byte web formats), but the test must document the deliberate change. If unsure, scope this task to PNG-only and handle JPEG/WEBP scale in a follow-up — PNG is the path the spec/Helios case exercises. - -- [ ] **Step 4: Run the test to verify it passes** - -Dispatch a Task subagent (Docker): -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.web.WebMercatorTileTest' --log heavy-rescale-op-options.log -``` -Expected: the two new tests PASS; all pre-existing `WebMercatorTileTest` tests still PASS (PNG default unchanged). - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/operator/OperatorOptions.scala src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/WebMercatorTileTest.scala -git commit -m "feat(rasterx): OperatorOptions accepts optional -scale for byte tile output - -The PNG (and JPEG/WEBP) byte-output branches now append a pre-formatted -per-band -scale string when writeOptions carries a non-empty 'scale' key; -default output (no scale) is byte-identical to before. Enables the XYZ -tilers' data-aware 8-bit rescale without disturbing other translate callers. - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: `RST_TileXYZ` resolves `rescale` to per-band scale + threads it into the translate step - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/web/RST_TileXYZ.scala` -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/WebMercatorTileTest.scala` (extend) - -**Current state (real signatures):** -- `case class RST_TileXYZ(tileExpr, zExpr, xExpr, yExpr, formatExpr, sizeExpr, resamplingExpr)` (line 34-42), `children` includes `ExpressionConfigExpr()` (line 46), `withNewChildrenInternal` copies `nc(0)..nc(6)` (line 52). -- `evalBinary/evalPath` overloads (Int + Long) at lines 70-77 call `doInvoke(row, z, x, y, format, size, resampling, conf, dt)`. -- `doInvoke` (line 79) validates format/resampling/size, then `RasterSerializationUtil.rowToTile(row, dt)` → `(_, ds, options)` and calls `execute(ds, options, z, x, y, fmt, size, resampleLower)` (line 99). -- `execute(ds, options, z, x, y, format, size, resampling)` (line 111) warps to GTiff then `GDALTranslate.executeTranslate(translatePath, warpedDs, "gdal_translate", warpedOpts ++ Map("format" -> format, "extension" -> extension))` (line 139-144). -- Builder (line 206) is arity 4-7 with `Literal` defaults. -- `BandAccessors.getMinMax(band)` (in `operations/BandAccessors.scala` line 23) returns `(Double,Double)` via `band.ComputeRasterMinMax(minmax, 0)` (force=0 = exact) — REUSE THIS. - -**Design decision (stats API):** Use the existing `BandAccessors.getMinMax(band)` (`ComputeRasterMinMax(_, 0)`, exact/force) — it is the canonical min/max accessor in this codebase, operates on an open `Band` (no driver registration, concurrency-safe), and matches light's whole-dataset `ds.statistics(b, approx=False)`. We deliberately do NOT use `AsMDArray().GetStatistics()` (that path is per CLAUDE.md MDArray-only and returns mean/stddev, not a clean exact min/max on a plain Band) nor `ComputeRasterMinMax(_, 1)` (approx — would risk cross-tier drift vs light's exact). - -**Design decision (resolve helper):** Add `resolveScale(ds: Dataset, rescale: String): String` returning the pre-formatted `-scale_ ...` string (or `""` for pass-through). It mirrors light's `_resolve_in_range`: -- parse `rescale` (default/null → `"auto"`); `"none"` → `""`; `"min,max"` numeric → that pair for every band; `"auto"` → uint8 first band → `""`, else per-band `getMinMax`. -- Constant band (min == max) widened to `(min, min+1)` (matches light). - -- [ ] **Step 1: Write the failing tests** - -Add to `WebMercatorTileTest.scala`. Build a uint16 narrow-range fixture (mirroring light's `_make_uint16_narrow`), assert auto recovers contrast and none stays crushed, and uint8 passes through. We decode the PNG via GDAL (open the `/vsimem/` bytes and read the band) to assert the value spread. - -```scala - /** Decode PNG bytes via GDAL and return (min, max) of the first band's non-zero - * (i.e. data, ignoring transparent) pixels. */ - private def pngBandSpread(bytes: Array[Byte]): (Int, Int) = { - val path = s"/vsimem/parity_decode_${java.util.UUID.randomUUID().toString.replace("-", "")}.png" - gdal.FileFromMemBuffer(path, bytes) - val ds = gdal.Open(path) - try { - val band = ds.GetRasterBand(1) - val buf = Array.ofDim[Byte](ds.GetRasterXSize * ds.GetRasterYSize) - band.ReadRaster(0, 0, ds.GetRasterXSize, ds.GetRasterYSize, buf) - val vals = buf.map(_ & 0xff).filter(_ > 0) - if (vals.isEmpty) (0, 0) else (vals.min, vals.max) - } finally { - ds.delete(); gdal.Unlink(path) - } - } - - /** 16×16 uint16 raster, EPSG:4326, footprint (-1,-1)→(1,1), values ramped over [8000,12000]. */ - private def makeUint16Narrow(): Dataset = { - val drv = gdal.GetDriverByName("MEM") - val ds = drv.Create("/vsimem/rescale_u16", 16, 16, 1, gdalconstConstants.GDT_UInt16) - ds.SetGeoTransform(Array(-1.0, 0.125, 0.0, 1.0, 0.0, -0.125)) - val sr = new org.gdal.osr.SpatialReference(); sr.ImportFromEPSG(4326) - ds.SetProjection(sr.ExportToWkt()) - val n = 256 - val ramp = (0 until n).map(i => (8000.0 + (12000.0 - 8000.0) * i / (n - 1))).toArray - ds.GetRasterBand(1).WriteRaster(0, 0, 16, 16, ramp) - ds.GetRasterBand(1).FlushCache() - ds - } - - test("RST_TileXYZ rescale=auto recovers contrast for uint16 narrow-range") { - val ds = makeUint16Narrow() - try { - // z=2 tile (2,1) overlaps the (-1..1) footprint (see existing in-extent test). - val auto = RST_TileXYZ.execute(ds, Map.empty[String, String], 2, 2, 1, "PNG", 64, "near", "auto") - val (lo, hi) = pngBandSpread(auto) - // Auto maps [8000,12000] -> ~full 8-bit; expect a wide spread, NOT crushed [31,46]. - (hi - lo) should be > 100 - } finally ds.delete() - } - - test("RST_TileXYZ rescale=none stays crushed for uint16 narrow-range") { - val ds = makeUint16Narrow() - try { - val none = RST_TileXYZ.execute(ds, Map.empty[String, String], 2, 2, 1, "PNG", 64, "near", "none") - val (_, hi) = pngBandSpread(none) - // 8000..12000 / 65535 * 255 -> ~[31,46]; crushed. - hi should be < 80 - } finally ds.delete() - } - - test("RST_TileXYZ uint8 source: auto == none (byte-identical pass-through)") { - // srcDs is Float64 in this suite; build a uint8 source for the pass-through proof. - val drv = gdal.GetDriverByName("MEM") - val ds = drv.Create("/vsimem/rescale_u8", 16, 16, 1, gdalconstConstants.GDT_Byte) - ds.SetGeoTransform(Array(-1.0, 0.125, 0.0, 1.0, 0.0, -0.125)) - val sr = new org.gdal.osr.SpatialReference(); sr.ImportFromEPSG(4326) - ds.SetProjection(sr.ExportToWkt()) - ds.GetRasterBand(1).WriteRaster(0, 0, 16, 16, Array.fill(256)(100.0)) - ds.GetRasterBand(1).FlushCache() - try { - val auto = RST_TileXYZ.execute(ds, Map.empty[String, String], 2, 2, 1, "PNG", 64, "near", "auto") - val none = RST_TileXYZ.execute(ds, Map.empty[String, String], 2, 2, 1, "PNG", 64, "near", "none") - java.util.Arrays.equals(auto, none) shouldBe true // no -scale emitted for uint8 auto - } finally ds.delete() - } -``` - -> The existing `RST_TileXYZ.execute(...)` 3 tests call the OLD 8-arg arity (no rescale). They will be updated to the new arity in Step 3 (add a trailing `"none"` to preserve their meaning, or rely on a defaulted overload — see Step 3). - -- [ ] **Step 2: Run the tests to verify they fail** - -Dispatch a Task subagent (Docker): -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.web.WebMercatorTileTest' --log heavy-rescale-tilexyz.log -``` -Expected: COMPILE FAILURE — `execute` does not take a 9th `rescale` arg yet. (That is an acceptable red: it proves the new signature is required.) - -- [ ] **Step 3: Implement — add `rescaleExpr`, `resolveScale`, and thread into `execute`** - -**(a) Constructor + children + builder.** Add `rescaleExpr: Expression` as the 8th field. Update: - -```scala -case class RST_TileXYZ( - tileExpr: Expression, - zExpr: Expression, - xExpr: Expression, - yExpr: Expression, - formatExpr: Expression, - sizeExpr: Expression, - resamplingExpr: Expression, - rescaleExpr: Expression -) extends InvokedExpression { - - private def rasterType = RST_ExpressionUtil.rasterType(tileExpr) - override def children: Seq[Expression] = - Seq(tileExpr, zExpr, xExpr, yExpr, formatExpr, sizeExpr, resamplingExpr, rescaleExpr, ExpressionConfigExpr()) - override def dataType: DataType = BinaryType - override def nullable: Boolean = true - override def prettyName: String = RST_TileXYZ.name - override def replacement: Expression = rstInvoke(RST_TileXYZ, rasterType) - override protected def withNewChildrenInternal(nc: IndexedSeq[Expression]): Expression = - copy(nc(0), nc(1), nc(2), nc(3), nc(4), nc(5), nc(6), nc(7)) -} -``` - -> **Verify the InvokedExpression `rstInvoke` reflection contract:** `replacement = rstInvoke(RST_TileXYZ, rasterType)` reflectively dispatches to the `evalBinary/evalPath` overloads by matching the (non-config) children to method params. Adding `rescaleExpr` as a child means the `evalBinary/evalPath` overloads MUST gain a matching trailing `UTF8String rescale` param so reflection still binds. Read `InvokedExpression`/`rstInvoke` before implementing to confirm arg ordering (children minus the trailing `ExpressionConfigExpr` map positionally to the eval-method params, with `conf` last). This is the single highest-risk wiring step — see Risks. - -Update the four eval overloads (lines 70-77) to take `rescale: UTF8String` BEFORE `conf` (matching child order: ..., resampling, rescale, conf): - -```scala - def evalBinary(row: InternalRow, z: Int, x: Int, y: Int, format: UTF8String, size: Int, resampling: UTF8String, rescale: UTF8String, conf: UTF8String): Array[Byte] = - doInvoke(row, z, x, y, format, size, resampling, rescale, conf, BinaryType) - def evalBinary(row: InternalRow, z: Long, x: Long, y: Long, format: UTF8String, size: Long, resampling: UTF8String, rescale: UTF8String, conf: UTF8String): Array[Byte] = - doInvoke(row, z.toInt, x.toInt, y.toInt, format, size.toInt, resampling, rescale, conf, BinaryType) - def evalPath(row: InternalRow, z: Int, x: Int, y: Int, format: UTF8String, size: Int, resampling: UTF8String, rescale: UTF8String, conf: UTF8String): Array[Byte] = - doInvoke(row, z, x, y, format, size, resampling, rescale, conf, StringType) - def evalPath(row: InternalRow, z: Long, x: Long, y: Long, format: UTF8String, size: Long, resampling: UTF8String, rescale: UTF8String, conf: UTF8String): Array[Byte] = - doInvoke(row, z.toInt, x.toInt, y.toInt, format, size.toInt, resampling, rescale, conf, StringType) -``` - -Update `doInvoke` (line 79) to accept `rescale: UTF8String` and pass it to `execute`: - -```scala - private def doInvoke( - row: InternalRow, - z: Int, x: Int, y: Int, - format: UTF8String, size: Int, resampling: UTF8String, - rescale: UTF8String, conf: UTF8String, dt: DataType - ): Array[Byte] = { - val safe: () => Array[Byte] = () => { - val exprConf = ExpressionConfig.fromB64(conf.toString) - RST_ExpressionUtil.init(exprConf) - val fmtStr = if (format == null) "PNG" else format.toString - val resampleStr = if (resampling == null) "bilinear" else resampling.toString - val rescaleStr = if (rescale == null) "auto" else rescale.toString - // scalastyle:off caselocale - val fmt = fmtStr.toUpperCase - val resampleLower = resampleStr.toLowerCase - // scalastyle:on caselocale - require(AllowedFormats.contains(fmt), s"rst_tilexyz: format must be one of ${AllowedFormats.mkString(", ")}; got '$fmtStr'") - require(AllowedResampling.contains(resampleLower), - s"rst_tilexyz: unsupported resampling '$resampleStr'; allowed: ${AllowedResampling.toSeq.sorted.mkString(", ")}") - require(size > 0 && size <= 4096, s"rst_tilexyz: size must be in (0, 4096]; got $size") - val (_, ds, options) = RasterSerializationUtil.rowToTile(row, dt) - try execute(ds, options, z, x, y, fmt, size, resampleLower, rescaleStr) - finally RasterDriver.releaseDataset(ds) - } - val result = Try(safe()).toOption.flatMap(Option(_)) - result.getOrElse(transparentPng(size)) - } -``` - -**(b) `resolveScale` helper + `execute` overloads.** Add the resolver (uses `BandAccessors`; add `import com.databricks.labs.gbx.rasterx.operations.BandAccessors` and `import org.gdal.gdalconst.gdalconstConstants`): - -```scala - /** Resolve the user `rescale` arg + open source `ds` into a pre-formatted GDAL - * `-scale_ min max 0 255` string for the byte-output translate step, or "" for - * pass-through (uint8 "auto", or "none"). Mirrors the light tier `_resolve_in_range`. - * - * - "none" -> "" (today's full-dtype-range behavior). - * - "min,max" pair -> that pair repeated for every band. - * - "auto": - * * uint8 source -> "" (already display-ready; pass through unchanged). - * * non-uint8 -> per-band whole-dataset (min,max) via BandAccessors.getMinMax - * (ComputeRasterMinMax exact). A constant band (min==max) widened to (min, min+1). - */ - private[web] def resolveScale(ds: Dataset, rescale: String): String = { - val mode = if (rescale == null) "auto" else rescale.trim - // scalastyle:off caselocale - val modeLower = mode.toLowerCase - // scalastyle:on caselocale - val nbands = ds.GetRasterCount - - def fmtBands(pairs: Seq[(Double, Double)]): String = - pairs.zipWithIndex.map { case ((lo, hi), i) => - s"-scale_${i + 1} $lo $hi 0 255" - }.mkString(" ") - - if (modeLower == "none") { - "" - } else if (modeLower == "auto") { - val firstDt = ds.GetRasterBand(1).GetRasterDataType - if (firstDt == gdalconstConstants.GDT_Byte) { - "" // uint8 pass-through - } else { - val pairs = (1 to nbands).map { b => - val (lo, hi) = BandAccessors.getMinMax(ds.GetRasterBand(b)) - if (!(lo < hi)) (lo, lo + 1.0) else (lo, hi) - } - fmtBands(pairs) - } - } else { - // explicit "min,max" pair (e.g. "8000,12000"), repeated per band. - val parts = mode.split(",").map(_.trim) - require(parts.length == 2, s"rst_tilexyz: rescale must be 'auto', 'none', or 'min,max'; got '$rescale'") - val lo = parts(0).toDouble - val hi = parts(1).toDouble - require(lo < hi, s"rst_tilexyz: rescale (min,max) must have min < max; got ($lo, $hi)") - fmtBands(Seq.fill(nbands)((lo, hi))) - } - } -``` - -Add a new `execute` overload that resolves the scale and a back-compat overload that defaults to `"none"` (so the 3 existing tests and any other caller compile unchanged — they keep today's behavior). Replace the existing `execute` (line 111) signature/body so the scale is threaded into the translate options: - -```scala - /** Back-compat: callers that do not specify rescale get today's behavior ("none"). */ - def execute( - ds: Dataset, - options: Map[String, String], - z: Int, x: Int, y: Int, - format: String, size: Int, resampling: String - ): Array[Byte] = execute(ds, options, z, x, y, format, size, resampling, "none") - - def execute( - ds: Dataset, - options: Map[String, String], - z: Int, x: Int, y: Int, - format: String, size: Int, resampling: String, rescale: String - ): Array[Byte] = { - val scaleFlags = resolveScale(ds, rescale) - executeWithScale(ds, options, z, x, y, format, size, resampling, scaleFlags) - } - - /** Render with a PRE-RESOLVED scale string (RST_XYZPyramid resolves once and passes it - * here for every tile so all tiles share one mapping — no seams, stats read once). */ - private[web] def executeWithScale( - ds: Dataset, - options: Map[String, String], - z: Int, x: Int, y: Int, - format: String, size: Int, resampling: String, scaleFlags: String - ): Array[Byte] = { - val (xmin, ymin, xmax, ymax) = TileMath.tileBboxWebMerc(z, x, y) - if (!datasetIntersectsWebMercBbox(ds, xmin, ymin, xmax, ymax)) { - return transparentPng(size) - } - val uuid = java.util.UUID.randomUUID().toString.replace("-", "") - val warpPath = s"/vsimem/tilexyz_warp_$uuid.tif" - val (warpedDs, warpedOpts) = GDALWarp.executeWarp( - warpPath, - Array(ds), - options ++ Map("format" -> "GTiff"), - command = s"gdalwarp -t_srs EPSG:3857 -te $xmin $ymin $xmax $ymax -ts $size $size -r $resampling" - ) - try { - val extension = format.toLowerCase(Locale.ROOT) match { - case "png" => "png" - case "jpeg" => "jpg" - case "webp" => "webp" - case other => throw new IllegalArgumentException(s"rst_tilexyz: unknown format $other") - } - val translatePath = s"/vsimem/tilexyz_out_$uuid.$extension" - // Inject the pre-resolved per-band -scale (empty => no rescale; today's behavior). - val translateOpts = warpedOpts ++ Map("format" -> format, "extension" -> extension) ++ - (if (scaleFlags.isEmpty) Map.empty[String, String] else Map("scale" -> scaleFlags)) - val (resDs, _) = GDALTranslate.executeTranslate( - translatePath, warpedDs, command = "gdal_translate", translateOpts - ) - Try(resDs.FlushCache()) - Try(resDs.delete()) - val bytes = gdal.GetMemFileBuffer(translatePath) - gdal.Unlink(translatePath) - if (bytes == null || bytes.isEmpty) transparentPng(size) else bytes - } finally { - RasterDriver.releaseDataset(warpedDs) - } - } -``` - -> **CRITICAL parity detail:** `resolveScale` reads stats from the ORIGINAL source `ds` (pre-warp), exactly as light reads whole-dataset stats from the open source — NOT from the per-tile warped raster. This guarantees every tile shares one mapping (no seams) AND that heavy and light derive the same `(min,max)` from the same source pixels. `executeWithScale` then applies the resolved scale to each warped tile. - -**(c) Builder arity 4→8.** Replace the builder (line 206): - -```scala - /** Builder: 4 to 8 args (tile, z, x, y, [format, [size, [resampling, [rescale]]]]). */ - override def builder(): FunctionBuilder = (c: Seq[Expression]) => { - c.length match { - case 4 => RST_TileXYZ(c(0), c(1), c(2), c(3), Literal("PNG"), Literal(256), Literal("bilinear"), Literal("auto")) - case 5 => RST_TileXYZ(c(0), c(1), c(2), c(3), c(4), Literal(256), Literal("bilinear"), Literal("auto")) - case 6 => RST_TileXYZ(c(0), c(1), c(2), c(3), c(4), c(5), Literal("bilinear"), Literal("auto")) - case 7 => RST_TileXYZ(c(0), c(1), c(2), c(3), c(4), c(5), c(6), Literal("auto")) - case 8 => RST_TileXYZ(c(0), c(1), c(2), c(3), c(4), c(5), c(6), c(7)) - case n => throw new IllegalArgumentException( - s"gbx_rst_tilexyz takes 4 to 8 arguments (tile, z, x, y, [format, [size, [resampling, [rescale]]]]); got $n" - ) - } - } -``` - -> **DEFAULT NOTE:** The SQL/builder default is `Literal("auto")` (matches light's default and the spec). The back-compat `execute(...)` 8-arg overload defaults to `"none"` ONLY to keep existing internal Scala callers/tests behaviorally unchanged at the call site — but the *public SQL surface* defaults to `"auto"`. These are intentionally different: the public contract is `auto`; the internal back-compat overload preserves legacy direct-`execute` callers. Confirm `RST_XYZPyramid` (Task 3) calls the new rescale-aware path with `"auto"` default, not the back-compat overload. - -**(d) Update the 3 existing `execute` tests.** They call the 8-arg `execute` (no rescale) which now defaults to `"none"` — behavior preserved (PNG magic bytes, transparent fallback). No change needed if the back-compat overload exists. Confirm they still pass in Step 4. - -- [ ] **Step 4: Run the tests to verify they pass** - -Dispatch a Task subagent (Docker): -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.web.WebMercatorTileTest' --log heavy-rescale-tilexyz.log -``` -Expected: all `WebMercatorTileTest` tests PASS (3 existing + Task-1 string tests + 3 new rescale tests). - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/web/RST_TileXYZ.scala src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/WebMercatorTileTest.scala -git commit -m "feat(rasterx): rescale param for RST_TileXYZ (data-aware 8-bit) - -Adds a trailing rescale expression (default auto). resolveScale derives a -per-band -scale string from whole-dataset min/max (BandAccessors.getMinMax, -exact) for non-8-bit sources; uint8 and none pass through unchanged. Stats -read from the source ds (pre-warp) so the mapping is source-global. - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: `RST_XYZPyramid` threads `rescale`, resolves the scale ONCE for the whole pyramid - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/web/RST_XYZPyramid.scala` -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/WebMercatorTileTest.scala` (extend) - -**Current state (real):** -- `case class RST_XYZPyramid(tileExpr, minZExpr, maxZExpr, formatExpr, sizeExpr, resamplingExpr, exprConfExpr = ExpressionConfigExpr())` (line 33-40); `children` (line 53) is `Seq(tileExpr, minZExpr, maxZExpr, formatExpr, sizeExpr, resamplingExpr, exprConfExpr)`; `withNewChildrenInternal` copies `nc(0)..nc(6)` (line 56). -- `doEval` (line 61) reads format/size/resampling (lines 77-79), opens the source via `rowToTile` (line 81), computes the WGS84 bbox once (line 84), guards tile count, then loops calling `RST_TileXYZ.execute(ds, options, zz, xx, yy, format, size, resampling)` per tile (line 115). -- Builder (line 163) arity 3-6 with `Literal` defaults. - -- [ ] **Step 1: Write the failing test** - -Add to `WebMercatorTileTest.scala`. Since the generator's `doEval` needs an `InternalRow` tile + `ExpressionConfig` (Spark-ish), test the pyramid's resolve-once + contrast contract at the `execute`/`resolveScale` level instead (the generator delegates to these), plus a direct guard that the resolved scale is shared. A focused, Spark-free assertion: - -```scala - test("RST_XYZPyramid resolves ONE scale for the source and reuses it per tile") { - val ds = makeUint16Narrow() - try { - // The pyramid resolves the scale once from the source, then renders each tile - // with that same string. Simulate the loop: resolve once, render two tiles. - val scale = RST_TileXYZ.resolveScale(ds, "auto") - scale should not be empty - scale should include("-scale_1") - val t1 = RST_TileXYZ.executeWithScale(ds, Map.empty[String, String], 2, 2, 1, "PNG", 64, "near", scale) - val t2 = RST_TileXYZ.executeWithScale(ds, Map.empty[String, String], 3, 4, 2, "PNG", 64, "near", scale) - // Both tiles produced with the SAME mapping (no seams). Spot-check one is contrast-recovered. - val (lo, hi) = pngBandSpread(t1) - (hi - lo) should be > 50 - t2 should not be null - } finally ds.delete() - } -``` - -> `resolveScale` and `executeWithScale` are `private[web]` (Task 2) so this same-package test can call them. This proves the resolve-once contract that the generator will use; the generator's own Spark-level exercise is covered by the bindings/SQL surface (Task 4) and the cross-tier parity test (Task 6). - -- [ ] **Step 2: Run the test to verify it fails** - -Dispatch a Task subagent (Docker): -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.web.WebMercatorTileTest' --log heavy-rescale-pyramid.log -``` -Expected: PASS already IF Task 2 exposed `resolveScale`/`executeWithScale` — in that case this test is a guard. If `executeWithScale` is not yet reachable (`private`), it FAILs to compile. Adjust visibility to `private[web]` (Task 2 already specifies this) so it passes; then proceed to wire the generator (the generator wiring itself is verified by Task 6's cross-tier test + Task 4's bindings). - -- [ ] **Step 3: Implement — add `rescaleExpr`, resolve once, pass per tile** - -Add `rescaleExpr: Expression` to the case class BEFORE `exprConfExpr` (so the config stays last, matching `RST_TileXYZ`): - -```scala -case class RST_XYZPyramid( - tileExpr: Expression, - minZExpr: Expression, - maxZExpr: Expression, - formatExpr: Expression, - sizeExpr: Expression, - resamplingExpr: Expression, - rescaleExpr: Expression, - exprConfExpr: Expression = ExpressionConfigExpr() -) extends CollectionGenerator with Serializable with CodegenFallback { -``` - -Update `children` (line 53) and `withNewChildrenInternal` (line 55-56): - -```scala - override def children: Seq[Expression] = - Seq(tileExpr, minZExpr, maxZExpr, formatExpr, sizeExpr, resamplingExpr, rescaleExpr, exprConfExpr) - override def withNewChildrenInternal(nc: IndexedSeq[Expression]): Expression = - copy(nc(0), nc(1), nc(2), nc(3), nc(4), nc(5), nc(6), nc(7)) -``` - -In `doEval`, after reading `resampling` (line 79) add: - -```scala - val rescale = Option(rescaleExpr.eval(input)).map(_.asInstanceOf[UTF8String].toString).getOrElse("auto") -``` - -Then after opening the source `ds` (line 81), resolve the scale ONCE (before the tile loop), and change the per-tile call (line 115) to use `executeWithScale` with the shared string: - -```scala - val (_, ds, options) = RasterSerializationUtil.rowToTile(rawTile, rasterType) - try { - // Resolve the 8-bit rescale mapping ONCE from the source (stats read once; every - // tile shares one mapping => no tile-to-tile seams). Mirrors the light tier. - val scaleFlags = RST_TileXYZ.resolveScale(ds, rescale) - - // ... (unchanged: WGS84 bbox, count guard) ... - - // in the emit loop, replace RST_TileXYZ.execute(...) with: - val bytes = RST_TileXYZ.executeWithScale(ds, options, zz, xx, yy, format, size, resampling, scaleFlags) - } -``` - -Update the builder (line 163) arity 3→7: - -```scala - /** Builder: 3 to 7 args (tile, min_z, max_z, [format, [size, [resampling, [rescale]]]]). */ - override def builder(): FunctionBuilder = (c: Seq[Expression]) => { - c.length match { - case 3 => RST_XYZPyramid(c(0), c(1), c(2), Literal("PNG"), Literal(256), Literal("bilinear"), Literal("auto")) - case 4 => RST_XYZPyramid(c(0), c(1), c(2), c(3), Literal(256), Literal("bilinear"), Literal("auto")) - case 5 => RST_XYZPyramid(c(0), c(1), c(2), c(3), c(4), Literal("bilinear"), Literal("auto")) - case 6 => RST_XYZPyramid(c(0), c(1), c(2), c(3), c(4), c(5), Literal("auto")) - case 7 => RST_XYZPyramid(c(0), c(1), c(2), c(3), c(4), c(5), c(6)) - case n => throw new IllegalArgumentException( - s"gbx_rst_xyzpyramid takes 3 to 7 arguments (tile, min_z, max_z, [format, [size, [resampling, [rescale]]]]); got $n" - ) - } - } -``` - -> Note: `RST_XYZPyramid` is a `CollectionGenerator`/`CodegenFallback` (no `rstInvoke` reflection) — the new child just needs to thread through `children`/`withNewChildrenInternal` and `doEval`. No eval-overload reflection concern here (unlike `RST_TileXYZ`). - -- [ ] **Step 4: Run the tests to verify they pass** - -Dispatch a Task subagent (Docker): -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.web.WebMercatorTileTest' --log heavy-rescale-pyramid.log -``` -Expected: all PASS (including the existing pyramid guard tests, which use TileMath directly and are unaffected by the new child). - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/web/RST_XYZPyramid.scala src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/WebMercatorTileTest.scala -git commit -m "feat(rasterx): rescale param for RST_XYZPyramid (resolve once) - -The pyramid generator threads a trailing rescale arg (default auto) and -resolves the per-band -scale string ONCE from the source before the tile -loop, passing it to every executeWithScale call so all tiles share one -8-bit mapping (no seams) and source stats are read a single time. - -Co-authored-by: Isaac" -``` - ---- - -### Task 4: Bindings / parity surface — SQL examples + function-info regeneration - -**Files:** -- Modify: `docs/tests/python/api/rasterx_functions_sql.py` (`rst_tilexyz_sql_example`, `rst_xyzpyramid_sql_example`) -- Regenerate: `src/main/resources/com/databricks/labs/gbx/function-info.json` (via `gbx:docs:function-info`) -- Confirm (no edit expected): `docs/tests-function-info/registered_functions.txt` — `gbx_rst_tilexyz` (line 76) and `gbx_rst_xyzpyramid` (line 78) already present; **no new function**, so no new row. The binding-parity check asserts function NAME presence across Scala `override def name`, Python `functions.py`, and `function-info.json` — all unchanged names. This is an arg-only change; parity should hold without a registered_functions.txt edit. VERIFY in Step 3. -- Confirm: Python `rst_tilexyz`/`rst_xyzpyramid` bindings already carry `rescale` (Phase 1) — no Python change in Phase 2. - -> **Why no registered_functions.txt / Python edit:** binding-parity (`docs/scripts/check-binding-parity.py`) keys on function *names*, not arity. The names are unchanged. The only doc artifact that should change is the SQL example (to demonstrate the new arg) and the regenerated function-info (which is derived from the example). Phase 1 already added `rescale` to the Python bindings. - -- [ ] **Step 1: Update the SQL examples to demonstrate `rescale`** - -In `docs/tests/python/api/rasterx_functions_sql.py`, update `rst_tilexyz_sql_example()` (line 1515) to include the trailing `rescale` arg: - -```python -def rst_tilexyz_sql_example(): - """Render a single web-mercator XYZ tile to PNG bytes""" - return """ --- Render tile (z=10, x=512, y=512) as 256x256 PNG bytes. --- rescale='auto' (default) rescales non-8-bit imagery by whole-dataset min/max --- for display contrast; 'none' keeps the raw full-dtype-range mapping; a --- 'min,max' string sets explicit bounds. -SELECT - path, - gbx_rst_tilexyz(tile, 10, 512, 512, 'PNG', 256, 'bilinear', 'auto') as tile_png -FROM rasters; -""" -``` - -Update `rst_xyzpyramid_sql_example()` (line 1535) docstring/example to mention the optional trailing `rescale`: - -```python -def rst_xyzpyramid_sql_example(): - """Generate one row per (z, x, y) tile across a zoom range""" - return """ --- Explode a raster into per-tile rows across zoom levels 4..6 (PNG, 256px). --- Optional trailing rescale arg (default 'auto') controls 8-bit display contrast: --- gbx_rst_xyzpyramid(tile, 4, 6, 'PNG', 256, 'bilinear', 'auto') -SELECT - path, - t.tile.z as z, - t.tile.x as x, - t.tile.y as y, - t.tile.bytes as png_bytes -FROM rasters -LATERAL VIEW gbx_rst_xyzpyramid(tile, 4, 6) AS t; -""" -``` - -(Leave the `*_sql_example_output` blocks unchanged — the output shape is identical.) - -- [ ] **Step 2: Regenerate function-info.json** - -Dispatch a Task subagent (Docker — runs `generate-function-info.py` + pytest): -``` -gbx:docs:function-info --log heavy-rescale-function-info.log -``` -Expected: `src/main/resources/com/databricks/labs/gbx/function-info.json` regenerated; `gbx_rst_tilexyz` (line ~277) and `gbx_rst_xyzpyramid` (line ~325) examples now show the `rescale` arg. (If the SQL example doc-test executes against sample data, ensure the new arg literal is valid SQL — it is a plain string arg.) - -- [ ] **Step 3: Run the binding-parity gate** - -Dispatch a Task subagent (Docker): -``` -gbx:test:bindings --log heavy-rescale-bindings.log -``` -Expected: PASS. Every name in `registered_functions.txt` still resolves to a Scala `override def name`, a Python binding, and a `function-info.json` key. If it FAILs, read the failure — an arg-only change should not break it; a failure means the regeneration changed a key. Fix upstream (the example), never add a placeholder. - -- [ ] **Step 4: Commit** - -```bash -git add docs/tests/python/api/rasterx_functions_sql.py src/main/resources/com/databricks/labs/gbx/function-info.json -git commit -m "docs(rasterx): document rescale arg in XYZ tiler SQL examples + function-info - -rst_tilexyz/rst_xyzpyramid SQL examples now show the trailing rescale arg -(default auto). Regenerated function-info.json. Binding parity unchanged -(arg-only; function names identical). - -Co-authored-by: Isaac" -``` - ---- - -### Task 5: Scalastyle lint (CI gate) - -**Files:** none (verification gate). - -- [ ] **Step 1: Run scalastyle (Docker — matches CI)** - -Dispatch a Task subagent (Docker): -``` -gbx:lint:scalastyle --log heavy-rescale-scalastyle.log -``` -Expected: clean. Watch for: `caselocale` (wrap `.toUpperCase`/`.toLowerCase` in `// scalastyle:off caselocale` as the existing code does — already applied in the `resolveScale` and `doInvoke` edits), line length, and unused imports (`BandAccessors`, `gdalconstConstants`). - -- [ ] **Step 2: Fix any findings, re-run, confirm clean.** Commit only if files changed: - -```bash -git add -A -git commit -m "style(rasterx): scalastyle fixes for XYZ rescale - -Co-authored-by: Isaac" -``` - ---- - -### Task 6: Cross-tier pixel/value-distribution parity gate (Docker — gates completion) - -**Files:** -- Test (NEW): `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/XYZRescaleParityTest.scala` - -> **SCOPE REFINEMENT (from the Task 2 review, binding for Tasks 6 + 7):** -> The cross-tier parity gate asserts parity for the **`"auto"` path** and the **uint8 pass-through** ONLY. -> **EXCLUDE `"none"` from cross-tier parity** and document it as a known per-tier-raw difference: -> heavy `"none"` (bare `gdal_translate -ot Byte`, no `-scale`) **CLIPS** values > 255 to 255, -> whereas light `"none"` (rio-tiler render with no `in_range`) does NOT clip the same way. This is a -> PRE-EXISTING difference in each tier's raw passthrough, predates this feature, and does NOT touch the -> `"auto"` contract (which is the whole point of the feature). Do NOT change heavy `"none"` to force a -> match — that would break heavy back-compat. Assert heavy `"none"` against HEAVY's real behavior (clip). -> Also: the spec's claim that light `"none"` yields a proportional `[31,46]` mapping is SUSPECT — -> rio-tiler `render()` on single-band uint16 with no `in_range` may 16-bit-passthrough, not proportionally -> squash. Task 7 (cross-language) MUST validate light `"none"` EMPIRICALLY rather than trust that number; -> Task 6 (in-Scala) does not assert light's `"none"` value at all. - -**Goal:** Prove both tiers feed the SAME per-band `(min,max)` and produce equivalent value distributions **on the `"auto"` path** (plus uint8 pass-through). The heavy side is asserted directly here; the cross-tier equivalence is asserted by reproducing light's resolved `(min,max)` and confirming heavy derives the identical pair from the same source pixels, then asserting the decoded tile's value distribution matches the expected linear `[min,max]->[0,255]` mapping within a tolerance. (A live light-vs-heavy byte comparison is NOT possible inside a Scala suite; the value-distribution-vs-expected-mapping assertion is the in-suite proxy for the documented pixel-level parity contract. The light tier's own tests, Phase 1, already assert its side of the same mapping.) - -**Design rationale for an in-Scala parity proxy:** the spec's parity contract is "both tiers derive the same `(min,max)` and feed it identically." Light feeds `in_range=[(min,max)]` to rio-tiler; heavy feeds `-scale min max 0 255`. Both are the SAME linear map `v -> round((v-min)/(max-min)*255)`. So a parity gate that (a) confirms heavy's resolved `(min,max)` equals the source's exact whole-dataset min/max (which is what light computes too), and (b) confirms heavy's decoded output matches that linear map within tolerance, proves the contract without crossing the language boundary in one process. A true cross-language byte/value diff belongs in the Python doc-test/bench harness (note below). - -- [ ] **Step 1: Write the failing parity test** - -```scala -package com.databricks.labs.gbx.rasterx.expressions.web - -import com.databricks.labs.gbx.rasterx.gdal.GDALManager -import org.gdal.gdal.{Dataset, gdal} -import org.gdal.gdalconst.gdalconstConstants -import org.scalatest.BeforeAndAfterAll -import org.scalatest.funsuite.AnyFunSuite -import org.scalatest.matchers.should.Matchers._ - -/** Cross-tier parity gate for the XYZ rescale feature. - * - * Both tiers MUST derive the same per-band (min,max) for a source and apply the same - * linear map v -> (v-min)/(max-min)*255. Light feeds rio-tiler in_range; heavy feeds - * gdal_translate -scale. Parity is pixel/value-distribution-level, NOT byte-level - * (heavy re-encodes a GTiff per tile; PNG encoders differ between GDAL and rio-tiler). - * uint8 pass-through is the one byte-identical-within-tier assertion (auto == none). - */ -class XYZRescaleParityTest extends AnyFunSuite with BeforeAndAfterAll { - - override def beforeAll(): Unit = { - GDALManager.loadSharedObjects(Iterable.empty[String]) - GDALManager.configureGDAL("/tmp", "/tmp", logCPL = true, CPL_DEBUG = "OFF") - gdal.AllRegister() - import com.databricks.labs.gbx.util.NodeFilePathUtil - java.nio.file.Files.createDirectories(NodeFilePathUtil.rootPath) - } - - private def makeUint16Narrow(lo: Int = 8000, hi: Int = 12000): Dataset = { - val drv = gdal.GetDriverByName("MEM") - val ds = drv.Create("/vsimem/parity_u16", 16, 16, 1, gdalconstConstants.GDT_UInt16) - ds.SetGeoTransform(Array(-1.0, 0.125, 0.0, 1.0, 0.0, -0.125)) - val sr = new org.gdal.osr.SpatialReference(); sr.ImportFromEPSG(4326) - ds.SetProjection(sr.ExportToWkt()) - val n = 256 - val ramp = (0 until n).map(i => (lo.toDouble + (hi - lo).toDouble * i / (n - 1))).toArray - ds.GetRasterBand(1).WriteRaster(0, 0, 16, 16, ramp) - ds.GetRasterBand(1).FlushCache() - ds - } - - private def pngBandSpread(bytes: Array[Byte]): (Int, Int) = { - val path = s"/vsimem/parity_decode_${java.util.UUID.randomUUID().toString.replace("-", "")}.png" - gdal.FileFromMemBuffer(path, bytes) - val ds = gdal.Open(path) - try { - val band = ds.GetRasterBand(1) - val buf = Array.ofDim[Byte](ds.GetRasterXSize * ds.GetRasterYSize) - band.ReadRaster(0, 0, ds.GetRasterXSize, ds.GetRasterYSize, buf) - val vals = buf.map(_ & 0xff).filter(_ > 0) - if (vals.isEmpty) (0, 0) else (vals.min, vals.max) - } finally { ds.delete(); gdal.Unlink(path) } - } - - test("heavy auto resolves source whole-dataset min/max (same statistic light uses)") { - val ds = makeUint16Narrow(8000, 12000) - try { - val scale = RST_TileXYZ.resolveScale(ds, "auto") - // Expect "-scale_1 <~8000> <~12000> 0 255". Parse and bound-check. - val parts = scale.trim.split("\\s+") - parts(0) shouldBe "-scale_1" - val lo = parts(1).toDouble - val hi = parts(2).toDouble - lo shouldBe (8000.0 +- 5.0) - hi shouldBe (12000.0 +- 5.0) - parts(3) shouldBe "0" - parts(4) shouldBe "255" - } finally ds.delete() - } - - test("heavy auto recovers contrast (value distribution spans most of 8-bit range)") { - val ds = makeUint16Narrow(8000, 12000) - try { - val png = RST_TileXYZ.execute(ds, Map.empty[String, String], 2, 2, 1, "PNG", 64, "near", "auto") - val (lo, hi) = pngBandSpread(png) - (hi - lo) should be > 100 // NOT crushed into ~[31,46] - } finally ds.delete() - } - - test("heavy none reproduces today's crushed full-dtype-range output") { - val ds = makeUint16Narrow(8000, 12000) - try { - val png = RST_TileXYZ.execute(ds, Map.empty[String, String], 2, 2, 1, "PNG", 64, "near", "none") - val (_, hi) = pngBandSpread(png) - hi should be < 80 // 8000..12000 / 65535 * 255 -> ~[31,46] - } finally ds.delete() - } - - test("heavy explicit pair maps exactly the given bounds") { - val ds = makeUint16Narrow(8000, 12000) - try { - val scale = RST_TileXYZ.resolveScale(ds, "8000,12000") - scale shouldBe "-scale_1 8000.0 12000.0 0 255" - } finally ds.delete() - } - - test("uint8 source: auto == none (byte-identical pass-through within tier)") { - val drv = gdal.GetDriverByName("MEM") - val ds = drv.Create("/vsimem/parity_u8", 16, 16, 1, gdalconstConstants.GDT_Byte) - ds.SetGeoTransform(Array(-1.0, 0.125, 0.0, 1.0, 0.0, -0.125)) - val sr = new org.gdal.osr.SpatialReference(); sr.ImportFromEPSG(4326) - ds.SetProjection(sr.ExportToWkt()) - ds.GetRasterBand(1).WriteRaster(0, 0, 16, 16, Array.fill(256)(100.0)) - ds.GetRasterBand(1).FlushCache() - try { - val auto = RST_TileXYZ.execute(ds, Map.empty[String, String], 2, 2, 1, "PNG", 64, "near", "auto") - val none = RST_TileXYZ.execute(ds, Map.empty[String, String], 2, 2, 1, "PNG", 64, "near", "none") - java.util.Arrays.equals(auto, none) shouldBe true - } finally ds.delete() - } -} -``` - -- [ ] **Step 2: Run to verify it fails / then passes** - -Dispatch a Task subagent (Docker): -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.web.XYZRescaleParityTest' --log heavy-rescale-parity.log -``` -Expected on first run after Tasks 1-3: PASS (the implementation is already in place). The test is a GATE — if any assertion fails, it surfaces a parity/semantics regression. (If `resolveScale` emits a different numeric format than `-scale_1 8000.0 12000.0 0 255`, adjust the explicit-pair assertion to match the actual `Double.toString` rendering — verify the exact string in the log and lock it in.) - -- [ ] **Step 3: Run BOTH web suites together as the final heavy gate** - -Dispatch a Task subagent (Docker): -``` -gbx:test:scala --suites 'com.databricks.labs.gbx.rasterx.expressions.web.WebMercatorTileTest,com.databricks.labs.gbx.rasterx.expressions.web.XYZRescaleParityTest' --log heavy-rescale-final.log -``` -Expected: all PASS. - -- [ ] **Step 4: Commit** - -```bash -git add src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/XYZRescaleParityTest.scala -git commit -m "test(rasterx): cross-tier value-distribution parity gate for XYZ rescale - -Asserts heavy auto resolves the source whole-dataset min/max (the same -statistic the light tier computes), recovers contrast, reproduces today's -crushed output under none, maps explicit pairs exactly, and that uint8 -auto==none byte-identical within tier. Parity is value-distribution-level -per the established convention. - -Co-authored-by: Isaac" -``` - ---- - -### Task 7 (optional, recommended): Cross-language live parity in the Python doc-test/bench harness - -**Files:** (only if a Docker doc-test harness with both tiers + a JAR is available — see memory "Docker volumes for integration tests") -- A Python test that tiles the SAME uint16 narrow-range fixture with BOTH the light `rst_tilexyz` and the registered heavy `gbx_rst_tilexyz` (JAR loaded), decodes both PNGs, and asserts equivalent per-band value distribution within a tolerance (Wasserstein/quantile match), plus uint8 auto==none within each tier. - -> Deferred-but-recommended because the in-Scala Task 6 proves the contract algebraically (same `(min,max)`, same linear map). A true cross-language byte/value diff needs both runtimes in one harness (the bench/doc-test container with a staged JAR). If that harness is not readily available, Task 6 is the accepted gate and this is a follow-up. DECISION NEEDED from the user (see Open Questions). - ---- - -## Self-Review - -- **Spec coverage:** - - `rescale` param + default `auto` on both heavy functions — Tasks 2, 3 (builders default `Literal("auto")`). - - uint8 pass-through (`auto` → no `-scale`) — Task 2 `resolveScale`, asserted Tasks 2 & 6. - - non-8-bit whole-dataset per-band min/max via `BandAccessors.getMinMax` (exact `ComputeRasterMinMax`) — Task 2. - - `"none"` escape hatch (today's full-dtype-range) — Task 2/6 (`hi < 80`). - - explicit `(min,max)` (as `"min,max"` string) — Task 2 `resolveScale`, asserted Task 6. - - resolve-once / no seams — Task 3 (`resolveScale` before loop, `executeWithScale` per tile), asserted Task 3. - - `-scale min max 0 255` injection into the byte-output translate step — Task 1 (`OperatorOptions`) + Task 2 (thread `scale` option). - - GDALManager-guarded registration (stats on open Band, no new registration) — Global Constraints + Task 2 design note. - - Bindings/parity surface — Task 4 (SQL examples + function-info regen + binding-parity gate; registered_functions.txt/Python confirmed unchanged with rationale). - - Heavy tests + lint + bindings in Docker — Tasks 2-6 commands; lint Task 5. - - Cross-tier pixel/value-distribution parity gate — Task 6 (+ optional Task 7 live cross-language). -- **No placeholders:** every step has concrete code or an exact `gbx:*` command. Task 7 is explicitly optional/deferred with a stated decision point, not a stub in the critical path. -- **Type/name consistency:** `resolveScale(ds, rescale: String): String` / `execute(..., rescale: String)` / `executeWithScale(..., scaleFlags: String)` / `rescaleExpr: Expression` / SQL `rescale` (string) / builder `Literal("auto")` — consistent across Tasks 2, 3, 6. `OperatorOptions` key is `"scale"` (the pre-formatted flag string) consistently in Tasks 1, 2, 3. -- **Number of tasks:** 7 (6 required + 1 optional). - -## Heavy-tier-specific RISKS / OPEN QUESTIONS (need user decision before execution) - -1. **`InvokedExpression.rstInvoke` reflection binding (HIGHEST RISK).** `RST_TileXYZ.replacement = rstInvoke(RST_TileXYZ, rasterType)` reflectively maps children (minus the trailing `ExpressionConfigExpr`) to the `evalBinary/evalPath` overload params positionally. Adding `rescaleExpr` REQUIRES the eval overloads to gain a matching trailing `UTF8String rescale` param in the exact child order (…, resampling, **rescale**, conf). If `rstInvoke` matches by arg COUNT and TYPE, the four overloads (Int/Long × Binary/Path) must all be updated symmetrically (plan does this). **I have NOT read `InvokedExpression`/`rstInvoke` — confirm the exact reflection contract (positional vs by-name, how `conf` is appended) before Task 2.** If it binds by a fixed arg list, a mismatch surfaces as a runtime `NoSuchMethodException`/`UNRESOLVED_ROUTINE` only when the SQL function is actually called — so the SQL-level exercise in Task 4/6 is essential, not just the direct-`execute` unit tests. - -2. **`-scale` + `-a_nodata none` + NoData interaction.** The PNG branch strips NoData (`-a_nodata none`) to avoid tRNS. With `-scale`, GDAL scales the raw values including any NoData sentinel (e.g. a uint16 NoData of 0 or 65535 would be included in the source min/max and skew the mapping). Light reads `ds.statistics(b, approx=False)` which RESPECTS the dataset mask/NoData; `BandAccessors.getMinMax` (`ComputeRasterMinMax`) does NOT mask NoData unless the band has a NoData value set. **Potential cross-tier divergence on NoData rasters.** Mitigation options (DECISION NEEDED): (a) accept for now — the Helios/EO fixtures are NoData-free; (b) set band NoData awareness in `getMinMax` (use `ComputeStatistics` which honors NoData, or mask first). The plan's fixtures are NoData-free so tests pass; flag for real-data validation. - -3. **JPEG has no alpha channel.** The transparent-PNG fallback and the PNG `-a_nodata none` path assume RGBA-capable output. JPEG (and the existing default branch) cannot carry alpha; out-of-extent JPEG tiles can't be transparent. This is PRE-EXISTING (heavy already returns a PNG transparent fallback regardless of `fmt` — see `transparentPng`), but adding `-scale` to JPEG/WEBP is new. **DECISION:** scope `-scale` to PNG-only in Task 1 (safest, matches the Helios case), or include JPEG/WEBP (plan includes them with a caution). Recommend PNG-only for Phase 2 unless you want all three. - -4. **`ComputeRasterMinMax` on full-res vs overviews.** `getMinMax` with `force=0` computes EXACT min/max over the full-resolution band (no approximation, no overview sampling) — this matches light's `approx=False` and is the correct parity choice, but on very large sources it reads every pixel once per band. Acceptable per the spec ("one stats read per source, negligible vs tiling"). No overview-sampling divergence risk since both tiers are exact. Confirmed low-risk; noted for completeness. - -5. **Thread-safety of stats computation.** `ComputeRasterMinMax` mutates the band's cached statistics (PAM) but operates on a per-task open `Dataset` (each Spark task deserializes its own tile via `rowToTile`) — no shared/process-global state, no driver registration. Safe under concurrency given the GDALManager guard already gates registration. No new global mutation introduced. Confirmed safe. - -6. **`Double.toString` formatting in `-scale` flags.** `resolveScale` interpolates `Double`s, so `8000.0` renders as `"8000.0"` and `getMinMax` results render with full precision (e.g. `7999.0` or `8000.000000001`). GDAL `-scale` accepts floats, so this is functionally fine, but the explicit-pair test asserts an exact string (`-scale_1 8000.0 12000.0 0 255`). **Lock the exact rendering after the first Docker run** (Task 6 Step 2 note). Not a correctness risk, just a test-brittleness note. - -7. **SQL surface for the explicit pair.** The plan encodes the `(min,max)` pair as a single string literal `'min,max'` to keep the SQL signature one extra `STRING` arg (parallel to `format`/`resampling`). The Python API (Phase 1) accepts a real `(min,max)` tuple via the core path. **DECISION NEEDED:** is a string `'8000,12000'` acceptable as the heavy SQL surface for the explicit pair, or do you want a different encoding (e.g. two extra numeric args `min`, `max`)? The string form keeps arity/registration simple and parity trivial; two numeric args would change the builder arity scheme and the binding signature. Recommend the string form. - -8. **Optional Task 7 (live cross-language parity).** Whether to add the in-harness light-vs-heavy value-distribution test now (needs the bench/doc-test Docker container with a staged JAR per memory "Docker volumes for integration tests") or defer it. The in-Scala Task 6 proves the contract algebraically. **DECISION NEEDED:** run Task 7 now or accept Task 6 as the gate? diff --git a/docs/superpowers/plans/2026-06-29-xyz-tile-data-aware-rescale-light.md b/docs/superpowers/plans/2026-06-29-xyz-tile-data-aware-rescale-light.md deleted file mode 100644 index 9feb75b51..000000000 --- a/docs/superpowers/plans/2026-06-29-xyz-tile-data-aware-rescale-light.md +++ /dev/null @@ -1,760 +0,0 @@ -# XYZ Tile Data-Aware Rescale — Light Tier Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make the light-tier XYZ tilers (`rst_tilexyz`, `rst_xyzpyramid`) recover contrast for non-8-bit imagery by default, via a new `rescale` parameter (default `"auto"`), so the washed-out Helios NB02 raster basemap renders correctly with zero customer config. - -**Architecture:** Add `rescale` resolution to `pyrx/core/xyz.py`: a helper computes the effective per-band `(min,max)` once per source (uint8 → pass-through; non-8-bit → whole-dataset min/max; `"none"` → today's behavior; explicit `(min,max)` → use as-is) and threads it into rio-tiler's `img.render(in_range=...)`. The scalar UDF and pyramid UDTF gain a trailing `rescale` arg with the same default. No Spark needed to test the core logic. - -**Tech Stack:** Python 3.12, rasterio, rio-tiler 9.x (`ImageData.render(in_range=...)`), morecantile, PySpark UDF/UDTF, pytest in `.venv-pyrx`. - -## Global Constraints - -- Cross-language naming: the new parameter is `rescale` in Python (Scala/SQL parity comes in the Phase 2 heavy plan). Keep `_geom` / canonical-name rules — N/A here (no new function). -- No aliases; one canonical parameter name `rescale`. -- This is the **light tier only**. Both tiers currently MATCH (full-dtype-range). Landing light first KNOWINGLY diverges from heavy until the Phase 2 heavy plan reconciles parity. This is an accepted, temporary divergence per the approved sequencing — do NOT attempt heavy/Scala changes in this plan. -- Tiling parity is pixel/value-level, not byte-level (documented convention). -- Run all tests in the project venv: `/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pytest ...` (NOT Docker — these are pure-Python core/UDF tests). -- rio-tiler is imported LAZILY inside functions in `core/xyz.py` (Serverless import constraint) — keep new rio-tiler usage lazy too. -- `rescale` accepted values: the string `"auto"` (default), the string `"none"`, or a 2-element `(min, max)` numeric tuple/list. Anything else raises `ValueError` from `_validate`. - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/core/xyz.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` -- Test: `python/geobrix/test/pyrx/test_core_xyz.py` (extend) - ---- - -### Task 1: `rescale` validation + range-resolution helper in `core/xyz.py` - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/core/xyz.py` -- Test: `python/geobrix/test/pyrx/test_core_xyz.py` - -**Interfaces:** -- Produces: - - `_validate_rescale(rescale) -> "auto" | "none" | (float, float)` — normalizes/validates the rescale arg. `None` is treated as `"auto"`. - - `_resolve_in_range(ds, rescale) -> list[tuple[float,float]] | None` — returns the per-band `in_range` list to pass to rio-tiler, or `None` when no rescale should be applied (uint8 pass-through, or `"none"`). For `"auto"` on a non-uint8 dataset: per-band whole-dataset `(min,max)` via `ds.statistics(b, approx=False)` (rasterio) → `[(min_b, max_b), ...]` for each band. For an explicit `(min,max)` tuple: the SAME pair repeated for every band. Constant band (min==max) → widen to `(min, min+1)` to avoid a zero-width range. - -- [ ] **Step 1: Write the failing tests** - -Add to `python/geobrix/test/pyrx/test_core_xyz.py` (it already imports `xyz`, `np`, `MemoryFile`, `from_origin`, and defines `_make_rgb`, `_open`). Add a uint16 narrow-range fixture and the helper tests: - -```python -def _make_uint16_narrow(width=64, height=64, epsg=4326, lo=8000, hi=12000): - """Single-band uint16 raster with values spread across [lo, hi] (narrow band).""" - transform = from_origin(10.0, 50.0, 0.03125, 0.03125) - profile = dict( - driver="GTiff", width=width, height=height, count=1, dtype="uint16", - crs=f"EPSG:{epsg}", transform=transform, - ) - ramp = np.linspace(lo, hi, width * height).astype("uint16").reshape(height, width) - with MemoryFile() as mf: - with mf.open(**profile) as ds: - ds.write(ramp, 1) - return mf.read() - - -def test_validate_rescale_normalizes(): - assert xyz._validate_rescale(None) == "auto" - assert xyz._validate_rescale("auto") == "auto" - assert xyz._validate_rescale("AUTO") == "auto" - assert xyz._validate_rescale("none") == "none" - assert xyz._validate_rescale((10, 200)) == (10.0, 200.0) - assert xyz._validate_rescale([10, 200]) == (10.0, 200.0) - - -def test_validate_rescale_rejects_bad(): - with pytest.raises(ValueError): - xyz._validate_rescale("stretch") - with pytest.raises(ValueError): - xyz._validate_rescale((1, 2, 3)) - with pytest.raises(ValueError): - xyz._validate_rescale((200, 10)) # min must be < max - - -def test_resolve_in_range_uint8_passthrough_is_none(): - mf, ds = _open(_make_rgb()) # uint8 - try: - assert xyz._resolve_in_range(ds, "auto") is None - finally: - ds.close(); mf.close() - - -def test_resolve_in_range_none_is_none(): - mf, ds = _open(_make_uint16_narrow()) - try: - assert xyz._resolve_in_range(ds, "none") is None - finally: - ds.close(); mf.close() - - -def test_resolve_in_range_auto_uint16_uses_data_minmax(): - mf, ds = _open(_make_uint16_narrow(lo=8000, hi=12000)) - try: - rng = xyz._resolve_in_range(ds, "auto") - assert rng is not None and len(rng) == 1 - lo, hi = rng[0] - # Whole-dataset min/max ~ [8000, 12000], NOT the dtype range [0, 65535]. - assert 7900 <= lo <= 8100 - assert 11900 <= hi <= 12100 - finally: - ds.close(); mf.close() - - -def test_resolve_in_range_explicit_pair_repeats_per_band(): - mf, ds = _open(_make_rgb()) # 3-band uint8 - try: - rng = xyz._resolve_in_range(ds, (10, 200)) - assert rng == [(10.0, 200.0), (10.0, 200.0), (10.0, 200.0)] - finally: - ds.close(); mf.close() -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/test_core_xyz.py -k "rescale or resolve_in_range" -v` -Expected: FAIL with `AttributeError: module ... has no attribute '_validate_rescale'` / `_resolve_in_range`. - -- [ ] **Step 3: Implement the helpers in `core/xyz.py`** - -Add after `_validate` (around line 79), before `render_tile`: - -```python -def _validate_rescale(rescale): - """Normalize/validate the rescale arg. - - Returns the string ``"auto"`` / ``"none"``, or a normalized ``(min, max)`` - float tuple. ``None`` -> ``"auto"``. Raises ValueError on anything else. - """ - if rescale is None: - return "auto" - if isinstance(rescale, str): - r = rescale.lower() - if r in ("auto", "none"): - return r - raise ValueError( - f"rst_tilexyz: rescale must be 'auto', 'none', or a (min, max) pair; " - f"got string '{rescale}'" - ) - # Sequence -> (min, max) - try: - lo, hi = rescale # unpacks exactly two; else ValueError - except (TypeError, ValueError): - raise ValueError( - f"rst_tilexyz: rescale tuple must have exactly two numbers (min, max); " - f"got {rescale!r}" - ) - lo, hi = float(lo), float(hi) - if not (lo < hi): - raise ValueError( - f"rst_tilexyz: rescale (min, max) must have min < max; got ({lo}, {hi})" - ) - return (lo, hi) - - -def _resolve_in_range(ds, rescale): - """Resolve the per-band ``in_range`` for rio-tiler render, or None for no rescale. - - - ``"none"`` -> None (today's full-dtype-range behavior). - - explicit ``(min, max)`` -> that pair repeated for every band. - - ``"auto"``: - * uint8 source -> None (already display-ready; pass through unchanged). - * non-uint8 -> per-band whole-dataset (min, max) via rasterio statistics. - A constant band (min == max) is widened to (min, min + 1). - """ - mode = _validate_rescale(rescale) - if mode == "none": - return None - nbands = ds.count - if isinstance(mode, tuple): - return [mode] * nbands - # mode == "auto" - if np.dtype(ds.dtypes[0]) == np.uint8: - return None - out = [] - for b in range(1, nbands + 1): - stats = ds.statistics(b, approx=False) - lo, hi = float(stats.min), float(stats.max) - if not (lo < hi): - hi = lo + 1.0 - out.append((lo, hi)) - return out -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/test_core_xyz.py -k "rescale or resolve_in_range" -v` -Expected: PASS (6 tests). - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/pyrx/core/xyz.py python/geobrix/test/pyrx/test_core_xyz.py -git commit -m "feat(pyrx): add rescale resolution helpers for XYZ tiling - -_validate_rescale + _resolve_in_range resolve the new rescale arg to a -per-band rio-tiler in_range: uint8 pass-through, non-8-bit auto whole- -dataset min/max, none = today, explicit (min,max). Core logic only. - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: Thread `rescale` through `render_tile` - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/core/xyz.py:82-106` (`render_tile`) -- Test: `python/geobrix/test/pyrx/test_core_xyz.py` - -**Interfaces:** -- Consumes: `_resolve_in_range` (Task 1). -- Produces: `render_tile(ds, z, x, y, fmt="PNG", size=256, resampling="bilinear", rescale="auto")` — same return (image bytes / transparent PNG), now applying `in_range` to `img.render` when resolved. Accepts an optional precomputed `in_range=` keyword (used by Task 3's pyramid to avoid recomputing stats per tile); when `in_range` is passed it OVERRIDES `rescale` resolution. - -- [ ] **Step 1: Write the failing tests** - -Add to `test_core_xyz.py`. These decode the rendered PNG and assert the auto path spans the 8-bit range while `none` stays crushed: - -```python -def _decode_png_rgb(png_bytes): - import io - from PIL import Image - img = Image.open(io.BytesIO(png_bytes)).convert("RGBA") - a = np.asarray(img) - # data pixels = alpha > 0 - mask = a[..., 3] > 0 - rgb = a[..., :3][mask] - return rgb # (N, 3) uint8 of covered pixels - - -def _center_tile_zxy(ds): - # Pick a zoom/tile that intersects the fixture extent (lon~10-12, lat~48-50). - import morecantile - tms = morecantile.tms.get("WebMercatorQuad") - west, south, east, north = xyz._wgs84_bounds(ds) - t = next(iter(tms.tiles(west, south, east, north, [8]))) - return t.z, t.x, t.y - - -def test_render_tile_auto_uint16_spans_full_range(): - mf, ds = _open(_make_uint16_narrow(lo=8000, hi=12000)) - try: - z, x, y = _center_tile_zxy(ds) - png = xyz.render_tile(ds, z, x, y, rescale="auto") - rgb = _decode_png_rgb(png) - assert rgb.size > 0 - # Auto rescale maps [8000,12000] -> ~full 8-bit; expect a wide spread, - # NOT crushed into the ~[31,46] full-dtype-range band. - assert int(rgb.max()) - int(rgb.min()) > 100 - finally: - ds.close(); mf.close() - - -def test_render_tile_none_uint16_stays_crushed(): - mf, ds = _open(_make_uint16_narrow(lo=8000, hi=12000)) - try: - z, x, y = _center_tile_zxy(ds) - png = xyz.render_tile(ds, z, x, y, rescale="none") - rgb = _decode_png_rgb(png) - assert rgb.size > 0 - # Full-dtype-range: 8000..12000 / 65535 * 255 -> ~[31, 46]; crushed. - assert int(rgb.max()) < 80 - finally: - ds.close(); mf.close() - - -def test_render_tile_uint8_auto_matches_none(): - mf, ds = _open(_make_rgb()) - try: - z, x, y = _center_tile_zxy(ds) - auto = xyz.render_tile(ds, z, x, y, rescale="auto") - none = xyz.render_tile(ds, z, x, y, rescale="none") - assert auto == none # uint8 pass-through: byte-identical - finally: - ds.close(); mf.close() -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/test_core_xyz.py -k "render_tile_auto or render_tile_none or uint8_auto" -v` -Expected: FAIL — `render_tile()` got an unexpected keyword argument `rescale`. - -- [ ] **Step 3: Implement — add `rescale`/`in_range` to `render_tile`** - -Replace the current `render_tile` body (lines 82-106) with: - -```python -def render_tile( - ds, z, x, y, fmt="PNG", size=256, resampling="bilinear", - rescale="auto", in_range=None, -) -> bytes: - """Render a single web-mercator (z, x, y) tile from open dataset ``ds``. - - Validates inputs (raises ValueError on bad format/size/resampling/rescale). - Out-of-extent / empty tiles, or any hard render failure, return a transparent - PNG of ``size`` x ``size`` (mirrors heavyweight: PNG regardless of ``fmt``). - - ``rescale`` controls 8-bit encoding contrast (see _resolve_in_range): "auto" - (default) rescales non-8-bit rasters by whole-dataset min/max and passes uint8 - through unchanged; "none" keeps the raw full-dtype-range mapping; a (min, max) - pair sets explicit bounds. ``in_range`` (internal) lets the pyramid path pass a - precomputed per-band range so stats are read once, not per tile; when given it - overrides ``rescale``. - """ - from rio_tiler.errors import TileOutsideBounds # lazy: see module-top note - from rio_tiler.io import Reader - - fmt_u, s, resamp_name = _validate(fmt, size, resampling) - if in_range is None: - in_range = _resolve_in_range(ds, rescale) # may raise ValueError on bad rescale - try: - with Reader(None, dataset=ds) as cog: - img = cog.tile( - int(x), int(y), int(z), tilesize=s, resampling_method=resamp_name - ) - if in_range is not None: - out = img.render(img_format=fmt_u, in_range=in_range) - else: - out = img.render(img_format=fmt_u) - if not out: - return transparent_png(s) - return out - except TileOutsideBounds: - return transparent_png(s) - except Exception: - # Slippy-map servers need a non-null 200 body even on failure. - return transparent_png(s) -``` - -Note: `_resolve_in_range` is called OUTSIDE the try/except so a bad `rescale` value raises ValueError (fail fast) rather than being swallowed into a transparent PNG. - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/test_core_xyz.py -v` -Expected: PASS (all existing xyz tests + the new render_tile tests). If an existing test calls `render_tile` positionally it still works (new args are trailing with defaults). - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/pyrx/core/xyz.py python/geobrix/test/pyrx/test_core_xyz.py -git commit -m "feat(pyrx): apply rescale in_range in render_tile - -render_tile resolves rescale to a per-band in_range and passes it to -rio-tiler's render; auto recovers contrast for non-8-bit rasters, uint8 -passes through byte-identical, none keeps today's behavior. Accepts a -precomputed in_range for the pyramid path (stats read once). - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: Thread `rescale` through `iter_pyramid` / `pyramid` (stats read once) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/core/xyz.py:141-175` (`iter_pyramid`, `pyramid`) -- Test: `python/geobrix/test/pyrx/test_core_xyz.py` - -**Interfaces:** -- Consumes: `_resolve_in_range` (Task 1), `render_tile(..., in_range=)` (Task 2). -- Produces: `iter_pyramid(ds, min_z, max_z, fmt="PNG", size=256, resampling="bilinear", rescale="auto")` and `pyramid(...)` with the same trailing `rescale`. The pyramid resolves `in_range` ONCE before the tile loop and passes it to every `render_tile`, so all tiles share one mapping (no seams) and stats are read once. - -- [ ] **Step 1: Write the failing test** - -```python -def test_pyramid_shares_one_mapping_no_per_tile_stats(monkeypatch): - """All pyramid tiles use ONE resolved in_range; stats resolved once, not per tile.""" - mf, ds = _open(_make_uint16_narrow(lo=8000, hi=12000)) - try: - calls = {"n": 0} - real = xyz._resolve_in_range - - def _spy(dataset, rescale): - calls["n"] += 1 - return real(dataset, rescale) - - monkeypatch.setattr(xyz, "_resolve_in_range", _spy) - tiles = xyz.pyramid(ds, 6, 8, rescale="auto") - assert len(tiles) >= 2 # multiple tiles across the range - # Resolved exactly once for the whole pyramid (not once per tile). - assert calls["n"] == 1 - # And the tiles are contrast-recovered (spot check one non-empty tile). - nonempty = [t for t in tiles if t["bytes"]] - assert nonempty - finally: - ds.close(); mf.close() -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/test_core_xyz.py::test_pyramid_shares_one_mapping_no_per_tile_stats -v` -Expected: FAIL — `pyramid()` got an unexpected keyword argument `rescale` (and/or `_resolve_in_range` called per tile). - -- [ ] **Step 3: Implement — resolve once, pass `in_range` per tile** - -Replace `iter_pyramid` (lines 141-163) and `pyramid` (lines 166-175): - -```python -def iter_pyramid( - ds, min_z, max_z, fmt="PNG", size=256, resampling="bilinear", rescale="auto" -): - """Render every intersecting (z, x, y) tile across [min_z, max_z], streaming. - - Yields ``(z, x, y, bytes)`` tuples one tile at a time — never buffers the full - pyramid (large-fan-out OOM guard). Validates zoom guards, the render args, the - rescale arg, and the tile-count guard BEFORE rendering any tile. The rescale - ``in_range`` is resolved ONCE so every tile shares one 8-bit mapping (no seams) - and source statistics are read a single time. - """ - lo, hi = _validate_zoom_range(min_z, max_z) - # Validate render args up front (so bad format/size fails fast, not per-tile). - _validate(fmt, size, resampling) - in_range = _resolve_in_range(ds, rescale) # once; also validates rescale - west, south, east, north = _wgs84_bounds(ds) - - # Count guard first — never materialize a giant list to count. - total = 0 - for z in range(lo, hi + 1): - total += _zoom_tile_count(west, south, east, north, z) - if total > MAX_TILE_COUNT: - _raise_count(lo, hi) - - for z in range(lo, hi + 1): - for t in _TMS.tiles(west, south, east, north, [z]): - b = render_tile( - ds, t.z, t.x, t.y, fmt, size, resampling, in_range=in_range - ) - yield (t.z, t.x, t.y, b) - - -def pyramid( - ds, min_z, max_z, fmt="PNG", size=256, resampling="bilinear", rescale="auto" -) -> list: - """Render every intersecting (z, x, y) tile across [min_z, max_z]. - - Returns a list of ``{"z","x","y","bytes"}`` dicts. List-materializing wrapper - around :func:`iter_pyramid`. - """ - return [ - {"z": z, "x": x, "y": y, "bytes": b} - for z, x, y, b in iter_pyramid( - ds, min_z, max_z, fmt, size, resampling, rescale - ) - ] -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/test_core_xyz.py -v` -Expected: PASS (all xyz core tests). - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/pyrx/core/xyz.py python/geobrix/test/pyrx/test_core_xyz.py -git commit -m "feat(pyrx): thread rescale through pyramid, resolve in_range once - -iter_pyramid/pyramid take rescale and resolve the per-band in_range a -single time, passing it to every render_tile so all tiles share one 8-bit -mapping (no tile-to-tile seams) and stats are read once. - -Co-authored-by: Isaac" -``` - ---- - -### Task 4: Expose `rescale` on the public UDF/UDTF surface (`functions.py`) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` - - `_tilexyz_udf` (lines 2487-2499) - - `_RstXyzPyramidUDTF.eval` (lines 2521-2538) - - `rst_tilexyz` public binding (lines 2541-2571) - - `rst_xyzpyramid` public docstring (lines 2574-2609) — add `rescale` to Args -- Test: `python/geobrix/test/pyrx/test_core_xyz.py` (UDF-level via the registered function is in test_functions_spark.py; here we add a direct `_tilexyz_udf` call test that needs no SparkSession by passing a plain tile dict) - -**Interfaces:** -- Consumes: `xyz.render_tile(..., rescale=)` (Task 2), `xyz.iter_pyramid(..., rescale=)` (Task 3). -- Produces: - - `_tilexyz_udf(tile, z, x, y, format, size, resampling, rescale=None)` — trailing optional arg; `None` → `"auto"`. - - `_RstXyzPyramidUDTF.eval(self, tile, min_z, max_z, format=None, size=None, resampling=None, rescale=None)`. - - `rst_tilexyz(tile, z, x, y, format="PNG", size=256, resampling="bilinear", rescale="auto")`. - - The SQL registrations pick up the new arity automatically (scalar UDF registered from the bare function; UDTF from the class). - -- [ ] **Step 1: Write the failing test** - -Add to `test_core_xyz.py` (it can build a tile dict from raster bytes the same way `_serde.open_tile` expects — but to avoid Spark/serde coupling, test the `_tilexyz_udf` path through a tiny tile dict). Place this test in `test_core_xyz.py`: - -```python -def test_tilexyz_udf_accepts_rescale_and_recovers_contrast(): - import io - from PIL import Image - from databricks.labs.gbx.pyrx import functions as fns - - raster = _make_uint16_narrow(lo=8000, hi=12000) - mf, ds = _open(raster) - try: - z, x, y = _center_tile_zxy(ds) - finally: - ds.close(); mf.close() - - tile = {"raster": raster} - auto = fns._tilexyz_udf(tile, z, x, y, "PNG", 256, "bilinear", "auto") - none = fns._tilexyz_udf(tile, z, x, y, "PNG", 256, "bilinear", "none") - - def _spread(png): - a = np.asarray(Image.open(io.BytesIO(png)).convert("RGBA")) - rgb = a[..., :3][a[..., 3] > 0] - return 0 if rgb.size == 0 else int(rgb.max()) - int(rgb.min()) - - assert _spread(auto) > 100 # contrast recovered - assert _spread(none) < 80 # today's crushed behavior preserved - - -def test_tilexyz_udf_rescale_defaults_to_auto(): - from databricks.labs.gbx.pyrx import functions as fns - raster = _make_uint16_narrow(lo=8000, hi=12000) - mf, ds = _open(raster) - try: - z, x, y = _center_tile_zxy(ds) - finally: - ds.close(); mf.close() - tile = {"raster": raster} - # rescale omitted -> defaults to auto (contrast recovered) - default = fns._tilexyz_udf(tile, z, x, y, "PNG", 256, "bilinear") - explicit_auto = fns._tilexyz_udf(tile, z, x, y, "PNG", 256, "bilinear", "auto") - assert default == explicit_auto -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/test_core_xyz.py -k "tilexyz_udf" -v` -Expected: FAIL — `_tilexyz_udf()` takes 7 positional arguments but 8 were given. - -- [ ] **Step 3: Implement — add `rescale` to the UDF, UDTF, and public binding** - -In `functions.py`, replace `_tilexyz_udf` (lines 2487-2499): - -```python -def _tilexyz_udf(tile, z, x, y, format, size, resampling, rescale=None): - # Mirror heavyweight: rst_tilexyz NEVER returns null — a null/empty tile or - # any hard failure yields a transparent PNG (slippy-map servers need a 200). - sz = int(size) if size is not None else 256 - if tile is None or tile["raster"] is None: - return xyz.transparent_png(sz) - from databricks.labs.gbx.pyrx import _env - - _env.configure_gdal_env() - fmt = str(format) if format is not None else "PNG" - resamp = str(resampling) if resampling is not None else "bilinear" - rsc = rescale if rescale is not None else "auto" - with _serde.open_tile(bytes(tile["raster"])) as ds: - return xyz.render_tile(ds, int(z), int(x), int(y), fmt, sz, resamp, rescale=rsc) -``` - -Replace `_RstXyzPyramidUDTF.eval` (lines 2521-2538): - -```python - def eval(self, tile, min_z, max_z, format=None, size=None, resampling=None, rescale=None): - # Defaults make format/size/resampling/rescale optional in the SQL UDTF call - # (gbx_rst_xyzpyramid(tile, min_z, max_z)). None maps to PNG/256/bilinear/auto. - if tile is None or tile["raster"] is None: - return - from databricks.labs.gbx.pyrx import _env - - _env.configure_gdal_env() - fmt = str(format) if format is not None else "PNG" - sz = int(size) if size is not None else 256 - resamp = str(resampling) if resampling is not None else "bilinear" - rsc = rescale if rescale is not None else "auto" - with _serde.open_tile(bytes(tile["raster"])) as ds: - for z, x, y, b in xyz.iter_pyramid( - ds, int(min_z), int(max_z), fmt, sz, resamp, rsc - ): - yield (z, x, y, b) -``` - -Replace the `rst_tilexyz` signature (lines 2541-2549) and its tail (lines 2568-2571). New signature: - -```python -def rst_tilexyz( - tile: ColLike, - z: ColLike, - x: ColLike, - y: ColLike, - format: ColLike = "PNG", - size: ColLike = 256, - resampling: ColLike = "bilinear", - rescale: ColLike = "auto", -) -> Column: -``` - -And add to its Args docstring (after the `resampling:` line, before `Returns:`): - -```python - rescale: 8-bit encoding contrast. "auto" (default) rescales non-8-bit - rasters by whole-dataset per-band min/max and passes uint8 - through unchanged; "none" keeps the raw full-dtype-range - mapping; a (min, max) pair sets explicit bounds. -``` - -New tail (replace lines 2568-2571): - -```python - fmt = f.lit(format) if isinstance(format, str) else _col(format) - sz = f.lit(size) if isinstance(size, int) else _col(size) - resamp = f.lit(resampling) if isinstance(resampling, str) else _col(resampling) - rsc = f.lit(rescale) if isinstance(rescale, str) else _col(rescale) - return _tilexyz_udf(_col(tile), _col(z), _col(x), _col(y), fmt, sz, resamp, rsc) -``` - -Add `rescale` to the `rst_xyzpyramid` docstring Args (after `resampling:` near line 2603): - -```python - rescale: 8-bit encoding contrast: "auto" (default), "none", or a - (min, max) pair. See rst_tilexyz. -``` - -Note on the explicit `(min, max)` pair via SQL/Column: the `rescale` Column path passes through `_col(rescale)` when not a string. A tuple literal is not a plain `str`, so `f.lit("auto")`/`f.lit("none")` cover the string cases and a Column expression covers dynamic values; an explicit numeric pair through the Python API is supported by passing it as `rescale=(min,max)` only to the core/UDF path (not the Column wrapper) — document that the Column wrapper supports the string modes and a Column, while the (min,max) tuple is for the direct/core API. (Heavy-tier SQL pair support is a Phase 2 concern.) - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/test_core_xyz.py -v` -Expected: PASS (all, including the two new `_tilexyz_udf` tests). - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/pyrx/functions.py python/geobrix/test/pyrx/test_core_xyz.py -git commit -m "feat(pyrx): expose rescale on rst_tilexyz/rst_xyzpyramid (light) - -Adds trailing rescale arg (default auto) to the scalar UDF, the pyramid -UDTF eval, and the public rst_tilexyz binding + docstrings. SQL picks up -the new arity automatically. - -Co-authored-by: Isaac" -``` - ---- - -### Task 5: Regression-guard the SQL surface + run the full pyrx light suite - -**Files:** -- Test: `python/geobrix/test/pyrx/test_functions_spark.py` (extend, if a local SparkSession is already used there) OR `test_sql_registration.py` -- No source changes expected (this task is a verification gate; only add a test if a gap exists). - -**Interfaces:** -- Consumes: the registered `gbx_rst_tilexyz` / `gbx_rst_xyzpyramid` (Task 4). - -- [ ] **Step 1: Inspect how the existing Spark tests invoke the tilers** - -Run: `/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/test_functions_spark.py -k "tilexyz or xyzpyramid" -v` -Expected: existing tests PASS (the new trailing arg is optional, so existing calls are unaffected). If there are NO such tests, note it and skip adding Spark-level ones here (covered by the core + UDF tests); the SQL arity is exercised by registration. - -- [ ] **Step 2: If a Spark tiler test exists, add a `rescale` regression** - -Only if `test_functions_spark.py` already builds a local SparkSession + a tile column, add (mirroring its existing fixture style — adapt names to that file's helpers): - -```python -def test_sql_rst_tilexyz_accepts_rescale(spark, sample_uint16_tile_df): - # gbx_rst_tilexyz(tile, z, x, y, format, size, resampling, rescale) - from databricks.labs.gbx.pyrx import functions as fns - fns.register(spark) - row = ( - sample_uint16_tile_df - .selectExpr("gbx_rst_tilexyz(tile, 8, 41, 90, 'PNG', 256, 'bilinear', 'auto') AS png") - .collect()[0] - ) - assert row["png"] is not None and len(row["png"]) > 0 -``` - -(If the file has no such fixture, DO NOT invent one — the core/UDF tests already prove the behavior. Skip to Step 3.) - -- [ ] **Step 3: Run the full pyrx light suite** - -Run: `/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m pytest python/geobrix/test/pyrx/ -v --tb=short` -Expected: PASS (no regressions across the pyrx light tier). - -- [ ] **Step 4: Lint check (CI gate, host)** - -Run: `bash scripts/commands/gbx-lint:python.sh --check 2>/dev/null || bash scripts/commands/gbx-lint-python.sh --check` -Expected: clean. If it reports formatting, run the `--fix` variant ONLY for these files, re-check, and confirm. (Note: per project memory, the host black may differ from CI; the Docker `--check` is authoritative, but for a pure-Python diff the host check is a reasonable pre-push gate.) - -- [ ] **Step 5: Commit (only if Step 2/4 changed files)** - -```bash -git add -A -git commit -m "test(pyrx): regression-guard rescale on the SQL tiler surface - -Co-authored-by: Isaac" -``` - ---- - -### Task 6: Rebuild + restage the light wheel; re-run Helios NB02 to visually confirm - -**Files:** none (operational verification). - -**Interfaces:** Consumes the registered light tier with `rescale="auto"` default. - -- [ ] **Step 1: Build the wheel (pure-Python; no JAR rebuild)** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -rm -rf python/geobrix/dist -/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python -m build --no-isolation /Users/mjohns/IdeaProjects/geobrix/python/geobrix -ls -la python/geobrix/dist/geobrix-*.whl -``` -Expected: `geobrix-0.4.0-py3-none-any.whl` produced. - -- [ ] **Step 2: Upload to the canonical sample-data Volume wheel path** - -```bash -/Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/python - <<'PY' -import glob -from databricks.sdk import WorkspaceClient -whl = sorted(glob.glob("/Users/mjohns/IdeaProjects/geobrix/python/geobrix/dist/geobrix-*.whl"))[-1] -dest = "/Volumes/geospatial_docs/geobrix/sample-data/geobrix-0.4.0-py3-none-any.whl" -WorkspaceClient(profile="oauth-fe").files.upload_from( - file_path=dest, source_path=whl, overwrite=True, use_parallel=False -) -print("uploaded", whl, "->", dest) -PY -``` -Expected: `uploaded ... -> /Volumes/.../geobrix-0.4.0-py3-none-any.whl`. - -- [ ] **Step 3: Re-run Helios NB02 on Serverless** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -bash scripts/commands/gbx-test-notebooks-serverless.sh \ - --notebook "notebooks/examples/helios/02. Visual Basemap (XYZ).ipynb" \ - --ws-dir "/Users/mjohns@databricks.com/GeoBrix/helios" \ - --extra-deps rich \ - --log notebooks-serverless-rescale.log -``` -Expected: `RunResultState.SUCCESS`; capture the `run_page_url`. - -- [ ] **Step 4: Report the run_page_url for visual confirmation** - -Surface the `run_page_url` so the user opens section 7 and confirms the basemap now has full contrast (no longer washed-out). This is the customer-facing definition of done for Phase 1. - -- [ ] **Step 5: No commit (operational task).** Hand off to the user for live notebook testing. - ---- - -## Phase 2 (separate plan — heavy/classic tier) - -NOT in this plan. After the light tier is confirmed, a follow-up plan will: -- Mirror `rescale` in `RST_TileXYZ.scala` / `RST_XYZPyramid.scala` (compute band stats once, inject `gdal_translate -scale min max 0 255`). -- Add the `rescale` arg to `OperatorOptions` PNG/JPEG/WEBP branches. -- Update bindings/parity: `registered_functions.txt`, `function-info.json` (via `gbx:docs:function-info`), the `*_sql_example()` in `docs/tests/python/api/rasterx_functions_sql.py`, and run `gbx:test:bindings`. -- Add the Docker cross-tier pixel-parity test (same uint16 narrow-range fixture; assert equivalent value distribution for "auto", byte-identical uint8 pass-through within each tier). -- Reconcile the temporary light-vs-heavy divergence introduced by this plan. - -## Self-Review - -- **Spec coverage:** `rescale` param + default auto (Tasks 1-4); uint8 pass-through (Task 1/2); non-8-bit whole-dataset min/max (Task 1); "none" escape hatch (Task 1/2); explicit (min,max) (Task 1/4); no seams via resolve-once (Task 3); light-tier TDD locally (Tasks 1-5); wheel restage + NB02 visual confirm (Task 6). Heavy tier + Docker parity test + bindings explicitly deferred to Phase 2 (matches approved light-first sequencing). Covered. -- **Placeholder scan:** none — all steps have concrete code/commands. Task 5 Step 2 is conditional by design (don't invent a fixture) and is explicitly gated, not a placeholder. -- **Type consistency:** `_validate_rescale` / `_resolve_in_range` / `render_tile(..., rescale, in_range)` / `iter_pyramid(..., rescale)` / `pyramid(..., rescale)` / `_tilexyz_udf(..., rescale=None)` / `_RstXyzPyramidUDTF.eval(..., rescale=None)` / `rst_tilexyz(..., rescale="auto")` are consistent across tasks. `in_range` is the precomputed per-band list `[(min,max),...]`; `rescale` is the user-facing knob. Consistent. diff --git a/docs/superpowers/plans/2026-06-30-dem-downloader-3dep.md b/docs/superpowers/plans/2026-06-30-dem-downloader-3dep.md deleted file mode 100644 index 268b92d91..000000000 --- a/docs/superpowers/plans/2026-06-30-dem-downloader-3dep.md +++ /dev/null @@ -1,620 +0,0 @@ -# DemDownloader (3DEP DEM Downloader) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a product `DemDownloader` (+ one-shot `download_dem_aoi`) to `gbx.sample` that stages USGS 3DEP elevation for an AOI via Planetary Computer STAC, mirroring `NaipDownloader` with resolution (gsd) selection instead of year (vintage) selection. - -**Architecture:** New module `sample/dem.py` mirroring `sample/naip.py` — a driver-side `discover` (metadata only) + a distributed `download` that wraps `StacClient.download`'s windowed, fanned-out fetch, + a `read` (raster_gbx). Selection axis is `gsd`: `download(resolution="finest")` picks the minimum gsd (10 m over 30 m); an int picks that exact gsd; graceful no-op when a source lacks `gsd`. - -**Tech Stack:** Python 3.12, PySpark, `StacClient` (existing), pytest. Tests on the host venv. - -## Global Constraints - -- **Mirror `NaipDownloader`** (`python/geobrix/src/databricks/labs/gbx/sample/naip.py`) closely — same method shapes, docstrings-style, injection seam. -- **Serverless-safe:** no `spark.conf.set` / `_jvm` / `sparkContext` / `.rdd` / `.cache()` / `.persist()`. Parallelism only from `StacClient.download`'s `spark.range` fan-out. -- **Online-only:** requires `pystac-client` + `planetary-computer`; no offline/synthetic fallback. `_stac_client` param is the offline-test injection seam. -- **No new SQL function**; do NOT touch `function-info.json` or `docs/tests-function-info/registered_functions.txt`. -- 3DEP specifics: collection `"3dep-seamless"`, asset `"data"` (both `__init__` params, overridable); gsd from `item_properties["gsd"]`. -- **Refinement vs spec:** `discover(resolution=None)` = show all gsd tiers (int filters) — mirrors `NaipDownloader.discover(year=None)`; `download(resolution="finest")` = pick min gsd. (The spec wrote `discover(resolution="finest")`; `None` is used for the "show all" default to avoid "finest" meaning two things. Same intent.) -- **Tests on the host venv:** `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest python/geobrix/test/sample/test_dem.py -v`. -- Local commits only (pushes held this session). Commit messages end with `Co-authored-by: Isaac`. - -**Follow-ups (NOT in this plan):** wiring NB-03 cell-5 to `download_dem_aoi`; the docs "3DEP Downloader (DEM)" page; the live Serverless smoke test. The deliverable here is `sample/dem.py` + its offline mock test suite. - ---- - -### Task 1: `DemDownloader` module — discover / read / export - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/sample/dem.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/__init__.py` -- Test: `python/geobrix/test/sample/test_dem.py` - -**Interfaces:** -- Produces: - - `_bbox_to_geojson_polygon(bbox) -> str` — closed GeoJSON Polygon ring from `(minx,miny,maxx,maxy)`. - - `class DemDownloader(catalog=PLANETARY_COMPUTER, sign="planetary_computer", collection="3dep-seamless", asset="data", _stac_client=None)` - - `DemDownloader.discover(bbox, resolution=None, spark=None) -> DataFrame` — columns `item_id, gsd, item_bbox, href`; `resolution=None` returns all tiers, `resolution=` filters `gsd==int`. - - `DemDownloader.read(out_dir, spark=None) -> DataFrame` — `raster_gbx` reader → `tile` column. - - `download_dem_aoi(spark, bbox, out_dir, resolution="finest", max_mpp=None, **kw)` — one-shot (delegates to `download`, delivered in Task 2). -- Consumes: `databricks.labs.gbx.stac.StacClient`; the `raster_gbx` DataSource (registered by `ds.register`). - -- [ ] **Step 1: Write the failing tests** - -Create `python/geobrix/test/sample/test_dem.py`: - -```python -"""Tests for DemDownloader — no network; StacClient is fully stubbed. - -Mirrors test_naip.py's driver-path stubbing. Selection axis is gsd (resolution), -not year: 3dep-seamless offers the same area at 10 m and 30 m. - -Coverage: - D1 — discover() returns [item_id, gsd, item_bbox, href], only the "data" asset - D2 — discover(resolution=10) filters to that gsd - D3 — discover() deduplicates overlapping items - DL1 — download(resolution="finest") picks the minimum gsd (10 m) - DL2 — download(resolution=30) selects that exact gsd - DL3 — download() passes bbox / bbox_crs / max_mpp / partitions to StacClient.download - DL4 — download() returned DataFrame carries StacClient.download schema - DL5 — download() empty result carries the canonical schema (no href, Boolean valid) - DL6 — download(resolution="finest") with no gsd property keeps all items (graceful) - E1 — export: DemDownloader + download_dem_aoi importable from sample -""" - -from __future__ import annotations - -import pytest - -pyspark = pytest.importorskip("pyspark") - -from pyspark.sql import SparkSession # noqa: E402 -from pyspark.sql import functions as F # noqa: E402 -from pyspark.sql.types import ( # noqa: E402 - ArrayType, BooleanType, DoubleType, LongType, MapType, StringType, - StructField, StructType, -) - -from databricks.labs.gbx.sample.dem import DemDownloader, _bbox_to_geojson_polygon # noqa: E402 - - -@pytest.fixture(scope="module") -def spark(): - s = ( - SparkSession.builder.master("local[2]") - .appName("dem-test") - .config("spark.sql.shuffle.partitions", "4") - .getOrCreate() - ) - yield s - s.stop() - - -# Two gsd tiers (10 m + 30 m), two AOI items each; asset "data" + a non-data asset. -_FAKE_ITEMS = [ - {"id": "dep_10_a", "bbox": [-122.45, 37.74, -122.40, 37.78], - "properties": {"datetime": "2021-01-01T00:00:00Z", "gsd": "10"}, - "assets": {"data": {"href": "file:///fake/dep_10_a.tif"}, - "rendered_preview": {"href": "file:///fake/prev_a.png"}}}, - {"id": "dep_10_b", "bbox": [-122.50, 37.70, -122.42, 37.76], - "properties": {"datetime": "2021-01-01T00:00:00Z", "gsd": "10"}, - "assets": {"data": {"href": "file:///fake/dep_10_b.tif"}}}, - {"id": "dep_30_a", "bbox": [-122.45, 37.74, -122.40, 37.78], - "properties": {"datetime": "2019-01-01T00:00:00Z", "gsd": "30"}, - "assets": {"data": {"href": "file:///fake/dep_30_a.tif"}}}, - {"id": "dep_30_b", "bbox": [-122.50, 37.70, -122.42, 37.76], - "properties": {"datetime": "2019-01-01T00:00:00Z", "gsd": "30"}, - "assets": {"data": {"href": "file:///fake/dep_30_b.tif"}}}, -] - -_FAKE_ITEMS_NO_GSD = [ - {"id": "dep_nogsd", "bbox": [-122.45, 37.74, -122.40, 37.78], - "properties": {"datetime": "2020-01-01T00:00:00Z"}, - "assets": {"data": {"href": "file:///fake/dep_nogsd.tif"}}}, -] - - -class _MockStacClient: - """Captures search + download calls; returns controlled DataFrames.""" - - def __init__(self, search_df, download_df=None): - self._search_df = search_df - self._download_df = download_df - self.download_calls = [] - - def search(self, df, geojson_col, collections, datetime, partitions=512): - return self._search_df - - def download(self, df, out_dir, **kwargs): - rows = df.select("item_id", "asset_name", "href").collect() - self.download_calls.append( - {"item_ids": sorted(r["item_id"] for r in rows), "out_dir": out_dir, **kwargs} - ) - spark = SparkSession.getActiveSession() - schema = StructType([ - StructField("item_id", StringType()), - StructField("asset_name", StringType()), - StructField("out_file_path", StringType()), - StructField("out_file_sz", LongType()), - StructField("is_out_file_valid", BooleanType()), - ]) - if self._download_df is not None: - return self._download_df - _rows = [(iid, "data", f"/fake/out/{iid}.tif", 1000, True) - for iid in self.download_calls[-1]["item_ids"]] - return spark.createDataFrame(_rows, schema).withColumn("last_update", F.current_timestamp()) - - -def _make_search_df(spark, items): - rows = [] - for d in items: - props = {k: str(v) for k, v in d["properties"].items()} - bbox = list(d["bbox"]) - date = d["properties"].get("datetime", "")[:10] - for asset_name, asset in d["assets"].items(): - rows.append((d["id"], date, bbox, props, asset_name, asset["href"])) - schema = StructType([ - StructField("item_id", StringType()), - StructField("date", StringType()), - StructField("item_bbox", ArrayType(DoubleType())), - StructField("item_properties", MapType(StringType(), StringType())), - StructField("asset_name", StringType()), - StructField("href", StringType()), - ]) - return spark.createDataFrame(rows if rows else [], schema) - - -def _make_download_df(spark, item_ids): - rows = [(iid, "data", f"/fake/out/{iid}.tif", 12345, True) for iid in item_ids] - schema = StructType([ - StructField("item_id", StringType()), - StructField("asset_name", StringType()), - StructField("out_file_path", StringType()), - StructField("out_file_sz", LongType()), - StructField("is_out_file_valid", BooleanType()), - ]) - return spark.createDataFrame(rows, schema).withColumn("last_update", F.current_timestamp()) - - -# --- D1 --- -def test_discover_returns_expected_columns_and_rows(spark): - mock = _MockStacClient(_make_search_df(spark, _FAKE_ITEMS)) - dd = DemDownloader(_stac_client=mock) - df = dd.discover((-122.52, 37.70, -122.36, 37.83), spark=spark) - assert set(df.columns) == {"item_id", "gsd", "item_bbox", "href"}, df.columns - rows = df.collect() - assert {r["item_id"] for r in rows} == {"dep_10_a", "dep_10_b", "dep_30_a", "dep_30_b"} - # only "data" asset hrefs (rendered_preview filtered out) - assert all("prev" not in r["href"] for r in rows) - - -# --- D2 --- -def test_discover_resolution_filter(spark): - mock = _MockStacClient(_make_search_df(spark, _FAKE_ITEMS)) - dd = DemDownloader(_stac_client=mock) - rows = dd.discover((-122.52, 37.70, -122.36, 37.83), resolution=10, spark=spark).collect() - assert len(rows) == 2 and all(r["gsd"] == 10 for r in rows) - assert {r["item_id"] for r in rows} == {"dep_10_a", "dep_10_b"} - - -# --- D3 --- -def test_discover_deduplicates_items(spark): - mock = _MockStacClient(_make_search_df(spark, _FAKE_ITEMS + _FAKE_ITEMS)) - dd = DemDownloader(_stac_client=mock) - ids = [r["item_id"] for r in dd.discover((-122.52, 37.70, -122.36, 37.83), spark=spark).collect()] - assert len(ids) == len(set(ids)), ids - - -# --- E1 --- -def test_export_dem_downloader_from_sample_init(): - from databricks.labs.gbx.sample import DemDownloader as DD - from databricks.labs.gbx.sample import download_dem_aoi as dda - assert DD is DemDownloader - assert callable(dda) - - -def test_bbox_to_geojson_polygon_shape(): - import json - d = json.loads(_bbox_to_geojson_polygon((-1.0, 2.0, 3.0, 4.0))) - assert d["type"] == "Polygon" - ring = d["coordinates"][0] - assert len(ring) == 5 and ring[0] == ring[-1] -``` - -- [ ] **Step 2: Run the tests to verify they fail** - -Run: `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest python/geobrix/test/sample/test_dem.py -v` -Expected: FAIL — `ModuleNotFoundError: ... sample.dem` (and `download_dem_aoi` import error in E1; that test also fails until Task 2, run it with `-k "discover or export or bbox"` for now — the download tests come in Task 2). - -- [ ] **Step 3: Create the module (discover / read / scaffold)** - -Create `python/geobrix/src/databricks/labs/gbx/sample/dem.py`: - -```python -"""DemDownloader — AOI-driven USGS 3DEP elevation staging via Planetary Computer STAC. - -Mirrors NaipDownloader's shape: a driver-side discovery step (metadata-only), then -DISTRIBUTED asset I/O via StacClient.download(). The selection axis is resolution (gsd): -``download(resolution="finest")`` picks the minimum gsd (10 m over 30 m); an int picks -that exact gsd. Signing is handled by StacClient (``planetary_computer`` modifier). - -ONLINE-ONLY — no offline fallback. Requires pystac-client and planetary-computer. - -Injection seam (offline tests): pass ``_stac_client`` (a pre-built or mock StacClient) -to bypass catalog network access. - -Serverless-safe: no spark.conf.set, _jvm, .rdd, cache, or persist. Parallelism via -StacClient.download()'s spark.range fan-out. -""" - -from __future__ import annotations - -from typing import TYPE_CHECKING, Optional, Sequence, Union - -if TYPE_CHECKING: - from pyspark.sql import DataFrame - -PLANETARY_COMPUTER = "https://planetarycomputer.microsoft.com/api/stac/v1" -DEM_COLLECTION = "3dep-seamless" -# 3DEP-seamless exposes its DEM raster under the "data" asset (not "image"). -_DEM_ASSET = "data" -# 3dep-seamless is a mosaic; a wide datetime bracket avoids guessing vintages. -_DEM_DATETIME = "2000-01-01/2030-01-01" - - -def _bbox_to_geojson_polygon(bbox: Sequence[float]) -> str: - """Convert (minx, miny, maxx, maxy) to a GeoJSON Polygon string.""" - import json - - minx, miny, maxx, maxy = bbox - coords = [ - [minx, miny], [maxx, miny], [maxx, maxy], [minx, maxy], [minx, miny], - ] - return json.dumps({"type": "Polygon", "coordinates": [coords]}) - - -class DemDownloader: - """Distributed, AOI-driven 3DEP DEM downloader via Planetary Computer STAC. - - Discovery (``discover``) is driver-side, metadata-only. Download (``download``) - fans out via StacClient.download() — Serverless-safe. Selection is by resolution - (gsd): ``"finest"`` picks the minimum gsd; an int picks that exact gsd. - - Parameters - ---------- - catalog: STAC API root URL (default: Planetary Computer). - sign: Signing modifier for StacClient (``"planetary_computer"``). - collection: STAC collection ID (default ``"3dep-seamless"``). - asset: Asset name to download (default ``"data"``). - _stac_client: Injectable StacClient (or mock) for offline unit tests. - """ - - def __init__( - self, - catalog: str = PLANETARY_COMPUTER, - sign: str = "planetary_computer", - collection: str = DEM_COLLECTION, - asset: str = _DEM_ASSET, - _stac_client=None, - ): - self.catalog = catalog - self.sign = sign - self.collection = collection - self.asset = asset - self._stac_client = _stac_client - - def _get_stac_client(self): - if self._stac_client is not None: - return self._stac_client - from databricks.labs.gbx.stac import StacClient - - return StacClient(catalog=self.catalog, sign=self.sign) - - def _aoi_dataframe(self, bbox: Sequence[float], spark=None) -> "DataFrame": - from pyspark.sql import SparkSession - - spark = spark or SparkSession.getActiveSession() - return spark.createDataFrame( - [(_bbox_to_geojson_polygon(bbox),)], ["geojson"] - ) - - def _gsd_col(self): - """Column expr: item_properties['gsd'] as an int (nullable).""" - from pyspark.sql import functions as F - from pyspark.sql.types import IntegerType - - return F.col("item_properties")["gsd"].cast(IntegerType()) - - def discover( - self, bbox: Sequence[float], resolution: Optional[int] = None, spark=None - ) -> "DataFrame": - """Search Planetary Computer for 3DEP items intersecting bbox. - - Returns one row per distinct DEM ``data`` asset: item_id (str), gsd (int), - item_bbox (array), href (str). ``resolution=None`` returns all gsd - tiers; an int keeps only items whose gsd equals it. - """ - from pyspark.sql import SparkSession - from pyspark.sql import functions as F - - spark = spark or SparkSession.getActiveSession() - client = self._get_stac_client() - aoi_df = self._aoi_dataframe(bbox, spark) - - raw = client.search( - aoi_df, geojson_col="geojson", - collections=[self.collection], datetime=_DEM_DATETIME, - ) - img = raw.filter(F.col("asset_name") == self.asset) - out = ( - img.withColumn("gsd", self._gsd_col()) - .select("item_id", "gsd", "item_bbox", "href") - .distinct() - ) - if resolution is not None: - out = out.filter(F.col("gsd") == int(resolution)) - return out - - def read(self, out_dir: str, spark=None) -> "DataFrame": - """Load downloaded DEM GeoTIFFs from out_dir into a raster tile DataFrame. - - Mirrors NaipDownloader.read(): the ``raster_gbx`` reader, filtered to ``*.tif``, - repartitioned by source path (Serverless-safe, column-hash repartition). - """ - from pyspark.sql import SparkSession - from pyspark.sql import functions as F - - spark = spark or SparkSession.getActiveSession() - return ( - spark.read.format("raster_gbx") - .option("filterRegex", r".*\.tif$") - .load(out_dir) - .repartition(64, F.col("source")) - .select("tile") - ) -``` - -Then add to `python/geobrix/src/databricks/labs/gbx/sample/__init__.py` (next to the NAIP import, and in `__all__`): - -```python -from databricks.labs.gbx.sample.dem import DemDownloader, download_dem_aoi -``` - -(Add `"DemDownloader"` and `"download_dem_aoi"` to the `__all__` list.) - -Note: `download_dem_aoi` is delivered in Task 2 but the import + `__all__` entry are added now; the export test (E1) is in Task 2's run. For Task 1, run the discover/bbox tests only. - -- [ ] **Step 4: Run the Task-1 tests to verify they pass** - -Run: `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest python/geobrix/test/sample/test_dem.py -v -k "discover or bbox"` -Expected: PASS (D1, D2, D3, helper). (The `__init__` import line references `download_dem_aoi`, which Task 2 adds — so add a temporary stub OR sequence Task 2 immediately; see Step 5.) - -- [ ] **Step 5: Add a minimal `download_dem_aoi` stub + `download` so the module imports, then commit** - -To keep the module importable at the Task-1 boundary, add the `download` method and `download_dem_aoi` now (they are Task 2's focus, but a module that imports a missing name breaks Task 1's own tests). Add the full `download` + `download_dem_aoi` from Task 2 Step 3. Then: - -```bash -chmod -R u+rwX .git/objects 2>/dev/null || true -git add python/geobrix/src/databricks/labs/gbx/sample/dem.py python/geobrix/src/databricks/labs/gbx/sample/__init__.py python/geobrix/test/sample/test_dem.py -git commit -m "feat(sample): DemDownloader — 3DEP DEM discover/read + module scaffold - -AOI-driven 3DEP elevation downloader mirroring NaipDownloader; discover() returns -gsd tiers, read() via raster_gbx. Selection axis is resolution (gsd), not year. - -Co-authored-by: Isaac" -``` - -(Task 1 and Task 2 both touch `dem.py`; because a Python module can't import a name that doesn't exist, `download`/`download_dem_aoi` land in the same commit as the scaffold. The reviewer treats Task 2's `download` logic + its tests as the second gate.) - ---- - -### Task 2: `download` (gsd selection) + `download_dem_aoi` one-shot - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/dem.py` (add `download` + `download_dem_aoi`) -- Test: `python/geobrix/test/sample/test_dem.py` (add DL1–DL6 + E1) - -**Interfaces:** -- Consumes: `DemDownloader` (Task 1); `StacClient.download(df, out_dir, bbox=, bbox_crs=, max_mpp=, partitions=)`. -- Produces: - - `DemDownloader.download(bbox, out_dir, resolution="finest", bbox_crs="EPSG:4326", max_mpp=None, partitions=None, spark=None) -> DataFrame` — returns `StacClient.download`'s result. - - `download_dem_aoi(spark, bbox, out_dir, resolution="finest", max_mpp=None, **kw) -> DataFrame`. - -- [ ] **Step 1: Write the failing download tests** - -Append to `python/geobrix/test/sample/test_dem.py`: - -```python -# --- DL1: finest picks the minimum gsd (10 m) --- -def test_download_picks_finest_gsd(spark, tmp_path): - mock = _MockStacClient(_make_search_df(spark, _FAKE_ITEMS), - _make_download_df(spark, ["dep_10_a", "dep_10_b"])) - dd = DemDownloader(_stac_client=mock) - result = dd.download((-122.52, 37.70, -122.36, 37.83), str(tmp_path / "o"), - resolution="finest", spark=spark) - assert len(mock.download_calls) == 1 - assert set(mock.download_calls[0]["item_ids"]) == {"dep_10_a", "dep_10_b"} - assert {"item_id", "asset_name", "out_file_path", "last_update"} <= {f.name for f in result.schema} - - -# --- DL2: resolution=int selects that gsd --- -def test_download_resolution_int_selects_tier(spark, tmp_path): - mock = _MockStacClient(_make_search_df(spark, _FAKE_ITEMS), - _make_download_df(spark, ["dep_30_a", "dep_30_b"])) - dd = DemDownloader(_stac_client=mock) - dd.download((-122.52, 37.70, -122.36, 37.83), str(tmp_path / "o"), resolution=30, spark=spark) - assert set(mock.download_calls[0]["item_ids"]) == {"dep_30_a", "dep_30_b"} - - -# --- DL3: kwargs forwarded --- -def test_download_passes_kwargs(spark, tmp_path): - mock = _MockStacClient(_make_search_df(spark, _FAKE_ITEMS), - _make_download_df(spark, ["dep_10_a"])) - bbox = (-122.52, 37.70, -122.36, 37.83) - dd = DemDownloader(_stac_client=mock) - dd.download(bbox, str(tmp_path / "o"), resolution="finest", - bbox_crs="EPSG:4326", max_mpp=5.0, partitions=8, spark=spark) - call = mock.download_calls[0] - assert call["bbox"] == list(bbox) - assert call["bbox_crs"] == "EPSG:4326" - assert call["max_mpp"] == 5.0 - assert call["partitions"] == 8 - - -# --- DL4: result carries StacClient.download schema --- -def test_download_returns_stac_schema(spark, tmp_path): - mock = _MockStacClient(_make_search_df(spark, _FAKE_ITEMS), - _make_download_df(spark, ["dep_10_a"])) - dd = DemDownloader(_stac_client=mock) - result = dd.download((-122.52, 37.70, -122.36, 37.83), str(tmp_path / "o"), spark=spark) - required = {"item_id", "asset_name", "out_file_path", "out_file_sz", "is_out_file_valid", "last_update"} - assert required <= {f.name for f in result.schema} - - -# --- DL5: empty result carries the canonical schema --- -def test_download_empty_returns_canonical_schema(spark, tmp_path): - mock = _MockStacClient(_make_search_df(spark, [])) - dd = DemDownloader(_stac_client=mock) - result = dd.download((-122.52, 37.70, -122.36, 37.83), str(tmp_path / "o"), spark=spark) - assert result.count() == 0 - schema_map = {f.name: f.dataType for f in result.schema} - assert {"item_id", "asset_name", "out_file_path", "out_file_sz", "is_out_file_valid", "last_update"} <= set(schema_map) - assert "href" not in schema_map - assert isinstance(schema_map["is_out_file_valid"], BooleanType) - assert isinstance(schema_map["out_file_sz"], LongType) - - -# --- DL6: finest with no gsd property keeps all items (graceful) --- -def test_download_finest_no_gsd_keeps_all(spark, tmp_path): - mock = _MockStacClient(_make_search_df(spark, _FAKE_ITEMS_NO_GSD), - _make_download_df(spark, ["dep_nogsd"])) - dd = DemDownloader(_stac_client=mock) - dd.download((-122.52, 37.70, -122.36, 37.83), str(tmp_path / "o"), - resolution="finest", spark=spark) - assert mock.download_calls[0]["item_ids"] == ["dep_nogsd"] -``` - -(E1 export + the helper test are already in the file from Task 1 and will now pass once `download_dem_aoi` exists.) - -- [ ] **Step 2: Run to verify the download tests fail** - -Run: `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest python/geobrix/test/sample/test_dem.py -v -k "download or export"` -Expected: FAIL — `AttributeError: 'DemDownloader' object has no attribute 'download'` / `ImportError: download_dem_aoi` (if not already added in Task 1 Step 5). - -- [ ] **Step 3: Implement `download` + `download_dem_aoi`** - -Add to `DemDownloader` in `sample/dem.py` (after `read`), and the module-level function at the end: - -```python - def download( - self, - bbox: Sequence[float], - out_dir: str, - resolution: Union[int, str] = "finest", - bbox_crs: str = "EPSG:4326", - max_mpp: Optional[float] = None, - partitions: Optional[int] = None, - spark=None, - ) -> "DataFrame": - """Search, select a gsd tier, and download 3DEP tiles to out_dir. - - resolution="finest" (default) picks the minimum gsd (e.g. 10 m over 30 m); - an int picks that exact gsd. When a source has no gsd property, "finest" - keeps all matching items (graceful no-op). Returns StacClient.download's - result: item_id, asset_name, out_file_path, out_file_sz, is_out_file_valid, - last_update. - """ - from pyspark.sql import SparkSession - from pyspark.sql import functions as F - - spark = spark or SparkSession.getActiveSession() - client = self._get_stac_client() - aoi_df = self._aoi_dataframe(bbox, spark) - - raw = client.search( - aoi_df, geojson_col="geojson", - collections=[self.collection], datetime=_DEM_DATETIME, - ) - img = raw.filter(F.col("asset_name") == self.asset).withColumn( - "_gsd", self._gsd_col() - ) - - if resolution == "finest": - min_row = img.agg(F.min("_gsd").alias("m")).first() - selected = min_row["m"] if min_row is not None else None - # A gsd tier exists -> keep the finest; else (no gsd property, or no - # items at all) keep the matching set as-is (empty stays empty). - vintage = img.filter(F.col("_gsd") == selected) if selected is not None else img - else: - vintage = img.filter(F.col("_gsd") == int(resolution)) - - vintage = vintage.select("item_id", "asset_name", "href") - return client.download( - vintage, out_dir, - bbox=list(bbox), bbox_crs=bbox_crs, max_mpp=max_mpp, partitions=partitions, - ) -``` - -Module-level one-shot (end of file): - -```python -def download_dem_aoi( - spark, - bbox: Sequence[float], - out_dir: str, - resolution: Union[int, str] = "finest", - max_mpp: Optional[float] = None, - **kw, -) -> "DataFrame": - """One-shot: construct a default DemDownloader and download a DEM for an AOI. - - Convenience wrapper — Planetary Computer catalog, planetary_computer signing, - 3dep-seamless collection, "data" asset. Forwards **kw (e.g. partitions, bbox_crs). - """ - downloader = DemDownloader() - return downloader.download( - bbox, out_dir, resolution=resolution, max_mpp=max_mpp, spark=spark, **kw - ) -``` - -- [ ] **Step 4: Run the full test file to verify all pass** - -Run: `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest python/geobrix/test/sample/test_dem.py -v` -Expected: PASS (D1–D3, DL1–DL6, E1, helper). - -- [ ] **Step 5: Serverless-safe source scan** - -Run: `grep -nE "spark\.conf\.set|_jvm|sparkContext|\.rdd|\.cache\(|\.persist\(" python/geobrix/src/databricks/labs/gbx/sample/dem.py` -Expected: no matches (empty output). If any match, remove it — the module must be Serverless-safe. - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects 2>/dev/null || true -git add python/geobrix/src/databricks/labs/gbx/sample/dem.py python/geobrix/test/sample/test_dem.py -git commit -m "feat(sample): DemDownloader.download gsd selection + download_dem_aoi - -resolution='finest' picks the minimum gsd (10 m over 30 m); int picks that exact gsd; -graceful no-op when a source lacks gsd. One-shot download_dem_aoi mirrors download_naip_aoi. - -Co-authored-by: Isaac" -``` - ---- - -## Self-Review - -**1. Spec coverage:** -- `DemDownloader` in `sample/dem.py`, exported → Task 1. ✓ -- discover/download/read + one-shot, mirroring NaipDownloader → Tasks 1+2. ✓ -- Selection axis = gsd: finest→min, int→exact, no-op if no gsd → Task 2 `download` + DL1/DL2/DL6. ✓ -- collection/asset `__init__` params → Task 1 `__init__`. ✓ -- Serverless-safe (no spark.conf/_jvm/.rdd) → Task 2 Step 5 scan. ✓ -- Online-only + `_stac_client` injection seam → Task 1 `_get_stac_client`. ✓ -- No new SQL function / no function-info change → nothing in the plan touches those. ✓ -- Empty-result canonical schema → DL5. ✓ -- Follow-ups (NB-03 wiring, docs page, live Serverless smoke) explicitly out of scope → header. ✓ - -**2. Placeholder scan:** No TBD/TODO. The only cross-task nuance (`download`/`download_dem_aoi` land in the Task-1 commit so the module imports) is stated explicitly with the reason, not left vague. - -**3. Type consistency:** `discover` → `(item_id, gsd, item_bbox, href)`; `download`/`download_dem_aoi` signatures match between the interface blocks, the impl, and the tests. `resolution` is `Optional[int]` for `discover` (None=all) and `Union[int,str]` for `download` (default `"finest"`). `_gsd_col()` used identically in `discover` and `download`. `download` returns `StacClient.download`'s schema (item_id, asset_name, out_file_path, out_file_sz, is_out_file_valid, last_update), asserted by DL4/DL5. diff --git a/docs/superpowers/plans/2026-06-30-pmtiles-reader-scalable-mosaic-pyramid.md b/docs/superpowers/plans/2026-06-30-pmtiles-reader-scalable-mosaic-pyramid.md deleted file mode 100644 index e17d84338..000000000 --- a/docs/superpowers/plans/2026-06-30-pmtiles-reader-scalable-mosaic-pyramid.md +++ /dev/null @@ -1,570 +0,0 @@ -# `pmtiles_gbx` Reader — Scalable Per-Tile Mosaic Pyramid + Archive Read — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a `reader()` to the `pmtiles_gbx` DataSource that produces a `(z,x,y,bytes)` tile stream — `source="raster"` builds an XYZ pyramid from source COGs via per-tile mosaic reads (scalable, seamless, memory-bounded), `source="archive"` reads an existing `.pmtiles` back to tiles. - -**Architecture:** A pure, Spark-free core (`ds/_xyz_mosaic.py`) does tile enumeration (morecantile) + per-tile mosaic compositing (rio-tiler `mosaic_reader` over in-memory rasterio datasets). Two thin `DataSourceReader`s (raster, archive) wrap it; fan-out comes from `InputPartition`s (scan partitions, not AQE-coalesced). `PMTilesGbxDataSource.reader()` dispatches on the `source` option. - -**Tech Stack:** Python 3.12, rio-tiler, morecantile, rasterio, the `pmtiles` lib, PySpark DataSource V2, pytest. Tests run via the project venv on the host. - -## Global Constraints - -- **Serverless-safe:** call `databricks.labs.gbx.pyrx._env.configure_gdal_env()` at the start of every executor `read()`; no `spark.conf.set`/`_jvm`/`sparkContext`/`.rdd`. -- **FUSE-safe:** read source/archive bytes **sequentially** (`open(path,"rb").read()`) into an in-memory dataset (`rasterio.io.MemoryFile`) / `pmtiles.reader.MemorySource`; never random-seek a Volume path. -- **Thread-safe:** `render_tile` calls `mosaic_reader(..., threads=0)` (serial) — rasterio datasets aren't safe for concurrent reads; the reader pre-loads source datasets single-threaded before rendering. -- **Output schema is exactly `(z int, x int, y int, bytes binary)`** — identical to the `pmtiles_gbx` writer's required input (`assert_input_schema`, `ds/pmtiles.py:55`), so reader → writer needs no glue. -- **No new SQL function**; do NOT touch `function-info.json` or `docs/tests-function-info/registered_functions.txt`. -- **Tests on the host venv:** `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest -v`. -- v1 raster mode targets uint8 imagery; `pixelSelection` v1 supports only `"first"`. OUT OF SCOPE: `source="vector"`, rescale-for-EO, infer-source-from-path, overview simplification, heavy-tier parity. - ---- - -### Task 1: Pure per-tile mosaic core (`ds/_xyz_mosaic.py`) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/_xyz_mosaic.py` -- Test: `python/geobrix/test/ds/test_xyz_mosaic.py` - -**Interfaces:** -- Produces: - - `enumerate_tiles(bbox, min_z, max_z) -> list[tuple[int,int,int]]` — bbox is `(minx,miny,maxx,maxy)` EPSG:4326; returns `(z,x,y)` for every WebMercatorQuad tile intersecting bbox across `min_z..max_z`. - - `source_bounds_union(paths) -> tuple[float,float,float,float]` — EPSG:4326 union of the rasters' bounds. - - `render_tile(z, x, y, datasets, tile_format="PNG") -> bytes | None` — composite the open rasterio `datasets` for tile `(z,x,y)` via `mosaic_reader` (serial); PNG bytes, or `None` if no dataset covers the tile. - -- [ ] **Step 1: Write the failing tests** - -```python -# python/geobrix/test/ds/test_xyz_mosaic.py -import io -import numpy as np -import rasterio -from rasterio.io import MemoryFile -from rasterio.transform import from_bounds -from PIL import Image - -from databricks.labs.gbx.ds._xyz_mosaic import ( - enumerate_tiles, source_bounds_union, render_tile, -) - - -def _cog_bytes(w, s, e, n, px=128, val=200): - """A uint8 RGB EPSG:4326 raster filling [w,s,e,n] with a constant value.""" - data = np.full((3, px, px), val, dtype="uint8") - profile = dict(driver="GTiff", width=px, height=px, count=3, dtype="uint8", - crs="EPSG:4326", transform=from_bounds(w, s, e, n, px, px)) - with MemoryFile() as mf: - with mf.open(**profile) as ds: - ds.write(data) - return mf.read() - - -def _open(b): - return MemoryFile(b).open() - - -def test_enumerate_tiles_covers_bbox(): - tiles = enumerate_tiles((-122.52, 37.70, -122.35, 37.83), 12, 13) - zs = {z for z, x, y in tiles} - assert zs == {12, 13} - assert all(isinstance(v, int) for t in tiles for v in t) - assert len(tiles) >= 4 # multiple tiles across the AOI - - -def test_source_bounds_union(): - a = _open(_cog_bytes(10.0, 50.0, 11.0, 51.0)) - b = _open(_cog_bytes(11.0, 50.0, 12.0, 51.0)) - try: - u = source_bounds_union([a, b]) if False else None - finally: - a.close(); b.close() - # union takes PATHS in production; here assert via the path-based helper below - import tempfile, os - paths = [] - for bb in (_cog_bytes(10.0, 50.0, 11.0, 51.0), _cog_bytes(11.0, 49.0, 12.0, 51.0)): - fd, p = tempfile.mkstemp(suffix=".tif"); os.write(fd, bb); os.close(fd); paths.append(p) - u = source_bounds_union(paths) - assert u[0] == 10.0 and u[1] == 49.0 and u[2] == 12.0 and u[3] == 51.0 - - -def test_render_tile_composites_all_covering_sources(): - # Two adjacent quads; a tile spanning the seam must composite BOTH (the cluster bug). - left = _open(_cog_bytes(-122.50, 37.74, -122.45, 37.79, val=120)) - right = _open(_cog_bytes(-122.45, 37.74, -122.40, 37.79, val=220)) - try: - import morecantile - tms = morecantile.tms.get("WebMercatorQuad") - # a high zoom tile near the seam lon=-122.45, lat~37.765 - t = next(iter(tms.tiles(-122.46, 37.76, -122.44, 37.77, [16]))) - png = render_tile(t.z, t.x, t.y, [left, right]) - assert png is not None - arr = np.asarray(Image.open(io.BytesIO(png)).convert("RGBA")) - assert float(np.mean(arr[:, :, 3] == 255)) > 0.99 # fully covered, no seam gap - # both source values appear (left ~120, right ~220) -> composited from both - lo = float(np.mean((arr[:, :, 0] > 90) & (arr[:, :, 0] < 150))) - hi = float(np.mean(arr[:, :, 0] > 190)) - assert lo > 0 and hi > 0 - finally: - left.close(); right.close() - - -def test_render_tile_none_when_no_source_covers(): - only = _open(_cog_bytes(-122.50, 37.74, -122.45, 37.79)) - try: - import morecantile - tms = morecantile.tms.get("WebMercatorQuad") - far = next(iter(tms.tiles(10.0, 50.0, 10.1, 50.1, [16]))) # far away - assert render_tile(far.z, far.x, far.y, [only]) is None - finally: - only.close() -``` - -- [ ] **Step 2: Run the tests to verify they fail** - -Run: `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest python/geobrix/test/ds/test_xyz_mosaic.py -v` -Expected: FAIL — `ModuleNotFoundError: ... ds._xyz_mosaic`. - -- [ ] **Step 3: Implement the core** - -```python -# python/geobrix/src/databricks/labs/gbx/ds/_xyz_mosaic.py -"""Pure (Spark-free) per-tile XYZ mosaic core for the pmtiles_gbx raster reader. - -Enumerate slippy-map tiles for an AOI (morecantile WebMercatorQuad) and render each -tile by compositing the covering source rasters with rio-tiler's mosaic_reader. No -full mosaic is built: each tile reads only its 256x256 window, so memory is bounded -per tile and the work distributes one (z,x,y) at a time. -""" - -from __future__ import annotations - -from typing import List, Optional, Sequence, Tuple - -BBox = Tuple[float, float, float, float] - - -def _tms(): - import morecantile - - return morecantile.tms.get("WebMercatorQuad") - - -def enumerate_tiles(bbox: BBox, min_z: int, max_z: int) -> List[Tuple[int, int, int]]: - """Every (z, x, y) WebMercatorQuad tile intersecting bbox (EPSG:4326) across z.""" - tms = _tms() - w, s, e, n = bbox - out: List[Tuple[int, int, int]] = [] - for z in range(int(min_z), int(max_z) + 1): - for t in tms.tiles(w, s, e, n, [z]): - out.append((int(t.z), int(t.x), int(t.y))) - return out - - -def source_bounds_union(paths: Sequence[str]) -> BBox: - """EPSG:4326 union of the source rasters' bounds.""" - import rasterio - from rasterio.warp import transform_bounds - - ws = ss = es = ns = None - for p in paths: - with rasterio.open(p) as ds: - w, s, e, n = transform_bounds(ds.crs, "EPSG:4326", *ds.bounds) - ws = w if ws is None else min(ws, w) - ss = s if ss is None else min(ss, s) - es = e if es is None else max(es, e) - ns = n if ns is None else max(ns, n) - if ws is None: - raise ValueError("source_bounds_union: no source rasters") - return (ws, ss, es, ns) - - -def render_tile(z, x, y, datasets, tile_format: str = "PNG") -> Optional[bytes]: - """Composite the open rasterio `datasets` for tile (z,x,y); PNG bytes or None. - - Uses rio-tiler mosaic_reader serially (threads=0) — rasterio datasets are not safe - for concurrent reads, and the inputs are already in memory so serial is cheap. - Returns None when no dataset covers the tile (caller skips it). - """ - from rio_tiler.errors import EmptyMosaicError - from rio_tiler.io import Reader - from rio_tiler.mosaic import mosaic_reader - - def _read(ds, tx, ty, tz): - with Reader(None, dataset=ds) as cog: - return cog.tile(tx, ty, tz) - - try: - img, _ = mosaic_reader(list(datasets), _read, int(x), int(y), int(z), threads=0) - except EmptyMosaicError: - return None - return bytes(img.render(img_format=tile_format)) -``` - -- [ ] **Step 4: Run the tests to verify they pass** - -Run: `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest python/geobrix/test/ds/test_xyz_mosaic.py -v` -Expected: PASS (4 tests). The composite test is the regression for the cluster western-quad bug. - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects 2>/dev/null || true -git add python/geobrix/src/databricks/labs/gbx/ds/_xyz_mosaic.py python/geobrix/test/ds/test_xyz_mosaic.py -git commit -m "feat(ds): _xyz_mosaic core — per-tile mosaic compositing + tile enumeration - -Spark-free core for the pmtiles_gbx raster reader: morecantile tile enumeration, -source-bounds union, and per-tile mosaic_reader compositing (serial, None on no -coverage). Unit-tested incl. the boundary-composite regression. - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: `source="raster"` reader (per-tile mosaic pyramid) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/_pmtiles_read.py` (the reader classes; keeps `ds/pmtiles.py` focused) -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/pmtiles.py` (add `reader()` dispatch to `PMTilesGbxDataSource`) -- Test: `python/geobrix/test/ds/test_pmtiles_reader.py` - -**Interfaces:** -- Consumes: `_xyz_mosaic.enumerate_tiles`, `source_bounds_union`, `render_tile` (Task 1); `_listing.list_files` + `_listing.to_spark_uri` (`ds/_listing.py`, as used by `RasterGbxReader`); `databricks.labs.gbx.pyrx._env.configure_gdal_env`. -- Produces: `PMtilesRasterReader(DataSourceReader)`; `PMTilesGbxDataSource.reader(schema)` returns it when `options.get("source","raster")=="raster"`. Output rows `(z:int,x:int,y:int,bytes:binary)`. - -- [ ] **Step 1: Write the failing integration tests** - -```python -# python/geobrix/test/ds/test_pmtiles_reader.py -import io -import numpy as np -import rasterio -from rasterio.transform import from_bounds -from PIL import Image - -from databricks.labs.gbx.ds.pmtiles import PMTilesGbxDataSource - - -def _write_cog(path, w, s, e, n, px=256, val=200): - data = np.full((3, px, px), val, dtype="uint8") - with rasterio.open(path, "w", driver="GTiff", width=px, height=px, count=3, - dtype="uint8", crs="EPSG:4326", - transform=from_bounds(w, s, e, n, px, px)) as ds: - ds.write(data) - - -def test_raster_reader_schema_and_fanout(spark, tmp_path): - # two adjacent quads over a small AOI - _write_cog(str(tmp_path / "a.tif"), -122.50, 37.74, -122.45, 37.79, val=120) - _write_cog(str(tmp_path / "b.tif"), -122.45, 37.74, -122.40, 37.79, val=220) - spark.dataSource.register(PMTilesGbxDataSource) - df = (spark.read.format("pmtiles_gbx") - .option("source", "raster").option("path", str(tmp_path)) - .option("bbox", "-122.50,37.74,-122.40,37.79") - .option("minZoom", "14").option("maxZoom", "16") - .option("tilesPerPartition", "20") - .load()) - assert [f.name for f in df.schema.fields] == ["z", "x", "y", "bytes"] - rows = df.collect() - assert len(rows) > 0 - assert df.rdd.getNumPartitions() >= 2 # fans out via InputPartitions - # no (z,x,y) duplicates (each tile produced once) - keys = [(r["z"], r["x"], r["y"]) for r in rows] - assert len(keys) == len(set(keys)) - - -def test_raster_reader_composites_seam_tile(spark, tmp_path): - _write_cog(str(tmp_path / "a.tif"), -122.50, 37.74, -122.45, 37.79, val=120) - _write_cog(str(tmp_path / "b.tif"), -122.45, 37.74, -122.40, 37.79, val=220) - spark.dataSource.register(PMTilesGbxDataSource) - rows = (spark.read.format("pmtiles_gbx") - .option("source", "raster").option("path", str(tmp_path)) - .option("bbox", "-122.50,37.74,-122.40,37.79") - .option("minZoom", "16").option("maxZoom", "16") - .load().collect()) - # find a fully-covered tile; assert it carries BOTH source values (composited) - both = 0 - for r in rows: - arr = np.asarray(Image.open(io.BytesIO(bytes(r["bytes"]))).convert("RGBA")) - if float(np.mean(arr[:, :, 3] == 255)) > 0.99: - lo = (arr[:, :, 0] > 90) & (arr[:, :, 0] < 150) - hi = arr[:, :, 0] > 190 - if lo.any() and hi.any(): - both += 1 - assert both >= 1, "no tile composited both quads -> the western-quad bug" - - -def test_raster_reader_bbox_defaults_to_source_union(spark, tmp_path): - _write_cog(str(tmp_path / "a.tif"), -122.50, 37.74, -122.45, 37.79) - spark.dataSource.register(PMTilesGbxDataSource) - df = (spark.read.format("pmtiles_gbx") - .option("source", "raster").option("path", str(tmp_path)) - .option("minZoom", "15").option("maxZoom", "15").load()) # no bbox - assert len(df.collect()) > 0 -``` - -(The `spark` fixture comes from `python/geobrix/test/ds/conftest.py`.) - -- [ ] **Step 2: Run to verify they fail** - -Run: `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest python/geobrix/test/ds/test_pmtiles_reader.py -v` -Expected: FAIL — `PMTilesGbxDataSource` has no `reader()` (Spark raises "does not support read" / `NotImplementedError`). - -- [ ] **Step 3: Implement the raster reader** - -Create `python/geobrix/src/databricks/labs/gbx/ds/_pmtiles_read.py`: - -```python -"""DataSourceReaders for pmtiles_gbx: raster (per-tile mosaic pyramid) + archive.""" - -from __future__ import annotations - -import re -from typing import Dict, Iterator, List, Sequence, Tuple - -from pyspark.sql.datasource import DataSourceReader, InputPartition - -from databricks.labs.gbx.ds import _listing, _xyz_mosaic - - -class _TilesPartition(InputPartition): - def __init__(self, tiles: List[Tuple[int, int, int]], sources: List[str]): - self.tiles = tiles - self.sources = sources - - -def _chunk(seq, n): - for i in range(0, len(seq), n): - yield seq[i : i + n] - - -class PMtilesRasterReader(DataSourceReader): - def __init__(self, options: Dict[str, str]): - self.path = options.get("path") - if not self.path: - raise ValueError("pmtiles_gbx raster reader requires a 'path' (dir of COGs).") - self.filter_regex = options.get("filterRegex", r".*\.tif$") - self.min_z = int(options.get("minZoom", "0")) - self.max_z = int(options.get("maxZoom", "0")) - self.tiles_per_partition = int(options.get("tilesPerPartition", "64")) - self.tile_format = options.get("tileFormat", "png").upper() - ps = options.get("pixelSelection", "first").lower() - if ps != "first": - raise ValueError(f"pmtiles_gbx v1 supports pixelSelection='first' only; got {ps!r}") - bbox_opt = options.get("bbox") - self.bbox = ( - tuple(float(v) for v in bbox_opt.split(",")) if bbox_opt else None - ) - if self.bbox is not None and len(self.bbox) != 4: - raise ValueError("pmtiles_gbx bbox must be 'minx,miny,maxx,maxy'") - - def partitions(self) -> Sequence[InputPartition]: - import rasterio - from rasterio.warp import transform_bounds - - sources = _listing.list_files(self.path, self.filter_regex) - if not sources: - raise ValueError(f"pmtiles_gbx raster reader: no rasters under {self.path}") - bbox = self.bbox or _xyz_mosaic.source_bounds_union(sources) - # per-source WGS84 bounds, to attach only intersecting sources to each chunk - src_bounds = [] - for p in sources: - with rasterio.open(p) as ds: - src_bounds.append((p, transform_bounds(ds.crs, "EPSG:4326", *ds.bounds))) - tiles = _xyz_mosaic.enumerate_tiles(bbox, self.min_z, self.max_z) - # spatial grouping: sort by (z, y, x) so chunks are contiguous - tiles.sort() - parts: List[InputPartition] = [] - tms = _xyz_mosaic._tms() - for chunk in _chunk(tiles, self.tiles_per_partition): - # combined WGS84 bbox of the chunk's tiles - cb = [tms.bounds(__import__("morecantile").Tile(x, y, z)) for z, x, y in chunk] - cw = min(b.left for b in cb); cs = min(b.bottom for b in cb) - ce = max(b.right for b in cb); cn = max(b.top for b in cb) - needed = [p for p, (w, s, e, n) in src_bounds - if not (e < cw or w > ce or n < cs or s > cn)] - parts.append(_TilesPartition(chunk, needed or sources)) - return parts - - def read(self, partition: "_TilesPartition") -> Iterator[Tuple]: - import rasterio - from rasterio.io import MemoryFile - - from databricks.labs.gbx.pyrx import _env - - _env.configure_gdal_env() - # FUSE-safe: sequential byte read -> in-memory dataset, pre-loaded single-threaded - mfs, datasets = [], [] - try: - for p in partition.sources: - with open(p, "rb") as fh: - mf = MemoryFile(fh.read()) - mfs.append(mf) - datasets.append(mf.open()) - for (z, x, y) in partition.tiles: - png = _xyz_mosaic.render_tile(z, x, y, datasets, tile_format=self.tile_format) - if png is not None: - yield (z, x, y, png) - finally: - for ds in datasets: - ds.close() - for mf in mfs: - mf.close() -``` - -In `python/geobrix/src/databricks/labs/gbx/ds/pmtiles.py`, add a `reader()` to `PMTilesGbxDataSource` (the class at line 70; it already has `name()`, `schema()`, `writer()`): - -```python - def reader(self, schema: StructType) -> "DataSourceReader": # noqa: F821 - # Per-branch (lazy) imports so the raster reader works before the archive - # reader exists (Task 2 lands before Task 3). - source = self.options.get("source", "raster").lower() - if source == "raster": - from databricks.labs.gbx.ds._pmtiles_read import PMtilesRasterReader - return PMtilesRasterReader(self.options) - if source == "archive": - from databricks.labs.gbx.ds._pmtiles_read import PMtilesArchiveReader - return PMtilesArchiveReader(self.options) - raise ValueError(f"pmtiles_gbx: unknown source={source!r} (use 'raster' or 'archive')") -``` - -(Add `from pyspark.sql.datasource import DataSourceReader` to the imports if not present. `schema()` already returns `(z,x,y,bytes)` — the shared read/write schema; confirm and reuse it.) - -- [ ] **Step 4: Run the raster tests to verify they pass** - -Run: `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest python/geobrix/test/ds/test_pmtiles_reader.py -v -k raster` -Expected: PASS. (Task 3 adds the archive test to the same file.) - -- [ ] **Step 5: Run the existing pmtiles writer tests for regression** - -Run: `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest python/geobrix/test/ds/test_pmtiles_parity.py python/geobrix/test/ds/test_pmtiles.py -v` -Expected: PASS (the writer + schema are unchanged; `reader()` is additive). - -- [ ] **Step 6: Commit** - -```bash -chmod -R u+rwX .git/objects 2>/dev/null || true -git add python/geobrix/src/databricks/labs/gbx/ds/_pmtiles_read.py python/geobrix/src/databricks/labs/gbx/ds/pmtiles.py python/geobrix/test/ds/test_pmtiles_reader.py -git commit -m "feat(ds): pmtiles_gbx source=raster reader (scalable per-tile mosaic pyramid) - -reader() dispatches on source; the raster reader enumerates tiles, groups them into -InputPartitions (fan-out, AQE-coalesce-proof), and renders each via the _xyz_mosaic -core with configure_gdal_env + FUSE-safe pre-loaded in-memory datasets. Output schema -matches the writer so reader->writer needs no glue. - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: `source="archive"` reader (read an existing `.pmtiles`) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/_pmtiles_read.py` (add `PMtilesArchiveReader`) -- Test: `python/geobrix/test/ds/test_pmtiles_reader.py` (add round-trip test) - -**Interfaces:** -- Consumes: the `pmtiles` lib (`pmtiles.reader.MemorySource`, `all_tiles`); the `pmtiles_gbx` writer (to build a fixture archive for the round-trip). -- Produces: `PMtilesArchiveReader(DataSourceReader)` — reads a `.pmtiles` file → `(z,x,y,bytes)`. - -- [ ] **Step 1: Write the failing round-trip test** - -```python -# add to python/geobrix/test/ds/test_pmtiles_reader.py -def test_archive_reader_roundtrip(spark, tmp_path): - from pyspark.sql import Row - spark.dataSource.register(PMTilesGbxDataSource) - # build a small archive via the writer - tiles = [Row(z=14, x=2615 + i, y=6330, bytes=_png_bytes(i)) for i in range(3)] - out = str(tmp_path / "rt.pmtiles") - spark.createDataFrame(tiles).write.format("pmtiles_gbx").option("shardZoom", "0").mode("overwrite").save(out) - # read it back - back = (spark.read.format("pmtiles_gbx").option("source", "archive").option("path", out).load().collect()) - got = {(r["z"], r["x"], r["y"]): bytes(r["bytes"]) for r in back} - assert set(got) == {(14, 2615, 6330), (14, 2616, 6330), (14, 2617, 6330)} - - -def _png_bytes(i): - import io - import numpy as np - from PIL import Image - a = np.full((256, 256, 3), 40 + i * 20, dtype="uint8") - buf = io.BytesIO(); Image.fromarray(a).save(buf, format="PNG"); return buf.getvalue() -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest python/geobrix/test/ds/test_pmtiles_reader.py::test_archive_reader_roundtrip -v` -Expected: FAIL — `ImportError: cannot import name 'PMtilesArchiveReader'` (referenced by `reader()` but not yet defined). - -- [ ] **Step 3: Implement the archive reader** - -Append to `ds/_pmtiles_read.py`: - -```python -class PMtilesArchiveReader(DataSourceReader): - def __init__(self, options: Dict[str, str]): - self.path = options.get("path") - if not self.path: - raise ValueError("pmtiles_gbx archive reader requires a 'path' (.pmtiles file).") - self.tiles_per_partition = int(options.get("tilesPerPartition", "2048")) - - def _entries(self) -> List[Tuple[int, int, int]]: - from pmtiles.reader import MemorySource, all_tiles - - with open(self.path, "rb") as fh: - raw = fh.read() - return [(z, x, y) for (z, x, y), _ in all_tiles(MemorySource(raw))] - - def partitions(self) -> Sequence[InputPartition]: - entries = self._entries() - return [_TilesPartition(list(c), [self.path]) for c in _chunk(entries, self.tiles_per_partition)] - - def read(self, partition: "_TilesPartition") -> Iterator[Tuple]: - from pmtiles.reader import MemorySource, Reader - - # FUSE-safe: read archive bytes sequentially, then serve tiles in memory - with open(partition.sources[0], "rb") as fh: - raw = fh.read() - reader = Reader(MemorySource(raw)) - for (z, x, y) in partition.tiles: - data = reader.get(z, x, y) - if data is not None: - yield (z, x, y, bytes(data)) -``` - -(Confirm the `pmtiles` lib's tile-fetch API — `Reader(MemorySource(raw)).get(z, x, y)`; if the installed version exposes it differently, mirror the call used elsewhere in `ds/pmtiles.py`/tests, e.g. iterate `all_tiles` and select. Keep the FUSE-safe sequential read either way.) - -- [ ] **Step 4: Run the round-trip test to verify it passes** - -Run: `source /Users/mjohns/IdeaProjects/geobrix/.venv-pyrx/bin/activate && python -m pytest python/geobrix/test/ds/test_pmtiles_reader.py -v` -Expected: PASS (all raster + archive tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects 2>/dev/null || true -git add python/geobrix/src/databricks/labs/gbx/ds/_pmtiles_read.py python/geobrix/test/ds/test_pmtiles_reader.py -git commit -m "feat(ds): pmtiles_gbx source=archive reader (read existing .pmtiles -> tiles) - -Round-trips with the writer; FUSE-safe sequential archive read via the pmtiles lib -MemorySource. Completes the read/write symmetry of the pmtiles_gbx tile family. - -Co-authored-by: Isaac" -``` - ---- - -## Self-Review - -**1. Spec coverage:** -- Pure core (enumerate/union/render) + boundary-composite regression → Task 1. ✓ -- `source="raster"` reader (options, partitions fan-out, configure_gdal_env, FUSE-safe pre-loaded cache, skip-empty, schema==writer) → Task 2. ✓ -- `source="archive"` reader (read existing .pmtiles, round-trip) → Task 3. ✓ -- `reader()` dispatch on `source`; shared `schema()`; no register change → Task 2. ✓ -- Serverless-safe / FUSE-safe / thread-safe (threads=0) / no SQL function → Global Constraints + baked into each task. ✓ -- Out of scope (vector, rescale, infer-path, overview simplification, heavy parity) → Global Constraints. ✓ - -**2. Placeholder scan:** No TBD/TODO. The two "confirm the exact API" notes (schema() reuse; pmtiles `.get` vs `all_tiles`) name the precise file/call to check and a concrete fallback — not open-ended placeholders. - -**3. Type consistency:** `enumerate_tiles`/`source_bounds_union`/`render_tile` signatures match between Task 1 (definition) and Task 2 (use). `_TilesPartition(tiles, sources)` is defined in Task 2 and reused in Task 3. Output rows `(z,x,y,bytes)` consistent across both readers and equal to the writer's `assert_input_schema`. `reader()` (Task 2) references `PMtilesArchiveReader` delivered in Task 3 — Task 2's raster tests run with `-k raster` so the missing import doesn't block; Task 3 completes it (noted in Task 2 Step 4). diff --git a/docs/superpowers/plans/2026-06-30-raster-bbox-window-read.md b/docs/superpowers/plans/2026-06-30-raster-bbox-window-read.md deleted file mode 100644 index 2aaba0bea..000000000 --- a/docs/superpowers/plans/2026-06-30-raster-bbox-window-read.md +++ /dev/null @@ -1,609 +0,0 @@ -# Raster `bbox` Window-on-Read Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a correct-by-construction `bbox` window-on-read option to the light raster readers (`raster_gbx`/`gtiff_gbx`) and a `StacClient.download(bbox=)` companion, so customers stop hand-rolling rasterio windowed reads (and the clip-vs-`window_transform` georef footgun). - -**Architecture:** One shared geometry primitive `ds/_window.py: window_for_bbox(src, bbox, bbox_crs)` returns a **clipped, integer, in-bounds `rasterio.windows.Window`** (or `None` on no-overlap). Because the window is guaranteed in-bounds, any caller doing `ds.read(win)` + `ds.window_transform(win)` gets agreeing pixels and georeference — the footgun cannot occur. The reader hands that window to the existing `_encode.encode_tile`; `StacClient`'s fetch opens the (signed) href with rasterio (`/vsicurl` for https → only AOI byte ranges), windows it, and writes a windowed GeoTIFF. - -**Tech Stack:** Python 3.12, rasterio, PySpark DataSource V2, pytest. Tests run via `gbx:test:python --path python/geobrix/test/ds/` and `.../test/stac/`. - -> **Refinement vs the committed spec:** the spec sketched the primitive returning `(data, transform, profile)`; this plan returns the clipped **`Window`** instead, so the reader reuses `_encode.encode_tile` (no duplicate encode logic) and the StacClient does its own read+write. The correctness principle is identical — the clip is centralized in the primitive, and an in-bounds window makes read/transform agree everywhere. - -## Global Constraints - -- **Serverless-safe (light tier):** no `spark.conf.set`, no `_jvm`/`sparkContext`/`.rdd`; GDAL only via `rasterio` (never raw `osgeo.gdal`); call `databricks.labs.gbx.pyrx._env.configure_gdal_env()` before executor reads (the reader already does). -- **No new SQL function:** reader/client option surface only — do NOT touch `function-info.json` or `docs/tests-function-info/registered_functions.txt`. -- **`bbox` string format:** `"minx,miny,maxx,maxy"` (same as the vector reader, `ds/vector.py:540`). A non-4 value raises `ValueError`. -- **CRS convention:** plain `bbox` is in the source CRS; `bboxCrs` (e.g. `"EPSG:4326"`) declares the bbox CRS and the primitive reprojects the bbox to the source CRS (mirrors `rst_clip`'s SRID rule). -- **Georef correctness is the headline:** the window is clipped to the dataset BEFORE the transform is derived; an overhang regression test is required. -- **Volume FUSE can't seek:** windowed reads in the reader must stage the source to worker-local disk with a sequential copy first (the reader's phase-2 already does this — mirror it for the bbox path). -- **Non-overlap:** reader yields no tile for a non-overlapping source file (skip); `StacClient.download` with a non-overlapping bbox raises a clear error. -- **Tier scope:** light only. Heavy `gdal`/`gtiff_gdal` parity, vector-reader `bboxCrs` backport, on-read decimation (`rst_resample_to_res` exists), and the Helios notebook rework are OUT OF SCOPE. - -## File Structure - -- `python/geobrix/src/databricks/labs/gbx/ds/_window.py` — **new**; the `window_for_bbox` primitive. One responsibility: bbox → clipped in-bounds Window. -- `python/geobrix/test/ds/test_window.py` — **new**; primitive unit tests (overhang regression, CRS transform, no-overlap). -- `python/geobrix/src/databricks/labs/gbx/ds/raster.py` — modify `RasterGbxReader.__init__` (parse `bbox`/`bboxCrs`) and `RasterGbxReader.read` (bbox branch: stage-to-local + window + encode). `GTiffGbxReader` inherits both (ds/gtiff.py:16), so `gtiff_gbx` gets the option for free. -- `python/geobrix/test/ds/test_raster_bbox.py` — **new**; Spark integration tests for `raster_gbx` + `gtiff_gbx`. -- `python/geobrix/src/databricks/labs/gbx/stac/_download.py` — modify `fetch_validate_publish` (add `bbox`/`bbox_crs`; windowed-fetch branch). -- `python/geobrix/src/databricks/labs/gbx/stac/client.py` — modify `StacClient.download` (add `bbox`/`bbox_crs`, thread into the `_fetch` UDF). -- `python/geobrix/test/stac/test_download_bbox.py` — **new**; windowed fetch tests (local file as href). - ---- - -### Task 1: `window_for_bbox` primitive - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/_window.py` -- Test: `python/geobrix/test/ds/test_window.py` - -**Interfaces:** -- Produces: `window_for_bbox(src, bbox: tuple[float,float,float,float], bbox_crs: str | None = None) -> rasterio.windows.Window | None`. `src` is an open `rasterio.DatasetReader`. Returns a clipped, integer, in-bounds Window, or `None` if the bbox does not overlap `src`. - -- [ ] **Step 1: Write the failing tests** - -```python -# python/geobrix/test/ds/test_window.py -import rasterio -from rasterio.io import MemoryFile -from rasterio.warp import transform_bounds - -from databricks.labs.gbx.ds._window import window_for_bbox -from databricks.labs.gbx.test.ds.conftest import make_geotiff_bytes # if importable; else use the local helper below - - -def _open(width=4, height=3, epsg=4326): - # Fixture extent: origin (10.0, 50.0), 0.5 px -> x[10,12], y[48.5,50] in EPSG:4326. - mf = MemoryFile(make_geotiff_bytes(width=width, height=height, epsg=epsg)) - return mf, mf.open() - - -def test_fully_inside_window_matches_bbox(): - mf, ds = _open() - try: - win = window_for_bbox(ds, (10.5, 49.0, 11.5, 50.0)) # inside x[10,12], y[48.5,50] - assert win is not None - b = rasterio.windows.bounds(win, ds.transform) # (left, bottom, right, top) - assert b == (10.5, 49.0, 11.5, 50.0) - finally: - ds.close(); mf.close() - - -def test_north_overhang_is_clipped_not_shifted(): - # Regression for the NB-02 georef bug: a bbox whose top (51.0) is north of the - # dataset top (50.0) must clip to the dataset top, NOT report row 0 at 51.0. - mf, ds = _open() - try: - win = window_for_bbox(ds, (10.5, 49.0, 11.5, 51.0)) - assert win is not None - top = rasterio.windows.bounds(win, ds.transform)[3] - assert top == 50.0, f"top should clip to dataset top 50.0, got {top}" - assert win.row_off == 0 - finally: - ds.close(); mf.close() - - -def test_no_overlap_returns_none(): - mf, ds = _open() - try: - assert window_for_bbox(ds, (20.0, 20.0, 21.0, 21.0)) is None - finally: - ds.close(); mf.close() - - -def test_bbox_crs_is_reprojected(): - # Source in EPSG:3857 over a known SF extent; a WGS84 bbox inside it must be - # transformed to 3857 before windowing (proves bbox_crs is applied). - w, s, e, n = transform_bounds("EPSG:4326", "EPSG:3857", -122.5, 37.7, -122.4, 37.8) - from rasterio.transform import from_bounds as _affine_from_bounds - profile = dict(driver="GTiff", width=100, height=100, count=1, dtype="uint8", - crs="EPSG:3857", transform=_affine_from_bounds(w, s, e, n, 100, 100)) - with MemoryFile() as src_mf: - with src_mf.open(**profile) as out: - import numpy as np - out.write(np.zeros((1, 100, 100), dtype="uint8")) - data = src_mf.read() - mf = MemoryFile(data); ds = mf.open() - try: - win = window_for_bbox(ds, (-122.47, 37.72, -122.43, 37.78), bbox_crs="EPSG:4326") - assert win is not None - b = rasterio.windows.bounds(win, ds.transform) # in source CRS (3857) - exp = transform_bounds("EPSG:4326", "EPSG:3857", -122.47, 37.72, -122.43, 37.78) - # within one source pixel (rounding to whole pixels) - px = abs(ds.transform.a) - assert all(abs(a - c) <= px for a, c in zip(b, exp)) - finally: - ds.close(); mf.close() -``` - -(If `make_geotiff_bytes` is not importable as a module path, copy the 12-line builder from `python/geobrix/test/ds/conftest.py:58` into this test file.) - -- [ ] **Step 2: Run the tests to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_window.py` -Expected: FAIL with `ModuleNotFoundError: ... ds._window` / `ImportError: cannot import name 'window_for_bbox'`. - -- [ ] **Step 3: Implement the primitive** - -```python -# python/geobrix/src/databricks/labs/gbx/ds/_window.py -"""Clip-safe AOI windowing for raster reads. - -window_for_bbox computes the pixel Window of a dataset covering a bbox and CLIPS it -to the dataset BEFORE returning it. Callers do ds.read(win) + ds.window_transform(win) -on an in-bounds window, so pixels and georeference always agree -- the clip-vs- -window_transform footgun (read clips to the dataset, but window_transform used the -unclipped window's origin -> raster shifted by the overhang) cannot occur. -""" - -from __future__ import annotations - -from typing import Optional, Tuple - -from rasterio.windows import Window, from_bounds as _from_bounds - - -def window_for_bbox( - src, - bbox: Tuple[float, float, float, float], - bbox_crs: Optional[str] = None, -) -> Optional[Window]: - """Clipped, integer, in-bounds Window of ``src`` covering ``bbox``. - - bbox is (minx, miny, maxx, maxy). bbox_crs (e.g. "EPSG:4326") declares the bbox - CRS; None means the bbox is already in src.crs. Returns None if the bbox does not - overlap the dataset. - """ - minx, miny, maxx, maxy = bbox - if bbox_crs is not None and str(bbox_crs) != str(src.crs): - from rasterio.warp import transform_bounds as _transform_bounds - - minx, miny, maxx, maxy = _transform_bounds( - bbox_crs, src.crs, minx, miny, maxx, maxy - ) - win = _from_bounds(minx, miny, maxx, maxy, transform=src.transform) - # Whole-pixel coverage of the bbox, then clip to the dataset extent. - win = win.round_offsets(op="floor").round_lengths(op="ceil") - try: - win = win.intersection(Window(0, 0, src.width, src.height)) - except Exception: # rasterio.errors.WindowError when disjoint - return None - if win.width < 1 or win.height < 1: - return None - return Window(int(win.col_off), int(win.row_off), int(win.width), int(win.height)) -``` - -- [ ] **Step 4: Run the tests to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_window.py` -Expected: PASS (4 tests). - -- [ ] **Step 5: Commit** - -```bash -chmod -R u+rwX .git/objects 2>/dev/null || true -git add python/geobrix/src/databricks/labs/gbx/ds/_window.py python/geobrix/test/ds/test_window.py -git commit -m "feat(ds): window_for_bbox clip-safe AOI windowing primitive - -Returns a clipped, in-bounds rasterio Window so read + window_transform agree -(fixes the clip-vs-window_transform georef footgun class). Source-CRS bbox with -an optional bbox_crs reprojection. - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: `bbox`/`bboxCrs` options on the raster readers - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/raster.py` (`RasterGbxReader.__init__` ~line 76-81; `RasterGbxReader.read` ~line 87) -- Test: `python/geobrix/test/ds/test_raster_bbox.py` - -**Interfaces:** -- Consumes: `window_for_bbox` (Task 1); existing `_encode.encode_tile(ds, window=(col,row,w,h), source_path, all_parents)` and `_listing.to_spark_uri`. -- Produces: reader options `.option("bbox", "minx,miny,maxx,maxy")` and `.option("bboxCrs", "EPSG:4326")` on `raster_gbx` and (inherited) `gtiff_gbx`. - -- [ ] **Step 1: Write the failing integration tests** - -```python -# python/geobrix/test/ds/test_raster_bbox.py -import numpy as np -import rasterio -from rasterio.io import MemoryFile -from rasterio.transform import from_origin - -from databricks.labs.gbx.ds.raster import RasterGbxDataSource -from databricks.labs.gbx.ds.gtiff import GTiffGbxDataSource - - -def _write_sample(path, width=4, height=3, epsg=4326): - # extent: origin (10.0, 50.0), 0.5 px -> x[10,12], y[48.5,50] - data = np.arange(width * height, dtype="float32").reshape(height, width) - with rasterio.open(path, "w", driver="GTiff", width=width, height=height, count=1, - dtype="float32", crs=f"EPSG:{epsg}", - transform=from_origin(10.0, 50.0, 0.5, 0.5), nodata=-9999.0) as ds: - ds.write(data, 1) - - -def _tile_bounds(row): - with MemoryFile(bytes(row["tile"]["raster"])) as mf, mf.open() as out: - b = out.bounds - return (b.left, b.bottom, b.right, b.top), (out.width, out.height) - - -def test_bbox_windows_to_aoi(spark, tmp_path): - f = tmp_path / "s.tif"; _write_sample(str(f)) - spark.dataSource.register(RasterGbxDataSource) - df = spark.read.format("raster_gbx").option("bbox", "10.5,49.0,11.5,50.0").load(str(f)) - rows = df.collect() - assert len(rows) == 1 - bounds, (w, h) = _tile_bounds(rows[0]) - assert bounds == (10.5, 49.0, 11.5, 50.0) - assert (w, h) == (2, 2) # 1.0 deg / 0.5 px - - -def test_bbox_north_overhang_clips(spark, tmp_path): - f = tmp_path / "s.tif"; _write_sample(str(f)) - spark.dataSource.register(RasterGbxDataSource) - df = spark.read.format("raster_gbx").option("bbox", "10.5,49.0,11.5,51.0").load(str(f)) - bounds, _ = _tile_bounds(df.collect()[0]) - assert bounds[3] == 50.0 # top clipped to dataset top, not 51.0 - - -def test_non_overlapping_file_is_skipped(spark, tmp_path): - f = tmp_path / "s.tif"; _write_sample(str(f)) - spark.dataSource.register(RasterGbxDataSource) - df = spark.read.format("raster_gbx").option("bbox", "20,20,21,21").load(str(f)) - assert df.collect() == [] - - -def test_gtiff_gbx_parity(spark, tmp_path): - f = tmp_path / "s.tif"; _write_sample(str(f)) - spark.dataSource.register(RasterGbxDataSource) - spark.dataSource.register(GTiffGbxDataSource) - opt = ("bbox", "10.5,49.0,11.5,50.0") - r1 = spark.read.format("raster_gbx").option(*opt).load(str(f)).collect()[0] - r2 = spark.read.format("gtiff_gbx").option(*opt).load(str(f)).collect()[0] - assert _tile_bounds(r1) == _tile_bounds(r2) - - -def test_malformed_bbox_raises(spark, tmp_path): - f = tmp_path / "s.tif"; _write_sample(str(f)) - spark.dataSource.register(RasterGbxDataSource) - import pytest - with pytest.raises(Exception): - spark.read.format("raster_gbx").option("bbox", "1,2,3").load(str(f)).collect() -``` - -(The `spark` fixture is provided by `python/geobrix/test/ds/conftest.py`.) - -- [ ] **Step 2: Run the tests to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_raster_bbox.py` -Expected: FAIL — `bbox` is ignored, so `test_bbox_windows_to_aoi` returns the whole 4x3 raster (bounds `(10,48.5,12,50)`, not `(10.5,49,11.5,50)`), and `test_non_overlapping_file_is_skipped` returns 1 row. - -- [ ] **Step 3: Parse the options in `RasterGbxReader.__init__`** - -In `python/geobrix/src/databricks/labs/gbx/ds/raster.py`, extend `__init__` (after the `filter_regex` line, ~line 81): - -```python - self.filter_regex = options.get("filterRegex", ".*") - # Optional AOI window-on-read. `bbox` is "minx,miny,maxx,maxy" in the source - # CRS by default; `bboxCrs` (e.g. "EPSG:4326") declares the bbox CRS and the - # window primitive reprojects it. None = read the whole raster (prior behavior). - bbox_opt = options.get("bbox") - if bbox_opt: - parts = [float(v) for v in str(bbox_opt).split(",")] - if len(parts) != 4: - raise ValueError( - "raster bbox option must be 'minx,miny,maxx,maxy'; got " - f"'{bbox_opt}'" - ) - self.bbox = tuple(parts) - else: - self.bbox = None - self.bbox_crs = options.get("bboxCrs") -``` - -- [ ] **Step 4: Add the bbox branch to `RasterGbxReader.read`** - -In `read`, right after `source = _listing.to_spark_uri(partition.file_path)` (~line 102) and before the `size_bytes = ...` line, insert the short-circuit branch: - -```python - # AOI window-on-read: stage to worker-local disk (FUSE-safe sequential copy -- - # Volume FUSE cannot serve the per-window seeks), then window from local disk. - # bbox disables the whole-image fast path and the multi-tile split. - if self.bbox is not None: - from databricks.labs.gbx.ds._window import window_for_bbox - - staged_dir = tempfile.mkdtemp(prefix="gbx_raster_") - try: - local_path = os.path.join( - staged_dir, os.path.basename(partition.file_path) or "raster.tif" - ) - with ( - open(partition.file_path, "rb") as _src, - open(local_path, "wb") as _dst, - ): - shutil.copyfileobj(_src, _dst, length=8 * 1024 * 1024) - with rasterio.open(local_path) as ds: - win = window_for_bbox(ds, self.bbox, self.bbox_crs) - if win is None: - return # source does not overlap the AOI -> emit nothing - cellid, raster_bytes, meta = _encode.encode_tile( - ds, - window=( - int(win.col_off), - int(win.row_off), - int(win.width), - int(win.height), - ), - source_path=partition.file_path, - all_parents="", - ) - yield (source, (cellid, raster_bytes, meta)) - finally: - shutil.rmtree(staged_dir, ignore_errors=True) - return -``` - -(`os`, `shutil`, `tempfile`, and `rasterio` are already imported at the top of `read`.) - -- [ ] **Step 5: Run the tests to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_raster_bbox.py` -Expected: PASS (5 tests). - -- [ ] **Step 6: Run the existing raster reader tests to confirm no regression** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_raster_datasource.py` -Expected: PASS (the no-bbox path is unchanged — `self.bbox is None` short-circuits nothing). - -- [ ] **Step 7: Commit** - -```bash -chmod -R u+rwX .git/objects 2>/dev/null || true -git add python/geobrix/src/databricks/labs/gbx/ds/raster.py python/geobrix/test/ds/test_raster_bbox.py -git commit -m "feat(ds): bbox/bboxCrs window-on-read on raster_gbx/gtiff_gbx - -A bbox option windows each source to the AOI via window_for_bbox (clip-safe), -staging to local disk first (FUSE-safe). Non-overlapping sources are skipped. -gtiff_gbx inherits the option. Mirrors the vector reader's bbox; source-CRS -default with a bboxCrs override. - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: `StacClient.download(bbox=, bbox_crs=)` windowed fetch - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/stac/_download.py` (`fetch_validate_publish`) -- Modify: `python/geobrix/src/databricks/labs/gbx/stac/client.py` (`StacClient.download`, ~line 209) -- Test: `python/geobrix/test/stac/test_download_bbox.py` - -**Interfaces:** -- Consumes: `window_for_bbox` (Task 1). -- Produces: `fetch_validate_publish(..., bbox=None, bbox_crs=None)` and `StacClient.download(..., bbox=None, bbox_crs=None)`. When `bbox` is set, the fetch opens the (signed) href with rasterio (`/vsicurl` for https → AOI byte ranges only), windows it, and writes a windowed GeoTIFF; a non-overlapping bbox raises `ValueError`. - -- [ ] **Step 1: Write the failing tests** - -```python -# python/geobrix/test/stac/test_download_bbox.py -import os - -import numpy as np -import pytest -import rasterio -from rasterio.transform import from_origin - -from databricks.labs.gbx.stac._download import fetch_validate_publish - - -def _write_gtiff(path, width=8, height=8): - # extent: origin (0, 8), 1.0 px -> x[0,8], y[0,8] - with rasterio.open(path, "w", driver="GTiff", height=height, width=width, count=1, - dtype="uint8", crs="EPSG:4326", transform=from_origin(0, 8, 1, 1)) as dst: - dst.write(np.arange(width * height, dtype="uint8").reshape(1, height, width)) - - -def test_windowed_fetch_clips_to_bbox(tmp_path): - src = tmp_path / "src.tif"; _write_gtiff(str(src)) - out_dir = tmp_path / "out" - res = fetch_validate_publish( - lambda: str(src), str(out_dir), "win.tif", bbox=(2, 2, 5, 6) - ) - assert res == os.path.join(str(out_dir), "win.tif") - with rasterio.open(res) as ds: - b = ds.bounds - assert (b.left, b.bottom, b.right, b.top) == (2, 2, 5, 6) - assert (ds.width, ds.height) == (3, 4) - - -def test_windowed_fetch_north_overhang_clips(tmp_path): - src = tmp_path / "src.tif"; _write_gtiff(str(src)) - out_dir = tmp_path / "out" - res = fetch_validate_publish( - lambda: str(src), str(out_dir), "win.tif", bbox=(2, 2, 10, 12) # E+N overhang - ) - with rasterio.open(res) as ds: - assert ds.bounds.top == 8 # clipped to dataset top, not 12 - assert ds.bounds.right == 8 - - -def test_windowed_fetch_no_overlap_raises(tmp_path): - src = tmp_path / "src.tif"; _write_gtiff(str(src)) - out_dir = tmp_path / "out" - res = fetch_validate_publish( - lambda: str(src), str(out_dir), "win.tif", bbox=(20, 20, 21, 21), max_tries=1 - ) - assert res is None # no overlap -> all attempts fail -> None (no file published) - assert not os.path.exists(os.path.join(str(out_dir), "win.tif")) - - -def test_no_bbox_path_unchanged(tmp_path): - # bbox=None must keep the byte-faithful download path. - src = tmp_path / "src.tif"; _write_gtiff(str(src)) - out_dir = tmp_path / "out" - - def get(href, timeout=None, stream=None): - class R: - def raise_for_status(self): pass - def iter_content(self, n): yield open(str(src), "rb").read() - return R() - - res = fetch_validate_publish(lambda: "http://x/ok.tif", str(out_dir), "ok.tif", get=get) - assert res == os.path.join(str(out_dir), "ok.tif") - assert os.path.getsize(res) == os.path.getsize(str(src)) # byte-identical (no window) -``` - -- [ ] **Step 2: Run the tests to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/stac/test_download_bbox.py` -Expected: FAIL — `fetch_validate_publish` has no `bbox` parameter (`TypeError: unexpected keyword argument 'bbox'`). - -- [ ] **Step 3: Add the windowed-fetch branch to `fetch_validate_publish`** - -In `python/geobrix/src/databricks/labs/gbx/stac/_download.py`, add a helper and extend the signature + the per-attempt fetch: - -```python -def windowed_download(href: str, outpath: str, bbox, bbox_crs=None) -> str: - """Open href (rasterio /vsicurl for https; any path locally), window to bbox, and - write a windowed GeoTIFF. The window is clipped to the dataset, so the output is - correctly georeferenced. Raises ValueError if the bbox does not overlap the asset.""" - import rasterio - - from databricks.labs.gbx.ds._window import window_for_bbox - - with rasterio.open(href) as src: - win = window_for_bbox(src, bbox, bbox_crs) - if win is None: - raise ValueError(f"bbox {bbox} does not overlap the asset {href!r}") - data = src.read(window=win) - profile = src.profile.copy() - profile.update( - driver="GTiff", - width=int(win.width), - height=int(win.height), - transform=src.window_transform(win), - ) - with rasterio.open(outpath, "w", **profile) as dst: - dst.write(data) - return outpath -``` - -Update `fetch_validate_publish`'s signature to add `bbox=None, bbox_crs=None` (after `validate: bool = True`), and replace the per-attempt download line: - -```python - local = os.path.join(tmpd, safe_filename) - if bbox is not None: - # Windowed read decodes-on-read; a successful write IS the validation, - # so publish directly (skip the separate read_validate window-decode). - windowed_download(href_fn(), local, bbox, bbox_crs) - shutil.copyfile(local, outpath) - return outpath - download_href(href_fn(), local, get=get) - if validate: - if read_validate(local): - shutil.copyfile(local, outpath) # publish only validated files - return outpath - else: - shutil.copyfile(local, outpath) - return outpath -``` - -(The existing idempotency short-circuit at the top still applies; a windowed re-run that finds a valid `outpath` returns it.) - -- [ ] **Step 4: Run the `_download` tests to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/stac/test_download_bbox.py` -Expected: PASS (4 tests). - -- [ ] **Step 5: Thread `bbox`/`bbox_crs` through `StacClient.download`** - -In `python/geobrix/src/databricks/labs/gbx/stac/client.py`, add `bbox: Optional[Sequence[float]] = None, bbox_crs: Optional[str] = None` to `download`'s signature (after `partitions`), capture them into locals next to `sign`/`_validate`, and pass them into the `fetch_validate_publish` call inside the `_fetch` UDF: - -```python - sign = self.sign - _validate = validate - _bbox = tuple(bbox) if bbox is not None else None - _bbox_crs = bbox_crs - _injected_get = _get_fn - ... - return fetch_validate_publish( - href_fn, - out_dir, - filename, - max_tries=max_tries, - validate=_validate, - bbox=_bbox, - bbox_crs=_bbox_crs, - **kwargs, - ) -``` - -Add to `download`'s docstring: "bbox=(minx,miny,maxx,maxy) windows each asset to the AOI on read (source CRS by default; bbox_crs declares the bbox CRS)." - -- [ ] **Step 6: Add a client-level test** - -Append to `python/geobrix/test/stac/test_download_bbox.py`: - -```python -def test_client_download_threads_bbox(spark, tmp_path, monkeypatch): - # End-to-end via StacClient.download with a local file as the (unsigned) href. - from databricks.labs.gbx.stac.client import StacClient - src = tmp_path / "src.tif"; _write_gtiff(str(src)) - out_dir = tmp_path / "out" - df = spark.createDataFrame( - [("item1", "image", str(src))], ["item_id", "asset_name", "href"] - ) - client = StacClient.__new__(StacClient) # bypass __init__ (no network/catalog) - client.sign = "none" # resolve_signer("none") must be identity; assert below - res = client.download(df, str(out_dir), bbox=(2, 2, 5, 6)).collect() - assert len(res) == 1 and res[0]["is_out_file_valid"] - with rasterio.open(res[0]["out_file_path"]) as ds: - assert (ds.width, ds.height) == (3, 4) -``` - -If `resolve_signer("none")` is not already an identity signer, the test should use whatever sign value `StacClient` treats as "no signing" (check `python/geobrix/src/databricks/labs/gbx/stac/_sign.py`); adjust `client.sign` accordingly. The `spark` fixture for the stac tests comes from `python/geobrix/test/stac/` conftest (or create a local `local[2]` session fixture mirroring `test/ds/conftest.py:spark` if none exists). - -- [ ] **Step 7: Run the full stac download + client tests** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/stac/test_download_bbox.py` then `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/stac/test_download.py` -Expected: PASS (windowed tests + the pre-existing `_download` tests unchanged). - -- [ ] **Step 8: Commit** - -```bash -chmod -R u+rwX .git/objects 2>/dev/null || true -git add python/geobrix/src/databricks/labs/gbx/stac/_download.py python/geobrix/src/databricks/labs/gbx/stac/client.py python/geobrix/test/stac/test_download_bbox.py -git commit -m "feat(stac): StacClient.download(bbox=) windowed AOI fetch - -Windowed-read variant of fetch_validate_publish opens the signed href with -rasterio (/vsicurl range reads for https) and writes a windowed GeoTIFF via the -clip-safe window_for_bbox, inside the existing re-sign/retry loop. Removes the -need for hand-rolled rasterio windowed staging. Source-CRS bbox with a bbox_crs -override; non-overlapping bbox raises. - -Co-authored-by: Isaac" -``` - ---- - -## Self-Review - -**1. Spec coverage:** -- Shared windowing primitive + footgun fix → Task 1. ✓ -- Reader `bbox`/`bboxCrs`, fast-path disable, non-overlap skip, gtiff_gbx parity → Task 2. ✓ -- `StacClient.download(bbox=)` windowed `/vsicurl` read → Task 3. ✓ -- CRS convention (source default + bboxCrs) → Task 1 primitive + Task 2/3 threading. ✓ -- Decimation out of scope, heavy parity out of scope, notebook rework out of scope → Global Constraints. ✓ -- Overhang regression test → Task 1 Step 1 (`test_north_overhang_is_clipped_not_shifted`) + Task 2 (`test_bbox_north_overhang_clips`) + Task 3 (`test_windowed_fetch_north_overhang_clips`). ✓ - -**2. Placeholder scan:** No TBD/TODO; every code step has complete code. The one conditional ("if `resolve_signer('none')` is not identity…") names the exact file to check and the concrete adjustment — not a placeholder. - -**3. Type consistency:** `window_for_bbox(src, bbox, bbox_crs) -> Window | None` is consumed identically in Tasks 2 and 3 (`win.col_off/row_off/width/height`, `None` → skip/raise). `bbox` is a 4-float tuple throughout. `encode_tile(ds, window=(col,row,w,h), source_path, all_parents)` matches its definition at `ds/_encode.py:19`. diff --git a/docs/superpowers/plans/2026-07-10-netcdf-gbx-reader.md b/docs/superpowers/plans/2026-07-10-netcdf-gbx-reader.md deleted file mode 100644 index f6c74907c..000000000 --- a/docs/superpowers/plans/2026-07-10-netcdf-gbx-reader.md +++ /dev/null @@ -1,1364 +0,0 @@ -# netcdf_gbx Lightweight Reader Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add an initial NetCDF reader to the GeoBrix lightweight (`pyrx`) API as a Spark Python DataSource named `netcdf_gbx`, with a raster mode (CF grids → GeoTIFF tile) and a vector mode (point/swath → per-cell points), plus a `TropomiDownloader` that stages real Sentinel-5P granules to validate vector mode. - -**Architecture:** One `NetcdfGbxDataSource(DataSource)` branches on a `mode` option (default `raster`) in both `schema()` and `reader()` — the exact pattern `VectorGbxDataSource.schema()` uses to read `self.options`. Raster mode reuses the existing `_encode.encode_tile` path (NetCDF variable → in-memory rasterio dataset → GeoTIFF bytes → the shared `(source, tile)` struct). Vector mode builds shapely points → plain WKB rows matching the light vector reader's schema convention (attribute columns + `geom_0` WKB + `geom_0_srid` + `geom_0_srid_proj`). Pure NetCDF/CF logic lives in a helper module `ds/_netcdf.py`; the DataSource + two readers live in `ds/netcdf.py`. - -**Tech Stack:** Python 3.12, PySpark 4 Python DataSource API, `xarray` (already in `[light]` via `xarray-spatial`), `netcdf4` (one new `[light]` dep), `rasterio` (existing), `shapely` (existing), `pyproj`/`rasterio.crs` (existing). Design reference: `docs/superpowers/specs/2026-07-10-netcdf-gbx-reader-design.md`. - -## Global Constraints - -- **Serverless-safe product code:** no `spark.conf.set`, `_jvm`, `.rdd`, `.cache()`, or `.persist()` anywhere in `ds/netcdf.py`, `ds/_netcdf.py`, or `sample/tropomi.py`. -- **GDAL-free light tier:** decode NetCDF only via `xarray` + `netcdf4`; encode rasters only via `rasterio` (its bundled GDAL). Never import `osgeo`. -- **Tile contract:** raster mode emits `struct>>` — identical to `raster_gbx`/`gtiff_gbx`. `cellid` = `-1` (`_encode.CELLID_FRESH`); metadata is the exact 11-key set. -- **Vector schema convention:** attribute columns (one per requested variable, typed) in requested order, then geometry column `geom_0` as **plain WKB** (`BinaryType`), then `geom_0_srid` (`StringType`) and `geom_0_srid_proj` (`StringType`). SRID is carried in the string columns, **not** embedded as EWKB. -- **Honesty:** never resample and never apply quality thresholds — pass every requested variable through (e.g. S5P `qa_value` travels as its own column). Class 4 (raw sensor geometry + GLT) is **rejected in both modes**. -- **Not a SQL function:** no `registered_functions.txt` / `function-info.json` / binding-parity changes. -- **Naming:** DataSource format name is exactly `netcdf_gbx`. New reader option names: `mode`, `variable`/`variables`, `group`. -- **Dependency pinning — three environments, kept in lockstep (Task 1):** a new light dep must be pinned in all three places or one environment breaks: - 1. **CI light tier** — `requirements-pyrx-ci.in` → regenerate hashed `requirements-pyrx-ci.txt` (`uv pip compile --generate-hashes --python-version 3.12`). CI installs it with `--require-hashes` (`.github/actions/pyrx_build/action.yml`); a stale lock fails closed. - 2. **Docker dev/test** (`geobrix-dev`, used by `gbx:test:python`) — `requirements-dev-container.in` → regenerate hashed `requirements-dev-container.txt`. - 3. **Local `.venv-pyrx`** (`gbx:venv:sync`, floor-only resolve) — the floor pin in the pyproject extra that `gbx:venv:sync` installs. -- **Tier test gating:** heavy-tier CI skips light-only tests by **directory**, not marker — `test/conftest.py::_LIGHT_TEST_DIRS` + `collect_ignore` drops those dirs when `rasterio` is absent (the heavy env). `ds` and `sample` are **already** in `_LIGHT_TEST_DIRS` and in the explicit pytest dir list in `.github/actions/pyrx_build/action.yml`, so netcdf tests placed under `test/ds/` and `test/sample/` need **no** conftest or action change. Do not place them anywhere else. -- **Commit hygiene:** subject ≤72 chars + a WHY body; end commit messages with `Co-authored-by: Isaac`. Branch: `examples/vapor-eyes` (already checked out). - ---- - -### Task 1: Pin `netcdf4` across all three light environments - -**Files:** -- Modify: `python/geobrix/pyproject.toml` (the `light = [...]` array, ends at line 116) — floor pin for the local `.venv-pyrx` -- Modify: `python/geobrix/requirements-pyrx-ci.in` (+ regenerate hashed `requirements-pyrx-ci.txt`) — CI light tier -- Modify: `python/geobrix/requirements-dev-container.in` (+ regenerate hashed `requirements-dev-container.txt`) — `geobrix-dev` Docker -- Modify: `scripts/commands/gbx-venv-sync.sh:51` — reconcile the `[pyrx,test]` extra (see Step 4) - -**Interfaces:** -- Consumes: nothing. -- Produces: `import netCDF4` and `import xarray` importable in **all three** light environments (CI pyrx job, `geobrix-dev` Docker, local `.venv-pyrx`). Every later task's tests rely on this. - -- [ ] **Step 1: Add `netcdf4` to the `[light]` extra** - -In `python/geobrix/pyproject.toml`, inside the `light = [` array, after the `"xarray-spatial>=0.4,<1",` line (116), add: - -```toml - # NetCDF decode for the netcdf_gbx reader. xarray is already pulled by - # xarray-spatial; netcdf4 is the one new engine — a single wheel bundling - # netcdf-c + HDF5, reading both NetCDF-3 and NetCDF-4/HDF5 (S5P, ERA5). - # Folded into [light] (not a separate [netcdf] extra) — one wheel, so - # geobrix[light] reads NetCDF out of the box. Floor pinned; add a ceiling - # here if a future release floats/breaks on Serverless env v5 (Py 3.12), - # same discipline as the rio-tiler / mapbox-vector-tile pins above. - "netcdf4>=1.6,<2", -``` - -- [ ] **Step 2: Add `netcdf4` to both hash-pinned requirement inputs** - -These `.in` files use exact `==` pins (e.g. `scipy==1.15.1`, `xarray-spatial==0.9.9`). Add the same exact pin to both. `xarray` is already locked transitively via `xarray-spatial`; `netcdf4` pulls `cftime` (also locked on regenerate). - -In `python/geobrix/requirements-pyrx-ci.in` (CI light tier), add near the other decode deps: - -``` -# NetCDF decode for the netcdf_gbx reader (NetCDF-3 + NetCDF-4/HDF5: S5P, ERA5). -# xarray comes transitively via xarray-spatial. Pin exact; bump alongside the -# pyproject [light] floor. Verify cp312 manylinux wheels exist for the pin. -netcdf4==1.7.2 -``` - -In `python/geobrix/requirements-dev-container.in` (the `geobrix-dev` Docker used by `gbx:test:python`), add the identical line so the Docker test env can run the netcdf tests: - -``` -netcdf4==1.7.2 -``` - -- [ ] **Step 3: Regenerate BOTH hashed locks** - -Each `.in` header documents its exact regenerate command (uv, Python 3.12, `--generate-hashes`). Run both: - -```bash -cd python/geobrix -uv pip compile --generate-hashes --python-version 3.12 \ - --output-file requirements-pyrx-ci.txt requirements-pyrx-ci.in -uv pip compile --generate-hashes --python-version 3.12 \ - --output-file requirements-dev-container.txt requirements-dev-container.in -``` -Expected: each `.txt` gains `netcdf4==1.7.2` and transitive `cftime==...`, each with `--hash=sha256:...` lines. (There is no CI recompile-diff check; `--require-hashes` fails closed at install if a `.txt` is stale, so both must be committed in sync with their `.in`.) - -- [ ] **Step 4: Reconcile the local `.venv-pyrx` extra so its floor pin installs** - -The pyproject floor pin (Step 1) only reaches `.venv-pyrx` if `gbx:venv:sync` installs an extra that includes `netcdf4`. `scripts/commands/gbx-venv-sync.sh:51` currently installs `-e "./python/geobrix[pyrx,test]"`, but **no `pyrx` extra exists** in `pyproject.toml` (only `light`, `test`, `vizx`, `stac`, `overture`, `databricks`) — so that extra is a silent no-op and `.venv-pyrx` gets no light-tier floor deps. Fix the command to install the real umbrella extra: - -```bash -# scripts/commands/gbx-venv-sync.sh line 51 -uv pip install --python "$VENV_DIR/bin/python" -e "./python/geobrix[light,test]" \ -``` -(If a `pyrx` alias extra is intended instead, add `pyrx = [...]` mirroring `light` in `pyproject.toml`. Either way, the extra `gbx:venv:sync` installs MUST contain `netcdf4`.) - -- [ ] **Step 5: Verify imports resolve under the hash-pinned install** - -Run (host with uv, or inside the dev container): - -```bash -cd python/geobrix -uv pip install --require-hashes -r requirements-pyrx-ci.txt -python -c "import xarray, netCDF4; print(xarray.__version__, netCDF4.__version__)" -``` -Expected: `--require-hashes` install succeeds (lock is valid), then prints two version strings, no ImportError. - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/pyproject.toml \ - python/geobrix/requirements-pyrx-ci.in python/geobrix/requirements-pyrx-ci.txt \ - python/geobrix/requirements-dev-container.in python/geobrix/requirements-dev-container.txt \ - scripts/commands/gbx-venv-sync.sh -git commit -m "build(light): pin netcdf4 across CI, Docker, and venv - -xarray is already present via xarray-spatial; netcdf4 is the single new -engine (NetCDF-3 + NetCDF-4/HDF5). Pinned in all three light envs that -resolve independently: requirements-pyrx-ci (CI, hashed), -requirements-dev-container (Docker, hashed), and the [light] floor for -.venv-pyrx. Also fixes gbx-venv-sync to install a real extra ([light, -test]); [pyrx] was a silent no-op leaving .venv-pyrx without light deps. - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: CF helper module `ds/_netcdf.py` (pure functions, no Spark) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/_netcdf.py` -- Test: `python/geobrix/test/ds/test_netcdf_helpers.py` - -**Interfaces:** -- Consumes: `netCDF4`, `xarray`, `numpy`, `affine`, `shapely` (all available after Task 1 / existing). -- Produces (used by Tasks 3–4): - - Constants `GRID = "grid"`, `POINTS = "points"`, `CURVILINEAR = "curvilinear"`, `UNSUPPORTED = "unsupported"`. - - `open_dataset(path: str, group: Optional[str]) -> "xarray.Dataset"` - - `classify(ds: "xarray.Dataset", variable: str) -> str` (one of the four constants) - - `grid_transform_crs(ds, variable: str) -> Tuple["affine.Affine", str]` (CRS as `"EPSG:4326"` etc.) - - `array_2d(ds, variable: str) -> "numpy.ndarray"` (the 2-D slice, north-up) - - `nodata_of(ds, variable: str) -> Optional[float]` - - `point_arrays(ds, variables: List[str]) -> Tuple["np.ndarray", "np.ndarray", Dict[str, "np.ndarray"], str]` → `(lon_1d, lat_1d, {var: values_1d}, srid)` - - `np_to_spark(dtype: "numpy.dtype") -> "pyspark.sql.types.DataType"` - -- [ ] **Step 1: Write the failing tests** - -Create `python/geobrix/test/ds/test_netcdf_helpers.py`: - -```python -"""Unit tests for CF NetCDF helpers (no Spark).""" - -import numpy as np -import pytest -from netCDF4 import Dataset - -from databricks.labs.gbx.ds import _netcdf - - -def _write_regular_grid(path): - with Dataset(path, "w") as ds: - ds.createDimension("lat", 3) - ds.createDimension("lon", 4) - lat = ds.createVariable("lat", "f8", ("lat",)) - lon = ds.createVariable("lon", "f8", ("lon",)) - lat.standard_name = "latitude" - lon.standard_name = "longitude" - lat[:] = [50.0, 49.5, 49.0] # descending (north-up) - lon[:] = [10.0, 10.5, 11.0, 11.5] - v = ds.createVariable("ch4", "f4", ("lat", "lon"), fill_value=-9999.0) - v[:] = np.arange(12, dtype="float32").reshape(3, 4) - - -def _write_points(path): - with Dataset(path, "w") as ds: - ds.createDimension("obs", 5) - lat = ds.createVariable("latitude", "f8", ("obs",)) - lon = ds.createVariable("longitude", "f8", ("obs",)) - lat.standard_name = "latitude" - lon.standard_name = "longitude" - lat[:] = [50.0, 50.1, 50.2, 50.3, 50.4] - lon[:] = [10.0, 10.1, 10.2, 10.3, 10.4] - v = ds.createVariable("value", "f4", ("obs",)) - v[:] = np.arange(5, dtype="float32") - - -def _write_curvilinear(path): - with Dataset(path, "w") as ds: - ds.createDimension("y", 2) - ds.createDimension("x", 3) - lat = ds.createVariable("latitude", "f8", ("y", "x")) - lon = ds.createVariable("longitude", "f8", ("y", "x")) - lat.standard_name = "latitude" - lon.standard_name = "longitude" - lat[:] = np.array([[50.0, 50.0, 50.0], [49.0, 49.0, 49.0]]) - lon[:] = np.array([[10.0, 11.0, 12.0], [10.0, 11.0, 12.0]]) - v = ds.createVariable("ch4", "f4", ("y", "x")) - v[:] = np.arange(6, dtype="float32").reshape(2, 3) - - -def test_classify_grid(tmp_path): - p = str(tmp_path / "grid.nc") - _write_regular_grid(p) - with _netcdf.open_dataset(p, None) as ds: - assert _netcdf.classify(ds, "ch4") == _netcdf.GRID - - -def test_classify_points(tmp_path): - p = str(tmp_path / "pts.nc") - _write_points(p) - with _netcdf.open_dataset(p, None) as ds: - assert _netcdf.classify(ds, "value") == _netcdf.POINTS - - -def test_classify_curvilinear(tmp_path): - p = str(tmp_path / "curv.nc") - _write_curvilinear(p) - with _netcdf.open_dataset(p, None) as ds: - assert _netcdf.classify(ds, "ch4") == _netcdf.CURVILINEAR - - -def test_grid_transform_crs_north_up(tmp_path): - p = str(tmp_path / "grid.nc") - _write_regular_grid(p) - with _netcdf.open_dataset(p, None) as ds: - transform, crs = _netcdf.grid_transform_crs(ds, "ch4") - assert crs == "EPSG:4326" - # origin at (lon min - half px, lat max + half px); px = 0.5 - assert transform.a == pytest.approx(0.5) # x pixel size - assert transform.e == pytest.approx(-0.5) # y pixel size (north-up => negative) - assert transform.c == pytest.approx(9.75) # ulx = 10.0 - 0.25 - assert transform.f == pytest.approx(50.25) # uly = 50.0 + 0.25 - - -def test_array_2d_is_north_up(tmp_path): - p = str(tmp_path / "grid.nc") - _write_regular_grid(p) - with _netcdf.open_dataset(p, None) as ds: - arr = _netcdf.array_2d(ds, "ch4") - np.testing.assert_allclose(arr, np.arange(12, dtype="float32").reshape(3, 4)) - - -def test_point_arrays_flatten(tmp_path): - p = str(tmp_path / "pts.nc") - _write_points(p) - with _netcdf.open_dataset(p, None) as ds: - lon, lat, attrs, srid = _netcdf.point_arrays(ds, ["value"]) - assert srid == "4326" - assert lon.shape == (5,) and lat.shape == (5,) - np.testing.assert_allclose(attrs["value"], np.arange(5, dtype="float32")) - - -def test_point_arrays_curvilinear_ravel(tmp_path): - p = str(tmp_path / "curv.nc") - _write_curvilinear(p) - with _netcdf.open_dataset(p, None) as ds: - lon, lat, attrs, srid = _netcdf.point_arrays(ds, ["ch4"]) - assert lon.shape == (6,) and lat.shape == (6,) - np.testing.assert_allclose(attrs["ch4"], np.arange(6, dtype="float32")) - - -def test_point_arrays_grid_meshgrid(tmp_path): - # A regular grid coerced to points: lon(4) x lat(3) -> 12 aligned points. - p = str(tmp_path / "grid.nc") - _write_regular_grid(p) - with _netcdf.open_dataset(p, None) as ds: - lon, lat, attrs, srid = _netcdf.point_arrays(ds, ["ch4"]) - assert lon.shape == (12,) and lat.shape == (12,) and attrs["ch4"].shape == (12,) - # first cell is (lon=10.0, lat=50.0) - assert lon[0] == pytest.approx(10.0) and lat[0] == pytest.approx(50.0) - - -def test_np_to_spark_types(): - from pyspark.sql.types import DoubleType, FloatType, IntegerType - assert isinstance(_netcdf.np_to_spark(np.dtype("float32")), FloatType) - assert isinstance(_netcdf.np_to_spark(np.dtype("float64")), DoubleType) - assert isinstance(_netcdf.np_to_spark(np.dtype("int32")), IntegerType) -``` - -- [ ] **Step 2: Run the tests to verify they fail** - -Run: `python/geobrix` → `pytest test/ds/test_netcdf_helpers.py -v` -Expected: FAIL — `ModuleNotFoundError: databricks.labs.gbx.ds._netcdf` (module not created yet). - -- [ ] **Step 3: Implement `ds/_netcdf.py`** - -Create `python/geobrix/src/databricks/labs/gbx/ds/_netcdf.py`: - -```python -"""CF-convention NetCDF helpers for the netcdf_gbx reader (pure, no Spark). - -Classifies a variable's geometry (regular grid / DSG points / curvilinear swath / -unsupported), derives an affine+CRS for grids, and flattens point/swath data to -1-D lon/lat/value arrays. No resampling, no quality filtering — the caller decides. -""" - -from __future__ import annotations - -from contextlib import contextmanager -from typing import Dict, Iterator, List, Optional, Tuple - -GRID = "grid" # class 1/2 -> raster -POINTS = "points" # CF discrete sampling geometries -> vector -CURVILINEAR = "curvilinear" # class 3 (2-D lat/lon) -> vector (per-cell points) -UNSUPPORTED = "unsupported" # class 4 (sensor geometry + GLT) / unknown - -_LAT_NAMES = {"lat", "latitude", "y"} -_LON_NAMES = {"lon", "longitude", "x"} - - -@contextmanager -def open_dataset(path: str, group: Optional[str]) -> Iterator["object"]: - """Open a NetCDF file as an xarray.Dataset (netcdf4 engine), optional HDF5 group.""" - import xarray as xr - - kw = {"engine": "netcdf4", "decode_coords": "all", "mask_and_scale": True} - if group: - kw["group"] = group - ds = xr.open_dataset(path, **kw) - try: - yield ds - finally: - ds.close() - - -def _is_lat(var) -> bool: - sn = str(getattr(var, "standard_name", "")).lower() - un = str(getattr(var, "units", "")).lower() - return sn == "latitude" or un in ("degrees_north", "degree_north") or var.name.lower() in _LAT_NAMES - - -def _is_lon(var) -> bool: - sn = str(getattr(var, "standard_name", "")).lower() - un = str(getattr(var, "units", "")).lower() - return sn == "longitude" or un in ("degrees_east", "degree_east") or var.name.lower() in _LON_NAMES - - -def _find_lat_lon(ds): - """Return (lat_var, lon_var) among ds coords/vars, or (None, None).""" - lat = lon = None - for name in list(ds.variables): - v = ds[name] - if lat is None and _is_lat(v): - lat = v - elif lon is None and _is_lon(v): - lon = v - return lat, lon - - -def classify(ds, variable: str) -> str: - lat, lon = _find_lat_lon(ds) - if lat is None or lon is None: - return UNSUPPORTED - if lat.ndim == 2 and lon.ndim == 2: - return CURVILINEAR - if lat.ndim == 1 and lon.ndim == 1: - var = ds[variable] - # DSG points: the value var shares the single obs dimension with lat/lon. - if var.ndim == 1 and lat.dims == lon.dims == var.dims: - return POINTS - # Regular grid: the value var's last two dims are the lat and lon dims. - if lat.dims[0] in var.dims and lon.dims[0] in var.dims: - return GRID - return UNSUPPORTED - - -def grid_transform_crs(ds, variable: str) -> Tuple["object", str]: - """Affine transform (north-up) + CRS string for a regular grid variable.""" - from affine import Affine - - lat, lon = _find_lat_lon(ds) - lats = lat.values - lons = lon.values - px = float(abs(lons[1] - lons[0])) - py = float(abs(lats[1] - lats[0])) - ulx = float(min(lons)) - px / 2.0 - uly = float(max(lats)) + py / 2.0 - transform = Affine.translation(ulx, uly) * Affine.scale(px, -py) - crs = _crs_string(ds) - return transform, crs - - -def _crs_string(ds) -> str: - # Look for a CF grid_mapping variable carrying an EPSG/authority code. - for name in list(ds.variables): - v = ds[name] - epsg = getattr(v, "epsg_code", None) or getattr(v, "spatial_epsg", None) - if epsg is not None: - return f"EPSG:{int(epsg)}" - # Default: geographic lon/lat. - return "EPSG:4326" - - -def array_2d(ds, variable: str) -> "object": - """The variable's 2-D slice as a north-up numpy array (lat descending).""" - import numpy as np - - da = ds[variable] - # Squeeze any length-1 leading dims (e.g. time); take the first index otherwise. - while da.ndim > 2: - da = da.isel({da.dims[0]: 0}) - lat, _ = _find_lat_lon(ds) - latdim = lat.dims[0] - # Ensure north-up: descending latitude along the lat dimension. - if latdim in da.dims and float(ds[lat.name].values[0]) < float(ds[lat.name].values[-1]): - da = da.isel({latdim: slice(None, None, -1)}) - return np.asarray(da.values) - - -def nodata_of(ds, variable: str) -> Optional[float]: - v = ds[variable] - for attr in ("_FillValue", "missing_value"): - val = v.attrs.get(attr) - if val is not None: - return float(val) - enc = v.encoding.get("_FillValue") - return float(enc) if enc is not None else None - - -def point_arrays(ds, variables: List[str]) -> Tuple["object", "object", Dict[str, "object"], str]: - """Flatten point (DSG), grid, or curvilinear data to aligned 1-D arrays. - - - POINTS/CURVILINEAR: lat/lon already align with the value array, so ravel each. - - GRID: 1-D lat (H) and 1-D lon (W) must be meshgridded to H*W before ravel so - they align with the 2-D value array's ravel. - """ - import numpy as np - - lat, lon = _find_lat_lon(ds) - kind = classify(ds, variables[0]) - if kind == GRID: - lon2d, lat2d = np.meshgrid( - np.asarray(ds[lon.name].values), np.asarray(ds[lat.name].values) - ) # indexing="xy" -> shape (H, W), matching the value array - lon_flat = lon2d.ravel() - lat_flat = lat2d.ravel() - else: # POINTS or CURVILINEAR - lon_flat = np.asarray(ds[lon.name].values).ravel() - lat_flat = np.asarray(ds[lat.name].values).ravel() - attrs: Dict[str, object] = {} - for name in variables: - attrs[name] = np.asarray(ds[name].values).ravel() - return lon_flat, lat_flat, attrs, "4326" - - -def np_to_spark(dtype) -> "object": - import numpy as np - from pyspark.sql import types as T - - kind = np.dtype(dtype).kind - itemsize = np.dtype(dtype).itemsize - if kind == "f": - return T.FloatType() if itemsize <= 4 else T.DoubleType() - if kind in ("i", "u"): - return T.IntegerType() if itemsize <= 4 else T.LongType() - if kind == "b": - return T.BooleanType() - return T.StringType() -``` - -- [ ] **Step 4: Run the tests to verify they pass** - -Run: `pytest test/ds/test_netcdf_helpers.py -v` -Expected: PASS (8 tests). - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/_netcdf.py python/geobrix/test/ds/test_netcdf_helpers.py -git commit -m "feat(ds): CF NetCDF helpers for netcdf_gbx (classify/transform/points) - -Pure, Spark-free geometry classification (grid/points/curvilinear/ -unsupported), north-up affine+CRS derivation for regular grids, and -1-D flattening for point + swath data. Foundation for the raster and -vector reader modes. - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: Raster mode — `NetcdfGbxDataSource` + `NetcdfRasterReader` - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/netcdf.py` -- Test: `python/geobrix/test/ds/test_netcdf_datasource.py` - -**Interfaces:** -- Consumes: `_netcdf.*` (Task 2); `raster.RasterGbxReader`, `raster.reader_schema`, `raster._FilePartition`; `_encode.encode_tile`; `_serde.TILE_SCHEMA`. -- Produces (used by Tasks 4–5): `NetcdfGbxDataSource` (`name()=="netcdf_gbx"`, `schema()`/`reader()` branch on `mode`), `NetcdfRasterReader`. - -- [ ] **Step 1: Write the failing tests (raster mode)** - -Create `python/geobrix/test/ds/test_netcdf_datasource.py`: - -```python -"""Integration tests for the netcdf_gbx DataSource (uses local Spark).""" - -import numpy as np -import pytest -from netCDF4 import Dataset -from rasterio.io import MemoryFile - -from databricks.labs.gbx.ds.netcdf import NetcdfGbxDataSource -from databricks.labs.gbx.pyrx import _serde - -EXPECTED_METADATA_KEYS = { - "path", "sourcePath", "driver", "format", "last_command", "last_error", - "all_parents", "size", "compression", "isZipped", "isSubset", -} - - -def _write_regular_grid(path): - with Dataset(path, "w") as ds: - ds.createDimension("lat", 3) - ds.createDimension("lon", 4) - lat = ds.createVariable("lat", "f8", ("lat",)) - lon = ds.createVariable("lon", "f8", ("lon",)) - lat.standard_name = "latitude" - lon.standard_name = "longitude" - lat[:] = [50.0, 49.5, 49.0] - lon[:] = [10.0, 10.5, 11.0, 11.5] - v = ds.createVariable("ch4", "f4", ("lat", "lon"), fill_value=-9999.0) - v[:] = np.arange(12, dtype="float32").reshape(3, 4) - - -def _write_curvilinear(path): - with Dataset(path, "w") as ds: - ds.createDimension("y", 2) - ds.createDimension("x", 3) - lat = ds.createVariable("latitude", "f8", ("y", "x")) - lon = ds.createVariable("longitude", "f8", ("y", "x")) - lat.standard_name = "latitude" - lon.standard_name = "longitude" - lat[:] = np.array([[50.0, 50.0, 50.0], [49.0, 49.0, 49.0]]) - lon[:] = np.array([[10.0, 11.0, 12.0], [10.0, 11.0, 12.0]]) - v = ds.createVariable("ch4", "f4", ("y", "x")) - v[:] = np.arange(6, dtype="float32").reshape(2, 3) - - -def test_raster_schema_matches_tile_schema(): - ds = NetcdfGbxDataSource(options={"path": "/tmp/none", "variable": "ch4"}) - schema = ds.schema() - assert [f.name for f in schema.fields] == ["source", "tile"] - assert schema["tile"].dataType == _serde.TILE_SCHEMA - - -def test_raster_read_round_trip(spark, tmp_path): - f = tmp_path / "grid.nc" - _write_regular_grid(str(f)) - spark.dataSource.register(NetcdfGbxDataSource) - df = ( - spark.read.format("netcdf_gbx") - .option("variable", "ch4") - .load(str(f)) - ) - rows = df.collect() - assert len(rows) == 1 - row = rows[0] - assert row["tile"]["cellid"] == -1 - assert set(row["tile"]["metadata"].keys()) == EXPECTED_METADATA_KEYS - with MemoryFile(bytes(row["tile"]["raster"])) as mf, mf.open() as out: - arr = out.read(1) - assert out.crs.to_epsg() == 4326 - np.testing.assert_allclose( - arr, np.arange(12, dtype="float32").reshape(3, 4), rtol=1e-6 - ) - - -def test_raster_mode_rejects_curvilinear(spark, tmp_path): - f = tmp_path / "curv.nc" - _write_curvilinear(str(f)) - from databricks.labs.gbx.ds.netcdf import NetcdfRasterReader - from databricks.labs.gbx.ds.raster import _FilePartition - - reader = NetcdfRasterReader({"path": str(f), "variable": "ch4"}) - with pytest.raises(ValueError, match="vector"): - list(reader.read(_FilePartition(str(f), reader.size_mib))) -``` - -- [ ] **Step 2: Run the tests to verify they fail** - -Run: `pytest test/ds/test_netcdf_datasource.py -v` -Expected: FAIL — `ModuleNotFoundError: databricks.labs.gbx.ds.netcdf`. - -- [ ] **Step 3: Implement `ds/netcdf.py` (raster mode + DataSource skeleton)** - -Create `python/geobrix/src/databricks/labs/gbx/ds/netcdf.py`: - -```python -"""netcdf_gbx — lightweight NetCDF reader. - -One DataSource, two modes (the `mode` option, default "raster"): - * raster — CF regular/projected grids -> the shared (source, tile) GeoTIFF struct. - * vector — DSG points, or any 2-D field (incl. curvilinear swath) coerced to - per-cell points -> the light vector schema (attrs + geom_0 WKB + srid cols). - -Class 4 (raw sensor geometry + GLT) is rejected in both modes. -Serverless-safe: no spark.conf/_jvm/.rdd/cache/persist. -""" - -from __future__ import annotations - -from typing import Dict, Iterator, List, Optional, Tuple - -from pyspark.sql.datasource import DataSource, DataSourceReader -from pyspark.sql.types import StructType - -from databricks.labs.gbx.ds import _encode, _netcdf -from databricks.labs.gbx.ds.raster import ( - RasterGbxReader, - _FilePartition, - reader_schema, -) - - -def _requested_variables(options: Dict[str, str]) -> List[str]: - raw = options.get("variables") or options.get("variable") - if not raw: - raise ValueError( - "netcdf_gbx requires a 'variable' (or 'variables') option naming the " - "NetCDF variable(s) to read." - ) - return [v.strip() for v in str(raw).split(",") if v.strip()] - - -class NetcdfRasterReader(RasterGbxReader): - """Raster mode: transcode a CF grid variable to a GeoTIFF tile.""" - - def __init__(self, options: Dict[str, str]): - super().__init__(options) # path/sizeInMB/filterRegex/bbox/bboxCrs - self.variables = _requested_variables(options) - self.group = options.get("group") - - def read(self, partition: "_FilePartition") -> Iterator[Tuple]: - import numpy as np - from rasterio.io import MemoryFile - - from databricks.labs.gbx.ds import _listing - - source = _listing.to_spark_uri(partition.file_path) - var = self.variables[0] # raster mode reads a single variable per tile - with _netcdf.open_dataset(partition.file_path, self.group) as ds: - kind = _netcdf.classify(ds, var) - if kind == _netcdf.CURVILINEAR: - raise ValueError( - f"netcdf_gbx: variable '{var}' in {partition.file_path} is " - f"curvilinear/swath (2-D lat/lon); read it with " - f"option('mode','vector') to get per-cell points." - ) - if kind != _netcdf.GRID: - raise ValueError( - f"netcdf_gbx: variable '{var}' is not a regular grid " - f"({kind}); raster mode supports CF regular/projected grids only." - ) - transform, crs = _netcdf.grid_transform_crs(ds, var) - arr = _netcdf.array_2d(ds, var) - nodata = _netcdf.nodata_of(ds, var) - - h, w = arr.shape[-2], arr.shape[-1] - profile = dict( - driver="GTiff", width=w, height=h, count=1, dtype=str(arr.dtype), - crs=crs, transform=transform, - ) - if nodata is not None: - profile["nodata"] = nodata - # Build an in-memory rasterio dataset, then reuse the shared encode_tile so - # the 11-key metadata + GTiff re-encode stay DRY with the other readers. - with MemoryFile() as mf: - with mf.open(**profile) as out: - out.write(arr.astype(profile["dtype"]), 1) - with mf.open() as rds: - cellid, raster_bytes, meta = _encode.encode_tile( - rds, window=(0, 0, w, h), - source_path=partition.file_path, all_parents="", - ) - yield (source, (cellid, raster_bytes, meta)) - - -class NetcdfGbxDataSource(DataSource): - @classmethod - def name(cls) -> str: - return "netcdf_gbx" - - def _mode(self) -> str: - return self.options.get("mode", "raster").lower() - - def schema(self) -> StructType: - mode = self._mode() - if mode == "raster": - return reader_schema() - if mode == "vector": - from databricks.labs.gbx.ds._netcdf_vector import NetcdfVectorReader - return NetcdfVectorReader(self.options).schema() - raise ValueError( - f"netcdf_gbx: unknown mode={mode!r} (use 'raster' or 'vector')." - ) - - def reader(self, schema: StructType) -> DataSourceReader: - mode = self._mode() - if mode == "raster": - return NetcdfRasterReader(self.options) - if mode == "vector": - from databricks.labs.gbx.ds._netcdf_vector import NetcdfVectorReader - return NetcdfVectorReader(self.options) - raise ValueError( - f"netcdf_gbx: unknown mode={mode!r} (use 'raster' or 'vector')." - ) -``` - -- [ ] **Step 4: Run the tests to verify they pass** - -Run: `pytest test/ds/test_netcdf_datasource.py -v` -Expected: PASS for `test_raster_schema_matches_tile_schema`, `test_raster_read_round_trip`, `test_raster_mode_rejects_curvilinear`. (Vector tests do not exist yet.) - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/netcdf.py python/geobrix/test/ds/test_netcdf_datasource.py -git commit -m "feat(ds): netcdf_gbx raster mode (CF grid -> GeoTIFF tile) - -NetcdfGbxDataSource branches on the mode option (schema + reader), like -VectorGbxDataSource. Raster mode transcodes a regular-grid variable to -the shared (source, tile) struct via the existing encode_tile path; -curvilinear/unsupported geometries raise an actionable error. - -Co-authored-by: Isaac" -``` - ---- - -### Task 4: Vector mode — `NetcdfVectorReader` - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/_netcdf_vector.py` -- Modify: `python/geobrix/test/ds/test_netcdf_datasource.py` (add vector tests) - -**Interfaces:** -- Consumes: `_netcdf.point_arrays`, `_netcdf.np_to_spark`, `_netcdf.classify`, `_listing`; `shapely`. -- Produces: `NetcdfVectorReader` (`schema()` returns attrs + `geom_0` WKB + `geom_0_srid` + `geom_0_srid_proj`; `read()` yields row tuples). Consumed by `NetcdfGbxDataSource` (already wired in Task 3). - -- [ ] **Step 1: Write the failing tests (vector mode)** - -Append to `python/geobrix/test/ds/test_netcdf_datasource.py`: - -```python -def _write_points(path): - with Dataset(path, "w") as ds: - ds.createDimension("obs", 5) - lat = ds.createVariable("latitude", "f8", ("obs",)) - lon = ds.createVariable("longitude", "f8", ("obs",)) - lat.standard_name = "latitude" - lon.standard_name = "longitude" - lat[:] = [50.0, 50.1, 50.2, 50.3, 50.4] - lon[:] = [10.0, 10.1, 10.2, 10.3, 10.4] - v = ds.createVariable("ch4", "f4", ("obs",)) - v[:] = np.arange(5, dtype="float32") - qa = ds.createVariable("qa_value", "i4", ("obs",)) - qa[:] = np.array([0, 1, 0, 1, 1], dtype="int32") - - -def test_vector_schema_columns(tmp_path): - from databricks.labs.gbx.ds._netcdf_vector import NetcdfVectorReader - f = tmp_path / "pts.nc" - _write_points(str(f)) - reader = NetcdfVectorReader( - {"path": str(f), "variables": "ch4,qa_value"} - ) - schema = reader.schema() - assert [f.name for f in schema.fields] == [ - "ch4", "qa_value", "geom_0", "geom_0_srid", "geom_0_srid_proj", - ] - from pyspark.sql.types import BinaryType, FloatType, IntegerType, StringType - assert isinstance(schema["ch4"].dataType, FloatType) - assert isinstance(schema["qa_value"].dataType, IntegerType) - assert isinstance(schema["geom_0"].dataType, BinaryType) - assert isinstance(schema["geom_0_srid"].dataType, StringType) - - -def test_vector_read_dsg_points(spark, tmp_path): - import shapely - f = tmp_path / "pts.nc" - _write_points(str(f)) - spark.dataSource.register(NetcdfGbxDataSource) - df = ( - spark.read.format("netcdf_gbx") - .option("mode", "vector") - .option("variables", "ch4,qa_value") - .load(str(f)) - ) - rows = df.orderBy("ch4").collect() - assert len(rows) == 5 - assert rows[0]["geom_0_srid"] == "4326" - pt = shapely.from_wkb(bytes(rows[0]["geom_0"])) - assert pt.x == pytest.approx(10.0) and pt.y == pytest.approx(50.0) - assert rows[1]["qa_value"] == 1 # ch4==1 -> qa 1 - - -def test_vector_read_curvilinear_to_points(spark, tmp_path): - f = tmp_path / "curv.nc" - _write_curvilinear(str(f)) - spark.dataSource.register(NetcdfGbxDataSource) - df = ( - spark.read.format("netcdf_gbx") - .option("mode", "vector") - .option("variables", "ch4") - .load(str(f)) - ) - assert df.count() == 6 # one point per cell (2x3) -``` - -- [ ] **Step 2: Run the tests to verify they fail** - -Run: `pytest test/ds/test_netcdf_datasource.py -k vector -v` -Expected: FAIL — `ModuleNotFoundError: databricks.labs.gbx.ds._netcdf_vector`. - -- [ ] **Step 3: Implement `ds/_netcdf_vector.py`** - -Create `python/geobrix/src/databricks/labs/gbx/ds/_netcdf_vector.py`: - -```python -"""netcdf_gbx vector mode: DSG points, or any 2-D field coerced to per-cell points. - -Output schema mirrors the light vector reader convention: attribute columns (one -per requested variable, typed) then geom_0 (plain WKB) + geom_0_srid + -geom_0_srid_proj string columns. SRID travels in the string column, not as EWKB. -""" - -from __future__ import annotations - -from typing import Dict, Iterator, List, Optional, Sequence, Tuple - -from pyspark.sql.datasource import DataSourceReader, InputPartition -from pyspark.sql.types import BinaryType, StringType, StructField, StructType - -from databricks.labs.gbx.ds import _listing, _netcdf - - -class _NcFilePartition(InputPartition): - def __init__(self, file_path: str): - self.file_path = file_path - - -class NetcdfVectorReader(DataSourceReader): - def __init__(self, options: Dict[str, str]): - self.path = options.get("path") - if not self.path: - raise ValueError("netcdf_gbx requires a 'path' (e.g. .load(path)).") - raw = options.get("variables") or options.get("variable") - if not raw: - raise ValueError( - "netcdf_gbx vector mode requires a 'variables' option naming the " - "NetCDF variable(s) to emit as point attributes." - ) - self.variables: List[str] = [v.strip() for v in str(raw).split(",") if v.strip()] - self.group: Optional[str] = options.get("group") - self.filter_regex = options.get("filterRegex", ".*") - - def _members(self) -> List[str]: - return _listing.list_files(self.path, self.filter_regex) - - def schema(self) -> StructType: - member = self._members()[0] - fields: List[StructField] = [] - with _netcdf.open_dataset(member, self.group) as ds: - for name in self.variables: - fields.append( - StructField(name, _netcdf.np_to_spark(ds[name].values.dtype), True) - ) - fields.append(StructField("geom_0", BinaryType(), True)) - fields.append(StructField("geom_0_srid", StringType(), True)) - fields.append(StructField("geom_0_srid_proj", StringType(), True)) - return StructType(fields) - - def partitions(self) -> Sequence[InputPartition]: - return [_NcFilePartition(f) for f in self._members()] - - def read(self, partition: "_NcFilePartition") -> Iterator[Tuple]: - import shapely - - with _netcdf.open_dataset(partition.file_path, self.group) as ds: - kind = _netcdf.classify(ds, self.variables[0]) - if kind == _netcdf.UNSUPPORTED: - raise ValueError( - f"netcdf_gbx: variable '{self.variables[0]}' in " - f"{partition.file_path} has no per-pixel lon/lat (sensor " - f"geometry / unsupported); orthorectify it first." - ) - lon, lat, attrs, srid = _netcdf.point_arrays(ds, self.variables) - - wkb = shapely.to_wkb(shapely.points(lon, lat)) - proj = f"EPSG:{srid}" - n = len(lon) - for i in range(n): - row = tuple(attrs[name][i].item() for name in self.variables) - yield row + (bytes(wkb[i]), srid, proj) -``` - -- [ ] **Step 4: Run the tests to verify they pass** - -Run: `pytest test/ds/test_netcdf_datasource.py -v` -Expected: PASS (all raster + vector tests). - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/_netcdf_vector.py python/geobrix/test/ds/test_netcdf_datasource.py -git commit -m "feat(ds): netcdf_gbx vector mode (points + swath->per-cell points) - -DSG point data reads natively; any 2-D field (incl. curvilinear swath) -is coerced to one point per cell. Output matches the light vector schema -(attrs + geom_0 WKB + srid columns), so gbx_st_*/H3 compose directly. -Requested variables pass through untouched (e.g. S5P qa_value). - -Co-authored-by: Isaac" -``` - ---- - -### Task 5: Register `netcdf_gbx` - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/register.py` (imports 8-25; `_SOURCES` 27-37) -- Test: `python/geobrix/test/ds/test_netcdf_datasource.py` (add a register test) - -**Interfaces:** -- Consumes: `NetcdfGbxDataSource` (Task 3). -- Produces: `netcdf_gbx` resolvable via `spark.read.format("netcdf_gbx")` after `register(spark)` and via `register(spark, only=["netcdf"])`. - -- [ ] **Step 1: Write the failing test** - -Append to `python/geobrix/test/ds/test_netcdf_datasource.py`: - -```python -def test_register_exposes_netcdf_gbx(spark, tmp_path): - from databricks.labs.gbx.ds.register import register - f = tmp_path / "grid.nc" - _write_regular_grid(str(f)) - register(spark, only=["netcdf"]) - df = spark.read.format("netcdf_gbx").option("variable", "ch4").load(str(f)) - assert df.count() == 1 -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `pytest test/ds/test_netcdf_datasource.py::test_register_exposes_netcdf_gbx -v` -Expected: FAIL — `register(... only=["netcdf"])` raises `ValueError` (unknown format) because `netcdf_gbx` is not in `_SOURCES`. - -- [ ] **Step 3: Wire into `register.py`** - -Add the import alongside the others (after the `gtiff` import, line 15): - -```python -from databricks.labs.gbx.ds.netcdf import NetcdfGbxDataSource -``` - -Add it to `_SOURCES` (after `GTiffGbxDataSource,`): - -```python - NetcdfGbxDataSource, -``` - -- [ ] **Step 4: Run to verify it passes** - -Run: `pytest test/ds/test_netcdf_datasource.py -v` -Expected: PASS (all tests, including the register test). - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/register.py python/geobrix/test/ds/test_netcdf_datasource.py -git commit -m "feat(ds): register netcdf_gbx DataSource - -Adds NetcdfGbxDataSource to the light _SOURCES tuple so -spark.read.format('netcdf_gbx') resolves after register(spark). - -Co-authored-by: Isaac" -``` - ---- - -### Task 6: `TropomiDownloader` (`gbx.sample`) - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/sample/tropomi.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/__init__.py` -- Test: `python/geobrix/test/sample/test_tropomi.py` - -**Interfaces:** -- Consumes: `StacClient` (injectable via `_stac_client`); `netcdf_gbx` reader (Task 5) in `read()`. -- Produces: `TropomiDownloader` (`discover`/`download`/`read`) + `download_tropomi_aoi`, re-exported from `sample/__init__.py`. - -**Verify before finalizing (spec risk R4):** confirm the Planetary Computer collection id (`sentinel-5p-l2-netcdf`), the CH4 asset name, and the `/PRODUCT/` group + variable names (`methane_mixing_ratio_bias_corrected`, `qa_value`) against a real granule. Adjust the constants below if they differ. - -- [ ] **Step 1: Write the failing test (offline, injected StacClient)** - -Create `python/geobrix/test/sample/test_tropomi.py`: - -```python -"""Offline unit tests for TropomiDownloader (injected StacClient, no network).""" - -from unittest.mock import MagicMock - -from databricks.labs.gbx.sample.tropomi import TropomiDownloader - - -def test_discover_filters_to_ch4_asset(spark): - fake = MagicMock() - search_df = spark.createDataFrame( - [("S5P_1", "ch4", [10.0, 49.0, 12.0, 51.0], "https://x/ch4.nc"), - ("S5P_1", "other", [10.0, 49.0, 12.0, 51.0], "https://x/other.nc")], - ["item_id", "asset_name", "item_bbox", "href"], - ) - fake.search.return_value = search_df - dl = TropomiDownloader(_stac_client=fake) - out = dl.discover([10.0, 49.0, 12.0, 51.0], spark=spark) - assert out.count() == 1 - assert out.first()["asset_name"] == "ch4" - - -def test_download_delegates_to_stacclient(spark): - fake = MagicMock() - search_df = spark.createDataFrame( - [("S5P_1", "ch4", "https://x/ch4.nc")], - ["item_id", "asset_name", "href"], - ) - fake.search.return_value = search_df - fake.download.return_value = spark.createDataFrame( - [("S5P_1", "ch4", "/vol/ch4.nc", 123, True)], - ["item_id", "asset_name", "out_file_path", "out_file_sz", "is_out_file_valid"], - ) - dl = TropomiDownloader(_stac_client=fake) - res = dl.download([10.0, 49.0, 12.0, 51.0], "/vol/out", spark=spark) - assert res.count() == 1 - fake.download.assert_called_once() -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `pytest test/sample/test_tropomi.py -v` -Expected: FAIL — `ModuleNotFoundError: databricks.labs.gbx.sample.tropomi`. - -- [ ] **Step 3: Implement `sample/tropomi.py`** - -Create `python/geobrix/src/databricks/labs/gbx/sample/tropomi.py` (structure cloned from `sample/dem.py`): - -```python -"""TropomiDownloader — AOI-driven Sentinel-5P L2 CH4 staging via Planetary Computer. - -Mirrors DemDownloader: driver-side discovery (metadata-only), then DISTRIBUTED -asset download via StacClient.download(). S5P L2 CH4 is netCDF-4 swath — read() -loads it through the netcdf_gbx reader in VECTOR mode (per-pixel points). - -ONLINE-ONLY (pystac-client + planetary-computer). Injection seam: _stac_client. -Serverless-safe: no spark.conf.set/_jvm/.rdd/cache/persist. -""" - -from __future__ import annotations - -from typing import Optional, Sequence - -PLANETARY_COMPUTER = "https://planetarycomputer.microsoft.com/api/stac/v1" -# VERIFY (R4): collection id, CH4 asset name, group, and variable names. -S5P_COLLECTION = "sentinel-5p-l2-netcdf" -_CH4_ASSET = "ch4" -_S5P_GROUP = "/PRODUCT" -_S5P_VARIABLES = "methane_mixing_ratio_bias_corrected,qa_value" -_S5P_DATETIME = "2018-01-01/2030-01-01" - - -def _bbox_to_geojson_polygon(bbox: Sequence[float]) -> str: - import json - - minx, miny, maxx, maxy = bbox - coords = [[minx, miny], [maxx, miny], [maxx, maxy], [minx, maxy], [minx, miny]] - return json.dumps({"type": "Polygon", "coordinates": [coords]}) - - -class TropomiDownloader: - def __init__( - self, - catalog: str = PLANETARY_COMPUTER, - sign: str = "planetary_computer", - collection: str = S5P_COLLECTION, - asset: str = _CH4_ASSET, - _stac_client=None, - ): - self.catalog = catalog - self.sign = sign - self.collection = collection - self.asset = asset - self._stac_client = _stac_client - - def _get_stac_client(self): - if self._stac_client is not None: - return self._stac_client - from databricks.labs.gbx.stac import StacClient - - return StacClient(catalog=self.catalog, sign=self.sign) - - def _aoi_dataframe(self, bbox: Sequence[float], spark=None): - from pyspark.sql import SparkSession - - spark = spark or SparkSession.getActiveSession() - return spark.createDataFrame([(_bbox_to_geojson_polygon(bbox),)], ["geojson"]) - - def discover(self, bbox: Sequence[float], spark=None): - from pyspark.sql import SparkSession - from pyspark.sql import functions as F - - spark = spark or SparkSession.getActiveSession() - client = self._get_stac_client() - raw = client.search( - self._aoi_dataframe(bbox, spark), - geojson_col="geojson", - collections=[self.collection], - datetime=_S5P_DATETIME, - ) - return ( - raw.filter(F.col("asset_name") == self.asset) - .select("item_id", "asset_name", "item_bbox", "href") - .distinct() - ) - - def download( - self, - bbox: Sequence[float], - out_dir: str, - bbox_crs: str = "EPSG:4326", - partitions: Optional[int] = None, - spark=None, - ): - from pyspark.sql import SparkSession - from pyspark.sql import functions as F - - spark = spark or SparkSession.getActiveSession() - client = self._get_stac_client() - raw = client.search( - self._aoi_dataframe(bbox, spark), - geojson_col="geojson", - collections=[self.collection], - datetime=_S5P_DATETIME, - ) - granules = raw.filter(F.col("asset_name") == self.asset).select( - "item_id", "asset_name", "href" - ) - return client.download( - granules, out_dir, bbox=list(bbox), bbox_crs=bbox_crs, partitions=partitions - ) - - def read( - self, - out_dir: str, - variables: str = _S5P_VARIABLES, - group: str = _S5P_GROUP, - spark=None, - ): - from pyspark.sql import SparkSession - from pyspark.sql import functions as F - - spark = spark or SparkSession.getActiveSession() - return ( - spark.read.format("netcdf_gbx") - .option("mode", "vector") - .option("group", group) - .option("variables", variables) - .option("filterRegex", r".*\.nc$") - .load(out_dir) - .repartition(64, F.col("geom_0_srid")) - ) - - -def download_tropomi_aoi(spark, bbox: Sequence[float], out_dir: str, **kw): - """One-shot: default TropomiDownloader + download S5P CH4 for an AOI.""" - return TropomiDownloader().download(bbox, out_dir, spark=spark, **kw) -``` - -- [ ] **Step 4: Export from `sample/__init__.py`** - -In `python/geobrix/src/databricks/labs/gbx/sample/__init__.py`, add the import (after the `dem` import) and the `__all__` entries: - -```python -from databricks.labs.gbx.sample.tropomi import TropomiDownloader, download_tropomi_aoi -``` -and add `"TropomiDownloader",` and `"download_tropomi_aoi",` to the `__all__` list (keep it sorted with the existing entries). - -- [ ] **Step 5: Run tests to verify they pass** - -Run: `pytest test/sample/test_tropomi.py -v` -Expected: PASS (2 tests). Also `python -c "from databricks.labs.gbx.sample import TropomiDownloader, download_tropomi_aoi"` succeeds. - -- [ ] **Step 6: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/sample/tropomi.py python/geobrix/src/databricks/labs/gbx/sample/__init__.py python/geobrix/test/sample/test_tropomi.py -git commit -m "feat(sample): TropomiDownloader for S5P L2 CH4 (vector-mode proof) - -Clones the DemDownloader shape (discover/download/read, Serverless-safe, -injectable StacClient). read() loads granules via netcdf_gbx vector mode -(S5P is netCDF-4 swath -> per-pixel points). Constants flagged for -against-granule verification (R4). - -Co-authored-by: Isaac" -``` - ---- - -### Task 7: Docs — `readers/netcdf.mdx` + doc-test + sidebar - -**Files:** -- Create: `docs/docs/readers/netcdf.mdx` -- Create: `docs/tests/python/readers/netcdf_gbx_read_examples.py` -- Create: `docs/tests/python/readers/test_netcdf_gbx_read_examples.py` -- Modify: `docs/sidebars.js` (the "Named" readers array) - -**Interfaces:** -- Consumes: the registered `netcdf_gbx` reader (Task 5). -- Produces: a docs page rendering raster + vector examples, and a doc-test asserting they run. - -- [ ] **Step 1: Write the doc-example source (named constants + verifiers)** - -Create `docs/tests/python/readers/netcdf_gbx_read_examples.py` (mirror `raster_gbx_read_examples.py`): - -```python -"""Executable examples for the netcdf_gbx reader docs (imported via raw-loader).""" - -# --8<-- [start:register] -from databricks.labs.gbx.ds.register import register - -register(spark) # noqa: F821 (spark provided by the notebook/test) -# --8<-- [end:register] - - -# --8<-- [start:read_raster] -grid = ( - spark.read.format("netcdf_gbx") # noqa: F821 - .option("variable", "t2m") # a regular lat/lon grid variable (e.g. ERA5) - .load("/Volumes/main/geobrix_samples/netcdf/era5_sample.nc") -) -# grid has the standard (source, tile) raster schema -# --8<-- [end:read_raster] - - -# --8<-- [start:read_vector] -points = ( - spark.read.format("netcdf_gbx") # noqa: F821 - .option("mode", "vector") - .option("group", "/PRODUCT") - .option("variables", "methane_mixing_ratio_bias_corrected,qa_value") - .load("/Volumes/main/geobrix_samples/netcdf/s5p_ch4_sample.nc") -) -# points has: , geom_0 (WKB), geom_0_srid, geom_0_srid_proj -# --8<-- [end:read_vector] - - -def netcdf_gbx_raster_example(spark, path): - register(spark) - return spark.read.format("netcdf_gbx").option("variable", "t2m").load(path) - - -def netcdf_gbx_vector_example(spark, path, variables, group): - register(spark) - return ( - spark.read.format("netcdf_gbx") - .option("mode", "vector").option("group", group) - .option("variables", variables).load(path) - ) -``` - -- [ ] **Step 2: Write the doc-test wrapper (fails first — no fixtures yet)** - -Create `docs/tests/python/readers/test_netcdf_gbx_read_examples.py`: - -```python -"""Doc-test: netcdf_gbx examples run against synthesized NetCDF fixtures.""" - -import numpy as np -from netCDF4 import Dataset - -from netcdf_gbx_read_examples import ( - netcdf_gbx_raster_example, - netcdf_gbx_vector_example, -) - - -def _write_grid(path): - with Dataset(path, "w") as ds: - ds.createDimension("lat", 3) - ds.createDimension("lon", 4) - lat = ds.createVariable("lat", "f8", ("lat",)); lat.standard_name = "latitude" - lon = ds.createVariable("lon", "f8", ("lon",)); lon.standard_name = "longitude" - lat[:] = [50.0, 49.5, 49.0]; lon[:] = [10.0, 10.5, 11.0, 11.5] - v = ds.createVariable("t2m", "f4", ("lat", "lon")); v[:] = np.arange(12).reshape(3, 4) - - -def _write_points(path): - with Dataset(path, "w") as ds: - ds.createDimension("obs", 4) - lat = ds.createVariable("latitude", "f8", ("obs",)); lat.standard_name = "latitude" - lon = ds.createVariable("longitude", "f8", ("obs",)); lon.standard_name = "longitude" - lat[:] = [50.0, 50.1, 50.2, 50.3]; lon[:] = [10.0, 10.1, 10.2, 10.3] - ds.createVariable("methane_mixing_ratio_bias_corrected", "f4", ("obs",))[:] = np.arange(4) - ds.createVariable("qa_value", "i4", ("obs",))[:] = np.array([1, 1, 0, 1]) - - -def test_raster_example_runs(spark, tmp_path): - p = str(tmp_path / "grid.nc"); _write_grid(p) - assert netcdf_gbx_raster_example(spark, p).count() == 1 - - -def test_vector_example_runs(spark, tmp_path): - p = str(tmp_path / "pts.nc"); _write_points(p) - df = netcdf_gbx_vector_example( - spark, p, "methane_mixing_ratio_bias_corrected,qa_value", None - ) - assert df.count() == 4 - assert "geom_0" in df.columns -``` - -- [ ] **Step 3: Run the doc-test to verify it passes** - -Run (in Docker per the doc-test convention): `gbx:test:python-docs --path docs/tests/python/readers/test_netcdf_gbx_read_examples.py` -Expected: PASS (2 tests). (If run outside Docker, use `pytest docs/tests/python/readers/test_netcdf_gbx_read_examples.py`.) - -- [ ] **Step 4: Write the MDX page** - -Create `docs/docs/readers/netcdf.mdx` (mirror `docs/docs/readers/geotiff.mdx` front-matter + `!!raw-loader!` import + Options table). Include: -- front-matter `sidebar_label: netcdf_gbx`, an appropriate `sidebar_position`. -- Intro: lightweight NetCDF reader, two modes, class-1/2 grids in raster mode, points/swath in vector mode, class 4 rejected. -- Options table: `mode` (raster|vector), `variable`/`variables`, `group`, and the inherited `bbox`/`bboxCrs`/`filterRegex`/`sizeInMB`. -- Two `` snippets pulling the `read_raster` and `read_vector` regions from `netcdf_gbx_read_examples.py`. -- A short "GeoBrix voice" note (no internal vocabulary): S5P swath is read as points; ERA5 grids as tiles. - -- [ ] **Step 5: Add the sidebar entry** - -In `docs/sidebars.js`, add `"readers/netcdf"` to the "Named" readers array (next to `"readers/geotiff"`). - -- [ ] **Step 6: Commit** - -```bash -git add docs/docs/readers/netcdf.mdx docs/tests/python/readers/netcdf_gbx_read_examples.py docs/tests/python/readers/test_netcdf_gbx_read_examples.py docs/sidebars.js -git commit -m "docs(readers): document netcdf_gbx (raster + vector modes) - -Single-source doc-test examples (raster grid + vector points), MDX page -with an options table, and a sidebar entry under Named readers. - -Co-authored-by: Isaac" -``` - ---- - -## Notes for the implementer - -- **R1 (mode-dependent schema)** is handled: `NetcdfGbxDataSource.schema()` reads `self.options["mode"]` and returns either the tile schema or the vector schema — the same mechanism `VectorGbxDataSource.schema()` uses (`ds/vector.py:779`). PySpark passes options to `schema()` via `self.options`. -- **R2 (vector schema)** matches the light vector reader exactly: attribute columns in requested order, then `geom_0` (plain WKB `BinaryType`), `geom_0_srid`, `geom_0_srid_proj` (both `StringType`). Not EWKB. -- **R3 (Serverless `netcdf4` pin)** is a deploy-time concern; Task 1 pins a floor and documents adding a ceiling if env-v5 floats it. -- **R4 (S5P names)** — verify collection/asset/group/variable names against a real granule in Task 6 before finalizing the constants. -- **Local test env:** the `ds/conftest.py` fixture sets `PYSPARK_PYTHON`/`PYSPARK_DRIVER_PYTHON = sys.executable`; a `test/sample/conftest.py` may need the same `spark` fixture — reuse or import the `ds` one. If `test/sample/` lacks a Spark fixture, add a minimal `conftest.py` mirroring `test/ds/conftest.py`'s `spark` fixture. -- **Raster encode reuse:** `NetcdfRasterReader.read` builds an in-memory rasterio dataset then calls `_encode.encode_tile(...)` so the 11-key metadata and GTiff re-encode stay DRY. The extra in-memory encode is negligible for the initial reader. -- **`sizeInMB` in raster mode:** the option is accepted (inherited via `RasterGbxReader.__init__`) but the initial raster read emits a **single whole tile per variable/file** — it does not sub-tile. NetCDF grids in scope (ERA5-class) are modest, so this is acceptable for the initial cut; sub-tiling large grids is a follow-up (pairs naturally with the multi-time-step fan-out increment). -- **`array_2d` north-up vs vector order:** raster mode reorders to north-up (descending latitude) to match the GeoTIFF convention; vector mode leaves points in stored order (each point carries its own coordinate, so order is irrelevant). Both are correct. -- **Tier gating — do NOT add markers or import-skips.** The repo separates tiers by directory: `test/conftest.py::_LIGHT_TEST_DIRS` + `collect_ignore` drop `ds/`, `sample/`, etc. when `rasterio` is absent (heavy env). `test_gtiff_datasource.py` does an unguarded `import rasterio`; match that convention — a plain `from netCDF4 import Dataset` at module top is fine because `ds/` and `sample/` are already gated. Since netcdf4 is now pinned in all three light envs (Task 1), it is always present wherever `ds/`/`sample/` are collected. Keep the new test files under `test/ds/` and `test/sample/` — placing them elsewhere would run them in the heavy tier and fail on the module-level netCDF4 import. diff --git a/docs/superpowers/plans/2026-07-10-vapor-eyes-B1-downloaders.md b/docs/superpowers/plans/2026-07-10-vapor-eyes-B1-downloaders.md deleted file mode 100644 index e8cbb31be..000000000 --- a/docs/superpowers/plans/2026-07-10-vapor-eyes-B1-downloaders.md +++ /dev/null @@ -1,468 +0,0 @@ -# vapor-eyes Plan B1 — sample-data downloaders (EmitDownloader + WellsDownloader) - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax. - -**Goal:** Add two `databricks.labs.gbx.sample` downloaders that stage vapor-eyes data to a Unity Catalog Volume with read-validation + idempotent missing-asset recovery: `EmitDownloader` (NASA EMIT CH4 via `earthaccess`) and `WellsDownloader` (TX RRC well pads via ArcGIS REST). - -**Architecture:** Both mirror the existing `NaipDownloader`/`DemDownloader`/`TropomiDownloader` shape (`discover`/`download`/`read`/`repair`, Serverless-safe, injectable client for offline tests) and reproduce `StacClient.download`'s 6-column result contract. They differ only in the fetch engine: EMIT uses `earthaccess` (Earthdata-Login-authenticated, **driver-side** download — fine at demo scale), wells use a paged ArcGIS-REST GeoJSON query. COG validation reuses `stac._download.read_validate`; GeoJSON gets a small JSON validator. This is Plan B1 of Spec B; Plan B2 (config_nb + NB01–05 + README/diagrams) consumes these. - -**Tech Stack:** Python 3.12, PySpark 4, `earthaccess`, `requests` (both new-ish for `[light]`/sample — see Task 0), `rasterio`/`shapely` (existing), the `geojson_gbx`/`raster_gbx` light readers. Design ref: `docs/superpowers/specs/2026-07-10-vapor-eyes-series-design.md` §5. - -## Global Constraints - -- **Serverless-safe:** no `spark.conf.set`, `_jvm`, `.rdd`, `.cache()`, `.persist()` in `sample/emit.py` or `sample/wells.py`. -- **Result contract (reproduce StacClient.download exactly):** the `download()` DataFrame has columns `item_id: string`, `asset_name: string`, `out_file_path: string`, `out_file_sz: long`, `is_out_file_valid: boolean`, `last_update: timestamp`. `is_out_file_valid` is true iff the file exists, is above a 1 KB floor, and passes a type-appropriate read (rasterio for `.tif`, JSON parse + non-empty `features` for `.json`). -- **Idempotent + repair:** skip assets already present and valid; `repair(target, where="is_out_file_valid = false")` re-fetches only invalid/missing rows and MERGEs into a Delta table by `(item_id, asset_name)` (mirrors `StacClient.repair`). `FORCE_REBUILD`-style full refetch via a `force` flag. -- **Injection seam:** `_earthaccess=` (EmitDownloader) / `_get=` (WellsDownloader) constructor params let offline tests bypass network — mirror `_MockStacClient` record-and-return. -- **Credential:** EMIT auth via `earthaccess.login(strategy="environment")` reading `EARTHDATA_TOKEN` (set from a Databricks secret). Never require username/password or `.netrc` on workers. -- **Tests live in** `python/geobrix/test/sample/` (already in `_LIGHT_TEST_DIRS`); no conftest/marker change. Each test module defines its own `spark` fixture (mirror `test_naip.py`) and uses `pytest.importorskip("pyspark")`. -- **Commit hygiene:** subject ≤72 chars + WHY body; end with `Co-authored-by: Isaac`. Branch: `examples/vapor-eyes`. - ---- - -### Task 0: Add `earthaccess` to the light dependency set (three envs) - -**Files:** `python/geobrix/pyproject.toml` (`[light]`... or a new note), `requirements-pyrx-ci.in`, `requirements-dev-container.in` (+ regenerate both hashed `.txt`). - -**Interfaces:** Produces `import earthaccess` in all three light envs (CI, Docker, `.venv-pyrx`). `requests` is already pinned (2.32.3). Follows [[new-feature-dep-and-tier-checklist]] / [[light-ci-lock-completeness]]. - -- [ ] **Step 1:** Add to `pyproject.toml` `[light]` (after the netcdf4 line): `"earthaccess>=0.11,<1",` with a comment (NASA EMIT auth+download for the sample EmitDownloader). -- [ ] **Step 2:** Add `earthaccess==` to both `requirements-pyrx-ci.in` and `requirements-dev-container.in` (exact pin; check cp312 wheels + that it doesn't float `requests`/`fsspec` off the DBR base — cap if needed). -- [ ] **Step 3:** Regenerate both hashed locks (container, Linux): - ```bash - docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && for f in requirements-pyrx-ci requirements-dev-container; do uv pip compile --generate-hashes --python-version 3.12 --index-url https://pypi-proxy.dev.databricks.com/simple --output-file $f.txt $f.in; done' - ``` - Inspect `git diff` for version-line changes — only `earthaccess` + its new transitives should appear; investigate any bump to an existing pin. -- [ ] **Step 4:** Install into the container test env + verify: `docker exec geobrix-dev bash -lc 'python3 -m pip install --break-system-packages --quiet earthaccess== && python3 -c "import earthaccess; print(earthaccess.__version__)"'` -- [ ] **Step 5:** Commit (`build(light): add earthaccess for EmitDownloader`). - ---- - -### Task 1: `EmitDownloader.discover` — earthaccess search → asset rows - -**Files:** Create `python/geobrix/src/databricks/labs/gbx/sample/emit.py`; Test `python/geobrix/test/sample/test_emit.py`. - -**Interfaces:** -- Consumes: injected `_earthaccess` (a module-like object with `login`, `search_data`, `download`); PySpark. -- Produces: `EmitDownloader(enh_short="EMITL2BCH4ENH", plm_short="EMITL2BCH4PLM", version="002", _earthaccess=None)`; `discover(bbox, temporal=None, spark=None) -> DataFrame[item_id, asset_name, href]`. `asset_name` ∈ {`"ch4enh"`, `"plm_cog"`, `"plm_geojson"`}. Used by Tasks 2–3. - -- [ ] **Step 1: Write the failing test.** Create `test/sample/test_emit.py`: -```python -"""Offline unit tests for EmitDownloader (injected earthaccess; no network).""" -from __future__ import annotations -import pytest -pyspark = pytest.importorskip("pyspark") -from pyspark.sql import SparkSession # noqa: E402 -from databricks.labs.gbx.sample.emit import EmitDownloader # noqa: E402 - - -@pytest.fixture(scope="module") -def spark(): - s = (SparkSession.builder.master("local[2]").appName("emit-test") - .config("spark.sql.shuffle.partitions", "4").getOrCreate()) - yield s - s.stop() - - -class _FakeGranule: - def __init__(self, name, links): - self._name = name - self._links = links - def __getitem__(self, k): # earthaccess granules are dict-like for umm - raise KeyError(k) - def data_links(self, access=None, in_region=False): - return self._links - - -class _FakeEarthaccess: - """Records login/search/download; returns controlled granules.""" - def __init__(self, granules_by_short): - self._by_short = granules_by_short - self.login_calls = 0 - self.download_calls = [] - def login(self, strategy="all", persist=False): - self.login_calls += 1 - return object() - def search_data(self, short_name=None, version=None, bounding_box=None, - temporal=None, count=-1, **kw): - return list(self._by_short.get(short_name, [])) - def download(self, granules, local_path=None, threads=8, **kw): - self.download_calls.append({"n": len(granules), "local_path": local_path}) - return [f"{local_path}/f{i}" for i in range(len(granules))] - - -def _fake_ea(): - enh = _FakeGranule("EMIT_ENH_1", [ - "https://x/EMIT_L2B_CH4ENH_002_20240823T1_o_s.tif", - "https://x/EMIT_L2B_CH4UNCERT_002_20240823T1_o_s.tif"]) - plm = _FakeGranule("EMIT_PLM_1", [ - "https://x/EMIT_L2B_CH4PLM_002_20240823T1_p.tif", - "https://x/EMIT_L2B_CH4PLM_002_20240823T1_p.json"]) - return _FakeEarthaccess({"EMITL2BCH4ENH": [enh], "EMITL2BCH4PLM": [plm]}) - - -def test_discover_extracts_enh_and_plm_assets(spark): - dl = EmitDownloader(_earthaccess=_fake_ea()) - df = dl.discover([-103.9, 31.65, -103.4, 32.15], spark=spark) - rows = {(r["asset_name"], r["href"].split("/")[-1]) for r in df.collect()} - assert ("ch4enh", "EMIT_L2B_CH4ENH_002_20240823T1_o_s.tif") in rows - assert ("plm_cog", "EMIT_L2B_CH4PLM_002_20240823T1_p.tif") in rows - assert ("plm_geojson", "EMIT_L2B_CH4PLM_002_20240823T1_p.json") in rows - assert [f.name for f in df.schema.fields] == ["item_id", "asset_name", "href"] -``` - -- [ ] **Step 2: Run → fail** (`ModuleNotFoundError: ...sample.emit`). `pytest test/sample/test_emit.py -k discover -v`. - -- [ ] **Step 3: Implement `discover` in `sample/emit.py`:** -```python -"""EmitDownloader — AOI-driven NASA EMIT CH4 staging via earthaccess. - -Mirrors DemDownloader/NaipDownloader (discover/download/read/repair, -Serverless-safe). EMIT is on NASA LP DAAC (not Planetary Computer), so the search -step uses `earthaccess` (Earthdata Login via EARTHDATA_TOKEN, a Databricks secret) -and downloads are DRIVER-SIDE (fine at demo AOI scale; EMIT auth mints short-lived -per-request credentials that don't fan out cleanly). The result frame reproduces -StacClient.download's 6-column contract so repair()/read() behave identically. -""" -from __future__ import annotations -from typing import List, Optional, Sequence - -ENH_SHORT = "EMITL2BCH4ENH" -PLM_SHORT = "EMITL2BCH4PLM" -VERSION = "002" - - -def _asset_of(url: str) -> Optional[str]: - u = url.lower() - if "ch4enh" in u and u.endswith(".tif"): - return "ch4enh" - if "ch4plm" in u and u.endswith(".tif"): - return "plm_cog" - if "ch4plm" in u and u.endswith(".json"): - return "plm_geojson" - return None - - -class EmitDownloader: - def __init__(self, enh_short=ENH_SHORT, plm_short=PLM_SHORT, version=VERSION, - _earthaccess=None): - self.enh_short = enh_short - self.plm_short = plm_short - self.version = version - self._earthaccess = _earthaccess - - def _ea(self): - if self._earthaccess is not None: - return self._earthaccess - import earthaccess - return earthaccess - - def _login(self): - # strategy="environment" reads EARTHDATA_TOKEN (Databricks secret -> env). - self._ea().login(strategy="environment") - - def _rows(self, bbox, temporal): - ea = self._ea() - w, s, e, n = bbox - out = [] - for short in (self.enh_short, self.plm_short): - for g in ea.search_data(short_name=short, version=self.version, - bounding_box=(w, s, e, n), temporal=temporal): - for url in g.data_links(access="external"): - a = _asset_of(url) - if a is None: - continue - # item_id = the granule file stem (unique per scene/plume). - item_id = url.split("/")[-1].rsplit(".", 1)[0] - out.append((item_id, a, url)) - return out - - def discover(self, bbox: Sequence[float], temporal=None, spark=None): - from pyspark.sql import SparkSession - from pyspark.sql.types import StringType, StructField, StructType - spark = spark or SparkSession.getActiveSession() - schema = StructType([ - StructField("item_id", StringType(), False), - StructField("asset_name", StringType(), False), - StructField("href", StringType(), False)]) - return spark.createDataFrame(self._rows(bbox, temporal), schema) -``` - -- [ ] **Step 4: Run → pass.** `pytest test/sample/test_emit.py -k discover -v` (in the container). -- [ ] **Step 5: Commit** (`feat(sample): EmitDownloader.discover via earthaccess`). - ---- - -### Task 2: `EmitDownloader.download` — driver-side fetch + validate + result frame - -**Files:** Modify `sample/emit.py`; Modify `test/sample/test_emit.py`. - -**Interfaces:** -- Consumes: `discover`; `stac._download.read_validate` (COG) + a new `_valid_geojson`. -- Produces: `download(bbox, out_dir, temporal=None, force=False, spark=None) -> DataFrame` with the 6-column result contract. Used by Task 3 (`read`/`repair`). - -- [ ] **Step 1: Write the failing test.** Append to `test/sample/test_emit.py`: -```python -def test_download_validates_and_builds_result_frame(spark, tmp_path, monkeypatch): - # Fake earthaccess.download writes a real tiny GeoTIFF + a real GeoJSON so - # validation passes; assert the 6-col contract + is_out_file_valid True. - import json, numpy as np, rasterio - from rasterio.transform import from_origin - fake = _fake_ea() - outdir = str(tmp_path / "emit") - - def _dl(granules, local_path=None, threads=8, **kw): - import os - os.makedirs(local_path, exist_ok=True) - paths = [] - for i, _ in enumerate(granules): - p = os.path.join(local_path, f"g{i}.tif") - with rasterio.open(p, "w", driver="GTiff", width=4, height=3, count=1, - dtype="float32", crs="EPSG:4326", - transform=from_origin(-103.9, 32.15, 0.01, 0.01)) as ds: - ds.write(np.ones((3, 4), "float32"), 1) - paths.append(p) - return paths - fake.download = _dl - dl = EmitDownloader(_earthaccess=fake) - res = dl.download([-103.9, 31.65, -103.4, 32.15], outdir, spark=spark) - assert [f.name for f in res.schema.fields] == [ - "item_id", "asset_name", "out_file_path", "out_file_sz", - "is_out_file_valid", "last_update"] - rows = res.collect() - assert len(rows) >= 1 - assert all(r["is_out_file_valid"] for r in rows) - assert all(r["out_file_sz"] > 1024 for r in rows) -``` -(Also add `test_download_marks_missing_invalid`: a `_dl` that writes nothing → `is_out_file_valid == False`, `out_file_path` null.) - -- [ ] **Step 2: Run → fail** (`AttributeError: ... has no attribute 'download'` on EmitDownloader). -- [ ] **Step 3: Implement `download` + validators in `sample/emit.py`:** -```python - def download(self, bbox, out_dir, temporal=None, force=False, spark=None): - import os - from datetime import datetime - from pyspark.sql import SparkSession - from pyspark.sql.types import (BooleanType, LongType, StringType, - StructField, StructType, TimestampType) - spark = spark or SparkSession.getActiveSession() - self._login() - ea = self._ea() - rows = self._rows(bbox, temporal) - os.makedirs(out_dir, exist_ok=True) - results = [] - for item_id, asset, href in rows: - dest = os.path.join(out_dir, os.path.basename(href)) - if not force and _existing_valid(dest, asset): - sz = os.path.getsize(dest) - results.append((item_id, asset, dest, sz, True, datetime.now())) - continue - # earthaccess.download works on granule objects; re-fetch the granule - # whose data_links contains this href, download to out_dir, then map. - try: - ea.download([_HrefGranule(href)], local_path=out_dir) - except Exception: - pass - if _existing_valid(dest, asset): - results.append((item_id, asset, dest, os.path.getsize(dest), True, - datetime.now())) - else: - results.append((item_id, asset, None, 0, False, datetime.now())) - schema = StructType([ - StructField("item_id", StringType(), False), - StructField("asset_name", StringType(), False), - StructField("out_file_path", StringType(), True), - StructField("out_file_sz", LongType(), True), - StructField("is_out_file_valid", BooleanType(), True), - StructField("last_update", TimestampType(), True)]) - return spark.createDataFrame(results, schema) -``` -Add module helpers: -```python -_FLOOR = 1024 - -def _valid_geojson(path: str) -> bool: - import json - try: - with open(path) as fh: - gj = json.load(fh) - return bool(gj.get("features")) - except Exception: - return False - -def _existing_valid(path: str, asset: str) -> bool: - import os - if not (os.path.exists(path) and os.path.getsize(path) > _FLOOR): - return False - if asset == "plm_geojson": - return _valid_geojson(path) - from databricks.labs.gbx.stac._download import read_validate - return read_validate(path) # rasterio window read for the COGs - -class _HrefGranule: - """Minimal granule adapter so earthaccess.download can fetch a single href.""" - def __init__(self, href): - self._href = href - def data_links(self, access=None, in_region=False): - return [self._href] -``` -(Implementer note: confirm the installed `earthaccess.download` accepts objects exposing `data_links`; if it strictly requires real `DataGranule`s, switch `download` to fetch the granule list from `search_data` once and pass matching granules through — the injected fake already models `download(granules, local_path=...)`.) - -- [ ] **Step 4: Run → pass.** `pytest test/sample/test_emit.py -v`. -- [ ] **Step 5: Commit** (`feat(sample): EmitDownloader.download with validate + result frame`). - ---- - -### Task 3: `EmitDownloader` `read_enh` / `read_plumes` / `repair` + export - -**Files:** Modify `sample/emit.py`, `sample/__init__.py`; Modify `test/sample/test_emit.py`, `test/sample/test_sample_bundle.py`. - -**Interfaces:** Produces `read_enh(out_dir, spark=None)` (ENH COGs → `raster_gbx` tiles), `read_plumes(out_dir, spark=None)` (PLM GeoJSON → `geojson_gbx`), `repair(target, where=..., spark=None, out_dir=None)`, and `download_emit_aoi(spark, bbox, out_dir, **kw)`. - -- [ ] **Step 1: Write failing tests** — `test_read_enh_uses_raster_gbx` (write two `.tif` via rasterio into a dir, assert `read_enh` returns a `tile` column, count 2 after filtering to `CH4ENH` files), `test_read_plumes_uses_geojson_gbx` (write a `.json` FeatureCollection, assert a geometry column), and `test_exports_from_sample` (import `EmitDownloader`, `download_emit_aoi` from `databricks.labs.gbx.sample`). -- [ ] **Step 2: Run → fail.** -- [ ] **Step 3: Implement:** -```python - def read_enh(self, out_dir, spark=None): - from pyspark.sql import SparkSession, functions as F - spark = spark or SparkSession.getActiveSession() - return (spark.read.format("raster_gbx") - .option("filterRegex", r".*CH4ENH.*\.tif$") - .load(out_dir).repartition(64, F.col("source")).select("source", "tile")) - - def read_plumes(self, out_dir, spark=None): - from pyspark.sql import SparkSession - spark = spark or SparkSession.getActiveSession() - return (spark.read.format("geojson_gbx") - .option("filterRegex", r".*CH4PLM.*\.json$").load(out_dir)) - - def repair(self, target, where="is_out_file_valid = false", spark=None, out_dir=None): - from pyspark.sql import SparkSession - spark = spark or SparkSession.getActiveSession() - is_table = isinstance(target, str) - df = spark.table(target) if is_table else target - invalid = df.filter(where) - if invalid.count() == 0: - return invalid - # Re-derive bbox/out_dir from context; re-download the invalid hrefs. - # (Store href in the table if you intend to repair — see note.) - ... # implement MERGE mirroring StacClient.repair (by item_id, asset_name) -``` -**Design note for `repair`:** `StacClient.repair` needs `href` present on the table. So the vapor-eyes `emit_scenes` table must carry `href` (add it to the download result or a joined column) for repair to re-fetch. Simplest: `download()` also returns `href`, and NB03 persists it; `repair` re-runs `download` on the invalid subset and MERGEs on `(item_id, asset_name)`. Implement `repair` to accept a DataFrame/table that includes `item_id, asset_name, href` and MERGE the 4 mutable columns, exactly like `StacClient.repair` (reuse its MERGE shape). -```python -def download_emit_aoi(spark, bbox, out_dir, **kw): - return EmitDownloader().download(bbox, out_dir, spark=spark, **kw) -``` -Add both to `sample/__init__.py` imports + `__all__`; add `"EmitDownloader"` + `"download_emit_aoi"` to the `test_sample_package_all` assertion set. -- [ ] **Step 4: Run → pass** (`pytest test/sample/test_emit.py test/sample/test_sample_bundle.py::test_sample_package_all -v`). -- [ ] **Step 5: Commit** (`feat(sample): EmitDownloader read_enh/read_plumes/repair + export`). - ---- - -### Task 4: `WellsDownloader` — paged TX RRC ArcGIS GeoJSON + validate + read - -**Files:** Create `sample/wells.py`; Test `test/sample/test_wells.py`; Modify `sample/__init__.py`, `test/sample/test_sample_bundle.py`. - -**Interfaces:** `WellsDownloader(service_url=WELLSHL_URL, _get=None)`; `download(bbox, out_dir, page_size=1000, spark=None) -> DataFrame[out_file_path, feature_count, is_out_file_valid, last_update]`; `read(out_dir, spark=None)` (GeoJSON → `geojson_gbx`); `discover(bbox, spark=None)` (count only). `_get` injects a fake HTTP getter for offline tests. - -- [ ] **Step 1: Write the failing test.** `test/sample/test_wells.py`: a fake `_get(url, params)` returns a two-page GeoJSON (page 1 `exceededTransferLimit: true` with N features, page 2 fewer, no flag). Assert `download` writes one merged `wells.geojson`, `feature_count` == total, `is_out_file_valid` True; and `_get` was called twice (paging). Plus `test_exports_from_sample`. -- [ ] **Step 2: Run → fail.** -- [ ] **Step 3: Implement `sample/wells.py`:** -```python -"""WellsDownloader — TX RRC well surface-hole locations via ArcGIS REST (GeoJSON). - -Open, no auth. Pages the FeatureServer (resultOffset/resultRecordCount, honoring -exceededTransferLimit), requests f=geojson (ArcGIS reprojects native EPSG:2277 -> -WGS84 lon/lat automatically), merges pages into one GeoJSON on the Volume, and -validates it. read() loads via geojson_gbx. Serverless-safe (driver-side fetch; -one merged file). -""" -from __future__ import annotations -import json -from typing import Optional, Sequence - -WELLSHL_URL = ("https://services3.arcgis.com/8jYUORGmDUL39WkJ/arcgis/rest/" - "services/WellSHL/FeatureServer/0/query") - - -def _default_get(url, params): - import requests - r = requests.get(url, params=params, timeout=120) - r.raise_for_status() - return r.json() - - -class WellsDownloader: - def __init__(self, service_url=WELLSHL_URL, _get=None): - self.service_url = service_url - self._get = _get or _default_get - - def _fetch_all(self, bbox, page_size): - w, s, e, n = bbox - feats, offset = [], 0 - while True: - params = { - "where": "1=1", "geometry": f"{w},{s},{e},{n}", - "geometryType": "esriGeometryEnvelope", "inSR": "4326", - "spatialRel": "esriSpatialRelIntersects", "outFields": "*", - "f": "geojson", "resultOffset": offset, - "resultRecordCount": page_size} - gj = self._get(self.service_url, params) - page = gj.get("features", []) - feats.extend(page) - if not gj.get("exceededTransferLimit") or not page: - break - offset += len(page) - return feats - - def download(self, bbox: Sequence[float], out_dir: str, page_size=1000, spark=None): - import os - from datetime import datetime - from pyspark.sql import SparkSession - spark = spark or SparkSession.getActiveSession() - os.makedirs(out_dir, exist_ok=True) - feats = self._fetch_all(bbox, page_size) - dest = os.path.join(out_dir, "wells.geojson") - with open(dest, "w") as fh: - json.dump({"type": "FeatureCollection", "features": feats}, fh) - valid = len(feats) > 0 and os.path.getsize(dest) > 1024 - return spark.createDataFrame( - [(dest, len(feats), valid, datetime.now())], - "out_file_path string, feature_count long, " - "is_out_file_valid boolean, last_update timestamp") - - def read(self, out_dir, spark=None): - from pyspark.sql import SparkSession - spark = spark or SparkSession.getActiveSession() - return (spark.read.format("geojson_gbx") - .option("filterRegex", r".*wells\.geojson$").load(out_dir)) - - -def download_wells_aoi(spark, bbox, out_dir, **kw): - return WellsDownloader().download(bbox, out_dir, spark=spark, **kw) -``` -(`repair` for wells = re-run `download` when `is_out_file_valid = false`; add a thin `repair(out_dir, bbox, spark=None)` that re-fetches if the file is missing/invalid.) -- [ ] **Step 4:** Export from `sample/__init__.py` + add to `test_sample_package_all`. Run → pass (`pytest test/sample/test_wells.py test/sample/test_sample_bundle.py -v`). -- [ ] **Step 5: Commit** (`feat(sample): WellsDownloader (TX RRC ArcGIS GeoJSON, paged)`). - ---- - -### Task 5: Lint + regression + integration smoke - -**Files:** none new (verification task). - -- [ ] **Step 1: Format** — container black/isort on the new files (from `python/geobrix`): `python3 -m isort ... && python3 -m black ...` then `--check` + `flake8` (repo config, E501 ignored). Fix any drift. -- [ ] **Step 2: Full regression** — `docker exec geobrix-dev bash -lc 'cd /root/geobrix/python/geobrix && python3 -m pytest test/sample -q -m "not integration"'`. Expected: all green (new emit/wells tests + the updated `test_sample_package_all`). -- [ ] **Step 3 (optional, gated on creds/network): integration smoke** — a `@pytest.mark.integration` test that, given `EARTHDATA_TOKEN`, pulls one EMIT ENH COG + one PLM GeoJSON for the SMALL bbox and validates; and a wells fetch over the SMALL bbox asserting ≥100 features. Skips when the token/network is absent. -- [ ] **Step 4: Commit** any format fixes (`style: format emit/wells downloaders`). - ---- - -## Notes for the implementer - -- **`repair` needs `href`.** To keep `EmitDownloader.repair` real, `download()` should also surface `href` (add a 7th column, or re-`discover` the invalid subset by `item_id`). Persist `href` in the `emit_scenes` table (Plan B2) so repair can re-fetch. Mirror `StacClient.repair`'s MERGE on `(item_id, asset_name)`. -- **Driver-side EMIT download** is a deliberate simplification (EMIT auth mints short-lived per-request creds that don't fan out cleanly like anonymous PC hrefs). Acceptable at demo AOI scale (1 scene / ~15 plume granules SMALL). Note it in the class docstring; a distributed variant is a future enhancement. -- **`earthaccess.download` granule contract** — verify the installed version accepts the `_HrefGranule` adapter, or fetch real `DataGranule`s once via `search_data` and pass the matching subset. The injected `_earthaccess` fake models `download(granules, local_path=...)`. -- **Wells geometry** arrives WGS84 lon/lat via `f=geojson` (ArcGIS reprojects the native EPSG:2277); no `outSR` needed. `geojson_gbx` yields the light vector schema (`geom_0` WKB + `geom_0_srid`/`geom_0_srid_proj`). -- **Tests** stay under `test/sample/` (already tier-gated); each module defines its own `spark` fixture + `pytest.importorskip("pyspark")`; do not add markers. -- After B1 lands and a real SMALL-AOI pull is eyeballed, **Plan B2** (config_nb + NB01–05 + README + 5 diagrams) is written against the actual staged data. diff --git a/docs/superpowers/plans/2026-07-10-vapor-eyes-B2-notebooks.md b/docs/superpowers/plans/2026-07-10-vapor-eyes-B2-notebooks.md deleted file mode 100644 index c3fe8882b..000000000 --- a/docs/superpowers/plans/2026-07-10-vapor-eyes-B2-notebooks.md +++ /dev/null @@ -1,139 +0,0 @@ -# vapor-eyes Plan B2 — config_nb + notebook series + README/diagrams - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans. Notebook tasks are **author + run against real staged data**, not pre-written cell-by-cell; each ends with a rendered payoff from real data (the verification). - -**Goal:** Build the five-notebook vapor-eyes methane series (`config_nb` + NB01–05) plus its `README.md` and five per-notebook diagrams, consuming the Plan B1 downloaders + the `netcdf_gbx` reader, on the lightweight/Serverless tier. - -**Architecture:** Mirrors `eo-series`/`helios`: `%run ./config_nb` from each NB establishes catalog/schema + the Volume ETL tree + toggles + downloaders + `finalize_delta`. Each NB adds one source, writes managed Delta tables (path-bearing metadata tables → Volume assets) per the Spec B §6 inventory, and ends in a payoff visualization. The additive cascade: S5P hotspots → S2 MBMP plume → EMIT IME+rate → nearest-well attribution → portfolio PMTiles synthesis. Design ref: `docs/superpowers/specs/2026-07-10-vapor-eyes-series-design.md`. - -**Tech Stack:** GeoBrix light tier (`pyrx` `rx.rst_*`, `pyvx`, `gbx.ds` readers, `gbx.vizx`), the B1 downloaders (`TropomiDownloader`/`EmitDownloader`/`WellsDownloader`) + `StacClient`, Databricks-native ST/H3, Delta, MapLibre (`plot_interactive`). Serverless (env v5, Python 3.12). - -## Global Constraints - -- **Location:** `notebooks/examples/vapor-eyes/` — `config_nb.ipynb`, `01. S5P Screening.ipynb`, `02. Sentinel-2 Detection.ipynb`, `03. EMIT Quantification.ipynb`, `04. Facility Attribution.ipynb`, `05. Portfolio Synthesis.ipynb`. -- **UC layout (Spec B §6):** `catalog_name="geospatial_docs"`, `schema_name="vapor_eyes"`; Volume `data`; `VAPOR_EYES_DIR=/Volumes/geospatial_docs/vapor_eyes/data/vapor-eyes` with `s5p/ sentinel2/ emit/ wells/ tiles/`. Managed Delta tables (unqualified) per the §6 inventory; metadata tables carry `*_path` columns → Volume assets. -- **Toggles in `config_nb`:** `FULL_AOI` (False default → SMALL bbox `[-103.90,31.65,-103.40,32.15]`; True → FULL `[-104.4,31.3,-103.0,32.7]`) drives bbox + datetime across downloaders; `FORCE_REBUILD`; `INTERACTIVE_PLOTS`. -- **EMIT credential:** `config_nb` reads an Earthdata token from a Databricks secret into `EARTHDATA_TOKEN` (e.g. `os.environ["EARTHDATA_TOKEN"] = dbutils.secrets.get(scope, key)`), documented; discovery stays anonymous. -- **Serverless discipline:** no runtime `spark.conf.set` (use `set_conf_safe` + `repartition(N, col)`); no `.cache()` (write managed tables via `finalize_delta`); ~1 GB per-UDF cap (one item/tile per task); sequential Volume I/O. -- **Notebook hygiene:** `.ipynb` cell last source line must not end in `\n`; every NB code edit updates its section's markdown narrative in the same stroke; no internal vocabulary (wave numbers etc.) in any markdown. -- **Commit hygiene:** subject ≤72 chars + WHY body; end with `Co-authored-by: Isaac`. Branch: `examples/vapor-eyes`. - ---- - -### Task 0: Real SMALL-AOI data pull (grounding — do FIRST) - -**Deliverable:** real staged data under `VAPOR_EYES_DIR` on a dev workspace so every subsequent NB is authored against actual data shapes. No repo files. - -- [ ] Provision `EARTHDATA_TOKEN` (user supplies the NASA Earthdata token). -- [ ] On a Serverless notebook (or the dev cluster), for the SMALL bbox: `TropomiDownloader().download(...)` (S5P `.nc`), `StacClient` Sentinel-2 B11/B12 (COG), `EmitDownloader().download(...)` (ENH COG + PLM GeoJSON), `WellsDownloader().download(...)` (wells GeoJSON). -- [ ] Eyeball each: S5P `netcdf_gbx` vector output columns (`methane_mixing_ratio_bias_corrected`, `qa_value`, geom); S2 SWIR value ranges; EMIT ENH raster + PLM GeoJSON properties (emission-rate field name, plume-origin coords); wells GeoJSON fields (`API`, `CompanyName`, …). Record the exact field/column names — they parameterize the NB cells below. - ---- - -### Task 1: `config_nb.ipynb` - -**Files:** Create `notebooks/examples/vapor-eyes/config_nb.ipynb`. -**Interfaces:** Produces the shared state every NB uses: `spark`, `rx`, registered readers, `catalog_name`/`schema_name`, `VAPOR_EYES_DIR` (+ subdir consts), `SMALL_BBOX`/`FULL_BBOX`/`AOI_BBOX`, `DATE_WINDOW`, `FULL_AOI`/`FORCE_REBUILD`/`INTERACTIVE_PLOTS`, `finalize_delta`, `set_conf_safe`, and instantiated `tropomi`/`emit`/`wells`/`stac_client`. - -- [ ] **Cells (mirror `helios/config_nb`):** - 1. 2-step wheel install (`--force-reinstall --no-deps` then `[light,stac,vizx]`) from the sample-data Volume wheel; `%restart_python`. - 2. Imports (`os`, delta, `DBF`, `F`, types); `from databricks.labs.gbx.pyrx import functions as rx` (option-1 default; option-2 heavyweight commented); `rx.register(spark)`; `from databricks.labs.gbx.ds.register import register; register(spark)`. - 3. USER SETTINGS: `catalog_name="geospatial_docs"`, `schema_name="vapor_eyes"`; toggles `FULL_AOI=False`, `FORCE_REBUILD=False`, `INTERACTIVE_PLOTS=False`; `SMALL_BBOX`/`FULL_BBOX`; `AOI_BBOX = FULL_BBOX if FULL_AOI else SMALL_BBOX`; `DATE_WINDOW` (anchored near an EMIT overpass — pick from Task 0, e.g. `"2024-08-01/2024-09-30"`). - 4. Earthdata secret → `os.environ["EARTHDATA_TOKEN"]` (guarded: only if the scope/key exist; print a clear note if absent so NB03 fails with guidance not a stack trace). - 5. `set_conf_safe(...)` (as helios). - 6. `USE CATALOG`/`CREATE DATABASE IF NOT EXISTS`/`USE DATABASE`. - 7. `VAPOR_EYES_DIR` + subdir consts; `dbutils.fs.mkdirs` each; `os.environ` exports. - 8. `finalize_delta(df, tbl, do_display=True)` (copy verbatim from `helios/config_nb`). - 9. Downloader instantiation: `tropomi = TropomiDownloader()`, `emit = EmitDownloader()`, `wells = WellsDownloader()`, `stac_client = StacClient()`; import vizx helpers (`plot_raster`, `plot_pmtiles`, `plot_interactive`, `pmtiles_layer`). -- [ ] **Verify:** `%run ./config_nb` in a scratch notebook prints the catalog/schema/dir banner and the AOI bbox; `spark.catalog.currentDatabase() == "vapor_eyes"`. -- [ ] **Commit** (`feat(vapor-eyes): config_nb (catalog/schema/Volume/toggles/downloaders)`). - ---- - -### Task 2: `01. S5P Screening.ipynb` - -**Files:** Create the notebook. **Tables:** `s5p_granules` (item_id, date, **ch4_path**, is_out_file_valid), `s5p_hotspots` (h3_cellid, ch4_mean, ch4_max, n_obs, geom_wkb). - -- [ ] **Cells:** `%run ./config_nb`; `tropomi.download(AOI_BBOX, S5P_DIR, ...)` → `finalize_delta(..., "s5p_granules")`; `tropomi.read(S5P_DIR)` → `netcdf_gbx` vector points (`methane_mixing_ratio_bias_corrected`, `qa_value`, geom); **quality filter** on `qa_value` (threshold from Task 0, document it); `h3_longlatash3` bin points → group → mean/max CH4 per H3 cell → `finalize_delta(..., "s5p_hotspots")`; flag top-N hotspot cells (the AOIs NB02 consumes). -- [ ] **Payoff:** H3 choropleth of CH4 enhancement over the AOI (matplotlib/`plot_interactive`), super-emitter cells highlighted. Assert the table is non-empty + a hotspot cell exists. -- [ ] **Narrative markdown** per section; **commit** (`feat(vapor-eyes): NB01 S5P screening -> H3 hotspots`). - ---- - -### Task 3: `02. Sentinel-2 Detection.ipynb` - -**Files:** Create. **Tables:** `s2_swir_assets` (item_id, date, **b11_path**, **b12_path**), `s2_plume_cells` (h3_cellid, mbmp_frac, geom_wkb). - -- [ ] **Cells:** `%run ./config_nb`; scope to a hotspot cell's bbox from NB01 (`spark.table("s5p_hotspots")`); `StacClient` search+download `sentinel-2-l2a` B11/B12 for that sub-AOI (low cloud) → `finalize_delta(..., "s2_swir_assets")`; `gtiff_gbx` read → **MBMP band ratio** via `rx.rst_*` map-algebra (B12/B11 fractional absorption; document the illustrative nature — R3); `rst_h3_tessellate` the fraction raster → `s2_plume_cells`. -- [ ] **Payoff:** the MBMP plume-fraction raster over the hotspot (`plot_raster`), a candidate plume visible. Assert a plume-fraction cell exceeds a documented threshold. -- [ ] Narrative; **commit** (`feat(vapor-eyes): NB02 Sentinel-2 SWIR MBMP detection`). - ---- - -### Task 4: `03. EMIT Quantification.ipynb` - -**Files:** Create. **Tables:** `emit_scenes` (plume_id, date, **enh_cog_path**, **plm_geojson_path**, href, is_out_file_valid), `plume_quant` (plume_id, ime, emission_rate, rate_uncertainty, outline_wkb, origin_lon, origin_lat). - -- [ ] **Cells:** `%run ./config_nb`; **EMIT token guard** (clear message if `EARTHDATA_TOKEN` absent); `emit.download(AOI_BBOX, EMIT_DIR, temporal=DATE_WINDOW)` → `finalize_delta(..., "emit_scenes")` (persist `href` for `repair`); `emit.read_enh(EMIT_DIR)` (ENH COG tiles) + `emit.read_plumes(EMIT_DIR)` (PLM GeoJSON); compute **IME** = zonal sum of enhancement × pixel-area within each PLM plume polygon via `rx.rst_clip` + `rst_summary` (GeoBrix raster); read the **emission rate** + origin from the PLM GeoJSON properties (field names from Task 0) → `plume_quant`. -- [ ] **Payoff:** high-res EMIT plume raster + its quantified rate label (`plot_raster` + the `plume_quant` row). Assert a plume with a positive emission_rate + IME. -- [ ] Narrative; **commit** (`feat(vapor-eyes): NB03 EMIT COG IME + PLM emission rate`). - ---- - -### Task 5: `04. Facility Attribution.ipynb` - -**Files:** Create. **Tables:** `wells` (api, operator, lease, county, geom_wkb), `plume_attribution` (plume_id, origin_lon, origin_lat, nearest_well_api, operator, dist_m). - -- [ ] **Cells:** `%run ./config_nb`; `wells.download(AOI_BBOX, WELLS_DIR)` → `wells.read(WELLS_DIR)` → `finalize_delta(..., "wells")` (geom + `CompanyName`/`API`/…); join `plume_quant` origin points to `wells` via **native ST nearest** (`st_distancesphere`/`st_distance` + window `row_number` over ascending distance = 1) → `plume_attribution`. -- [ ] **Payoff:** attribution map — plume origin(s), the nearest well highlighted, operator labeled (`plot_interactive`). Assert every plume maps to exactly one well with a finite distance. -- [ ] Narrative; **commit** (`feat(vapor-eyes): NB04 nearest-well facility attribution`). - ---- - -### Task 6: `05. Portfolio Synthesis.ipynb` - -**Files:** Create. **Volume:** `tiles/*.pmtiles`, `tiles/mosaic.json`. **Tables:** `super_emitters` (operator, total_rate, plume_count, well_count), `pmtiles_catalog` (layer, **pmtiles_path**, bounds). - -- [ ] **Cells:** `%run ./config_nb`; rank operators by summed `emission_rate` (`plume_attribution` ⋈ `plume_quant`) → `super_emitters`; build the **multi-layer PMTiles map**: encode hotspots/plumes/wells/attribution to MVT (`gbx_st_asmvt` + `gbx_st_asmvt_pyramid`) → `gbx_pmtiles_agg` per layer → `pmtiles_catalog` + a `mosaic.json`; render with `plot_pmtiles`/`plot_interactive([...])`. -- [ ] **Payoff:** the super-emitter leaderboard table + the unified interactive map; a short ESG-style summary cell. -- [ ] Narrative; **commit** (`feat(vapor-eyes): NB05 portfolio synthesis + PMTiles map`). - ---- - -### Task 7: `README.md` - -**Files:** Create `notebooks/examples/vapor-eyes/README.md`, mirroring `eo-series`/`helios`. - -- [ ] Sections: intro + narrative; lightweight-tier/Serverless note; data-source note (S5P/S2 anon PC, EMIT via earthaccess + Earthdata token, TX RRC wells); **"Notebooks at a glance"** with an embedded diagram + 2-3 bullets per NB; a Files table; Prerequisites (incl. the Earthdata secret + Volume `data`); Run order; a Data-flow ASCII block; Serverless execution-strategy; Gotchas (EMIT opportunistic coverage, MBMP illustrative, TX-only wells, EMIT token); "Key GeoBrix / Databricks functions shown". -- [ ] **Commit** (`docs(vapor-eyes): series README`). - ---- - -### Task 8: Per-notebook diagrams (5) - -**Files:** `resources/images/diagrams/vapor-eyes/vapor-eyes-0{1..5}.png` (+ `.svg`); Modify `resources/images/generators/example-diagrams.py`. - -- [ ] Extend `example-diagrams.py` with a `vapor-eyes` series (5 diagrams in the established visual style: source → GeoBrix path → table/asset → payoff, one per NB). Generate PNG+SVG. -- [ ] Embed each in the README ("Notebooks at a glance"). -- [ ] Run `docs/scripts/check-diagram-coverage.py` → passes for `vapor-eyes`. -- [ ] **Commit** (`docs(vapor-eyes): per-notebook diagrams`). - ---- - -### Task 9: Validation - -- [ ] Run the series end-to-end on Serverless (SMALL default) via `gbx:test:notebooks` where applicable + a manual pass; each NB's payoff cell renders from real staged data. -- [ ] `FULL_AOI=True` smoke (larger counts resolve; no code path breaks). -- [ ] Diagram-coverage check + README link audit ([[docs-link-audit-pending]]). -- [ ] Lint any helper `.py`; ensure no internal vocabulary in markdown (`grep -rn -iE "wave [0-9]+" docs/ notebooks/`). - ---- - -## Notes for the implementer - -- **Author against Task 0 data.** The exact S5P `qa_value` threshold, the PLM GeoJSON emission-rate + origin field names, the S2 MBMP threshold, and the wells operator field are read from the real pull, not guessed — fill them into NB01/02/03/04 cells from Task 0. -- **`fsspec`/`s3fs` on Serverless (from B1):** the `[stac]` install now pulls `fsspec`/`s3fs==2026.6.0`; on env v5 verify no "core package changed" hard-fail — pin to the base if needed (same discipline as the `[light]` idna/rio-tiler pins). Check during Task 0's first Serverless install. -- **MBMP is illustrative** (R3) — frame NB02 as a demonstrative SWIR proxy, not an operational retrieval. -- **EMIT is opportunistic** (R2) — anchor `DATE_WINDOW` near a real EMIT overpass over the SMALL cluster (2023-06-08 / 2023-10-08 / 2023-12-24 / 2024-08-23 from coverage verification). -- **Netcdf raster mode** is intentionally not used here (Spec B §2/R5) — S5P uses vector mode; raster mode is documented/tested in Spec A. -- Notebooks are built + validated against real data, so tasks are "author + run", not pre-written cells — the payoff render is the per-task verification. diff --git a/docs/superpowers/plans/2026-07-14-vapor-eyes-lakeflow-sdp-aibi.md b/docs/superpowers/plans/2026-07-14-vapor-eyes-lakeflow-sdp-aibi.md deleted file mode 100644 index b6c135fc8..000000000 --- a/docs/superpowers/plans/2026-07-14-vapor-eyes-lakeflow-sdp-aibi.md +++ /dev/null @@ -1,985 +0,0 @@ -# Vapor-Eyes Lakeflow SDP + AI/BI Dashboard Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Ship a self-contained, incremental (daily + backfill) Lakeflow Declarative Pipeline that reproduces the vapor-eyes methane cascade into a dedicated `geospatial_docs.vapor_eyes_lf` schema, materializes latest + trend analytics as gold MVs, produces an app-ready fanout PMTiles product plus a light overview, and drives an AI/BI dashboard — all packaged as a Databricks Asset Bundle, deployed and run live, with docs + screenshots. - -**Architecture:** A Databricks Asset Bundle defines one **job** (`vapor_eyes_lf_job`, daily schedule) with Task 1 = a date-parameterized Python **land** task (runs the GeoBrix sample downloaders idempotently, stages raw files to a Volume subtree) → Task 2 = the **Lakeflow pipeline** (Serverless, environment version 5). The pipeline is medallion-layered: Auto Loader streaming **bronze** (file inventory, append-only, bi-temporal), append **silver** partitioned by `observation_date` (with SCD2 wells for as-of attribution), **gold** materialized views split into *latest* (operational maps) and *daily/trend* (time series), and a **tiles** layer (MVT pyramid → fanout PMTiles shards + light overview). An AI/BI dashboard resource binds to the gold MVs. - -**Tech Stack:** Databricks Asset Bundles (`databricks bundle`), Lakeflow Declarative Pipelines Python (`from pyspark import pipelines as dp`), Auto Loader (`cloudFiles`), GeoBrix lightweight tier (`geobrix[light,stac,vizx]` — `databricks.labs.gbx.pyrx`/`pyvx`/`ds`/`sample`/`stac`/`pmtiles`), Databricks-native spatial SQL (`st_*`, `h3_*` via `pyspark.databricks.sql.functions`), AI/BI (Lakeview) dashboards, `pytest` for pure-Python units, Python 3.12 / Spark 4 / Scala 2.13. - -## Global Constraints - -Every task's requirements implicitly include these (copied verbatim from the spec): - -- **Target schema:** `geospatial_docs.vapor_eyes_lf` (dedicated; NEVER write to the notebook series' `geospatial_docs.vapor_eyes`). -- **Compute:** Serverless, **environment version 5** (`environment: { spec: { environment_version: "5" }}` / pipeline `serverless: true`). -- **Tier:** lightweight only. Pipeline/transformation code MUST NOT call `spark.conf.set`, `spark._jvm`, or `.rdd`. `repartition(N, col)` is the only allowed parallelism lever (Serverless). -- **Lakeflow API:** `from pyspark import pipelines as dp`; datasets defined with `@dp.table` (streaming), `@dp.materialized_view` (batch), `@dp.temporary_view`, `@dp.expect*`, and `dp.create_auto_cdc_flow` (SCD). Dataset functions return a Spark DataFrame. **No side effects at module scope** (module code is evaluated repeatedly during planning) — all imperative work lives in the land task or inside decorated function bodies. **No `%pip` / `%restart_python` / `dbutils` / `%run`** in pipeline source files (those are notebook-only); dependencies come from the pipeline environment, secrets from the land task. -- **Dependency:** `geobrix[light,stac,vizx]` installed from the staged wheel on a UC Volume, declared as a **pipeline environment dependency** (Lakeflow's supported mechanism — see https://docs.databricks.com/aws/en/ldp/developer/external-dependencies; serverless pipelines do NOT support init scripts, `%pip`, `dbutils.library.restartPython()`, or JVM libraries). A bare Volume-path dependency (`"/Volumes/.../geobrix-0.4.0-py3-none-any.whl"`) installs the wheel; to also pull the `[light,stac,vizx]` extras, use the PEP 508 direct-reference form `geobrix[light,stac,vizx] @ file:///Volumes/.../geobrix-0.4.0-py3-none-any.whl`. -- **Registration:** every pipeline module that uses GeoBrix SQL/readers calls, inside a helper invoked from function bodies: `from databricks.labs.gbx.pyrx import functions as rx; rx.register(spark)`, `from databricks.labs.gbx.pyvx import functions as vx; vx.register(spark)`, `from databricks.labs.gbx.ds.register import register; register(spark)`. -- **AOI:** full AOI `FULL_BBOX = (-103.60, 31.05, -102.60, 31.85)`, EPSG:4326, `(minx, miny, maxx, maxy)`. -- **Bi-temporal:** every fact row carries `observation_date` (DATE, event time) and `_ingested_at` (TIMESTAMP, `current_timestamp()`). -- **Maps:** AI/BI cannot render WKB or H3 — gold exposes native `GEOMETRY` (via `st_geomfromwkb(wkb)`) and/or lat/lon columns. -- **Docs voice:** anything under `docs/docs/` or a README is user-facing — no internal/wave vocabulary. `grep -rn -iE "wave [0-9]+" docs/docs/` must print nothing. -- **Auth for push:** `gh auth switch --user mjohns-databricks` before any push/PR (not needed for local commits). Local git commits per task are fine. -- **Workspace auth for deploy:** use the Databricks CLI profile authorized for the workspace (confirm with `databricks auth describe` in Phase 1 Task 1). - ---- - -## File Structure - -``` -notebooks/examples/vapor-eyes/lakeflow/ - databricks.yml # bundle root: job (land->pipeline) + pipeline + dashboard + vars + target - README.md # deploy / run / schedule / backfill; params; caveats; screenshots - land/ - land.py # Task 1: date-parameterized downloader driver (CLI, no dp import) - _dates.py # pure helpers: window parsing, observation_date derivation, sharding math - transformations/ - _config.py # params (spark.conf.get), Volume paths, GeoBrix registration helper - bronze_ingest.py # s5p_granules, s2_swir_assets, emit_scenes, wells_raw (Auto Loader @dp.table) - silver_cascade.py # s5p_hotspots, s2_plume_cells, emit_plumes, plume_quant, wells_shl(SCD2), plume_candidate_wells - gold_analytics.py # *_latest MVs + *_daily/trend MVs + aoi_kpis_latest - portfolio_tiles.py # portfolio_mvt_tiles, pmtiles_shards (fanout), vapor_eyes_overview.pmtiles - dashboards/ - vapor_eyes_lf.lvdash.json # AI/BI dashboard (date filter + 4 pages) - tests/ - test_dates.py # pytest for land/_dates.py helpers - test_land.py # pytest for land.py argument handling (mocked downloaders) - validate/ - phase1.sql ... phase6.sql # SQL assertion queries run post-deploy per phase -``` - -Each `transformations/*.py` is a pipeline source file (referenced by the pipeline `libraries`/`root_path`). `land/*.py` is NOT part of the pipeline — it is the job's first task and must not `import` `pyspark.pipelines`. - ---- - -## Phase 1 — Vertical slice (S5P only) - -Goal: prove the entire pattern end-to-end on one source — DAB deploy on Serverless env v5, land-task idempotency, Auto Loader bronze, append silver partitioned by `observation_date` — and resolve the two Lakeflow unknowns (environment dependency install; dir-reader → incremental silver). - -### Task 1.1: Bundle scaffold + workspace auth check - -**Files:** -- Create: `notebooks/examples/vapor-eyes/lakeflow/databricks.yml` -- Create: `notebooks/examples/vapor-eyes/lakeflow/transformations/_config.py` - -**Interfaces:** -- Produces: bundle name `vapor_eyes_lf`; variables `catalog`, `schema`, `volume`, `full_aoi`, `date_window`, `s5p_temporal`, and the rest of the Global-Constraints parameters; pipeline `vapor_eyes_lf_pipeline` (target `${var.catalog}.${var.schema}`); `_config.py` helpers `cfg(spark)` → dict of resolved params, `paths(spark)` → dict of Volume dirs, `register_gbx(spark)`. - -- [ ] **Step 1: Confirm workspace auth + Serverless availability** - -Run: -```bash -databricks auth describe -databricks pipelines list-pipelines --max-results 1 -``` -Expected: prints the authenticated host/user (the profile authorized for the workspace) and returns without error. If it errors, stop and resolve auth before continuing. - -- [ ] **Step 2: Write the bundle root** - -Create `notebooks/examples/vapor-eyes/lakeflow/databricks.yml`: -```yaml -bundle: - name: vapor_eyes_lf - -variables: - catalog: {default: geospatial_docs} - schema: {default: vapor_eyes_lf} - volume: {default: data} - full_aoi: {default: "true"} - date_window: {default: "2023-07-15/2023-08-20"} - s5p_temporal: {default: "2024-08-23/2024-08-24"} - h3_res: {default: "6"} - qa_min: {default: "0.5"} - cloud_max: {default: "20"} - s2_h3_res: {default: "10"} - k_candidates: {default: "5"} - min_z: {default: "6"} - max_z: {default: "13"} - overview_max_z: {default: "12"} - earthdata_secret: {default: "geospatial_docs.vapor_eyes.earthdata_token"} - gbx_wheel: {default: "/Volumes/geospatial_docs/geobrix/sample-data/geobrix-0.4.0-py3-none-any.whl"} - -resources: - pipelines: - vapor_eyes_lf_pipeline: - name: vapor_eyes_lf_pipeline - serverless: true - catalog: ${var.catalog} - schema: ${var.schema} - root_path: ./transformations - libraries: - - glob: - include: transformations/** - environment: - dependencies: - - "geobrix[light,stac,vizx] @ file://${var.gbx_wheel}" - configuration: - vapor_eyes.catalog: ${var.catalog} - vapor_eyes.schema: ${var.schema} - vapor_eyes.volume: ${var.volume} - vapor_eyes.full_aoi: ${var.full_aoi} - vapor_eyes.date_window: ${var.date_window} - vapor_eyes.s5p_temporal: ${var.s5p_temporal} - vapor_eyes.h3_res: ${var.h3_res} - vapor_eyes.qa_min: ${var.qa_min} - vapor_eyes.cloud_max: ${var.cloud_max} - vapor_eyes.s2_h3_res: ${var.s2_h3_res} - vapor_eyes.k_candidates: ${var.k_candidates} - vapor_eyes.min_z: ${var.min_z} - vapor_eyes.max_z: ${var.max_z} - vapor_eyes.overview_max_z: ${var.overview_max_z} - -targets: - dev: - mode: development - default: true -``` - -> Environment dependency (mechanism confirmed — https://docs.databricks.com/aws/en/ldp/developer/external-dependencies): serverless Lakeflow pipelines install packages via **pipeline environment dependencies** only (no init scripts / `%pip` / restartPython / JVM libs). The `environment.dependencies` list above uses the PEP 508 direct-reference form to carry the `[light,stac,vizx]` extras. Residual to verify against `bundle validate`: the exact DAB key for the pipeline environment block (`environment.dependencies` vs `environment.spec.dependencies`) and that the extras resolve from the direct reference — if extras don't resolve, list the wheel path plus the extra dependencies explicitly. - -- [ ] **Step 3: Write `_config.py`** - -Create `notebooks/examples/vapor-eyes/lakeflow/transformations/_config.py`: -```python -"""Shared config for the vapor-eyes Lakeflow pipeline. Parameters come from the -pipeline `configuration` block (spark.conf.get); NO side effects at import time.""" - - -def cfg(spark): - g = spark.conf.get - full_aoi = g("vapor_eyes.full_aoi", "true").lower() == "true" - return { - "catalog": g("vapor_eyes.catalog", "geospatial_docs"), - "schema": g("vapor_eyes.schema", "vapor_eyes_lf"), - "volume": g("vapor_eyes.volume", "data"), - "full_aoi": full_aoi, - "bbox": (-103.60, 31.05, -102.60, 31.85) if full_aoi - else (-103.25, 31.30, -102.85, 31.62), - "date_window": g("vapor_eyes.date_window", "2023-07-15/2023-08-20"), - "s5p_temporal": g("vapor_eyes.s5p_temporal", "2024-08-23/2024-08-24"), - "h3_res": int(g("vapor_eyes.h3_res", "6")), - "qa_min": float(g("vapor_eyes.qa_min", "0.5")), - "cloud_max": int(g("vapor_eyes.cloud_max", "20")), - "s2_h3_res": int(g("vapor_eyes.s2_h3_res", "10")), - "k_candidates": int(g("vapor_eyes.k_candidates", "5")), - "min_z": int(g("vapor_eyes.min_z", "6")), - "max_z": int(g("vapor_eyes.max_z", "13")), - "overview_max_z": int(g("vapor_eyes.overview_max_z", "12")), - } - - -def paths(spark): - c = cfg(spark) - root = f"/Volumes/{c['catalog']}/{c['schema']}/{c['volume']}/vapor-eyes-lf" - return { - "root": root, - "s5p": f"{root}/s5p", - "s2": f"{root}/sentinel2", - "emit": f"{root}/emit", - "wells": f"{root}/wells", - "tiles": f"{root}/tiles", - "schema_loc": f"{root}/_schema", # Auto Loader schema locations - } - - -def register_gbx(spark): - """Register GeoBrix light SQL functions + DS readers. Call from function bodies.""" - from databricks.labs.gbx.pyrx import functions as rx - from databricks.labs.gbx.pyvx import functions as vx - from databricks.labs.gbx.ds.register import register as register_ds - rx.register(spark) - vx.register(spark) - register_ds(spark) -``` - -- [ ] **Step 4: Validate the bundle skeleton** - -Run: -```bash -cd notebooks/examples/vapor-eyes/lakeflow && databricks bundle validate -``` -Expected: `Validation OK!` (or the concrete error that resolves Phase-1 unknown #1 — fix per Step 2 note, re-run until OK). - -- [ ] **Step 5: Commit** - -```bash -git add notebooks/examples/vapor-eyes/lakeflow/databricks.yml notebooks/examples/vapor-eyes/lakeflow/transformations/_config.py -git commit -m "feat(vapor-eyes-lf): bundle scaffold + pipeline config" -``` - -### Task 1.2: Pure-Python date/window helpers (TDD) - -**Files:** -- Create: `notebooks/examples/vapor-eyes/lakeflow/land/_dates.py` -- Test: `notebooks/examples/vapor-eyes/lakeflow/tests/test_dates.py` - -**Interfaces:** -- Produces: `parse_window(window: str) -> tuple[date, date]` (accepts `"YYYY-MM-DD/YYYY-MM-DD"`); `asof_window(asof: str, days: int = 1) -> str` (returns a `"start/end"` string ending at `asof`); `observation_date_from_item(item_id: str, source: str) -> date | None` (parses the acquisition date token out of an S5P/S2/EMIT item id). - -- [ ] **Step 1: Write the failing tests** - -Create `notebooks/examples/vapor-eyes/lakeflow/tests/test_dates.py`: -```python -from datetime import date -from land._dates import parse_window, asof_window, observation_date_from_item - - -def test_parse_window_splits_range(): - assert parse_window("2023-07-15/2023-08-20") == (date(2023, 7, 15), date(2023, 8, 20)) - - -def test_asof_window_single_day(): - assert asof_window("2024-08-24", days=1) == "2024-08-23/2024-08-24" - - -def test_observation_date_s5p(): - # S5P item id embeds the sensing date as ..._YYYYMMDDT... - got = observation_date_from_item( - "S5P_OFFL_L2__CH4____20240823T193456_20240823T211626_...", "s5p") - assert got == date(2024, 8, 23) - - -def test_observation_date_none_when_absent(): - assert observation_date_from_item("no-date-here", "s5p") is None -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `cd notebooks/examples/vapor-eyes/lakeflow && python -m pytest tests/test_dates.py -v` -Expected: FAIL — `ModuleNotFoundError: No module named 'land._dates'`. - -- [ ] **Step 3: Implement `_dates.py`** - -Create `notebooks/examples/vapor-eyes/lakeflow/land/_dates.py`: -```python -"""Pure date/window helpers for the land task. No Spark, no side effects.""" -import re -from datetime import date, timedelta - -_DATE8 = re.compile(r"(\d{4})(\d{2})(\d{2})T\d{6}") - - -def parse_window(window: str) -> tuple[date, date]: - start, end = window.split("/") - return date.fromisoformat(start), date.fromisoformat(end) - - -def asof_window(asof: str, days: int = 1) -> str: - end = date.fromisoformat(asof) - start = end - timedelta(days=days) - return f"{start.isoformat()}/{end.isoformat()}" - - -def observation_date_from_item(item_id: str, source: str) -> date | None: - m = _DATE8.search(item_id) - if not m: - return None - y, mo, d = (int(x) for x in m.groups()) - return date(y, mo, d) -``` - -- [ ] **Step 4: Run to verify pass** - -Run: `cd notebooks/examples/vapor-eyes/lakeflow && python -m pytest tests/test_dates.py -v` -Expected: 4 passed. - -> Note: `observation_date_from_item` is verified against a real item id in Task 1.4 Step 3 (S5P) and adjusted per-source in Phase 2 if a source's id format differs. - -- [ ] **Step 5: Commit** - -```bash -git add notebooks/examples/vapor-eyes/lakeflow/land/_dates.py notebooks/examples/vapor-eyes/lakeflow/tests/test_dates.py -git commit -m "feat(vapor-eyes-lf): land date/window helpers (TDD)" -``` - -### Task 1.3: Land task — S5P download driver - -**Files:** -- Create: `notebooks/examples/vapor-eyes/lakeflow/land/land.py` -- Test: `notebooks/examples/vapor-eyes/lakeflow/tests/test_land.py` -- Modify: `notebooks/examples/vapor-eyes/lakeflow/databricks.yml` (add the job with the land task) - -**Interfaces:** -- Consumes: `land._dates.parse_window`, `asof_window`. -- Produces: `land.py` CLI `--sources s5p[,s2,emit,wells] --window |--asof --catalog --schema --volume --s5p-temporal`; function `run_land(spark, sources, *, catalog, schema, volume, date_window, s5p_temporal)` that stages files to the Volume subtree and returns a summary dict `{source: staged_count}`. Job task key `land`. - -- [ ] **Step 1: Write the failing test (arg handling, mocked downloaders)** - -Create `notebooks/examples/vapor-eyes/lakeflow/tests/test_land.py`: -```python -import sys, types -from unittest import mock - - -def _install_fakes(): - # Fake databricks.labs.gbx.sample so land.py imports without the wheel. - sample = types.ModuleType("databricks.labs.gbx.sample") - for name in ("TropomiDownloader", "EmitDownloader", "WellsDownloader"): - setattr(sample, name, mock.MagicMock()) - sys.modules["databricks"] = types.ModuleType("databricks") - sys.modules["databricks.labs"] = types.ModuleType("databricks.labs") - sys.modules["databricks.labs.gbx"] = types.ModuleType("databricks.labs.gbx") - sys.modules["databricks.labs.gbx.sample"] = sample - stac = types.ModuleType("databricks.labs.gbx.stac") - stac.StacClient = mock.MagicMock() - sys.modules["databricks.labs.gbx.stac"] = stac - return sample - - -def test_run_land_s5p_calls_tropomi_download(): - sample = _install_fakes() - from land.land import run_land - fake_spark = mock.MagicMock() - tropomi = sample.TropomiDownloader.return_value - tropomi.download.return_value.count.return_value = 1 - run_land(fake_spark, ["s5p"], catalog="c", schema="s", volume="data", - date_window="2023-07-15/2023-08-20", s5p_temporal="2024-08-23/2024-08-24") - assert tropomi.download.called - # staged to the vapor-eyes-lf s5p subtree with the s5p_temporal window - _, kwargs = tropomi.download.call_args - assert kwargs.get("temporal") == "2024-08-23/2024-08-24" -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `cd notebooks/examples/vapor-eyes/lakeflow && python -m pytest tests/test_land.py -v` -Expected: FAIL — `ModuleNotFoundError: No module named 'land.land'`. - -- [ ] **Step 3: Implement `land.py`** - -Create `notebooks/examples/vapor-eyes/lakeflow/land/land.py`: -```python -"""Task 1 of the vapor-eyes-lf job: date-parameterized downloader driver. - -Runs the GeoBrix sample downloaders idempotently (their own skip-guards avoid -re-downloading valid staged files) and lands raw files into the pipeline's own -Volume subtree. Emits NO Delta tables — the pipeline's Auto Loader bronze layer -inventories the staged files. This file is NOT part of the pipeline and must not -import pyspark.pipelines.""" -import argparse - -from land._dates import asof_window - - -def _subtree(catalog, schema, volume): - root = f"/Volumes/{catalog}/{schema}/{volume}/vapor-eyes-lf" - return { - "root": root, "s5p": f"{root}/s5p", "s2": f"{root}/sentinel2", - "emit": f"{root}/emit", "wells": f"{root}/wells", - } - - -def run_land(spark, sources, *, catalog, schema, volume, date_window, - s5p_temporal, bbox=(-103.60, 31.05, -102.60, 31.85)): - from databricks.labs.gbx.sample import ( - EmitDownloader, TropomiDownloader, WellsDownloader) - dirs = _subtree(catalog, schema, volume) - for d in dirs.values(): - _mkdir(spark, d) - staged = {} - if "s5p" in sources: - df = TropomiDownloader().download(bbox, dirs["s5p"], temporal=s5p_temporal, spark=spark) - staged["s5p"] = df.count() - if "s2" in sources: - from databricks.labs.gbx.stac import StacClient - staged["s2"] = _land_s2(spark, StacClient(), bbox, dirs["s2"], date_window) - if "emit" in sources: - df = EmitDownloader().download(bbox, dirs["emit"], temporal=date_window, spark=spark) - staged["emit"] = df.count() - if "wells" in sources: - df = WellsDownloader().download(bbox, dirs["wells"], spark=spark) - staged["wells"] = int(df.first()["feature_count"]) - print(f"... landed: {staged}") - return staged - - -def _mkdir(spark, path): - try: - import os - os.makedirs(path, exist_ok=True) - except Exception as e: - print(f"... mkdir skipped {path}: {type(e).__name__}") - - -def _land_s2(spark, stac, bbox, out_dir, date_window): - # Filled in Phase 2 (S2 needs a computed top-hotspot window); Phase 1 no-op. - return 0 - - -def main(): - from pyspark.sql import SparkSession - ap = argparse.ArgumentParser() - ap.add_argument("--sources", default="s5p") - ap.add_argument("--window") - ap.add_argument("--asof") - ap.add_argument("--catalog", default="geospatial_docs") - ap.add_argument("--schema", default="vapor_eyes_lf") - ap.add_argument("--volume", default="data") - ap.add_argument("--s5p-temporal", default="2024-08-23/2024-08-24") - a = ap.parse_args() - window = a.window or (asof_window(a.asof) if a.asof else "2023-07-15/2023-08-20") - spark = SparkSession.builder.getOrCreate() - run_land(spark, a.sources.split(","), catalog=a.catalog, schema=a.schema, - volume=a.volume, date_window=window, s5p_temporal=a.s5p_temporal) - - -if __name__ == "__main__": - main() -``` - -- [ ] **Step 4: Run to verify pass** - -Run: `cd notebooks/examples/vapor-eyes/lakeflow && python -m pytest tests/test_land.py -v` -Expected: 1 passed. - -- [ ] **Step 5: Add the job (land task) to the bundle** - -In `databricks.yml`, under `resources:` add: -```yaml - jobs: - vapor_eyes_lf_job: - name: vapor_eyes_lf_job - tasks: - - task_key: land - spark_python_task: - python_file: ./land/land.py - parameters: - ["--sources", "s5p", "--s5p-temporal", "${var.s5p_temporal}"] - environment_key: land_env - - task_key: pipeline - depends_on: [{task_key: land}] - pipeline_task: - pipeline_id: ${resources.pipelines.vapor_eyes_lf_pipeline.id} - environments: - - environment_key: land_env - spec: - environment_version: "5" - dependencies: - - "geobrix[light,stac,vizx] @ file://${var.gbx_wheel}" - schedule: - quartz_cron_expression: "0 0 7 * * ?" - timezone_id: "America/Chicago" - pause_status: PAUSED -``` - -- [ ] **Step 6: Validate + commit** - -Run: `cd notebooks/examples/vapor-eyes/lakeflow && databricks bundle validate` -Expected: `Validation OK!` -```bash -git add notebooks/examples/vapor-eyes/lakeflow/land/land.py notebooks/examples/vapor-eyes/lakeflow/tests/test_land.py notebooks/examples/vapor-eyes/lakeflow/databricks.yml -git commit -m "feat(vapor-eyes-lf): S5P land task + job wiring" -``` - -### Task 1.4: Bronze `s5p_granules` (Auto Loader streaming table) - -**Files:** -- Create: `notebooks/examples/vapor-eyes/lakeflow/transformations/bronze_ingest.py` -- Create: `notebooks/examples/vapor-eyes/lakeflow/tests/validate/phase1.sql` - -**Interfaces:** -- Produces: streaming table `s5p_granules` with columns `path STRING, file_size LONG, source_file STRING, observation_date DATE, _ingested_at TIMESTAMP`. -- Consumes: `_config.paths`, `_config.cfg`. - -- [ ] **Step 1: Write `bronze_ingest.py` (S5P only for Phase 1)** - -Create `notebooks/examples/vapor-eyes/lakeflow/transformations/bronze_ingest.py`: -```python -"""Bronze: Auto Loader file inventory per source. Append-only, exactly-once per -staged file. observation_date parsed from the filename; _ingested_at = now.""" -from pyspark import pipelines as dp -from pyspark.sql import functions as F - -from _config import paths - -# S5P sensing date token in the granule filename: ..._YYYYMMDDT...... -_S5P_DATE = r".*_(\d{8})T\d{6}.*" - - -def _autoload(spark, src_dir, schema_loc, glob): - return ( - spark.readStream.format("cloudFiles") - .option("cloudFiles.format", "binaryFile") - .option("cloudFiles.schemaLocation", schema_loc) - .option("pathGlobFilter", glob) - .load(src_dir) - .select( - F.col("path"), - F.col("length").alias("file_size"), - F.element_at(F.split(F.col("path"), "/"), -1).alias("source_file"), - ) - .withColumn("_ingested_at", F.current_timestamp()) - ) - - -@dp.table(name="s5p_granules", comment="Staged Sentinel-5P CH4 granule inventory") -def s5p_granules(): - from pyspark.sql import SparkSession - spark = SparkSession.getActiveSession() - p = paths(spark) - df = _autoload(spark, p["s5p"], f"{p['schema_loc']}/s5p", "*.nc") - return df.withColumn( - "observation_date", - F.to_date(F.regexp_extract("source_file", _S5P_DATE, 1), "yyyyMMdd"), - ) -``` - -- [ ] **Step 2: Write the Phase-1 validation SQL** - -Create `notebooks/examples/vapor-eyes/lakeflow/tests/validate/phase1.sql`: -```sql --- Bronze populated, exactly-once, dated. -SELECT count(*) AS n_granules, - count(DISTINCT source_file) AS n_files, - count(DISTINCT observation_date) AS n_dates, - max(observation_date) AS latest_obs -FROM geospatial_docs.vapor_eyes_lf.s5p_granules; --- Expect: n_granules > 0, n_granules = n_files (no dup ingest), observation_date NOT NULL. - --- Silver hotspots present and per-observation_date. -SELECT observation_date, count(*) AS n_cells, - round(max(ch4_max), 1) AS peak_ch4 -FROM geospatial_docs.vapor_eyes_lf.s5p_hotspots -GROUP BY observation_date ORDER BY observation_date; -``` - -- [ ] **Step 3: Deploy, run land, inspect a real S5P item id, run pipeline** - -Run (deploy + land + confirm the date regex against a real filename): -```bash -cd notebooks/examples/vapor-eyes/lakeflow && databricks bundle deploy -databricks bundle run vapor_eyes_lf_job --python-params '["--sources","s5p","--s5p-temporal","2024-08-23/2024-08-24"]' -databricks fs ls dbfs:/Volumes/geospatial_docs/vapor_eyes_lf/data/vapor-eyes-lf/s5p -``` -Expected: the job's `land` task stages ≥1 `.nc` file; `fs ls` prints a filename containing a `_YYYYMMDDT` token. If the token position differs from `_S5P_DATE`, fix the regex in `bronze_ingest.py` and `land/_dates.py`, re-run `tests/test_dates.py`, redeploy. - -- [ ] **Step 4: Validate bronze + silver populated** - -Run: -```bash -databricks sql query --warehouse-id --query "$(cat tests/validate/phase1.sql | sed -n '1,7p')" -``` -Expected: `n_granules > 0`, `n_granules = n_files`, `latest_obs = 2024-08-23`. - -- [ ] **Step 5: Verify idempotency (run job twice → no new rows)** - -Run: -```bash -databricks bundle run vapor_eyes_lf_job --python-params '["--sources","s5p"]' -databricks sql query --warehouse-id --query "SELECT count(*) FROM geospatial_docs.vapor_eyes_lf.s5p_granules" -``` -Expected: same `count(*)` as Step 4 (downloader skip-guard + Auto Loader exactly-once → no duplicates). - -- [ ] **Step 6: Commit** - -```bash -git add notebooks/examples/vapor-eyes/lakeflow/transformations/bronze_ingest.py notebooks/examples/vapor-eyes/lakeflow/tests/validate/phase1.sql -git commit -m "feat(vapor-eyes-lf): S5P Auto Loader bronze inventory" -``` - -### Task 1.5: Silver `s5p_hotspots` (append, partitioned by observation_date) — resolves reader→incremental unknown - -**Files:** -- Create: `notebooks/examples/vapor-eyes/lakeflow/transformations/silver_cascade.py` - -**Interfaces:** -- Consumes: `s5p_granules` (bronze), `_config.cfg`, `_config.register_gbx`. -- Produces: streaming table `s5p_hotspots` with `h3_cellid LONG, observation_date DATE, ch4_mean DOUBLE, ch4_max DOUBLE, n_obs LONG, geom_wkb BINARY, _ingested_at TIMESTAMP`, partitioned by `observation_date`. - -- [ ] **Step 1: Confirm the netcdf_gbx reader exposes a source-path column** - -Run (in a scratch Serverless notebook or `databricks` exec, using the wheel): -```python -from databricks.labs.gbx.sample import TropomiDownloader -df = TropomiDownloader().read("/Volumes/geospatial_docs/vapor_eyes_lf/data/vapor-eyes-lf/s5p") -print(df.columns) -``` -Expected: confirms the point columns `methane_mixing_ratio_bias_corrected, qa_value, geom_0` and whether a source/path column exists. **Branch:** -- If a source-path column exists → silver joins reader points to bronze on that path to attach `observation_date` (Step 2a). -- If NOT → silver derives `observation_date` per file by reading each new granule directory-scoped via the reader's `filterRegex` on the date token, looping over new dates from bronze (Step 2b). Pick the branch that matches reality; implement only that one. - -- [ ] **Step 2: Implement `silver_cascade.py` (s5p_hotspots)** - -Create `notebooks/examples/vapor-eyes/lakeflow/transformations/silver_cascade.py`: -```python -"""Silver: the methane cascade, append-only and partitioned by observation_date. -GeoBrix light readers are directory-scoped; we attach observation_date from the -bronze inventory (Step 1 branch) so each overpass's points aggregate into that -date's hotspot cells.""" -from pyspark import pipelines as dp -from pyspark.sql import functions as F -from pyspark.databricks.sql import functions as DBF - -from _config import cfg, paths, register_gbx - - -@dp.table( - name="s5p_hotspots", - comment="Per-H3-cell CH4 mean/max per overpass (S5P screening surface)", - partition_cols=["observation_date"], -) -@dp.expect("has_obs", "n_obs > 0") -@dp.expect("ch4_present", "ch4_mean IS NOT NULL") -def s5p_hotspots(): - from pyspark.sql import SparkSession - spark = SparkSession.getActiveSession() - register_gbx(spark) - c = cfg(spark) - p = paths(spark) - minx, miny, maxx, maxy = c["bbox"] - - from databricks.labs.gbx.sample import TropomiDownloader - pts = TropomiDownloader().read(p["s5p"]) # cols: methane_mixing_ratio_bias_corrected, qa_value, geom_0(+ source path) - pts = pts.filter( - (F.col("qa_value") >= c["qa_min"]) - & F.col("methane_mixing_ratio_bias_corrected").isNotNull() - ) - # observation_date: Step-1 branch. Branch-2a (join to bronze on source path): - granules = spark.read.table("s5p_granules").select("path", "observation_date") - pts = pts.join(granules, pts[""] == granules["path"], "left") - - pts = ( - pts.withColumn("_g", DBF.st_geomfromwkb(F.col("geom_0"))) - .withColumn("lon", DBF.st_x("_g")).withColumn("lat", DBF.st_y("_g")) - .filter((F.col("lon") >= minx) & (F.col("lon") <= maxx)) - .filter((F.col("lat") >= miny) & (F.col("lat") <= maxy)) - .withColumn("h3_cellid", DBF.h3_longlatash3("lon", "lat", F.lit(c["h3_res"]))) - ) - return ( - pts.groupBy("h3_cellid", "observation_date") - .agg( - F.mean("methane_mixing_ratio_bias_corrected").alias("ch4_mean"), - F.max("methane_mixing_ratio_bias_corrected").alias("ch4_max"), - F.count("*").alias("n_obs"), - ) - .withColumn("geom_wkb", DBF.h3_centeraswkb("h3_cellid")) - .withColumn("_ingested_at", F.current_timestamp()) - ) -``` -Replace `` with the confirmed column from Step 1 (Branch 2a), or replace the join with the Branch-2b per-date read. - -> Phase-1 unknown #2 note: `s5p_hotspots` is defined with `@dp.table` (streaming). If aggregation over a directory-read (non-streaming) source is rejected by the pipeline, switch to `@dp.materialized_view` and add `observation_date` as a MERGE key so history is retained across runs (documented fallback). Resolve against the live pipeline run. - -- [ ] **Step 3: Deploy + run + validate** - -Run: -```bash -cd notebooks/examples/vapor-eyes/lakeflow && databricks bundle deploy -databricks bundle run vapor_eyes_lf_job --python-params '["--sources","s5p"]' -databricks sql query --warehouse-id --query "$(sed -n '9,14p' tests/validate/phase1.sql)" -``` -Expected: one row per `observation_date` with `n_cells > 0` and a plausible `peak_ch4`. - -- [ ] **Step 4: Commit** - -```bash -git add notebooks/examples/vapor-eyes/lakeflow/transformations/silver_cascade.py -git commit -m "feat(vapor-eyes-lf): S5P hotspots silver (append by observation_date)" -``` - -**Phase 1 gate:** DAB deploys on Serverless env v5; `land` stages S5P idempotently; `s5p_granules` (Auto Loader) and `s5p_hotspots` (append by `observation_date`) populate; a second job run adds no duplicate rows. The environment-install and reader→incremental unknowns are resolved and the resolved approach is encoded. STOP for review before Phase 2. - ---- - -## Phase 2 — All downloads + full bronze - -Goal: extend `land.py` and `bronze_ingest.py` to S2, EMIT, wells. Each bronze table follows the S5P Auto Loader pattern; land uses the real per-source downloader signatures. - -### Task 2.1: Land — EMIT + wells - -**Files:** Modify `land/land.py` (EMIT + wells already scaffolded — wire the Earthdata secret), `databricks.yml` (land task `--sources s5p,emit,wells`; inject `EARTHDATA_TOKEN` from the UC secret). - -- [ ] **Step 1:** In `databricks.yml` land task, add the secret env and expand sources: -```yaml - spark_python_task: - python_file: ./land/land.py - parameters: ["--sources","s5p,emit,wells","--window","${var.date_window}","--s5p-temporal","${var.s5p_temporal}"] - environment_key: land_env -``` -And under the `land` task add (bundle job secret reference): -```yaml - # EARTHDATA_TOKEN for EMIT (NASA LP DAAC). UC secret scope.key from var.earthdata_secret. - spark_env_vars: - EARTHDATA_TOKEN: "{{secrets/vapor_eyes/earthdata_token}}" -``` -> Verify the scope/key path against `databricks secrets list-scopes`; the secret was created for the notebook series (`geospatial_docs.vapor_eyes.earthdata_token`). - -- [ ] **Step 2:** Confirm `run_land` already calls `EmitDownloader().download(bbox, dir, temporal=date_window, spark=spark)` and `WellsDownloader().download(bbox, dir, spark=spark)` (implemented in Task 1.3). Add a pytest to `tests/test_land.py` asserting both are called for `--sources emit,wells`: -```python -def test_run_land_emit_wells(): - sample = _install_fakes() - from land.land import run_land - fake = mock.MagicMock() - sample.EmitDownloader.return_value.download.return_value.count.return_value = 2 - sample.WellsDownloader.return_value.download.return_value.first.return_value = {"feature_count": 500} - out = run_land(fake, ["emit","wells"], catalog="c", schema="s", volume="data", - date_window="2023-07-15/2023-08-20", s5p_temporal="x") - assert sample.EmitDownloader.return_value.download.called - assert sample.WellsDownloader.return_value.download.called - assert out["wells"] == 500 -``` -Run: `python -m pytest tests/test_land.py -v` → 3 passed. - -- [ ] **Step 3:** Commit: `git commit -am "feat(vapor-eyes-lf): land EMIT + wells with Earthdata secret"`. - -### Task 2.2: Bronze — s2_swir_assets, emit_scenes, wells_raw - -**Files:** Modify `transformations/bronze_ingest.py`. - -- [ ] **Step 1:** Add three Auto Loader tables mirroring `s5p_granules`, with per-source globs and date tokens: -```python -@dp.table(name="emit_scenes", comment="Staged EMIT L2B CH4 product inventory") -def emit_scenes(): - from pyspark.sql import SparkSession - spark = SparkSession.getActiveSession() - p = paths(spark) - df = _autoload(spark, p["emit"], f"{p['schema_loc']}/emit", "*") - return df.withColumn( - "observation_date", - F.to_date(F.regexp_extract("source_file", r".*_(\d{8})T\d{6}.*", 1), "yyyyMMdd")) - - -@dp.table(name="wells_raw", comment="Staged TX RRC WellSHL snapshot inventory") -def wells_raw(): - from pyspark.sql import SparkSession - spark = SparkSession.getActiveSession() - p = paths(spark) - # wells snapshot has no acquisition date; observation_date = ingest date. - return (_autoload(spark, p["wells"], f"{p['schema_loc']}/wells", "*.geojson") - .withColumn("observation_date", F.to_date(F.col("_ingested_at")))) - - -@dp.table(name="s2_swir_assets", comment="Staged Sentinel-2 B11/B12 SWIR COG inventory") -def s2_swir_assets(): - from pyspark.sql import SparkSession - spark = SparkSession.getActiveSession() - p = paths(spark) - return (_autoload(spark, p["s2"], f"{p['schema_loc']}/s2", "*.tif") - .withColumn("observation_date", - F.to_date(F.regexp_extract("source_file", r".*_(\d{8})T\d{6}.*", 1), "yyyyMMdd"))) -``` -> Confirm each source's real date-token position with `databricks fs ls` after the first land run (as in Phase 1 Task 1.4 Step 3); adjust regexes if needed. - -- [ ] **Step 2:** Deploy + run job (`--sources s5p,emit,wells`); validate each bronze table `count(*) > 0` via a `phase2.sql` mirroring `phase1.sql` for the three tables. (S2 lands in Task 3.2 once the top-hotspot window exists.) - -- [ ] **Step 3:** Commit: `git commit -am "feat(vapor-eyes-lf): EMIT/wells/S2 Auto Loader bronze"`. - ---- - -## Phase 3 — Silver cascade (temporal) - -Goal: complete silver — `emit_plumes`, `plume_quant`, `wells_shl` (SCD2), `plume_candidate_wells` (as-of), `s2_plume_cells` — all append/partitioned by `observation_date`, mirroring the notebook transforms exactly (API pinned). - -### Task 3.1: `emit_plumes` + `plume_quant` - -**Files:** Modify `transformations/silver_cascade.py`. - -**Interfaces:** Produces `emit_plumes` (`plume_id, observation_date, max_conc_ppmm, emission_rate_kg_hr, emission_rate_uncert_kg_hr, wind_speed_ms, fetch_length_m, lon_max, lat_max, plume_geom, _ingested_at`) and `plume_quant` (adds `gbx_mean_ppmm, gbx_max_ppmm`). - -- [ ] **Step 1:** Add `emit_plumes` (append, partition by observation_date). Read via `EmitDownloader().read_plumes(p["emit"])`; attach `observation_date` from `emit_scenes` (same Step-1 branch as S5P); `@dp.expect("rate_nonneg","emission_rate_kg_hr >= 0")`, `@dp.expect("has_geom","plume_geom IS NOT NULL")`: -```python -@dp.table(name="emit_plumes", partition_cols=["observation_date"], - comment="EMIT plume outlines + JPL emission estimates") -@dp.expect("rate_nonneg", "emission_rate_kg_hr >= 0") -@dp.expect("has_geom", "plume_geom IS NOT NULL") -def emit_plumes(): - from pyspark.sql import SparkSession - spark = SparkSession.getActiveSession() - register_gbx(spark) - p = paths(spark) - from databricks.labs.gbx.sample import EmitDownloader - plumes = EmitDownloader().read_plumes(p["emit"]) - scenes = spark.read.table("emit_scenes").select("path", "observation_date").distinct() - # attach observation_date via the plume product's source path (Step-1 branch) - return (plumes.join(scenes, plumes[""] == scenes["path"], "left") - .withColumn("_ingested_at", F.current_timestamp())) -``` -- [ ] **Step 2:** Add `plume_quant` — port NB03 CELL 10 verbatim (crossJoin `read_enh`, `rx.rst_clip("scene","plume_geom",F.lit(True))`, `rx.rst_summary("clip")`, `get_json_object` for mean/max, `row_number` max-per-plume). Carry `observation_date`. Full code mirrors the pinned NB03 CELL 10 with `plumes = spark.read.table("emit_plumes")` as the source. -- [ ] **Step 3:** Deploy + run; validate `emit_plumes`/`plume_quant` row counts > 0 and `gbx_max_ppmm` non-null. Commit. - -### Task 3.2: `s2_plume_cells` (+ S2 land with computed top-hotspot window) - -**Files:** Modify `land/land.py` (`_land_s2`), `transformations/silver_cascade.py`. - -- [ ] **Step 1:** Implement `_land_s2`: read `s5p_hotspots` (latest date), pick top cell by `ch4_max`, derive its H3 boundary bbox, `stac_client.search(aoi, geojson_col="geojson", collections=["sentinel-2-l2a"], datetime=date_window)`, filter `eo:cloud_cover <= cloud_max`, take least-cloudy `item_id`, filter `asset_name in (B11,B12)`, `stac_client.download(bands, s2_dir, bbox=list(cell_bbox), bbox_crs="EPSG:4326")`. (Ports NB02 CELL 7; the top-hotspot is computed here in the land task, not cross-module.) -- [ ] **Step 2:** Add `s2_plume_cells` — port NB02 CELL 9 (`_band_tile` via `gtiff_gbx` + `filterRegex`, `rx.rst_mapalgebra(F.array("tile_b12","tile_b11"), "(B - A) / (A + B)")`, LATERAL `gbx_rst_h3_tessellate(tile, s2_h3_res)`, `gbx_rst_summary(named_struct(...))`). Carry `observation_date` from `s2_swir_assets`. -- [ ] **Step 3:** Add S2 to the land task sources in `databricks.yml` (`--sources s5p,emit,wells,s2`; S2 must run after a hotspot exists — it reads `s5p_hotspots`, so within the same land run S5P is processed first by the pipeline; if S2 needs the pipeline's hotspots, split land into `land_s5p` before pipeline and `land_s2` after — decide against the live DAG in review). Deploy + validate. Commit. - -### Task 3.3: `wells_shl` (SCD2) + `plume_candidate_wells` (as-of) - -**Files:** Modify `transformations/silver_cascade.py`. - -- [ ] **Step 1:** Define a streaming view over `wells_raw` that reads each snapshot via `WellsDownloader().read(p["wells"])` with the NB04 CELL 7 column projection (`API→api`, `CompanyName→operator`, `LeaseName→lease`, `WellNbr→well_no`, `FieldName→field`, `County→county`, `WellURL→well_url`, `geom_0→well_geom`), plus `observation_date` (snapshot date) and `_ingested_at`. -- [ ] **Step 2:** Create the SCD2 target with `dp.create_auto_cdc_flow`: -```python -dp.create_streaming_table("wells_shl") -dp.create_auto_cdc_flow( - target="wells_shl", - source="wells_snapshots", # the streaming view from Step 1 - keys=["api"], - sequence_by=F.col("observation_date"), - stored_as_scd_type=2, -) -``` -- [ ] **Step 3:** Add `plume_candidate_wells` (append, partition by observation_date) — port NB04 CELL 9 but join **as-of**: for each plume at `observation_date d`, join to `wells_shl` rows where `__START_AT <= d AND (__END_AT IS NULL OR __END_AT > d)`. Then `st_point(lon_max,lat_max)`, `st_geomfromwkb(well_geom)`, `st_distancesphere`, `row_number` ≤ `k_candidates`. Carry plume `observation_date`. -- [ ] **Step 4:** Deploy + validate (`wells_shl` has `__START_AT/__END_AT`; `plume_candidate_wells` = `k_candidates` rows per plume). Commit. - ---- - -## Phase 4 — Gold analytics (latest + trend) - -**Files:** Create `transformations/gold_analytics.py`. - -**Interfaces:** Produces MVs `plume_leaderboard_latest`, `operator_emissions_latest`, `field_county_emissions_latest`, `hotspot_latest`, `aoi_kpis_latest`, `emissions_trend_daily`, `operator_emissions_daily`, `hotspot_trend`. Each map-facing MV exposes native `GEOMETRY`/lat-lon. - -### Task 4.1: Latest MVs - -- [ ] **Step 1:** `plume_leaderboard_latest` — `@dp.materialized_view`; from `plume_quant` join `plume_candidate_wells` (rank=1) on `plume_id`+`observation_date`; keep the latest `observation_date` per `plume_id` via `row_number() over (partition by plume_id order by observation_date desc) = 1`; expose `origin_geom = st_point(lon_max, lat_max)`, `plume_geom_native = st_geomfromwkb(plume_geom)`, `lead_operator/lead_lease/lead_field/lead_dist_m`, emission cols, `observation_date`. Complete code: -```python -from pyspark import pipelines as dp -from pyspark.sql import functions as F -from pyspark.sql.window import Window - - -@dp.materialized_view(name="plume_leaderboard_latest", - comment="Latest per-plume emission + leading candidate operator (map-ready)") -def plume_leaderboard_latest(): - from pyspark.sql import SparkSession - spark = SparkSession.getActiveSession() - q = spark.read.table("plume_quant") - lead = (spark.read.table("plume_candidate_wells") - .filter("rank = 1") - .select("plume_id", "observation_date", - F.col("operator").alias("lead_operator"), - F.col("lease").alias("lead_lease"), - F.col("field").alias("lead_field"), - F.col("dist_m").alias("lead_dist_m"))) - j = q.join(lead, ["plume_id", "observation_date"], "left") - latest = j.withColumn("_r", F.row_number().over( - Window.partitionBy("plume_id").orderBy(F.col("observation_date").desc()))).filter("_r = 1").drop("_r") - return latest.select( - "plume_id", "observation_date", "emission_rate_kg_hr", "emission_rate_uncert_kg_hr", - "max_conc_ppmm", "gbx_mean_ppmm", "gbx_max_ppmm", "wind_speed_ms", "fetch_length_m", - "lead_operator", "lead_lease", "lead_field", "lead_dist_m", - F.expr("st_x(st_point(lon_max, lat_max))").alias("lon_max"), - F.expr("st_y(st_point(lon_max, lat_max))").alias("lat_max"), - F.expr("st_point(lon_max, lat_max)").alias("origin_geom"), - F.expr("st_geomfromwkb(plume_geom)").alias("plume_geom_native")) -``` -- [ ] **Step 2:** `operator_emissions_latest` — group latest `plume_leaderboard_latest` by `lead_operator`: `sum/max(emission_rate_kg_hr)`, `count(plume_id)` as `plume_count`, `approx_count_distinct` of wells from `plume_candidate_wells`. -- [ ] **Step 3:** `field_county_emissions_latest` — group by `lead_field, county` (county via join to `plume_candidate_wells` rank=1): `total_emission_kg_hr, plume_count`. -- [ ] **Step 4:** `hotspot_latest` — from `s5p_hotspots` latest `observation_date`; `center_lon/center_lat = st_x/st_y(st_geomfromwkb(geom_wkb))`, `hex_geom = st_geomfromwkb(h3_boundaryaswkb(h3_cellid))`, ranked by `ch4_max`. -- [ ] **Step 5:** `aoi_kpis_latest` — single-row MV: `total_plumes` (count `plume_leaderboard_latest`), `total_emission_kg_hr` (sum), `wells_scanned` (count distinct `api` in `wells_shl` current), `hotspot_cells` (count `hotspot_latest`), `aoi_area_km2` (from bbox), `latest_observation_date`. -- [ ] **Step 6:** Deploy + validate all 5 non-empty; native GEOMETRY columns non-null (`SELECT count(*) FROM ... WHERE origin_geom IS NOT NULL`). Commit. - -### Task 4.2: Trend MVs - -- [ ] **Step 1:** `emissions_trend_daily` — `plume_quant` group by `observation_date`: `sum/max(emission_rate_kg_hr)`, `count(*) plume_count`. -- [ ] **Step 2:** `operator_emissions_daily` — `plume_candidate_wells` (rank=1) join `plume_quant`, group by `observation_date, operator`. -- [ ] **Step 3:** `hotspot_trend` — `s5p_hotspots` group by `observation_date, h3_cellid`: `ch4_mean, ch4_max` (+ `center_lon/lat` for optional animated maps). -- [ ] **Step 4:** Deploy + validate; commit. - ---- - -## Phase 5 — Tiles / synthesis (fanout PMTiles + overview) - -**Files:** Create `transformations/portfolio_tiles.py`; add sharding math to `land/_dates.py` companion or a new `transformations/_shard.py` with a pytest. - -### Task 5.1: `portfolio_mvt_tiles` - -- [ ] **Step 1:** `@dp.materialized_view portfolio_mvt_tiles` — build the three layers from **latest** silver (`hotspot_latest`, `plume_leaderboard_latest`, `wells_shl` current), each `(geom_wkb, attrs)` carrying `observation_date` in `attrs`; `repartition(N, "geom_wkb")`; LATERAL `gbx_st_asmvt_pyramid(geom_wkb, attrs, min_z, max_z, '')` per NB05 CELL 7; union. Columns `layer, z, x, y, mvt_bytes`. -- [ ] **Step 2:** Deploy + validate row count > 0 across `z` in `[min_z, max_z]`. Commit. - -### Task 5.2: Sharding math (TDD) + `pmtiles_shards` fanout - -**Files:** Create `transformations/_shard.py`, `tests/test_shard.py`. - -- [ ] **Step 1:** Write failing test for `tile_shard(z, x, y, shard_zoom) -> str` (shard key = the ancestor tile at `shard_zoom`, e.g. `"z5/x/y"`), and `shard_bounds(shard_key) -> (minx,miny,maxx,maxy)` (Web-Mercator tile → lon/lat bbox). Provide concrete assertions (e.g. `tile_shard(13, 1600, 3000, 6) == "6/25/46"`). -- [ ] **Step 2:** Implement `_shard.py` (pure math: `x >> (z - shard_zoom)`, standard XYZ→lon/lat). Run tests → pass. -- [ ] **Step 3:** `pmtiles_shards` — group `portfolio_mvt_tiles` by `tile_shard(...)`, per shard `pmtiles_agg("mvt_bytes","z","x","y", META)` (META = the NB05 TileJSON `vector_layers` for hotspots/plumes/wells), write each archive to `{tiles}/shards/.pmtiles` (FUSE-safe sequential write in the body), and return the catalog rows `shard_id, min_x, min_y, max_x, max_y, archive_path, layer_feature_counts, min_z, max_z`. Binary-free (no `tile-join`). -- [ ] **Step 4:** Deploy + validate `pmtiles_shards` rows > 0 and archives exist on the Volume (`databricks fs ls .../tiles/shards`). Commit. - -### Task 5.3: `vapor_eyes_overview.pmtiles` (light single archive) - -- [ ] **Step 1:** In `portfolio_tiles.py`, a final `@dp.materialized_view overview_manifest` (1 row) whose body filters `portfolio_mvt_tiles` to `z <= overview_max_z`, `pmtiles_agg` the whole set, and `open(f"{tiles}/vapor_eyes_overview.pmtiles","wb").write(archive)`; return `path, byte_size, max_zoom`. -- [ ] **Step 2:** Deploy + validate the file exists and `byte_size` is small; commit. - ---- - -## Phase 6 — AI/BI dashboard - -**Files:** Create `dashboards/vapor_eyes_lf.lvdash.json`; add the dashboard resource to `databricks.yml`. - -**REQUIRED SUB-SKILL:** use the `fe-databricks-tools:databricks-lakeview-dashboard` skill to author + deploy the `.lvdash.json` (datasets, widgets, map layers, filters) rather than hand-writing the widget schema. - -### Task 6.1: Datasets + pages - -- [ ] **Step 1:** Define dashboard datasets (SQL queries over the gold MVs): - - `ds_kpis` → `aoi_kpis_latest` - - `ds_hotspots` → `hotspot_latest` (`hex_geom`, `ch4_max`, `center_lon/lat`) - - `ds_plumes` → `plume_leaderboard_latest` (`origin_geom`, `plume_geom_native`, `emission_rate_kg_hr`, `lead_operator`) - - `ds_operators` → `operator_emissions_latest` - - `ds_fields` → `field_county_emissions_latest` - - `ds_wells` → `plume_candidate_wells` (`well_lon`, `well_lat`, `dist_m`, `rank`, `operator`) - - `ds_trend` → `emissions_trend_daily`; `ds_operator_trend` → `operator_emissions_daily` -- [ ] **Step 2:** Add a **date-range filter** widget bound to `observation_date` (default = latest), scoping the operational pages. -- [ ] **Step 3:** Build the 4 pages: - - **Regional screen (latest):** counter tiles (`ds_kpis`), choropleth map (`ds_hotspots.hex_geom` colored by `ch4_max`), top-cells table. - - **Quantify & attribute (latest):** point map (`ds_plumes.origin_geom` sized/colored by `emission_rate_kg_hr`) + polygon choropleth (`plume_geom_native`), leaderboard table, operator bar (`ds_operators`), field/county bar (`ds_fields`). - - **Wells & candidates (latest):** point map (`ds_wells.well_lon/well_lat`), nearest-candidate table. - - **Trends:** line (`ds_trend` total emission over `observation_date`), multi-series line (`ds_operator_trend`). - Map viz: **point map** on lat/lon or GEOMETRY POINT; **choropleth** on GEOMETRY POLYGON (WKB/H3 not renderable — that's why gold exposes native GEOMETRY). -- [ ] **Step 4:** Add the dashboard resource to `databricks.yml`: -```yaml - dashboards: - vapor_eyes_lf_dashboard: - display_name: "Vapor-Eyes — Methane Cascade (Lakeflow)" - file_path: ./dashboards/vapor_eyes_lf.lvdash.json - warehouse_id: ${var.warehouse_id} -``` -(add `warehouse_id` var). Deploy: `databricks bundle deploy`. -- [ ] **Step 5:** Open the deployed dashboard; verify each map/chart/filter renders against the populated MVs. Capture screenshots to `resources/images/...` for the docs. Commit. - ---- - -## Phase 7 — Docs - -**Files:** Create `notebooks/examples/vapor-eyes/lakeflow/README.md`; modify `notebooks/examples/vapor-eyes/README.md`; create a page under `docs/docs/`. - -### Task 7.1: lakeflow/README.md - -- [ ] **Step 1:** Write deploy/run/schedule/backfill instructions (`databricks bundle deploy`, `bundle run vapor_eyes_lf_job`, daily schedule note, backfill = `--window `), the full parameter table (from Global Constraints), the **portability caveat** (downloads its own full-AOI data), the SCD2/as-of note, prerequisites (Volume `data`, Earthdata secret), and embed the dashboard screenshots. No wave/internal vocabulary. -- [ ] **Step 2:** Commit. - -### Task 7.2: series README + docs page - -- [ ] **Step 1:** Add a "Lakeflow pipeline + AI/BI dashboard" section to `notebooks/examples/vapor-eyes/README.md` cross-linking `lakeflow/`, explaining it as the productionized, incremental, as-of counterpart to the notebook cascade. -- [ ] **Step 2:** Add/update a page under `docs/docs/` (near the vapor-eyes example docs) covering: incremental Lakeflow SDP, bi-temporal as-of modeling, gold latest/trend MVs, AI/BI native-spatial maps, fanout PMTiles for apps. Screenshots + cross-links. -- [ ] **Step 3:** Run `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/ notebooks/examples/vapor-eyes/` → expect no output. Run `gbx:lint:python --check` over `lakeflow/` Python. Commit. -- [ ] **Step 4:** (Optional, on user go) `gh auth switch --user mjohns-databricks` then push per "Hold pushes, batch more" — push only at a clear stopping point or on user request. - ---- - -## Self-Review (writing-plans) - -**Spec coverage:** -- Ingestion (land→pipeline job): Phase 1 T1.3, Phase 2 T2.1 ✓ -- Dedicated `vapor_eyes_lf` schema + own Volume subtree: Global Constraints, `_config.paths` ✓ -- Full AOI, downloaded once/idempotent: `_config.cfg` bbox, land skip-guards, Phase 1 T1.5 idempotency check ✓ -- Bi-temporal (observation_date + _ingested_at): every bronze/silver task ✓ -- SCD2 wells + as-of attribution: Phase 3 T3.3 ✓ -- Gold latest + trend MVs (all 4 families + trend): Phase 4 ✓ -- Native GEOMETRY/lat-lon for maps: Phase 4 T4.1, Phase 6 ✓ -- Fanout PMTiles + light overview: Phase 5 T5.2/T5.3 ✓ -- DAB packaging + live deploy/run + dashboard + screenshots: Phases 1–6 ✓ -- 7-phase build: matches ✓ -- Docs (3 targets) + voice check: Phase 7 ✓ - -**Placeholder scan:** Two intentional `<...>` markers remain and are *resolved by an explicit adjacent step*, not left vague: `` (resolved in Phase 1 T1.5 Step 1 branch) and ``/`` CLI values (workspace-specific, filled at run). All code-bearing steps contain complete code. No "add error handling"/"similar to Task N"/"TBD". - -**Type/name consistency:** table names, column names (`observation_date`, `_ingested_at`, `ch4_max`, `plume_geom`, `well_geom`, `lon_max/lat_max`, `dist_m`, `rank`), and function names (`cfg`, `paths`, `register_gbx`, `run_land`, `tile_shard`, `shard_bounds`) are used consistently across tasks. GeoBrix API (`rx.rst_mapalgebra/rst_clip/rst_summary`, `gbx_rst_h3_tessellate`, `gbx_st_asmvt_pyramid`, `pmtiles_agg`, `DBF.h3_*/st_*`) matches the pinned notebook signatures. - -**Known workspace-resolved unknowns (front-loaded into Phase 1):** (1) exact DAB key for the pipeline environment block + extras resolution from the direct-reference wheel (mechanism confirmed = environment dependencies; https://docs.databricks.com/aws/en/ldp/developer/external-dependencies); (2) netcdf_gbx source-path column for the reader→incremental join; (3) `@dp.table` streaming vs `@dp.materialized_view` for directory-read aggregation; (4) S2 land ordering vs the pipeline's hotspot table. Each has a specified fallback. diff --git a/docs/superpowers/plans/2026-07-15-vapor-eyes-context-geometries.md b/docs/superpowers/plans/2026-07-15-vapor-eyes-context-geometries.md deleted file mode 100644 index c4fa760f4..000000000 --- a/docs/superpowers/plans/2026-07-15-vapor-eyes-context-geometries.md +++ /dev/null @@ -1,417 +0,0 @@ -# Vapor-Eyes Context Geometries Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add EIA Permian shale-play and TIGER county context geometries to the Vapor-Eyes Lakeflow SDP, roll up Carbon Mapper plumes by play and county via point-in-polygon, and surface both as choropleths on a new AI/BI "Regional Context" dashboard page. - -**Architecture:** Two static reference sources land once per run to `context/` on the Volume; two reference MVs read them with the GeoBrix light vector reader (`geojson_gbx` / `shapefile_gbx`) emitting native `GEOMETRY` at SRID 4326; two gold MVs join `cm_plume_attributed` points into the polygons with native `st_contains`; a fourth dashboard page renders the two rollup choropleths. - -**Tech Stack:** Databricks Lakeflow Declarative Pipelines (`from pyspark import pipelines as dp`), GeoBrix light tier (`pyrx`/`pyvx`/`ds`), Databricks native ST functions, Databricks Asset Bundle, AI/BI (Lakeview) dashboards, pyogrio-backed light vector readers. - -## Global Constraints - -- **Light tier only.** Product/transform paths never call `spark.conf.set` / `_jvm` / `.rdd`. `repartition(N, col)` only if keyed (memory `serverless-fanout-repartition-by-column`). These rollups are tiny (≤ a few hundred polygons) — no repartition needed. -- **GEOMETRY at SRID 4326.** Every map-facing geometry column is native `GEOMETRY` tagged SRID 4326. -- **Choropleth render contract (memory `aibi-custom-geometry-choropleth`):** the widget's `encodings.region.fieldName` = `"geo(

`; lint `bash scripts/commands/gbx-lint-python.sh --check` (format in-container with `black`/`isort`). Scala tests run via Maven in the container (`gbx:test:scala --suite ...`). -- Branch `beta/0.4.0`; commit locally, do NOT push (controller pushes on the user's go). - ---- - -### Task 1: Light — generalize `_writer_col_roles` with optional overrides - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (`_writer_col_roles`) -- Test: `python/geobrix/test/ds/test_writer_col_roles.py` (new) - -**Interfaces:** -- Produces: `_writer_col_roles(schema, geom_col=None, srid_col=None, proj_col=None) -> (geom, srid, proj, attr_cols)`. geom & srid required (raise `ValueError`); proj optional. No-override call behaves as today. - -- [ ] **Step 1: Write the failing tests** - -Create `python/geobrix/test/ds/test_writer_col_roles.py`: - -```python -import pytest -from pyspark.sql.types import ( - BinaryType, - LongType, - StringType, - StructField, - StructType, -) - -from databricks.labs.gbx.ds.vector import _writer_col_roles - - -def _schema(*names_types): - return StructType([StructField(n, t, True) for n, t in names_types]) - - -CONV = _schema( - ("name", StringType()), - ("geom_0", BinaryType()), - ("geom_0_srid", StringType()), - ("geom_0_srid_proj", StringType()), -) - - -def test_default_convention(): - g, s, p, attrs = _writer_col_roles(CONV) - assert (g, s, p) == ("geom_0", "geom_0_srid", "geom_0_srid_proj") - assert attrs == ["name"] - - -def test_explicit_geom_and_srid_arbitrary_names(): - sch = _schema( - ("v", LongType()), - ("the_geom", BinaryType()), - ("epsg", StringType()), - ("proj4", StringType()), - ) - g, s, p, attrs = _writer_col_roles( - sch, geom_col="the_geom", srid_col="epsg", proj_col="proj4" - ) - assert (g, s, p) == ("the_geom", "epsg", "proj4") - assert attrs == ["v"] - - -def test_srid_defaults_off_geom_when_only_geomcol_given(): - g, s, p, _ = _writer_col_roles(CONV, geom_col="geom_0") - assert (g, s, p) == ("geom_0", "geom_0_srid", "geom_0_srid_proj") - - -def test_geomcol_missing_column_raises(): - with pytest.raises(ValueError): - _writer_col_roles(CONV, geom_col="nope") - - -def test_sridcol_missing_column_raises(): - with pytest.raises(ValueError): - _writer_col_roles(CONV, srid_col="nope") - - -def test_projcol_missing_column_raises(): - with pytest.raises(ValueError): - _writer_col_roles(CONV, proj_col="nope") - - -def test_srid_unresolvable_raises(): - # geomCol given, but no sridCol and no _srid present - sch = _schema(("the_geom", BinaryType()), ("v", LongType())) - with pytest.raises(ValueError): - _writer_col_roles(sch, geom_col="the_geom") - - -def test_proj_optional_absent_is_fine(): - sch = _schema( - ("name", StringType()), - ("geom_0", BinaryType()), - ("geom_0_srid", StringType()), - ) - g, s, p, attrs = _writer_col_roles(sch) - assert (g, s) == ("geom_0", "geom_0_srid") - assert p == "geom_0_srid_proj" # default name, not present -> harmless - assert attrs == ["name"] -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_writer_col_roles.py` -Expected: FAIL — `_writer_col_roles() got an unexpected keyword argument 'geom_col'`. - -- [ ] **Step 3: Implement** - -Replace `_writer_col_roles` in `python/geobrix/src/databricks/labs/gbx/ds/vector.py` with: - -```python -def _writer_col_roles(schema, geom_col=None, srid_col=None, proj_col=None): - """(geom_col, srid_col, proj_col, attr_cols) for the vector writers. - - By default the column ``X`` paired with ``X_srid`` is the geometry, - ``X_srid_proj`` is its PROJ4 fallback, and everything else is an attribute. - The geomCol / sridCol / projCol options override these by name so the frame - need not use the convention: each option, when given, must name an existing - column; when omitted it falls back to its convention name. geom and srid are - required (clear error if unresolvable); proj is optional. - """ - names = [f.name for f in schema.fields] - - # geometry (required) - if geom_col is not None: - if geom_col not in names: - raise ValueError(f"vector writer geomCol={geom_col!r} is not a column; got {names}") - geom = geom_col - else: - srid_named = [n for n in names if n.endswith("_srid")] - if not srid_named: - raise ValueError( - "vector writer input needs a geometry/'*_srid' column pair (from a " - f"*_gbx reader) or an explicit geomCol option; got columns {names}" - ) - geom = srid_named[0][: -len("_srid")] - if geom not in names: - raise ValueError(f"no geometry column {geom!r} for srid {srid_named[0]!r}") - - # srid (required: option, else _srid) - if srid_col is not None: - if srid_col not in names: - raise ValueError(f"vector writer sridCol={srid_col!r} is not a column; got {names}") - srid = srid_col - else: - srid = geom + "_srid" - if srid not in names: - raise ValueError( - f"vector writer needs a SRID column: pass sridCol, or add a {srid!r} " - f"column (authority code, '0' if unknown). Columns: {names}" - ) - - # proj (optional: an explicit projCol must exist; the default may be absent) - if proj_col is not None: - if proj_col not in names: - raise ValueError(f"vector writer projCol={proj_col!r} is not a column; got {names}") - proj = proj_col - else: - proj = geom + "_srid_proj" # optional; may be absent - - attr_cols = [n for n in names if n not in (geom, srid, proj)] - return geom, srid, proj, attr_cols -``` - -- [ ] **Step 4: Run to verify pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_writer_col_roles.py` -Expected: PASS (8 tests). - -- [ ] **Step 5: Lint + commit** - -```bash -bash scripts/commands/gbx-docker-exec.sh "cd /root/geobrix && black python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_writer_col_roles.py && isort python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_writer_col_roles.py" -bash scripts/commands/gbx-lint-python.sh --check -git add python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_writer_col_roles.py -git commit -m "feat(ds): _writer_col_roles accepts geom/srid/proj column overrides" -``` - ---- - -### Task 2: Light — thread `geomCol`/`sridCol`/`projCol` through both writers - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (`VectorGbxWriter.__init__`, `GeoJSONLGbxWriter.__init__`) -- Test: `python/geobrix/test/ds/test_geojsonl_writer.py` - -**Interfaces:** -- Consumes: `_writer_col_roles(schema, geom_col, srid_col, proj_col)` (Task 1). -- Produces: both writers read `geomcol`/`sridcol`/`projcol` from their lowercased `opts` and pass them to `_writer_col_roles`. - -- [ ] **Step 1: Write the failing test** - -Append to `python/geobrix/test/ds/test_geojsonl_writer.py`: - -```python -def test_geomcol_sridcol_options_avoid_renaming(spark, tmp_path): - # A frame with non-convention column names writes via the geomCol/sridCol - # options without the user renaming anything. - register(spark) - out = str(tmp_path / "renamed") - rows = [ - ("a", bytearray(to_wkb(Point(-73.9, 40.7))), "4326", ""), - ("b", bytearray(to_wkb(Point(-0.1, 51.5))), "4326", ""), - ] - df = spark.createDataFrame( - rows, schema="name string, the_geom binary, epsg string, p4 string" - ).repartition(2, F.col("the_geom")) - ( - df.write.format("geojsonl_gbx") - .mode("overwrite") - .option("geomCol", "the_geom") - .option("sridCol", "epsg") - .option("projCol", "p4") - .save(out) - ) - back = spark.read.format("geojson_gbx").option("multi", "true").load(out) - assert back.count() == 2 - assert {r["name"] for r in back.collect()} == {"a", "b"} -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path "python/geobrix/test/ds/test_geojsonl_writer.py::test_geomcol_sridcol_options_avoid_renaming"` -Expected: FAIL — the writer still auto-derives by convention and errors (no `*_srid` column). - -- [ ] **Step 3: Implement — read the options in both writers** - -In `VectorGbxWriter.__init__`, change the `_writer_col_roles(schema)` call to pass the options. The block currently reads: - -```python - self.geometry_type_override = opts.get("geometrytype") - self.layer_name = opts.get("layername") - self.geom_col, self.srid_col, self.proj_col, self.attr_cols = _writer_col_roles( - schema - ) -``` - -becomes: - -```python - self.geometry_type_override = opts.get("geometrytype") - self.layer_name = opts.get("layername") - self.geom_col, self.srid_col, self.proj_col, self.attr_cols = _writer_col_roles( - schema, - geom_col=opts.get("geomcol"), - srid_col=opts.get("sridcol"), - proj_col=opts.get("projcol"), - ) -``` - -Make the identical change in `GeoJSONLGbxWriter.__init__` (same two lines — -`opts` is already the lowercased dict there too). - -- [ ] **Step 4: Run to verify pass + no regression** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_geojsonl_writer.py` -Expected: PASS (all, including the new test). -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_vector_writer.py` -Expected: PASS (no regression — default path unchanged). - -- [ ] **Step 5: Lint + commit** - -```bash -bash scripts/commands/gbx-docker-exec.sh "cd /root/geobrix && black python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_geojsonl_writer.py && isort python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_geojsonl_writer.py" -bash scripts/commands/gbx-lint-python.sh --check -git add python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_geojsonl_writer.py -git commit -m "feat(ds): light vector writers accept geomCol/sridCol/projCol options" -``` - ---- - -### Task 3: Light — per-format output geometry name - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (a `_output_geom_name` helper + the two `geometry_name=` pyogrio sites + the osgeo FileGDB `CreateLayer`) -- Test: `python/geobrix/test/ds/test_vector_writer.py` - -**Interfaces:** -- Consumes: `self.driver`, `self.geom_col`. -- Produces: `_output_geom_name(driver, geom_col) -> str` — `"geom"` for GPKG, `"SHAPE"` for OpenFileGDB, else `geom_col` (structural drivers: inert). - -**Note:** for GPKG this changes the on-disk geometry column from the input name to `geom`, so `gpkg_gbx` reads it back as `geom` (not `geom_0`). Update the affected round-trip expectation in this task. - -- [ ] **Step 1: Write the failing test** - -Append to `python/geobrix/test/ds/test_vector_writer.py` (mirror its existing imports / `register` usage): - -```python -def test_gpkg_output_uses_format_default_geom_name(spark, tmp_path): - # GPKG output should use the format-default geometry column name `geom`, - # not the input column name, so an arbitrary input name doesn't leak out. - from databricks.labs.gbx.ds.register import register - - register(spark) - out = str(tmp_path / "out.gpkg") - rows = [("a", bytearray(_to_wkb(_Point(1.0, 2.0))), "4326", "")] - df = spark.createDataFrame( - rows, schema="name string, the_geom binary, epsg string, p4 string" - ) - ( - df.write.format("gpkg_gbx") - .mode("overwrite") - .option("geomCol", "the_geom") - .option("sridCol", "epsg") - .save(out) - ) - import pyogrio - - info = pyogrio.read_info(out) - assert info["geometry_name"] == "geom" -``` - -(If `test_vector_writer.py` does not already import `to_wkb`/`Point`, add -`from shapely import Point as _Point` and `from shapely import to_wkb as _to_wkb` -at the top in Step 3.) - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path "python/geobrix/test/ds/test_vector_writer.py::test_gpkg_output_uses_format_default_geom_name"` -Expected: FAIL — `geometry_name` is `the_geom` (the input column name), not `geom`. - -- [ ] **Step 3: Implement the helper + use it** - -Add near `_writer_col_roles` in `vector.py`: - -```python -# Output geometry field name per driver. GeoJSON/GeoJSONSeq/Shapefile geometry -# is structural (no named field), so the value is inert there; GPKG/FileGDB name -# the geometry column, so use the format default rather than the input column -# name (which may be arbitrary once geomCol is in play). -_OUTPUT_GEOM_NAME = {"GPKG": "geom", "OpenFileGDB": "SHAPE"} - - -def _output_geom_name(driver, geom_col): - return _OUTPUT_GEOM_NAME.get(driver, geom_col) -``` - -In `VectorGbxWriter._write_local` (the pyogrio Arrow path), change the -`geometry_name=self.geom_col` in the `kw = dict(...)` to: - -```python - geometry_name=_output_geom_name(self.driver, self.geom_col), -``` - -In `VectorGbxWriter._write_local_osgeo_gdb`, name the created geometry field by -passing `geometry_name` where the layer is created — set the FileGDB geometry -field name to `_output_geom_name(self.driver, self.geom_col)` (i.e. `SHAPE`). -(Use the `geometry_name=` argument GDAL's `CreateLayer` options accept, or the -layer-creation option `["GEOMETRY_NAME=SHAPE"]`.) - -The classic pyogrio fallback path (`_write_local_classic`) and `GeoJSONLGbxWriter` -write GeoJSON/GeoJSONSeq where geometry is structural, so leave their -`geometry_name=self.geom_col` as-is (inert) for minimal change. - -- [ ] **Step 4: Run + fix the affected round-trip expectation** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_vector_writer.py` -Expected: the new test PASSES. If an existing GPKG round-trip test now reads the -geometry column back as `geom` (it derived `geom_0` before), update that test's -expected geometry-column name to `geom`. Re-run until green. - -Also run the parity suite to catch any GPKG geom-name assumption: -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_vector_parity.py` -Expected: PASS (update any GPKG geom-column-name expectation the same way). - -- [ ] **Step 5: Lint + commit** - -```bash -bash scripts/commands/gbx-docker-exec.sh "cd /root/geobrix && black python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_vector_writer.py && isort python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_vector_writer.py" -bash scripts/commands/gbx-lint-python.sh --check -git add python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_vector_writer.py -git commit -m "feat(ds): vector writers name output geometry per format (GPKG geom, FileGDB SHAPE)" -``` - ---- - -### Task 4: Heavy — `geojsonl` `resolveRoles` overrides + option threading - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/vectorx/ds/geojsonl/GeoJSONL_DataSource.scala` (`resolveRoles`) -- Modify: `src/main/scala/com/databricks/labs/gbx/vectorx/ds/geojsonl/GeoJSONL_Table.scala` (`newWriteBuilder` validation call) -- Modify: `src/main/scala/com/databricks/labs/gbx/vectorx/ds/geojsonl/GeoJSONL_RowWriter.scala` (read options, pass to `resolveRoles`) -- Test: `src/test/scala/com/databricks/labs/gbx/vectorx/ds/geojsonl/GeoJSONLWriterTest.scala` - -**Interfaces:** -- Produces: `resolveRoles(schema, geomCol: Option[String] = None, sridCol: Option[String] = None, projCol: Option[String] = None): ColRoles` with the same rules as the light tier (geom & srid required; proj optional; an explicit override must name an existing column). - -- [ ] **Step 1: Write the failing test** - -Append to `src/test/scala/com/databricks/labs/gbx/vectorx/ds/geojsonl/GeoJSONLWriterTest.scala` a case mirroring the existing write tests but with non-convention column names driven by options: - -```scala - test("geomCol/sridCol options write a non-convention frame") { - import spark.implicits._ - val rows = Seq( - ("a", wkb(-73.9, 40.7), "4326", ""), - ("b", wkb(-0.1, 51.5), "4326", "") - ).toDF("name", "the_geom", "epsg", "p4") - val out = s"$tmpDir/renamed" - rows.write - .format("geojsonl") - .mode("overwrite") - .option("geomCol", "the_geom") - .option("sridCol", "epsg") - .option("projCol", "p4") - .save(out) - val back = spark.read.format("geojson_ogr").option("multi", "true").load(out) - back.count() shouldEqual 2 - } -``` - -(Reuse the test's existing `wkb(...)` / WKB helper and `tmpDir` fixture; match -the file's exact helper names when implementing.) - -Also add a `resolveRoles` unit assertion (no Spark write) in the same file: - -```scala - test("resolveRoles honors overrides and requires srid") { - import org.apache.spark.sql.types._ - val sch = StructType(Seq( - StructField("the_geom", BinaryType), StructField("epsg", StringType), - StructField("p4", StringType), StructField("v", LongType))) - val r = GeoJSONL_DataSource.resolveRoles( - sch, Some("the_geom"), Some("epsg"), Some("p4")) - r.geomCol shouldEqual "the_geom" - r.sridCol shouldEqual "epsg" - // geomCol given but no srid resolvable -> error - val bad = StructType(Seq(StructField("the_geom", BinaryType))) - an[IllegalArgumentException] should be thrownBy - GeoJSONL_DataSource.resolveRoles(bad, Some("the_geom"), None, None) - } -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.vectorx.ds.geojsonl.GeoJSONLWriterTest'` -Expected: FAIL — `resolveRoles` does not take the extra args / the renamed-frame write errors. - -- [ ] **Step 3: Implement `resolveRoles` overrides** - -Replace `resolveRoles` in `GeoJSONL_DataSource.scala` with: - -```scala - def resolveRoles( - schema: StructType, - geomColOpt: Option[String] = None, - sridColOpt: Option[String] = None, - projColOpt: Option[String] = None - ): ColRoles = { - val names = schema.fieldNames.toSeq - - // geometry (required) - val geomCol = geomColOpt match { - case Some(g) => - if (!names.contains(g)) - throw new IllegalArgumentException( - s"`geojsonl` writer geomCol='$g' is not a column; got ${names.mkString("[", ", ", "]")}.") - g - case None => - val sridCols = names.filter(_.endsWith("_srid")) - if (sridCols.isEmpty) - throw new IllegalArgumentException( - "`geojsonl` writer input needs a geometry/'*_srid' column pair (from a *_ogr " + - s"reader) or an explicit geomCol option; got ${names.mkString("[", ", ", "]")}.") - val g = sridCols.head.dropRight("_srid".length) - if (!names.contains(g)) - throw new IllegalArgumentException( - s"`geojsonl` writer found srid column '${sridCols.head}' but no geometry column '$g'.") - g - } - - // srid (required: option, else _srid) - val sridCol = sridColOpt match { - case Some(s) => - if (!names.contains(s)) - throw new IllegalArgumentException( - s"`geojsonl` writer sridCol='$s' is not a column; got ${names.mkString("[", ", ", "]")}.") - s - case None => - val s = geomCol + "_srid" - if (!names.contains(s)) - throw new IllegalArgumentException( - s"`geojsonl` writer needs a SRID column: pass sridCol, or add a '$s' column " + - "(authority code, '0' if unknown).") - s - } - - // proj (optional: explicit must exist; default may be absent) - val projCol = projColOpt match { - case Some(p) => - if (!names.contains(p)) - throw new IllegalArgumentException( - s"`geojsonl` writer projCol='$p' is not a column; got ${names.mkString("[", ", ", "]")}.") - p - case None => geomCol + "_srid_proj" - } - - val attrCols = names.filterNot(n => n == geomCol || n == sridCol || n == projCol) - val geomType = schema(geomCol).dataType - val geomIsWkb = geomType match { - case BinaryType => true - case StringType => false - case other => - throw new IllegalArgumentException( - s"`geojsonl` writer geometry column '$geomCol' must be BINARY (WKB) or STRING (WKT); got $other.") - } - ColRoles(geomCol, sridCol, projCol, attrCols, geomIsWkb) - } -``` - -- [ ] **Step 4: Thread the options at both call sites** - -In `GeoJSONL_RowWriter.scala`, after the existing `ciOptions` reads (around the -`layerNameOpt` line), add: - -```scala - private val geomColOpt: Option[String] = ciOptions.get("geomcol") - private val sridColOpt: Option[String] = ciOptions.get("sridcol") - private val projColOpt: Option[String] = ciOptions.get("projcol") -``` - -and change the role resolution line to: - -```scala - private val roles = GeoJSONL_DataSource.resolveRoles(schema, geomColOpt, sridColOpt, projColOpt) -``` - -In `GeoJSONL_Table.scala` `newWriteBuilder`, the early validation call must read -the same options (the `info.options()` `CaseInsensitiveStringMap` is -case-insensitive, so look up the camelCase keys): - -```scala - val o = info.options() - GeoJSONL_DataSource.resolveRoles( - info.schema(), - Option(o.get("geomCol")), - Option(o.get("sridCol")), - Option(o.get("projCol")) - ) - new GeoJSONL_WriteBuilder(info.schema(), properties ++ info.options().asScala) -``` - -- [ ] **Step 5: Run to verify pass (in Docker/Maven)** - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.vectorx.ds.geojsonl.GeoJSONLWriterTest'` -Expected: PASS (existing + the two new cases). This compiles + runs in the -`geobrix-dev` container; allow a few minutes. - -- [ ] **Step 6: Scalastyle + commit** - -```bash -bash scripts/commands/gbx-lint-scalastyle.sh -git add src/main/scala/com/databricks/labs/gbx/vectorx/ds/geojsonl/GeoJSONL_DataSource.scala src/main/scala/com/databricks/labs/gbx/vectorx/ds/geojsonl/GeoJSONL_Table.scala src/main/scala/com/databricks/labs/gbx/vectorx/ds/geojsonl/GeoJSONL_RowWriter.scala src/test/scala/com/databricks/labs/gbx/vectorx/ds/geojsonl/GeoJSONLWriterTest.scala -git commit -m "feat(vectorx): heavy geojsonl writer accepts geomCol/sridCol/projCol options" -``` - ---- - -### Task 5: Docs — document the options on the writer pages - -**Files:** -- Modify: `docs/docs/writers/geojsonl.mdx`, `docs/docs/writers/geojson.mdx`, `docs/docs/writers/geopackage.mdx`, `docs/docs/writers/shapefile.mdx`, `docs/docs/writers/filegdb.mdx` (whichever document write options) - -**Interfaces:** docs only. - -- [ ] **Step 1: Add an options note** - -On each vector writer page that lists write options, document `geomCol` / -`sridCol` / `projCol`: "Override the geometry / SRID / PROJ4 column names -(default to `` / `_srid` / `_srid_proj`); srid is required -(`"0"` if unknown), proj optional." Link the shared model to the -[Named Vector Formats](./overview#named-vector-formats) section. For -`geojsonl.mdx`, note the options work in both the lightweight and heavyweight -tiers; for the other formats, note the writer is lightweight-tier. - -- [ ] **Step 2: Verify no internal-vocabulary leak** - -Run: `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/writers/` -Expected: no output. - -- [ ] **Step 3: Commit** - -```bash -git add docs/docs/writers/ -git commit -m "docs(writers): document geomCol/sridCol/projCol options" -``` - ---- - -## Out of scope / follow-ups - -- Adding *write* support for the other heavy OGR formats (`shapefile`/`gpkg`/`geojson`/`file_gdb`) — they are heavy read-only today; that is a separate net-new effort. -- A JAR rebuild + cluster restage is needed before the heavy `geojsonl` change is usable on a cluster (the light change ships in the wheel). - -## Self-Review - -**Spec coverage:** light `_writer_col_roles` overrides (Task 1) ✓; thread through both light writers (Task 2) ✓; per-format output geom name (Task 3) ✓; heavy `geojsonl` `resolveRoles` + option threading at both call sites (Task 4) ✓; srid required / proj optional / defaults-if-present encoded in both tiers ✓; identical option names + semantics ✓; tests light + heavy ✓; docs ✓; heavy-only-geojsonl scoping respected (no other heavy writers touched) ✓. - -**Placeholder scan:** none — every code step has complete code. Task 3's osgeo `CreateLayer` geometry-name uses the documented GDAL `GEOMETRY_NAME=` layer option; Task 4's test helper names defer to the existing fixture (called out explicitly). - -**Type consistency:** `_writer_col_roles(schema, geom_col, srid_col, proj_col)` signature identical across Tasks 1-3; Scala `resolveRoles(schema, geomColOpt, sridColOpt, projColOpt)` identical across Task 4 sites; option keys `geomcol`/`sridcol`/`projcol` (lowercased) consistent in both light writers and the heavy `ciOptions`; `_output_geom_name(driver, geom_col)` consistent between definition and call sites. diff --git a/docs/superpowers/plans/2026-06-26-single-file-vector-io-ux.md b/docs/superpowers/plans/2026-06-26-single-file-vector-io-ux.md deleted file mode 100644 index 2a24b4698..000000000 --- a/docs/superpowers/plans/2026-06-26-single-file-vector-io-ux.md +++ /dev/null @@ -1,369 +0,0 @@ -# Single-File / Bundle Vector I/O UX — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Give GeoBrix's single-file/bundle vector writers a consistent `fileName` option + adaptive output naming, make the shapefile reader's `.load()` contract identical across tiers, and replace the cryptic heavy read-only-write failure with a clear error. - -**Architecture:** One pure-function path-resolution helper in the light writer (`ds/vector.py`) drives all four light single-file writers; the heavy shapefile reader gains bare-`.shp` sidecar staging + recursive-dir union (light reader gains recursive listing) with a shared schema-divergence error; heavy read-only OGR formats reject writes before `inferSchema` runs. - -**Tech Stack:** Light = Python 3.12 / PySpark DataSource V2 (pyogrio); heavy = Scala 2.13 / Spark 4 DataSource V2 (GDAL/OGR JNI). Tests/build in the `geobrix-dev` Docker container via `gbx:*` commands. - -## Global Constraints - -- Branch: **`beta/0.4.0`** (commit directly; no per-feature branch). -- Light tier is **pure Python/PySpark** — no `_jvm` / `spark.conf` / `sparkContext` in product code. -- Heavy tier is **Scala 2.13 / Spark 4.0**. -- **DRY:** one shared `_resolve_single_file_output` helper drives all light single-file writers; one shared schema-divergence error message string reused across tiers (same wording). -- **No new `gbx_` functions** → binding parity unaffected. -- Canonical extensions: `gpkg`→`.gpkg`, `geojson`→`.geojson`, shapefile+`zip`→`.shp.zip`, file_gdb→`.gdb` (or `.gdb.zip` with `zip=true`). -- Prefer **pure-unit tests** for path-resolution logic (no Spark); use real `/Volumes` sample data only for round-trip tests that need a cluster/Docker. -- Doc-tests are the documentation source (`docs/tests/...`); user-facing docs under `docs/docs/`. -- **Out of scope:** adding heavy single-file *vector* writers (heavy vector stays read-only); implementing PMTiles `fileName` (contract only — helper must be reusable); sharded/dir-writer naming (`geojsonl_gbx`/`geojsonl_ogr`/raster tile dirs). - -## File Structure - -| File | Responsibility | Change | -|---|---|---| -| `python/geobrix/src/databricks/labs/gbx/ds/vector.py` | light writers + reader | add helper; wire 4 writers; recursive reader listing + schema-divergence error | -| `python/geobrix/test/ds/test_vector_filename.py` | light unit + round-trip tests | **create** | -| `python/geobrix/test/ds/test_vector_reader_contract.py` | light reader contract tests | **create** | -| `src/main/scala/com/databricks/labs/gbx/util/HadoopUtils.scala` | heavy listing/staging | bare-`.shp` sidecar discovery; shared schema-divergence message | -| `src/main/scala/com/databricks/labs/gbx/vectorx/ds/ogr/OGR_DataSource.scala` | heavy reader schema infer | stage siblings for bare `.shp`; read-only-write guard | -| `src/main/scala/com/databricks/labs/gbx/vectorx/ds/ogr/OGR_Batch.scala` | heavy partition plan | schema-divergence detection | -| `src/main/scala/com/databricks/labs/gbx/vectorx/ds/ogr/OGR_Table.scala` | heavy capabilities | (read-only guard support if needed) | -| `src/test/scala/com/databricks/labs/gbx/vectorx/ds/ogr/OgrReaderContractTest.scala` | heavy reader/write-guard tests | **create** | -| `docs/docs/writers/*.mdx`, `docs/docs/readers/*.mdx` | user docs | `fileName`/naming + `.load()` contract | - -Canonical names used across tasks (define once, reuse): -- `_resolve_single_file_output(path: str, file_name: str | None, ext: str) -> str` -- `_canonical_ext(driver: str, zip_enabled: bool) -> str` -- `_complete_ext(name: str, ext: str) -> str` -- Schema-divergence message: `"shapefile reader: shapefiles under have differing schemas; load them separately or use a single-stem directory. Stems: , ."` - ---- - -## Workstream A — Light writer `fileName` + adaptive naming - -### Task A1: `_complete_ext` + `_canonical_ext` helpers (pure unit) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (new module-level functions near `zip_shapefile`, ~line 160) -- Test: `python/geobrix/test/ds/test_vector_filename.py` (create) - -**Interfaces — Produces:** -- `_canonical_ext(driver: str, zip_enabled: bool) -> str` -- `_complete_ext(name: str, ext: str) -> str` - -- [ ] **Step 1: Write failing tests** - -```python -# python/geobrix/test/ds/test_vector_filename.py -from databricks.labs.gbx.ds.vector import _canonical_ext, _complete_ext - -def test_canonical_ext(): - assert _canonical_ext("GPKG", False) == ".gpkg" - assert _canonical_ext("GeoJSON", False) == ".geojson" - assert _canonical_ext("ESRI Shapefile", True) == ".shp.zip" - assert _canonical_ext("OpenFileGDB", False) == ".gdb" - assert _canonical_ext("OpenFileGDB", True) == ".gdb.zip" - -def test_complete_ext_appends_when_missing(): - assert _complete_ext("roads", ".shp.zip") == "roads.shp.zip" - assert _complete_ext("roads.shp", ".shp.zip") == "roads.shp.zip" # partial -> complete - assert _complete_ext("roads.shp.zip", ".shp.zip") == "roads.shp.zip" # already complete - assert _complete_ext("city", ".gpkg") == "city.gpkg" - assert _complete_ext("city.gpkg", ".gpkg") == "city.gpkg" - -def test_complete_ext_rejects_wrong_geo_ext(): - import pytest - with pytest.raises(ValueError, match="expected .shp.zip"): - _complete_ext("roads.gpkg", ".shp.zip") -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_vector_filename.py` -Expected: FAIL (ImportError: cannot import `_canonical_ext`). - -- [ ] **Step 3: Implement** - -```python -# in ds/vector.py -_CANONICAL_EXT = { - "GPKG": ".gpkg", - "GeoJSON": ".geojson", - "ESRI Shapefile": ".shp.zip", # single-file form is zip; non-zip is a dir bundle (out of scope) - "OpenFileGDB": ".gdb", -} -# Recognized geo extensions, longest-first, so multi-part suffixes match before their parts. -_RECOGNIZED_EXTS = (".shp.zip", ".gdb.zip", ".gpkg", ".geojson", ".gdb", ".shp") - -def _canonical_ext(driver: str, zip_enabled: bool) -> str: - if driver == "OpenFileGDB": - return ".gdb.zip" if zip_enabled else ".gdb" - return _CANONICAL_EXT[driver] - -def _complete_ext(name: str, ext: str) -> str: - low = name.lower() - if low.endswith(ext): - return name - # Incremental completion for multi-part ext (e.g. "roads.shp" -> "roads.shp.zip"). - for k in range(1, ext.count(".") + 1): - suffix = "." + ".".join(ext.strip(".").split(".")[:k]) - if low.endswith(suffix) and ext.startswith(suffix): - return name + ext[len(suffix):] - # Reject a DIFFERENT recognized geo extension rather than double-append. - for other in _RECOGNIZED_EXTS: - if other != ext and low.endswith(other): - raise ValueError( - f"output name '{name}' ends with '{other}' but this writer expects '{ext}'." - ) - return name + ext -``` - -- [ ] **Step 4: Run to verify pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_vector_filename.py` -Expected: PASS (3 tests). - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_vector_filename.py -git commit -m "feat(vector): canonical-ext + ext-completion helpers for single-file writers" -m "Co-authored-by: Isaac" -``` - -### Task A2: `_resolve_single_file_output` (the 3-case contract, pure unit) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/vector.py` -- Test: `python/geobrix/test/ds/test_vector_filename.py` - -**Interfaces — Consumes:** `_complete_ext`. **Produces:** `_resolve_single_file_output(path, file_name, ext) -> str` (returns the resolved output path; creates parent dirs as a side effect). - -- [ ] **Step 1: Write failing tests** (use `tmp_path` for the existing-dir case) - -```python -import os -from databricks.labs.gbx.ds.vector import _resolve_single_file_output as R - -def test_case1_filename_given(tmp_path): - out = R(str(tmp_path / "newdir"), "roads", ".shp.zip") - assert out == str(tmp_path / "newdir" / "roads.shp.zip") - assert os.path.isdir(tmp_path / "newdir") # parent created - -def test_case2_existing_dir_no_filename(tmp_path): - d = tmp_path / "roads_dir"; d.mkdir() - out = R(str(d), None, ".shp.zip") - assert out == str(d / "roads_dir.shp.zip") # named after the dir, under it - -def test_case3_stem_path_no_filename(tmp_path): - out = R(str(tmp_path / "sub" / "roads"), None, ".gpkg") - assert out == str(tmp_path / "sub" / "roads.gpkg") # complete ext on the stem - assert os.path.isdir(tmp_path / "sub") # parent created - -def test_filename_extension_completed(tmp_path): - out = R(str(tmp_path), "roads.shp", ".shp.zip") - assert out == str(tmp_path / "roads.shp.zip") -``` - -- [ ] **Step 2: Run to verify failure** — `gbx:test:python --path .../test_vector_filename.py` → FAIL (import). - -- [ ] **Step 3: Implement** - -```python -def _resolve_single_file_output(path: str, file_name, ext: str) -> str: - """Resolve the output path for a single-file/single-unit writer. See - docs/superpowers/specs/2026-06-26-writer-filename-naming-design.md (3-case contract). - Creates the parent directory as needed. Pure path logic + one mkdirs side effect.""" - path = path.rstrip("/") - if file_name: # case 1: path is the parent dir - os.makedirs(path, exist_ok=True) - return os.path.join(path, _complete_ext(file_name, ext)) - if os.path.isdir(path): # case 2: existing dir -> name after it, under it - return os.path.join(path, _complete_ext(os.path.basename(path), ext)) - # case 3: file-like target -> complete ext, create parent - parent = os.path.dirname(path) or "." - os.makedirs(parent, exist_ok=True) - return _complete_ext(path, ext) -``` - -- [ ] **Step 4: Run to verify pass** — expected PASS (4 tests). -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_vector_filename.py -git commit -m "feat(vector): _resolve_single_file_output adaptive-naming helper" -m "Co-authored-by: Isaac" -``` - -### Task A3: Wire the helper + `fileName` into `VectorGbxWriter` - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (`VectorGbxWriter.__init__`, currently `:721-784` — replace the inline zip-extension block `:738-753`) - -**Interfaces — Consumes:** `_resolve_single_file_output`, `_canonical_ext`. - -- [ ] **Step 1: Replace the inline naming block.** Delete lines `738-753` (the `if self.zip:` extension block) and the bare `self.path = to_local_path(path)` assignment's downstream use, replacing with: - -```python -self.zip = opts.get("zip", "false").lower() == "true" and self.driver in ( - "ESRI Shapefile", "OpenFileGDB", -) -self._file_name = opts.get("filename") # .option("fileName", ...) (opts are lower-cased) -# Single-file/unit writers (gpkg/geojson, shapefile+zip, file_gdb): adaptive naming. -# Non-zip shapefile remains a directory bundle (existing behavior; not single-file). -if self.driver in ("GPKG", "GeoJSON") or self.zip or self.driver == "OpenFileGDB": - ext = _canonical_ext(self.driver, self.zip) - self.path = _resolve_single_file_output(self.path, self._file_name, ext) -``` - -> NOTE: non-zip `ESRI Shapefile` keeps the prior directory-bundle path handling — do not route it through the helper (out of scope). Confirm the `else` branch preserves `self.path = to_local_path(path)` for that case. - -- [ ] **Step 2: Add a unit test asserting the writer resolves paths** (construct the writer with a fake schema; assert `self.path`): - -```python -def test_writer_resolves_gpkg_stem(tmp_path): - from databricks.labs.gbx.ds.vector import VectorGbxWriter - from pyspark.sql.types import StructType, StructField, BinaryType, IntegerType - sch = StructType([StructField("geom", BinaryType()), StructField("geom_srid", IntegerType())]) - w = VectorGbxWriter(str(tmp_path / "city"), sch, "GPKG", {}, overwrite=True) - assert w.path == str(tmp_path / "city.gpkg") -``` - -- [ ] **Step 3: Run** — `gbx:test:python --path python/geobrix/test/ds/test_vector_filename.py` → PASS. -- [ ] **Step 4: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/vector.py python/geobrix/test/ds/test_vector_filename.py -git commit -m "feat(vector): fileName option + adaptive naming in light single-file writers" -m "Co-authored-by: Isaac" -``` - -### Task A4: Round-trip tests on a Volume (Docker) - -**Files:** Test: `python/geobrix/test/ds/test_vector_filename.py` - -- [ ] **Step 1: Add round-trip tests** (gated on Docker `/Volumes` per existing corpus-test pattern; see [[docker-volumes-for-integration-tests]]). For each of `gpkg_gbx`, `geojson_gbx`, `shapefile_gbx`(zip), `file_gdb_gbx`: write with (a) stem path, (b) existing dir, (c) `.option("fileName", ...)`, then read back and assert row count + resolved filename matches the contract. - -- [ ] **Step 2: Run** — `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_vector_filename.py` (in container, Volumes mounted). Expected PASS. -- [ ] **Step 3: Commit.** - -### Task A5: Writer docs - -**Files:** Modify `docs/docs/writers/overview.mdx` + each single-file writer page. - -- [ ] Document the `fileName` option + the 3-case naming behavior (one table mirroring the spec). Run `gbx:docs:restart`; verify rendering. Commit. - ---- - -## Workstream B — Shapefile reader `.load()` contract (light + heavy) - -### Task B1: Light — recursive directory listing + stem grouping - -**Files:** Modify `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (`VectorGbxReader._members`, `:457-473`). Test: `python/geobrix/test/ds/test_vector_reader_contract.py` (create). - -- [ ] **Step 1: Failing tests** — dir-of-dirs of shapefiles should enumerate all `.shp` recursively: - -```python -def test_members_recursive_shapefiles(tmp_path): - # build sub1/a.shp, sub2/b.shp (+ sidecars) ; assert both enumerated - ... - members = reader._members() - assert {os.path.basename(m) for m in members} == {"a.shp", "b.shp"} -``` - -- [ ] **Step 2: Run → FAIL** (current `os.listdir` is non-recursive). -- [ ] **Step 3: Implement** — replace `os.listdir(self.path)` with `os.walk`-based recursive collection, still filtered by `_EXT_FOR_DRIVER` and still returning `[self.path]` for a file / `.gdb`: - -```python -def _members(self): - if not os.path.isdir(self.path) or self.path.lower().rstrip("/").endswith(".gdb"): - return [self.path] - exts = self._EXT_FOR_DRIVER.get(self.driver) or () - members = [] - for root, _dirs, files in os.walk(self.path): - for n in sorted(files): - low = n.lower() - if (exts and low.endswith(exts)) or low.rstrip("/").endswith(".gdb"): - members.append(os.path.join(root, n)) - return sorted(members) or [self.path] -``` - -- [ ] **Step 4: Run → PASS.** Also re-run existing reader tests to confirm flat-dir + bare-file unchanged. -- [ ] **Step 5: Commit.** - -### Task B2: Light — schema-divergence error - -**Files:** Modify `ds/vector.py` (reader schema inference path). Test: `test_vector_reader_contract.py`. - -- [ ] **Step 1: Failing test** — a dir with two shapefiles of differing schemas raises the shared message. -- [ ] **Step 2: Run → FAIL.** -- [ ] **Step 3: Implement** — when `> 1` member, infer each member's schema (pyogrio `read_info`), and if any differs from the first, raise `ValueError` with the shared message (Global Constraints). Single member → unchanged. -- [ ] **Step 4: Run → PASS** (+ same-schema union still works). -- [ ] **Step 5: Commit.** - -### Task B3: Heavy — bare-`.shp` sidecar staging - -**Files:** Modify `src/main/scala/.../util/HadoopUtils.scala` (`stageHeadForSchemaSpark` / `listDataFilesSpark`) + `OGR_DataSource.scala`. Test: `OgrReaderContractTest.scala` (create). - -- [ ] **Step 1: Failing test** — `spark.read.format("shapefile_ogr").load("

)"` AND the dataset query MUST include a field `{"name":"geo()","expression":"ST_ASGEOJSON(``)"}`. Both halves required or the map is blank. -- **Ranking metric = detection count + MEAN kg/hr, never summed** (memory `operator_emissions_leaderboard`). -- **Guarded downloads:** a context download failure logs a WARNING and continues (context is additive; core demo unaffected) — same pattern as EMIT/CM in `land.py`. -- **No aliases; user-facing docs voice** — nothing under `docs/docs/` leaks internal vocabulary. -- **No push until user go** (memory `hold-pushes-batch-more`); PR via `gh auth switch --user mjohns-databricks` (memory `gh_account_for_geobrix`). -- **Verification reality:** land helpers (Tasks 1–2) are locally unit-tested with pytest. The SDP transforms and dashboard (Tasks 3–5) run only on Databricks Serverless — their verification is *bundle deploy + pipeline run + live row query* (and chrome-devtools for the dashboard), driven by the orchestrator, not local pytest. Implementer subagents author the code and run the local unit tests that exist; the orchestrator runs the cluster verification between tasks. - -## Key interfaces (verified against current code) - -- **Light vector reader** (`from databricks.labs.gbx.ds.register import register`): `spark.read.format("geojson_gbx").load(path)` / `format("shapefile_gbx").load(path.zip)`. Output schema = source attribute columns (preserved by name) + `geom_0` (WKB `BinaryType`, `asWKB` default true) + `geom_0_srid` (String) + `geom_0_srid_proj` (String). Convert with `st_setsrid(st_geomfromwkb(geom_0), 4326)`. -- **`_config.paths(spark)`** returns a dict of Volume dirs under `/Volumes/{catalog}/{schema}/{volume}/vapor-eyes-lf`. **`_config.cfg(spark)`** returns config incl. `bbox`. **`register_gbx(spark)`** registers pyrx + pyvx + ds readers. -- **Point layer** `cm_plume_attributed` (gold, ~3724 rows): columns include `plume_id`, `emission_rate_kg_hr`, `lead_operator`, `lon`, `lat`, `plume_geom` (native GEOMETRY 4326), `observation_date`. Build point via `st_setsrid(st_point(lon, lat), 4326)`. -- **EIA plays GeoJSON** attributes: `Shale_play`, `Basin` (filter `='Permian'` → 7 rows), `Lithology`, `Age_shale`, `Area_sq_km`. -- **TIGER counties** attributes: `STATEFP` (TX=`48`, NM=`35`), `GEOID` (FIPS), `NAME`. - ---- - -### Task 1: Config + Volume subtree for the `context` source - -**Files:** -- Modify: `notebooks/examples/vapor-eyes/lakeflow/transformations/_config.py` (add `context` to `paths`) -- Modify: `notebooks/examples/vapor-eyes/lakeflow/land/land.py:22` (`_subtree` add `context`) -- Modify: `notebooks/examples/vapor-eyes/lakeflow/databricks.yml` (job `--sources` add `context`) -- Test: `notebooks/examples/vapor-eyes/lakeflow/tests/test_land.py` (extend), or `tests/test_dates.py` sibling for config - -**Interfaces:** -- Produces: `paths(spark)["context"]` → `f"{root}/context"`; `_subtree(...)` dict includes `"context"`. - -- [ ] **Step 1: Write the failing test** in `tests/test_land.py` (add a test): - -```python -def test_subtree_includes_context(): - from land.land import _subtree - dirs = _subtree("cat", "sch", "vol") - assert dirs["context"].endswith("/vapor-eyes-lf/context") -``` - -- [ ] **Step 2: Run it to confirm it fails** - -Run: `cd notebooks/examples/vapor-eyes/lakeflow && python -m pytest tests/test_land.py::test_subtree_includes_context -q` -Expected: FAIL (KeyError: 'context'). - -- [ ] **Step 3: Add `context` to `_subtree`** in `land/land.py` — locate the dict returned by `_subtree` (around line 22) and add a `"context"` entry mirroring the others, e.g. `"context": f"{root}/context",`. - -- [ ] **Step 4: Add `context` to `_config.paths`** — in `_config.py`, add `"context": f"{root}/context",` to the returned dict. - -- [ ] **Step 5: Add `context` to the job sources** — in `databricks.yml`, change the `land` task `parameters` `"--sources", "s5p,emit,wells,s2,cm"` → `"s5p,emit,wells,s2,cm,context"`. - -- [ ] **Step 6: Run test to confirm it passes** - -Run: `cd notebooks/examples/vapor-eyes/lakeflow && python -m pytest tests/test_land.py::test_subtree_includes_context -q` -Expected: PASS. - -- [ ] **Step 7: Commit** - -```bash -git add notebooks/examples/vapor-eyes/lakeflow/transformations/_config.py notebooks/examples/vapor-eyes/lakeflow/land/land.py notebooks/examples/vapor-eyes/lakeflow/databricks.yml notebooks/examples/vapor-eyes/lakeflow/tests/test_land.py -git commit -m "feat(vapor-eyes): add context Volume subtree + source toggle" -``` - ---- - -### Task 2: `context` land source — download EIA plays + TIGER counties - -**Files:** -- Modify: `notebooks/examples/vapor-eyes/lakeflow/land/land.py` (add `_land_context`, `_dl_eia_plays`, `_dl_tiger_counties`; dispatch from `run_land`) -- Test: `notebooks/examples/vapor-eyes/lakeflow/tests/test_land.py` - -**Interfaces:** -- Consumes: `_subtree(...)["context"]`. -- Produces: `run_land(..., sources=["context"])` writes `context/plays/plays.geojson` and `context/counties/cb_2024_us_county_500k.zip` (+ unzipped `.shp` set); returns `staged["context"]` = count of files landed. Guarded: on any download error, logs `WARNING` and sets `staged["context"] = 0`. - -- [ ] **Step 1: Write the failing tests** in `tests/test_land.py`: - -```python -def test_eia_plays_url_is_permian_geojson(): - from land.land import _EIA_PLAYS_URL - assert _EIA_PLAYS_URL.startswith("https://hub.arcgis.com/api/download/v1/items/") - assert "geojson" in _EIA_PLAYS_URL - -def test_tiger_counties_url(): - from land.land import _TIGER_COUNTIES_URL - assert _TIGER_COUNTIES_URL == ( - "https://www2.census.gov/geo/tiger/GENZ2024/shp/cb_2024_us_county_500k.zip" - ) - -def test_land_context_guarded_on_download_error(tmp_path, monkeypatch): - """A download failure must not raise — it logs and returns 0.""" - import land.land as L - def boom(*a, **k): - raise RuntimeError("network down") - monkeypatch.setattr(L, "_http_get_to_file", boom) - n = L._land_context(str(tmp_path)) - assert n == 0 -``` - -- [ ] **Step 2: Run to confirm failure** - -Run: `cd notebooks/examples/vapor-eyes/lakeflow && python -m pytest tests/test_land.py -q -k context or url` -Expected: FAIL (ImportError / AttributeError). - -- [ ] **Step 3: Implement the download helpers** in `land/land.py` (near `_land_cm`): - -```python -_EIA_PLAYS_URL = ( - "https://hub.arcgis.com/api/download/v1/items/" - "3f001fba00dc4add8dbd00542d61e4da/geojson?redirect=true&layers=0" -) -_TIGER_COUNTIES_URL = ( - "https://www2.census.gov/geo/tiger/GENZ2024/shp/cb_2024_us_county_500k.zip" -) - - -def _http_get_to_file(url, dst, timeout=180): - """Stream an HTTP GET to a local/Volume file path. Raises on non-200.""" - import requests - resp = requests.get(url, timeout=timeout, stream=True) - resp.raise_for_status() - with open(dst, "wb") as fh: - for chunk in resp.iter_content(chunk_size=1 << 20): - if chunk: - fh.write(chunk) - - -def _land_context(context_dir): - """Download the two static Permian context geometry sources into - context_dir/{plays,counties}. Returns the number of files landed. - - Guarded: any download failure logs a WARNING and is skipped — context is - additive, so a failure must not abort the (already-valuable) core demo.""" - import os - landed = 0 - plays_dir = os.path.join(context_dir, "plays") - counties_dir = os.path.join(context_dir, "counties") - os.makedirs(plays_dir, exist_ok=True) - os.makedirs(counties_dir, exist_ok=True) - try: - dst = os.path.join(plays_dir, "plays.geojson") - _http_get_to_file(_EIA_PLAYS_URL, dst) - print(f"... context: EIA plays -> {dst}") - landed += 1 - except Exception as e: # noqa: BLE001 - guarded, additive source - print(f"... WARNING: EIA plays download failed ({e}); skipping") - try: - dst = os.path.join(counties_dir, "cb_2024_us_county_500k.zip") - _http_get_to_file(_TIGER_COUNTIES_URL, dst) - print(f"... context: TIGER counties -> {dst}") - landed += 1 - except Exception as e: # noqa: BLE001 - guarded, additive source - print(f"... WARNING: TIGER counties download failed ({e}); skipping") - return landed -``` - -- [ ] **Step 4: Dispatch from `run_land`** — add, alongside the other `if "" in sources:` blocks: - -```python - if "context" in sources: - staged["context"] = _land_context(dirs["context"]) - _list_dir(dirs["context"], "context") -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `cd notebooks/examples/vapor-eyes/lakeflow && python -m pytest tests/test_land.py -q` -Expected: PASS (all, including existing). - -- [ ] **Step 6: Commit** - -```bash -git add notebooks/examples/vapor-eyes/lakeflow/land/land.py notebooks/examples/vapor-eyes/lakeflow/tests/test_land.py -git commit -m "feat(vapor-eyes): land EIA plays + TIGER counties context geometries" -``` - ---- - -### Task 3: Reference tables `ref_shale_plays` + `ref_counties` - -**Files:** -- Create: `notebooks/examples/vapor-eyes/lakeflow/transformations/context_reference.py` - -**Interfaces:** -- Consumes: `context/plays/plays.geojson`, `context/counties/cb_2024_us_county_500k.zip` on the Volume; `_config.paths`, `register_gbx`. -- Produces MVs: `ref_shale_plays(play_name STRING, area_sq_km DOUBLE, play_geom GEOMETRY)`, `ref_counties(county_name STRING, state_fp STRING, geoid STRING, county_geom GEOMETRY)`. Both geometries SRID 4326. - -- [ ] **Step 1: Create the transform file** - -```python -"""Context reference geometries: EIA Permian shale plays + TIGER counties. - -Static reference (not observations) — read once per run from the Volume with the -GeoBrix light vector reader (geojson_gbx / shapefile_gbx, pyogrio-backed, no JAR) -and emit native GEOMETRY at SRID 4326 for the AI/BI choropleths and the gold -point-in-polygon rollups. The reader emits geometry as WKB in `geom_0`.""" -from pyspark import pipelines as dp -from pyspark.sql import functions as F - -from _config import paths, register_gbx - - -@dp.materialized_view( - name="ref_shale_plays", - comment="EIA tight-oil/shale plays for the Permian basin (named play polygons)", -) -def ref_shale_plays(): - from pyspark.sql import SparkSession - spark = SparkSession.getActiveSession() - register_gbx(spark) - p = paths(spark) - src = f"{p['context']}/plays/plays.geojson" - return ( - spark.read.format("geojson_gbx").load(src) - .filter(F.col("Basin") == "Permian") - .select( - F.col("Shale_play").alias("play_name"), - F.col("Area_sq_km").cast("double").alias("area_sq_km"), - F.expr("st_setsrid(st_geomfromwkb(geom_0), 4326)").alias("play_geom"), - ) - ) - - -@dp.materialized_view( - name="ref_counties", - comment="US Census TIGER counties (TX + NM) clipped to the AOI", -) -def ref_counties(): - from pyspark.sql import SparkSession - spark = SparkSession.getActiveSession() - register_gbx(spark) - p = paths(spark) - src = f"{p['context']}/counties/cb_2024_us_county_500k.zip" - return ( - spark.read.format("shapefile_gbx").load(src) - .filter(F.col("STATEFP").isin("48", "35")) - .select( - F.col("NAME").alias("county_name"), - F.col("STATEFP").alias("state_fp"), - F.col("GEOID").alias("geoid"), - F.expr("st_setsrid(st_geomfromwkb(geom_0), 4326)").alias("county_geom"), - ) - ) -``` - -- [ ] **Step 2: Local syntax check** (no local Spark for DLP): - -Run: `cd notebooks/examples/vapor-eyes/lakeflow && python -c "import ast; ast.parse(open('transformations/context_reference.py').read()); print('ok')"` -Expected: `ok`. - -- [ ] **Step 3: Commit** - -```bash -git add notebooks/examples/vapor-eyes/lakeflow/transformations/context_reference.py -git commit -m "feat(vapor-eyes): ref_shale_plays + ref_counties reference tables" -``` - -- [ ] **Step 4: Cluster verification (orchestrator-driven, after commit)** — the orchestrator lands the context source and runs the pipeline, then confirms: - - `ref_shale_plays` has 7 rows, all `ST_SRID(play_geom)=4326`, `play_name` ∈ {Delaware, Bone Spring, Wolfcamp, Wolfcamp - Midland, Spraberry, Abo-Yeso, Glorieta-Yeso}. - - `ref_counties` non-empty (TX+NM AOI counties), all `ST_SRID(county_geom)=4326`. - - If `geom_0` is not the emitted column name at runtime, adjust the `select` to the actual reader geometry column (the reader default is `geom_0`; confirm from the failure message and fix in this file, not downstream). - ---- - -### Task 4: Gold rollups `emissions_by_play` + `detections_by_county` - -**Files:** -- Modify: `notebooks/examples/vapor-eyes/lakeflow/transformations/gold_analytics.py` (append two MVs) - -**Interfaces:** -- Consumes: `cm_plume_attributed` (lon, lat, emission_rate_kg_hr, lead_operator, plume_id), `ref_shale_plays`, `ref_counties`. -- Produces MVs (map-ready, geometry SRID 4326): - - `emissions_by_play(play_name, plume_count LONG, mean_emission_kg_hr DOUBLE, max_emission_kg_hr DOUBLE, active_operators LONG, play_geom GEOMETRY)` - - `detections_by_county(county_name, state_fp, geoid, plume_count LONG, mean_emission_kg_hr DOUBLE, max_emission_kg_hr DOUBLE, county_geom GEOMETRY)` - -- [ ] **Step 1: Append the two MVs** to `gold_analytics.py`: - -```python -@dp.materialized_view( - name="emissions_by_play", - comment="Carbon Mapper plume detections rolled up to EIA shale plays (map-ready)", -) -def emissions_by_play(): - from pyspark.sql import SparkSession - spark = SparkSession.getActiveSession() - plumes = spark.read.table("cm_plume_attributed").select( - "plume_id", "emission_rate_kg_hr", "lead_operator", - F.expr("st_setsrid(st_point(lon, lat), 4326)").alias("pt"), - ) - plays = spark.read.table("ref_shale_plays") - joined = plays.join( - plumes, F.expr("st_contains(play_geom, pt)"), "left" - ) - return joined.groupBy("play_name", "play_geom").agg( - F.count("plume_id").alias("plume_count"), - F.avg("emission_rate_kg_hr").alias("mean_emission_kg_hr"), - F.max("emission_rate_kg_hr").alias("max_emission_kg_hr"), - F.countDistinct("lead_operator").alias("active_operators"), - ) - - -@dp.materialized_view( - name="detections_by_county", - comment="Carbon Mapper plume detections rolled up to TX/NM counties (map-ready)", -) -def detections_by_county(): - from pyspark.sql import SparkSession - spark = SparkSession.getActiveSession() - plumes = spark.read.table("cm_plume_attributed").select( - "plume_id", "emission_rate_kg_hr", - F.expr("st_setsrid(st_point(lon, lat), 4326)").alias("pt"), - ) - counties = spark.read.table("ref_counties") - joined = counties.join( - plumes, F.expr("st_contains(county_geom, pt)"), "left" - ) - return joined.groupBy("county_name", "state_fp", "geoid", "county_geom").agg( - F.count("plume_id").alias("plume_count"), - F.avg("emission_rate_kg_hr").alias("mean_emission_kg_hr"), - F.max("emission_rate_kg_hr").alias("max_emission_kg_hr"), - ) -``` - -- [ ] **Step 2: Local syntax check** - -Run: `cd notebooks/examples/vapor-eyes/lakeflow && python -c "import ast; ast.parse(open('transformations/gold_analytics.py').read()); print('ok')"` -Expected: `ok`. - -- [ ] **Step 3: Commit** - -```bash -git add notebooks/examples/vapor-eyes/lakeflow/transformations/gold_analytics.py -git commit -m "feat(vapor-eyes): emissions_by_play + detections_by_county rollups" -``` - -- [ ] **Step 4: Cluster verification (orchestrator-driven)** — run pipeline; confirm both MVs non-empty, `plume_count` sums are plausible vs the 3724 attributed plumes (points outside all plays/counties simply don't join), geometry SRID 4326, mean/max populated. Grouping by the geometry column is required so each row carries its polygon for the choropleth (grouping a small reference set by geometry is safe). - ---- - -### Task 5: "Regional Context" dashboard page - -**Files:** -- Modify: `notebooks/examples/vapor-eyes/lakeflow/dashboards/vapor_eyes_lf.lvdash.json` - -**Interfaces:** -- Consumes: `emissions_by_play`, `detections_by_county`. -- Produces: a fourth page `page_regional_context` with two choropleth widgets + their datasets, following the proven render contract. - -- [ ] **Step 1: Add two datasets** to the `datasets` array — `ds_emissions_by_play` (`SELECT * FROM emissions_by_play`) and `ds_detections_by_county` (`SELECT * FROM detections_by_county`), mirroring the existing dataset entries' shape. - -- [ ] **Step 2: Add the page + two choropleth widgets.** Each choropleth widget MUST use the render contract verbatim (memory `aibi-custom-geometry-choropleth`): - - `encodings.region` = `{"regionType": "custom", "fieldName": "geo(play_geom)"}` (county widget: `"geo(county_geom)"`). - - The widget query `fields` MUST include `{"name": "geo(play_geom)", "expression": "ST_ASGEOJSON(`play_geom`)"}` (county: `geo(county_geom)` / `ST_ASGEOJSON(`county_geom`)`), plus the metric/tooltip fields (`plume_count`, `mean_emission_kg_hr`, `max_emission_kg_hr`, `play_name`/`county_name`, `active_operators`). - - `encodings.color` = `plume_count` (both, primary metric). - - Page title "Regional Context"; description noting the play/county rollups; a Carbon Mapper attribution text widget (copy the attribution string used on the other CM pages). - - Copy an existing working choropleth widget from `page_regional_screen` as the structural template and swap dataset/field names — do NOT hand-write the widget JSON from scratch. - -- [ ] **Step 3: Validate the JSON** - -Run: `cd notebooks/examples/vapor-eyes/lakeflow && python -c "import json; json.load(open('dashboards/vapor_eyes_lf.lvdash.json')); print('ok')"` -Expected: `ok`. - -- [ ] **Step 4: Commit** - -```bash -git add notebooks/examples/vapor-eyes/lakeflow/dashboards/vapor_eyes_lf.lvdash.json -git commit -m "feat(vapor-eyes): Regional Context dashboard page (play + county choropleths)" -``` - -- [ ] **Step 5: Cluster+browser verification (orchestrator-driven)** — `databricks bundle deploy --force -p oauth-fe`, publish, then chrome-devtools screenshot to confirm BOTH choropleths paint polygons over the Permian (not a blank world map). If blank, re-check the geo()/ST_ASGEOJSON field pairing before anything else. - ---- - -### Task 6: Docs + dependency/tier verification - -**Files:** -- Modify: `notebooks/examples/vapor-eyes/lakeflow/README.md` -- Modify: `docs/docs/notebooks/vapor-eyes-lakeflow.mdx` -- Capture: `resources/images/diagrams/vapor-eyes/lakeflow-dashboard-regional-context.png` - -**Interfaces:** none (docs only). - -- [ ] **Step 1: Verify pyogrio is pinned** in the light extra and does not need a new pin. Confirm the light vector reader dep (`pyogrio`) is already present in the geobrix `[light]`/`[stac,vizx]` resolution used by the pipeline env (memory `pyogrio-for-pyvx-vector`, `light-ci-lock-completeness`). Record the finding; add a pin only if genuinely missing. - -Run: `grep -rn "pyogrio" python/geobrix/pyproject.toml python/geobrix/requirements*/ 2>/dev/null` -Expected: pyogrio appears in the light dependency set. If absent, STOP and escalate (do not silently add). - -- [ ] **Step 2: Add a "Regional Context" section** to `README.md` (after the Regional Screen section) describing the two rollup choropleths and the two sources (EIA plays, TIGER counties) with the `![...](../../../resources/images/diagrams/vapor-eyes/lakeflow-dashboard-regional-context.png)` image reference. Note the data sources + licenses. - -- [ ] **Step 3: Mirror the section** in `docs/docs/notebooks/vapor-eyes-lakeflow.mdx`, keeping user-facing voice (no internal vocabulary; `grep -rn -iE "wave [0-9]+" docs/docs/` stays empty). - -- [ ] **Step 4: Capture the screenshot (orchestrator-driven)** from the published Regional Context page (same method as the other three: hide nav, hide Ask Genie, expand scroll container, fullPage). - -- [ ] **Step 5: Commit** - -```bash -git add notebooks/examples/vapor-eyes/lakeflow/README.md docs/docs/notebooks/vapor-eyes-lakeflow.mdx resources/images/diagrams/vapor-eyes/lakeflow-dashboard-regional-context.png -git commit -m "docs(vapor-eyes): document Regional Context page + sources" -``` - ---- - -## Final review + wrap - -- [ ] Whole-branch review of the context-geometry commits (spec compliance + quality). -- [ ] Confirm nothing pushed; summarize for the user and await go before push/PR (squash the redundant S2 commit `47785515` at that point, run `gbx:lint`, PR via `mjohns-databricks`). diff --git a/docs/superpowers/plans/2026-07-16-genie-map-app-mvp.md b/docs/superpowers/plans/2026-07-16-genie-map-app-mvp.md deleted file mode 100644 index 4caef4e52..000000000 --- a/docs/superpowers/plans/2026-07-16-genie-map-app-mvp.md +++ /dev/null @@ -1,1526 +0,0 @@ -# Genie Map App — Phase 0 + Phase 1 (MVP) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Adapt the `isaac_work/genie_map` kepler.gl + `@databricks/appkit` prototype into a deployable Databricks App (`apps/genie_map/`) that renders the vapor-eyes methane gold data — H3 hexagon layers (CH4 hotspots + wells density), wells/plume point layers, and a curated Genie NLP path — with implementer + slide-ware storytelling artifacts. - -**Architecture:** Copy the prototype into `apps/genie_map/`, replace its build-time `VITE_*` taxi canonical-schema with a static **layer registry** (`config/datasets/vapor-eyes.ts` + per-layer SQL templates) featuring **density-aware dynamic H3 resolution**, point every layer at Lakeflow gold `geospatial_docs.vapor_eyes_lf`, add one new gold MV (`wells_enriched_latest`) to the vapor-eyes SDP, curate a Genie Space, and deploy via a DAB bundle wired to `gbx:app:*` commands. - -**Tech Stack:** React 18 + TypeScript + kepler.gl 3.2.5; `@databricks/appkit` 0.41.6 (Node/Express); Vite 6; pnpm@10, node>=20; Databricks Asset Bundles (DAB); Lakeflow Declarative Pipeline (`pyspark.pipelines`); Databricks-native `st_*`/`h3_*` SQL. - -## Global Constraints - -- **App location:** new top-level `apps/genie_map/` on branch `apps/genie-map` (cut from `examples/vapor-eyes`, which carries the SDP + spec this depends on). -- **Data spine:** all gold tables in `geospatial_docs.vapor_eyes_lf`. Map-facing geometry is native `GEOMETRY` at **SRID 4326** — never raw WKB or H3 cell ids. Re-tag geometry that round-trips through an MV with `st_setsrid(..., 4326)` (SRID-0 is silently unrendered). -- **Exec context:** workspace `e2-demo-field-eng`, CLI profile **`oauth-fe`** (the default profile's PAT is dead). SQL warehouse = GeoBrix **`82e587bd93c6cbcf`**. -- **SDP gold conventions** (match `notebooks/examples/vapor-eyes/lakeflow/transformations/gold_analytics.py`): `@dp.materialized_view`; no driver `.collect()`/`.rdd`/`spark.conf.set` (use window/crossJoin-broadcast); current well inventory = `wells_shl` filtered `__END_AT IS NULL`; no GeoBrix SQL needed in gold (pure native `st_*`/`h3_*`). -- **`gbx:*` command pattern:** each command is a `.md` + `.sh` pair under `scripts/commands/`, `.sh` sources `common.sh`, supports `--help`/`--log`, resolves `SCRIPT_DIR`/`PROJECT_ROOT` via `$SCRIPT_DIR/../..`, exits non-zero on failure. Fix the command, never work around it. -- **User-facing voice:** no internal planning vocabulary (no "wave N", no subagent/dispatch references) in anything under `apps/genie_map/docs/` or `docs/docs/`. Behavior, not process. -- **No aliases / single canonical name** per repo policy. -- **Serving model** (`DATABRICKS_SERVING_ENDPOINT_NAME`) is baked at Vite build time into `__LLM_MODEL__` — changing it requires a rebuild, not just runtime config. Document, don't fight it. -- **Commits:** frequent, one deliverable each; end commit messages with `Co-authored-by: Isaac`. - -## File Structure - -**New app (copied then modified from `isaac_work/genie_map/kepler-demo/`):** -``` -apps/genie_map/ -├── README.md # quickstart (Task 15) -├── app.yaml # Databricks Apps runtime (Task 3, edited) -├── package.json, pnpm-lock.yaml # copied verbatim (Task 1) -├── vite.config.ts, tsconfig*.json # copied verbatim (Task 1) -├── .env.example # renamed from taxi.env.example, vapor-eyes values (Task 3) -├── shared/types.ts # + registry types (Task 4) -├── config/queries/ -│ ├── hotspot_h3.sql # Task 6 (cell-sourced dynamic H3, coarsen-only) -│ ├── wells_h3.sql # Task 8 (point-sourced dynamic H3, refine+coarsen) -│ ├── plume_points.sql # Task 7 (replaces point_data.sql) -│ └── wells_points.sql # Task 9 -├── client/src/ -│ ├── config/datasets/vapor-eyes.ts # THE layer registry (Task 5) -│ ├── config/datasets/index.ts # active-dataset selector (Task 5) -│ ├── config/h3-layer-config.ts # generalized factory (Task 4) -│ ├── config/point-layer-config.ts # converted to factory (Task 4) -│ ├── hooks/useLayerData.ts # generic registry-driven hook (Task 6) -│ ├── hooks/useViewportBounds.ts # copied verbatim -│ ├── hooks/useLayerVisibility.ts # copied verbatim -│ ├── tools/databricks/genie-tool.ts # widen geometry detection (Task 11) -│ ├── tools/databricks/types.ts # copied (parseGeoJsonFromRows reused) -│ └── App.tsx # registry-driven layer wiring (Task 10) -├── bundle/databricks.yml # DAB bundle (Task 12) -└── docs/ - ├── BUILD.md # implementer narrative (Task 14, accrues) - └── diagrams/*.py + *.png # slide-ware (Task 13) -``` - -**SDP (existing pipeline, gold layer):** -``` -notebooks/examples/vapor-eyes/lakeflow/transformations/gold_analytics.py # + 1 MV (Task 16) -notebooks/examples/vapor-eyes/lakeflow/tests/validate/ # + MV validator -``` - -**`gbx:*` commands:** -``` -scripts/commands/gbx-app-dev.{md,sh} # Task 12 -scripts/commands/gbx-app-deploy.{md,sh} # Task 12 -``` - ---- - -## Task ordering rationale - -Tasks 1–15 are the **app + deploy + docs** track. Tasks 16 + 18 are the **SDP gold MV / Genie Space** track (Task 17 was merged into 16 — see below). The SDP track is independent of the app track and can run in parallel; the app's wells layers (Tasks 8, 9) *consume* the `wells_enriched_latest` MV from Task 16, so **run 16 before verifying 8–9 against live data** — but the app code (SQL templates + hooks) can be written against the documented MV schema first (TDD-style) and verified once the MV materializes. The Genie Space (Task 18) depends on all gold tables existing. - -**Dynamic H3 (density-aware) — cross-cutting note.** Per spec §4 "Dynamic H3", H3 resolution is chosen at query time from **zoom (a max-resolution ceiling) AND density (a target of ~300 on-screen cells)**, not zoom alone. Two source modes: `ch4_hotspots` is **cell-sourced** (coarsen-only from `hotspot_latest.h3_cellid` via `h3_toparent`; never finer than native res 6); `well_density` is **point-sourced** (refine *and* coarsen from `wells_enriched_latest` points via `h3_longlatash3`). The originally-planned fixed `wells_h3_density_latest` MV is **dropped** (old Task 17 removed); `wells_enriched_latest` (Task 16) is the single wells source for both the wells H3 layer and the wells point layer. - ---- - -### Task 1: Copy prototype into `apps/genie_map/` and cut the branch - -**Files:** -- Create branch `apps/genie-map` from `examples/vapor-eyes` -- Create: `apps/genie_map/**` (copy of `isaac_work/genie_map/kepler-demo/`, minus taxi-specific + dead files) - -**Interfaces:** -- Produces: the working tree that all later tasks edit. No code interface. - -- [ ] **Step 1: Cut the branch** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -git checkout examples/vapor-eyes && git pull --ff-only -git checkout -b apps/genie-map -``` - -- [ ] **Step 2: Copy the app, excluding node_modules/dist and dead taxi files** - -```bash -mkdir -p apps/genie_map -rsync -a --exclude node_modules --exclude dist --exclude '.env' --exclude '.env.local' \ - /Users/mjohns/isaac_work/genie_map/kepler-demo/ apps/genie_map/ -# Remove dead-weight SQL (superseded duplicate + taxi templates replaced later) -rm -f apps/genie_map/config/queries/chart_operators.sql -# Rename env template -git -C apps/genie_map mv taxi.env.example .env.example 2>/dev/null || mv apps/genie_map/taxi.env.example apps/genie_map/.env.example -``` - -- [ ] **Step 3: Verify the tree copied and taxi notebooks were NOT dragged in** - -Run: `ls apps/genie_map && ls apps/genie_map/config/queries && test ! -d apps/genie_map/notebooks && echo "clean"` -Expected: app files present; `config/queries/` has `h3_aggregation.sql point_data.sql chart_groups.sql` (chart_operators removed; taxi templates removed in Task 6/7); prints `clean`. - -- [ ] **Step 4: Add an app-scoped `.gitignore` note and commit the raw copy** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -git add apps/genie_map -git commit -m "chore(genie-map): copy kepler-demo prototype into apps/genie_map - -Verbatim copy of isaac_work/genie_map/kepler-demo minus node_modules, -dist, the dead chart_operators.sql, and the taxi env template (renamed -to .env.example). Subsequent tasks retarget it onto vapor-eyes gold. - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: Establish local build baseline (green before changes) - -**Files:** -- Modify: none (verification only) - -**Interfaces:** -- Produces: confidence that the copy builds, so later failures are attributable to our changes. - -- [ ] **Step 1: Install dependencies** - -Run: `cd apps/genie_map && pnpm install --frozen-lockfile` -Expected: install completes; `node_modules/` created. (If lockfile mismatch, run `pnpm install` and note it in BUILD.md.) - -- [ ] **Step 2: Type-check + build the client and server** - -Run: `cd apps/genie_map && pnpm build` -Expected: `build:server` (tsc) + `build:client` (vite) complete; `dist/server/index.js` and `dist/client/` produced. A missing `.env`/`VITE_DATASET_TABLE` is fine at build time — it fails only at query time. - -- [ ] **Step 3: Record the baseline in BUILD.md (create it)** - -Create `apps/genie_map/docs/BUILD.md` with a "Baseline" section noting: source prototype, versions built against, that build is green with no env. (This file accrues through Task 14.) - -- [ ] **Step 4: Commit** - -```bash -git add apps/genie_map/docs/BUILD.md -git commit -m "docs(genie-map): record green build baseline of copied app - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: Retarget env template + app.yaml to vapor-eyes - -**Files:** -- Modify: `apps/genie_map/.env.example` -- Modify: `apps/genie_map/app.yaml` - -**Interfaces:** -- Produces: the env contract (`DATABRICKS_*` + one `VITE_*` for the active dataset id + mapbox token) that Task 5's registry and Task 12's bundle read. - -- [ ] **Step 1: Rewrite `.env.example` for vapor-eyes** - -Replace the taxi dataset block. New contents: - -```dotenv -# --- Databricks connection (local dev) --- -DATABRICKS_HOST=https://e2-demo-field-eng.cloud.databricks.com -DATABRICKS_TOKEN= -DATABRICKS_WAREHOUSE_ID=82e587bd93c6cbcf -DATABRICKS_GENIE_SPACE_ID= -DATABRICKS_SERVING_ENDPOINT_NAME=databricks-gpt-5-2 -VITE_MAPBOX_TOKEN= -PORT=3000 - -# --- Active dataset (selects a registry entry in client/src/config/datasets) --- -VITE_ACTIVE_DATASET=vapor-eyes -``` - -The per-layer schema is no longer env-driven — it lives in the registry (Task 5). Only the active-dataset *id* is env-selectable. - -- [ ] **Step 2: Update `app.yaml` comment + keep resource wiring** - -`app.yaml` already injects `sql-warehouse-id` and `genie-space-id` via `valueFrom` and hardcodes the serving endpoint. Add `VITE_ACTIVE_DATASET` note as a comment (it is baked at build time, not injected at runtime, so it belongs to the build, not app.yaml). Verify the file still reads: - -```yaml -command: - - node - - dist/server/index.js -env: - - name: DATABRICKS_WAREHOUSE_ID - valueFrom: sql-warehouse-id - - name: DATABRICKS_GENIE_SPACE_ID - valueFrom: genie-space-id - - name: DATABRICKS_SERVING_ENDPOINT_NAME - value: databricks-gpt-5-2 -``` - -- [ ] **Step 3: Verify** - -Run: `cat apps/genie_map/.env.example && echo '---' && cat apps/genie_map/app.yaml` -Expected: vapor-eyes values; warehouse `82e587bd93c6cbcf`; `VITE_ACTIVE_DATASET=vapor-eyes`. - -- [ ] **Step 4: Commit** - -```bash -git add apps/genie_map/.env.example apps/genie_map/app.yaml -git commit -m "feat(genie-map): retarget env + app.yaml to vapor_eyes_lf warehouse - -Co-authored-by: Isaac" -``` - ---- - -### Task 4: Registry types + generalize the layer-config factories - -**Files:** -- Modify: `apps/genie_map/shared/types.ts` (add registry interfaces) -- Modify: `apps/genie_map/client/src/config/h3-layer-config.ts` (parameterize palette/fields) -- Modify: `apps/genie_map/client/src/config/point-layer-config.ts` (convert static object → factory) -- Test: `apps/genie_map/client/src/config/__tests__/layer-config.test.ts` - -**Interfaces:** -- Consumes: existing `createH3LayerConfig(options: H3LayerConfigOptions)` (h3-layer-config.ts:35). -- Produces: - - `LayerDef`, `DatasetConfig` types (in `shared/types.ts`) — shapes below. - - `createH3LayerConfig(options)` extended with `colorField`/`palette`/`tooltipFields`. - - `createPointLayerConfig(options: PointLayerConfigOptions)` — new factory returning a kepler `addDataToMap` config, analogous to `createH3LayerConfig`. - -- [ ] **Step 1: Write the failing test for registry types + point factory** - -Create `apps/genie_map/client/src/config/__tests__/layer-config.test.ts`: - -```ts -import { describe, it, expect } from 'vitest'; -import { createH3LayerConfig } from '../h3-layer-config'; -import { createPointLayerConfig } from '../point-layer-config'; - -describe('layer config factories', () => { - it('H3 factory honors a custom color field and dataset id', () => { - const cfg = createH3LayerConfig({ - datasetId: 'ch4_hotspots', hexField: 'hex', valueField: 'ch4_max', - label: 'CH4 Hotspots', enable3d: true, - }) as any; - const layer = cfg.config.visState.layers[0]; - expect(layer.config.dataId).toBe('ch4_hotspots'); - expect(layer.visualChannels.colorField.name).toBe('ch4_max'); - expect(layer.config.columns.hex_id).toBe('hex'); - }); - - it('point factory returns a point layer bound to the given dataset id + coords', () => { - const cfg = createPointLayerConfig({ - datasetId: 'wells', label: 'Wells', - latField: 'latitude', lngField: 'longitude', - tooltipFields: ['record_id', 'operator'], - }) as any; - const layer = cfg.config.visState.layers[0]; - expect(layer.type).toBe('point'); - expect(layer.config.dataId).toBe('wells'); - expect(layer.config.columns.lat).toBe('latitude'); - expect(layer.config.columns.lng).toBe('longitude'); - }); -}); -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `cd apps/genie_map && pnpm vitest run client/src/config/__tests__/layer-config.test.ts` -Expected: FAIL — `createPointLayerConfig` is not exported. - -- [ ] **Step 3: Add registry types to `shared/types.ts`** - -Append: - -```ts -export type LayerKind = 'h3' | 'point'; -export type H3Source = 'cells' | 'points'; - -// Dynamic, density-aware H3 config (spec §4). Resolution is picked at query time -// from BOTH zoom (a max-res ceiling) AND density (target on-screen cell count). -export interface H3ResConfig { - source: H3Source; // 'cells' → coarsen-only via h3_toparent (from a stored cell col); - // 'points' → refine+coarsen via h3_longlatash3 (from lon/lat) - cellIdCol?: string; // source==='cells': the stored H3 cell column, e.g. 'h3_cellid' - nativeRes?: number; // source==='cells': stored resolution = hard finest ceiling - lonCol?: string; // source==='points' - latCol?: string; // source==='points' - minRes: number; // coarsest resolution ever rendered - maxRes: number; // finest resolution allowed (points may exceed a cell's nativeRes) - zoomResBreaks: number[]; // exactly 4 ascending zoom thresholds - resByBreak: number[]; // exactly 5 resolutions (per zoom band); each <= maxRes - aggExpr: string; // child aggregation, e.g. 'MAX(ch4_max)' or 'COUNT(*)' - targetCells?: number; // density target (default 300); coarsen when in-view count exceeds it -} - -export interface LayerDef { - id: string; // kepler dataId + layer key, e.g. 'ch4_hotspots' - kind: LayerKind; - label: string; - queryName: string; // key into config/queries (file name without .sql) - hexField?: string; // kind==='h3': the string hex column returned by SQL - h3?: H3ResConfig; // kind==='h3': dynamic-resolution config (required for h3) - valueField: string; // color/size metric column - lngField?: string; // kind==='point' - latField?: string; // kind==='point' - tooltipFields: string[]; - palette?: string; // kepler named colorRange, default 'Global Warming' - enable3d?: boolean; - zoomVisible: { min: number; max: number }; // hard visibility band: min <= z < max - fadeBand?: [number, number]; // optional opacity ramp across [start,end] for smooth swap -} - -export interface DatasetConfig { - id: string; // matches VITE_ACTIVE_DATASET - displayName: string; - genieSpaceAlias: string; // 'default' server-side; label only - defaultViewport: { longitude: number; latitude: number; zoom: number }; - layers: LayerDef[]; -} -``` - -- [ ] **Step 4: Extend `createH3LayerConfig` for palette + tooltip fields** - -In `h3-layer-config.ts`, widen `H3LayerConfigOptions` with `palette?: string` and `tooltipFields?: string[]`, default `palette='Global Warming'`, and use them in the returned config's `visConfig.colorRange` name lookup and `interactionConfig.tooltip.fieldsToShow`. Keep the existing `colorField`/`sizeField` = `valueField` behavior. - -- [ ] **Step 5: Convert `point-layer-config.ts` to a factory** - -Replace the static `POINT_LAYER_CONFIG` object with: - -```ts -export interface PointLayerConfigOptions { - datasetId: string; - label: string; - latField: string; - lngField: string; - tooltipFields: string[]; - radius?: number; - color?: [number, number, number]; -} - -export function createPointLayerConfig(options: PointLayerConfigOptions) { - const { datasetId, label, latField, lngField, tooltipFields, - radius = 20, color = [255, 195, 0] } = options; - return { - version: 'v1', - config: { visState: { filters: [], layers: [{ - id: `point-layer-${datasetId}`, type: 'point', - config: { dataId: datasetId, label, color, - columns: { lat: latField, lng: lngField, altitude: null }, - isVisible: true, - visConfig: { radius, fixedRadius: false, opacity: 0.8, outline: false, - thickness: 2, strokeColor: null, colorRange: GLOBAL_WARMING, - strokeColorRange: GLOBAL_WARMING, radiusRange: [0, 50], filled: true }, - textLabel: [] }, - visualChannels: { colorField: null, colorScale: 'quantile', - strokeColorField: null, strokeColorScale: 'quantile', - sizeField: null, sizeScale: 'linear' } }], - interactionConfig: { tooltip: { fieldsToShow: { [datasetId]: tooltipFields }, - enabled: true }, brush: { size: 0.5, enabled: false }, - geocoder: { enabled: false } }, - layerBlending: 'normal', splitMaps: [] } }, - }; -} -``` - -(Define `GLOBAL_WARMING` as the existing 6-color array already present in the file.) - -- [ ] **Step 6: Run tests to verify they pass** - -Run: `cd apps/genie_map && pnpm vitest run client/src/config/__tests__/layer-config.test.ts` -Expected: PASS (2 tests). - -- [ ] **Step 7: Commit** - -```bash -git add apps/genie_map/shared/types.ts apps/genie_map/client/src/config -git commit -m "feat(genie-map): registry types + parameterized layer factories - -Adds LayerDef/DatasetConfig types and a createPointLayerConfig factory -(mirrors createH3LayerConfig) so layer styling/fields are per-dataset. - -Co-authored-by: Isaac" -``` - ---- - -### Task 5: The vapor-eyes layer registry + active-dataset selector - -**Files:** -- Create: `apps/genie_map/client/src/config/datasets/vapor-eyes.ts` -- Create: `apps/genie_map/client/src/config/datasets/index.ts` -- Test: `apps/genie_map/client/src/config/datasets/__tests__/registry.test.ts` - -**Interfaces:** -- Consumes: `DatasetConfig`, `LayerDef` (Task 4). -- Produces: - - `vaporEyes: DatasetConfig` (the one shipped config). - - `getActiveDataset(): DatasetConfig` — reads `import.meta.env.VITE_ACTIVE_DATASET`, defaults `'vapor-eyes'`, throws on unknown id. - - `DATASETS: Record`. - -- [ ] **Step 1: Write the failing test** - -Create `registry.test.ts`: - -```ts -import { describe, it, expect } from 'vitest'; -import { vaporEyes } from '../vapor-eyes'; -import { getActiveDataset, DATASETS } from '../index'; - -describe('vapor-eyes registry', () => { - it('declares the four MVP layers', () => { - const ids = vaporEyes.layers.map((l) => l.id); - expect(ids).toEqual(['ch4_hotspots', 'well_density', 'wells', 'plumes']); - }); - it('ch4_hotspots is a cell-sourced H3 layer capped at native res 6', () => { - const l = vaporEyes.layers.find((x) => x.id === 'ch4_hotspots')!; - expect(l.kind).toBe('h3'); - expect(l.queryName).toBe('hotspot_h3'); - expect(l.valueField).toBe('ch4_max'); - expect(l.h3!.source).toBe('cells'); - expect(l.h3!.nativeRes).toBe(6); - expect(l.h3!.maxRes).toBe(6); // never finer than S5P native footprint - expect(l.h3!.zoomResBreaks).toHaveLength(4); - expect(l.h3!.resByBreak).toHaveLength(5); - }); - it('well_density is a point-sourced H3 layer that can refine past res 6', () => { - const l = vaporEyes.layers.find((x) => x.id === 'well_density')!; - expect(l.h3!.source).toBe('points'); - expect(l.h3!.lonCol).toBe('longitude'); - expect(l.h3!.maxRes).toBeGreaterThan(6); - expect(l.queryName).toBe('wells_h3'); - }); - it('wells H3 and wells points share the wells_enriched source + overlap-swap', () => { - const density = vaporEyes.layers.find((x) => x.id === 'well_density')!; - const wells = vaporEyes.layers.find((x) => x.id === 'wells')!; - // ~1-level overlap band: density fades out where wells fades in. - expect(density.zoomVisible.max).toBeGreaterThan(wells.zoomVisible.min); - expect(density.fadeBand).toBeDefined(); - }); - it('ch4 hexes and plumes coexist (plumes appear on zoom-in, hexes stay)', () => { - const ch4 = vaporEyes.layers.find((x) => x.id === 'ch4_hotspots')!; - const plumes = vaporEyes.layers.find((x) => x.id === 'plumes')!; - expect(ch4.zoomVisible.max).toBe(24); // ch4 stays visible at all zooms - expect(plumes.zoomVisible.min).toBeGreaterThan(0); // plumes only on zoom-in - }); - it('getActiveDataset defaults to vapor-eyes', () => { - expect(getActiveDataset().id).toBe('vapor-eyes'); - expect(DATASETS['vapor-eyes']).toBe(vaporEyes); - }); -}); -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `cd apps/genie_map && pnpm vitest run client/src/config/datasets/__tests__/registry.test.ts` -Expected: FAIL — modules not found. - -- [ ] **Step 3: Write `vapor-eyes.ts`** - -```ts -import type { DatasetConfig } from '@shared/types'; - -const T = (name: string) => `geospatial_docs.vapor_eyes_lf.${name}`; - -export const vaporEyes: DatasetConfig = { - id: 'vapor-eyes', - displayName: 'Vapor-Eyes — Permian Basin Methane', - genieSpaceAlias: 'default', - // Delaware Basin (full AOI center, from SDP _config bbox -104.5,30.8,-101.0,33.0) - defaultViewport: { longitude: -102.75, latitude: 31.9, zoom: 8 }, - layers: [ - // CH4 hexes: cell-sourced, coarsen-ONLY (never finer than S5P native res 6). - // Always-on wide-area context; density heuristic keeps it readable at low zoom. - { id: 'ch4_hotspots', kind: 'h3', label: 'CH₄ Hotspots (latest)', - queryName: 'hotspot_h3', hexField: 'hex', valueField: 'ch4_max', - tooltipFields: ['hex', 'ch4_max', 'ch4_mean', 'n_obs'], - palette: 'Global Warming', enable3d: true, - h3: { source: 'cells', cellIdCol: 'h3_cellid', nativeRes: 6, - minRes: 2, maxRes: 6, zoomResBreaks: [5, 7, 9, 11], - resByBreak: [3, 4, 5, 6, 6], aggExpr: 'MAX(ch4_max)', targetCells: 300 }, - zoomVisible: { min: 0, max: 24 } }, - // Well density: point-sourced, refine (finer on zoom-in) AND coarsen (only if dense). - // Owns low/mid zoom; fades out over [11,12] as the wells point layer fades in. - { id: 'well_density', kind: 'h3', label: 'Well Density (H3)', - queryName: 'wells_h3', hexField: 'hex', valueField: 'well_count', - tooltipFields: ['hex', 'well_count', 'operator_count'], - palette: 'Uber Viz Sequential', enable3d: true, - h3: { source: 'points', lonCol: 'longitude', latCol: 'latitude', - minRes: 3, maxRes: 9, zoomResBreaks: [5, 7, 9, 11], - resByBreak: [4, 5, 6, 7, 9], aggExpr: 'COUNT(*)', targetCells: 300 }, - zoomVisible: { min: 0, max: 12 }, fadeBand: [11, 12] }, - // Wells points: fade in over [11,12] — ~1-level overlap with well_density. - { id: 'wells', kind: 'point', label: 'Wells', - queryName: 'wells_points', valueField: 'well_count', - lngField: 'longitude', latField: 'latitude', - tooltipFields: ['record_id', 'operator', 'field', 'county', 'play_name'], - zoomVisible: { min: 11, max: 24 }, fadeBand: [11, 12] }, - // EMIT plumes: coexist with the CH4 hex screen; appear once zoomed in to resolve sources. - { id: 'plumes', kind: 'point', label: 'EMIT Plumes', - queryName: 'plume_points', valueField: 'max_conc_ppmm', - lngField: 'longitude', latField: 'latitude', - tooltipFields: ['record_id', 'max_conc_ppmm', 'lead_operator', 'lead_county'], - zoomVisible: { min: 9, max: 24 } }, - ], -}; - -export const VAPOR_EYES_TABLES = { - hotspot: T('hotspot_latest'), - wellsEnriched: T('wells_enriched_latest'), // feeds BOTH well_density (H3) and wells (points) - plumes: T('plume_leaderboard_latest'), -}; -``` - -- [ ] **Step 4: Write `index.ts`** - -```ts -import type { DatasetConfig } from '@shared/types'; -import { vaporEyes } from './vapor-eyes'; - -export const DATASETS: Record = { 'vapor-eyes': vaporEyes }; - -export function getActiveDataset(): DatasetConfig { - const id = (import.meta.env.VITE_ACTIVE_DATASET as string) || 'vapor-eyes'; - const ds = DATASETS[id]; - if (!ds) throw new Error(`Unknown VITE_ACTIVE_DATASET '${id}'. Known: ${Object.keys(DATASETS).join(', ')}`); - return ds; -} -``` - -- [ ] **Step 5: Run tests to verify pass** - -Run: `cd apps/genie_map && pnpm vitest run client/src/config/datasets/__tests__/registry.test.ts` -Expected: PASS (6 tests). - -- [ ] **Step 6: Commit** - -```bash -git add apps/genie_map/client/src/config/datasets -git commit -m "feat(genie-map): vapor-eyes layer registry + active-dataset selector - -Four MVP layers with density-aware dynamic H3: cell-sourced ch4_hotspots -(coarsen-only, capped at native res 6) + point-sourced well_density -(refine+coarsen). Visibility choreography: wells H3<->points overlap-swap, -CH4 hexes + plumes coexist. getActiveDataset() is the seam for helios. - -Co-authored-by: Isaac" -``` - ---- - -### Task 6: `hotspot_h3.sql` template + generic `useLayerData` hook - -**Files:** -- Create: `apps/genie_map/config/queries/hotspot_h3.sql` -- Delete: `apps/genie_map/config/queries/h3_aggregation.sql` -- Create: `apps/genie_map/client/src/hooks/useLayerData.ts` -- Test: `apps/genie_map/client/src/hooks/__tests__/useLayerData.test.ts` - -**Interfaces:** -- Consumes: `useKeplerDataset` (useKeplerDataset.ts:74), `sql` from `@databricks/appkit-ui/js`, `LayerDef`, `createH3LayerConfig`/`createPointLayerConfig`, `ViewportBounds`. -- Produces: `useLayerData(layer: LayerDef, bounds: ViewportBounds | null): { data: unknown[]; isLoading: boolean; error: Error | null }` — resolves the SQL params + kepler layerConfig from the `LayerDef.kind` and delegates to `useKeplerDataset`. - -- [ ] **Step 1: Write `hotspot_h3.sql` (cell-sourced, density-aware coarsen-only)** - -Reads `hotspot_latest` (native res-6 cells). A CTE picks the zoom-ceiling resolution -(clamped to `native_res` — never finer), then coarsens further by -`floor(log₇(in_view_count / target_cells))` `h3_toparent` steps when the data is dense. -Sparse data → 0 coarsening steps → stays at res 6. Children are re-aggregated: - -```sql --- Params: x_min,x_max,y_min,y_max DOUBLE ; zoom_level INT ; --- zoom_break_1..4 INT ; res_1..5 INT ; min_res INT ; native_res INT ; --- target_cells INT ; table_name STRING via IDENTIFIER() -WITH in_view AS ( - SELECT h3_cellid, ch4_max, ch4_mean, n_obs - FROM IDENTIFIER(:table_name) - WHERE center_lon BETWEEN :x_min AND :x_max - AND center_lat BETWEEN :y_min AND :y_max - AND ch4_max IS NOT NULL -), -zoom_ceiling AS ( - SELECT CASE - WHEN :zoom_level <= :zoom_break_1 THEN :res_1 - WHEN :zoom_level <= :zoom_break_2 THEN :res_2 - WHEN :zoom_level <= :zoom_break_3 THEN :res_3 - WHEN :zoom_level <= :zoom_break_4 THEN :res_4 - ELSE :res_5 END AS zc -), -counted AS (SELECT COUNT(*) AS n FROM in_view), -target_res AS ( - -- ceiling capped at native_res; density subtracts coarsening levels (each ≈ ÷7); - -- floored at min_res. Sparse (n <= target) → levels = 0 → stays at ceiling. - SELECT GREATEST(:min_res, - LEAST(zc.zc, :native_res) - - GREATEST(0, CAST(FLOOR(LOG(7.0, GREATEST(c.n, 1) / CAST(:target_cells AS DOUBLE))) AS INT)) - ) AS res - FROM zoom_ceiling zc CROSS JOIN counted c -) -SELECT - h3_h3tostring(h3_toparent(v.h3_cellid, t.res)) AS hex, - CAST(MAX(v.ch4_max) AS DOUBLE) AS ch4_max, - CAST(AVG(v.ch4_mean) AS DOUBLE) AS ch4_mean, - CAST(SUM(v.n_obs) AS DOUBLE) AS n_obs -FROM in_view v CROSS JOIN target_res t -GROUP BY h3_toparent(v.h3_cellid, t.res) -``` - -(kepler's `hexagonId` layer draws the hexagon from the H3 string id itself, so returning a -coarser *parent* id renders a correctly-sized hex — no `hex_geom` needed on this path.) - -- [ ] **Step 2: Delete the taxi template** - -```bash -git -C /Users/mjohns/IdeaProjects/geobrix rm apps/genie_map/config/queries/h3_aggregation.sql -``` - -- [ ] **Step 3: Write the failing test for `useLayerData` param resolution** - -The pure param-building logic is unit-testable. Create `useLayerData.ts` exporting -`buildLayerParams(layer, bounds, tableName)` and test that H3 layers get the dynamic-res -params and point layers get only bbox+table: - -```ts -import { describe, it, expect } from 'vitest'; -import { buildLayerParams } from '../useLayerData'; -import type { LayerDef } from '@shared/types'; - -const bounds = { x_min: -103, x_max: -102, y_min: 31, y_max: 32, zoom_level: 8 }; -const h3Layer: LayerDef = { id: 'ch4_hotspots', kind: 'h3', label: 'x', - queryName: 'hotspot_h3', hexField: 'hex', valueField: 'ch4_max', tooltipFields: [], - h3: { source: 'cells', cellIdCol: 'h3_cellid', nativeRes: 6, minRes: 2, maxRes: 6, - zoomResBreaks: [5, 7, 9, 11], resByBreak: [3, 4, 5, 6, 6], - aggExpr: 'MAX(ch4_max)', targetCells: 300 }, - zoomVisible: { min: 0, max: 24 } }; -const pointLayer: LayerDef = { id: 'plumes', kind: 'point', label: 'x', - queryName: 'plume_points', valueField: 'max_conc_ppmm', lngField: 'longitude', - latField: 'latitude', tooltipFields: ['record_id'], zoomVisible: { min: 9, max: 24 } }; - -describe('buildLayerParams', () => { - it('returns null when bounds are null', () => { - expect(buildLayerParams(h3Layer, null, 't')).toBeNull(); - }); - it('emits the dynamic-H3 params for an H3 layer', () => { - const p = buildLayerParams(h3Layer, bounds, 'db.sch.hotspot_latest') as any; - expect(p.x_min).toBeDefined(); - expect(p.table_name).toBeDefined(); - expect(p.zoom_level).toBeDefined(); - expect(p.zoom_break_1).toBeDefined(); - expect(p.res_1).toBeDefined(); - expect(p.res_5).toBeDefined(); - expect(p.native_res).toBeDefined(); // cells source carries native_res - expect(p.target_cells).toBeDefined(); - }); - it('emits only bbox+table for a point layer', () => { - const p = buildLayerParams(pointLayer, bounds, 'db.sch.plume_leaderboard_latest') as any; - expect(p.x_min).toBeDefined(); - expect(p.table_name).toBeDefined(); - expect(p.zoom_break_1).toBeUndefined(); - }); -}); -``` - -- [ ] **Step 4: Run to verify it fails** - -Run: `cd apps/genie_map && pnpm vitest run client/src/hooks/__tests__/useLayerData.test.ts` -Expected: FAIL — `buildLayerParams` not exported. - -- [ ] **Step 5: Implement `useLayerData.ts`** - -```ts -import { useMemo } from 'react'; -import { sql } from '@databricks/appkit-ui/js'; -import { useKeplerDataset } from './useKeplerDataset'; -import { createH3LayerConfig } from '../config/h3-layer-config'; -import { createPointLayerConfig } from '../config/point-layer-config'; -import type { LayerDef, ViewportBounds } from '@shared/types'; - -export function buildLayerParams( - layer: LayerDef, bounds: ViewportBounds | null, tableName: string, -): Record | null { - if (!bounds || !tableName) return null; - const base: Record = { - x_min: sql.double(bounds.x_min), x_max: sql.double(bounds.x_max), - y_min: sql.double(bounds.y_min), y_max: sql.double(bounds.y_max), - table_name: sql.string(tableName), - }; - if (layer.kind !== 'h3' || !layer.h3) return base; - - const h = layer.h3; - const p: Record = { - ...base, - zoom_level: sql.number(bounds.zoom_level), - zoom_break_1: sql.number(h.zoomResBreaks[0]), zoom_break_2: sql.number(h.zoomResBreaks[1]), - zoom_break_3: sql.number(h.zoomResBreaks[2]), zoom_break_4: sql.number(h.zoomResBreaks[3]), - res_1: sql.number(h.resByBreak[0]), res_2: sql.number(h.resByBreak[1]), - res_3: sql.number(h.resByBreak[2]), res_4: sql.number(h.resByBreak[3]), - res_5: sql.number(h.resByBreak[4]), - min_res: sql.number(h.minRes), - target_cells: sql.number(h.targetCells ?? 300), - }; - if (h.source === 'cells') p.native_res = sql.number(h.nativeRes ?? h.maxRes); - else p.max_res = sql.number(h.maxRes); // points source: finer allowed than any stored cell - return p; -} - -export function useLayerData(layer: LayerDef, bounds: ViewportBounds | null, tableName: string) { - const params = useMemo(() => buildLayerParams(layer, bounds, tableName), [layer, bounds, tableName]); - - const layerConfig = useMemo(() => ( - layer.kind === 'h3' - ? createH3LayerConfig({ datasetId: layer.id, hexField: layer.hexField ?? 'hex', - valueField: layer.valueField, label: layer.label, enable3d: layer.enable3d ?? true, - palette: layer.palette, tooltipFields: layer.tooltipFields }) - : createPointLayerConfig({ datasetId: layer.id, label: layer.label, - latField: layer.latField!, lngField: layer.lngField!, tooltipFields: layer.tooltipFields }) - ), [layer]); - - const fields = useMemo(() => ( - layer.kind === 'h3' - ? [{ name: layer.hexField ?? 'hex', type: 'string' }, { name: layer.valueField, type: 'real' }] - : [{ name: 'longitude', type: 'real' }, { name: 'latitude', type: 'real' }, - ...layer.tooltipFields.map((f) => ({ name: f, type: 'string' }))] - ), [layer]); - - return useKeplerDataset>({ - queryName: layer.queryName, params, - transformRows: (raw) => raw as Record[], - toKeplerRow: (row) => fields.map((f) => (row as any)[f.name]), - fields, datasetId: layer.id, datasetLabel: layer.label, layerConfig, - }); -} -``` - -- [ ] **Step 6: Run tests to verify pass** - -Run: `cd apps/genie_map && pnpm vitest run client/src/hooks/__tests__/useLayerData.test.ts` -Expected: PASS (3 tests). - -- [ ] **Step 7: Commit** - -```bash -git add apps/genie_map/config/queries apps/genie_map/client/src/hooks/useLayerData.ts apps/genie_map/client/src/hooks/__tests__ -git commit -m "feat(genie-map): density-aware dynamic hotspot_h3 SQL + useLayerData hook - -Cell-sourced coarsen-only H3: zoom sets a res ceiling (capped at native -res 6), density coarsens further via h3_toparent only when the in-view -cell count exceeds ~300. buildLayerParams emits the dynamic-res params for -H3 layers, bbox-only for points. - -Co-authored-by: Isaac" -``` - ---- - -### Task 7: `plume_points.sql` template - -**Files:** -- Create: `apps/genie_map/config/queries/plume_points.sql` -- Delete: `apps/genie_map/config/queries/point_data.sql` - -**Interfaces:** -- Consumes: `useLayerData` (Task 6) via the `plumes` LayerDef. -- Produces: SQL returning `longitude, latitude, record_id, max_conc_ppmm, lead_operator, lead_county`. - -- [ ] **Step 1: Write `plume_points.sql`** - -```sql --- Params: x_min,x_max,y_min,y_max DOUBLE ; table_name STRING via IDENTIFIER() -SELECT - lon_max AS longitude, - lat_max AS latitude, - CAST(plume_id AS STRING) AS record_id, - CAST(max_conc_ppmm AS DOUBLE) AS max_conc_ppmm, - lead_operator, lead_county -FROM IDENTIFIER(:table_name) -WHERE lon_max BETWEEN :x_min AND :x_max - AND lat_max BETWEEN :y_min AND :y_max -LIMIT 10000 -``` - -(Uses `plume_leaderboard_latest`'s `lon_max`/`lat_max` — see `gold_analytics.py:82-85`.) - -- [ ] **Step 2: Delete the taxi template** - -```bash -git -C /Users/mjohns/IdeaProjects/geobrix rm apps/genie_map/config/queries/point_data.sql -``` - -- [ ] **Step 3: Verify SQL parses (syntax-only lint)** - -Run: `grep -c 'IDENTIFIER(:table_name)' apps/genie_map/config/queries/plume_points.sql` -Expected: `1`. (Live execution verified in Task 19.) - -- [ ] **Step 4: Commit** - -```bash -git add apps/genie_map/config/queries -git commit -m "feat(genie-map): plume_points SQL over plume_leaderboard_latest - -Co-authored-by: Isaac" -``` - ---- - -### Task 8: `wells_h3.sql` template (point-sourced, refine + coarsen) - -**Files:** -- Create: `apps/genie_map/config/queries/wells_h3.sql` - -**Interfaces:** -- Consumes: `wells_enriched_latest` MV (Task 16) — the **points** source. `useLayerData` via `well_density` LayerDef (`h3.source==='points'`). -- Produces: SQL returning `hex, well_count, operator_count`. - -- [ ] **Step 1: Write `wells_h3.sql`** - -Unlike the cell-sourced hotspot query, this aggregates the **well points on the fly** via -`h3_longlatash3(lon, lat, res)` — so it can go *finer as you zoom in* (up to `max_res`), -not just coarser. Same density heuristic: zoom sets the ceiling, density lowers it toward -`min_res` only when the in-view well count is dense. Two-pass: bin at the zoom ceiling to -estimate density, then re-bin at the density-adjusted resolution. - -```sql --- Params: x_min,x_max,y_min,y_max DOUBLE ; zoom_level INT ; zoom_break_1..4 INT ; --- res_1..5 INT ; min_res INT ; max_res INT ; target_cells INT ; --- table_name STRING via IDENTIFIER() -WITH in_view AS ( - SELECT longitude, latitude, operator - FROM IDENTIFIER(:table_name) - WHERE longitude BETWEEN :x_min AND :x_max - AND latitude BETWEEN :y_min AND :y_max - AND longitude IS NOT NULL AND latitude IS NOT NULL -), -zoom_ceiling AS ( - SELECT LEAST(:max_res, CASE - WHEN :zoom_level <= :zoom_break_1 THEN :res_1 - WHEN :zoom_level <= :zoom_break_2 THEN :res_2 - WHEN :zoom_level <= :zoom_break_3 THEN :res_3 - WHEN :zoom_level <= :zoom_break_4 THEN :res_4 - ELSE :res_5 END) AS zc -), --- Estimate density at the ceiling resolution (distinct cells occupied in view). -ceiling_cells AS ( - SELECT COUNT(DISTINCT h3_longlatash3(longitude, latitude, (SELECT zc FROM zoom_ceiling))) AS n - FROM in_view -), -target_res AS ( - SELECT GREATEST(:min_res, - (SELECT zc FROM zoom_ceiling) - - GREATEST(0, CAST(FLOOR(LOG(7.0, GREATEST(c.n, 1) / CAST(:target_cells AS DOUBLE))) AS INT)) - ) AS res - FROM ceiling_cells c -) -SELECT - h3_h3tostring(h3_longlatash3(v.longitude, v.latitude, t.res)) AS hex, - CAST(COUNT(*) AS DOUBLE) AS well_count, - CAST(COUNT(DISTINCT v.operator) AS DOUBLE) AS operator_count -FROM in_view v CROSS JOIN target_res t -GROUP BY h3_longlatash3(v.longitude, v.latitude, t.res) -``` - -- [ ] **Step 2: Verify** - -Run: `grep -c 'h3_longlatash3' apps/genie_map/config/queries/wells_h3.sql` -Expected: `3`. - -- [ ] **Step 3: Commit** - -```bash -git add apps/genie_map/config/queries/wells_h3.sql -git commit -m "feat(genie-map): point-sourced dynamic wells_h3 SQL over wells_enriched_latest - -Aggregates well points via h3_longlatash3 at a zoom+density-driven -resolution — refines finer on zoom-in (up to max_res 9), coarsens only -when wells are genuinely dense. Single wells source, no fixed density MV. - -Co-authored-by: Isaac" -``` - ---- - -### Task 9: `wells_points.sql` template - -**Files:** -- Create: `apps/genie_map/config/queries/wells_points.sql` - -**Interfaces:** -- Consumes: `wells_enriched_latest` MV (Task 16). `useLayerData` via `wells` LayerDef. -- Produces: SQL returning `longitude, latitude, record_id, operator, field, county, play_name`. - -- [ ] **Step 1: Write `wells_points.sql`** - -```sql --- Params: x_min,x_max,y_min,y_max DOUBLE ; table_name STRING via IDENTIFIER() -SELECT - longitude, - latitude, - CAST(api AS STRING) AS record_id, - operator, field, county_name AS county, play_name -FROM IDENTIFIER(:table_name) -WHERE longitude BETWEEN :x_min AND :x_max - AND latitude BETWEEN :y_min AND :y_max -LIMIT 10000 -``` - -- [ ] **Step 2: Verify** - -Run: `grep -c 'longitude' apps/genie_map/config/queries/wells_points.sql` -Expected: `2`. - -- [ ] **Step 3: Commit** - -```bash -git add apps/genie_map/config/queries/wells_points.sql -git commit -m "feat(genie-map): wells_points SQL over wells_enriched_latest MV - -Co-authored-by: Isaac" -``` - ---- - -### Task 10: Rewire `App.tsx` to drive layers from the registry - -**Files:** -- Modify: `apps/genie_map/client/src/App.tsx` -- Delete: `apps/genie_map/client/src/hooks/useH3AggregationData.ts` -- Delete: `apps/genie_map/client/src/hooks/usePointData.ts` - -**Interfaces:** -- Consumes: `getActiveDataset` (Task 5), `useLayerData` (Task 6), `VAPOR_EYES_TABLES` (Task 5), `useViewportBounds`, `useLayerVisibility` (`LayerRule`). -- Produces: the rendered app; no exported code interface. - -- [ ] **Step 1: Delete the two taxi-specific data hooks** - -```bash -cd /Users/mjohns/IdeaProjects/geobrix -git rm apps/genie_map/client/src/hooks/useH3AggregationData.ts apps/genie_map/client/src/hooks/usePointData.ts -``` - -- [ ] **Step 2: Rewrite the layer wiring in `App.tsx`** - -Replace the taxi `LAYER_RULES` block (App.tsx:115-118) and the two hook calls (App.tsx:135-146) with registry-driven wiring. Map each `LayerDef` to a table (via `VAPOR_EYES_TABLES` keyed by `layer.id`), call `useLayerData` per layer, build `LAYER_RULES` from `zoomVisible`: - -```tsx -import { getActiveDataset, VAPOR_EYES_TABLES } from './config/datasets'; -import { useLayerData } from './hooks/useLayerData'; -import type { LayerRule } from './hooks/useLayerVisibility'; - -const DATASET = getActiveDataset(); -const TABLE_BY_LAYER: Record = { - ch4_hotspots: VAPOR_EYES_TABLES.hotspot, - well_density: VAPOR_EYES_TABLES.wellsEnriched, // well_density H3 aggregates well points - wells: VAPOR_EYES_TABLES.wellsEnriched, // ...same source as the wells point layer - plumes: VAPOR_EYES_TABLES.plumes, -}; -const LAYER_RULES: LayerRule[] = DATASET.layers.map((l) => ({ - layerId: l.kind === 'h3' ? `h3-layer-${l.id}` : `point-layer-${l.id}`, - activeWhen: (z: number) => z >= l.zoomVisible.min && z < l.zoomVisible.max, -})); -``` - -Inside `App()`, replace the two hook calls with a loop that is React-hooks-safe (fixed-length registry → stable call order): - -```tsx -const { bounds, onViewStateChange } = useViewportBounds(); -// Registry length is compile-time fixed, so per-layer hook calls keep stable order. -DATASET.layers.forEach((layer) => { - useLayerData(layer, bounds, TABLE_BY_LAYER[layer.id]); -}); -useLayerVisibility(bounds?.zoom_level ?? null, LAYER_RULES); -``` - -(If a linter flags hooks-in-loop, replace with explicit per-layer calls — the registry is fixed at 4; unroll them. Add a comment either way.) Set the initial kepler viewport from `DATASET.defaultViewport`. Remove the now-dead `useFilterState`/AnalyticsDashboard props that referenced taxi `h3Data`/`pointData` if they break the build; keep panels that still compile. - -- [ ] **Step 3: Type-check + build** - -Run: `cd apps/genie_map && pnpm build` -Expected: build green. Fix any dangling taxi imports (dataset-config constants, `H3_LAYER_ID`, `POINT_LAYER_ID`) revealed by tsc — replace references with registry-derived ids. - -- [ ] **Step 4: Run the full unit test suite** - -Run: `cd apps/genie_map && pnpm vitest run` -Expected: PASS (Tasks 4/5/6 tests). - -- [ ] **Step 5: Commit** - -```bash -git add apps/genie_map/client/src/App.tsx apps/genie_map/client/src/hooks -git commit -m "feat(genie-map): drive kepler layers from the vapor-eyes registry - -Replaces the two taxi-specific data hooks + hardcoded LAYER_RULES with a -registry loop (useLayerData per LayerDef, LAYER_RULES from zoomVisible). - -Co-authored-by: Isaac" -``` - ---- - -### Task 11: Widen Genie geometry-column detection - -**Files:** -- Modify: `apps/genie_map/client/src/tools/databricks/genie-tool.ts` (findGeometryColumn, line 113-118) -- Test: `apps/genie_map/client/src/tools/databricks/__tests__/genie-geom.test.ts` - -**Interfaces:** -- Consumes: nothing new. -- Produces: `findGeometryColumn(columns)` recognizing curated vapor-eyes geometry column names. - -- [ ] **Step 1: Write the failing test** - -```ts -import { describe, it, expect } from 'vitest'; -import { findGeometryColumn } from '../genie-tool'; - -describe('findGeometryColumn', () => { - it('matches curated ST_ASGEOJSON alias names', () => { - expect(findGeometryColumn(['operator', 'geojson'])).toBe('geojson'); - expect(findGeometryColumn(['hex_geojson', 'x'])).toBe('hex_geojson'); - expect(findGeometryColumn(['geometry'])).toBe('geometry'); - expect(findGeometryColumn(['geom_geojson'])).toBe('geom_geojson'); - expect(findGeometryColumn(['operator', 'ch4_max'])).toBeUndefined(); - }); -}); -``` - -(Requires exporting `findGeometryColumn` — add `export` if not already.) - -- [ ] **Step 2: Run to verify it fails** - -Run: `cd apps/genie_map && pnpm vitest run client/src/tools/databricks/__tests__/genie-geom.test.ts` -Expected: FAIL — not exported / `geom_geojson` handled but assertion on export missing. - -- [ ] **Step 3: Widen the matcher** - -```ts -export function findGeometryColumn(columns: string[]): string | undefined { - return columns.find((col) => { - const lower = col.toLowerCase(); - return lower.includes('geojson') || lower === 'geometry' || lower === 'geom'; - }); -} -``` - -(Curated Genie SQL always aliases geometry to a `*_geojson` name via `ST_ASGEOJSON`, so the substring match is the primary path; `geometry`/`geom` are fallbacks.) - -- [ ] **Step 4: Run tests to verify pass** - -Run: `cd apps/genie_map && pnpm vitest run client/src/tools/databricks/__tests__/genie-geom.test.ts` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add apps/genie_map/client/src/tools/databricks/genie-tool.ts apps/genie_map/client/src/tools/databricks/__tests__ -git commit -m "feat(genie-map): widen Genie geometry-column detection for *_geojson aliases - -Co-authored-by: Isaac" -``` - ---- - -### Task 12: DAB bundle + `gbx:app:dev` / `gbx:app:deploy` commands - -**Files:** -- Create: `apps/genie_map/bundle/databricks.yml` -- Create: `scripts/commands/gbx-app-dev.md`, `scripts/commands/gbx-app-dev.sh` -- Create: `scripts/commands/gbx-app-deploy.md`, `scripts/commands/gbx-app-deploy.sh` - -**Interfaces:** -- Consumes: `common.sh` helpers (`print_banner`, `resolve_log_path`). -- Produces: two runnable commands; a deployable bundle. - -> **Extraction note** (see `2026-07-16-geobrix-deploy-helpers-vision.md`): keep the bundle's -> resource set (warehouse, genie-space, app name) **declarative** — a simple named list, not -> imperative glue — so a future `scaffold_map_app(resources)` helper can be lifted from this -> hand-built bundle mechanically. Don't build the helper now; just don't entangle the data. - -- [ ] **Step 1: Write the DAB bundle** - -`apps/genie_map/bundle/databricks.yml`: - -```yaml -bundle: - name: genie-map - -targets: - dev: - mode: development - default: true - workspace: - host: https://e2-demo-field-eng.cloud.databricks.com - -resources: - apps: - genie_map: - name: genie-map - description: "Genie Map — vapor-eyes methane on GeoBrix" - source_code_path: ../ - resources: - - name: sql-warehouse-id - sql_warehouse: - id: 82e587bd93c6cbcf - permission: CAN_USE - - name: genie-space-id - genie_space: - id: ${var.genie_space_id} - permission: CAN_RUN - -variables: - genie_space_id: - description: "Curated vapor-eyes Genie Space id (from Task 18)" - default: "" -``` - -(If the DAB `genie_space` resource type is unavailable in the installed CLI, fall back to injecting `genie-space-id` as an app secret/env and note it in BUILD.md — verified live in Task 20.) - -- [ ] **Step 2: Write `gbx-app-dev.sh`** - -```bash -#!/bin/bash -# gbx:app:dev - Run Genie Map locally (pnpm dev) -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -source "$SCRIPT_DIR/common.sh" -APP_DIR="$PROJECT_ROOT/apps/genie_map" -show_help() { cat <] [--log ] [--help] -OPTIONS: --profile (default oauth-fe), --log, --help -EOF -exit 0; } -while [ $# -gt 0 ]; do case "$1" in - --help|-h) show_help;; - --profile) PROFILE="$2"; shift 2;; - --log) LOG_ARG="$2"; shift 2;; - *) echo "Unknown option: $1"; exit 1;; -esac; done -cd "$APP_DIR" || exit 1 -[ -d node_modules ] || pnpm install -pnpm build || exit 1 -cd "$APP_DIR/bundle" || exit 1 -databricks bundle deploy --profile "$PROFILE" || exit 1 -databricks bundle run genie_map --profile "$PROFILE" -``` - -- [ ] **Step 4: Write the two `.md` registrations** - -Each `.md`: short title, 1-2 sentence description, `USAGE`, options, one example — matching the existing `gbx-docker-attach.md` shape. - -- [ ] **Step 5: Make executable + verify `--help`** - -```bash -chmod +x scripts/commands/gbx-app-dev.sh scripts/commands/gbx-app-deploy.sh -bash scripts/commands/gbx-app-dev.sh --help && bash scripts/commands/gbx-app-deploy.sh --help -``` -Expected: both print banners and exit 0. - -- [ ] **Step 6: Commit** - -```bash -git add apps/genie_map/bundle scripts/commands/gbx-app-dev.* scripts/commands/gbx-app-deploy.* -git commit -m "feat(genie-map): DAB bundle + gbx:app:dev / gbx:app:deploy commands - -Co-authored-by: Isaac" -``` - ---- - -### Task 13: Slide-ware diagrams (sources authored; batch-rendered) - -**Files:** -- Create: `apps/genie_map/docs/diagrams/genie-map.py` (generator, mirrors `resources/images/generators/vapor-eyes.py` conventions) -- Create (rendered, when online): `apps/genie_map/docs/diagrams/*.png` - -**Interfaces:** -- Consumes: the vapor-eyes accent palette + Chrome-render SVG→PNG→PIL-crop pipeline. -- Produces: 4 diagram PNGs referenced by BUILD.md + the docs page. - -- [ ] **Step 1: Read the existing generator conventions** - -Read `resources/images/generators/vapor-eyes.py` (THEMES dict, render helper) so the new generator matches palette + output naming. - -- [ ] **Step 2: Author `genie-map.py` with four diagram specs** - -`genie-map-architecture` (User → client/server → {warehouse←gold, Genie Space}); `genie-map-two-paths` (viewport vs NLP); `genie-map-lineage` (GeoBrix/vapor-eyes gold → the new `wells_enriched_latest` MV → map layers); `genie-map-registry` (one config → many layers, helios as future plug-in). Consider a fifth `genie-map-dynamic-h3` explainer (zoom+density → resolution; cell-coarsen vs point-refine) — it's a strong slide. Reuse the vapor-eyes accent progression. - -- [ ] **Step 3: Batch-render (online)** - -Run the generator per the vapor-eyes generator's render path; produce the 4 PNGs. -Expected: 4 PNGs in `apps/genie_map/docs/diagrams/`, visually consistent with vapor-eyes diagrams. - -- [ ] **Step 4: Commit** - -```bash -git add apps/genie_map/docs/diagrams -git commit -m "docs(genie-map): slide-ware diagrams (architecture, two-paths, lineage, registry) - -Co-authored-by: Isaac" -``` - ---- - -### Task 14: `BUILD.md` implementer narrative + reproduce-it runbook - -**Files:** -- Modify: `apps/genie_map/docs/BUILD.md` (started in Task 2) - -**Interfaces:** -- Produces: the technical-implementer artifact (§9a of the spec). - -- [ ] **Step 1: Write the narrative sections** - -Sections: (1) What we adapted (prototype → vapor-eyes); (2) The layer-registry contract (`LayerDef`/`DatasetConfig`/`H3ResConfig`, how to add a layer, how helios plugs in later); (3) The **density-aware dynamic H3** design (why zoom-only regressed on sparse data; cell-coarsen via `h3_toparent` vs point-refine via `h3_longlatash3`; the target-cell-count heuristic) and the new `wells_enriched_latest` MV and *why* (single wells source for both H3 + points, basin/county joins); (4) Genie Space curation decisions; (5) Deploy wiring (`gbx:app:*`, DAB); (6) Gotchas with fixes (SRID-0 re-tag, `*_geojson` detection, `__LLM_MODEL__` build-time bake, `oauth-fe` profile). No internal planning vocabulary. - -- [ ] **Step 2: Write the reproduce-it runbook** - -Ordered, verifiable steps: rerun SDP for the 2 MVs (Task 18) → create/curate Genie Space (Task 18) → set bundle `genie_space_id` var → `gbx:app:deploy`. Each step states its verification. - -- [ ] **Step 3: Embed provenance** - -Reference the Task-19/20 screenshots (working viewport H3, wells layers, an NL query rendering) and the actual SQL templates. - -- [ ] **Step 4: Commit** - -```bash -git add apps/genie_map/docs/BUILD.md -git commit -m "docs(genie-map): implementer build narrative + reproduce-it runbook - -Co-authored-by: Isaac" -``` - ---- - -### Task 15: `README.md` quickstart + docs-site example page - -**Files:** -- Create: `apps/genie_map/README.md` -- Create: `docs/docs/examples/genie-map.mdx` - -**Interfaces:** -- Produces: the app quickstart + the site page embedding Task 13 diagrams. - -- [ ] **Step 1: Write `README.md`** - -Quickstart: prerequisites (vapor_eyes_lf gold materialized, a curated Genie Space id, mapbox token), `cp .env.example .env`, `gbx:app:dev`, `gbx:app:deploy`. Link to `docs/BUILD.md` for the deep dive. - -- [ ] **Step 2: Write `docs/docs/examples/genie-map.mdx`** - -Short user-facing page: what Genie Map is, the two paths, embed the 4 diagrams, link into the vapor-eyes story. Enforce user-facing voice (no internal vocabulary). - -- [ ] **Step 3: Voice check** - -Run: `grep -rn -iE "wave [0-9]+|wave-[0-9]+|subagent|dispatch" docs/docs/examples/genie-map.mdx apps/genie_map/README.md` -Expected: no output. - -- [ ] **Step 4: Commit** - -```bash -git add apps/genie_map/README.md docs/docs/examples/genie-map.mdx -git commit -m "docs(genie-map): README quickstart + docs-site example page - -Co-authored-by: Isaac" -``` - ---- - -### Task 16: New gold MV `wells_enriched_latest` (basin/play + county/state) - -> This is the **single** new wells MV. The originally-planned fixed -> `wells_h3_density_latest` MV was dropped (see the Dynamic-H3 note in "Task ordering -> rationale"): the wells H3 layer aggregates *these points* on the fly at a -> zoom+density-driven resolution (Task 8's `wells_h3.sql`), so `wells_enriched_latest` -> feeds both the wells point layer AND the wells H3 layer. - -**Files:** -- Modify: `notebooks/examples/vapor-eyes/lakeflow/transformations/gold_analytics.py` (append MV) -- Test: `notebooks/examples/vapor-eyes/lakeflow/tests/validate/test_wells_gold.py` (new) - -**Interfaces:** -- Consumes: `wells_shl` (SCD2), `ref_shale_plays` (`play_name`, `play_geom`), `ref_counties` (`county_name`, `state_fp`, `geoid`, `county_geom`). -- Produces: MV `wells_enriched_latest` with `api, operator, lease, field, well_url, longitude, latitude, well_geom_native (GEOMETRY 4326), play_name, county_name, state_fp, geoid, county_rrc`. (Point columns are aliased `longitude`/`latitude` so the app's `wells_points.sql` and the point-sourced `wells_h3.sql` read them directly.) - -- [ ] **Step 1: Write the failing validator test** - -Add `tests/validate/test_wells_gold.py` (validators in this repo run against a -materialized dev pipeline; follow the sibling `tests/validate/` pattern): - -```python -def test_wells_enriched_latest_schema(spark): - df = spark.read.table("geospatial_docs.vapor_eyes_lf.wells_enriched_latest") - cols = set(df.columns) - assert {"api", "operator", "longitude", "latitude", - "play_name", "county_name", "state_fp", "geoid"} <= cols - # one row per api (multi-play containment deduped to a single deterministic play) - assert df.count() == df.select("api").distinct().count() - # map-facing geometry re-tagged 4326 (SRID-0 silently unrendered) - srid = df.selectExpr("st_srid(well_geom_native) AS s").first()["s"] - assert srid == 4326 -``` - -- [ ] **Step 2: Run to verify it fails** - -Run (once the dev pipeline is materialized): the repo's validate command against this test. -Expected: FAIL — table does not exist yet. - -- [ ] **Step 3: Append the MV** - -```python -@dp.materialized_view( - name="wells_enriched_latest", - comment="Current well inventory spatially tagged with shale play + county/state (map-ready points)", -) -def wells_enriched_latest(): - from pyspark.sql import SparkSession - from pyspark.sql.window import Window - spark = SparkSession.getActiveSession() - - wells = ( - spark.read.table("wells_shl").filter(F.col("__END_AT").isNull()) - .select( - "api", "operator", "lease", "field", "well_url", - F.col("county").alias("county_rrc"), - F.expr("st_x(st_geomfromwkb(well_geom))").alias("longitude"), - F.expr("st_y(st_geomfromwkb(well_geom))").alias("latitude"), - F.expr("st_setsrid(st_geomfromwkb(well_geom), 4326)").alias("well_geom_native"), - ) - ) - plays = spark.read.table("ref_shale_plays").select("play_name", "play_geom") - counties = spark.read.table("ref_counties").select( - "county_name", "state_fp", "geoid", "county_geom" - ) - - # Play: a well may fall in multiple/zero plays. Keep one deterministic play - # (first by play_name) via a windowed row_number; NULL when outside all plays. - with_play = ( - wells.join(plays, F.expr("st_contains(play_geom, well_geom_native)"), "left") - .withColumn( - "_pr", - F.row_number().over(Window.partitionBy("api").orderBy(F.col("play_name").asc_nulls_last())), - ) - .filter("_pr = 1").drop("_pr", "play_geom") - ) - # County expected unique per point. - with_county = ( - with_play.join(counties, F.expr("st_contains(county_geom, well_geom_native)"), "left") - .drop("county_geom") - ) - return with_county.select( - "api", "operator", "lease", "field", "well_url", "county_rrc", - "longitude", "latitude", "well_geom_native", - "play_name", "county_name", "state_fp", "geoid", - ) -``` - -- [ ] **Step 4: Verify (materialize + run test)** - -Deploy + run the vapor-eyes SDP dev target (existing bundle) and run the validator. -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add notebooks/examples/vapor-eyes/lakeflow/transformations/gold_analytics.py notebooks/examples/vapor-eyes/lakeflow/tests/validate/test_wells_gold.py -git commit -m "feat(vapor-eyes): wells_enriched_latest gold MV with play + county/state joins - -One row per well tagged with shale play (first-containing, deterministic) -and authoritative county/state from ref_counties; SRID 4326 point. - -Co-authored-by: Isaac" -``` - ---- - -### Task 18: Materialize gold + curate the vapor-eyes Genie Space - -**Files:** -- Modify: none (workspace-side operation; record the space id in BUILD.md + bundle var) - -**Interfaces:** -- Consumes: all gold tables incl. the new `wells_enriched_latest` MV. -- Produces: a Genie Space id (recorded for Task 12's `genie_space_id` var + `.env`). - -> **Extraction note** (see `2026-07-16-geobrix-deploy-helpers-vision.md`): keep the space's -> table list + curated instructions/example-SQL **declarative** (data, not one-off glue) so a -> future `create_genie_space(tables, warehouse, …)` helper can be lifted from this manual -> curation. Don't build the helper now. - -- [ ] **Step 1: Deploy + run the vapor-eyes SDP so the new MV materializes** - -Run the existing vapor-eyes Lakeflow bundle (`-p oauth-fe`) to update the pipeline with Task 16 (`wells_enriched_latest`). Verify the MV exists and is non-empty via a warehouse query. - -- [ ] **Step 2: Create the Genie Space** - -Create a Genie Space (workspace UI or API) over `geospatial_docs.vapor_eyes_lf`, adding tables: `hotspot_latest`, `plume_leaderboard_latest`, `wells_enriched_latest`, `plume_candidate_wells`, `ref_shale_plays`, `ref_counties`, `operator_intensity_latest`, `detections_by_county`, `emissions_by_play`. (If a space already exists, reuse its id — the goal is wiring the app to *a* space, not necessarily creating one.) - -- [ ] **Step 3: Add instructions + example SQL that emit `*_geojson` geometry** - -Curate: column descriptions; join hints (wells↔plumes via `plume_candidate_wells`; ↔basin via `st_contains(play_geom, ...)`; ↔county via `st_contains(county_geom, ...)`); the concentration-led framing (rank by `max_conc_ppmm`, never sum emission rates). Add example NL→SQL pairs whose SELECT aliases geometry as `hex_geojson`/`plume_geojson` via `ST_ASGEOJSON(...)` so the app renders answers. Example prompts: "well density in Loving County, TX"; "operators with the most wells in the Delaware Basin"; "highest-concentration plumes and their nearest operator". - -- [ ] **Step 4: Smoke-test the space** - -Ask one geometry prompt and one aggregate prompt in the space; confirm the geometry answer returns a `*_geojson` column. - -- [ ] **Step 5: Record the space id** - -Put the id in `apps/genie_map/bundle/databricks.yml` `genie_space_id` default + note in `apps/genie_map/.env.example` (leave blank there, real value in local `.env`) and BUILD.md runbook. - -- [ ] **Step 6: Commit the recorded id** - -```bash -git add apps/genie_map/bundle/databricks.yml apps/genie_map/docs/BUILD.md -git commit -m "chore(genie-map): record curated vapor-eyes Genie Space id - -Co-authored-by: Isaac" -``` - ---- - -### Task 19: Live smoke — viewport SQL templates against `vapor_eyes_lf` - -**Files:** -- Modify: none (verification; capture provenance into BUILD.md) - -**Interfaces:** -- Consumes: the four SQL templates + materialized gold. -- Produces: evidence each layer query returns rows with the expected columns. - -- [ ] **Step 1: Run each SQL template on the warehouse** - -For each of `hotspot_h3.sql`, `wells_h3.sql`, `plume_points.sql`, `wells_points.sql`: substitute a Delaware Basin bbox + the fully-qualified table name (and, for the two H3 queries, the dynamic-res params — pick a low-zoom and a high-zoom case), run on warehouse `82e587bd93c6cbcf` (`-p oauth-fe`, or the databricks-query skill). Verify each returns rows and the columns the hooks expect (`hex`+metric, or `longitude/latitude`+tooltips), and that the H3 queries return **coarser hexes at low zoom / finer at high zoom** (dynamic resolution working). - -- [ ] **Step 2: Capture results into BUILD.md** - -Record row counts + a sample row per query as provenance. - -- [ ] **Step 3: Commit** - -```bash -git add apps/genie_map/docs/BUILD.md -git commit -m "docs(genie-map): capture live SQL-template smoke results - -Co-authored-by: Isaac" -``` - ---- - -### Task 20: Deploy + end-to-end demo verification - -**Files:** -- Modify: none (verification; capture screenshots for BUILD.md/diagrams) - -**Interfaces:** -- Consumes: everything. -- Produces: a deployed app + screenshots proving the MVP paths. - -- [ ] **Step 1: Set local `.env` and run locally** - -`cp apps/genie_map/.env.example apps/genie_map/.env`, fill token + mapbox + genie space id, `gbx:app:dev`. Confirm the map renders CH4 hotspots at low zoom, well-density H3, and wells/plume points at high zoom. - -- [ ] **Step 2: Exercise the Genie NLP path** - -Ask "well density in Loving County, TX" and "highest-concentration plumes and their nearest operator"; confirm at least one renders a layer via the `*_geojson` path. - -- [ ] **Step 3: Deploy** - -Run `bash scripts/commands/gbx-app-deploy.sh --profile oauth-fe`. Confirm the app comes up in the workspace and the deployed URL renders. - -- [ ] **Step 4: Capture screenshots + finalize BUILD.md provenance** - -Screenshot each working path; embed in BUILD.md; verify diagrams (Task 13) render. - -- [ ] **Step 5: Commit** - -```bash -git add apps/genie_map/docs -git commit -m "docs(genie-map): end-to-end demo verification + screenshots - -Co-authored-by: Isaac" -``` - ---- - -## Follow-on (separate plans, not this MVP) - -- **Phase 2 — PMTiles:** consume the SDP's PMTiles fanout shards + light overview as a MapLibre/deck.gl vector layer. New plan. -- **Phase 3+ — Raster/EMIT overlay; helios as a selectable Genie Space + dataset config** (proves the registry seam); multi-layer selection UI. New plan(s). - -## Self-Review notes - -- **Spec coverage:** §1 goals → Tasks 1–20; §3 data spine → registry (5) + SQL (6–9); §4 registry + dynamic H3 → Tasks 4–6, 8; §5 gold MV → Task 16; §6 Genie Space → Task 18; §7 packaging/deploy → Tasks 1,3,12; §8 phasing → this plan = P0+P1, follow-on noted; §9 storytelling → Tasks 13 (slide-ware), 14 (BUILD.md), 15 (README/site), with provenance capture in 19–20; §9 constraint (render batched online) honored in Task 13. All spec sections mapped. -- **Type consistency:** `LayerDef`/`DatasetConfig` (+ `H3ResConfig`) defined in Task 4, consumed in 5/6/10; `createPointLayerConfig`/`createH3LayerConfig` signatures consistent across 4/6; `buildLayerParams`/`useLayerData` signatures consistent 6/10; the dynamic-H3 SQL params emitted by `buildLayerParams` (Task 6: `zoom_level`, `zoom_break_1..4`, `res_1..5`, `min_res`, `native_res`/`max_res`, `target_cells`) match the `:param` names consumed in `hotspot_h3.sql` (Task 6) and `wells_h3.sql` (Task 8); `wells_enriched_latest` columns (Task 16: `longitude`, `latitude`, `operator`, `county_name`, `play_name`, `api`) match the reads in `wells_h3.sql` (Task 8) and `wells_points.sql` (Task 9). Registry query names (`hotspot_h3`, `wells_h3`, `plume_points`, `wells_points`) match the SQL file names in Tasks 6–9. -- **Placeholder scan:** no TBD/TODO; every code step shows code; SQL/bundle/command bodies are complete. One documented fallback (DAB `genie_space` resource type) is a real contingency with a stated alternative, not a placeholder. -- **Known live-verify dependency:** Tasks 8/9 SQL and Task 10 wiring are written against the documented MV schema (16/17); their live correctness is verified in Task 19 after Task 18 materializes gold — ordering noted at top. diff --git a/docs/superpowers/plans/2026-07-19-issue-59-nodata-reducer-null.md b/docs/superpowers/plans/2026-07-19-issue-59-nodata-reducer-null.md deleted file mode 100644 index 6cd9b9030..000000000 --- a/docs/superpowers/plans/2026-07-19-issue-59-nodata-reducer-null.md +++ /dev/null @@ -1,736 +0,0 @@ -# Issue #59 — NULL for Zero-Valid-Pixel Raster Reducers — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make `gbx_rst_max` / `gbx_rst_min` / `gbx_rst_avg` / `gbx_rst_median` return SQL `NULL` (not NaN on light, not `0.0` on heavy) for a raster band with zero valid pixels, identically on both tiers, and align light `rst_isempty` to heavy's all-nodata-aware semantics. - -**Architecture:** Light tier (pyrx) is pure Python — reducers live in `core/accessors.py` and already detect the empty case via `if vals.size`; the change is `float("nan")` → `None` plus an all-nodata branch in `isempty`. Heavy tier (rasterx) reducers return dense `Array[Double]` which cannot carry a per-band NULL, so all four `execute` methods change to `Array[java.lang.Double]` and gain a `stats.getValid_count == 0` guard (the same signal `RST_PixelCount` uses). No internal Scala code calls these `execute` methods, so the signature change is contained. Docs (release notes, docstrings, function-info, reducer pages) are updated to state NULL-on-empty. - -**Tech Stack:** Scala 2.13.16 / Spark 4.0.0 / GDAL Java bindings (heavy, built + tested in the `geobrix-dev` Docker container via Maven); Python 3.12 / rasterio / numpy / PySpark (light); Docusaurus MDX docs; `gbx:*` command palette. - -## Global Constraints - -- **Version is 0.4.1 (beta).** Beta = APIs may break to stabilize; **no function aliases** — one canonical name per function. -- **Convention (the invariant this plan enforces):** a per-band element of `rst_max`/`rst_min`/`rst_avg`/`rst_median` is SQL `NULL` **iff** that band has zero valid pixels. `rst_pixelcount` stays `0` for an empty band — do **not** change it. -- **Binding parity is enforced** — every function name must exist as a Scala `override def name`, a Python `functions.py` binding, and a `function-info.json` key. The four reducers already exist in all three; this plan changes behavior/return representation, not the name set, so no new names are added. -- **User-facing docs voice** — no internal planning vocabulary (no "wave N", no subagent/dispatch references) anywhere under `docs/docs/`. QC judge enforces via `internals-leak`. -- **Heavy work runs in Docker.** Never run Maven/Scala suites inline on the host; dispatch via a Task subagent using the `gbx:*` commands, and give a one-line progress update ~every 30s on long runs. -- **gh account** — `gh auth switch --user mjohns-databricks` before any push/PR/comment to `databrickslabs/geobrix`. -- **Deferred (do NOT touch in this plan):** covering-mode tessellation divergence for geometrically-overlapping-but-all-nodata chips (`RasterTessellate.scala` / `tessellate.py:183-184`). Filed as a separate follow-up issue. - ---- - -## Task 1: Light tier — reducers return None + isempty all-nodata parity - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/core/accessors.py` (reducers `:120-153`, `isempty` `:93-94`) -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` (docstrings `:2314-2350`, `rst_isempty` `:2263-2264`) -- Test: `python/geobrix/test/pyrx/test_core_accessors_stats.py` (replace `:70-78`), and a new `test_core_accessors_isempty.py` (or extend the stats file) for isempty - -**Interfaces:** -- Consumes: existing `_valid_values(ds, band_index) -> np.ndarray` (`accessors.py:113-117`), which returns an empty array for an all-masked band. -- Produces: `accessors.avg/minimum/maximum/median(ds) -> List[Optional[float]]` where an empty band yields `None`; `accessors.pixelcount(ds) -> List[int]` **unchanged** (empty → `0`); `accessors.isempty(ds) -> bool` returns `True` when width/height/count is 0 **or** every band has zero valid pixels. - -- [ ] **Step 1: Replace the NaN-asserting test with a None-asserting test** - -In `python/geobrix/test/pyrx/test_core_accessors_stats.py`, replace the whole `test_stats_all_invalid_band_is_nan_zero` function (lines 70-78) with: - -```python -def test_stats_all_invalid_band_is_null_zero(): - # A band that is entirely NoData has zero valid pixels: reducers must return - # None (SQL NULL), never NaN or 0.0 (issue #59). pixelcount stays 0. - data = np.full((2, 2), -9999.0, dtype="float32") - raster = _custom_raster(data) - with _serde.open_tile(raster) as ds: - assert accessors.avg(ds) == [None] - assert accessors.minimum(ds) == [None] - assert accessors.maximum(ds) == [None] - assert accessors.median(ds) == [None] - assert accessors.pixelcount(ds) == [0] - - -def test_stats_genuine_zero_is_not_null(): - # A band of genuine 0.0 valid pixels must return 0.0, not None — the - # zero-not-null trap. nodata is a sentinel that no pixel equals. - data = np.zeros((2, 2), dtype="float32") - raster = _custom_raster(data) # nodata defaults to -9999.0, unmatched - with _serde.open_tile(raster) as ds: - assert accessors.avg(ds) == [pytest.approx(0.0)] - assert accessors.minimum(ds) == [pytest.approx(0.0)] - assert accessors.maximum(ds) == [pytest.approx(0.0)] - assert accessors.median(ds) == [pytest.approx(0.0)] - assert accessors.pixelcount(ds) == [4] -``` - -- [ ] **Step 2: Run the tests to verify they fail** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_core_accessors_stats.py -k "all_invalid_band_is_null_zero or genuine_zero_is_not_null"` -Expected: `test_stats_all_invalid_band_is_null_zero` FAILS (reducers currently return NaN, `[nan] == [None]` is False); `test_stats_genuine_zero_is_not_null` PASSES (0.0 already works). - -- [ ] **Step 3: Change the four reducers from NaN to None** - -In `python/geobrix/src/databricks/labs/gbx/pyrx/core/accessors.py`, edit the four reducers so the empty-band branch yields `None` and update each docstring. Replace lines 120-153 with: - -```python -def avg(ds) -> List[Optional[float]]: - """Per-band mean of valid pixels; None (SQL NULL) for empty/all-invalid bands.""" - out: List[Optional[float]] = [] - for bi in range(1, ds.count + 1): - vals = _valid_values(ds, bi) - out.append(float(np.mean(vals)) if vals.size else None) - return out - - -def minimum(ds) -> List[Optional[float]]: - """Per-band min of valid pixels; None (SQL NULL) for empty/all-invalid bands.""" - out: List[Optional[float]] = [] - for bi in range(1, ds.count + 1): - vals = _valid_values(ds, bi) - out.append(float(np.min(vals)) if vals.size else None) - return out - - -def maximum(ds) -> List[Optional[float]]: - """Per-band max of valid pixels; None (SQL NULL) for empty/all-invalid bands.""" - out: List[Optional[float]] = [] - for bi in range(1, ds.count + 1): - vals = _valid_values(ds, bi) - out.append(float(np.max(vals)) if vals.size else None) - return out - - -def median(ds) -> List[Optional[float]]: - """Per-band median of valid pixels; None (SQL NULL) for empty/all-invalid bands.""" - out: List[Optional[float]] = [] - for bi in range(1, ds.count + 1): - vals = _valid_values(ds, bi) - out.append(float(np.median(vals)) if vals.size else None) - return out -``` - -(`Optional` is already imported in this module — it is used by `getnodata` at `:102`. If a flake8 run reports it missing, add `Optional` to the existing `from typing import ...` line.) - -- [ ] **Step 4: Run the tests to verify they pass** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_core_accessors_stats.py -k "all_invalid_band_is_null_zero or genuine_zero_is_not_null"` -Expected: both PASS. - -- [ ] **Step 5: Write the failing isempty parity test** - -Create `python/geobrix/test/pyrx/test_core_accessors_isempty.py`: - -```python -"""isempty parity: an all-nodata raster is empty (matches heavy RasterAccessors.isEmpty).""" -import numpy as np -from rasterio.io import MemoryFile -from rasterio.transform import from_origin - -from databricks.labs.gbx.pyrx import _serde -from databricks.labs.gbx.pyrx.core import accessors - - -def _raster(data, nodata=-9999.0): - h, w = data.shape - profile = dict( - driver="GTiff", width=w, height=h, count=1, dtype="float32", - crs="EPSG:4326", transform=from_origin(10.0, 50.0, 0.5, 0.5), nodata=nodata, - ) - with MemoryFile() as mf: - with mf.open(**profile) as dst: - dst.write(data.astype("float32"), 1) - return mf.read() - - -def test_isempty_all_nodata_is_empty(): - data = np.full((2, 2), -9999.0, dtype="float32") - with _serde.open_tile(_raster(data)) as ds: - assert accessors.isempty(ds) is True - - -def test_isempty_has_valid_pixels_is_not_empty(): - data = np.array([[1.0, 2.0], [3.0, 4.0]], dtype="float32") - with _serde.open_tile(_raster(data)) as ds: - assert accessors.isempty(ds) is False -``` - -- [ ] **Step 6: Run the isempty test to verify the all-nodata case fails** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_core_accessors_isempty.py -v` -Expected: `test_isempty_all_nodata_is_empty` FAILS (current isempty only checks dimensions, returns False); `test_isempty_has_valid_pixels_is_not_empty` PASSES. - -- [ ] **Step 7: Extend isempty with an all-nodata branch** - -In `python/geobrix/src/databricks/labs/gbx/pyrx/core/accessors.py`, replace lines 93-94: - -```python -def isempty(ds) -> bool: - """True if the raster has no size, or every band has zero valid pixels. - - Mirrors heavyweight RasterAccessors.isEmpty (null / no size / all bands - fully NoData). A dimensionally-valid raster whose every band is NoData is - still empty (issue #59). - """ - if int(ds.width) == 0 or int(ds.height) == 0 or int(ds.count) == 0: - return True - return all(_valid_values(ds, bi).size == 0 for bi in range(1, ds.count + 1)) -``` - -- [ ] **Step 8: Run the isempty test to verify it passes** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_core_accessors_isempty.py -v` -Expected: both PASS. - -- [ ] **Step 9: Update the public docstrings in functions.py** - -In `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py`, update the four reducer docstrings (lines 2314-2342) — change each `Empty / all-invalid bands return NaN.` to `Empty / all-invalid bands return NULL.`. Leave `rst_pixelcount` (`:2346-2348`, "return 0") unchanged. Then give `rst_isempty` (`:2263-2264`) a docstring: - -```python -def rst_isempty(tile: ColLike) -> Column: - """True if the raster has no size or every band is entirely NoData; BOOLEAN.""" - return _u_isempty(_raster_field(_col(tile))) -``` - -- [ ] **Step 10: Run the full pyrx accessors suite + lint** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_core_accessors_stats.py --path python/geobrix/test/pyrx/test_core_accessors_isempty.py -v` -Then: `bash scripts/commands/gbx-lint-python.sh --check` -Expected: all tests PASS; lint clean (run `--fix` on host first if black/isort/flake8 complain, then re-run `--check`). - -- [ ] **Step 11: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/pyrx/core/accessors.py \ - python/geobrix/src/databricks/labs/gbx/pyrx/functions.py \ - python/geobrix/test/pyrx/test_core_accessors_stats.py \ - python/geobrix/test/pyrx/test_core_accessors_isempty.py -git commit -m "fix(pyrx): reducers return NULL (not NaN) for zero-valid-pixel bands - -gbx_rst_max/min/avg/median now emit None for an all-nodata band, and -rst_isempty is all-nodata-aware, matching heavy RasterAccessors.isEmpty. -Addresses issue #59 on the light tier. pixelcount unchanged (empty -> 0). - -Co-authored-by: Isaac" -``` - ---- - -## Task 2: Heavy tier — RST_Avg returns NULL for zero-valid-pixel bands - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/accessors/RST_Avg.scala` (`execute` `:54-64`) -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_AccessorsExecuteTest.scala` - -**Interfaces:** -- Consumes: `band.AsMDArray().GetStatistics()` (GDAL) — a `Statistics` object with `getMean` and `getValid_count`; already called in `RST_Avg.execute`. -- Produces: `RST_Avg.execute(ds: Dataset): Array[java.lang.Double]` — element is `null` for a null band or `getValid_count == 0`, else the boxed mean. `ArrayData.toArrayData` accepts `Array[java.lang.Double]` and maps `null` to a SQL NULL element (the expression already declares `dataType = ArrayType(DoubleType)`, `nullable = true`). - -- [ ] **Step 1: Add an all-nodata + genuine-zero test to the Scala accessor suite** - -The suite opens a MODIS TIF in `beforeAll`. Add a helper that builds a synthetic all-nodata raster in `/vsimem`, and two tests. Append inside `RST_AccessorsExecuteTest` (before the closing brace at line 246): - -```scala - /** Build a 4x4 single-band Float32 /vsimem raster where every pixel == nodata. */ - private def allNodataDs(nodata: Double = -9999.0): Dataset = { - val path = s"/vsimem/all_nodata_${java.util.UUID.randomUUID().toString.replace("-", "")}.tif" - val drv = gdal.GetDriverByName("GTiff") - val d = drv.Create(path, 4, 4, 1, org.gdal.gdalconst.gdalconstConstants.GDT_Float32) - val band = d.GetRasterBand(1) - band.SetNoDataValue(nodata) - val buf = Array.fill[Double](16)(nodata) - band.WriteRaster(0, 0, 4, 4, buf) - band.FlushCache() - d.FlushCache() - band.delete() - d - } - - /** Build a 4x4 single-band Float32 /vsimem raster of genuine 0.0 pixels. */ - private def allZeroDs(nodata: Double = -9999.0): Dataset = { - val path = s"/vsimem/all_zero_${java.util.UUID.randomUUID().toString.replace("-", "")}.tif" - val drv = gdal.GetDriverByName("GTiff") - val d = drv.Create(path, 4, 4, 1, org.gdal.gdalconst.gdalconstConstants.GDT_Float32) - val band = d.GetRasterBand(1) - band.SetNoDataValue(nodata) - val buf = Array.fill[Double](16)(0.0) - band.WriteRaster(0, 0, 4, 4, buf) - band.FlushCache() - d.FlushCache() - band.delete() - d - } - - test("RST_Avg returns null for an all-nodata band (issue #59)") { - val empty = allNodataDs() - RST_Avg.execute(empty).head shouldBe null - empty.delete() - } - - test("RST_Avg returns 0.0 (not null) for a genuine-zero band") { - val zeros = allZeroDs() - RST_Avg.execute(zeros).head shouldBe (0.0: java.lang.Double) - zeros.delete() - } -``` - -- [ ] **Step 2: Run the RST_Avg tests to verify the all-nodata case fails** (Docker — dispatch a Task subagent) - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_AccessorsExecuteTest' --log issue59-avg.log` -Expected: `RST_Avg returns null for an all-nodata band` FAILS — today `execute` returns `Array[Double]` (primitive), so `.head` is `0.0`, and also the code returns `stats.getMean` (0.0) for the empty band; the boxed-null assertion cannot even compile against `Array[Double]`. This step confirms the compile/behavior gap. The genuine-zero test may fail to compile until Step 3 changes the return type — expect a compile error naming `RST_Avg.execute` return type; that is the failing signal. - -- [ ] **Step 3: Change RST_Avg.execute to Array[java.lang.Double] with a valid-count guard** - -In `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/accessors/RST_Avg.scala`, replace `execute` (lines 54-64): - -```scala - def execute(ds: Dataset): Array[java.lang.Double] = { - (1 to ds.GetRasterCount()).map { bandIndex => - val band = ds.GetRasterBand(bandIndex) - if (band == null) null - else { - val md = band.AsMDArray() - val stats = md.GetStatistics() - val res: java.lang.Double = - if (stats == null || stats.getValid_count == 0) null - else stats.getMean - if (stats != null) stats.delete() - md.delete() - band.delete() - res - } - }.toArray - } -``` - -Also update the class doc comment (line 14) to note: `an all-nodata band (zero valid pixels) yields a NULL element`. - -- [ ] **Step 4: Run the RST_Avg tests to verify they pass** (Docker) - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_AccessorsExecuteTest' --log issue59-avg.log` -Expected: both new tests PASS. The pre-existing `RST_Avg should return the average px value` test (line 28-32) still PASSES — `avg.foreach(a => a shouldBe expected)` compares boxed `java.lang.Double` to a primitive `Double`; scalatest `shouldBe` handles the boxing. If it fails on a type mismatch, change that assertion to `a shouldBe (expected: java.lang.Double)`. - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/accessors/RST_Avg.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_AccessorsExecuteTest.scala -git commit -m "fix(rasterx): RST_Avg returns NULL for zero-valid-pixel bands (#59) - -execute now returns Array[java.lang.Double] and emits null when a band -has zero valid pixels (getValid_count == 0), replacing the leaked 0.0. - -Co-authored-by: Isaac" -``` - ---- - -## Task 3: Heavy tier — RST_Max and RST_Min return NULL for zero-valid-pixel bands - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/accessors/RST_Max.scala` (`execute` `:54-64`) -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/accessors/RST_Min.scala` (`execute` `:54-64`) -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_AccessorsExecuteTest.scala` - -**Interfaces:** -- Consumes: `BandAccessors.getMinMax(band): (Double, Double)` (`BandAccessors.scala:23-30`) — **leave this helper unchanged**; it is also used by `RST_TileXYZ.scala:157`. The empty-band decision is made inside the reducer via a `getValid_count` check, not inside `getMinMax`. -- Produces: `RST_Max.execute(ds): Array[java.lang.Double]` and `RST_Min.execute(ds): Array[java.lang.Double]` — `null` element for a null band or `getValid_count == 0`, else the boxed max/min. - -- [ ] **Step 1: Add all-nodata + genuine-zero tests for Max and Min** - -In `RST_AccessorsExecuteTest.scala`, append (the `allNodataDs`/`allZeroDs` helpers from Task 2 already exist): - -```scala - test("RST_Max returns null for an all-nodata band (issue #59)") { - val empty = allNodataDs() - RST_Max.execute(empty).head shouldBe null - empty.delete() - } - - test("RST_Max returns 0.0 (not null) for a genuine-zero band") { - val zeros = allZeroDs() - RST_Max.execute(zeros).head shouldBe (0.0: java.lang.Double) - zeros.delete() - } - - test("RST_Min returns null for an all-nodata band (issue #59)") { - val empty = allNodataDs() - RST_Min.execute(empty).head shouldBe null - empty.delete() - } - - test("RST_Min returns 0.0 (not null) for a genuine-zero band") { - val zeros = allZeroDs() - RST_Min.execute(zeros).head shouldBe (0.0: java.lang.Double) - zeros.delete() - } -``` - -- [ ] **Step 2: Run to verify failure/compile gap** (Docker) - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_AccessorsExecuteTest' --log issue59-minmax.log` -Expected: the new tests FAIL or fail to compile (execute still returns `Array[Double]`, all-nodata leaks `0.0`, and the null assertion needs boxed elements). - -- [ ] **Step 3: Change RST_Max.execute with a valid-count guard** - -In `RST_Max.scala`, replace `execute` (lines 54-64): - -```scala - def execute(ds: Dataset): Array[java.lang.Double] = { - (1 to ds.GetRasterCount()).map { bandIndex => - val band = ds.GetRasterBand(bandIndex) - if (band == null) null - else { - val md = band.AsMDArray() - val stats = md.GetStatistics() - val res: java.lang.Double = - if (stats == null || stats.getValid_count == 0) null - else { - val (_, max) = BandAccessors.getMinMax(band) - max - } - if (stats != null) stats.delete() - md.delete() - band.delete() - res - } - }.toArray - } -``` - -Update the class doc comment (line 15) to note the NULL-on-empty behavior. - -- [ ] **Step 4: Change RST_Min.execute with a valid-count guard** - -In `RST_Min.scala`, replace `execute` (lines 54-64): - -```scala - def execute(ds: Dataset): Array[java.lang.Double] = { - (1 to ds.GetRasterCount()).map { bandIndex => - val band = ds.GetRasterBand(bandIndex) - if (band == null) null - else { - val md = band.AsMDArray() - val stats = md.GetStatistics() - val res: java.lang.Double = - if (stats == null || stats.getValid_count == 0) null - else { - val (min, _) = BandAccessors.getMinMax(band) - min - } - if (stats != null) stats.delete() - md.delete() - band.delete() - res - } - }.toArray - } -``` - -Update the class doc comment (line 15) to note the NULL-on-empty behavior. - -- [ ] **Step 5: Run to verify all Max/Min tests pass** (Docker) - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_AccessorsExecuteTest' --log issue59-minmax.log` -Expected: the four new tests PASS; the pre-existing `RST_Max`/`RST_Min` tests (lines 92-95, 128-131) still PASS (boxed-vs-primitive `shouldBe` handles boxing; if not, box the expected value as in Task 2 Step 4). - -- [ ] **Step 6: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/accessors/RST_Max.scala \ - src/main/scala/com/databricks/labs/gbx/rasterx/expressions/accessors/RST_Min.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_AccessorsExecuteTest.scala -git commit -m "fix(rasterx): RST_Max/RST_Min return NULL for zero-valid-pixel bands (#59) - -execute methods now return Array[java.lang.Double] and emit null when a -band has zero valid pixels (getValid_count == 0), replacing the leaked -0.0 that ComputeRasterMinMax's zero-initialized array produced. -BandAccessors.getMinMax is unchanged (still used by RST_TileXYZ). - -Co-authored-by: Isaac" -``` - ---- - -## Task 4: Heavy tier — RST_Median returns NULL for zero-valid-pixel bands - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/accessors/RST_Median.scala` (`execute` `:54-65`) -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_AccessorsExecuteTest.scala` - -**Interfaces:** -- Consumes: `GDALWarp.executeWarp` (warps to a 1×1 raster via `gdalwarp -r med -ts 1 1`), then `resDs.GetRasterBand(i).AsMDArray().GetStatistics()`. -- Produces: `RST_Median.execute(ds: Dataset, options: Map[String, String]): Array[java.lang.Double]` — `null` element when the warped 1×1 band's stats are null or `getValid_count == 0`, else the boxed median (`getMax` of the 1×1 med-warp). Adds the currently-missing null-check on `GetStatistics()`. - -- [ ] **Step 1: Add all-nodata + genuine-zero tests for Median** - -In `RST_AccessorsExecuteTest.scala`, append: - -```scala - test("RST_Median returns null for an all-nodata band (issue #59)") { - val empty = allNodataDs() - RST_Median.execute(empty, Map.empty).head shouldBe null - empty.delete() - } - - test("RST_Median returns 0.0 (not null) for a genuine-zero band") { - val zeros = allZeroDs() - RST_Median.execute(zeros, Map.empty).head shouldBe (0.0: java.lang.Double) - zeros.delete() - } -``` - -- [ ] **Step 2: Run to verify failure** (Docker) - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_AccessorsExecuteTest' --log issue59-median.log` -Expected: the all-nodata Median test FAILS (today `getMax` of the 1×1 med-warp of an all-nodata input is not null, and the return type is primitive `Array[Double]`). - -- [ ] **Step 3: Change RST_Median.execute with a null/valid-count guard** - -In `RST_Median.scala`, replace `execute` (lines 54-65): - -```scala - def execute(ds: Dataset, options: Map[String, String]): Array[java.lang.Double] = { - val outShortName = ds.GetDriver().getShortName - val uuid = java.util.UUID.randomUUID().toString.replace("-", "") - val extension = GDAL.getExtension(outShortName) - val resultPath = s"/vsimem/rst_median_$uuid.$extension" - val cmd = s"gdalwarp -r med -ts 1 1" - val (resDs, _) = GDALWarp.executeWarp(resultPath, Array(ds), options, cmd) - val medians: Array[java.lang.Double] = (1 to resDs.GetRasterCount()).map { i => - val md = resDs.GetRasterBand(i).AsMDArray() - val stats = md.GetStatistics() - val res: java.lang.Double = - if (stats == null || stats.getValid_count == 0) null - else stats.getMax - if (stats != null) stats.delete() - md.delete() - res - }.toArray - resDs.delete() - gdal.Unlink(resultPath) - medians - } -``` - -Update the class doc comment (line 15) to note NULL-on-empty behavior. - -- [ ] **Step 4: Run to verify Median tests pass** (Docker) - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_AccessorsExecuteTest' --log issue59-median.log` -Expected: both new tests PASS; the pre-existing `RST_Median should return approximated median` test (lines 97-106) still PASSES. - -- [ ] **Step 5: Run the broader eval suites to catch ArrayData/nullable regressions** (Docker) - -Run: `bash scripts/commands/gbx-test-scala.sh --suites 'com.databricks.labs.gbx.rasterx.expressions.RST_AccessorsEvalTest,com.databricks.labs.gbx.rasterx.expressions.RST_AccessorsExecuteTest,com.databricks.labs.gbx.rasterx.RasterXFunctionsTest' --log issue59-eval.log` -Expected: PASS. These exercise the `eval` → `ArrayData.toArrayData` path end-to-end, confirming a boxed `null` element round-trips to a SQL NULL array element for all four reducers. - -- [ ] **Step 6: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/accessors/RST_Median.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_AccessorsExecuteTest.scala -git commit -m "fix(rasterx): RST_Median returns NULL for zero-valid-pixel bands (#59) - -execute returns Array[java.lang.Double], adds the previously-missing null -check on GetStatistics, and emits null when the med-warped band has zero -valid pixels. Completes the reducer NULL convention on the heavy tier. - -Co-authored-by: Isaac" -``` - ---- - -## Task 5: Cross-tier seam-reconciliation regression test - -**Files:** -- Test (light): `python/geobrix/test/pyrx/test_issue59_seam_reconciliation.py` (new) - -**Interfaces:** -- Consumes: the light `accessors.maximum` (now None-on-empty) and standard Spark `GROUP BY ... MAX()`. This is the concrete harm from the issue: NaN sorts high and overwrites a real value in `MAX()`; NULL is ignored by `MAX()`. - -- [ ] **Step 1: Write the regression test that models the seam GROUP BY** - -Create `python/geobrix/test/pyrx/test_issue59_seam_reconciliation.py`: - -```python -"""Issue #59 regression: an all-nodata chip's reducer must not poison MAX() in a -seam-reconciliation GROUP BY. Pure-accessor level (Spark-free) — asserts that the -empty-band reducer yields None and that None is ignored by a max() over a group -where a real value also exists, whereas the old NaN would win. -""" -import numpy as np -from rasterio.io import MemoryFile -from rasterio.transform import from_origin - -from databricks.labs.gbx.pyrx import _serde -from databricks.labs.gbx.pyrx.core import accessors - - -def _raster(data, nodata=-9999.0): - h, w = data.shape - profile = dict( - driver="GTiff", width=w, height=h, count=1, dtype="float32", - crs="EPSG:4326", transform=from_origin(10.0, 50.0, 0.5, 0.5), nodata=nodata, - ) - with MemoryFile() as mf: - with mf.open(**profile) as dst: - dst.write(data.astype("float32"), 1) - return mf.read() - - -def test_all_nodata_chip_does_not_poison_group_max(): - # One H3 cell reconciled across two chips at a tile seam: a real-data chip - # (max 42.0) and an all-nodata chip (empty -> None). MAX over the group must - # be 42.0, not None and not NaN. - real = _raster(np.array([[42.0, 1.0], [2.0, 3.0]], dtype="float32")) - empty = _raster(np.full((2, 2), -9999.0, dtype="float32")) - - with _serde.open_tile(real) as r, _serde.open_tile(empty) as e: - real_max = accessors.maximum(r)[0] - empty_max = accessors.maximum(e)[0] - - assert empty_max is None - # SQL MAX() ignores NULL: emulate the reconciliation with a NULL-skipping max. - group = [v for v in (real_max, empty_max) if v is not None] - assert max(group) == 42.0 -``` - -- [ ] **Step 2: Run the regression test** (Task 1 must be merged/applied so `maximum` returns None) - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_issue59_seam_reconciliation.py -v` -Expected: PASS. (Before Task 1, `empty_max` was NaN — `empty_max is None` would be False — so this test also guards against regressing the light fix.) - -- [ ] **Step 3: Commit** - -```bash -git add python/geobrix/test/pyrx/test_issue59_seam_reconciliation.py -git commit -m "test(pyrx): guard seam-reconciliation MAX against all-nodata chips (#59) - -Regression test for the core harm in issue #59 — an all-nodata chip's -reducer must be NULL (ignored by MAX), never NaN (which poisons the -GROUP BY seam reconciliation). - -Co-authored-by: Isaac" -``` - ---- - -## Task 6: Docs — beta release notes, function-info regen, reducer/isempty pages - -**Files:** -- Modify: `docs/docs/beta-release-notes.mdx` (v0.4.1 section, after the last bullet in "What's new in v0.4.1") -- Modify: `docs/docs/api/raster-functions.mdx` (`rst_avg` `:164`, `rst_max` `:289`, `rst_median` `:303`, `rst_min` `:345`, `rst_isempty` `:1339` sections) -- Regenerate: `src/main/resources/com/databricks/labs/gbx/function-info.json` (via command, not by hand) - -**Interfaces:** -- Consumes: the doc SQL examples in `docs/tests/python/api/rasterx_functions_sql.py` (`rst_avg_sql_example` etc.) feed `function-info.json`. No new function names — only descriptive text and the regenerated file change. - -- [ ] **Step 1: Add the breaking-change entry to the v0.4.1 release notes** - -In `docs/docs/beta-release-notes.mdx`, append a bullet to the end of the "What's new in v0.4.1" list (before the `---` that closes the section): - -```markdown -- **Raster value reducers return `NULL` for all-nodata bands (behavior change).** `gbx_rst_max`, `gbx_rst_min`, `gbx_rst_avg`, and `gbx_rst_median` now return SQL `NULL` for a band with zero valid pixels — for example an H3 covering-tessellation cell that clips only NoData — on **both** the lightweight and heavyweight tiers. Previously the lightweight tier returned `NaN` and the heavyweight tier returned `0.0`; neither was catchable or aggregation-safe. `NaN` silently passed `WHERE measure IS NOT NULL` and could overwrite a real value in `MAX()` during a `GROUP BY` seam reconciliation (NaN sorts greater than everything); `0.0` was indistinguishable from a genuine zero. The new `NULL` is catchable via `WHERE measure IS NULL` and ignored by aggregates like `MAX`/`MIN`/`AVG`. `gbx_rst_pixelcount` is unchanged — an all-nodata band still returns `0` (a count of zero is meaningful). Relatedly, the lightweight `gbx_rst_isempty` is now all-nodata-aware: a dimensionally-valid raster whose every band is entirely NoData now returns `true`, matching the heavyweight tier. See [Raster Functions](./api/raster-functions). -``` - -- [ ] **Step 2: Update the reducer + isempty descriptions in raster-functions.mdx** - -In `docs/docs/api/raster-functions.mdx`, in each of the `### rst_avg`, `### rst_max`, `### rst_median`, `### rst_min` sections (near lines 164/289/303/345), add one sentence after the signature line describing empty-band behavior: - -```markdown -Returns `NULL` for a band with zero valid pixels (all NoData) on both tiers. -``` - -In the `### rst_isempty` section (near line 1339), add: - -```markdown -Returns `true` when the raster has no size **or** every band is entirely NoData. -``` - -Verify no internal vocabulary leaked: `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/` must print nothing. - -- [ ] **Step 3: Regenerate function-info.json** (Docker — needs the doc SQL example pipeline) - -Run: `bash scripts/commands/gbx-docs-function-info.sh` -Then confirm the reducer entries still have non-empty usage: -Run: `git diff --stat src/main/resources/com/databricks/labs/gbx/function-info.json` -Expected: the file regenerates cleanly (the SQL examples already exist for all four reducers + pixelcount + isempty; usage is non-empty). If the generator errors on empty usage, fix the upstream SQL example — do not hand-edit the JSON. - -- [ ] **Step 4: Verify docs render + no dead references** - -Run: `grep -n -iE "return NaN|returns NaN|-> NaN" docs/docs/api/raster-functions.mdx` -Expected: no reducer line still claims NaN. (There may be unrelated NaN mentions; scan that none refer to the four reducers.) - -- [ ] **Step 5: Commit** - -```bash -git add docs/docs/beta-release-notes.mdx docs/docs/api/raster-functions.mdx \ - src/main/resources/com/databricks/labs/gbx/function-info.json -git commit -m "docs(issue-59): document NULL-on-empty reducers + isempty change - -Beta release notes gain a behavior-change entry; raster-functions.mdx -reducer and isempty sections state the NULL-on-all-nodata contract; -function-info.json regenerated. - -Co-authored-by: Isaac" -``` - ---- - -## Task 7: Binding parity + affected-package verification before push - -**Files:** none (verification only) - -**Interfaces:** consumes the full changed tree; produces a green binding-parity + lint gate. - -- [ ] **Step 1: Run binding parity** (Docker) - -Run: `bash scripts/commands/gbx-test-bindings.sh --log issue59-bindings.log` -Expected: PASS — every reducer name resolves across Scala `override def name`, Python `functions.py`, and `function-info.json`. (No names changed, but the element-type/behavior change must not break the parity harness.) - -- [ ] **Step 2: Run the full affected light package suite** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/ --log issue59-pyrx.log` -Expected: PASS. Confirms no other pyrx test depended on the old NaN/dimension-only-isempty behavior. - -- [ ] **Step 3: Run scalastyle + python lint (matches CI)** - -Run: `bash scripts/commands/gbx-lint-scalastyle.sh` and `bash scripts/commands/gbx-lint-python.sh --check` -Expected: both clean. - -- [ ] **Step 4: Grep for any remaining callers/docs asserting the old behavior** - -Run: -```bash -grep -rn -iE "all-invalid.*NaN|return NaN|isnan\(accessors\.(avg|min|max|median)" python/geobrix docs/docs src/main -``` -Expected: no reducer still documents/asserts NaN. Fix any stragglers, then re-run the relevant suite. - -- [ ] **Step 5: Final commit if Step 4 changed anything, else proceed** - -```bash -git add -A && git commit -m "chore(issue-59): clean up residual NaN references" || echo "nothing to clean" -``` - ---- - -## Self-Review - -**Spec coverage:** -- Convention (NULL iff zero valid pixels, both tiers; pixelcount stays 0) → Tasks 1–4 + Task 6 docs. ✓ -- Light reducers `float("nan")`→`None` → Task 1. ✓ -- Light `isempty` all-nodata parity → Task 1. ✓ -- Heavy `Array[Double]`→`Array[java.lang.Double]` + `getValid_count == 0` guard, all four → Tasks 2–4. ✓ -- Heavy `RST_Median` missing null-check → Task 4 Step 3. ✓ -- Caller audit (NPE risk) → resolved during planning: `grep` found **no** internal callers of the four reducer `execute` methods, and `getMinMax` (shared with `RST_TileXYZ`) is left unchanged. The eval-path round-trip is covered by Task 4 Step 5. ✓ -- Tests: replace NaN test, zero-not-null trap (both tiers), seam-reconciliation regression, isempty parity → Tasks 1, 2, 3, 4, 5. ✓ -- Docs: beta release notes, docstrings, function-info regen, reducer pages, isempty note → Tasks 1 (docstrings) + 6. ✓ -- Binding parity + lint gate → Task 7. ✓ -- Deferred tessellation explicitly out of scope → Global Constraints. ✓ - -**Placeholder scan:** No TBD/TODO; every code step shows full code. ✓ - -**Type consistency:** All four heavy `execute` methods use `Array[java.lang.Double]` and the `val res: java.lang.Double = if (...) null else ` idiom uniformly; `RST_Median` keeps its `(ds, options)` signature. Light reducers return `List[Optional[float]]`; `pixelcount` stays `List[int]`. ✓ - -## Execution Handoff - -Plan complete and saved to `docs/superpowers/plans/2026-07-19-issue-59-nodata-reducer-null.md`. diff --git a/docs/superpowers/plans/2026-07-20-issue59-tessellation-nodata-standardization.md b/docs/superpowers/plans/2026-07-20-issue59-tessellation-nodata-standardization.md deleted file mode 100644 index 0c9246389..000000000 --- a/docs/superpowers/plans/2026-07-20-issue59-tessellation-nodata-standardization.md +++ /dev/null @@ -1,278 +0,0 @@ -# Issue #59 — Covering Tessellation All-Nodata Standardization (emit + NULL) — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Lock in and document that covering-mode H3 tessellation (`gbx_rst_h3_tessellate`) treats a cell that overlaps the raster bbox but clips to all-NoData identically on both tiers — the cell **is emitted** (a chip row is produced), and reducing that chip yields SQL `NULL` (already true after the issue #59 reducer fix). This is the "emit + NULL" (B1) end state, folded into the `issues/59` branch (no separate issue). - -**Architecture:** An empirical parity probe (already run, throwaway code removed) proved the two tiers **already emit the exact same covering-cell set** on interior-hole and swath-edge rasters (heavy_only=0, light_only=0 across 2 rasters × 2 resolutions), including all-nodata cells, with matching valid-pixel counts. Heavy's `polyfill(bbox.buffer)+intersects` and light's `polygon_to_cells_experimental(contain="overlap")` are functionally equivalent. **No product code change is needed.** The remaining work is (1) a both-tier regression test that locks the emit-all-nodata-cells + reduce-to-NULL contract so it can't silently regress, and (2) a doc sentence making the contract explicit for users. - -**Tech Stack:** Python 3.12 / rasterio / numpy / pytest (light, no Docker); Scala 2.13 / GDAL / ScalaTest in the `geobrix-dev` Docker container via `gbx:*` commands (heavy); Docusaurus MDX docs. - -## Global Constraints - -- **The emit + NULL contract (this plan's invariant):** covering mode emits a chip row for **every** H3 cell whose hexagon geometrically overlaps the raster bbox — including cells that clip to entirely NoData. It does **not** drop all-nodata-but-overlapping cells. Reducing an all-nodata chip (`gbx_rst_max/min/avg/median`) returns SQL `NULL` (from the issue #59 reducer fix already on this branch); `gbx_rst_pixelcount` returns `0`. A cell is dropped **only** on true geometric non-overlap. -- **No product code change.** Probe-confirmed the tiers already agree. If any task discovers a real behavioral divergence (a cell one tier emits and the other doesn't, or a disagreement on all-nodata → NULL), STOP and escalate — that contradicts the probe and changes scope. -- **Version 0.4.2 (in-flight beta).** Docs/release-notes target 0.4.2; do not bump the version banner. No function aliases/renames; no new functions. -- **User-facing docs voice** — no internal planning vocabulary under `docs/docs/` (no "wave N", no subagent/dispatch talk). QC judge enforces `internals-leak`. -- **Heavy work runs in Docker** via `gbx:*` commands; never `mvn` on the host. Long Scala/Maven runs get a progress line ~every 30s. -- **Heavy /vsimem test fixtures that warp/reproject must set a geotransform + projection** (`SetGeoTransform` + `SetProjection(EPSG:4326)`) — bare `Create`+`WriteRaster` rasters make GDAL warp/clip paths misbehave (learned in the reducer work). -- **gh account** — `gh auth switch --user mjohns-databricks` before any push/PR/comment to `databrickslabs/geobrix`. - ---- - -## Task 1: Light-tier parity test — covering emits all-nodata cells that reduce to NULL - -**Files:** -- Test: `python/geobrix/test/pyrx/test_core_tessellate.py` (append; imports already present: `h3`, `pytest`, `numpy as np`, `_serde`, `tessellate`, `make_geotiff_bytes`) - -**Interfaces:** -- Consumes: `tessellate.tessellate_h3(ds, resolution) -> [(cellid_int, gtiff_bytes)]` (covering mode); `accessors.maximum/pixelcount(ds) -> List[Optional[float]] / List[int]` (issue #59 behavior: empty band → `None` / `0`). `make_geotiff_bytes(width, height, epsg, nodata)` builds a north-up 4326 GTiff at origin (10.0, 50.0), 0.5° pixels. -- Produces: a regression test asserting covering emits an all-nodata-overlapping chip whose reducer is `None`. - -- [ ] **Step 1: Write the failing/guard test** - -Add to `python/geobrix/test/pyrx/test_core_tessellate.py`. The helper builds a raster with an interior NoData hole (a real value everywhere except a central block), tessellates in covering mode, and asserts that (a) at least one emitted chip is all-nodata (`pixelcount == 0`), (b) every such chip reduces to `None` (not NaN, not a number), and (c) chips with data still reduce to a real value. - -```python -def _raster_with_interior_hole(nodata=-9999.0): - """9x9 EPSG:4326 raster, all pixels = 42.0 except a 3x3 interior NoData block.""" - import numpy as np - from rasterio.io import MemoryFile - from rasterio.transform import from_origin - - data = np.full((9, 9), 42.0, dtype="float32") - data[3:6, 3:6] = nodata - profile = dict( - driver="GTiff", width=9, height=9, count=1, dtype="float32", - crs="EPSG:4326", transform=from_origin(10.0, 50.0, 0.05, 0.05), nodata=nodata, - ) - with MemoryFile() as mf: - with mf.open(**profile) as dst: - dst.write(data, 1) - return mf.read() - - -def test_covering_emits_all_nodata_cells_that_reduce_to_null(): - # Contract (issue #59, emit + NULL): covering mode emits a chip for every - # overlapping cell, INCLUDING cells that clip to entirely NoData; reducing - # such a chip yields None (SQL NULL), never NaN. Data-bearing cells keep a - # real value. A cell is dropped only on true geometric non-overlap. - from databricks.labs.gbx.pyrx.core import accessors - - src = _raster_with_interior_hole() - empty_seen = data_seen = 0 - with _serde.open_tile(src) as ds: - chips = tessellate.tessellate_h3(ds, 6) - assert chips, "covering must emit at least one chip" - for _cellid, raster in chips: - with _serde.open_tile(raster) as chip: - pc = accessors.pixelcount(chip)[0] - mx = accessors.maximum(chip)[0] - if pc == 0: - empty_seen += 1 - assert mx is None, f"all-nodata chip must reduce to None, got {mx!r}" - else: - data_seen += 1 - assert mx is not None and mx == 42.0 - # The hole must actually produce >=1 all-nodata cell, else the test proves nothing. - assert empty_seen > 0, "interior hole should yield >=1 all-nodata covering cell" - assert data_seen > 0 -``` - -- [ ] **Step 2: Run the test** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_core_tessellate.py -k "all_nodata_cells_that_reduce_to_null" -v` -Expected: PASS. (Tasks 1–5 of the reducer fix already make `accessors.maximum` return `None` for an empty band, so this passes now — it is a regression guard. If `empty_seen == 0`, adjust the hole size / resolution until the interior block yields at least one fully-NoData covering cell; res 6 on a 9×9 @ 0.05° raster with a 3×3 hole should. If `mx` comes back as `NaN`, the reducer fix regressed — STOP and escalate.) - -- [ ] **Step 3: Run the full tessellate test file to confirm no regression** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_core_tessellate.py -v` -Expected: all PASS (including the pre-existing `test_tessellate_drops_zero_coverage_fringe_cells`, which asserts the complementary invariant — no *zero-geometric-coverage* fringe cells; the two are consistent: overlap-but-nodata is emitted, no-overlap is not). - -- [ ] **Step 4: Lint** - -Run: `bash scripts/commands/gbx-lint-python.sh --check` (run `--fix` on host first if it flags formatting, then re-check). -Expected: clean. - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/test/pyrx/test_core_tessellate.py -git commit -m "test(pyrx): covering emits all-nodata cells that reduce to NULL (#59) - -Locks the emit+NULL contract on the light tier: covering-mode tessellation -emits a chip for every overlapping H3 cell including all-nodata ones, and -reducing such a chip yields None (not NaN). Guards against a silent -regression to dropping empty cells or to the pre-#59 NaN sentinel. - -Co-authored-by: Isaac" -``` - ---- - -## Task 2: Heavy-tier parity test — covering emits all-nodata cells with a null reducer - -**Files:** -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/operations/RasterTessellateTest.scala` (append; reuses `beforeAll` GDAL registration and the `validPixelCount(chip)` helper already in the suite) - -**Interfaces:** -- Consumes: `RasterTessellate.tessellateH3Iter(ds, options, resolution, mode) -> Iterator[(Long cellId, Dataset chip, Map)]` (covering default); `RST_Max.execute(chip): Array[java.lang.Double]` (issue #59: empty band → `null` element); the suite's existing `validPixelCount(chip: Dataset): Long`. -- Produces: a heavy regression test mirroring Task 1 — an all-nodata-overlapping chip is emitted and `RST_Max` on it is `null`. - -- [ ] **Step 1: Add a georeferenced interior-hole raster helper + the test** - -Append to `RasterTessellateTest.scala` (inside the class). The helper builds a `/vsimem` GTiff with a geotransform + EPSG:4326 projection (required for the covering clip/warp path) and an interior NoData block. `RST_Max` must be imported. - -```scala - /** 9x9 Float32 /vsimem raster (EPSG:4326, georeferenced) = 42.0 except a 3x3 interior NoData block. */ - private def interiorHoleDs(): Dataset = { - val path = s"/vsimem/tess_hole_${java.util.UUID.randomUUID().toString.replace("-", "")}.tif" - val drv = gdal.GetDriverByName("GTiff") - val d = drv.Create(path, 9, 9, 1, org.gdal.gdalconst.gdalconstConstants.GDT_Float32) - d.SetGeoTransform(Array(10.0, 0.05, 0.0, 50.0, 0.0, -0.05)) - val srs = new org.gdal.osr.SpatialReference() - srs.ImportFromEPSG(4326) - d.SetProjection(srs.ExportToWkt()) - srs.delete() - val band = d.GetRasterBand(1) - band.SetNoDataValue(-9999.0) - val buf = Array.fill[Double](81)(42.0) - for (r <- 3 to 5; c <- 3 to 5) buf(r * 9 + c) = -9999.0 // interior 3x3 hole - band.WriteRaster(0, 0, 9, 9, buf) - band.FlushCache() - d.FlushCache() - band.delete() - d - } - - test("covering emits all-nodata cells whose reducer is null (issue #59 emit+NULL)") { - val iter = RasterTessellate.tessellateH3Iter(interiorHoleDs(), Map.empty, 6, "covering") - var emptySeen = 0 - var dataSeen = 0 - try { - iter.foreach { case (_, chip, _) => - val vc = validPixelCount(chip) - val mx = RST_Max.execute(chip).headOption.orNull - if (vc == 0L) { emptySeen += 1; mx shouldBe null } - else { dataSeen += 1; mx should not be null } - RasterDriver.releaseDataset(chip) - } - } finally iter match { - case ac: AutoCloseable => ac.close() - case _ => - } - emptySeen should be > 0 // the hole must yield >=1 all-nodata covering cell - dataSeen should be > 0 - } -``` - -(If `RST_Max` isn't imported in this suite, add `import com.databricks.labs.gbx.rasterx.expressions.accessors.RST_Max`. `RasterDriver` is already imported.) - -- [ ] **Step 2: Run the test in Docker** - -Run: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.operations.RasterTessellateTest' --log tess-null.log` -Expected: PASS, including the new test and all pre-existing covering/centroid tests. (Compile + run takes a few minutes; be patient.) If `emptySeen == 0`, the hole didn't produce a fully-NoData covering cell at res 6 — bump resolution to 7 (finer cells → more likely an interior cell lands entirely in the hole) and re-run. If the empty chip's `RST_Max` is not `null`, the reducer fix regressed — STOP and escalate. - -- [ ] **Step 3: Scalastyle** - -Run: `bash scripts/commands/gbx-lint-scalastyle.sh` -Expected: 0 errors. - -- [ ] **Step 4: Commit** - -```bash -git add src/test/scala/com/databricks/labs/gbx/rasterx/operations/RasterTessellateTest.scala -git commit -m "test(rasterx): covering emits all-nodata cells with null reducer (#59) - -Heavy-tier counterpart to the pyrx test: covering-mode tessellation emits -a chip for every overlapping H3 cell including all-nodata ones, and -RST_Max on such a chip is a null element. Locks the emit+NULL contract on -the heavy tier so it stays cross-tier consistent. - -Co-authored-by: Isaac" -``` - ---- - -## Task 3: Document the covering all-nodata contract - -**Files:** -- Modify: `docs/docs/api/h3-raster-tessellation.mdx` (the covering-mode prose, around the `**covering:**` paragraph at line ~52 and the cross-tier note at ~169) - -**Interfaces:** none (docs only). Consumes nothing; produces user-facing contract text. - -- [ ] **Step 1: Add the emit + NULL sentence to the covering description** - -In `docs/docs/api/h3-raster-tessellation.mdx`, in the `**covering:**` explanation paragraph (~line 52, which currently ends with the border-cell union note), append: - -```markdown -A cell that overlaps the tile but clips to entirely NoData is still emitted as a chip (a cell is omitted only when its hexagon does not geometrically overlap the tile at all). Such an all-NoData chip has a valid-pixel count of `0`, and the value reducers (`gbx_rst_max`, `gbx_rst_min`, `gbx_rst_avg`, `gbx_rst_median`) return SQL `NULL` for it on both tiers — filter these with `WHERE measure IS NULL`. This lets a downstream query distinguish *missing data* (a chip is present but its measure is `NULL`) from *outside the coverage area* (no chip for that cell). -``` - -- [ ] **Step 2: Reinforce the cross-tier parity note** - -In the cross-tier mechanism note (~line 169, which already states the covering *set* is identical across tiers and "verified by the per-mode parity tests"), extend it so the all-nodata behavior is explicitly covered: - -```markdown -Both tiers emit the same covering set — including cells that clip to all-NoData — and both reduce an all-NoData chip to SQL `NULL`; this is verified by the per-mode parity tests and the all-nodata regression tests on each tier. -``` - -- [ ] **Step 3: Verify no internal-vocab leak** - -Run: `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/` -Expected: prints nothing. - -- [ ] **Step 4: Commit** - -```bash -git add docs/docs/api/h3-raster-tessellation.mdx -git commit -m "docs(issue-59): document covering emit+NULL for all-nodata cells - -State the covering-mode contract explicitly: an overlapping-but-all-NoData -cell is emitted (not dropped) and its reducers return SQL NULL on both -tiers; a cell is omitted only on true geometric non-overlap. Clarifies the -missing-data vs outside-coverage distinction for downstream filtering. - -Co-authored-by: Isaac" -``` - ---- - -## Task 4: Verification gate - -**Files:** none (verification only). - -- [ ] **Step 1: Run both affected test files** - -Light: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_core_tessellate.py -v` -Heavy (Docker): `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.operations.RasterTessellateTest' --log tess-final.log` -Expected: both green. - -- [ ] **Step 2: Lint both tiers** - -Run: `bash scripts/commands/gbx-lint-python.sh --check` and `bash scripts/commands/gbx-lint-scalastyle.sh` -Expected: both clean. - -- [ ] **Step 3: Confirm no product code changed in this plan** - -Run: `git diff --stat ..HEAD -- src/main python/geobrix/src` -Expected: **empty** — this plan adds only tests and docs. If any `src/main` or `pyrx/src` file changed, that contradicts the probe (which proved no code change is needed); review and escalate before proceeding. - ---- - -## Self-Review - -**Spec/decision coverage:** -- B1 "emit + NULL" end state, no separate issue, folded into issues/59 → whole plan. ✓ -- Probe finding (sets already match, no code change) → Global Constraints + Task 4 Step 3 guard. ✓ -- Both-tier parity test (emit all-nodata cell + reduce to NULL) → Task 1 (light) + Task 2 (heavy). ✓ -- Doc sentence on the covering contract + missing-vs-outside distinction → Task 3. ✓ -- Depends on the issue #59 reducer fix already on this branch (empty → None/null) → Tasks 1/2 assert it end-to-end; escalate if it regressed. ✓ - -**Placeholder scan:** none — every test/doc step has full code/prose. ✓ - -**Type consistency:** light asserts `accessors.maximum(...)[0] is None` / `== 42.0`, `pixelcount(...)[0] == 0`; heavy asserts `RST_Max.execute(chip).headOption.orNull shouldBe null` (boxed `java.lang.Double` from the #59 change) and uses the suite's existing `validPixelCount`. Consistent with the reducer work already on the branch. ✓ - -## Execution Handoff - -Plan complete and saved to `docs/superpowers/plans/2026-07-20-issue59-tessellation-nodata-standardization.md`. diff --git a/docs/superpowers/plans/2026-07-24-raster-bng-quadbin-h3-parity-phase1-heavy.md b/docs/superpowers/plans/2026-07-24-raster-bng-quadbin-h3-parity-phase1-heavy.md deleted file mode 100644 index 3f031a235..000000000 --- a/docs/superpowers/plans/2026-07-24-raster-bng-quadbin-h3-parity-phase1-heavy.md +++ /dev/null @@ -1,1093 +0,0 @@ -# Raster BNG + Quadbin (H3 parity) — Phase 1 (Heavy/Scala) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add the 9 heavy-tier (Scala/GDAL) RasterX functions that bring the quadbin and BNG discrete-grid families to full parity with the H3 raster surface. - -**Architecture:** Approach B from the spec — parallel per-grid expression families that clone the existing H3/quadbin shapes, sharing the raster→grid hot loop and reusing `Long`-keyed accumulators for all grids. BNG differs only by (a) reprojecting its input raster to EPSG:27700 before pixel→cell math and (b) rendering `String` cell ids at the output boundary via `BNG.format`. This plan is Phase 1 only (heavy tier); light-tier phases are separate plans (Phase 3 BNG is gated on pygx BNG phase 2). - -**Tech Stack:** Scala 2.13.16, Spark 4.0.0, Java 17, GDAL Java bindings. All build/test runs inside the `geobrix-dev` Docker container (`gbx:docker:*`, `gbx:test:scala`). - -**Spec:** `docs/superpowers/specs/2026-07-24-raster-bng-quadbin-h3-parity-design.md` - -**Closes:** [databrickslabs/geobrix#49](https://github.com/databrickslabs/geobrix/issues/49) — customer request for Mosaic-style BNG tessellate on rasters (CV image tiling). The `gbx_rst_bng_tessellate` deliverable (Tasks 3–4) is that function; its #49 acceptance criteria are in spec §1.1 and repeated in the Task 3/4 briefs. - -## Global Constraints - -- **No aliases.** One canonical name per function; fix upstream, never add an alias. -- **Cross-language naming:** SQL name `gbx_`; Scala `override def name` is the SQL literal. Registered names go in `docs/tests-function-info/registered_functions.txt`. -- **Binding parity enforced:** every new function must appear as (1) a Scala `override def name` literal, (2) a Python `functions.py` binding, (3) a `function-info.json` key. `gbx:test:bindings` fails otherwise. (Python bindings + function-info are Tasks 8–9.) -- **BNG resolution contract:** integer indices ±1..±6 (1=100km … 6=1m; negatives=quadrants) or string keys from `BNG.resolutionMap` (`"1km"`, `"100m"`, …), resolved via `BNG.getResolution(res: Any): Int`. Never metres-as-Int. -- **CRS contract:** quadbin/H3 assume raster is EPSG:4326 (caller reprojects upstream — no change). BNG raster fns auto-reproject the input to EPSG:27700 internally via `RasterProject.project`, using **nearest-neighbour** resampling; pixels outside the GB extent are silently dropped. -- **Empty-cell / NoData semantics (spec §2.6, SAFETY-CRITICAL):** - - `rastertogrid`: a cell is emitted only when ≥1 **valid** pixel lands in it (`maskBuf(idx) != 0`). Never emit a zero-valid-pixel cell; never substitute `0.0`/`NaN`/sentinel/NULL for an absent cell. - - `rasterize_agg`: the output raster MUST be built through `VectorRasterBridge.buildEmptyRaster` (which calls `SetNoDataValue(-9999.0)` then `Fill`), so `-9999.0` is registered band NoData — never a bare pixel value. -- **GDAL thread-safety:** register GDAL/OGR only via `GDALManager` guards (`RST_ExpressionUtil.init` already does this in the eval paths). Never raw `gdal.AllRegister()` per task. -- **GDAL resource management:** release every `Dataset`/`Band` via `RasterDriver.releaseDataset(ds)` in `try/finally` (the shared eval helpers already do this). -- **Docker:** dispatch long Scala builds/tests as a Task subagent; never run inline. Give a progress line ~every 30s while a suite runs. - -## File Structure - -**Create (Scala main):** -- `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_BNG_RasterToGrid.scala` — shared BNG raster→grid object (String-output, Long-keyed accumulator, 27700 warp). Mirrors `RST_Quadbin_RasterToGrid`. -- `.../grid/RST_BNG_RasterToGridAvg.scala`, `…Count.scala`, `…Max.scala`, `…Min.scala`, `…Median.scala` — 5 reducer expressions. -- `.../generators/RST_Quadbin_Tessellate.scala`, `.../generators/RST_BNG_Tessellate.scala` — 2 tessellate generators. -- `.../agg/RST_Quadbin_RasterizeAgg.scala`, `.../agg/RST_BNG_RasterizeAgg.scala` — 2 UDAFs. - -**Modify (Scala main):** -- `.../rasterx/operations/RasterTessellate.scala` — add `tessellateQuadbinIter` + `tessellateBngIter` (+ private covering/centroid helpers + a `getTileQuadbin`/`getTileBng`). -- `.../rasterx/functions.scala` — `rd.register(...)` × 9 and Scala `functions` column wrappers × 9. -- `docs/tests-function-info/registered_functions.txt` — +9 names. - -**Modify (Python bindings):** -- `python/geobrix/src/databricks/labs/gbx/rasterx/functions.py` — +9 `call_function` wrappers. - -**Create/Modify (tests):** -- `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_BNG_RasterToGridTest.scala` -- `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_Quadbin_TessellateTest.scala` (+ BNG tessellate cases) -- `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/agg/RST_GridRasterizeAggTest.scala` -- Doc-test SQL examples: `docs/tests/python/api/rasterx_functions_sql.py` (feeds function-info). - -**Modify (docs):** -- `docs/docs/api/raster-functions.mdx`, `docs/docs/api/execution-tiers.mdx`, `docs/docs/api/performance.mdx`, `docs/docs/beta-release-notes.mdx`, `README.md` (badges). - ---- - -## Task 1: BNG raster→grid shared object - -**Files:** -- Create: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_BNG_RasterToGrid.scala` -- Reference (read, do not modify): `.../grid/RST_Quadbin_RasterToGrid.scala`, `.../gridx/grid/BNG.scala`, `.../rasterx/operations/RasterProject.scala` - -**Interfaces:** -- Consumes: `BNG.getResolution(res: Any): Int`, `BNG.pointToCellID(e: Double, n: Double, res: Int): Long`, `BNG.format(id: Long): String`; `RasterProject.project(ds, options, dstSR): (Dataset, Map[String,String])`. -- Produces: `RST_BNG_RasterToGrid.execute[T](ds: Dataset, resolution: Int, fAgg: mutable.ArrayBuffer[Double] => T): Array[Array[(String, T)]]` and `RST_BNG_RasterToGrid.eval[T](row, resolution, conf, rdt, execute): ArrayData` — note the cell id type is **String** (vs quadbin's Long). - -- [ ] **Step 1: Write the failing test** - -Create `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_BNG_RasterToGridTest.scala`: - -```scala -package com.databricks.labs.gbx.rasterx.expressions.grid - -import com.databricks.labs.gbx.rasterx.gdal.RasterDriver -import com.databricks.labs.gbx.rasterx.test.RasterXTestBase // if a shared base exists; else mixin used by RST_Quadbin_RasterToGridTest -import org.gdal.gdal.gdal -import org.scalatest.funsuite.AnyFunSuite -import scala.collection.mutable.ArrayBuffer - -class RST_BNG_RasterToGridTest extends AnyFunSuite { - - // A 2x2 EPSG:27700 raster centred on London (530000,180000), 100m pixels, - // all pixels valid, values 1,2,3,4. Built in-memory via MEM driver. - private def londonDs = { - gdal.AllRegister() // test-only; production uses GDALManager - val drv = gdal.GetDriverByName("MEM") - val ds = drv.Create("", 2, 2, 1, org.gdal.gdalconst.gdalconstConstants.GDT_Float64) - ds.SetGeoTransform(Array(530000.0, 100.0, 0.0, 180200.0, 0.0, -100.0)) - val sr = new org.gdal.osr.SpatialReference(); sr.ImportFromEPSG(27700) - ds.SetProjection(sr.ExportToWkt()) - ds.GetRasterBand(1).WriteRaster(0, 0, 2, 2, Array(1.0, 2.0, 3.0, 4.0)) - ds.FlushCache(); ds - } - - test("bng rastertogrid: emits String cell ids and averages valid pixels") { - val ds = londonDs - val meanF = (v: ArrayBuffer[Double]) => v.sum / v.length - val out: Array[Array[(String, Double)]] = - RST_BNG_RasterToGrid.execute(ds, resolution = 3, fAgg = meanF) // 3 = 1km - RasterDriver.releaseDataset(ds) - val cells = out.flatten - assert(cells.nonEmpty) - assert(cells.forall(_._1.matches("^[A-Z]{2}\\d*$"))) // BNG string form - // all four pixels fall in the same 1km cell here -> mean 2.5 - assert(cells.map(_._1).distinct.length == 1) - assert(math.abs(cells.head._2 - 2.5) < 1e-9) - } - - test("bng rastertogrid: zero-valid-pixel cell is never emitted (spec 2.6)") { - // Mask all pixels nodata -> no cells at all. - val ds = londonDs - ds.GetRasterBand(1).SetNoDataValue(1.0) - ds.GetRasterBand(1).Fill(1.0) - val meanF = (v: ArrayBuffer[Double]) => v.sum / v.length - val out = RST_BNG_RasterToGrid.execute(ds, 3, meanF) - RasterDriver.releaseDataset(ds) - assert(out.flatten.isEmpty, "all-nodata raster must yield no cells") - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Dispatch a Task subagent to run in Docker: -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.grid.RST_BNG_RasterToGridTest' --log bng-r2g.log -``` -Expected: FAIL — `RST_BNG_RasterToGrid` not found (does not compile / unresolved). - -- [ ] **Step 3: Write the implementation** - -Create `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_BNG_RasterToGrid.scala`. This mirrors `RST_Quadbin_RasterToGrid` but: keys the accumulator on `Long` (BNG internal id), reprojects the dataset to 27700 up front, computes eastings/northings under the **warped** geotransform, and renders `BNG.format(cellId): String` into the output tuple. - -```scala -package com.databricks.labs.gbx.rasterx.expressions.grid - -import com.databricks.labs.gbx.expressions.ExpressionConfig -import com.databricks.labs.gbx.gridx.grid.BNG -import com.databricks.labs.gbx.rasterx.gdal.RasterDriver -import com.databricks.labs.gbx.rasterx.operations.RasterProject -import com.databricks.labs.gbx.rasterx.util.{RST_ExpressionUtil, RasterSerializationUtil} -import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.util.ArrayData -import org.apache.spark.sql.types.DataType -import org.apache.spark.unsafe.types.UTF8String -import org.gdal.gdal.Dataset -import org.gdal.osr.SpatialReference - -import scala.collection.mutable - -/** Shared helper for `RST_BNG_RasterToGrid*` expressions — mirrors `RST_Quadbin_RasterToGrid` - * but delegates per-pixel cell math to [[BNG.pointToCellID]] (EPSG:27700 eastings/northings). - * - * Unlike the H3/quadbin families (whose input contract is EPSG:4326 lon/lat), BNG has no lon/lat - * input path, so the raster is reprojected to EPSG:27700 up front via [[RasterProject]] using - * nearest-neighbour resampling. Cell ids are `Long` internally and rendered to the user-facing - * BNG `String` via [[BNG.format]] at the output boundary. Pixels outside the GB extent are dropped. - */ -object RST_BNG_RasterToGrid { - - /** Compute the BNG cell id for the centroid of pixel (x, y) under geotransform `gt` (EPSG:27700). */ - def cellPixel(gt: Array[Double], x: Int, y: Int, resolution: Int): Long = { - val offset = 0.5 - val xOffset = offset + x - val yOffset = offset + y - val eGeo = gt(0) + xOffset * gt(1) + yOffset * gt(2) - val nGeo = gt(3) + xOffset * gt(4) + yOffset * gt(5) - BNG.pointToCellID(eGeo, nGeo, resolution) - } - - def execute[T]( - ds: Dataset, - resolution: Int, - fAgg: mutable.ArrayBuffer[Double] => T - ): Array[Array[(String, T)]] = { - require( - BNG.resolutions.contains(resolution), - s"raster→bng: resolution must be one of ${BNG.resolutions.toSeq.sorted.mkString(", ")}; got $resolution" - ) - - // Reproject to EPSG:27700 (nearest-neighbour) unless already there. - val dstSR = new SpatialReference(); dstSR.ImportFromEPSG(27700) - val srcWkt = ds.GetProjection() - val (workDs, reprojected) = - if (srcWkt != null && srcWkt.nonEmpty && { - val s = new SpatialReference(); s.ImportFromWkt(Array(srcWkt)); val same = s.IsSame(dstSR) == 1; s.delete(); same - }) (ds, false) - else { - val (p, _) = RasterProject.project(ds, Map("resampling" -> "near"), dstSR) - (p, true) - } - dstSR.delete() - - try { - val gt = workDs.GetGeoTransform - val xSize = workDs.getRasterXSize - val ySize = workDs.getRasterYSize - val nPix = xSize * ySize - val bands = workDs.getRasterCount - - val bandBuf = new Array[Double](nPix) - val maskBuf = new Array[Byte](nPix) - - (1 to bands).iterator.map { bi => - val b = workDs.GetRasterBand(bi) - val m = b.GetMaskBand() - b.ReadRaster(0, 0, xSize, ySize, bandBuf) - m.ReadRaster(0, 0, xSize, ySize, maskBuf) - - var valid = 0; var i = 0 - while (i < nPix) { if (maskBuf(i) != 0) valid += 1; i += 1 } - - val acc = new mutable.LongMap[mutable.ArrayBuffer[Double]](valid) - var y = 0; var idx = 0 - while (y < ySize) { - var x = 0 - while (x < xSize) { - if (maskBuf(idx) != 0) { - val cell = cellPixel(gt, x, y, resolution) // Long id - val buf = acc.getOrElseUpdate(cell, new mutable.ArrayBuffer) - buf += bandBuf(idx) - } - idx += 1; x += 1 - } - y += 1 - } - - val out = new Array[(String, T)](acc.size) - var j = 0 - acc.foreach { case (cell, buf) => out(j) = (BNG.format(cell), fAgg(buf)); j += 1 } - out - }.toArray - } finally { - if (reprojected) RasterDriver.releaseDataset(workDs) - } - } - - def eval[T]( - row: InternalRow, - resolution: Int, - conf: UTF8String, - rdt: DataType, - execute: (Dataset, Int) => Array[Array[(String, T)]] - ): ArrayData = { - val exprConf = ExpressionConfig.fromB64(conf.toString) - RST_ExpressionUtil.init(exprConf) - val ds = RasterSerializationUtil.rowToDS(row, rdt) - val result = execute(ds, resolution) - RasterDriver.releaseDataset(ds) - ArrayData.toArrayData( - result.map(band => - ArrayData.toArrayData( - band.map { case (cellId, measure) => - InternalRow.fromSeq(Seq(UTF8String.fromString(cellId), measure)) - } - ) - ) - ) - } -} -``` - -Note: verify `RasterProject.project`'s options key for resampling against its source (Step relies on it honouring `"resampling" -> "near"`; if the key differs, adapt to the actual signature — read `RasterProject.scala` and `RST_ToWebMercator.scala` which calls it). If `RasterProject` takes a gdalwarp command string instead, build `s"gdalwarp -t_srs EPSG:27700 -r near"` as `RST_ToWebMercator` does. - -- [ ] **Step 4: Run the test to verify it passes** - -Dispatch Task subagent: -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.grid.RST_BNG_RasterToGridTest' --log bng-r2g.log -``` -Expected: PASS (both tests). - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_BNG_RasterToGrid.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_BNG_RasterToGridTest.scala -git commit -m "feat(rasterx): BNG raster-to-grid shared object (27700 warp, String cell ids)" -``` - ---- - -## Task 2: BNG rastertogrid reducer expressions (×5) - -**Files:** -- Create: `.../grid/RST_BNG_RasterToGridAvg.scala`, `…Count.scala`, `…Max.scala`, `…Min.scala`, `…Median.scala` -- Reference: `.../grid/RST_Quadbin_RasterToGridAvg.scala` (and its Count/Max/Min/Median siblings) - -**Interfaces:** -- Consumes: `RST_BNG_RasterToGrid.execute`/`.eval` (Task 1). -- Produces: 5 `WithExpressionInfo` objects with `name` = `gbx_rst_bng_rastertogrid{avg,count,max,min,median}`; each `dataType = ArrayType(ArrayType(StructType(cellID: StringType, measure: )))`. - -- [ ] **Step 1: Write the failing test** - -Add to `RST_BNG_RasterToGridTest.scala`: - -```scala -test("bng rastertogrid reducers: min/max/count/median on the london cell") { - val ds = londonDs - import scala.collection.mutable.ArrayBuffer - val minF = (v: ArrayBuffer[Double]) => v.min - val maxF = (v: ArrayBuffer[Double]) => v.max - val cntF = (v: ArrayBuffer[Double]) => v.length - val medF = (v: ArrayBuffer[Double]) => { val s = v.sorted; val m = s.length/2; if (s.length%2==0) (s(m-1)+s(m))/2.0 else s(m) } - assert(RST_BNG_RasterToGrid.execute(ds, 3, minF).flatten.head._2 == 1.0) - assert(RST_BNG_RasterToGrid.execute(ds, 3, maxF).flatten.head._2 == 4.0) - assert(RST_BNG_RasterToGrid.execute(ds, 3, cntF).flatten.head._2 == 4) - assert(math.abs(RST_BNG_RasterToGrid.execute(ds, 3, medF).flatten.head._2 - 2.5) < 1e-9) - RasterDriver.releaseDataset(ds) -} -``` - -- [ ] **Step 2: Run to verify it fails** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.grid.RST_BNG_RasterToGridTest' --log bng-r2g.log -``` -Expected: PASS actually (this test only exercises Task-1 `execute`). Its purpose is to lock reducer math; the reducer *expressions* are covered by the integration test in Task 7. If you prefer a failing gate here, assert on the expression objects' `name` instead: -```scala -test("bng reducer names are canonical") { - assert(RST_BNG_RasterToGridAvg.name == "gbx_rst_bng_rastertogridavg") - assert(RST_BNG_RasterToGridMedian.name == "gbx_rst_bng_rastertogridmedian") -} -``` -Run: expected FAIL (objects undefined). - -- [ ] **Step 3: Write the 5 reducer files** - -`RST_BNG_RasterToGridAvg.scala` (clone the quadbin Avg verbatim, swapping `Quadbin`→`BNG`, `LongType`→`StringType` for `cellID`, and `Long` resolution handling; keep the `execute` reducer lambdas identical): - -```scala -package com.databricks.labs.gbx.rasterx.expressions.grid - -import com.databricks.labs.gbx.expressions.{ExpressionConfigExpr, InvokedExpression, WithExpressionInfo} -import com.databricks.labs.gbx.gridx.grid.BNG -import com.databricks.labs.gbx.rasterx.util.{RST_ErrorHandler, RST_ExpressionUtil} -import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.analysis.FunctionRegistry.FunctionBuilder -import org.apache.spark.sql.catalyst.expressions.Expression -import org.apache.spark.sql.catalyst.util.ArrayData -import org.apache.spark.sql.types._ -import org.apache.spark.unsafe.types.UTF8String -import org.gdal.gdal.Dataset - -import scala.collection.mutable.ArrayBuffer - -/** Returns the average raster value within each BNG grid cell. */ -case class RST_BNG_RasterToGridAvg(tileExpr: Expression, resolution: Expression) extends InvokedExpression { - private def rasterType = RST_ExpressionUtil.rasterType(tileExpr) - override def children: Seq[Expression] = Seq(tileExpr, resolution, ExpressionConfigExpr()) - override def dataType: DataType = - ArrayType(ArrayType(StructType(Seq(StructField("cellID", StringType), StructField("measure", DoubleType))))) - override def nullable: Boolean = true - override def prettyName: String = RST_BNG_RasterToGridAvg.name - override def replacement: Expression = rstInvoke(RST_BNG_RasterToGridAvg, rasterType) - override protected def withNewChildrenInternal(nc: IndexedSeq[Expression]): Expression = copy(nc(0), nc(1)) -} - -object RST_BNG_RasterToGridAvg extends WithExpressionInfo { - def evalPath(row: InternalRow, resolution: Int, conf: UTF8String): ArrayData = doInvoke(row, resolution, conf, StringType) - def evalBinary(row: InternalRow, resolution: Int, conf: UTF8String): ArrayData = doInvoke(row, resolution, conf, BinaryType) - def evalPath(row: InternalRow, resolution: Long, conf: UTF8String): ArrayData = evalPath(row, resolution.toInt, conf) - def evalBinary(row: InternalRow, resolution: Long, conf: UTF8String): ArrayData = evalBinary(row, resolution.toInt, conf) - // BNG string-key resolution ("1km" etc.) — PySpark may send a UTF8String. - def evalPath(row: InternalRow, resolution: UTF8String, conf: UTF8String): ArrayData = evalPath(row, BNG.getResolution(resolution), conf) - def evalBinary(row: InternalRow, resolution: UTF8String, conf: UTF8String): ArrayData = evalBinary(row, BNG.getResolution(resolution), conf) - - private def doInvoke(row: InternalRow, resolution: Int, conf: UTF8String, rdt: DataType): ArrayData = - Option(RST_ErrorHandler.safeEval(() => RST_BNG_RasterToGrid.eval[Double](row, resolution, conf, rdt, this.execute), row, rdt, conf)) - .map(_.asInstanceOf[ArrayData]).orNull - - def execute(ds: Dataset, resolution: Int): Array[Array[(String, Double)]] = { - val meanF = (values: ArrayBuffer[Double]) => values.sum / values.length - RST_BNG_RasterToGrid.execute(ds, resolution, meanF) - } - override def name: String = "gbx_rst_bng_rastertogridavg" - override def builder(): FunctionBuilder = (c: Seq[Expression]) => new RST_BNG_RasterToGridAvg(c(0), c(1)) -} -``` - -Create the other four by the same clone, changing only: class/object name, `name` literal, the `execute` reducer lambda, and (for Count) the type param `[Int]` + `measure` field type `IntegerType`: -- `…Count`: `dataType` struct `measure: IntegerType`; `execute` returns `Array[Array[(String, Int)]]` with `(values) => values.length`, `eval[Int]`, `doInvoke` passes `eval[Int]`. Match `RST_Quadbin_RasterToGridCount` exactly for the Int plumbing. -- `…Max`: `(values) => values.max`. `…Min`: `(values) => values.min`. -- `…Median`: the median lambda from the test in Step 1. - -- [ ] **Step 4: Run to verify it passes** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.grid.RST_BNG_RasterToGridTest' --log bng-r2g.log -``` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_BNG_RasterToGrid{Avg,Count,Max,Min,Median}.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_BNG_RasterToGridTest.scala -git commit -m "feat(rasterx): BNG rastertogrid reducers (avg/count/max/min/median)" -``` - ---- - -## Task 3: Quadbin + BNG tessellate iterators in RasterTessellate - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/operations/RasterTessellate.scala` -- Reference: same file's `tessellateH3Iter`, `getTile`, `tessellateH3CoveringIter`, `tessellateH3CentroidIter`; `Quadbin.cellToBoundary`/bbox helpers and `BNG.cellIdToGeometry` for cell polygons. - -**Interfaces:** -- Consumes: `Quadbin` cell→geometry (verify method name in `Quadbin.scala`: bbox is `(lonMin,latMin,lonMax,latMax)`; build a JTS polygon from it), `BNG.cellIdToGeometry(cell: Long): Geometry`, `BNG.pointToCellID`, `BNG.format`, `Quadbin.pointToCell`. -- Produces: `RasterTessellate.tessellateQuadbinIter(ds, options, resolution, mode): Iterator[(Long, Dataset, Map[String,String])]` and `tessellateBngIter(ds, options, resolution, mode): Iterator[(String, Dataset, Map[String,String])]`. - -- [ ] **Step 1: Write the failing test** - -Create `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_Quadbin_TessellateTest.scala`: - -```scala -package com.databricks.labs.gbx.rasterx.expressions.grid - -import com.databricks.labs.gbx.rasterx.gdal.RasterDriver -import com.databricks.labs.gbx.rasterx.operations.RasterTessellate -import org.gdal.gdal.gdal -import org.scalatest.funsuite.AnyFunSuite - -class RST_Quadbin_TessellateTest extends AnyFunSuite { - - private def wgs84Ds = { // 4x4 EPSG:4326 raster over a small lon/lat box, all valid - gdal.AllRegister() - val drv = gdal.GetDriverByName("MEM") - val ds = drv.Create("", 4, 4, 1, org.gdal.gdalconst.gdalconstConstants.GDT_Float64) - ds.SetGeoTransform(Array(-0.2, 0.1, 0.0, 51.6, 0.0, -0.1)) - val sr = new org.gdal.osr.SpatialReference(); sr.ImportFromEPSG(4326) - ds.SetProjection(sr.ExportToWkt()) - ds.GetRasterBand(1).Fill(7.0) - ds.FlushCache(); ds - } - - test("quadbin tessellate covering: yields >=1 chip, each tagged with its cell id") { - val ds = wgs84Ds - val it = RasterTessellate.tessellateQuadbinIter(ds, Map.empty, resolution = 10, mode = "covering") - val chips = it.toList - assert(chips.nonEmpty) - chips.foreach { case (cell, d, _) => assert(cell != 0L); RasterDriver.releaseDataset(d) } - RasterDriver.releaseDataset(ds) - } - - test("quadbin tessellate rejects unknown mode") { - val ds = wgs84Ds - val ex = intercept[IllegalArgumentException] { - RasterTessellate.tessellateQuadbinIter(ds, Map.empty, 10, "nonsense").toList - } - assert(ex.getMessage.toLowerCase.contains("mode")) - RasterDriver.releaseDataset(ds) - } -} -``` - -- [ ] **Step 2: Run to verify it fails** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.grid.RST_Quadbin_TessellateTest' --log tess.log -``` -Expected: FAIL — `tessellateQuadbinIter` not a member of `RasterTessellate`. - -- [ ] **Step 3: Implement the iterators** - -In `RasterTessellate.scala`, add (mirroring the H3 trio). Reuse the existing covering/centroid structure; the only per-grid differences are (a) the cell-set enumeration for a bbox and (b) the cell→geometry function. Extract a private generic `getTileGeneric[C](ds, options, cell: C, cellGeom: Geometry, tagId: String)` if the H3 `getTile` body is easy to parameterize; otherwise add `getTileQuadbin`/`getTileBng` copies that differ only in the `RASTERX_CELL_ID` tag string (`cell.toString` for quadbin, `BNG.format(cell)` for BNG) and the geometry source. - -```scala -// --- Quadbin --- -def tessellateQuadbinIter( - ds: Dataset, options: Map[String, String], resolution: Int, mode: String = "covering" -): Iterator[(Long, Dataset, Map[String, String])] = { - require(Modes.contains(mode), s"gbx_rst_quadbin_tessellate mode must be one of ${Modes.mkString(", ")}; got '$mode'") - // Enumerate candidate cells covering the raster bbox via Quadbin.polyfill on the bbox - // (verify Quadbin.polyfill signature: takes bbox tuple + zoom, returns Seq[Long]). - // For covering: keep cells whose geometry intersects the raster bbox (as H3 covering does). - // For centroid: single-assign each valid pixel to Quadbin.pointToCell(lon,lat,z) (as H3 centroid does). - // Build cell geometry from Quadbin.cellToBBox -> JTS polygon. - ... // structure copied from tessellateH3Iter / tessellateH3CoveringIter / tessellateH3CentroidIter -} - -// --- BNG --- -def tessellateBngIter( - ds: Dataset, options: Map[String, String], resolution: Int, mode: String = "covering" -): Iterator[(String, Dataset, Map[String, String])] = { - require(Modes.contains(mode), s"gbx_rst_bng_tessellate mode must be one of ${Modes.mkString(", ")}; got '$mode'") - // BNG: reproject ds to EPSG:27700 first (RasterProject, nearest). Cell geometry = BNG.cellIdToGeometry(cell:Long), - // in EPSG:27700 (same CRS as the warped bbox). Enumerate via BNG.polyfill(bbox, resolution) if available, - // else derive the easting/northing cell range from the bbox and resolution. Emit BNG.format(cell) as the id. - ... -} -``` - -Because the H3 covering/centroid helpers are ~120 lines, the implementer MUST read `tessellateH3Iter`, `tessellateH3CoveringIter`, `tessellateH3CentroidIter`, and `getTile` in full and clone their control flow, substituting only the cell-enumeration + cell-geometry + id-tag. Do not invent a new tessellation algorithm. Keep `covering` = geometric-overlap keep-test (per the documented decision at the top of `getTile`), `centroid` = pixel-centroid single-assign. - -**Issue #49 acceptance (BNG tessellate is the originating customer ask — spec §1.1):** -- The BNG iterator MUST build cell geometry directly from `BNG.cellIdToGeometry(cell: Long)` and enumerate cells for the raster bbox itself. It MUST NOT route through the vector `bng_tessellate` expression — that path carries inherited Mosaic bugs (mosaic#423 spurious POINT/LINESTRING chips; mosaic#434/#580 half-size cells) tracked separately for pygx phase 2. The customer explicitly hit "issues with the bng_tessellate function from GridX"; the raster path must sidestep them, not reuse them. -- `covering` mode yields one clipped chip per BNG cell overlapping the raster — uniform-size image tiles, which is the customer's CV use case. -- Add a Task-3 test asserting the BNG iterator emits ≥1 chip for a GB raster and each chip is tagged with its BNG `String` id (via `RASTERX_CELL_ID`), and that only areal geometry is used (no POINT/LINESTRING chip leakage). - -- [ ] **Step 4: Run to verify it passes** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.grid.RST_Quadbin_TessellateTest' --log tess.log -``` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/operations/RasterTessellate.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_Quadbin_TessellateTest.scala -git commit -m "feat(rasterx): quadbin + BNG tessellate iterators in RasterTessellate" -``` - ---- - -## Task 4: Quadbin + BNG tessellate generator expressions - -**Files:** -- Create: `.../generators/RST_Quadbin_Tessellate.scala`, `.../generators/RST_BNG_Tessellate.scala` -- Reference: `.../generators/RST_H3_Tessellate.scala` - -**Interfaces:** -- Consumes: `RasterTessellate.tessellateQuadbinIter`/`tessellateBngIter` (Task 3). -- Produces: `RST_Quadbin_Tessellate` / `RST_BNG_Tessellate` case classes + companions; `name` = `gbx_rst_quadbin_tessellate` / `gbx_rst_bng_tessellate`; `elementSchema = StructType(Array(StructField("tile", tileType)))`; 2-or-3-arg builder, default `mode="covering"`. - -- [ ] **Step 1: Write the failing test** - -Add to `RST_Quadbin_TessellateTest.scala`: - -```scala -test("generator names + default mode arity") { - assert(RST_Quadbin_Tessellate.name == "gbx_rst_quadbin_tessellate") - assert(RST_BNG_Tessellate.name == "gbx_rst_bng_tessellate") - // 2-arg builder defaults mode to "covering" - import org.apache.spark.sql.catalyst.expressions.Literal - val e = RST_Quadbin_Tessellate.builder()(Seq(Literal("t"), Literal(10))) - assert(e.isInstanceOf[RST_Quadbin_Tessellate]) -} -``` - -- [ ] **Step 2: Run to verify it fails** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.grid.RST_Quadbin_TessellateTest' --log tess.log -``` -Expected: FAIL — generators undefined. - -- [ ] **Step 3: Implement generators** - -Clone `RST_H3_Tessellate.scala` verbatim into each new file, changing: class/object name, `name` literal, the `RasterTessellate.tessellate*Iter` call, and the error message. `RST_BNG_Tessellate` uses `BNG.getResolution` on the resolution arg if it may arrive as a string; otherwise `resolutionExpr.eval(input).asInstanceOf[Int]` as H3 does. The generator body (CollectionGenerator, `elementSchema`, cleanup listener, row wrapping) is identical. - -```scala -// RST_Quadbin_Tessellate.scala — identical structure to RST_H3_Tessellate, with: -// name = "gbx_rst_quadbin_tessellate" -// iter = RasterTessellate.tessellateQuadbinIter(ds, mtd, resolution, mode) // cellId Long; tag handled inside iter -// RST_BNG_Tessellate.scala — same, with: -// name = "gbx_rst_bng_tessellate" -// iter = RasterTessellate.tessellateBngIter(ds, mtd, resolution, mode) // cellId String -``` - -- [ ] **Step 4: Run to verify it passes** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.grid.RST_Quadbin_TessellateTest' --log tess.log -``` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/generators/RST_Quadbin_Tessellate.scala \ - src/main/scala/com/databricks/labs/gbx/rasterx/expressions/generators/RST_BNG_Tessellate.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/grid/RST_Quadbin_TessellateTest.scala -git commit -m "feat(rasterx): quadbin + BNG tessellate generator expressions" -``` - ---- - -## Task 5: Quadbin rasterize_agg UDAF - -**Files:** -- Create: `.../agg/RST_Quadbin_RasterizeAgg.scala` -- Reference: `.../agg/RST_H3_RasterizeAgg.scala` (full), `.../util/VectorRasterBridge.scala` (`buildEmptyRaster`), `Quadbin.scala` (`resolution`, centroid, bbox). - -**Interfaces:** -- Consumes: `VectorRasterBridge.buildEmptyRaster`, `Quadbin.resolution(cell: Long): Int`, `Quadbin.centroid`/bbox for gridspec sample points. -- Produces: `RST_Quadbin_RasterizeAgg` (TypedImperativeAggregate reusing a `Long`-keyed acc); `name` = `gbx_rst_quadbin_rasterize_agg`; same 12-arg signature as H3. - -- [ ] **Step 1: Write the failing test** - -Create `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/agg/RST_GridRasterizeAggTest.scala`: - -```scala -package com.databricks.labs.gbx.rasterx.expressions.agg - -import org.scalatest.funsuite.AnyFunSuite - -class RST_GridRasterizeAggTest extends AnyFunSuite { - test("quadbin rasterize_agg canonical name + 12-arg builder") { - assert(RST_Quadbin_RasterizeAgg.name == "gbx_rst_quadbin_rasterize_agg") - import org.apache.spark.sql.catalyst.expressions.Literal - val args = (0 until 12).map(i => Literal(i)).toSeq - assert(RST_Quadbin_RasterizeAgg.builder()(args).isInstanceOf[RST_Quadbin_RasterizeAgg]) - } -} -``` - -- [ ] **Step 2: Run to verify it fails** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.agg.RST_GridRasterizeAggTest' --log rasterize.log -``` -Expected: FAIL — undefined. - -- [ ] **Step 3: Implement the UDAF** - -Clone `RST_H3_RasterizeAgg.scala`. Keep the `Long`-keyed accumulator + serde (rename `H3RasterizeAcc` → `QuadbinRasterizeAcc` or reuse a shared acc class — prefer a shared `LongCellRasterizeAcc` if you extract one, but a straight clone is acceptable for this task). Substitute the four documented per-grid points (spec §3.3): -1. `resolutionOf`: replace `H3Core.h3GetResolution(c)` with `Quadbin.resolution(c)`. -2. gridspec sample points: replace `H3.cellIdToCenter`/`cellIdToBoundary` with quadbin centroid / bbox-corner coordinates (WGS84 — quadbin bbox is already lon/lat). -3. default `pixel_size`: derive from zoom (web-mercator tile edge in the target srid) instead of `H3.edgeLength`. -4. projection: same pixel-centre → WGS84 → `Quadbin.pointToCell` mapping (quadbin input is 4326, so identical structure to H3's WGS84 hop). -`buildEmptyRaster` usage, NoData `-9999.0`, last-wins fold order, serde — unchanged. - -- [ ] **Step 4: Run to verify it passes** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.agg.RST_GridRasterizeAggTest' --log rasterize.log -``` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/agg/RST_Quadbin_RasterizeAgg.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/agg/RST_GridRasterizeAggTest.scala -git commit -m "feat(rasterx): quadbin rasterize_agg UDAF" -``` - ---- - -## Task 6: BNG rasterize_agg UDAF - -**Files:** -- Create: `.../agg/RST_BNG_RasterizeAgg.scala` -- Reference: `RST_H3_RasterizeAgg.scala`, `RST_Quadbin_RasterizeAgg.scala` (Task 5), `BNG.scala` (`parse`, `format`, `getResolution`, `cellIdToCenter`, `cellIdToBoundary`). - -**Interfaces:** -- Consumes: `BNG.parse(cellID: String): Long`, `BNG.getResolution`, `BNG.cellIdToCenter(cell: Long): Coordinate`, `BNG.cellIdToBoundary(cell: Long): Seq[Coordinate]`, `VectorRasterBridge.buildEmptyRaster`. -- Produces: `RST_BNG_RasterizeAgg`; `name` = `gbx_rst_bng_rasterize_agg`; 12-arg signature; **cellid input is STRING** (parsed to internal Long on `update`). - -- [ ] **Step 1: Write the failing test** - -Add to `RST_GridRasterizeAggTest.scala`: - -```scala -test("bng rasterize_agg canonical name + string cellid parse") { - assert(RST_BNG_RasterizeAgg.name == "gbx_rst_bng_rasterize_agg") - import org.apache.spark.sql.catalyst.expressions.Literal - val args = (0 until 12).map(i => Literal(i)).toSeq - assert(RST_BNG_RasterizeAgg.builder()(args).isInstanceOf[RST_BNG_RasterizeAgg]) -} -``` - -- [ ] **Step 2: Run to verify it fails** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.agg.RST_GridRasterizeAggTest' --log rasterize.log -``` -Expected: FAIL — `RST_BNG_RasterizeAgg` undefined. - -- [ ] **Step 3: Implement the UDAF** - -Clone the Task-5 quadbin UDAF. Per-grid substitutions: -1. **cellid input type STRING:** in `update`, the raw cellid is a `UTF8String`; convert to internal Long via `BNG.parse(raw.toString)`. Keep the accumulator `Long`-keyed (so serde is unchanged). Reject non-String cellids with a clear error. -2. `resolutionOf`: `BNG.getResolution` semantics — read each cell's resolution from its Long id (use the same helper BNG uses internally; if only `getResolution(res: Any)` exists for the *argument* form, derive resolution from `cellDigits`/the id as `BNG.format`/`cellIdToGeometry` do — read `BNG.scala` to find the id→resolution path). Error on mixed resolutions. -3. gridspec sample points: `BNG.cellIdToCenter` / `cellIdToBoundary` — these return **EPSG:27700** coordinates, so there is **no WGS84 hop**. Set `srcSR`/`dstSR` to 27700 (identity) — the sample points and the output raster are both in 27700. This is the key BNG divergence. -4. projection in the burn loop: pixel centre is already 27700 → `BNG.pointToCellID(e, n, resolution)` directly (no reproject to WGS84). -5. default `pixel_size`: BNG resolution → metre edge (1=100000, 2=10000, 3=1000, 4=100, 5=10, 6=1; negative resolutions per `BNG` base-50 quadrant sizes — derive from the same divisor math `pointToCellID` uses). -`buildEmptyRaster` (srid 27700), NoData `-9999.0`, last-wins — unchanged. - -- [ ] **Step 4: Run to verify it passes** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.agg.RST_GridRasterizeAggTest' --log rasterize.log -``` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/expressions/agg/RST_BNG_RasterizeAgg.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/agg/RST_GridRasterizeAggTest.scala -git commit -m "feat(rasterx): BNG rasterize_agg UDAF (27700-native, String cellid)" -``` - ---- - -## Task 7: Register all 9 + Scala functions wrappers + integration test - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/functions.scala` -- Modify: `docs/tests-function-info/registered_functions.txt` -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_GridIntegrationTest.scala` (create — end-to-end SQL registration + a covering→rastertogrid round-trip) - -**Interfaces:** -- Consumes: all 9 expression objects (Tasks 2, 4, 5, 6) + the 5 BNG reducers. -- Produces: SQL-registered functions; Scala `functions` column wrappers. - -- [ ] **Step 1: Write the failing test** - -Create `RST_GridIntegrationTest.scala` — register functions on a `SparkSession`, assert all 9 resolve, and run one end-to-end assertion (spec §2.6 empty-cell + §5 round-trip). Use the existing `RST_H3IntegrationTest` as the pattern for session setup and sample-data loading. - -```scala -// Mirror RST_H3IntegrationTest: create/borrow the test SparkSession, call rasterx functions.register(spark), -// then: -test("all 9 grid functions are registered") { - val fns = spark.sessionState.functionRegistry.listFunction().map(_.funcName).toSet - Seq("gbx_rst_bng_rastertogridavg","gbx_rst_bng_rastertogridcount","gbx_rst_bng_rastertogridmax", - "gbx_rst_bng_rastertogridmin","gbx_rst_bng_rastertogridmedian","gbx_rst_bng_tessellate", - "gbx_rst_quadbin_tessellate","gbx_rst_quadbin_rasterize_agg","gbx_rst_bng_rasterize_agg") - .foreach(n => assert(fns.contains(n), s"$n not registered")) -} - -test("rasterize_agg output declares -9999 NoData (spec 2.6)") { - // build a tiny cell set, run gbx_rst_quadbin_rasterize_agg, read the tile band GetNoDataValue == -9999.0 - ... -} -``` - -- [ ] **Step 2: Run to verify it fails** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_GridIntegrationTest' --log grid-int.log -``` -Expected: FAIL — functions not registered. - -- [ ] **Step 3: Register + add wrappers** - -In `functions.scala`: -- Add to the imports the new agg classes: `RST_Quadbin_RasterizeAgg, RST_BNG_RasterizeAgg` in the `expressions.agg` import; generators `RST_Quadbin_Tessellate, RST_BNG_Tessellate`; grid reducers `RST_BNG_RasterToGrid{Avg,Count,Max,Min,Median}`. -- Add `rd.register(...)` next to the existing siblings: -```scala -rd.register(RST_Quadbin_RasterizeAgg) // near RST_H3_RasterizeAgg (line ~84) -rd.register(RST_BNG_RasterizeAgg) -rd.register(RST_Quadbin_Tessellate) // near RST_H3_Tessellate (line ~95) -rd.register(RST_BNG_Tessellate) -rd.register(RST_BNG_RasterToGridAvg) // after RST_Quadbin_RasterToGridMedian (line ~111) -rd.register(RST_BNG_RasterToGridCount) -rd.register(RST_BNG_RasterToGridMax) -rd.register(RST_BNG_RasterToGridMin) -rd.register(RST_BNG_RasterToGridMedian) -``` -- Add Scala `functions` column wrappers mirroring `rst_h3_tessellate` / `rst_quadbin_rastertogridavg` (2-arg + 3-arg where the H3 sibling has them). Example: -```scala -def rst_bng_rastertogridavg(tileExpr: Column, resolution: Column): Column = - ColumnAdapter(RST_BNG_RasterToGridAvg.name, Seq(tileExpr, resolution)) -def rst_quadbin_tessellate(tileExpr: Column, resolution: Column): Column = - ColumnAdapter(RST_Quadbin_Tessellate.name, Seq(tileExpr, resolution)) -def rst_quadbin_tessellate(tileExpr: Column, resolution: Column, mode: String): Column = - ColumnAdapter(RST_Quadbin_Tessellate.name, Seq(tileExpr, resolution, lit(mode))) -// ... and bng_tessellate (2+3 arg), the 4 other bng reducers, quadbin_rasterize_agg, bng_rasterize_agg -``` -For `rasterize_agg`, mirror however `rst_h3_rasterize_agg`'s wrapper is exposed (grep it in `functions.scala`; if H3 has no column wrapper because it's SQL-only UDAF, follow that same pattern — SQL registration alone). - -- Add the 9 names to `docs/tests-function-info/registered_functions.txt` (alphabetical within their prefix groups): -``` -gbx_rst_bng_rastertogridavg -gbx_rst_bng_rastertogridcount -gbx_rst_bng_rastertogridmax -gbx_rst_bng_rastertogridmedian -gbx_rst_bng_rastertogridmin -gbx_rst_bng_rasterize_agg -gbx_rst_bng_tessellate -gbx_rst_quadbin_rasterize_agg -gbx_rst_quadbin_tessellate -``` - -- [ ] **Step 4: Run to verify it passes** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.expressions.RST_GridIntegrationTest' --log grid-int.log -``` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/functions.scala \ - docs/tests-function-info/registered_functions.txt \ - src/test/scala/com/databricks/labs/gbx/rasterx/expressions/RST_GridIntegrationTest.scala -git commit -m "feat(rasterx): register 9 BNG/quadbin raster-grid functions + wrappers" -``` - ---- - -## Task 8: Python bindings - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/rasterx/functions.py` -- Reference: the existing `rst_h3_tessellate`, `rst_quadbin_rastertogridavg`, `rst_h3_rasterize_agg` bindings in the same file. - -**Interfaces:** -- Consumes: the 9 registered SQL names (Task 7). -- Produces: 9 Python `call_function` wrappers matching the Scala API names (`rst_bng_rastertogridavg`, …, `rst_quadbin_tessellate`, `rst_bng_tessellate`, `rst_quadbin_rasterize_agg`, `rst_bng_rasterize_agg`). - -- [ ] **Step 1: Write the failing test** - -Add to `python/geobrix/test/rasterx/` (heavy binding test dir; mirror an existing binding-presence test) — or if bindings are only asserted via `gbx:test:bindings`, write a light import/signature test: - -```python -def test_bng_quadbin_raster_grid_bindings_exist(): - from databricks.labs.gbx.rasterx import functions as rx - for name in [ - "rst_bng_rastertogridavg", "rst_bng_rastertogridcount", "rst_bng_rastertogridmax", - "rst_bng_rastertogridmin", "rst_bng_rastertogridmedian", "rst_bng_tessellate", - "rst_quadbin_tessellate", "rst_quadbin_rasterize_agg", "rst_bng_rasterize_agg", - ]: - assert hasattr(rx, name), f"missing binding {name}" -``` - -- [ ] **Step 2: Run to verify it fails** - -``` -gbx:test:python --path python/geobrix/test/rasterx/ # narrow to the new test node -``` -Expected: FAIL — attributes missing. - -- [ ] **Step 3: Implement bindings** - -Clone the existing wrappers. Reducers (2-arg, like `rst_quadbin_rastertogridavg`): -```python -def rst_bng_rastertogridavg(tile: ColLike, resolution: ColLike) -> Column: - """Average raster value within each BNG grid cell. - - Args: - tile: Raster tile column. - resolution: BNG resolution (±1..±6 or a resolution string like "1km"). - Returns: - Column of array of (bng_cell STRING, measure DOUBLE). - """ - return f.call_function("gbx_rst_bng_rastertogridavg", _col(tile), _col(resolution)) -``` -Repeat for count/max/min/median. Tessellate (mirror `rst_h3_tessellate`, 3-arg with `mode="covering"` default). `rasterize_agg` (mirror `rst_h3_rasterize_agg`'s wrapper — same 12 args; for BNG the cellid column is a STRING). - -- [ ] **Step 4: Run to verify it passes** - -``` -gbx:test:python --path python/geobrix/test/rasterx/ -``` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/rasterx/functions.py python/geobrix/test/rasterx/ -git commit -m "feat(rasterx): Python bindings for 9 BNG/quadbin raster-grid functions" -``` - ---- - -## Task 9: function-info SQL examples + regenerate + binding parity - -**Files:** -- Modify: `docs/tests/python/api/rasterx_functions_sql.py` (add `*_sql_example()` for each of the 9) -- Regenerate: `src/main/resources/com/databricks/labs/gbx/function-info.json` (via `gbx:docs:function-info`) - -**Interfaces:** -- Consumes: registered names (Task 7), Python bindings (Task 8). -- Produces: non-empty `function-info.json` entries; passing `gbx:test:bindings`. - -- [ ] **Step 1: Write the failing check** - -Run the parity gate first to see the 9 gaps: -``` -gbx:test:bindings --log bindings.log -``` -Expected: FAIL — 9 functions missing `function-info.json` examples. - -- [ ] **Step 2: Add SQL examples** - -For each of the 9, add a `*_sql_example()` in `docs/tests/python/api/rasterx_functions_sql.py` that executes **real** SQL against real sample data (BNG examples use British coordinates / a UK raster; quadbin/H3 use lon/lat). No placeholders, no empty usage — the coverage test asserts non-empty. Mirror the existing rasterx `*_sql_example()` functions in that file. - -- [ ] **Step 3: Regenerate function-info** - -``` -gbx:docs:function-info -``` -This runs `generate-function-info.py` and writes `function-info.json`. Dispatch in Docker if it needs the container. - -- [ ] **Step 4: Run parity to verify it passes** - -``` -gbx:test:bindings --log bindings.log -``` -Expected: PASS — all 9 present as Scala name + Python binding + function-info key. - -- [ ] **Step 5: Commit** - -```bash -git add docs/tests/python/api/rasterx_functions_sql.py \ - src/main/resources/com/databricks/labs/gbx/function-info.json -git commit -m "feat(rasterx): function-info examples for 9 BNG/quadbin raster-grid functions" -``` - ---- - -## Task 10: Cross-cutting tests (reproject correctness, round-trip, seam-safety) - -**Files:** -- Modify: `RST_BNG_RasterToGridTest.scala`, `RST_GridIntegrationTest.scala` -- These require real GDAL + sample data → run in Docker (integration). - -- [ ] **Step 1: BNG reproject-correctness test** - -Add: a fixture raster in EPSG:3857 (or 4326) and the same raster **pre-warped** to EPSG:27700 must yield identical BNG cell assignments + measures (within 1e-9). Proves the internal auto-warp matches an explicit upstream warp. - -```scala -test("bng rastertogrid: internal warp matches explicit upstream warp") { - val ds3857 = /* small raster over GB in EPSG:3857 */ ??? - val ds27700 = /* same raster warped to 27700 with gdalwarp -r near */ ??? - val meanF = (v: scala.collection.mutable.ArrayBuffer[Double]) => v.sum / v.length - val a = RST_BNG_RasterToGrid.execute(ds3857, 3, meanF).flatten.toMap // triggers internal warp - val b = RST_BNG_RasterToGrid.execute(ds27700, 3, meanF).flatten.toMap // already 27700, no warp - assert(a.keySet == b.keySet) - a.foreach { case (cell, v) => assert(math.abs(v - b(cell)) < 1e-9) } -} -``` - -- [ ] **Step 2: rasterize_agg round-trip + NoData** - -Add (per spec §5): `rastertogrid` then `rasterize_agg` on the same cell set recovers per-cell values; the output tile's band `GetNoDataValue == -9999.0`; feeding it back into a reducer excludes the filled pixels. - -- [ ] **Step 3: Run the full grid suites in Docker** - -``` -gbx:test:scala --suites 'com.databricks.labs.gbx.rasterx.expressions.grid.*,com.databricks.labs.gbx.rasterx.expressions.agg.RST_GridRasterizeAggTest,com.databricks.labs.gbx.rasterx.expressions.RST_GridIntegrationTest' --log grid-all.log -``` -Expected: PASS (all). - -- [ ] **Step 4: Commit** - -```bash -git add src/test/scala/com/databricks/labs/gbx/rasterx/ -git commit -m "test(rasterx): BNG reproject-correctness + rasterize round-trip + NoData regression" -``` - ---- - -## Task 11: Docs + README badges - -**Files:** -- Modify: `docs/docs/api/raster-functions.mdx`, `docs/docs/api/execution-tiers.mdx`, `docs/docs/api/performance.mdx`, `docs/docs/beta-release-notes.mdx`, `README.md` - -- [ ] **Step 1: raster-functions.mdx** — add reference entries for the 9 functions, grouped with their H3/quadbin siblings. BNG entries note the auto-reproject-to-27700 behaviour and the BNG resolution contract; all note the `-9999.0` NoData sentinel caveat that H3 already documents. No internal vocabulary (no wave numbers). - -- [ ] **Step 2: execution-tiers.mdx** — mark the 9 as heavy-tier now, with light quadbin (Phase 2) and light BNG (Phase 3, gated on pygx BNG) as "planned". Match the existing tier-badge style. - -- [ ] **Step 3: performance.mdx** — add the 9 to the existing execution-shape families (rastertogrid reducers, tessellate generator, rasterize UDAF) — classify into existing families, do not invent a new shape (per `performance-doc-update-on-new-function`). - -- [ ] **Step 4: beta-release-notes.mdx** — a feature entry: quadbin and BNG now have the full H3 raster surface (rastertogrid reducers, tessellate, rasterize_agg) on the heavy tier. Call out `gbx_rst_bng_tessellate` for raster tiling by BNG scale (the CV image-tiling use case from issue #49). User-facing voice, no internal vocabulary (QC judge `internals-leak` enforces). The PR body / merge commit should reference "Closes #49". - -- [ ] **Step 5: README.md badges** — RasterX 108 → 117, Functions 156 → 165. Update the two `img.shields.io` lines (the comment already documents the derivation). - -```bash -git add docs/docs/api/raster-functions.mdx docs/docs/api/execution-tiers.mdx \ - docs/docs/api/performance.mdx docs/docs/beta-release-notes.mdx README.md -git commit -m "docs(rasterx): document 9 BNG/quadbin raster-grid functions + badge bump" -``` - -- [ ] **Step 6: Run doc tests + grep for internals leak** - -``` -gbx:test:docs --log docs.log -grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/ ; echo "(should print nothing)" -``` - ---- - -## Task 12: Benchmark registration (heavy-tier, existing 20-node cluster) - -**Files:** -- Modify: `src/test/scala/com/databricks/labs/gbx/bench/BenchDispatch.scala` -- Modify: `docs/docs/api/benchmarking.mdx` -- Test: `src/test/scala/com/databricks/labs/gbx/bench/BenchDispatchTest.scala` -- Reference: `BenchDispatch.scala` (the `shape` map, input-classification sets, aggregate-branch sets), `HeavyBenchSuite.scala`, `scripts/commands/gbx-bench-cluster.sh`, `scripts/commands/gbx-bench-heavyweight.sh`. - -**Context — the 20-node config is reused, not redefined.** All cluster benches read the same cluster from `notebooks/tests/databricks_cluster_config.env` (via `CLUSTER_ID`, sourced by `gbx:bench:cluster`). "Same 20-node config as other benchmarks" means run the new functions through that existing config unchanged — do **not** author a new cluster spec. The only code change is registering the 9 functions in `BenchDispatch` so the existing harness discovers and dispatches them; the harness then benchmarks them on the same cluster with the same row ladder as every other `rst_*` function. - -**Interfaces:** -- Consumes: the 9 registered SQL/Scala names (Task 7). Bench uses the Scala-API form without the `gbx_` prefix (e.g. `rst_bng_rastertogridavg`, matching how `rst_h3_rastertogridavg` appears in `BenchDispatch`). -- Produces: `BenchDispatch` entries so `BenchDispatch.all` and `--set full` include the 9; the 2 rasterize_aggs routed to the grid-aggregate branch (the `rst_h3_rasterize_agg` path). - -- [ ] **Step 1: Write the failing test** - -Add to `BenchDispatchTest.scala` (mirror its existing assertions about `rst_h3_*` membership): - -```scala -test("BenchDispatch registers the 9 BNG/quadbin raster-grid functions") { - val expected = Seq( - "rst_bng_rastertogridavg", "rst_bng_rastertogridcount", "rst_bng_rastertogridmax", - "rst_bng_rastertogridmin", "rst_bng_rastertogridmedian", - "rst_bng_tessellate", "rst_quadbin_tessellate", - "rst_quadbin_rasterize_agg", "rst_bng_rasterize_agg") - expected.foreach(fn => assert(BenchDispatch.all.contains(fn), s"$fn not in BenchDispatch.all")) - // shape classification: reducers + tessellate are DGGS; the aggs route to the grid-aggregate branch - assert(BenchDispatch.shapeOf("rst_bng_rastertogridavg") == "DGGS") - assert(BenchDispatch.aggregateBranchOf("rst_bng_rasterize_agg") == "grid_aggregate") // or whatever h3's branch label is -} -``` -(Adapt `shapeOf`/`aggregateBranchOf` to the actual accessor names in `BenchDispatch` — read the file; if shape is a private `Map`, assert via the public dispatch entry point the test file already uses.) - -- [ ] **Step 2: Run to verify it fails** - -Dispatch a Task subagent (Docker): -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.bench.BenchDispatchTest' --log bench-dispatch.log -``` -Expected: FAIL — the 9 names absent from `BenchDispatch.all`. - -- [ ] **Step 3: Register the 9 in BenchDispatch** - -In `BenchDispatch.scala`, add to the `shape` map next to the existing H3/quadbin grid entries (lines ~169–191): -```scala -// BNG raster→grid reducers (DGGS shape, same as H3/quadbin reducers) -"rst_bng_rastertogridavg" -> DGGS, "rst_bng_rastertogridcount" -> DGGS, -"rst_bng_rastertogridmax" -> DGGS, "rst_bng_rastertogridmedian" -> DGGS, -"rst_bng_rastertogridmin" -> DGGS, -// tessellate generators (DGGS, same as rst_h3_tessellate) -"rst_quadbin_tessellate" -> DGGS, "rst_bng_tessellate" -> DGGS, -// grid rasterize aggregators (DGGS; routed through the grid-aggregate branch like rst_h3_rasterize_agg) -"rst_quadbin_rasterize_agg" -> DGGS, "rst_bng_rasterize_agg" -> DGGS -``` -Add the 2 rasterize_aggs to the same aggregate-branch set that holds `rst_h3_rasterize_agg` (the `h3Aggregate` set at line ~226 — rename to a grid-neutral `gridAggregate` if it now holds 3 grids, updating the `aggregateShape`/branch dispatch at line ~234 accordingly; keep the branch label stable or update the test in Step 1 to match). Ensure the pure-core/spark-path input classification for the new reducers matches the H3/quadbin reducers (they take a single tile + resolution — no special input set needed; verify they are NOT accidentally caught by `byteInput`/`tileArrayInput`/geometry-input sets). - -**Parity contract for the aggs:** `rst_h3_rasterize_agg` uses a fixed deterministic cell set with an explicit grid (the PARITY CONTRACT block at line ~237). Add the analogous fixed cell set for quadbin (a fixed zoom + tile range) and BNG (a fixed resolution + a handful of GB cells), so the bench's light-vs-heavy parity check has a deterministic input. Mirror the `h3RaggRes`/`h3RaggCenterLat` constants with quadbin/BNG equivalents. - -- [ ] **Step 4: Run to verify it passes** - -``` -gbx:test:scala --suite 'com.databricks.labs.gbx.bench.BenchDispatchTest' --log bench-dispatch.log -``` -Expected: PASS. - -- [ ] **Step 5: Document the bench coverage** - -In `docs/docs/api/benchmarking.mdx`, add the 9 functions to the results narrative/tables alongside their siblings (the `rst_h3_rastertogrid*` / `rst_quadbin_rastertogrid*` rows already there ~lines 380–400, and `rst_h3_tessellate` ~line 370). Note that BNG reducers/tessellate include an internal EPSG:27700 reproject in their timing (unlike the 4326-native H3/quadbin), so a like-for-like comparison should account for the warp. No actual numbers yet — those come from a cluster run; add the rows as "pending first cluster run" or leave the table and add a one-line note that the new grid functions are covered by the harness. Per `bench-changes-update-docs`, any bench change is reflected here in the same stroke. No internal vocabulary. - -- [ ] **Step 6: Commit** - -```bash -git add src/test/scala/com/databricks/labs/gbx/bench/BenchDispatch.scala \ - src/test/scala/com/databricks/labs/gbx/bench/BenchDispatchTest.scala \ - docs/docs/api/benchmarking.mdx -git commit -m "bench(rasterx): register 9 BNG/quadbin raster-grid functions in BenchDispatch" -``` - -- [ ] **Step 7: (Optional, user-gated) Run the actual cluster benchmark** - -This runs on the shared 20-node cluster — **do not launch without user go-ahead** (bench cluster is shared; guard against duplicate runs; the cluster memory notes apply: poll libs to INSTALLED, give a summary.md link at the end). When approved: -``` -gbx:bench:cluster --functions rst_bng_rastertogridavg,rst_bng_rastertogridcount,rst_bng_rastertogridmax,rst_bng_rastertogridmin,rst_bng_rastertogridmedian,rst_bng_tessellate,rst_quadbin_tessellate,rst_quadbin_rasterize_agg,rst_bng_rasterize_agg --row-counts 1000 --run-id bng-quadbin-parity -``` -(1000-scale for the spark path per the bench policy.) Then backfill the real numbers into `benchmarking.mdx` and give the user the run's `summary.md` link. - ---- - -## Final gate: Pre-push checks (after Task 12 — do NOT push, user batches pushes) - -- [ ] Run and report results. Hold for user go-ahead before any push. - -``` -gbx:lint:scalastyle -gbx:lint:python --check -gbx:test:bindings -``` - ---- - -## Self-Review notes (author) - -- **Spec §1 (+9 fns):** Tasks 2 (5 BNG reducers), 4 (2 tessellate), 5+6 (2 rasterize_agg) = 9. ✅ covered. -- **Spec §2.2 (Long ids, format/parse):** Task 1 uses `BNG.format` at output; Task 6 uses `BNG.parse` on input. ✅ -- **Spec §2.3 (CRS split):** Task 1/3/6 reproject BNG to 27700; quadbin untouched (4326 contract). ✅ -- **Spec §2.6 (empty-cell/NoData):** Task 1 Step-1 empty test; Task 10 Step-2 NoData/round-trip; global constraint restated. ✅ -- **Spec §3.4 (BNG resolution):** Task 2 `UTF8String` overloads via `BNG.getResolution`; global constraint. ✅ -- **Spec §4 (heavy-first):** this plan is Phase 1 only; light phases are separate plans. ✅ -- **Spec §5 (tests):** Tasks 1,2,3,4,5,6,7,10 unit+integration; Task 10 reproject-correctness + round-trip. ✅ -- **Spec §6 (surfaces):** registered_functions (T7), functions.scala (T7), Python (T8), function-info (T9), docs+README (T11), bindings (T9/T11), benchmark harness + benchmarking.mdx (T12). ✅ -- **Benchmarking (user-requested addition):** Task 12 registers all 9 in `BenchDispatch` (reducers+tessellate→DGGS; aggs→grid-aggregate branch) and reuses the existing 20-node cluster config (`notebooks/tests/databricks_cluster_config.env`) — no new cluster spec. Actual cluster run is user-gated (Task 12 Step 7). ✅ -- **Spec §7 risks:** scan-extraction is optional (Task 1 introduces BNG fresh, no H3 refactor — lowest-risk branch of §3.1/§8); nearest-neighbour warp pinned (global constraint + Task 10 test); GDALManager guard (global constraint). -- **Spec §8 open question (extract shared scan vs BNG-only helper):** plan chooses **BNG-only** (Task 1 writes a standalone object; no refactor of H3/quadbin reference), the lower-risk option. If a reviewer wants the shared `RasterGridScan` extraction, that is a follow-up refactor task, not a blocker. diff --git a/docs/superpowers/plans/2026-07-25-raster-bng-quadbin-h3-parity-phase2-light.md b/docs/superpowers/plans/2026-07-25-raster-bng-quadbin-h3-parity-phase2-light.md deleted file mode 100644 index 5a7ee7eb9..000000000 --- a/docs/superpowers/plans/2026-07-25-raster-bng-quadbin-h3-parity-phase2-light.md +++ /dev/null @@ -1,235 +0,0 @@ -# Raster BNG + Quadbin (H3 parity) — Phase 2 (Light/pyrx) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Give all 9 heavy-tier raster BNG/quadbin functions a lightweight (`pyrx`) implementation so every one flips `` → ``, completing both-tier parity for the feature. - -**Architecture:** Add grid-generic branches to the three light engine modules (`gridagg.py`, `tessellate.py`, `cellraster.py`), each keeping its existing H3 path unchanged and delegating new grids' cell math to `pygx` (`_bng`, `_quadbin`) — single source of truth, no cell-math duplication. Register 9 light UDTFs/UDFs. Quadbin first (engine already 4326-native), then BNG (adds EPSG:27700 warp + String cell ids), then docs/diagram/badges. - -**Tech Stack:** Python 3.12, `pyrx` (rasterio + shapely + pyproj + numpy), `pygx` (BNG/quadbin cell math). Cross-tier parity tests run in Docker (heavy needs the JAR). No new dependencies. - -**Spec:** `docs/superpowers/specs/2026-07-24-raster-bng-quadbin-h3-parity-design.md` §4.1 -**Branch:** `issues/49` (off `beta/0.4.0`). Builds toward **v0.4.3**. - -## Global Constraints - -- **Single source of truth for cell math:** BNG cell math comes ONLY from `pygx._bng` (`point_to_cell_id`, `format`, `parse`, `cell_id_to_geometry`, `is_valid`, `get_edge_size`, `get_resolution_from_digits`, `CRS_ID=27700`); quadbin from `pygx._quadbin`. NEVER reimplement/duplicate a grid's cell math in `pyrx/core/*` (the one existing exception, `gridagg._quadbin_cells`, is a pre-existing bit-exact numpy encoder — do NOT add a second BNG copy; BNG uses a scalar `pygx._bng` loop). -- **Each engine keeps its H3 path unchanged.** New grids are sibling branches/adapters. The `cellraster.py` grid-adapter refactor is refactor-only for H3 (no behavior change); existing H3 light tests are the regression gate. -- **BNG CRS:** warp input raster to EPSG:27700 (`rasterio.warp`, **nearest**) before cell math; read eastings/northings from the warped geotransform (0.5-px centroid); drop out-of-GB pixels/cells via `pygx._bng.is_valid`. Quadbin/H3 stay 4326 (no warp). -- **Cell id types:** BNG cell ids are **String** at the output boundary (`pygx._bng.format`), Long internally. Quadbin/H3 are Long. New `_GRID_FLAT_STRING_SCHEMA` (`cellID StringType`) for BNG reducers. -- **Parity bar (def-of-done):** light output == heavy output — **exact cell-set** (BNG String / quadbin Long ids) + measures within tolerance (1e-9 numeric, 1e-6 geometry). This is per the pygx exact-parity standard. -- **Empty-cell / NoData (spec §2.6):** rastertogrid emits a cell only for ≥1 valid pixel (never a zero-valid-pixel cell). rasterize_agg uses `_NODATA = -9999.0` written into a raster whose nodata is set to -9999 (masked on read-back) — mirror the existing `cellraster` H3 handling exactly. -- **Python wrappers:** `rst_*` reducer/tessellate wrappers stay SQL-LATERAL-invocation pointers raising `NotImplementedError` with the LATERAL example (matching existing H3/quadbin). `rasterize_agg` gets a callable wrapper mirroring `rst_h3_rasterize_agg`. -- **Light-CI checklist:** no new deps (pygx/rasterio/shapely/pyproj/numpy already present); Phase-2 tests land under `python/geobrix/test/pyrx/`, which is already in all three light-CI sync points (`_LIGHT_TEST_DIRS` in conftest, the light-run list in `pyrx_build/action.yml`, and the heavy `LIGHT_IGNORES` in `python_build/action.yml` — synced 2026-07-25 so light tests only run in the light phase). No new test dir is introduced, so no CI sync change is needed for this plan; if a future task adds a NEW light dir, update all three. If any dep is added, pin across all 3 envs + recompile hashed txt. -- **Commands:** light Python tests `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/`; cross-tier/doc tests in Docker via `gbx:test:*-docs`; lint `bash scripts/commands/gbx-lint-python.sh --check` (host black may differ from Docker — verify with Docker `--check` before push). Do NOT push (batch-push; user-gated). - -## File Structure - -**Modify (engines):** -- `python/geobrix/src/databricks/labs/gbx/pyrx/core/gridagg.py` — add `"bng"` branch: `_bng_cells` scalar encoder (calls `pygx._bng.point_to_cell_id`), 27700 warp for bng, `is_valid` drop, String-id yield. -- `python/geobrix/src/databricks/labs/gbx/pyrx/core/tessellate.py` — add `iter_tessellate_quadbin`, `iter_tessellate_bng` (mirror `iter_tessellate_h3`; reuse shared clip/emit; BNG warp+is_valid). -- `python/geobrix/src/databricks/labs/gbx/pyrx/core/cellraster.py` — refactor `_h3_str`/`_resolution`/`compute_gridspec`/`cell_bbox`/`cells_to_raster` to dispatch on a `grid` param via per-grid adapters backed by `pygx`; H3 path preserved. - -**Modify (registration/bindings):** -- `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` — `_GRID_FLAT_STRING_SCHEMA`; 5 `_RstBngRasterToGrid*UDTF` via factory; `_RstQuadbinTessellateUDTF` + `_RstBngTessellateUDTF`; quadbin/BNG `rasterize_agg` UDFs; register all 9; `rasterize_agg` callable wrappers. - -**Create (tests):** `python/geobrix/test/pyrx/test_gridagg_bng.py`, `test_tessellate_quadbin.py`, `test_tessellate_bng.py`, `test_cellraster_grids.py`; cross-tier parity in `python/geobrix/test/` doc/integration area (Docker). - -**Modify (docs, final task):** `docs/docs/api/raster-functions.mdx` (9 badges heavy→both; fix page-level "every function both-tier" claim + BNG-Grid reason), `docs/docs/api/execution-tiers.mdx`, `docs/docs/api/performance.mdx`, `resources/images/generators/rasterx-function-categories.py` (+ regenerate PNG/SVG), `docs/docs/api/beta-release-notes.mdx`. - ---- - -## Task 1: Light quadbin tessellate - -**Files:** -- Modify: `pyrx/core/tessellate.py` (add `iter_tessellate_quadbin`) -- Modify: `pyrx/functions.py` (`_RstQuadbinTessellateUDTF` + register + wrapper) -- Test: `python/geobrix/test/pyrx/test_tessellate_quadbin.py` -- Reference: `iter_tessellate_h3` / `_centroid_chips` in tessellate.py; `_RstH3TessellateUDTF` in functions.py; `pygx._quadbin` (`point_as_cell`, `as_wkb`, `resolution`). - -**Interfaces:** -- Consumes: `pygx._quadbin.point_as_cell(lon,lat,res)`, quadbin cell→polygon (via `_quadbin.as_wkb(cell)` → shapely, or the cell bbox); shared clip/emit helpers in tessellate.py. -- Produces: `iter_tessellate_quadbin(ds, resolution, mode="covering") -> Iterator[(int cellid, bytes raster)]`; `_RstQuadbinTessellateUDTF` registered as `gbx_rst_quadbin_tessellate`. - -- [ ] **Step 1: Write the failing test** — `test_tessellate_quadbin.py`: covering over a small EPSG:4326 raster yields ≥1 chip each tagged with a Long quadbin cell id; centroid mode single-assigns; unknown mode raises. Use the real sample-raster fixture pattern from the existing H3 tessellate test (read `test/pyrx/` for it). - -```python -def test_iter_tessellate_quadbin_covering_yields_tagged_chips(): - from databricks.labs.gbx.pyrx.core import tessellate as t - import rasterio - with rasterio.open(SMALL_4326_TIF) as ds: - chips = list(t.iter_tessellate_quadbin(ds, resolution=12, mode="covering")) - assert chips, "covering must yield >=1 chip" - for cellid, raster in chips: - assert isinstance(cellid, int) and cellid != 0 - assert raster # non-empty GTiff bytes -``` - -- [ ] **Step 2: Run to verify it fails** — `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_tessellate_quadbin.py`. Expected: FAIL (`iter_tessellate_quadbin` undefined). - -- [ ] **Step 3: Implement** — Clone `iter_tessellate_h3`'s structure into `iter_tessellate_quadbin`. Substitute only: cell enumeration (`pygx._quadbin.polyfill` on the raster bbox for covering; per-pixel `pygx._quadbin.point_as_cell` for centroid), cell geometry (shapely from `pygx._quadbin.as_wkb(cell)`), Long id tag. 4326-native (no warp). Reuse the module's shared clip/emit machinery. Add `_RstQuadbinTessellateUDTF` (mirror `_RstH3TessellateUDTF`), register `gbx_rst_quadbin_tessellate` in the `udtfs` list, add the `rst_quadbin_tessellate` NotImplementedError-pointer wrapper. - -- [ ] **Step 4: Run to verify it passes** — same command. Expected: PASS. - -- [ ] **Step 5: Commit** — `git add pyrx/core/tessellate.py pyrx/functions.py test/pyrx/test_tessellate_quadbin.py && git commit -m "feat(pyrx): light quadbin tessellate"` - ---- - -## Task 2: Light quadbin rasterize_agg + cellraster grid-adapter refactor - -**Files:** -- Modify: `pyrx/core/cellraster.py` (grid-adapter refactor + quadbin binding) -- Modify: `pyrx/functions.py` (`_rst_quadbin_rasterize_agg_udf` + register + wrapper) -- Test: `python/geobrix/test/pyrx/test_cellraster_grids.py` -- Reference: `cellraster.py` (`_h3_str`, `_resolution`, `compute_gridspec`, `cell_bbox`, `cells_to_raster`); `_rst_h3_rasterize_agg_udf` in functions.py; `pygx._quadbin` (`resolution`, `centroid`, `k_ring`). - -**Interfaces:** -- Consumes: a new per-grid adapter in cellraster (`grid` param → `to_str`/`resolution`/`cell_center`/`cell_boundary`/`edge_size`); `pygx._quadbin`. -- Produces: `cellraster` functions accept `grid="h3"|"quadbin"|"bng"`; `_rst_quadbin_rasterize_agg_udf`; `gbx_rst_quadbin_rasterize_agg` registered. - -- [ ] **Step 1: Write the failing test** — `test_cellraster_grids.py`: (a) H3 rasterize unchanged (regression: same output as before the refactor for a fixed cell set); (b) quadbin rasterize_agg burns a small quadbin cell set to a raster whose band nodata == -9999 and recovers the per-cell values via a round-trip through `gridagg.raster_to_grid(..., "quadbin", "avg")`. - -```python -def test_quadbin_rasterize_roundtrip_and_nodata(): - from databricks.labs.gbx.pyrx.core import cellraster as cr, gridagg - # fixed quadbin cell set + values -> raster -> read back -> exact recover; nodata==-9999 - ... -``` - -- [ ] **Step 2: Run to verify it fails** — `gbx-test-python.sh --path .../test_cellraster_grids.py`. Expected: FAIL. - -- [ ] **Step 3: Implement** — Refactor cellraster's H3-hardcoded helpers to dispatch on `grid` via a small adapter object/dict: `to_str(id)`, `resolution(ids)`, `cell_center(id)->(lon,lat or e,n)`, `cell_boundary(id)`, `edge_size(res)`, and the sample-point CRS (`4326` for h3/quadbin). H3 adapter binds the current `h3.*` calls (behavior identical). Quadbin adapter binds `pygx._quadbin`. `compute_gridspec`/`cell_bbox`/`cells_to_raster` take `grid` and use the adapter. Add `_rst_quadbin_rasterize_agg_udf` (mirror H3's, `grid="quadbin"`), register `gbx_rst_quadbin_rasterize_agg`, add the callable wrapper. - -- [ ] **Step 4: Run to verify it passes** — also run the existing H3 cellraster/rasterize tests to confirm the refactor didn't regress H3: `gbx-test-python.sh --path python/geobrix/test/pyrx/`. Expected: PASS (new + all existing H3). - -- [ ] **Step 5: Commit** — `feat(pyrx): light quadbin rasterize_agg + grid-adapter cellraster refactor` - ---- - -## Task 3: Light BNG rastertogrid reducers (×5) - -**Files:** -- Modify: `pyrx/core/gridagg.py` (`_bng_cells` + `"bng"` branch + 27700 warp + String yield) -- Modify: `pyrx/functions.py` (`_GRID_FLAT_STRING_SCHEMA`; 5 `_RstBngRasterToGrid*UDTF`; register; wrappers) -- Test: `python/geobrix/test/pyrx/test_gridagg_bng.py` -- Reference: `raster_to_grid` / `_h3_cells` / `_grouped_measures` in gridagg.py; `_make_rastertogrid_udtf` + `_GRID_FLAT_*` schemas in functions.py; `pygx._bng` (`point_to_cell_id`, `format`, `is_valid`). - -**Interfaces:** -- Consumes: `pygx._bng.point_to_cell_id(e,n,res)`, `pygx._bng.format(id)`, `pygx._bng.is_valid(id)`; `rasterio.warp` to 27700. -- Produces: `gridagg.raster_to_grid(ds, res, "bng", agg)` returns `[{"cellID": str, "measure": ...}]`; 5 `gbx_rst_bng_rastertogrid{avg,count,max,min,median}` registered UDTFs; `_GRID_FLAT_STRING_SCHEMA`. - -- [ ] **Step 1: Write the failing test** — `test_gridagg_bng.py`: (a) a small EPSG:27700 London raster (all valid) → `raster_to_grid(ds, 3, "bng", "avg")` yields String cell ids matching `^[A-Z]{2}\d*$` with correct mean; (b) all-nodata band yields `[]` (no zero-valid-pixel cell, §2.6); (c) a 4326 input raster is auto-warped to 27700 and yields valid BNG cells; (d) out-of-GB pixels dropped (`is_valid`). - -- [ ] **Step 2: Run to verify it fails** — `gbx-test-python.sh --path .../test_gridagg_bng.py`. Expected: FAIL (`"bng"` grid unknown). - -- [ ] **Step 3: Implement** — In gridagg: add `_bng_cells(e, n, res)` scalar loop over valid pixels calling `pygx._bng.point_to_cell_id` (like `_h3_cells`); in `raster_to_grid`, for `grid=="bng"` warp `ds` to EPSG:27700 (`rasterio.warp`, nearest) first, compute e/n from the warped geotransform, drop cells failing `pygx._bng.is_valid`, and render `pygx._bng.format(id)` as the String `cellID` at yield (keep grouping Long-keyed). Extend `_validate_resolution` for bng (±1..±6 / `pygx._bng.get_resolution`). In functions.py: `_GRID_FLAT_STRING_SCHEMA` (cellID StringType; count = String cellID + Integer measure), 5 `_RstBngRasterToGrid*UDTF` via `_make_rastertogrid_udtf("bng", agg, schema)` (the UDTF yield must emit String cellID — adjust the factory to not `int()` the cellID for bng, or add a String-cellID factory variant), register the 5, add NotImplementedError-pointer wrappers. - -- [ ] **Step 4: Run to verify it passes** — same, plus the existing H3/quadbin gridagg tests (confirm no regression). Expected: PASS. - -- [ ] **Step 5: Commit** — `feat(pyrx): light BNG rastertogrid reducers (avg/count/max/min/median)` - ---- - -## Task 4: Light BNG tessellate - -**Files:** -- Modify: `pyrx/core/tessellate.py` (`iter_tessellate_bng`) -- Modify: `pyrx/functions.py` (`_RstBngTessellateUDTF` + register + wrapper) -- Test: `python/geobrix/test/pyrx/test_tessellate_bng.py` -- Reference: `iter_tessellate_quadbin` (Task 1); `pygx._bng` (`polyfill`, `cell_id_to_geometry`, `format`, `is_valid`, `point_to_cell_id`). - -**Interfaces:** -- Produces: `iter_tessellate_bng(ds, resolution, mode="covering") -> Iterator[(str cellid, bytes raster)]`; `gbx_rst_bng_tessellate` registered. - -- [ ] **Step 1: Write the failing test** — `test_tessellate_bng.py`: covering over a GB raster (27700, or 4326 auto-warped) yields ≥1 chip each tagged with a BNG String id `^[A-Z]{2}\d*$`; only areal chips; centroid single-assigns; unknown mode raises; a 4326 input triggers the warp. - -- [ ] **Step 2: Run to verify it fails** — Expected: FAIL. - -- [ ] **Step 3: Implement** — Clone `iter_tessellate_quadbin` into `iter_tessellate_bng`; warp raster to 27700 first; enumerate via `pygx._bng.polyfill(bbox_poly, res)` (buffer the bbox before polyfill — mirror the heavy-tier fix so boundary cells aren't dropped; use `pygx._bng` buffer-radius helper if present, else a resolution-derived buffer), cell geometry from `pygx._bng.cell_id_to_geometry`, drop out-of-GB via `is_valid`, String id tag via `format`. covering = geometric-overlap keep-test; centroid = per-pixel `point_to_cell_id` on 27700 coords. Add `_RstBngTessellateUDTF`, register `gbx_rst_bng_tessellate`, wrapper. - -- [ ] **Step 4: Run to verify it passes** — Expected: PASS. - -- [ ] **Step 5: Commit** — `feat(pyrx): light BNG tessellate (27700 warp, boundary-complete covering)` - ---- - -## Task 5: Light BNG rasterize_agg - -**Files:** -- Modify: `pyrx/core/cellraster.py` (BNG adapter binding — 27700-native) -- Modify: `pyrx/functions.py` (`_rst_bng_rasterize_agg_udf` + register + wrapper) -- Test: `python/geobrix/test/pyrx/test_cellraster_grids.py` (extend) -- Reference: Task 2 grid-adapter; `pygx._bng` (`parse`, `format`, `cell_id_to_geometry`, `get_edge_size`, `get_resolution_from_digits`). - -**Interfaces:** -- Produces: cellraster BNG adapter (27700, no WGS84 hop); `_rst_bng_rasterize_agg_udf` (cellid input STRING, parsed via `pygx._bng.parse`); `gbx_rst_bng_rasterize_agg` registered. - -- [ ] **Step 1: Write the failing test** — extend `test_cellraster_grids.py`: BNG rasterize_agg burns a small BNG cell set (String ids) to a 27700 raster (band nodata == -9999); round-trip through `raster_to_grid(..., "bng", "avg")` recovers the per-cell values. Assert the srid used is 27700. - -- [ ] **Step 2: Run to verify it fails** — Expected: FAIL. - -- [ ] **Step 3: Implement** — Bind the BNG adapter in cellraster: `to_str` = `pygx._bng.format`, `resolution(ids)` via `pygx._bng.get_resolution_from_digits`, `cell_center`/`cell_boundary` from `pygx._bng.cell_id_to_geometry` (27700 coords — **no 4326 reproject**; sample-point CRS = 27700), `edge_size` = `pygx._bng.get_edge_size`. `_rst_bng_rasterize_agg_udf`: parse String cellid → Long via `pygx._bng.parse`, `grid="bng"`, srid forced 27700 (document the srid arg is a no-op for BNG). Register `gbx_rst_bng_rasterize_agg`, callable wrapper. - -- [ ] **Step 4: Run to verify it passes** — full `test/pyrx/` (confirm H3+quadbin+BNG all green). Expected: PASS. - -- [ ] **Step 5: Commit** — `feat(pyrx): light BNG rasterize_agg (27700-native, String cellid)` - ---- - -## Task 6: Cross-tier parity tests (all 9, Docker) - -**Files:** -- Test: cross-tier parity suite under `python/geobrix/test/` (follow the existing light-vs-heavy parity test pattern — grep for an existing `*parity*` or a test that registers both tiers). -- Docker: heavy tier needs the JAR + sample data. - -- [ ] **Step 1: Write the tests** — for each of the 9: run the light SQL function and the heavy SQL function on the SAME real sample raster + resolution, assert **exact cell-set equality** (BNG String / quadbin Long ids) and measures within tolerance (1e-9 numeric, 1e-6 geometry). For tessellate: same emitted cell-id set + per-chip pixel parity within tolerance. For rasterize_agg: same output raster (band-wise within NoData-aware tolerance) + band nodata == -9999. - -- [ ] **Step 2: Run in Docker** — `bash scripts/commands/gbx-test-scala.sh` is heavy; the cross-tier test is Python-driven but needs the JAR-backed heavy functions — run via the doc/integration Docker path (`gbx:test:*-docs` or the existing parity harness). Report the command. If a real production divergence appears, STOP and report BLOCKED with expected-vs-actual — do NOT weaken the tolerance. - -- [ ] **Step 3: Commit** — `test(pyrx): cross-tier light-vs-heavy parity for 9 BNG/quadbin raster-grid fns` - ---- - -## Task 7: function-info + binding parity (light entries) - -**Files:** -- The 9 already have function-info examples (Phase 1). Verify light registration doesn't break `gbx:test:bindings`; if the light SQL functions need any function-info/registered_functions adjustment, make it here. - -- [ ] **Step 1** — run `bash scripts/commands/gbx-test-bindings.sh --log bindings.log`. Expected: still PASS 165/165 (parity is name-level; light adds impls under the same names). If it fails, address upstream. -- [ ] **Step 2: Commit** if any change — else note "no change needed; parity holds". - ---- - -## Task 8: Docs + diagram + badges (flip heavy→both) - -**Files:** -- `docs/docs/api/raster-functions.mdx`, `docs/docs/api/execution-tiers.mdx`, `docs/docs/api/performance.mdx`, `docs/docs/api/beta-release-notes.mdx`, `resources/images/generators/rasterx-function-categories.py` - -- [ ] **Step 1: raster-functions.mdx** — flip all 9 `` → ``; add per-function `:::note Lightweight tier (pyrx)` admonitions (backing lib: `pygx._bng`/`_quadbin` + rasterio; BNG notes 27700 warp). **Fix the now-true page-level claim** (lines ~20, ~157): "every RasterX function is available in both tiers" is TRUE again once these land — remove the Phase-1 "except…" qualifier that Phase-1 docs added. Fix the BNG-Grid section's tier language (drop "heavy-tier only / pending pygx BNG raster integration"; it's both-tier now). -- [ ] **Step 2: execution-tiers.mdx** — move the 9 from heavy-only to both in the raster-grid tier table. -- [ ] **Step 3: performance.mdx** — the 9 now have light impls; update the "Heavy-tier variants" framing added in Phase 1 to reflect both-tier availability; classify into existing families (no new shape). -- [ ] **Step 4: rasterx-function-categories diagram** — update `resources/images/generators/rasterx-function-categories.py`: header comment 108→117; add the 2 tessellate to Generators, 2 rasterize_agg to Aggregators, and a **BNG Grid** card (mirror H3/Quadbin Grid) with the 5 BNG reducers. Regenerate PNG+SVG (portrait + landscape) via the Chrome-headless commands in the script docstring. Verify `bash docs/scripts/check-diagram-coverage.py` passes (coverage + count). -- [ ] **Step 5: beta-release-notes.mdx** — note the 9 raster BNG/quadbin functions are now both-tier (lightweight parity), building toward v0.4.3. -- [ ] **Step 6: verify** — `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/` (clean); `bash docs/scripts/check-diagram-coverage.py` (pass); user-facing voice. -- [ ] **Step 7: Commit** — `docs(rasterx): light-tier parity — flip 9 grid fns to both-tier + diagram/badges` - ---- - -## Final gate: pre-push checks (after Task 8 — do NOT push, user-gated) - -- [ ] Run and report: `bash scripts/commands/gbx-lint-python.sh --check` (verify with Docker `--check` too, host black may differ); `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/`; `bash scripts/commands/gbx-test-bindings.sh`; `bash docs/scripts/check-diagram-coverage.py`. Hold for user go-ahead before push to `issues/49`. - ---- - -## Self-Review notes (author) - -- **Spec §4.1 (light design):** Task 1 (quadbin tessellate), Task 2 (quadbin rasterize + adapter refactor), Task 3 (5 BNG reducers), Task 4 (BNG tessellate), Task 5 (BNG rasterize) = all 9. ✅ -- **Single-source cell math:** every task delegates to `pygx` (`_bng`/`_quadbin`); no numpy BNG reimpl (global constraint). ✅ -- **H3 unchanged:** Task 2's cellraster refactor is the only H3 restructure (refactor-only; existing H3 tests gate it in Step 4). Tasks 1/3/4 are additive. ✅ -- **BNG 27700 + String + is_valid:** Tasks 3/4/5 each warp to 27700, use `is_valid`, render String ids. ✅ -- **BNG tessellate boundary completeness:** Task 4 buffers bbox before polyfill (carries the heavy-tier final-review fix into light). ✅ -- **Parity bar:** Task 6 cross-tier exact-cell-set + tolerance. ✅ -- **§2.6:** Task 3 Step-1 (b) empty-band → [] ; Task 2/5 nodata==-9999 + round-trip. ✅ -- **Docs/diagram/badges (deferred from Phase 1):** Task 8 flips badges, fixes the page-level claim + BNG reason, regenerates the coverage-enforced diagram (108→117 + BNG card). ✅ -- **Light-CI:** no new deps; pyrx/pygx test dirs already in `_LIGHT_TEST_DIRS`. ✅ -- **Open impl-time decision:** the `_make_rastertogrid_udtf` factory currently `int()`s the cellID; Task 3 must add a String-cellID path (factory variant or param) — flagged in Task 3 Step 3, not a blocker. diff --git a/docs/superpowers/plans/2026-07-26-light-bng-encoder-vectorization.md b/docs/superpowers/plans/2026-07-26-light-bng-encoder-vectorization.md deleted file mode 100644 index 2f74f8ac6..000000000 --- a/docs/superpowers/plans/2026-07-26-light-bng-encoder-vectorization.md +++ /dev/null @@ -1,505 +0,0 @@ -# Light BNG raster→grid encoder vectorization — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Eliminate the two per-pixel Python loops in the light BNG raster→grid path (encoder + `is_valid` filter) by adding numpy array kernels in `pygx/_bng.py`, maintained as ONE shared numpy-polymorphic core per operation that both the scalar and vector entry points wrap — so scalar-vs-vector drift is impossible by construction. - -**Architecture:** Refactor `point_to_cell_id` / `get_quadrant` into numpy-polymorphic cores (`_point_to_cell_id_core`, `_get_quadrant_core`, `_encode_core`) that run identically on a Python scalar and a numpy array (via `np.trunc`/`np.floor`/`np.where`, all arithmetic in int64). Keep the existing scalar public names as thin wrappers; add `point_to_cell_id_vec`. Add a resolution-aware `is_valid_vec(cell_ids, resolution)` (single-resolution batch, so no per-cell resolution derivation). Rewire `gridagg._raster_to_bng`'s two per-pixel loops to the vector kernels. - -**Tech Stack:** Python 3.12, numpy (already a dependency), pytest. Light tier only — no Scala/JAR change, no new deps, no new functions (behavior identical, only faster). - -## Global Constraints - -- **Bit-exact cell ids.** The refactored scalar `point_to_cell_id` / `get_quadrant` / `is_valid` must reproduce their CURRENT output exactly (integer equality, never tolerance). `point_to_cell_id_vec` / `is_valid_vec` must equal the scalar element-for-element. These are integer cell ids — any divergence is a bug. -- **int64 accumulation.** The encode step accumulates in int64, never float64. (The packed id at res ±6 is a 16-digit number < 2×10¹⁵, within float64's 2⁵³ exact-integer ceiling AND int64's 9.2×10¹⁸ — but int64 removes all doubt and matches the current pure-integer `encode`.) -- **Truncation semantics.** The scalar uses `int(e_int/100000)` (truncate toward zero) for letter indices and `math.floor(.../divisor)` (floor) for bins. The core must mirror EACH exactly (`np.trunc` vs `np.floor`) — they differ for negative (out-of-GB) coords, which the encoder DOES see (encode runs before `is_valid`). -- **Single source of truth.** No second copy of the BNG codec math. Scalar and vector forms share one core. A reviewer must not "optimize" the scalar path back into a separate body (that reintroduces drift) — the scalar routing through numpy is marginally slower and that is accepted (only cold callers use it). -- **No public-surface change beyond the two new `*_vec` functions.** `format`/`parse`/geometry/neighborhood/polyfill/tessellate are untouched. No doc/badge/registered-function change. -- **`test/pygx` and `test/pyrx` are light dirs.** Tests run under `gbx:test:python`; the affected packages must be run before push. - ---- - -### Task 1: Shared numpy-polymorphic core + `point_to_cell_id_vec` - -Refactor the scalar encoder into a shared core and add the vector entry point. This is the dominant per-pixel cost. - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pygx/_bng.py` (`get_quadrant` ~L165-180, `encode` ~L183-200, `point_to_cell_id` ~L203-220; add cores + `point_to_cell_id_vec`) -- Test: `python/geobrix/test/pygx/test_bng_encoder_vec.py` (new) - -**Interfaces:** -- Consumes: nothing new (uses `numpy`, already imported? — add `import numpy as np` if absent). -- Produces: - - `_get_quadrant_core(resolution: int, eastings, northings, divisor) -> np.int64|np.ndarray` (numpy-polymorphic) - - `_encode_core(e_letter, n_letter, e_bin, n_bin, quadrant, n_positions, resolution) -> np.int64|np.ndarray` (int64) - - `_point_to_cell_id_core(e, n, resolution: int) -> np.int64|np.ndarray` (int64) - - `point_to_cell_id(eastings: float, northings: float, resolution: int) -> int` (thin wrapper, unchanged signature/behavior) - - `get_quadrant(resolution: int, eastings: float, northings: float, divisor: float) -> int` (thin wrapper, unchanged) - - `point_to_cell_id_vec(e: np.ndarray, n: np.ndarray, resolution: int) -> np.ndarray[int64]` - -- [ ] **Step 1: Write the failing regression + smoke test** - -Create `python/geobrix/test/pygx/test_bng_encoder_vec.py`: - -```python -"""Bit-exactness tests for the vectorized BNG encoder. - -The shared numpy core is a rewrite of the current scalar codec body, so the -gate is TWO checks: (1) the refactored scalar reproduces the CURRENT behavior -over a frozen baseline (no regression), and (2) the vec form equals the scalar -element-for-element (shared core => cannot drift, but this documents it and -guards the thin wrappers). Includes res +-6 to catch any int64 overflow / -float64 precision loss, and out-of-GB / negative coords (the encoder sees them -because encode runs BEFORE is_valid). -""" - -import numpy as np -import pytest - -from databricks.labs.gbx.pygx import _bng - -# Dense EPSG:27700 grid across GB + explicit out-of-GB / negative / boundary coords. -_EAST = np.concatenate( - [ - np.linspace(0.0, 700000.0, 43), - np.array([-150000.0, -1.0, 0.0, 99999.5, 100000.0, 529999.9, 530000.0, 700001.0]), - ] -) -_NORTH = np.concatenate( - [ - np.linspace(0.0, 1300000.0, 41), - np.array([-250000.0, -1.0, 0.0, 179999.9, 180000.0, 1300001.0]), - ] -) -_RESOLUTIONS = [-1, 1, -2, 2, -3, 3, -4, 4, -5, 5, -6, 6] - - -def _grid(): - EE, NN = np.meshgrid(_EAST, _NORTH) - return EE.ravel(), NN.ravel() - - -@pytest.mark.parametrize("res", _RESOLUTIONS) -def test_point_to_cell_id_vec_equals_scalar(res): - e, n = _grid() - vec = _bng.point_to_cell_id_vec(e, n, res) - assert vec.dtype == np.int64 - for ei, ni, c in zip(e, n, vec): - expected = _bng.point_to_cell_id(float(ei), float(ni), res) - assert int(c) == expected, f"e={ei} n={ni} res={res}: {int(c)} != {expected}" - - -@pytest.mark.parametrize("res", _RESOLUTIONS) -def test_scalar_matches_frozen_reference(res): - # Reference = an inlined copy of the ORIGINAL scalar body, proving the - # refactor introduced no regression. (Kept local so it can't drift with - # the module under test.) - import math - - def ref_get_quadrant(resolution, eastings, northings, divisor): - if resolution < -1: - e_q = eastings / divisor - n_q = northings / divisor - e_dec = e_q - math.floor(e_q) - n_dec = n_q - math.floor(n_q) - if e_dec < 0.5 and n_dec < 0.5: - return 1 - if e_dec < 0.5: - return 2 - if n_dec < 0.5: - return 4 - return 3 - return 0 - - def ref_encode(e_letter, n_letter, e_bin, n_bin, quadrant, n_positions, resolution): - id_placeholder = 10 ** (5 + 2 * n_positions - 2) - e_letter_shift = 10 ** (3 + 2 * n_positions - 2) - n_letter_shift = 10 ** (1 + 2 * n_positions - 2) - e_shift = 10 ** n_positions - n_shift = 10 - if resolution == -1: - val = (id_placeholder + e_letter * e_letter_shift) / 100 + quadrant - else: - val = ( - id_placeholder - + e_letter * e_letter_shift - + n_letter * n_letter_shift - + e_bin * e_shift - + n_bin * n_shift - + quadrant - ) - return int(val) - - def ref_point_to_cell_id(eastings, northings, resolution): - e_int = int(eastings) - n_int = int(northings) - e_letter = int(e_int / 100000) - n_letter = int(n_int / 100000) - if resolution < 0: - divisor = 10 ** (6 - abs(resolution) + 1) - else: - divisor = 10 ** (6 - resolution) - quadrant = ref_get_quadrant(resolution, e_int, n_int, divisor) - n_positions = abs(resolution) if resolution >= -1 else abs(resolution) - 1 - e_bin = math.floor((e_int % 100000) / divisor) - n_bin = math.floor((n_int % 100000) / divisor) - return ref_encode(e_letter, n_letter, e_bin, n_bin, quadrant, n_positions, resolution) - - e, n = _grid() - for ei, ni in zip(e, n): - assert _bng.point_to_cell_id(float(ei), float(ni), res) == ref_point_to_cell_id( - float(ei), float(ni), res - ), f"e={ei} n={ni} res={res}" - - -def test_vec_dtype_is_int64_at_high_resolution(): - # res 6 packed id ~1.x*10^15 -- must stay exact in int64, not round in float64. - e = np.array([529999.0, 530000.0, 123456.0]) - n = np.array([179999.0, 180000.0, 654321.0]) - vec = _bng.point_to_cell_id_vec(e, n, 6) - assert vec.dtype == np.int64 - for ei, ni, c in zip(e, n, vec): - assert int(c) == _bng.point_to_cell_id(float(ei), float(ni), 6) -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `gbx:test:python --path python/geobrix/test/pygx/test_bng_encoder_vec.py` (or `python -m pytest python/geobrix/test/pygx/test_bng_encoder_vec.py -v`) -Expected: FAIL — `AttributeError: module ... has no attribute 'point_to_cell_id_vec'`. - -- [ ] **Step 3: Add the shared cores and refactor the scalars** - -Ensure `import numpy as np` is present near the top of `_bng.py` (alongside `import math`). - -Replace `get_quadrant`, `encode`, and `point_to_cell_id` with cores + thin wrappers: - -```python -def _get_quadrant_core(resolution, eastings, northings, divisor): - """Numpy-polymorphic quadrant (0 for res >= -1; 1/2/4/3 for res < -1). - - ``eastings``/``northings`` are int64 (scalar or array). Mirrors the scalar - ``get_quadrant`` if-chain order via nested ``np.where``. - """ - if resolution >= -1: - return eastings * 0 # int64 zeros, scalar or array - e_q = eastings / divisor - n_q = northings / divisor - e_dec = e_q - np.floor(e_q) - n_dec = n_q - np.floor(n_q) - q = np.where( - (e_dec < 0.5) & (n_dec < 0.5), - 1, - np.where(e_dec < 0.5, 2, np.where(n_dec < 0.5, 4, 3)), - ) - return q.astype(np.int64) - - -def get_quadrant( - resolution: int, eastings: float, northings: float, divisor: float -) -> int: - """Scalar wrapper over :func:`_get_quadrant_core` (unchanged public behavior).""" - return int(_get_quadrant_core(resolution, eastings, northings, divisor)) - - -def _encode_core(e_letter, n_letter, e_bin, n_bin, quadrant, n_positions, resolution): - """Pack the BNG digit-id in int64 (scalar or array). Bit-exact with the - original pure-integer ``encode`` (the res==-1 ``/100`` is always exactly - divisible, so integer ``//100`` matches ``int(float/100)``).""" - id_placeholder = 10 ** (5 + 2 * n_positions - 2) - e_letter_shift = 10 ** (3 + 2 * n_positions - 2) - n_letter_shift = 10 ** (1 + 2 * n_positions - 2) - e_shift = 10 ** n_positions - n_shift = 10 - if resolution == -1: - val = (id_placeholder + e_letter * e_letter_shift) // 100 + quadrant - else: - val = ( - id_placeholder - + e_letter * e_letter_shift - + n_letter * n_letter_shift - + e_bin * e_shift - + n_bin * n_shift - + quadrant - ) - return val - - -def encode(e_letter, n_letter, e_bin, n_bin, quadrant, n_positions, resolution) -> int: - """Scalar wrapper over :func:`_encode_core` (unchanged public behavior).""" - return int(_encode_core(e_letter, n_letter, e_bin, n_bin, quadrant, n_positions, resolution)) - - -def _point_to_cell_id_core(e, n, resolution: int): - """Numpy-polymorphic BNG encoder core (int64 result; scalar or array). - - Mirrors the scalar ``point_to_cell_id`` EXACTLY, incl. truncation semantics: - letter indices use ``int(e_int/100000)`` (truncate toward zero -> ``np.trunc``); - bins use ``math.floor(.../divisor)`` (floor -> ``np.floor``). These differ for - negative (out-of-GB) coords, which the encoder sees (encode runs before is_valid). - """ - e_int = np.trunc(e).astype(np.int64) - n_int = np.trunc(n).astype(np.int64) - # int(e_int/100000): float division then truncate toward zero (NOT floor-div). - e_letter = np.trunc(e_int / 100000).astype(np.int64) - n_letter = np.trunc(n_int / 100000).astype(np.int64) - if resolution < 0: - divisor = 10 ** (6 - abs(resolution) + 1) - else: - divisor = 10 ** (6 - resolution) - quadrant = _get_quadrant_core(resolution, e_int, n_int, divisor) - n_positions = abs(resolution) if resolution >= -1 else abs(resolution) - 1 - # math.floor((e_int % 100000) / divisor): float division then floor. - e_bin = np.floor((e_int % 100000) / divisor).astype(np.int64) - n_bin = np.floor((n_int % 100000) / divisor).astype(np.int64) - return _encode_core(e_letter, n_letter, e_bin, n_bin, quadrant, n_positions, resolution) - - -def point_to_cell_id(eastings: float, northings: float, resolution: int) -> int: - """Scalar wrapper over :func:`_point_to_cell_id_core` (unchanged behavior).""" - if math.isnan(eastings) or math.isnan(northings): - raise ValueError("NaN coordinates are not supported.") - return int(_point_to_cell_id_core(float(eastings), float(northings), resolution)) - - -def point_to_cell_id_vec(e: np.ndarray, n: np.ndarray, resolution: int) -> np.ndarray: - """Vectorized BNG encoder: array of int64 cell ids from EPSG:27700 (e, n). - - Shares :func:`_point_to_cell_id_core` with the scalar ``point_to_cell_id``, so - the two forms cannot drift. Fed clean (post-mask) coords, so no NaN guard. - """ - e = np.asarray(e, dtype="float64") - n = np.asarray(n, dtype="float64") - return _point_to_cell_id_core(e, n, resolution).astype(np.int64) -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `python -m pytest python/geobrix/test/pygx/test_bng_encoder_vec.py -v` -Expected: PASS (all parametrizations). - -- [ ] **Step 5: Run the existing BNG codec/parity suites to confirm no regression in dependent callers** - -Run: `python -m pytest python/geobrix/test/pygx/test_bng_codec.py python/geobrix/test/pygx/test_parity_bng.py python/geobrix/test/pygx/test_bng_polyfill.py python/geobrix/test/pygx/test_bng_neighborhood.py -v` -Expected: PASS (polyfill/k-ring/point_as_cell all route through the rewritten scalar wrapper). - -- [ ] **Step 6: Lint** - -Run: `gbx:lint:python --check` (or `black --check` + `flake8` on `_bng.py` and the new test) -Expected: clean. - -- [ ] **Step 7: Commit** - -```bash -git add python/geobrix/src/databricks/labs/gbx/pygx/_bng.py python/geobrix/test/pygx/test_bng_encoder_vec.py -git commit -F # see commit-message-hygiene: subject <=72, WHY body, Co-authored-by: Isaac -``` -Subject: `perf(pygx): vectorize BNG encoder via shared numpy core` - ---- - -### Task 2: `is_valid_vec` (resolution-aware) — vectorize the second per-pixel loop - -The gridagg filter `[is_valid(int(c)) for c in cids]` is the OTHER per-pixel loop. Since the gridagg batch is single-resolution, a resolution-aware vector form avoids per-cell resolution derivation. - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pygx/_bng.py` (add `is_valid_vec` near `is_valid` ~L606-624) -- Test: `python/geobrix/test/pygx/test_bng_encoder_vec.py` (extend) - -**Interfaces:** -- Consumes: `point_to_cell_id_vec` (Task 1) to generate test ids. -- Produces: `is_valid_vec(cell_ids: np.ndarray, resolution: int) -> np.ndarray[bool]` — element-for-element equal to `[is_valid(int(c)) for c in cell_ids]` for ids produced at that resolution. - -- [ ] **Step 1: Write the failing test (append to `test_bng_encoder_vec.py`)** - -```python -@pytest.mark.parametrize("res", _RESOLUTIONS) -def test_is_valid_vec_equals_scalar(res): - e, n = _grid() # includes out-of-GB coords -> some ids are invalid - ids = _bng.point_to_cell_id_vec(e, n, res) - vec = _bng.is_valid_vec(ids, res) - assert vec.dtype == bool - for c, v in zip(ids, vec): - assert bool(v) == _bng.is_valid(int(c)), f"id={int(c)} res={res}" - - -def test_is_valid_vec_empty(): - assert _bng.is_valid_vec(np.array([], dtype=np.int64), 3).tolist() == [] -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `python -m pytest python/geobrix/test/pygx/test_bng_encoder_vec.py::test_is_valid_vec_equals_scalar -v` -Expected: FAIL — no `is_valid_vec` attribute. - -- [ ] **Step 3: Implement `is_valid_vec`** - -Faithful vectorization of `is_valid` for a fixed known resolution. The digit -layout is fixed by `resolution`, so extract the letter/coordinate digits with -integer arithmetic (`// 10**k % 10`) instead of string slicing, then apply the -same `get_x`/`get_y` index math as arrays. Iterate the digit-position loop only -`ndigits` (<=16) times — constant, NOT per-pixel. - -```python -def is_valid_vec(cell_ids: np.ndarray, resolution: int) -> np.ndarray: - """Vectorized :func:`is_valid` for a single-resolution batch of Long ids. - - Equivalent to ``[is_valid(int(c)) for c in cell_ids]`` when every id was - produced at ``resolution``. Extracts digit slices by integer arithmetic - (fixed digit layout per resolution), reproducing ``get_x``/``get_y`` and the - bounds/index checks on arrays. - """ - cell_ids = np.asarray(cell_ids, dtype=np.int64) - if cell_ids.size == 0: - return np.zeros(0, dtype=bool) - - # res -1 (500km) is a DEGENERATE 4-digit id (encode divides by 100) and its - # get_x/get_y math has k=-1 -- the generic array formula below does NOT apply. - # 500km cells never arise at raster-tile scale, so keep parity via the scalar - # path for this one resolution (still correct; not the hot case). - if resolution == -1: - return np.array( - [is_valid(int(c)) for c in cell_ids], dtype=bool - ) # vectorscan: ok (res -1 degenerate, non-hot) - - # For all other resolutions the digit count is fixed by the encode layout: - # ndigits = 4 + 2*n_positions (leading placeholder '1' + letter + coord fields). - n_positions = abs(resolution) if resolution >= -1 else abs(resolution) - 1 - ndigits = 4 + 2 * n_positions - - # digit(pos) with pos=0 the MOST-significant digit (matches cell_digits order). - def digit(pos): - p = ndigits - 1 - pos # power of 10 for that position - return (cell_ids // (10 ** p)) % 10 - - # Two-digit letter fields: cell_digits[1:3] -> y_letter, [3:5] -> x_letter. - y_letter = digit(1) * 10 + digit(2) - x_letter = digit(3) * 10 + digit(4) - - edge = get_edge_size(resolution) - k = (ndigits - 6) // 2 - quadrant = digit(ndigits - 1) - - # get_x: x_digits = digits[1:3] + digits[5:5+k]; value * edge_adj + x_offset. - def concat_coord(letter_pair, start): - val = letter_pair # digits[1:3] or [3:5] already combined by caller - for i in range(k): - val = val * 10 + digit(start + i) - return val - - x_val = concat_coord(x_letter, 5) # digits[3:5] ++ digits[5:5+k] - y_val = concat_coord(y_letter, 5 + k) # digits[1:3] ++ digits[5+k:5+2k] - edge_adj = np.where(quadrant > 0, 2 * edge, edge) - x_offset = np.where((quadrant == 3) | (quadrant == 4), edge, 0) - y_offset = np.where((quadrant == 2) | (quadrant == 3), edge, 0) - x = x_val * edge_adj + x_offset - y = y_val * edge_adj + y_offset - - return ( - (x >= 0) - & (x <= 700000) - & (y >= 0) - & (y <= 1300000) - & (x_letter < len(LETTER_MAP)) - & (y_letter < len(LETTER_MAP[0])) - ) -``` - -NOTE for the implementer: the digit-slice mapping in `get_x`/`get_y` is the -arbiter — `test_is_valid_vec_equals_scalar` (across every resolution, incl. -res −1 with `ndigits < 6`) is the gate. If a slice index is off, that test -fails loudly. Adjust `concat_coord`/`digit` indexing until green; do NOT loosen -the test. Watch the res −1 case (500km, `ndigits = 4`, `k = -1`) — if the -generic formula misbehaves there, special-case it to match the scalar exactly. - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `python -m pytest python/geobrix/test/pygx/test_bng_encoder_vec.py -v` -Expected: PASS (all, including the new `is_valid_vec` cases across every resolution). - -- [ ] **Step 5: Lint** - -Run: `gbx:lint:python --check` -Expected: clean. - -- [ ] **Step 6: Commit** - -Subject: `perf(pygx): add resolution-aware is_valid_vec for BNG` - ---- - -### Task 3: Rewire `gridagg._raster_to_bng` to the vector kernels + full parity - -Replace both per-pixel loops with the vector kernels and prove end-to-end parity is unchanged. - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/pyrx/core/gridagg.py` (`_bng_cells` ~L74-85; the `is_valid` filter in `_raster_to_bng._run` ~L292-301) -- Test: existing `python/geobrix/test/pyrx/test_gridagg_bng.py`, `python/geobrix/test/pyrx/test_parity_bng_quadbin_raster_grid.py` (run, do not modify unless a real gap surfaces) - -**Interfaces:** -- Consumes: `_bng.point_to_cell_id_vec`, `_bng.is_valid_vec` (Tasks 1–2). -- Produces: no signature change; `_raster_to_bng` output identical, just faster. - -- [ ] **Step 1: Rewire `_bng_cells`** - -Replace the scalar loop: - -```python -def _bng_cells(e: np.ndarray, n: np.ndarray, resolution: int) -> np.ndarray: - """Per-valid-pixel BNG cell ids (Long int64), fully vectorized. - - ``e``/``n`` are EPSG:27700 eastings/northings (pixel centroids of the WARPED - raster). Delegates to ``pygx._bng.point_to_cell_id_vec`` -- the SAME shared - numpy core the scalar ``point_to_cell_id`` wraps, so cell ids are identical. - """ - return _bng.point_to_cell_id_vec(e, n, resolution) -``` - -- [ ] **Step 2: Rewire the `is_valid` filter in `_raster_to_bng._run`** - -Replace the per-pixel `keep = np.array([_bng.is_valid(int(c)) for c in cids], ...)` block with: - -```python - cids = _bng_cells(e, n, resolution) - # Drop out-of-GB pixels (is_valid) BEFORE grouping so a cell is only - # emitted for >=1 valid, in-GB pixel (sec 2.6). Vectorized over the - # single-resolution batch. - keep = _bng.is_valid_vec(cids, resolution) - cids = cids[keep] - vals = vals[keep] -``` - -(Remove the now-unused `# vectorscan: ok (pygx._bng SSoT)` comments on those two sites; the SSoT is preserved via the shared core.) - -- [ ] **Step 3: Run the gridagg BNG + cross-tier parity suites** - -Run: `python -m pytest python/geobrix/test/pyrx/test_gridagg_bng.py python/geobrix/test/pyrx/test_parity_bng_quadbin_raster_grid.py python/geobrix/test/pyrx/test_tessellate_bng.py -v` -Expected: PASS — cell sets + measures unchanged (the vector path must not shift any cell id or drop a different set). - -- [ ] **Step 4: Run the full pygx + pyrx light suites for the touched packages** - -Run: `python -m pytest python/geobrix/test/pygx/ python/geobrix/test/pyrx/ -v` (in Docker if sample-data-backed tests are present: `gbx:test:python --path python/geobrix/test/pygx/` and `--path python/geobrix/test/pyrx/`) -Expected: PASS. - -- [ ] **Step 5: Lint** - -Run: `gbx:lint:python --check` -Expected: clean. - -- [ ] **Step 6: Commit** - -Subject: `perf(pyrx): use vectorized BNG kernels in raster->grid` -Body: note the ~6× light BNG rastertogrid gap this closes; a follow-on cluster bench re-measures (not part of this change's gate). - ---- - -## Post-plan follow-ons (NOT in this plan) - -- **Cluster bench re-measure** (own cycle): confirm light BNG rastertogrid is no longer ~6× slower (target: parity-class with quadbin). Update `docs/docs/api/benchmarking.mdx` BNG numbers. Per `bench-changes-update-docs` + `benchmarking-preflight-discipline`. -- **27700 warp cost** (secondary BNG perf term, spec §5): out of scope here; separate perf item if the encoder fix doesn't close the gap enough. -- quadbin (already vectorized) and h3 (already C-backed) need NO equivalent work — BNG was the sole grid paying per-pixel Python cost. - -## Self-Review - -- **Spec coverage:** §3.1 core → Task 1; §3.2 wrappers → Task 1; §3.3 `is_valid_vec` → Task 2; §3.4 `_bng_cells` rewire → Task 3; §4 parity gate (regression sweep + scalar-is-vec smoke + int64 res±6 + end-to-end) → Task 1 tests + Task 3 parity suites; §6 surfaces → all tasks; §7 risks (truncation, int64, scalar-through-numpy, out-of-GB) → Global Constraints + Task 1 test coverage. Covered. -- **Placeholders:** none — all code shown; `is_valid_vec` digit-index mapping explicitly gated by a cross-resolution equality test, with a res −1 caveat. -- **Type consistency:** `point_to_cell_id_vec` returns int64 (matches `_bng_cells`' historic `dtype="int64"`); `is_valid_vec` returns bool array (matches the old `keep` bool array); scalar wrappers return Python `int`/`bool` (unchanged). diff --git a/docs/superpowers/plans/2026-07-27-heavy-xyz-rgba-convergence.md b/docs/superpowers/plans/2026-07-27-heavy-xyz-rgba-convergence.md deleted file mode 100644 index 4a1f3804f..000000000 --- a/docs/superpowers/plans/2026-07-27-heavy-xyz-rgba-convergence.md +++ /dev/null @@ -1,530 +0,0 @@ -# Heavy XYZ RGBA Convergence Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make heavy `rst_tilexyz` / `rst_xyzpyramid` emit display RGB(A) web-map tiles matching the lightweight (rio-tiler) tier — RGBA for PNG/WEBP, RGB for JPEG — so an in-extent tile with internal NoData renders transparent (not opaque/black) on the heavy tier. - -**Architecture:** Insert an RGB(A) compositing step into `RST_TileXYZ.executeWithScale`, between the `gdal.Warp` and the `gdal.Translate`. The warp gains `-dstalpha` so GDAL derives a binary (0/255) alpha band from the source's valid-data mask — this is the same mask rio-tiler uses, and covers both out-of-extent and internal-NoData pixels in one step. A new private helper `toDisplayRGBA` reduces/expands the warped band set to the rio-tiler band mapping (1→grey RGB, 2→grey+alpha, 3→RGB, 4→RGB+alpha, ≥5→first-3 RGB) and attaches the alpha, producing a MEM dataset with explicit color interpretation that the translate step encodes. JPEG drops alpha (3-band RGB). The existing per-band `-scale` rescale applies to the RGB bands only. - -**Tech Stack:** Scala 2.13, Spark 4.0, GDAL Java bindings (`org.gdal.gdal`), scalatest (heavy unit tests), Python/pytest + rasterio + PIL (cross-tier parity test), Docker (`geobrix-dev`) for both test suites. - -## Global Constraints - -- Byte-parity across tiers is a NON-GOAL and impossible (different PNG/WEBP encoders). Cross-tier verification is decode + tolerance, never byte equality or a bench fingerprint. The bench stays `timing-only` — do not touch bench wiring. -- Band mapping matches rio-tiler EXACTLY: 1→grey replicated to R=G=B (+alpha); 2→band1 grey (R=G=B) + band2 as alpha; 3→R,G,B (+alpha); 4→R,G,B + band4 alpha; ≥5→first 3 bands as R,G,B (+alpha). -- Alpha is BINARY (0 or 255) from the warp's valid-data mask — one mask covers out-of-extent AND internal NoData. Alpha bands are NEVER passed to the `-scale` rescale. -- PNG → RGBA; WEBP → RGBA where the GDAL build's WEBP driver supports alpha, else 3-band RGB fallback with a logged note (never a hard failure); JPEG → RGB (no alpha). -- The 8-bit `rescale` behavior (`"auto"` default, `resolveScale`) is REUSED unchanged and applied to RGB bands only. Do not redesign it. -- Heavy output band-count changes (N→4 for PNG/WEBP, N→3 for JPEG): a beta behavior change, documented in release notes + function docs. -- Follow existing GDAL resource discipline: release every intermediate Dataset (`RasterDriver.releaseDataset` or `.delete()`) in `try/finally`; `gdal.Unlink` every `/vsimem` path. -- Scala style: matches CI scalastyle (`gbx:lint:scalastyle`). Run before any commit that touches Scala. -- The out-of-extent `transparentPng(size)` fallback already emits all-zero-alpha RGBA — keep it; it is the reference for the RGBA MEM-dataset construction idiom. - ---- - -## File Structure - -- `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/web/RST_TileXYZ.scala` — MODIFY. Add `-dstalpha` to the warp command; add private `toDisplayRGBA` helper; wire it into `executeWithScale` between warp and translate; JPEG path drops alpha. `RST_XYZPyramid.scala` is UNCHANGED (delegates per-tile to `executeWithScale`). -- `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/RST_TileXYZRgbaTest.scala` — CREATE. Heavy unit tests: per-source-band-count output shape, per-format band count, internal-NoData → 0-alpha, out-of-extent fallback still transparent. -- `python/geobrix/test/pyrx/test_cross_language_xyz_parity.py` — MODIFY (extend). Add the cross-tier RGBA parity test (decode + exact-alpha-position + tolerance-RGB) reusing the existing `spark_with_jar`/`heavy_registered` fixtures and `_decode_band`/`_make_uint16_narrow_bytes` helpers. -- `docs/docs/beta-release-notes.mdx` and `docs/docs/api/raster-functions.mdx` — MODIFY. Behavior-change note + band-mapping documentation. - ---- - -### Task 1: Warp emits binary alpha (`-dstalpha`) + `toDisplayRGBA` compositing helper - -Insert the RGBA compositing between warp and translate. This is the core change and carries its own heavy unit test. - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/web/RST_TileXYZ.scala` (`executeWithScale` ~L199-244; add helper + `-dstalpha` to the warp command L217) -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/RST_TileXYZRgbaTest.scala` (create) - -**Interfaces:** -- Consumes: existing `resolveScale(ds, rescale): String` (per-band `-scale` flags), `transparentPng(size)`, `GDALWarp.executeWarp`, `GDALTranslate.executeTranslate`, `RasterDriver.releaseDataset`. -- Produces: `private[web] def toDisplayRGBA(warpedDs: Dataset, format: String, scaleFlags: String): (Dataset, Boolean)` — returns a MEM `GDT_Byte` dataset with color-interp set (RGBA for PNG/WEBP, RGB for JPEG) plus a boolean = `true` when an alpha band is present. Caller encodes this dataset and releases it. - -- [ ] **Step 1: Write the failing heavy unit test** - -Create `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/RST_TileXYZRgbaTest.scala`: - -```scala -package com.databricks.labs.gbx.rasterx.expressions.web - -import com.databricks.labs.gbx.rasterx.gdal.RasterDriver -import com.databricks.labs.gbx.test.SparkSuite -import org.gdal.gdal.gdal -import org.gdal.gdalconst.gdalconstConstants -import org.scalatest.matchers.should.Matchers - -/** Heavy-tier RGBA output shape for rst_tilexyz. Byte-parity with the light tier - * is impossible (different encoders) and is NOT tested here; the cross-tier - * decode+tolerance parity test lives in the Python suite. These tests assert the - * HEAVY output STRUCTURE: band count / alpha per format / band count, and that an - * internal-NoData hole yields 0-alpha (the display bug being fixed). */ -class RST_TileXYZRgbaTest extends SparkSuite with Matchers { - - // Decode a PNG/WEBP byte array via GDAL (/vsimem) into a Dataset for band inspection. - private def openBytes(bytes: Array[Byte], ext: String) = { - val p = s"/vsimem/rgbatest_${java.util.UUID.randomUUID().toString.replace("-", "")}.$ext" - gdal.FileFromMemBuffer(p, bytes) - val ds = gdal.Open(p) - (ds, p) - } - - test("PNG output from a 1-band source is 4-band RGBA") { - val src = TileXYZTestFixtures.singleBandOverTile() // see Step 3 fixtures - try { - val png = RST_TileXYZ.execute( - src, Map.empty, TileXYZTestFixtures.z, TileXYZTestFixtures.x, - TileXYZTestFixtures.y, "PNG", 256, "near", "auto") - val (ds, p) = openBytes(png, "png") - try ds.GetRasterCount shouldBe 4 - finally { ds.delete(); gdal.Unlink(p) } - } finally RasterDriver.releaseDataset(src) - } - - test("PNG output from a 3-band source is 4-band RGBA") { - val src = TileXYZTestFixtures.threeBandOverTile() - try { - val png = RST_TileXYZ.execute( - src, Map.empty, TileXYZTestFixtures.z, TileXYZTestFixtures.x, - TileXYZTestFixtures.y, "PNG", 256, "near", "auto") - val (ds, p) = openBytes(png, "png") - try ds.GetRasterCount shouldBe 4 - finally { ds.delete(); gdal.Unlink(p) } - } finally RasterDriver.releaseDataset(src) - } - - test("JPEG output is 3-band RGB (no alpha)") { - val src = TileXYZTestFixtures.threeBandOverTile() - try { - val jpg = RST_TileXYZ.execute( - src, Map.empty, TileXYZTestFixtures.z, TileXYZTestFixtures.x, - TileXYZTestFixtures.y, "JPEG", 256, "near", "auto") - val (ds, p) = openBytes(jpg, "jpg") - try ds.GetRasterCount shouldBe 3 - finally { ds.delete(); gdal.Unlink(p) } - } finally RasterDriver.releaseDataset(src) - } - - test("internal-NoData hole yields a fully-transparent (alpha=0) region") { - val src = TileXYZTestFixtures.singleBandWithNoDataHole() // NoData square in the middle - try { - val png = RST_TileXYZ.execute( - src, Map.empty, TileXYZTestFixtures.z, TileXYZTestFixtures.x, - TileXYZTestFixtures.y, "PNG", 256, "near", "auto") - val (ds, p) = openBytes(png, "png") - try { - ds.GetRasterCount shouldBe 4 - // Read the alpha band (band 4); some pixels must be 0 (the hole) and some 255. - val alpha = ds.GetRasterBand(4) - val w = ds.GetRasterXSize; val h = ds.GetRasterYSize - val buf = Array.ofDim[Byte](w * h) - alpha.ReadRaster(0, 0, w, h, w, h, gdalconstConstants.GDT_Byte, buf) - val ints = buf.map(_ & 0xff) - ints.exists(_ == 0) shouldBe true // the NoData hole is transparent - ints.exists(_ == 255) shouldBe true // valid data is opaque - } finally { ds.delete(); gdal.Unlink(p) } - } finally RasterDriver.releaseDataset(src) - } -} -``` - -Also create the fixtures object `src/test/scala/com/databricks/labs/gbx/rasterx/expressions/web/TileXYZTestFixtures.scala`: - -```scala -package com.databricks.labs.gbx.rasterx.expressions.web - -import com.databricks.labs.gbx.rasterx.gdal.RasterDriver -import org.gdal.gdal.{Dataset, gdal} -import org.gdal.gdalconst.gdalconstConstants -import org.gdal.osr.SpatialReference - -/** In-memory GTiff fixtures for RST_TileXYZ tests, placed over a known WebMercator - * z=8 tile so execute() produces a data-carrying tile (not the transparent fallback). - * Footprint: lon 10..12, lat 48..50 (EPSG:4326) -- mirrors the Python parity fixture. */ -object TileXYZTestFixtures { - // z=8 tile covering lon~11, lat~49 (the fixture midpoint). Precomputed via morecantile. - val z = 8; val x = 134; val y = 86 - - private def wgs84Wkt: String = { - val srs = new SpatialReference(); srs.ImportFromEPSG(4326); srs.ExportToWkt() - } - - private def makeGeoTiff(nbands: Int, fill: (Int, Int, Int) => Int, - noDataBandVal: Option[Int] = None): Dataset = { - val w = 64; val h = 64 - val mem = gdal.GetDriverByName("MEM").Create("", w, h, nbands, gdalconstConstants.GDT_Byte) - // lon 10..12, lat 48..50 -> pixel size 2/64 in each axis (north-up). - mem.SetGeoTransform(Array(10.0, 2.0 / w, 0.0, 50.0, 0.0, -2.0 / h)) - mem.SetProjection(wgs84Wkt) - for (b <- 1 to nbands) { - val buf = Array.tabulate(w * h)(i => fill(b, i % w, i / w).toByte) - mem.GetRasterBand(b).WriteRaster(0, 0, w, h, w, h, gdalconstConstants.GDT_Byte, buf) - noDataBandVal.foreach(mem.GetRasterBand(b).SetNoDataValue(_)) - } - mem - } - - def singleBandOverTile(): Dataset = - makeGeoTiff(1, (_, px, py) => (px + py) % 200 + 20) - - def threeBandOverTile(): Dataset = - makeGeoTiff(3, (b, px, _) => (px * b) % 200 + 20) - - /** Single band with a NoData value of 0 and a 0-filled square in the center. */ - def singleBandWithNoDataHole(): Dataset = - makeGeoTiff(1, (_, px, py) => if (px >= 24 && px < 40 && py >= 24 && py < 40) 0 else (px + py) % 200 + 20, - noDataBandVal = Some(0)) -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run in Docker: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.web.RST_TileXYZRgbaTest' --log rgba-task1.log` -Expected: FAIL — the 1-band and 3-band PNG cases assert 4 bands but current output has 1 and 3 bands; the JPEG case may already pass (3-band source → 3-band); the NoData case fails (no alpha band today). - -- [ ] **Step 3: Add `-dstalpha` to the warp and the `toDisplayRGBA` helper** - -In `RST_TileXYZ.executeWithScale`, change the warp command (currently L217) to request a destination alpha band: - -```scala - val (warpedDs, warpedOpts) = GDALWarp.executeWarp( - warpPath, - Array(ds), - options ++ Map("format" -> "GTiff"), - command = s"gdalwarp -t_srs EPSG:3857 -te $xmin $ymin $xmax $ymax -ts $size $size -r $resampling -dstalpha" - ) -``` - -`-dstalpha` makes GDAL append a binary (0/255) alpha band derived from the source's valid-data mask: 255 where a source pixel maps in, 0 outside the footprint and at NoData pixels. So `warpedDs` now has `sourceBands + 1` bands, the last being alpha. - -Add the helper (place it near `transparentPng`, mirroring its MEM-dataset idiom): - -```scala - /** Build a display RGB(A) GDT_Byte MEM dataset from the warped tile, matching the - * rio-tiler band mapping. `warpedDs` has the source bands plus a trailing binary - * alpha band (from `-dstalpha`). Returns the MEM dataset ready to encode. - * - * Band mapping (rio-tiler parity), where N = source band count (warpedDs has N+1): - * N==1 -> grey replicated to R=G=B; N==2 -> band1 grey R=G=B, band2 as alpha; - * N==3 -> R,G,B; N==4 -> R,G,B + band4 alpha; N>=5 -> first 3 bands R,G,B. - * Alpha for PNG/WEBP is: the source's own alpha band (N in {2,4}) if present, else - * the warp's trailing -dstalpha band. JPEG drops alpha (3-band RGB). - * The `-scale` flags apply to RGB bands only (never alpha). */ - private[web] def toDisplayRGBA(warpedDs: Dataset, format: String, scaleFlags: String): Dataset = { - val total = warpedDs.GetRasterCount // = sourceBands + 1 (trailing -dstalpha) - val n = total - 1 // source band count - val w = warpedDs.GetRasterXSize; val h = warpedDs.GetRasterYSize - val wantAlpha = format.toUpperCase(Locale.ROOT) != "JPEG" - val outBands = if (wantAlpha) 4 else 3 - val mem = gdal.GetDriverByName("MEM").Create("", w, h, outBands, gdalconstConstants.GDT_Byte) - - // Choose the source band indices that become R,G,B, and the alpha source band. - val (rgbSrc, alphaSrc): (Seq[Int], Int) = n match { - case 1 => (Seq(1, 1, 1), total) // grey -> RGB; alpha = trailing -dstalpha band - case 2 => (Seq(1, 1, 1), 2) // band1 grey -> RGB; band2 IS alpha (rio-tiler) - case 3 => (Seq(1, 2, 3), total) // RGB; alpha = trailing -dstalpha band - case 4 => (Seq(1, 2, 3), 4) // RGB; band4 IS alpha - case _ => (Seq(1, 2, 3), total) // >=5: first 3 -> RGB; alpha = trailing band - } - - // Copy R,G,B (raw byte copy; the -scale rescale is applied at the translate step - // via scaleFlags, so we copy the ALREADY-warped-but-not-yet-rescaled bands and let - // translate rescale them -- see Step 4 note). Copy raw bytes band-for-band. - def copyBand(srcIdx: Int, dstIdx: Int): Unit = { - val buf = Array.ofDim[Byte](w * h) - warpedDs.GetRasterBand(srcIdx).ReadRaster(0, 0, w, h, w, h, gdalconstConstants.GDT_Byte, buf) - mem.GetRasterBand(dstIdx).WriteRaster(0, 0, w, h, w, h, gdalconstConstants.GDT_Byte, buf) - } - rgbSrc.zipWithIndex.foreach { case (srcIdx, i) => copyBand(srcIdx, i + 1) } - mem.GetRasterBand(1).SetColorInterpretation(gdalconstConstants.GCI_RedBand) - mem.GetRasterBand(2).SetColorInterpretation(gdalconstConstants.GCI_GreenBand) - mem.GetRasterBand(3).SetColorInterpretation(gdalconstConstants.GCI_BlueBand) - if (wantAlpha) { - copyBand(alphaSrc, 4) - mem.GetRasterBand(4).SetColorInterpretation(gdalconstConstants.GCI_AlphaBand) - } - mem.SetGeoTransform(warpedDs.GetGeoTransform()) - Option(warpedDs.GetProjection()).foreach(mem.SetProjection) - mem - } -``` - -IMPORTANT implementer note on rescale ordering: the source bands in `warpedDs` are the raw (possibly uint16) values; the `-scale` rescale must map them to Byte. Two correct orderings — pick the one that keeps `-scale` on RGB only: - (A) copy raw source bands into the MEM ds as their native dtype is NOT possible here (MEM ds is `GDT_Byte`); so instead, keep the MEM ds `GDT_Byte` and let the TRANSLATE step apply `-scale` — but translate would then rescale the alpha band too. To avoid that, prefer ordering (B). - (B) RECOMMENDED: create the MEM ds with the SAME dtype as the warped source bands for RGB, GDT_Byte for alpha — but mixed-dtype MEM bands are not allowed. Therefore the clean approach: apply the rescale to the RGB bands DURING the copy (compute the Byte value from the `scaleFlags` linear map in Scala), and copy the alpha band verbatim. Parse each `-scale lo hi 0 255` into (lo,hi) and map `byte = clamp(round((v-lo)/(hi-lo)*255), 0, 255)`; when `scaleFlags` is empty (uint8 auto / none) copy verbatim. Then the translate step needs NO `-scale`. - -Adopt ordering (B): add a `private def rescaleByteMap(scaleFlags: String, bandIndex: Int): Option[(Double, Double)]` that returns the (lo,hi) for a given RGB output band (the flags are per-band in order), and apply it in `copyBand` for RGB bands only. The implementer MUST make the cross-tier parity test (Task 3) pass — that test is the arbiter of correct rescale behavior; iterate the mapping until the decoded distributions match within tolerance. Do NOT weaken the parity tolerance. - -Now rewire `executeWithScale`'s translate section to composite first, then encode WITHOUT `-scale` (rescale already applied in the helper): - -```scala - try { - val extension = format.toLowerCase(Locale.ROOT) match { - case "png" => "png" - case "jpeg" => "jpg" - case "webp" => "webp" - case other => throw new IllegalArgumentException(s"rst_tilexyz: unknown format $other") - } - val rgbaDs = toDisplayRGBA(warpedDs, format, scaleFlags) - try { - val translatePath = s"/vsimem/tilexyz_out_$uuid.$extension" - val translateOpts = warpedOpts ++ Map("format" -> format, "extension" -> extension) - val (resDs, _) = GDALTranslate.executeTranslate( - translatePath, rgbaDs, command = "gdal_translate", translateOpts) - Try(resDs.FlushCache()); Try(resDs.delete()) - val bytes = gdal.GetMemFileBuffer(translatePath) - gdal.Unlink(translatePath) - if (bytes == null || bytes.isEmpty) transparentPng(size) else bytes - } finally RasterDriver.releaseDataset(rgbaDs) - } finally { - RasterDriver.releaseDataset(warpedDs) - } -``` - -Note: with rescale now applied in `toDisplayRGBA`, `OperatorOptions`'s PNG branch (`-ot Byte -a_nodata none$scaleSuffix`) receives no `scale` key, so `scaleSuffix` is empty — the MEM ds is already Byte RGBA and encodes directly. The `-a_nodata none` stays (harmless; alpha carries transparency now). - -- [ ] **Step 4: Run the heavy unit test to verify it passes** - -Run in Docker: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.web.RST_TileXYZRgbaTest' --log rgba-task1.log` -Expected: PASS — 1-band and 3-band PNG → 4 bands; JPEG → 3 bands; internal-NoData → alpha band has both 0 and 255. - -- [ ] **Step 5: Run the existing heavy XYZ suite (no regression)** - -Run in Docker: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.web.*' --log rgba-task1-regress.log` - -KNOWN INTERACTION — `XYZRescaleParityTest` will break and must be updated in THIS task. It asserts on the `resolveScale` FLAG STRING structure (verified: `RST_TileXYZ.scala` context, that test at ~L182 `segments.length shouldBe 3` and ~L189 `parts.length shouldBe 4 // lo hi 0 255`). Those assertions still hold IF you keep `resolveScale` producing the same per-band `-scale lo hi 0 255` string — which ordering (B) does: `resolveScale` is unchanged and still called; you PARSE its output inside `toDisplayRGBA` (via `rescaleByteMap`) instead of passing it to translate. So `resolveScale`'s output (what `XYZRescaleParityTest` inspects) is unchanged and those assertions stay green. If any assertion in that suite instead inspects the OUTPUT tile's band count, update it to expect RGBA (PNG→4, JPEG→3) and add a one-line comment pointing at this plan. Do NOT delete assertions — adjust the expected value. -Expected: PASS after any such adjustment. - -- [ ] **Step 6: Scalastyle + commit** - -Run in Docker: `bash scripts/commands/gbx-lint-scalastyle.sh` (or `gbx:lint:scalastyle`) -Expected: clean. -Commit (message to a temp file, `git commit -F`): subject `feat(rasterx): heavy rst_tilexyz emits RGBA via warp -dstalpha + band mapping`. - ---- - -### Task 2: WEBP alpha support detection + JPEG/WEBP wiring - -Handle the WEBP-alpha capability check and confirm JPEG/WEBP paths. - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/expressions/web/RST_TileXYZ.scala` (`toDisplayRGBA` / format branch) -- Test: `src/test/scala/.../web/RST_TileXYZRgbaTest.scala` (extend) - -**Interfaces:** -- Consumes: `toDisplayRGBA` from Task 1. -- Produces: WEBP emits RGBA when the driver supports alpha, else RGB; JPEG always RGB. - -- [ ] **Step 1: Write the failing WEBP/JPEG shape tests** - -Append to `RST_TileXYZRgbaTest.scala`: - -```scala - test("WEBP output is RGBA when the driver supports alpha, else RGB") { - val src = TileXYZTestFixtures.threeBandOverTile() - try { - val webp = RST_TileXYZ.execute( - src, Map.empty, TileXYZTestFixtures.z, TileXYZTestFixtures.x, - TileXYZTestFixtures.y, "WEBP", 256, "near", "auto") - val (ds, p) = openBytes(webp, "webp") - try { - val nb = ds.GetRasterCount - // 4 (alpha-capable build) or 3 (fallback) -- both acceptable; never the source's raw N. - (nb == 4 || nb == 3) shouldBe true - } finally { ds.delete(); gdal.Unlink(p) } - } finally RasterDriver.releaseDataset(src) - } -``` - -- [ ] **Step 2: Run to verify it fails or is inconclusive** - -Run in Docker: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.web.RST_TileXYZRgbaTest' --log rgba-task2.log` -Expected: the WEBP test either passes (if Task 1 already produces 4-band WEBP and the driver accepts it) or fails at encode (driver rejects 4-band) — the latter is what Step 3 fixes. - -- [ ] **Step 3: Add the WEBP-alpha capability check** - -In `toDisplayRGBA`, gate `wantAlpha` for WEBP on driver capability. GDAL's WEBP driver advertises alpha via its metadata; probe once: - -```scala - private lazy val webpSupportsAlpha: Boolean = - Try { - val drv = gdal.GetDriverByName("WEBP") - drv != null && { - val md = drv.GetMetadataItem("DMD_CREATIONOPTIONLIST") - md != null // WEBP driver present with creation options => alpha-capable in practice - } - }.getOrElse(false) -``` - -Then in `toDisplayRGBA`: - -```scala - val fmtU = format.toUpperCase(Locale.ROOT) - val wantAlpha = fmtU match { - case "JPEG" => false - case "WEBP" => webpSupportsAlpha - case _ => true // PNG - } -``` - -If WEBP does not support alpha, `wantAlpha=false` → 3-band RGB output. Add a one-line log (use the existing logging idiom in the package, e.g. an `RST_ErrorHandler`/logger already imported) noting the RGB fallback. If no logger is readily available, a scaladoc note on the branch suffices — do NOT add a new logging dependency. - -- [ ] **Step 4: Run the test to verify it passes** - -Run in Docker: `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.expressions.web.RST_TileXYZRgbaTest' --log rgba-task2.log` -Expected: PASS (4 or 3 band WEBP accepted). - -- [ ] **Step 5: Scalastyle + commit** - -Run: `bash scripts/commands/gbx-lint-scalastyle.sh` -Commit: `feat(rasterx): WEBP alpha-capability gate + RGB fallback for rst_tilexyz`. - ---- - -### Task 3: Cross-tier RGBA parity test (decode + tolerance) - -The primary gate: prove heavy now matches light on shape, alpha positions, and RGB distribution. - -**Files:** -- Modify: `python/geobrix/test/pyrx/test_cross_language_xyz_parity.py` (extend — reuse its fixtures + `spark_with_jar`/`heavy_registered` + `_decode_band`) - -**Interfaces:** -- Consumes (existing in that file): `spark_with_jar`, `heavy_registered`, `_make_uint16_narrow_bytes`, `_center_tile_zxy`, `xyz.render_tile`, `rx.rst_tilexyz`, `rx.rst_fromcontent`. -- Produces: two new tests exercising RGBA shape + alpha-position parity + RGB-distribution tolerance. - -- [ ] **Step 1: Write the failing parity test (append to the file)** - -Add a NoData-hole fixture builder and the parity tests: - -```python -def _make_uint16_with_nodata_hole(width=64, height=64, lo=8000, hi=12000, nodata=0): - """uint16 ramp with a NoData square in the center (tests internal-NoData transparency).""" - transform = from_origin(10.0, 50.0, 0.03125, 0.03125) - profile = dict(driver="GTiff", width=width, height=height, count=1, - dtype="uint16", crs="EPSG:4326", transform=transform, nodata=nodata) - ramp = np.linspace(lo, hi, width * height).astype("uint16").reshape(height, width) - ramp[height // 3:2 * height // 3, width // 3:2 * width // 3] = nodata # hole - with MemoryFile() as mf: - with mf.open(**profile) as ds: - ds.write(ramp, 1) - return mf.read() - - -def _decode_rgba(png_bytes): - """Decode to a full HxWx4 uint8 RGBA array (alpha as the 4th channel).""" - from PIL import Image - img = Image.open(io.BytesIO(png_bytes)).convert("RGBA") - return np.asarray(img) - - -def test_light_vs_heavy_rgba_shape_and_alpha_parity(heavy_registered): - """Both tiers emit RGBA PNGs whose transparent-pixel positions match exactly, - and whose RGB channels agree within tolerance, on a source with internal NoData.""" - from pyspark.sql import functions as f - from databricks.labs.gbx.rasterx import functions as rx - - spark = heavy_registered - raster_bytes = _make_uint16_with_nodata_hole() - - with MemoryFile(raster_bytes) as mf, mf.open() as ds: - z, x, y = _center_tile_zxy(ds) - with MemoryFile(raster_bytes) as mf, mf.open() as ds: - light_png = xyz.render_tile(ds, z, x, y, rescale="auto") - - df = spark.range(1).select( - rx.rst_tilexyz( - rx.rst_fromcontent(f.lit(raster_bytes), f.lit("GTiff")), - z, x, y, "PNG", 256, "near", "auto").alias("bytes")) - heavy_png = bytes(df.collect()[0]["bytes"]) - - light = _decode_rgba(light_png) - heavy = _decode_rgba(heavy_png) - - # Same dimensions and 4-band RGBA. - assert light.shape == heavy.shape, f"shape mismatch light={light.shape} heavy={heavy.shape}" - assert light.shape[2] == 4, "light not RGBA" - assert heavy.shape[2] == 4, "heavy not RGBA" - - # (a) Exact alpha-position parity: the set of transparent pixels is identical. - light_transparent = light[..., 3] == 0 - heavy_transparent = heavy[..., 3] == 0 - # There MUST be a transparent hole (the NoData square) and opaque data around it. - assert light_transparent.any() and (~light_transparent).any(), "light has no alpha variation" - assert heavy_transparent.any() and (~heavy_transparent).any(), "heavy has no alpha variation" - # Allow a thin disagreement fringe from warp-resampling edges (<=1% of pixels). - disagree = int(np.sum(light_transparent != heavy_transparent)) - frac = disagree / light_transparent.size - print(f"\n[rgba] alpha disagreement {disagree}/{light_transparent.size} = {frac:.4f}") - assert frac <= 0.01, ( - f"alpha-position parity: {frac:.4f} of pixels disagree on transparency " - f"(tolerance 0.01) -- heavy and light derive different NoData masks") - - # (b) RGB distribution within tolerance over the OPAQUE (data) pixels of each tier. - qs = (0.05, 0.25, 0.5, 0.75, 0.95) - light_rgb = light[..., 0][~light_transparent].astype(float) - heavy_rgb = heavy[..., 0][~heavy_transparent].astype(float) - light_q = np.quantile(light_rgb, qs) - heavy_q = np.quantile(heavy_rgb, qs) - max_q_diff = float(np.max(np.abs(light_q - heavy_q))) - print(f"[rgba] light R quantiles={light_q.round(1)} heavy={heavy_q.round(1)} maxdiff={max_q_diff:.1f}") - assert max_q_diff <= 20, ( - f"cross-tier RGB quantile mismatch {max_q_diff:.1f} > 20 " - f"(light {light_q.round(1)} vs heavy {heavy_q.round(1)})") -``` - -- [ ] **Step 2: Run to verify it fails (before the Task 1/2 JAR is staged) or passes (after)** - -The heavy side needs the freshly-built JAR staged under `python/geobrix/lib/`. Build + stage first: -Run in Docker: `bash scripts/commands/gbx-docker-exec.sh 'mvn clean package -PskipScoverage -DskipTests'` then copy `target/geobrix-*-jar-with-dependencies.jar` to `python/geobrix/lib/`. -Then run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/pyrx/test_cross_language_xyz_parity.py --with-integration --log rgba-parity.log` -Expected before the JAR carries Task 1/2: FAIL (heavy PNG is not RGBA / alpha positions differ). After staging the new JAR: PASS. - -- [ ] **Step 3: If the parity test fails on RGB distribution, fix the rescale mapping in `toDisplayRGBA`** - -This is the arbiter of Task 1's rescale-during-copy math. If quantiles diverge > 20, the Scala `rescaleByteMap` is wrong — fix the (lo,hi) parse / linear map to match `resolveScale`'s `-scale lo hi 0 255` semantics exactly. Do NOT loosen the tolerance. Rebuild + re-stage the JAR and re-run. - -- [ ] **Step 4: Commit** - -Commit: `test(pyrx): cross-tier RGBA shape + alpha-position + RGB parity for rst_tilexyz`. - ---- - -### Task 4: Docs — behavior change + band mapping - -**Files:** -- Modify: `docs/docs/beta-release-notes.mdx` (in-flight list) -- Modify: `docs/docs/api/raster-functions.mdx` (`rst_tilexyz` entry) - -**Interfaces:** none (documentation). - -- [ ] **Step 1: Add the beta release note** - -In `docs/docs/beta-release-notes.mdx`, under the current in-flight "What's new" list, add: - -```markdown -- **Heavy XYZ tiles now emit RGBA to match the lightweight tier (behavior change).** `gbx_rst_tilexyz` and `gbx_rst_xyzpyramid` on the heavyweight tier now produce display RGB(A) web-map tiles: **PNG and WEBP → RGBA**, **JPEG → RGB**, with a binary alpha channel derived from the source's valid-data mask. Previously the heavy tier emitted the source's raw band count with no alpha, so a tile with internal NoData rendered opaque (black) where the lightweight (`pyrx`) tier renders it transparent — the two tiers now agree. Band mapping matches the lightweight tier: a single-band source becomes greyscale RGB, three bands become RGB, and an existing alpha/4th band is preserved. This changes the heavy output band count (e.g. a 1-band source now yields a 4-band RGBA PNG); consumers that read heavy `gbx_rst_tilexyz` bytes as a raw single-band raster rather than a display tile are affected. WEBP alpha requires GDAL WEBP-alpha support in the runtime; where absent, WEBP falls back to RGB. See [Raster Functions](./api/raster-functions). -``` - -- [ ] **Step 2: Update the `rst_tilexyz` function doc** - -In `docs/docs/api/raster-functions.mdx`, find the `rst_tilexyz` (and `rst_xyzpyramid`) entry and add a note describing: display RGB(A) output, the band-mapping table (1→grey RGB, 3→RGB, 4→RGB+alpha, ≥5→first 3), binary alpha from NoData, PNG/WEBP RGBA vs JPEG RGB, and the WEBP-alpha-fallback caveat. Match the surrounding entry's prose style. - -- [ ] **Step 3: Internal-vocabulary + link check** - -Run: `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/api/raster-functions.mdx docs/docs/beta-release-notes.mdx` → expect no output. -Verify the `./api/raster-functions` link and any anchor resolve. - -- [ ] **Step 4: Commit** - -Commit: `docs: heavy XYZ RGBA output behavior change + band mapping`. - ---- - -## Self-Review - -- **Spec coverage:** §2 scope (PNG/WEBP RGBA, JPEG RGB, xyzpyramid inherits) → Tasks 1–2; §4.1 band mapping → Task 1 `toDisplayRGBA` + Task 1 tests; §4.2 binary alpha from warp mask → Task 1 `-dstalpha`; §4.4 WEBP fallback → Task 2; §5 behavior change/docs → Task 4; §6 decode+tolerance parity (exact alpha positions + tolerance RGB, internal-NoData fixture) → Task 3; §6 heavy unit tests per band count/format → Task 1–2; bench untouched → honored (no bench task). Covered. -- **Placeholder scan:** no TBD/TODO; all code shown incl. fixtures and the rescale-ordering decision (ordering B, made explicit). The one deliberately-iterative point (the exact `rescaleByteMap` linear-map math) is gated by the Task 3 parity test, which is named as the arbiter — not a placeholder but a test-driven convergence. -- **Type consistency:** `toDisplayRGBA(warpedDs, format, scaleFlags): Dataset` used consistently; `execute(...)` 9-arg signature matches the real current signature (verified against the file); `_decode_rgba`/`_decode_band` distinct helpers; fixture builders named consistently across Scala (`TileXYZTestFixtures.*`) and Python (`_make_uint16_*`). -- **Note on ordering (B):** the plan resolves the rescale-vs-alpha interaction by applying the RGB rescale during the band copy (so `-scale` never touches alpha) rather than at the translate step. This deviates from the spec's looser "then encode" sketch but satisfies the spec's hard constraint (§4.1: alpha never rescaled) and is the reviewable, correct ordering. Flagged here for the executor. diff --git a/docs/superpowers/plans/2026-07-27-netcdf-gridonly-enum-and-benchmark.md b/docs/superpowers/plans/2026-07-27-netcdf-gridonly-enum-and-benchmark.md deleted file mode 100644 index d2e51cc29..000000000 --- a/docs/superpowers/plans/2026-07-27-netcdf-gridonly-enum-and-benchmark.md +++ /dev/null @@ -1,451 +0,0 @@ -# NetCDF grid-only enumeration + scale/offset parity + at-scale benchmark — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make heavy `netcdf_gdal` enumerate only true georeferenced grid variables (kill the 85-subdataset fan-out and match the light `classify()` set), make it apply CF `scale_factor`/`add_offset` so its values match light on scaled grids, add a regular-grid raster corpus downloader (NASA-NEX via anonymous Planetary Computer), and split the reader benchmark into an honest raster (heavy-vs-light) leg and a light-only vector leg. - -**Architecture:** Three areas. (1) Scala reader: `NetCDF_Batch` gains a geotransform/CRS grid filter; `WindowedExtract` gains an opt-in `applyScale` (netcdf-only) that unscales via the proven `GDALTranslate` fallback; `NetCDF_Reader` opts in. (2) Python sample: a `NasaNexDownloader` mirroring `TropomiDownloader`. (3) Python bench: `run_format_read` passes `sizeInMB` to `netcdf_gdal`; the bench cell splits into `{CORPUS}/netcdf` (NASA-NEX raster) + `{CORPUS}/netcdf-swath` (S5P vector); the top-level `corpus.json` read is made lazy. - -**Tech Stack:** Scala 2.13.16 / Spark 4.0.0 / Java 17 / GDAL Java bindings (heavy); Python 3.12 / PySpark / xarray / pystac-client + planetary-computer (light + downloader). Tests: ScalaTest (`PlanTest with SilentSparkSession`) in Docker; pytest (local Spark) for light + downloader; cluster jobs.submit for the at-scale bench (human-gated). - -## Global Constraints - -- **No aliases** — one canonical name per reader/function. -- **GDAL/OGR registration only via `GDALManager` guards** (`GDALManager.init`); never raw `gdal.AllRegister()` per task. The executor path must init `NodeFileManager.init(exprConfig.hConf)` before `readRemote` (already fixed in `7ba0d2dd`; do not regress). -- **`WindowedExtract` is shared by ALL raster readers.** The `applyScale` change MUST default to `false` and only `NetCDF_Reader` may set it true. GeoTIFF/other readers keep raw-copy behavior byte-for-byte. -- **Heavy raster schema is fixed:** `struct>`. Do not change. -- **Serverless-safe light tier + downloader:** no `spark.conf.set`, `_jvm`, `.rdd`, `cache`, `persist`. Downloaders discover on the driver (metadata-only) and fan out via `StacClient.download`. -- **Heavy work runs in the `geobrix-dev` Docker container** via `gbx:*` commands; dispatch long Scala/Maven suites to a subagent, never inline. -- **Bench discipline** (`benchmarking-preflight-discipline`): non-empty corpus, logged granule counts (no silent truncation), stamped worker count, `summary.md` link at end. Cluster is the user's `0519-143423-0jwqt79u`; it is TERMINATED — the build/stage/bench tasks must (re)start it and stage the fat JAR + tests.jar BEFORE start (`jar-stage-before-cluster-start`), then poll RUNNING + libs INSTALLED before submitting (`poll-cluster-start-and-libs`). -- **Verified data facts:** S5P `/PRODUCT/methane_mixing_ratio` = 215×3736, no geotransform, no CRS, GEOLOCATION array (swath). coral `bleaching_alert_area` = 7200×3600 with geotransform (regular grid). NASA-NEX GDDP-CMIP6 = `application/netcdf`, regular 0.25° grid, anonymous on PC. -- **`bench-changes-update-docs`:** any bench change reflected in `docs/docs/api/benchmarking.mdx` the same cycle. **No internal vocabulary** in `docs/docs/` (QC judge greps `wave\s*\d+`). - ---- - -### Task 1: `NetCDF_Batch` grid-only enumeration filter - -Replace the "keep any >1×1 subdataset" test with "keep only true georeferenced grids" so S5P's 85 subdatasets collapse to 0 (matching light) and gridded files keep their grid variables. - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/ds/netcdf/NetCDF_Batch.scala:50-59` (the `grids` filter) -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/ds/NetCDF_DataSourceTest.scala` - -**Interfaces:** -- Consumes: `gdal.Open(selector, GA_ReadOnly)`, `Dataset.GetGeoTransform(Array[Double])`, `Dataset.GetProjectionRef`/`GetProjection`. -- Produces: enumeration keeps a subdataset iff `XSize>1 && YSize>1 && RasterCount>=1` AND (`GetProjectionRef` non-empty OR geotransform is non-identity). Same `(path, var)` partition shape as before. - -- [ ] **Step 1: Write the failing test (swath enumerates 0, grid enumerates its vars)** - -Add to `NetCDF_DataSourceTest.scala`. The coral fixture is a real grid (2 vars); there is no committed swath fixture, so build a tiny synthetic swath `.nc` in-test via a helper, OR assert on the existing grid fixture that the count is exactly its grid vars (2) and unchanged. Use the coral integration fixture for the positive case: - -```scala -test("netcdf_gdal enumerates only georeferenced grid variables (coral grid = 2)") { - import com.databricks.labs.gbx.rasterx.functions._ - rasterx.functions.register(spark) - val ncDir = this.getClass.getResource("/binary/netcdf-coral/").toString - val df = spark.read.format("netcdf_gdal").option("sizeInMB", "-1") - .option("filterRegex", ".*20220101\\.nc$").load(ncDir) - val vars = df.select("source").collect().map(_.getString(0).split(":").last).toSet - vars shouldBe Set("bleaching_alert_area", "mask") -} -``` - -For the swath-negative case, stage a synthetic swath fixture (Task 6 stages a real one; here use a small generated `.nc` with 2-D lat/lon and no geotransform). If generating in-test is impractical in Scala, defer the swath assertion to the Python cross-tier test (Task 4) and note it here. Minimum: the coral grid test above must pass and lock the georeferenced-only behavior. - -- [ ] **Step 2: Run to verify (coral still 2; pre-change this passes, so add a guard the change preserves)** - -Run (subagent, Docker): `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.ds.NetCDF_DataSourceTest' --log netcdf-gridfilter.log` -Expected: PASS on coral (coral vars have geotransform, so both old and new filter keep them — this test guards against the change dropping real grids). - -- [ ] **Step 3: Implement the georeferenced-grid filter** - -In `NetCDF_Batch.scala`, replace the `grids` filter body: - -```scala -val grids = vars.filter { v => - try { - val sub = gdal.Open(s"""NETCDF:"$localPath":$v""", GA_ReadOnly) - if (sub == null) false - else { - val bigEnough = sub.GetRasterXSize > 1 && sub.GetRasterYSize > 1 && sub.GetRasterCount >= 1 - // A true raster grid has real georeferencing: a CRS or a non-identity geotransform. - // Swath subdatasets (e.g. S5P /PRODUCT/methane_mixing_ratio) report an empty - // projection AND the identity transform [0,1,0,0,0,1] (their georeferencing lives in - // a GEOLOCATION array), so they are correctly dropped here — matching the light - // classify() which returns CURVILINEAR and excludes them from raster mode. - val hasCrs = { val p = sub.GetProjectionRef; p != null && p.nonEmpty } - val gt = new Array[Double](6); sub.GetGeoTransform(gt) - val identity = gt(0) == 0.0 && gt(1) == 1.0 && gt(2) == 0.0 && - gt(3) == 0.0 && gt(4) == 0.0 && gt(5) == 1.0 - val ok = bigEnough && (hasCrs || !identity) - sub.delete(); ok - } - } catch { case _: Throwable => false } -} -``` - -- [ ] **Step 4: Run to green** - -Run (subagent): `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.ds.NetCDF_DataSourceTest' --log netcdf-gridfilter.log` -Expected: PASS (coral still enumerates its 2 grid vars; the georeference test now locks the behavior). - -- [ ] **Step 5: Scalastyle + commit** - -Run (subagent): `bash scripts/commands/gbx-lint-scalastyle.sh` (confirm no new violations in NetCDF_Batch). - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/ds/netcdf/NetCDF_Batch.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/ds/NetCDF_DataSourceTest.scala -git commit -m "fix(rasterx): netcdf_gdal enumerates only georeferenced grid variables - -A subdataset is kept only when it has a CRS or a non-identity geotransform, -not merely >1x1. S5P swaths (85 subdatasets, no geotransform, GEOLOCATION -array) now enumerate to zero grid variables -- matching the light classify() -contract and eliminating the ~85-partitions-per-file fan-out that made the -reader unusable at scale. Regular grids (coral/CMIP/NASA-NEX) are unaffected. - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: opt-in `applyScale` in `WindowedExtract` (cross-tier value parity) - -Make heavy apply CF scale/offset when the reader opts in, so `netcdf_gdal` values match light's decoded physical values. Use the proven `GDALTranslate` fallback with `-unscale -ot Float64` when any band has non-identity scale/offset — avoids hand-rolling per-dtype buffer math and reuses the correct path. Default OFF; GeoTIFF unchanged. - -**Files:** -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/operations/WindowedExtract.scala` (branch at top of `extract`) -- Modify: `src/main/scala/com/databricks/labs/gbx/rasterx/ds/netcdf/NetCDF_Reader.scala:35` (pass `Map("applyScale" -> "true")` into `splitRasterIter`) -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/ds/NetCDF_DataSourceTest.scala` (GeoTIFF-unchanged regression) - -**Interfaces:** -- Consumes: `options: Map[String,String]` already threaded `NetCDF_Reader → BalancedSubdivision.splitRasterIter → ReTile.reTileIter → getTile → WindowedExtract.extract`. `GDALTranslate.executeTranslate` (existing fallback). `Band.GetScale`/`GetOffset`. -- Produces: when `options.getOrElse("applyScale","false").toBoolean` AND some band has `GetScale != 1.0` or `GetOffset != 0.0`, the extracted tile carries **decoded Float64** values (raw*scale+offset), NoData mapped to the decoded fill; scale/offset NOT re-copied as metadata (already applied). Otherwise behavior is byte-for-byte unchanged. - -- [ ] **Step 1: Write the failing GeoTIFF-unchanged regression test** - -This proves `applyScale` defaults off — a GeoTIFF read is unaffected. Add to `NetCDF_DataSourceTest.scala` (or a WindowedExtract-focused suite): - -```scala -test("gtiff_gdal read is unaffected by applyScale default (raw values preserved)") { - import com.databricks.labs.gbx.rasterx.functions._ - rasterx.functions.register(spark) - val tif = this.getClass.getResource("/modis/").toString - val df = spark.read.format("gtiff_gdal").option("sizeInMB", "1").load(tif).limit(1) - // Reading succeeds and produces a tile; no applyScale option is set anywhere. - df.count() shouldBe 1L -} -``` - -(The stronger scaled-value parity assertion is the Python cross-tier test in Task 4; this Scala test guards the shared-code default.) - -- [ ] **Step 2: Run to verify it passes pre-change (guard test)** - -Run (subagent): `bash scripts/commands/gbx-test-scala.sh --suite 'com.databricks.labs.gbx.rasterx.ds.NetCDF_DataSourceTest' --log netcdf-applyscale.log` -Expected: PASS (no behavior change yet; this test must stay green after Step 3). - -- [ ] **Step 3: Implement `applyScale` branch in `WindowedExtract.extract`** - -At the top of `extract`, before `simpleEnough`, add the opt-in unscale path: - -```scala -def extract(ds: Dataset, options: Map[String, String], - xStart: Int, yStart: Int, xOffset: Int, yOffset: Int): (Dataset, Map[String, String]) = { - val applyScale = options.getOrElse("applyScale", "false").toBoolean - if (applyScale && hasNonIdentityScale(ds)) { - // Decode CF scale_factor/add_offset to physical Float64 values, matching the light - // netcdf_gbx reader (xarray mask_and_scale=True). Use the proven gdal_translate path - // with -unscale -ot Float64 so per-dtype packing/nodata handling is GDAL's, not ours. - return GDALTranslate.executeTranslate( - ds, - options + ("translateOptions" -> - s"-srcwin $xStart $yStart $xOffset $yOffset -unscale -ot Float64"), - /* match the existing fallback's call shape */ ) - } - if (!simpleEnough(ds)) return fallback(ds, options, xStart, yStart, xOffset, yOffset) - // ... existing fast path unchanged ... -} - -private def hasNonIdentityScale(ds: Dataset): Boolean = { - val n = ds.getRasterCount - (1 to n).exists { b => - val sb = ds.GetRasterBand(b) - val s = new Array[java.lang.Double](1); sb.GetScale(s) - val o = new Array[java.lang.Double](1); sb.GetOffset(o) - (s(0) != null && s(0).doubleValue() != 1.0) || (o(0) != null && o(0).doubleValue() != 0.0) - } -} -``` - -NOTE: match `GDALTranslate.executeTranslate`'s actual signature — read `src/main/scala/com/databricks/labs/gbx/rasterx/operator/GDALTranslate.scala` and `WindowedExtract.fallback` to see exactly how the existing `-srcwin` fallback constructs its options/command, and mirror it (the existing `fallback` already builds a `-srcwin` translate — extend its options string with `-unscale -ot Float64` rather than inventing a new call). If `fallback` already accepts a srcwin, the cleanest implementation is: `if (applyScale && hasNonIdentityScale(ds)) return fallback(ds, options + ("unscale"->"true"), ...)` and have `fallback` append `-unscale -ot Float64` when that option is set. - -- [ ] **Step 4: Wire `NetCDF_Reader` to opt in** - -`NetCDF_Reader.scala:35` — pass the option into tiling: - -```scala -private val tilesIter = BalancedSubdivision.splitRasterIter(ds, Map("applyScale" -> "true"), partition.sizeInMB) -``` - -- [ ] **Step 5: Run Scala suite to green** - -Run (subagent): `bash scripts/commands/gbx-test-scala.sh --suites 'com.databricks.labs.gbx.rasterx.ds.NetCDF_DataSourceTest,com.databricks.labs.gbx.rasterx.ds.GTiff_DataSourceTest,com.databricks.labs.gbx.rasterx.ds.GDAL_DataSourceTest' --log netcdf-applyscale.log` -Expected: PASS — netcdf tests green, GeoTIFF/GDAL readers unchanged (applyScale off). - -- [ ] **Step 6: Scalastyle + commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/operations/WindowedExtract.scala \ - src/main/scala/com/databricks/labs/gbx/rasterx/ds/netcdf/NetCDF_Reader.scala \ - src/test/scala/com/databricks/labs/gbx/rasterx/ds/NetCDF_DataSourceTest.scala -git commit -m "feat(rasterx): opt-in applyScale so netcdf_gdal decodes CF scale/offset - -WindowedExtract gains an opt-in applyScale (default false): when set and a -band carries non-identity scale_factor/add_offset, the tile is decoded to -physical Float64 via gdal_translate -unscale, matching the light netcdf_gbx -reader (xarray mask_and_scale). NetCDF_Reader opts in; all other raster -readers keep raw-copy behavior byte-for-byte (regression-tested on gtiff_gdal). - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: `NasaNexDownloader` (regular-grid raster corpus) - -Mirror `TropomiDownloader` for `nasa-nex-gddp-cmip6` — anonymous PC, regular 0.25° grids. - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/sample/nasanex.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/sample/__init__.py` (export `NasaNexDownloader`, `download_nasanex_aoi`) -- Test: `python/geobrix/test/sample/test_nasanex.py` - -**Interfaces:** -- Consumes: `databricks.labs.gbx.stac.StacClient` (same as tropomi), `_stac_client` injection seam. -- Produces: `NasaNexDownloader(catalog=PLANETARY_COMPUTER, collection="nasa-nex-gddp-cmip6", sign="planetary_computer").download(bbox, out_dir, temporal=None, variables=("tas",), spark=None) -> DataFrame` with columns `out_file_path, out_file_sz, is_out_file_valid`; saves `{item_id}_{asset}.nc`. `download_nasanex_aoi(spark, bbox, out_dir, **kw)` wrapper. - -- [ ] **Step 1: Read the template** - -Read `python/geobrix/src/databricks/labs/gbx/sample/tropomi.py` in full and `python/geobrix/test/sample/` (find the tropomi test) to copy the injection-seam + test pattern exactly. - -- [ ] **Step 2: Write the failing downloader test (injected STAC client)** - -`python/geobrix/test/sample/test_nasanex.py` — mirror the tropomi test: inject a fake `_stac_client` returning a canned item with `pr`/`tas` netcdf assets, assert `download` filters to the requested `variables` and calls `StacClient.download` with the right hrefs. No network. - -```python -def test_nasanex_download_filters_to_requested_variables(monkeypatch, tmp_path): - from databricks.labs.gbx.sample.nasanex import NasaNexDownloader - dl = NasaNexDownloader() - # inject fake client (mirror tropomi test's seam); assert only 'tas' asset is downloaded - ... - assert selected_assets == ["tas"] -``` - -- [ ] **Step 3: Run to verify failure** - -Run: `gbx:test:python --path python/geobrix/test/sample/test_nasanex.py` -Expected: FAIL — module not found. - -- [ ] **Step 4: Implement `nasanex.py`** - -Copy `tropomi.py`'s structure; swap `S5P_COLLECTION` → `"nasa-nex-gddp-cmip6"`, drop the `/PRODUCT` group and swath-specific bits, and filter assets by the `variables` tuple (asset names are the climate-var ids `tas`, `pr`, ...). Save `{item_id}_{asset}.nc`. Keep it Serverless-safe (driver discovery + `StacClient.download` fan-out; no spark config mutation). Add `download_nasanex_aoi`. - -- [ ] **Step 5: Run to green + export** - -Add exports to `sample/__init__.py`. Run: `gbx:test:python --path python/geobrix/test/sample/test_nasanex.py` -Expected: PASS. - -- [ ] **Step 6: Lint + commit** - -Run: `gbx:lint:python --fix` then verify Docker `--check`. - -```bash -git add python/geobrix/src/databricks/labs/gbx/sample/nasanex.py \ - python/geobrix/src/databricks/labs/gbx/sample/__init__.py \ - python/geobrix/test/sample/test_nasanex.py -git commit -m "feat(sample): NasaNexDownloader for regular-grid NetCDF raster corpus - -Mirrors TropomiDownloader for nasa-nex-gddp-cmip6 (anonymous Planetary -Computer): AOI/temporal-driven discovery + distributed StacClient.download -of the requested climate variables as {item_id}_{asset}.nc regular-grid -granules. Provides the at-scale gridded raster corpus for the heavy-vs-light -netcdf reader benchmark (S5P swaths cannot serve raster). - -Co-authored-by: Isaac" -``` - ---- - -### Task 4: cross-tier parity — scaled-grid case - -Extend the existing parity test so it gates the Task 2 unscaling on a scaled grid, and asserts swath → empty on both tiers. - -**Files:** -- Modify: `python/geobrix/test/ds/test_netcdf_cross_tier.py` -- Test fixture: a synthetic packed-integer `.nc` with `scale_factor`/`add_offset` (built in-test with `netCDF4`, like the existing `_write_regular_grid` helpers). - -**Interfaces:** -- Consumes: both registered readers; `_netcdf.readable_variables`. Produces: assertions only. - -- [ ] **Step 1: Write the scaled-grid parity test** - -```python -def _write_scaled_grid(path): - from netCDF4 import Dataset - import numpy as np - with Dataset(path, "w") as ds: - ds.createDimension("lat", 4); ds.createDimension("lon", 5) - lat = ds.createVariable("lat","f8",("lat",)); lat.standard_name="latitude" - lon = ds.createVariable("lon","f8",("lon",)); lon.standard_name="longitude" - lat[:] = [50.0,49.5,49.0,48.5]; lon[:] = [10.0,10.5,11.0,11.5,12.0] - v = ds.createVariable("t","i2",("lat","lon"), fill_value=-32768) - v.scale_factor = 0.01; v.add_offset = 250.0 - v[:] = np.arange(20, dtype="i2").reshape(4,5) # physical = raw*0.01 + 250 - -@pytest.mark.integration -def test_netcdf_gdal_applies_scale_matches_light(spark, tmp_path): - if not _heavy_available(spark): pytest.skip("netcdf_gdal (heavy JAR) unavailable") - f = tmp_path/"scaled.nc"; _write_scaled_grid(str(f)) - from databricks.labs.gbx.ds.netcdf import NetcdfGbxDataSource - spark.dataSource.register(NetcdfGbxDataSource) - light = _tile_values(spark.read.format("netcdf_gbx").load(str(f))) # decoded physical - heavy = _tile_values(spark.read.format("netcdf_gdal").load(str(f))) # now decoded via applyScale - np.testing.assert_allclose(light["t"], heavy["t"], rtol=1e-4, atol=1e-4, equal_nan=True) -``` - -(`_tile_values` = the existing helper that reads a tile's band into a numpy array, keyed by the `source` variable.) - -- [ ] **Step 2: Run in Docker (needs heavy JAR)** - -Run (subagent, after Task 1+2 JAR built/staged): `gbx:test:python --path python/geobrix/test/ds/test_netcdf_cross_tier.py --with-integration --log netcdf-parity-scaled.log` -Expected: PASS (ran, not skipped) — heavy now matches light on the scaled grid. If it fails, the Task-2 unscale path is wrong; fix there, not by loosening tolerance. - -- [ ] **Step 3: Commit** - -```bash -git add python/geobrix/test/ds/test_netcdf_cross_tier.py -git commit -m "test(netcdf): cross-tier value parity on a scaled grid (applyScale gate) - -Adds a packed-integer scale_factor/add_offset fixture and asserts heavy -netcdf_gdal (applyScale) tile values match light netcdf_gbx decoded values -within tolerance -- the correctness gate for the heavy unscaling path. - -Co-authored-by: Isaac" -``` - ---- - -### Task 5: bench harness — raster/vector split + sizeInMB + lazy corpus.json - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/readers.py` (`run_format_read` sizeInMB passthrough; `stage_nasanex_corpus` + keep/rename swath stager) -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/cluster.py` (`_CELL_NETCDF` → raster leg + `_CELL_NETCDF_SWATH` vector leg; lazy top-level `corpus.json` read) -- Modify: `notebooks/tests/push_and_run_bench_on_cluster.py` (if it also reads corpus.json unconditionally for netcdf-only — make it consistent) -- Test: extend `python/geobrix/test/bench/` smoke tests (imports, skip-clean-when-empty). - -**Interfaces:** -- Consumes: `run_format_read(..., fmt, options, size_mib)`. Produces: `run_format_read` passes `sizeInMB` to `netcdf_gdal` (currently only `raster_gbx`); the netcdf raster leg calls both tiers over `{CORPUS}/netcdf` with `sizeInMB=-1` (one tile per grid var, fair granularity); the vector leg calls light `netcdf_gbx` mode=vector over `{CORPUS}/netcdf-swath`. - -- [ ] **Step 1: Write failing test — `run_format_read` passes sizeInMB to netcdf_gdal** - -In the bench test dir, assert (with a stub spark/reader) that `run_format_read(fmt="netcdf_gdal", size_mib=-1, ...)` sets `.option("sizeInMB","-1")`. Mirror any existing `run_format_read` test. - -- [ ] **Step 2: Run to verify failure** - -Run: `gbx:test:python --path python/geobrix/test/bench/` (the relevant test) -Expected: FAIL — sizeInMB only applied for `raster_gbx` today. - -- [ ] **Step 3: Implement sizeInMB passthrough** - -`readers.py:268` — broaden the condition: - -```python -if fmt in ("raster_gbx", "netcdf_gdal", "gdal", "gtiff_gdal"): - reader = reader.option("sizeInMB", str(size_mib)) -``` - -(Include the heavy raster formats so the knob is available; netcdf raster leg passes `size_mib=-1`.) - -- [ ] **Step 4: Split the bench cell (raster + vector legs) + lazy corpus.json** - -In `cluster.py`: rename/duplicate `_CELL_NETCDF` into: -- **raster leg** — reads `{CORPUS}/netcdf` (NASA-NEX grids), both tiers, `options={"filterRegex": r".*\.nc$"}`, heavy `fmt="netcdf_gdal"` + light `fmt="netcdf_gbx"` (raster mode default), passing `size_mib=-1`. -- **vector leg** `_CELL_NETCDF_SWATH` — reads `{CORPUS}/netcdf-swath` (S5P), light only `fmt="netcdf_gbx"` with `options={"mode":"vector","filterRegex": r".*\.nc$"}`; a comment/log line states heavy has no swath path (light-only throughput, not a comparison). -Make the top-level `corpus = _m.Corpus.read(f"{CORPUS}/corpus.json")` (cluster.py:257) lazy: only read it when a function-bench leg actually runs (guard behind `not (netcdf_only or netcdf_swath_only or other reader-only flags)`), so a reader-only run does not require the function-corpus scaffold. - -- [ ] **Step 5: Add `stage_nasanex_corpus` + swath stager** - -In `readers.py`: `stage_nasanex_corpus(spark, corpus_dir, bbox=None, temporal=None, variables=("tas",), partitions=None)` calling `NasaNexDownloader().download(...)`; keep `stage_netcdf_corpus` (S5P) but point it at `{CORPUS}/netcdf-swath`. Both log granule counts (no silent truncation) and skip-clean when the pool is empty. - -- [ ] **Step 6: Run bench smoke tests to green + commit** - -Run: `gbx:test:python --path python/geobrix/test/bench/` -Expected: PASS (imports, sizeInMB passthrough, skip-clean). - -```bash -git add python/geobrix/src/databricks/labs/gbx/bench/readers.py \ - python/geobrix/src/databricks/labs/gbx/bench/cluster.py \ - notebooks/tests/push_and_run_bench_on_cluster.py \ - python/geobrix/test/bench/ -git commit -m "bench(netcdf): split raster (NASA-NEX) vs vector (S5P) legs - -netcdf reader bench now has an honest split: a raster leg (heavy netcdf_gdal -vs light netcdf_gbx over NASA-NEX regular grids, sizeInMB=-1 for matching -one-tile-per-var granularity) and a light-only vector leg (netcdf_gbx vector -mode over S5P swaths -- heavy has no swath path). run_format_read now passes -sizeInMB to the heavy raster formats; the top-level corpus.json read is lazy -so reader-only runs do not require the function-bench scaffold. - -Co-authored-by: Isaac" -``` - ---- - -### Task 6: docs + at-scale cluster run (human-gated) - -**Files:** -- Modify: `docs/docs/api/benchmarking.mdx` (two corpora recipes + raster/vector split; NASA-NEX anonymous, S5P swath = vector-only; no internal vocabulary) -- Modify: `docs/docs/readers/netcdf.mdx` (reinforce: heavy = regular grids only; grid-only enumeration explicit; heavy now decodes scale/offset) -- Run: the actual bench on cluster `0519-143423-0jwqt79u` (rebuild+stage JAR/wheel, restart, stage a bounded NASA-NEX raster corpus + bounded S5P swath corpus, run both legs) - -- [ ] **Step 1: Update benchmarking.mdx + netcdf.mdx** - -Document the raster corpus (`stage_nasanex_corpus`, anonymous PC, regular 0.25° grids) and the swath vector corpus (S5P), the raster-vs-vector split, and that heavy `netcdf_gdal` now applies scale/offset (matches light). Reinforce in `netcdf.mdx` that heavy raster is regular-grid-only (swaths → light vector). Run `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/` → nothing new. - -- [ ] **Step 2: Commit docs** - -```bash -git add docs/docs/api/benchmarking.mdx docs/docs/readers/netcdf.mdx -git commit -m "docs(netcdf): raster/vector bench corpora + heavy scale-decode - -Co-authored-by: Isaac" -``` - -- [ ] **Step 3: Build + stage artifacts, (re)start cluster** - -Dispatch a subagent: `set -a; source notebooks/tests/databricks_cluster_config.env; set +a` then `bash scripts/commands/gbx-data-push-jar.sh` (fat JAR + tests.jar) and `gbx-data-push-wheel.sh`; sync the wheel to `sample-data/` (per `bench-wheel-path-divergence`). Then `databricks clusters start 0519-143423-0jwqt79u --profile oauth-fe` and poll RUNNING + libs INSTALLED. - -- [ ] **Step 4: Stage bounded corpora** - -Raster: `stage_nasanex_corpus(spark, f"{CORPUS}/netcdf", bbox=..., temporal=, variables=("tas",))` — bounded to ~20–50 grid granules; log the count. Vector: stage a bounded S5P subset (e.g. 20 granules from the existing pool) into `{CORPUS}/netcdf-swath`. - -- [ ] **Step 5: Run both legs + capture summary** - -`bash scripts/commands/gbx-bench-cluster.sh --netcdf-only --row-counts 1000` (raster + vector legs). Confirm it converges (grid-only enumeration → sane task count). Give the run's `summary.md` link (`bench-run-give-summary-link`). Record the heavy-vs-light raster throughput + light vector throughput in the ledger. - -- [ ] **Step 6: Stop the cluster** - -`databricks clusters delete 0519-143423-0jwqt79u --profile oauth-fe` (terminate) once the run is captured (`stop-clusters-you-start` — it's the user's cluster; terminate since we started it fresh, or leave per user preference). - ---- - -## Sequencing note - -Tasks 1→2 are the reader correctness core (Scala; one JAR build covers both). Task 3 (downloader) is independent Python. Task 4 needs Tasks 1+2's JAR. Task 5 is Python bench plumbing. Task 6 is docs + the human-gated at-scale run (needs everything). The one already-landed fix (`7ba0d2dd`, executor NPE) is a prerequisite that's done. - -## Loose ends to surface at "done" (per report-loose-ends-after-spec-execution) - -- Whether NASA-NEX GDDP variables actually carry scale/offset (if not, the scaled-parity test uses the synthetic fixture — the reader change ships regardless). -- The light-tier NetCDF **writer** remains a separate future cycle (from the prior spec). -- Wheel rebuild+restage after any light change (Tasks 3/5 touch light packages) per `whl-change-rebuild-and-stage`. -- The at-scale raster throughput number is only meaningful once Task 6 runs on-cluster; until then the raster bench is functionally-verified but un-numbered. diff --git a/docs/superpowers/plans/2026-07-27-netcdf-heavy-readers.md b/docs/superpowers/plans/2026-07-27-netcdf-heavy-readers.md deleted file mode 100644 index c00ec8931..000000000 --- a/docs/superpowers/plans/2026-07-27-netcdf-heavy-readers.md +++ /dev/null @@ -1,1019 +0,0 @@ -# NetCDF Heavy Readers + Light Auto-Enumeration Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add heavy-tier NetCDF readers (`netcdf_gdal` raster, `netcdf_ogr` vector) following the existing named-reader pattern, and unify the variable-selection contract so the shipped light `netcdf_gbx` reader auto-enumerates all readable variables (variable option becomes an optional filter, not a mandatory selector). - -**Architecture:** Three readers share one contract — a bare `load` returns *every* readable variable (raster: every georeferenced grid variable, one tile row each; vector: every DSG/curvilinear feature), and `variable`/`variables` narrows that set. `netcdf_gdal extends GDAL_DataSource` but overrides plan-time partitioning to enumerate GDAL subdatasets (one partition per `(file, variable)`) instead of assuming one raster per file. `netcdf_ogr extends OGR_DataSource` unchanged except `driverName=netCDF`. The light reader moves variable resolution from `__init__` (where it raised on absence) into the open-dataset path (`read()`/`schema()`), enumerating via the existing `classify()`. - -**Tech Stack:** Scala 2.13.16 / Spark 4.0.0 / Java 17 (heavy, GDAL Java bindings); Python 3.12+ / PySpark DataSource V2 / xarray+netcdf4 (light). Tests: ScalaTest (`PlanTest with SilentSparkSession`) in Docker; pytest (local Spark) for light. - -## Global Constraints - -- **No aliases** — one canonical name per reader (`netcdf_gdal`, `netcdf_ogr`, `netcdf_gbx`). Beta breaks API to stabilize. -- **Named-reader pattern:** a heavy named reader `extends _DataSource with DataSourceExtras`, overrides `shortName()`, `dsExtraMap(checkMap)`, `inferSchema`, `getTable`, and registers a fully-qualified class line in `src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister`. Pattern is `_`. -- **GDAL/OGR registration only via the synchronized `GDALManager` guards** — `GDALManager.init(config)` for GDAL, `GDALManager.initOgr()` for OGR. Never raw `gdal.AllRegister()`/`ogr.RegisterAll()` per task. -- **Heavy raster schema is fixed:** `struct>>` via `RST_ExpressionUtil.tileDataType(BinaryType)`. Do not change it. -- **Credential-aware listing/staging:** on `/Volumes`, a raw driver-thread Hadoop FS listing lacks the UC credential. Enumerate via `HadoopUtils.listDataFilesSpark(spark, path)` and stage per-executor via `NodeFileManager.readRemote(path)` (the OGR/GDAL batch pattern), never a raw driver FS call. -- **Serverless-safe light tier:** the Python reader may only `spark.dataSource.register` + build Column output — no `spark.conf.set`, `_jvm`, `.rdd`. -- **Swath→points is light-only** (non-goal for heavy): `netcdf_ogr` surfaces only native CF-DSG features; do not reimplement the per-pixel flatten on the JVM. -- **Heavy work runs in the `geobrix-dev` Docker container** via `gbx:*` commands. Dispatch long suites (Maven/Scala tests) to a Task subagent; never inline. -- **Binding parity** is enforced for *registered SQL functions*, not for readers — these readers add no `gbx_*` function, so `registered_functions.txt` is untouched. (Do not add reader shortNames to it.) - ---- - -### Task 1: Light `netcdf_gbx` auto-enumeration (establishes the shared contract) - -Move variable resolution from `__init__` (which raised when the option was absent) to the open-dataset path, so a bare `load` returns all readable variables. This is a **behavior change to the shipped v0.4.1 reader** — strictly more permissive; explicit-`variable` calls are unaffected. - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/_netcdf.py` (add `readable_variables` + `select_variables` helpers) -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/netcdf.py` (`NetcdfRasterReader`: enumerate + one tile per grid var; drop mandatory `_requested_variables`) -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/_netcdf_vector.py` (`NetcdfVectorReader`: enumerate at `schema()`/`read()`; optional filter) -- Test: `python/geobrix/test/ds/test_netcdf_helpers.py` (helper unit tests) -- Test: `python/geobrix/test/ds/test_netcdf_datasource.py` (auto-enumerate + back-compat) - -**Interfaces:** -- Produces (consumed by heavy parity test in Task 4 and by the readers here): - - `_netcdf.readable_variables(ds, mode: str) -> List[str]` — for `mode="raster"` returns data variables that `classify()` as `GRID`; for `mode="vector"` returns those classifying as `POINTS` or `CURVILINEAR`. Iterates `ds.data_vars` only (xarray excludes coordinate/grid-mapping variables, so `lat`/`lon`/`time_bnds`/`crs` are never returned). - - `_netcdf.select_variables(ds, options: Dict[str, str], mode: str) -> List[str]` — `readable_variables(ds, mode)` when no `variable`/`variables` option; otherwise the option's names intersected with the readable set (order follows the option). - - Light raster `source` column value = `f'NETCDF:"{path}":{var}"'` (the GDAL subdataset selector form) so multiple variables are disambiguable and the string matches the heavy `netcdf_gdal` source. - -- [ ] **Step 1: Write failing helper tests** - -Add to `python/geobrix/test/ds/test_netcdf_helpers.py`. Reuse the writer helpers already in `test_netcdf_datasource.py` — import them or inline equivalents; here we build datasets directly with `netCDF4` and open with `_netcdf.open_dataset`. - -```python -import numpy as np -from netCDF4 import Dataset -from databricks.labs.gbx.ds import _netcdf - - -def _grid_two_vars(path): - with Dataset(path, "w") as ds: - ds.createDimension("lat", 3) - ds.createDimension("lon", 4) - lat = ds.createVariable("lat", "f8", ("lat",)); lat.standard_name = "latitude" - lon = ds.createVariable("lon", "f8", ("lon",)); lon.standard_name = "longitude" - lat[:] = [50.0, 49.5, 49.0] - lon[:] = [10.0, 10.5, 11.0, 11.5] - for name in ("ch4", "co"): - v = ds.createVariable(name, "f4", ("lat", "lon"), fill_value=-9999.0) - v[:] = np.arange(12, dtype="float32").reshape(3, 4) - - -def test_readable_variables_raster_enumerates_all_grids(tmp_path): - f = tmp_path / "g.nc"; _grid_two_vars(str(f)) - with _netcdf.open_dataset(str(f), None) as ds: - assert sorted(_netcdf.readable_variables(ds, "raster")) == ["ch4", "co"] - # coordinate variables are never returned - assert "lat" not in _netcdf.readable_variables(ds, "raster") - - -def test_select_variables_absent_option_returns_all(tmp_path): - f = tmp_path / "g.nc"; _grid_two_vars(str(f)) - with _netcdf.open_dataset(str(f), None) as ds: - assert sorted(_netcdf.select_variables(ds, {}, "raster")) == ["ch4", "co"] - - -def test_select_variables_filters_to_named(tmp_path): - f = tmp_path / "g.nc"; _grid_two_vars(str(f)) - with _netcdf.open_dataset(str(f), None) as ds: - assert _netcdf.select_variables(ds, {"variable": "co"}, "raster") == ["co"] -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `gbx:test:python --path python/geobrix/test/ds/test_netcdf_helpers.py -k "readable_variables or select_variables"` -Expected: FAIL — `AttributeError: module ... has no attribute 'readable_variables'`. - -- [ ] **Step 3: Implement the helpers in `_netcdf.py`** - -Add after `classify()`: - -```python -def readable_variables(ds, mode: str) -> List[str]: - """Data variables readable in `mode` ('raster' -> GRID; 'vector' -> POINTS/CURVILINEAR). - - Iterates ds.data_vars only: with open_dataset's decode_coords="all", lat/lon, - grid-mapping, and bounds coordinate variables are xarray coords, not data_vars, - so they are never surfaced as readable fields. - """ - keep = {GRID} if mode == "raster" else {POINTS, CURVILINEAR} - return [name for name in list(ds.data_vars) if classify(ds, name) in keep] - - -def select_variables(ds, options: Dict[str, str], mode: str) -> List[str]: - """Auto-enumerate all readable variables, narrowed by an optional variable filter.""" - readable = readable_variables(ds, mode) - raw = options.get("variables") or options.get("variable") - if not raw: - return readable - requested = [v.strip() for v in str(raw).split(",") if v.strip()] - readable_set = set(readable) - return [v for v in requested if v in readable_set] -``` - -- [ ] **Step 4: Run helper tests to green** - -Run: `gbx:test:python --path python/geobrix/test/ds/test_netcdf_helpers.py -k "readable_variables or select_variables"` -Expected: PASS. - -- [ ] **Step 5: Write failing raster auto-enumerate test** - -Add to `python/geobrix/test/ds/test_netcdf_datasource.py`. Add a two-variable grid writer alongside the existing `_write_regular_grid`: - -```python -def _write_regular_grid_two(path): - with Dataset(path, "w") as ds: - ds.createDimension("lat", 3) - ds.createDimension("lon", 4) - lat = ds.createVariable("lat", "f8", ("lat",)); lat.standard_name = "latitude" - lon = ds.createVariable("lon", "f8", ("lon",)); lon.standard_name = "longitude" - lat[:] = [50.0, 49.5, 49.0] - lon[:] = [10.0, 10.5, 11.0, 11.5] - for name in ("ch4", "co"): - v = ds.createVariable(name, "f4", ("lat", "lon"), fill_value=-9999.0) - v[:] = np.arange(12, dtype="float32").reshape(3, 4) - - -def test_raster_bare_load_returns_all_grid_variables(spark, tmp_path): - f = tmp_path / "grid2.nc" - _write_regular_grid_two(str(f)) - spark.dataSource.register(NetcdfGbxDataSource) - df = spark.read.format("netcdf_gbx").load(str(f)) # NO variable option - rows = df.collect() - assert len(rows) == 2 # one tile per grid variable - sources = sorted(r["source"] for r in rows) - assert sources[0].endswith(":ch4") and sources[1].endswith(":co") - - -def test_raster_variable_option_filters(spark, tmp_path): - f = tmp_path / "grid2.nc" - _write_regular_grid_two(str(f)) - spark.dataSource.register(NetcdfGbxDataSource) - df = spark.read.format("netcdf_gbx").option("variable", "co").load(str(f)) - rows = df.collect() - assert len(rows) == 1 and rows[0]["source"].endswith(":co") -``` - -- [ ] **Step 6: Run to verify failure** - -Run: `gbx:test:python --path python/geobrix/test/ds/test_netcdf_datasource.py -k "bare_load_returns_all or variable_option_filters"` -Expected: FAIL — bare load raises `ValueError` (mandatory variable) today; filtered load returns 1 row but `source` lacks `:co` suffix. - -- [ ] **Step 7: Rewrite `NetcdfRasterReader` for enumeration** - -In `netcdf.py`, delete `_requested_variables` and rewrite the reader so selection happens inside `read()` (dataset is open there), emitting one tile per kept variable: - -```python -class NetcdfRasterReader(RasterGbxReader): - """Raster mode: transcode each CF grid variable to a GeoTIFF tile (one row per variable).""" - - def __init__(self, options: Dict[str, str]): - super().__init__(options) # path/sizeInMB/filterRegex/bbox/bboxCrs - self.options = dict(options) - self.group = options.get("group") - - def read(self, partition: "_FilePartition") -> Iterator[Tuple]: - from rasterio.io import MemoryFile - - with _netcdf.open_dataset(partition.file_path, self.group) as ds: - variables = _netcdf.select_variables(ds, self.options, "raster") - for var in variables: - transform, crs = _netcdf.grid_transform_crs(ds, var) - arr = _netcdf.array_2d(ds, var) - nodata = _netcdf.nodata_of(ds, var) - source = f'NETCDF:"{partition.file_path}":{var}' - h, w = arr.shape[-2], arr.shape[-1] - profile = dict(driver="GTiff", width=w, height=h, count=1, - dtype=str(arr.dtype), crs=crs, transform=transform) - if nodata is not None: - profile["nodata"] = nodata - with MemoryFile() as mf: - with mf.open(**profile) as out: - out.write(arr.astype(profile["dtype"]), 1) - with mf.open() as rds: - cellid, raster_bytes, meta = _encode.encode_tile( - rds, window=(0, 0, w, h), - source_path=partition.file_path, all_parents="") - yield (source, (cellid, raster_bytes, meta)) -``` - -Drop the now-unused `_listing` import if nothing else uses it (leave `_encode`, `_netcdf`). Keep the curvilinear-in-raster-mode guard behavior via `select_variables` — a curvilinear variable simply isn't in the raster readable set, so a bare load skips it silently (matches "filter, not selector"). If the user *explicitly* names a curvilinear variable it is filtered out (returns no row for it); this is the intended optional-filter semantics. - -- [ ] **Step 8: Update the pre-existing raster tests that assumed mandatory variable** - -`test_raster_read_round_trip` still passes an explicit `variable` and expects 1 row — verify it stays green (source now ends `:ch4`; the test asserts `cellid == -1` and metadata, unaffected). `test_raster_mode_rejects_curvilinear` constructs `NetcdfRasterReader({...,"variable":"ch4"})` and expects a `ValueError` matching "vector" — this behavior is **removed** (curvilinear is now silently filtered). Replace that test with: - -```python -def test_raster_mode_skips_curvilinear_variable(spark, tmp_path): - f = tmp_path / "curv.nc" - _write_curvilinear(str(f)) - spark.dataSource.register(NetcdfGbxDataSource) - # bare load: curvilinear var is not a readable GRID -> zero rows, no error - df = spark.read.format("netcdf_gbx").load(str(f)) - assert df.count() == 0 -``` - -- [ ] **Step 9: Run raster suite to green** - -Run: `gbx:test:python --path python/geobrix/test/ds/test_netcdf_datasource.py -k "raster"` -Expected: PASS (round-trip, bare-load-all, filter, skips-curvilinear). - -- [ ] **Step 10: Write failing vector auto-enumerate test** - -```python -def test_vector_bare_load_returns_all_dsg_variables(spark, tmp_path): - f = tmp_path / "pts.nc" - _write_points(str(f)) - spark.dataSource.register(NetcdfGbxDataSource) - df = spark.read.format("netcdf_gbx").option("mode", "vector").load(str(f)) # no variables - assert {"ch4", "qa_value"}.issubset(set(df.columns)) - assert df.count() == 5 -``` - -- [ ] **Step 11: Run to verify failure** - -Run: `gbx:test:python --path python/geobrix/test/ds/test_netcdf_datasource.py -k "vector_bare_load"` -Expected: FAIL — `ValueError` "vector mode requires a 'variables' option". - -- [ ] **Step 12: Rewrite `NetcdfVectorReader` for enumeration** - -In `_netcdf_vector.py`, stop raising in `__init__`; resolve variables from the open head file in `schema()` and per-file in `read()`: - -```python - def __init__(self, options: Dict[str, str]): - self.path = options.get("path") - if not self.path: - raise ValueError("netcdf_gbx requires a 'path' (e.g. .load(path)).") - self.options = dict(options) - self.group: Optional[str] = options.get("group") - self.filter_regex = options.get("filterRegex", ".*") - - def _variables(self, ds) -> List[str]: - return _netcdf.select_variables(ds, self.options, "vector") - - def schema(self) -> StructType: - members = self._members() - if not members: - raise ValueError( - f"netcdf_gbx: no files matched filterRegex {self.filter_regex!r} " - f"under {self.path!r} — nothing to infer a schema from.") - fields: List[StructField] = [] - with _netcdf.open_dataset(members[0], self.group) as ds: - for name in self._variables(ds): - fields.append(StructField(name, _netcdf.np_to_spark(ds[name].values.dtype), True)) - fields.append(StructField("geom_0", BinaryType(), True)) - fields.append(StructField("geom_0_srid", StringType(), True)) - fields.append(StructField("geom_0_srid_proj", StringType(), True)) - return StructType(fields) -``` - -And in `read()`, replace `self.variables` with a per-file resolution and drop the UNSUPPORTED-raises-on-first-var assumption by iterating the resolved set: - -```python - def read(self, partition: "_NcFilePartition") -> Iterator[Tuple]: - import shapely - with _netcdf.open_dataset(partition.file_path, self.group) as ds: - variables = self._variables(ds) - if not variables: - return - lon, lat, attrs, srid = _netcdf.point_arrays(ds, variables) - wkb = shapely.to_wkb(shapely.points(lon, lat)) - proj = f"EPSG:{srid}" - for i in range(len(lon)): - row = tuple(attrs[name][i].item() for name in variables) - yield row + (bytes(wkb[i]), srid, proj) -``` - -Keep the existing explicit-`variables` tests (`test_vector_schema_columns`, `test_vector_read_dsg_points`, `test_vector_read_curvilinear_to_points`) green — they pass `variables` so `select_variables` returns exactly the named set. - -- [ ] **Step 13: Run full light netcdf suite to green** - -Run: `gbx:test:python --path python/geobrix/test/ds/` -Expected: PASS (all helper + datasource tests, old and new). - -- [ ] **Step 14: Lint + commit** - -Run: `gbx:lint:python --fix` then `gbx:lint:python --check` (verify with Docker `--check` per host/Docker black mismatch). - -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/_netcdf.py \ - python/geobrix/src/databricks/labs/gbx/ds/netcdf.py \ - python/geobrix/src/databricks/labs/gbx/ds/_netcdf_vector.py \ - python/geobrix/test/ds/test_netcdf_helpers.py \ - python/geobrix/test/ds/test_netcdf_datasource.py -git commit -m "feat(netcdf): light netcdf_gbx auto-enumerates readable variables - -variable/variables option becomes an optional filter, not a mandatory -selector. A bare load returns every readable variable (raster: one tile -row per grid variable, source=NETCDF:\"file\":var; vector: all DSG/ -curvilinear features). Establishes the shared cross-tier contract for -the new heavy netcdf_gdal/netcdf_ogr readers. - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: `netcdf_gdal` heavy raster reader (subdataset enumeration) - -The one genuinely new mechanism: a NetCDF file has no top-level bands, only subdatasets, so plan-time partitioning enumerates subdatasets (one partition per `(file, variable)`) rather than one-per-file. - -**Files:** -- Create: `src/main/scala/com/databricks/labs/gbx/rasterx/ds/netcdf/NetCDF_DataSource.scala` -- Create: `src/main/scala/com/databricks/labs/gbx/rasterx/ds/netcdf/NetCDF_Table.scala` -- Create: `src/main/scala/com/databricks/labs/gbx/rasterx/ds/netcdf/NetCDF_Batch.scala` -- Create: `src/main/scala/com/databricks/labs/gbx/rasterx/ds/netcdf/NetCDF_Partition.scala` -- Create: `src/main/scala/com/databricks/labs/gbx/rasterx/ds/netcdf/NetCDF_Reader.scala` -- Modify: `src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister` (+1 line) -- Test: `src/test/scala/com/databricks/labs/gbx/rasterx/ds/NetCDF_DataSourceTest.scala` - -**Interfaces:** -- Consumes: `RasterAccessors.subdatasetsMap(ds: Dataset): Map[String, String]` (SUBDATASETS metadata, `SUBDATASET_N_NAME -> "NETCDF:\"file\":var"`), `RasterDriver.read(path, options)` / `releaseDataset`, `BalancedSubdivision.splitRasterIter(ds, Map.empty, sizeInMB)`, `RasterSerializationUtil.tileToRow`, `NodeFileManager.readRemote/releaseRemote`, `HadoopUtils.listDataFilesSpark`, `GDALManager.init`. -- Produces: reader shortName `"netcdf_gdal"`; `dsExtraMap` = `Map("driver" -> "netCDF")`; `NetCDF_Partition(filePath: String, subdatasetName: String, sizeInMB: Int, expressionConfig: ExpressionConfig)`; `source` column value = the subdataset selector `NETCDF:"file":var`. - -- [ ] **Step 1: Write failing unit test (shortName + dsExtraMap + inferSchema + is-a)** - -Create `src/test/scala/com/databricks/labs/gbx/rasterx/ds/NetCDF_DataSourceTest.scala` mirroring `GTiff_DataSourceTest`: - -```scala -package com.databricks.labs.gbx.rasterx.ds - -import com.databricks.labs.gbx.rasterx.ds.netcdf.NetCDF_DataSource -import org.apache.spark.sql.catalyst.plans.PlanTest -import org.apache.spark.sql.test.SilentSparkSession -import org.apache.spark.sql.types.StringType -import org.apache.spark.sql.util.CaseInsensitiveStringMap -import org.scalatest.matchers.should.Matchers._ - -import scala.jdk.CollectionConverters._ - -class NetCDF_DataSourceTest extends PlanTest with SilentSparkSession { - - test("NetCDF_DataSource short name is netcdf_gdal") { - new NetCDF_DataSource().shortName() shouldBe "netcdf_gdal" - } - - test("NetCDF_DataSource injects driver netCDF in dsExtraMap") { - new NetCDF_DataSource().dsExtraMap() shouldBe Map("driver" -> "netCDF") - } - - test("NetCDF_DataSource infers (source, tile) schema") { - val ds = new NetCDF_DataSource() - val schema = ds.inferSchema(new CaseInsensitiveStringMap(Map.empty[String, String].asJava)) - schema.fields.length shouldBe 2 - schema.fields(0).name shouldBe "source" - schema.fields(0).dataType shouldBe StringType - schema.fields(1).name shouldBe "tile" - } - - test("NetCDF_DataSource is a TableProvider and DataSourceRegister") { - val ds = new NetCDF_DataSource() - ds shouldBe a[org.apache.spark.sql.connector.catalog.TableProvider] - ds shouldBe a[org.apache.spark.sql.sources.DataSourceRegister] - } -} -``` - -- [ ] **Step 2: Run to verify failure (compile error — class absent)** - -Dispatch a Task subagent (Docker/Maven is long-running): -Run: `gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.ds.NetCDF_DataSourceTest' --log netcdf-gdal-unit.log` -Expected: FAIL — compilation error, `NetCDF_DataSource` not found. - -- [ ] **Step 3: Create `NetCDF_Partition`** - -```scala -package com.databricks.labs.gbx.rasterx.ds.netcdf - -import com.databricks.labs.gbx.expressions.ExpressionConfig -import org.apache.spark.sql.connector.read.InputPartition - -/** One partition of a netcdf_gdal scan: one (file, subdataset variable) pair. - * Opened by NetCDF_Reader as the GDAL subdataset selector NETCDF:"file":var. */ -case class NetCDF_Partition( - filePath: String, - subdatasetName: String, - sizeInMB: Int, - expressionConfig: ExpressionConfig -) extends InputPartition - with Serializable -``` - -- [ ] **Step 4: Create `NetCDF_Reader`** - -Opens the subdataset selector (staging the file locally first if remote), tiles via the shared `BalancedSubdivision` path, and yields `(selector, tile)` rows — `source` = the selector so the variable is recoverable. - -```scala -package com.databricks.labs.gbx.rasterx.ds.netcdf - -import com.databricks.labs.gbx.rasterx.gdal.RasterDriver -import com.databricks.labs.gbx.rasterx.operations.BalancedSubdivision -import com.databricks.labs.gbx.rasterx.util.{RST_ExpressionUtil, RasterSerializationUtil} -import com.databricks.labs.gbx.util.NodeFileManager -import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.connector.read.PartitionReader -import org.apache.spark.sql.types.BinaryType -import org.apache.spark.unsafe.types.UTF8String - -/** Reads one netcdf_gdal partition: opens its subdataset selector, splits into tiles, yields (source, tile). */ -class NetCDF_Reader(partition: NetCDF_Partition) extends PartitionReader[InternalRow] { - - RST_ExpressionUtil.init(partition.expressionConfig) - - // Stage the .nc locally if remote (subdataset selectors are not plain paths, so - // RasterDriver's own copyToLocal cannot recognize/stage them — do it explicitly). - private val isLocal = partition.filePath.startsWith("/") && - !partition.filePath.startsWith("/Volumes/") && !partition.filePath.startsWith("/dbfs/") - private val localPath = if (isLocal) partition.filePath else NodeFileManager.readRemote(partition.filePath) - private val selector = s"""NETCDF:"$localPath":${partition.subdatasetName}""" - - private val ds = RasterDriver.read(selector, Map("isSubdataset" -> "true")) - private val tilesIter = BalancedSubdivision.splitRasterIter(ds, Map.empty, partition.sizeInMB) - RST_ExpressionUtil.addCleanupListener(tilesIter) - private val hconf = partition.expressionConfig.hConf - // The result-facing source keeps the ORIGINAL (remote) path, not the local staging copy. - private val srcSelector = s"""NETCDF:"${partition.filePath}":${partition.subdatasetName}""" - - override def next(): Boolean = tilesIter.hasNext - - override def get(): InternalRow = { - val tile = tilesIter.next() - val tileRow = RasterSerializationUtil.tileToRow((-1L, tile._1, tile._2), BinaryType, hconf) - RasterDriver.releaseDataset(tile._1) - InternalRow.fromSeq(Seq(UTF8String.fromString(srcSelector), tileRow)) - } - - override def close(): Unit = { - if (!isLocal) NodeFileManager.releaseRemote(partition.filePath) - } -} -``` - -- [ ] **Step 5: Create `NetCDF_Batch` (subdataset enumeration at plan time)** - -Enumerate subdatasets executor-side via a UDF (credential-aware on `/Volumes`, mirroring `OGR_Batch`). Filter to real grid variables, apply the optional `variable`/`variables` filter, emit one partition per `(file, variable)`. - -```scala -package com.databricks.labs.gbx.rasterx.ds.netcdf - -import com.databricks.labs.gbx.expressions.ExpressionConfig -import com.databricks.labs.gbx.rasterx.gdal.{GDALManager, RasterDriver} -import com.databricks.labs.gbx.rasterx.operations.RasterAccessors -import com.databricks.labs.gbx.util.{HadoopUtils, NodeFileManager} -import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.connector.read.{Batch, InputPartition, PartitionReaderFactory, Scan} -import org.apache.spark.sql.functions.{col, explode, udf} -import org.apache.spark.sql.types.StructType - -/** Scan/Batch for netcdf_gdal: one partition per (file, grid-variable subdataset). */ -class NetCDF_Batch(schema: StructType, options: Map[String, String]) extends Scan with Batch { - - override def readSchema(): StructType = schema - override def toBatch: Batch = this - - override def planInputPartitions(): Array[InputPartition] = { - val inPath = options("path") - val sizeInMB = options.getOrElse("sizeInMB", "-1").toInt - val filterRegex = options.getOrElse("filterRegex", ".*\\.nc$") - // Optional variable filter (empty => keep all). Names, comma-separated. - val wanted = (options.get("variables").orElse(options.get("variable"))) - .map(_.split(",").map(_.trim).filter(_.nonEmpty).toSet).getOrElse(Set.empty[String]) - - val spark = SparkSession.builder.getOrCreate - val exprConfig = ExpressionConfig(spark) - import spark.implicits._ - - val files = HadoopUtils.listDataFilesSpark(spark, inPath) - .filter(_.matches(filterRegex)) - NodeFileManager.init(exprConfig.hConf) - - // Executor-side enumeration: open each file, list SUBDATASETS, keep grid variables. - val enumUDF = udf { (path: String) => - try { - GDALManager.init(exprConfig) - val localPath = NodeFileManager.readRemote(path) - val ds = RasterDriver.read(localPath, Map.empty) - val subs = RasterAccessors.subdatasetsMap(ds) - // SUBDATASET_i_NAME -> "NETCDF:\"file\":var"; take NAME entries only. - val vars = subs.toSeq.filter(_._1.endsWith("_NAME")).map { case (_, sel) => - sel.reverse.takeWhile(_ != ':').reverse // trailing :var - }.filter(v => !v.endsWith("_bnds") && !v.endsWith("_bounds")) - // Keep only subdatasets GDAL opens as a >1x1 raster (drops degenerate/coordinate arrays). - val grids = vars.filter { v => - try { - val sub = RasterDriver.read(s"""NETCDF:"$localPath":$v""", Map("isSubdataset" -> "true")) - val ok = sub.GetRasterXSize > 1 && sub.GetRasterYSize > 1 && sub.GetRasterCount >= 1 - RasterDriver.releaseDataset(sub); ok - } catch { case _: Throwable => false } - } - RasterDriver.releaseDataset(ds) - NodeFileManager.releaseRemote(path) - grids.map(v => (path, v)).toArray - } catch { case _: Throwable => Array.empty[(String, String)] } - } - - val pairs = files.toDF("path") - .select(explode(enumUDF(col("path"))).as("p")) - .select("p._1", "p._2").as[(String, String)].collect() - - pairs - .filter { case (_, v) => wanted.isEmpty || wanted.contains(v) } - .map { case (file, v) => NetCDF_Partition(file, v, sizeInMB, exprConfig) } - .toArray[InputPartition] - } - - override def createReaderFactory(): PartitionReaderFactory = - (partition: InputPartition) => new NetCDF_Reader(partition.asInstanceOf[NetCDF_Partition]) -} -``` - -- [ ] **Step 6: Create `NetCDF_Table`** - -Read-only raster table wiring `NetCDF_Batch` (mirror `GDAL_Table` but read-only — no write path for this reader). - -```scala -package com.databricks.labs.gbx.rasterx.ds.netcdf - -import org.apache.spark.sql.connector.catalog._ -import org.apache.spark.sql.connector.read.ScanBuilder -import org.apache.spark.sql.types.StructType -import org.apache.spark.sql.util.CaseInsensitiveStringMap - -import scala.jdk.CollectionConverters._ - -/** Read-only Table for netcdf_gdal: batch read via NetCDF_Batch. */ -class NetCDF_Table(schema: StructType, properties: Map[String, String]) extends Table with SupportsRead { - - override def name(): String = "netcdf_gdal" - // noinspection ScalaDeprecation - override def schema(): StructType = schema - override def columns(): Array[Column] = schema.fields.map(f => Column.create(f.name, f.dataType, f.nullable)) - override def newScanBuilder(options: CaseInsensitiveStringMap): ScanBuilder = { () => - new NetCDF_Batch(schema, properties ++ options.asScala) - } - override def capabilities(): java.util.Set[TableCapability] = - Set(TableCapability.BATCH_READ).asJava -} -``` - -- [ ] **Step 7: Create `NetCDF_DataSource`** - -Extends `GDAL_DataSource` (inherits the fixed `(source, tile)` `inferSchema`) with `DataSourceExtras`; overrides `getTable` to return the subdataset-aware `NetCDF_Table`. - -```scala -package com.databricks.labs.gbx.rasterx.ds.netcdf - -import com.databricks.labs.gbx.ds.DataSourceExtras -import com.databricks.labs.gbx.rasterx.ds.gdal.GDAL_DataSource -import org.apache.spark.sql.connector.catalog.Table -import org.apache.spark.sql.connector.expressions.Transform -import org.apache.spark.sql.types.StructType -import org.apache.spark.sql.util.CaseInsensitiveStringMap - -import scala.jdk.CollectionConverters._ - -/** GDAL TableProvider restricted to netCDF (driver = netCDF). Reads CF grid variables as - * one (source, tile) row per variable; source = the NETCDF:"file":var subdataset selector. - * Use format "netcdf_gdal". Read-only. */ -//noinspection ScalaUnusedSymbol -class NetCDF_DataSource extends GDAL_DataSource with DataSourceExtras { - - override def dsExtraMap(checkMap: Map[String, String] = Map.empty): Map[String, String] = - Map("driver" -> "netCDF") - - override def shortName(): String = "netcdf_gdal" - - override def inferSchema(options: CaseInsensitiveStringMap): StructType = - super.inferSchema(extraCaseInsensitiveStringMap(options)) - - override def getTable(schema: StructType, partitions: Array[Transform], properties: java.util.Map[String, String]): Table = - new NetCDF_Table(schema, extraJavaUtilMap(properties).asScala.toMap) -} -``` - -- [ ] **Step 8: Register in META-INF/services** - -Add one line to `src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister`: - -``` -com.databricks.labs.gbx.rasterx.ds.netcdf.NetCDF_DataSource -``` - -- [ ] **Step 9: Run unit test to green** - -Run (Task subagent): `gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.ds.NetCDF_DataSourceTest' --log netcdf-gdal-unit.log` -Expected: PASS (all 4 unit tests). - -- [ ] **Step 10: Write failing integration test (real fixture enumeration + filter)** - -Append to `NetCDF_DataSourceTest.scala`. Use the coral fixtures (`/binary/netcdf-coral/`). First confirm the variable name(s) — a quick probe test that prints subdatasets is fine, but write the assertion against the known CF variable. (If the coral file is single-variable, assert `>= 1` grid var and that a bogus filter yields 0 rows; if multi-variable, assert the enumerated count.) - -```scala -test("netcdf_gdal bare load enumerates grid variables into (source, tile) rows") { - import com.databricks.labs.gbx.rasterx.functions._ - rasterx.functions.register(spark) - val ncDir = this.getClass.getResource("/binary/netcdf-coral/").toString - val df = spark.read.format("netcdf_gdal").option("sizeInMB", "1") - .option("filterRegex", ".*20220101\\.nc$").load(ncDir) - val rows = df.select("source").collect() - rows.length should be >= 1 - all(rows.map(_.getString(0))) should startWith("NETCDF:") -} - -test("netcdf_gdal variable filter naming an absent variable yields no rows") { - import com.databricks.labs.gbx.rasterx.functions._ - rasterx.functions.register(spark) - val ncDir = this.getClass.getResource("/binary/netcdf-coral/").toString - val df = spark.read.format("netcdf_gdal") - .option("filterRegex", ".*20220101\\.nc$") - .option("variable", "no_such_variable_xyz").load(ncDir) - df.count() shouldBe 0L -} -``` - -- [ ] **Step 11: Run integration test to verify then green** - -Run (Task subagent): `gbx:test:scala --suite 'com.databricks.labs.gbx.rasterx.ds.NetCDF_DataSourceTest' --log netcdf-gdal-int.log` -Expected: first PASS after implementation; if the grid filter over/under-includes, tighten the `_bnds` skip / `>1x1` check in `NetCDF_Batch` (Step 5) until the enumerated set matches the file's real grid variables. This test is the filter's spec. - -- [ ] **Step 12: Bump the BenchDispatch count assert if tripped** - -Per the `scala-benchdispatch-count-assert` gotcha, adding RasterX surfaces can trip a hardcoded `BenchDispatch.all.size == N`. These readers add no bench function, so likely untouched — but verify: - -Run: `grep -rn "\.size shouldBe\|\.size ==" src/test/scala/**/BenchDispatch* 2>/dev/null; grep -rn "BenchDispatch.all.size" src/` -Expected: no change needed (no new bench dispatch). If a suite references reader counts, bump it. - -- [ ] **Step 13: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/rasterx/ds/netcdf/ \ - src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister \ - src/test/scala/com/databricks/labs/gbx/rasterx/ds/NetCDF_DataSourceTest.scala -git commit -m "feat(rasterx): netcdf_gdal heavy raster reader via subdataset enumeration - -Reads CF grid variables from a .nc file as one (source, tile) row per -variable. Plans one partition per (file, subdataset), enumerating -SUBDATASETS executor-side (credential-aware for /Volumes) and filtering -to real >1x1 grid variables. source = the NETCDF:\"file\":var selector, -matching the light netcdf_gbx raster contract. variable/variables is an -optional filter. Read-only. - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: `netcdf_ogr` heavy vector reader (CF-DSG features) - -A thin OGR reader — the OGR netCDF driver surfaces native CF Discrete Sampling Geometry features into the shared vector schema. No new plan-time mechanism. - -**Files:** -- Create: `src/main/scala/com/databricks/labs/gbx/vectorx/ds/netcdf/NetCDF_OGR_DataSource.scala` -- Modify: `src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister` (+1 line) -- Test: `src/test/scala/com/databricks/labs/gbx/vectorx/ds/NetCDF_OGR_DataSourceTest.scala` -- Possibly create: a small CF-DSG `.nc` fixture under `src/test/resources/binary/netcdf-dsg/` (existing fixtures are grids) - -**Interfaces:** -- Consumes: `OGR_DataSource` (inherited `inferSchema`, `getTable`, `supportsExternalMetadata`, write-guard), `DataSourceExtras`. -- Produces: shortName `"netcdf_ogr"`; `dsExtraMap` = `Map("driverName" -> "netCDF")`; emits the shared vector schema (attributes + `geom_0` WKB + `geom_0_srid` / `geom_0_srid_proj`). - -- [ ] **Step 1: Write failing unit test** - -Create `src/test/scala/com/databricks/labs/gbx/vectorx/ds/NetCDF_OGR_DataSourceTest.scala`: - -```scala -package com.databricks.labs.gbx.vectorx.ds - -import com.databricks.labs.gbx.vectorx.ds.netcdf.NetCDF_OGR_DataSource -import org.apache.spark.sql.catalyst.plans.PlanTest -import org.apache.spark.sql.test.SilentSparkSession -import org.scalatest.matchers.should.Matchers._ - -class NetCDF_OGR_DataSourceTest extends PlanTest with SilentSparkSession { - - test("netcdf_ogr short name is netcdf_ogr") { - new NetCDF_OGR_DataSource().shortName() shouldBe "netcdf_ogr" - } - - test("netcdf_ogr injects driverName netCDF") { - new NetCDF_OGR_DataSource().dsExtraMap() shouldBe Map("driverName" -> "netCDF") - } - - test("netcdf_ogr is a TableProvider and DataSourceRegister") { - val ds = new NetCDF_OGR_DataSource() - ds shouldBe a[org.apache.spark.sql.connector.catalog.TableProvider] - ds shouldBe a[org.apache.spark.sql.sources.DataSourceRegister] - } -} -``` - -- [ ] **Step 2: Run to verify failure (compile error)** - -Run (Task subagent): `gbx:test:scala --suite 'com.databricks.labs.gbx.vectorx.ds.NetCDF_OGR_DataSourceTest' --log netcdf-ogr-unit.log` -Expected: FAIL — class not found. - -- [ ] **Step 3: Create `NetCDF_OGR_DataSource`** - -Mirror `GeoJSON_DataSource` exactly, swapping the driver and messages: - -```scala -package com.databricks.labs.gbx.vectorx.ds.netcdf - -import com.databricks.labs.gbx.ds.DataSourceExtras -import com.databricks.labs.gbx.vectorx.ds.ogr.OGR_DataSource -import org.apache.spark.sql.connector.catalog.Table -import org.apache.spark.sql.connector.expressions.Transform -import org.apache.spark.sql.types.StructType -import org.apache.spark.sql.util.CaseInsensitiveStringMap - -/** OGR-based TableProvider for CF Discrete Sampling Geometry features in netCDF (driverName = netCDF). - * Surfaces native DSG point/profile/trajectory features into the shared vector schema. Read-only. - * Does NOT flatten swaths to per-cell points — that is the light netcdf_gbx vector mode only. */ -//noinspection ScalaUnusedSymbol -class NetCDF_OGR_DataSource extends OGR_DataSource with DataSourceExtras { - - override def dsExtraMap(checkMap: Map[String, String] = Map.empty): Map[String, String] = - Map("driverName" -> "netCDF") - - override def shortName(): String = "netcdf_ogr" - - override protected def writeGuardMessage(path: String): String = - "'netcdf_ogr' is a read-only reader; write vector data with the light geojson_gbx writer " + - "(or another _gbx vector writer)." - - override def inferSchema(options: CaseInsensitiveStringMap): StructType = - super.inferSchema(extraCaseInsensitiveStringMap(options)) - - override def getTable(schema: StructType, partitions: Array[Transform], properties: java.util.Map[String, String]): Table = - super.getTable(schema, partitions, extraJavaUtilMap(properties)) -} -``` - -- [ ] **Step 4: Register in META-INF/services** - -Add: -``` -com.databricks.labs.gbx.vectorx.ds.netcdf.NetCDF_OGR_DataSource -``` - -- [ ] **Step 5: Run unit test to green** - -Run (Task subagent): `gbx:test:scala --suite 'com.databricks.labs.gbx.vectorx.ds.NetCDF_OGR_DataSourceTest' --log netcdf-ogr-unit.log` -Expected: PASS. - -- [ ] **Step 6: Stage a CF-DSG fixture and write the integration test** - -The existing fixtures (CMIP5/coral/ECMWF) are grids — the OGR netCDF driver reads DSG features, not grids, so a grid `.nc` yields zero features. Create a tiny CF-DSG point file. Add a generator (committed as a helper script + its output `.nc`) or write it in-test via a temp file. Prefer an in-test temp `.nc` built with the JVM's netCDF write path if available; otherwise stage a small committed fixture at `src/test/resources/binary/netcdf-dsg/points.nc` produced by this Python snippet (run once, commit the `.nc`): - -```python -# scripts/testdata/make_netcdf_dsg.py (run once; commit the .nc, not required at test time) -from netCDF4 import Dataset -import numpy as np -with Dataset("src/test/resources/binary/netcdf-dsg/points.nc", "w") as ds: - ds.featureType = "point" # CF-DSG marker the OGR driver keys on - ds.createDimension("obs", 5) - lat = ds.createVariable("latitude", "f8", ("obs",)); lat.standard_name = "latitude"; lat.units = "degrees_north" - lon = ds.createVariable("longitude", "f8", ("obs",)); lon.standard_name = "longitude"; lon.units = "degrees_east" - val = ds.createVariable("ch4", "f4", ("obs",)); val.coordinates = "latitude longitude" - lat[:] = [50.0, 50.1, 50.2, 50.3, 50.4] - lon[:] = [10.0, 10.1, 10.2, 10.3, 10.4] - val[:] = np.arange(5, dtype="float32") -``` - -Then the integration test: - -```scala -test("netcdf_ogr reads CF-DSG point features into the shared vector schema") { - val dsgDir = this.getClass.getResource("/binary/netcdf-dsg/").toString - val df = spark.read.format("netcdf_ogr").load(dsgDir) - df.columns should contain allOf ("geom_0", "geom_0_srid", "geom_0_srid_proj") - df.count() shouldBe 5L -} - -test("netcdf_ogr on a grid file yields no features (empty, non-erroring)") { - val gridDir = this.getClass.getResource("/binary/netcdf-coral/").toString - val df = spark.read.format("netcdf_ogr").option("filterRegex", ".*20220101\\.nc$").load(gridDir) - df.count() shouldBe 0L -} -``` - -If the OGR netCDF driver in the container does not expose the DSG file as features (driver build variance), fall back to asserting only the schema-infer + empty-on-grid behavior and note the DSG-read case as environment-dependent in the test comment. Verify against the actual container GDAL/OGR build first. - -- [ ] **Step 7: Run integration test to green** - -Run (Task subagent): `gbx:test:scala --suite 'com.databricks.labs.gbx.vectorx.ds.NetCDF_OGR_DataSourceTest' --log netcdf-ogr-int.log` -Expected: PASS. - -- [ ] **Step 8: Commit** - -```bash -git add src/main/scala/com/databricks/labs/gbx/vectorx/ds/netcdf/ \ - src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister \ - src/test/scala/com/databricks/labs/gbx/vectorx/ds/NetCDF_OGR_DataSourceTest.scala \ - src/test/resources/binary/netcdf-dsg/points.nc scripts/testdata/make_netcdf_dsg.py -git commit -m "feat(vectorx): netcdf_ogr heavy DSG vector reader - -Thin OGR reader (driverName=netCDF) surfacing native CF Discrete -Sampling Geometry features into the shared vector schema. Read-only; -grid-only files yield no features. Does not flatten swaths to points -(light netcdf_gbx vector mode only). Adds a small CF-DSG test fixture. - -Co-authored-by: Isaac" -``` - ---- - -### Task 4: Cross-tier raster parity test (the correctness gate for the grid filter) - -On a shared gridded fixture, `netcdf_gdal` and light `netcdf_gbx` raster mode must enumerate the **same variable set** and produce the **same per-variable tile** (CRS + geotransform equal, cell values within tolerance). Byte parity is not expected (xarray/rasterio vs GDAL). - -**Files:** -- Test: `docs/tests/python/...` OR a dedicated cross-tier test. Since heavy needs the JVM + Docker and light is Python, the practical parity harness is a **Python doc/integration test that reads both** through Spark (heavy JAR present in the Docker env). Place at `python/geobrix/test/ds/test_netcdf_cross_tier.py` guarded to run only where the JAR + GDAL are available (Docker). Follow the `docker-volumes-for-integration-tests` gating (skip cleanly when the heavy reader/format is unregistered). - -**Interfaces:** -- Consumes: both registered formats (`netcdf_gdal`, `netcdf_gbx`); the coral/CMIP5 fixture copied to a readable path; `readable_variables` (light) as the reference variable set. - -- [ ] **Step 1: Write the parity test (skipped outside Docker/heavy env)** - -```python -import numpy as np -import pytest -from rasterio.io import MemoryFile - - -def _heavy_available(spark): - try: - spark.read.format("netcdf_gdal") - return True - except Exception: - return False - - -@pytest.mark.integration -def test_netcdf_gdal_matches_light_raster(spark, netcdf_grid_fixture): - if not _heavy_available(spark): - pytest.skip("netcdf_gdal (heavy JAR) not available in this environment") - from databricks.labs.gbx.ds.netcdf import NetcdfGbxDataSource - spark.dataSource.register(NetcdfGbxDataSource) - - path = netcdf_grid_fixture # a single gridded .nc with known variable(s) - light = spark.read.format("netcdf_gbx").load(path).collect() - heavy = spark.read.format("netcdf_gdal").load(path).collect() - - # same enumerated variable set (source ends in :var for both tiers) - light_vars = sorted(r["source"].rsplit(":", 1)[-1] for r in light) - heavy_vars = sorted(r["source"].rsplit(":", 1)[-1] for r in heavy) - assert light_vars == heavy_vars - - # per-variable tile: CRS equal, values within tolerance - def by_var(rows): - out = {} - for r in rows: - v = r["source"].rsplit(":", 1)[-1] - with MemoryFile(bytes(r["tile"]["raster"])) as mf, mf.open() as ds: - out[v] = (ds.crs.to_epsg(), ds.read(1)) - return out - - lm, hm = by_var(light), by_var(heavy) - for v in light_vars: - assert lm[v][0] == hm[v][0] # same EPSG - np.testing.assert_allclose(lm[v][1], hm[v][1], rtol=1e-4, atol=1e-4, - equal_nan=True) -``` - -Add a `netcdf_grid_fixture` fixture (module-scoped) that copies one coral/CMIP5 `.nc` into a temp path, or points at the mounted test-resources path in Docker. - -- [ ] **Step 2: Run in Docker; refine the heavy grid filter if the variable sets differ** - -Run (Task subagent, Docker with heavy JAR + volumes): `gbx:test:python --path python/geobrix/test/ds/test_netcdf_cross_tier.py --log netcdf-parity.log` -Expected: PASS. If `light_vars != heavy_vars`, the heavy `NetCDF_Batch` grid filter (Task 2 Step 5) is over/under-inclusive vs light `classify()` — adjust the `_bnds`/`>1x1` filter until they agree, re-run. - -- [ ] **Step 3: Commit** - -```bash -git add python/geobrix/test/ds/test_netcdf_cross_tier.py -git commit -m "test(netcdf): cross-tier raster parity netcdf_gdal vs netcdf_gbx - -Same gridded fixture -> same enumerated variable set and same per-variable -tile (EPSG equal, cell values within tolerance). This is the correctness -gate for the heavy subdataset grid filter. Skips cleanly where the heavy -JAR is unavailable. - -Co-authored-by: Isaac" -``` - ---- - -### Task 5: Documentation, release notes, reader tables - -**Files:** -- Modify: `docs/docs/readers/netcdf.mdx` (document `netcdf_gdal` + `netcdf_ogr` alongside `netcdf_gbx`; the optional-filter contract; the light behavior change; the swath→points light-only asymmetry). Create if it does not exist. -- Modify: `docs/docs/beta-release-notes.mdx` (new heavy readers + light behavior change). -- Modify: `CLAUDE.md` (add `netcdf_gdal`, `netcdf_ogr` to the Readers named-reader lists). - -**Interfaces:** none (docs). User-facing docs voice — no internal vocabulary (no "wave N", no subagent/dispatch references). - -- [ ] **Step 1: Check whether `docs/docs/readers/netcdf.mdx` exists** - -Run: `ls docs/docs/readers/netcdf.mdx 2>/dev/null && echo exists || echo missing` -If missing, create it modeled on a sibling reader page (e.g. `docs/docs/readers/geojson.mdx` or `gtiff.mdx`) — check `ls docs/docs/readers/`. - -- [ ] **Step 2: Write/extend the readers page** - -Document all three readers and the unified contract. Include runnable examples (per the doc-tests-are-source rule, code should come from `docs/tests/` — if the page uses executable imports, add the snippet to the corresponding doc-test module and import via raw-loader; otherwise keep examples minimal and accurate). Cover: -- Bare `load` returns all readable variables (raster: one tile row per grid variable; vector: all DSG features). -- `variable`/`variables` is an optional filter. -- `source` column is the `NETCDF:"file":var` selector. -- `netcdf_ogr` reads native CF-DSG features only; swath→points is `netcdf_gbx` vector-mode only. - -- [ ] **Step 3: Add release notes entry** - -In `docs/docs/beta-release-notes.mdx`, under the current unreleased/next section, add: -- New heavy readers `netcdf_gdal` (raster) and `netcdf_ogr` (DSG vector). -- **Behavior change:** light `netcdf_gbx` `variable`/`variables` option is now an optional filter; a bare load returns all readable variables (previously raised). Explicit-`variable` calls are unchanged. - -- [ ] **Step 4: Update CLAUDE.md reader lists** - -In `CLAUDE.md`, the "Readers are namespace-suffixed" section: add `netcdf_gdal` to the Raster (GDAL) list and `netcdf_ogr` to the Vector (OGR) list. - -- [ ] **Step 5: Internals-leak + link checks** - -Run: `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/ 2>/dev/null` (expect nothing). -Verify doc links resolve (per the queued `docs-link-audit-pending` note, sanity-check the readers page links). - -- [ ] **Step 6: Commit** - -```bash -git add docs/docs/readers/netcdf.mdx docs/docs/beta-release-notes.mdx CLAUDE.md -git commit -m "docs(netcdf): document netcdf_gdal + netcdf_ogr and the light behavior change - -Adds the readers page coverage for the two heavy readers and the unified -optional-filter contract; release-notes the light netcdf_gbx bare-load -behavior change; adds both readers to the CLAUDE.md reader tables. - -Co-authored-by: Isaac" -``` - ---- - -### Task 6: Benchmarking — NetCDF corpus + format-parameterized reader-bench path - -Add a same-corpus heavy-vs-light throughput bench over real S5P granules staged by the existing `TropomiDownloader`. The generic `readers.run_format_read(spark, dir, ..., fmt=...)` is already format-generic; the missing pieces are the corpus and the invocation cell. - -**Files:** -- Modify: bench reader harness — `readers.py` (the `_list_tifs` glob is `.tif`-only) and/or `cluster.py` (the reader cell hard-codes `rows/` + `filterRegex: .*\.tif$`). Add a `.nc`-capable path + a NetCDF corpus stager. -- Modify: `docs/docs/api/benchmarking.mdx` (per the `bench-changes-update-docs` rule). - -**Interfaces:** -- Consumes: `readers.run_format_read(spark, netcdf_dir, ..., fmt="netcdf_gdal" | "netcdf_gbx")`, `TropomiDownloader().download(bbox, out_dir, temporal=...)` (download-and-stop mode), the reader-bench corpus Volume (parallel `netcdf/` subdir alongside `rows/`). - -- [ ] **Step 1: Locate the bench harness reader path** - -Run: `grep -rn "_list_tifs\|run_format_read\|filterRegex.*tif\|rows/" scripts/ python/ docs/ 2>/dev/null | grep -i "bench\|reader\|cluster" | head -30` -Identify the exact files behind `readers.py` and `cluster.py` referenced in the spec (§6). - -- [ ] **Step 2: Add a format+glob-parameterized listing** - -Generalize the `.tif`-only glob so the reader bench can target `.nc`. Add a `filterRegex`/extension parameter (default `.*\.tif$`) rather than hard-coding, so `.*\.nc$` works. Keep the existing GeoTIFF path behavior identical (same default). - -- [ ] **Step 3: Add the NetCDF corpus stager** - -A helper (bench-only, not product) that calls `TropomiDownloader().download(bbox, out_dir=/netcdf, temporal=...)` in download-and-stop mode to stage real S5P L2 CH4 granules as `{item_id}.nc`. Guard: if the corpus dir is empty (no Planetary Computer token / download unavailable), the bench **skips cleanly** rather than failing (spec §8). - -- [ ] **Step 4: Add the NetCDF reader-bench invocation cell** - -Add a cell/path that runs `run_format_read(spark, netcdf_dir, fmt="netcdf_gdal")` and `fmt="netcdf_gbx"` (raster mode) over the same staged `.nc` dir with `filterRegex: .*\.nc$` — a true same-corpus heavy-vs-light comparison. Follow the bench pre-flight discipline (scope from real run output, non-empty corpus, correct worker count, truthful stamp, guard dups). Use the standing bench defaults (spark-path 1000 tiles / pure-core 1; `--row-counts 1000`). - -- [ ] **Step 5: Document the bench addition** - -Update `docs/docs/api/benchmarking.mdx`: the reader-bench now covers NetCDF, and document the `TropomiDownloader`-staged S5P corpus recipe (real granules, PC token at stage time, corpus decoupled from `read()`), and the swath-vs-grid caveat (S5P = throughput bench; parity uses a gridded fixture). - -- [ ] **Step 6: Commit** - -```bash -git add docs/docs/api/benchmarking.mdx -git commit -m "bench(netcdf): same-corpus netcdf_gdal vs netcdf_gbx reader bench - -Adds a format-parameterized .nc reader-bench path and a TropomiDownloader- -staged real S5P L2 CH4 granule corpus (download-and-stop; skips cleanly -when unavailable). Documents the corpus recipe and swath-vs-grid caveat -in benchmarking.mdx. - -Co-authored-by: Isaac" -``` - ---- - -## Sequencing note - -Tasks 1 → 2 → 3 → 4 are the spec's implementation order (light contract first, then heavy raster, then heavy vector, then the parity gate). Tasks 5 (docs) and 6 (bench) come last and can be done in either order. The parity test (Task 4) is what refines the heavy grid filter (Task 2 Step 5) — expect to revisit the filter once after Task 4 runs in Docker. - -## Loose ends to surface at "done" (per report-loose-ends-after-spec-execution) - -- The light NetCDF **writer** is a separate later cycle on this branch — NOT in this plan. -- Wheel rebuild + Volume staging is required after the Task 1 Python change (per `whl-change-rebuild-and-stage`). -- The heavy readers ship in the JAR; cluster benching needs the JAR staged before cluster start (per `jar-stage-before-cluster-start`). -- Re-verify memory-based gates live at completion: `gh auth switch --user mjohns-databricks` before any push; `gbx:lint:python --check` (Docker black) and `gbx:lint:scalastyle`; run affected package tests. diff --git a/docs/superpowers/plans/2026-07-28-light-netcdf-writer.md b/docs/superpowers/plans/2026-07-28-light-netcdf-writer.md deleted file mode 100644 index f87ec5b88..000000000 --- a/docs/superpowers/plans/2026-07-28-light-netcdf-writer.md +++ /dev/null @@ -1,533 +0,0 @@ -# Light `netcdf_gbx` writer Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a Serverless-safe light `netcdf_gbx` WRITER — the symmetric inverse of the light `netcdf_gbx` reader — with a raster mode (grid tiles → CF grid NetCDF, one `.nc` per row) and a vector mode (points → CF Discrete Sampling Geometry NetCDF, one `.nc` per partition), plus a light-only writer throughput benchmark and a reader `_FillValue` cross-tier parity defense. - -**Architecture:** A `writer(schema, overwrite)` method on `NetcdfGbxDataSource` dispatches on the `mode` option (same as the reader) to one of two `DataSourceWriter`s in a new `ds/_write_netcdf.py`. Both use the DataSource V2 `write(iterator)` per-partition path (no pandas_udf), encode with `netCDF4.Dataset` to a worker-local temp file, then `shutil.copyfile` to the scheme-stripped `/Volumes` path. Raster inverts `_netcdf.grid_transform_crs`/`array_2d`; vector inverts `point_arrays`. - -**Tech Stack:** Python 3.12 / PySpark DataSource V2 / `netCDF4` (encode) / `rasterio` (decode tiles) / `shapely` (point WKB) / `numpy`. Tests: pytest (local Spark) in Docker; cluster jobs.submit for the writer bench (human-gated). - -## Global Constraints - -- **Serverless-safe:** NO `spark.conf.set`, `_jvm`, `.rdd`, `cache`, `persist` in the writer module. The DataSource V2 `write(iterator)` path is per-partition Python; no Spark-config mutation. -- **FUSE write discipline** (`volumes-cleanpath-bare-not-file`): strip the path scheme with `_listing.to_local_path(path)` in the constructor; `netCDF4.Dataset` writes to a worker-local `tempfile` (it needs random-access construction — writing directly to a Volume FUSE path can corrupt); move to the Volume with `shutil.copyfile` (NOT `shutil.copy`/`copy2` — they `chmod` and FUSE rejects it; NOT `os.rename` — unreliable on FUSE). -- **No aliases** — one canonical writer per format; `mode` option selects raster/vector (mirrors the reader). -- **Mirror the existing writer** (`ds/writer.py` `RasterGbxWriter`): `write(iterator)` per partition, `commit` no-op, `abort` deletes written paths; `overwrite` globs+removes stale `*.nc` under the scheme-stripped path. -- **Round-trip is the primary test gate:** write → re-read with the reader → assert values/CRS/transform/nodata (raster) and lon/lat/attrs (vector) match. -- **No CI plumbing change:** `netcdf4` is already in `requirements-pyrx-ci.in` + `pyproject.toml [light]`; `test/ds/` is already in `_LIGHT_TEST_DIRS` and both CI action lists. Do NOT add dependency/test-dir entries. -- **Docker:** Python tests run via `bash scripts/commands/gbx-test-python.sh --path --log .log` in the `geobrix-dev` container. RUN TESTS SYNCHRONOUSLY to completion (Python tests are seconds); do NOT background-monitor and return early. Lint via `gbx-lint-python.sh --fix` then verify Docker `--check` (host/Docker black can differ — Docker is authoritative). -- **Commit mechanics** (a repo pre-commit guard false-positives on piping + leading-dash tokens): write each commit message to a temp file and `git commit -F /tmp/.txt`; do NOT pipe the commit through `tail`/`2>&1`; do NOT put leading-dash tokens in the message body. End every message with the `Co-authored-by: Isaac` trailer. Do NOT push. Do NOT switch git accounts. -- **Wheel:** after the light package change, rebuild + restage the wheel (`whl-change-rebuild-and-stage`) — done in the bench task (Task 6) before the cluster run. - ---- - -### Task 1: Raster writer — `NetcdfRasterGbxWriter` + `NetcdfGbxDataSource.writer` dispatch - -Grid tile → CF grid NetCDF, one `.nc` per row. Inverts the raster reader. - -**Files:** -- Create: `python/geobrix/src/databricks/labs/gbx/ds/_write_netcdf.py` -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/netcdf.py` (add `writer()` dispatch) -- Test: `python/geobrix/test/ds/test_netcdf_writer.py` - -**Interfaces:** -- Consumes: `ds.writer.assert_write_schema(schema)` (exact `(source, tile)`); `ds._listing.to_local_path(path)`; `rasterio.io.MemoryFile`; `netCDF4.Dataset`. Reader inverse target: `_netcdf.grid_transform_crs` (Affine: `ulx=min(lons)-px/2`, `uly=max(lats)+py/2`) and `_netcdf.array_2d` (north-up). -- Produces: `NetcdfRasterGbxWriter(options: dict, schema: StructType, overwrite: bool)` with `write(iterator) -> NetcdfCommitMessage(paths=[...])`, `commit`, `abort`. `NetcdfGbxDataSource.writer(schema, overwrite)` returns it for `mode="raster"`. Output var name = parsed from `source` selector `NETCDF:"…":var`, else `varNameCol` option, else `"data"`. Filename = `nameCol` (basename), else var name, else content-hash+uuid. - -- [ ] **Step 1: Write the failing raster round-trip test** - -`python/geobrix/test/ds/test_netcdf_writer.py`. Build a known grid, read it via `netcdf_gbx`, write via `netcdf_gbx`, re-read, compare. Reuse the `_write_regular_grid` helper pattern from `test_netcdf_datasource.py`. - -```python -import numpy as np -import pytest -from netCDF4 import Dataset -from databricks.labs.gbx.ds.netcdf import NetcdfGbxDataSource - - -def _write_regular_grid(path, var="ch4"): - with Dataset(path, "w") as ds: - ds.createDimension("lat", 3); ds.createDimension("lon", 4) - lat = ds.createVariable("lat", "f8", ("lat",)); lat.standard_name = "latitude" - lon = ds.createVariable("lon", "f8", ("lon",)); lon.standard_name = "longitude" - lat[:] = [50.0, 49.5, 49.0]; lon[:] = [10.0, 10.5, 11.0, 11.5] - v = ds.createVariable(var, "f4", ("lat", "lon"), fill_value=-9999.0) - v[:] = np.arange(12, dtype="float32").reshape(3, 4) - - -def test_raster_write_roundtrip(spark, tmp_path): - src = tmp_path / "in.nc"; _write_regular_grid(str(src)) - outdir = tmp_path / "out" - spark.dataSource.register(NetcdfGbxDataSource) - df = spark.read.format("netcdf_gbx").load(str(src)) # (source, tile), 1 grid var - df.write.format("netcdf_gbx").mode("overwrite").save(str(outdir)) - # re-read the written .nc - re = spark.read.format("netcdf_gbx").load(str(outdir)).collect() - assert len(re) == 1 - from rasterio.io import MemoryFile - with MemoryFile(bytes(re[0]["tile"]["raster"])) as mf, mf.open() as ds: - arr = ds.read(1); epsg = ds.crs.to_epsg() - np.testing.assert_allclose(arr, np.arange(12, dtype="float32").reshape(3, 4), rtol=1e-6) - assert epsg == 4326 -``` - -- [ ] **Step 2: Run to verify failure** - -Run (synchronous): `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_netcdf_writer.py -k roundtrip --log netcdf-writer.log` -Expected: FAIL — `netcdf_gbx` has no writer (`writer()` not implemented / unsupported save). - -- [ ] **Step 3: Implement `NetcdfRasterGbxWriter` in `_write_netcdf.py`** - -```python -"""netcdf_gbx writer (DataSource V2). Inverse of the netcdf_gbx reader. - -Raster mode: (source, tile) grid tiles -> CF grid NetCDF, one .nc per row. -Serverless-safe: write(iterator) + netCDF4 encode to worker-local temp then -shutil.copyfile to the FUSE path. No spark.conf/_jvm/.rdd. -""" -from __future__ import annotations - -import glob -import os -import shutil -import tempfile -import uuid -from dataclasses import dataclass -from typing import Iterator, List, Optional - -from pyspark.sql.datasource import DataSourceWriter, WriterCommitMessage -from pyspark.sql.types import StructType - - -@dataclass -class NetcdfCommitMessage(WriterCommitMessage): - paths: List[str] - - -def _var_from_source(source: Optional[str]) -> Optional[str]: - # source is NETCDF:"/path/file.nc":var -> return the trailing var, else None. - if source and source.startswith("NETCDF:") and ":" in source: - return source.rsplit(":", 1)[-1] or None - return None - - -class NetcdfRasterGbxWriter(DataSourceWriter): - def __init__(self, options: dict, schema: StructType, overwrite: bool): - from databricks.labs.gbx.ds.writer import assert_write_schema - from databricks.labs.gbx.ds._listing import to_local_path - - assert_write_schema(schema) # exact (source, tile) - self.path = to_local_path(options.get("path")) - self.overwrite = overwrite - self.name_col = options.get("nameCol") - self.var_name_col = options.get("varNameCol") - if overwrite and os.path.isdir(self.path): - for stale in glob.glob(os.path.join(self.path, "*.nc")): - try: - os.remove(stale) - except OSError: - pass - - def write(self, iterator: Iterator) -> WriterCommitMessage: - from rasterio.io import MemoryFile - from netCDF4 import Dataset - import numpy as np - - os.makedirs(self.path, exist_ok=True) - written: List[str] = [] - for row in iterator: - source = row["source"] - raster_bytes = bytes(row["tile"]["raster"]) - with MemoryFile(raster_bytes) as mf, mf.open() as ds: - arr = ds.read(1) - transform = ds.transform - epsg = ds.crs.to_epsg() if ds.crs else None - nodata = ds.nodata - h, w = arr.shape[-2], arr.shape[-1] - # pixel-center 1-D coords; array_2d is north-up so transform.e < 0 -> lat descending - lon = np.array([transform.c + transform.a * (i + 0.5) for i in range(w)]) - lat = np.array([transform.f + transform.e * (j + 0.5) for j in range(h)]) - # variable name: varNameCol override -> source selector -> "data" - var = None - if self.var_name_col and row[self.var_name_col]: - var = os.path.basename(str(row[self.var_name_col])) - var = var or _var_from_source(source) or "data" - # filename - if self.name_col and row[self.name_col]: - stem = os.path.basename(str(row[self.name_col])) - else: - stem = var if _var_from_source(source) else uuid.uuid4().hex[:12] - - tmp = tempfile.NamedTemporaryFile(suffix=".nc", delete=False) - tmp.close() - try: - nc = Dataset(tmp.name, "w") - try: - nc.createDimension("lat", h) - nc.createDimension("lon", w) - vlat = nc.createVariable("lat", "f8", ("lat",)) - vlat.standard_name = "latitude"; vlat.units = "degrees_north"; vlat[:] = lat - vlon = nc.createVariable("lon", "f8", ("lon",)) - vlon.standard_name = "longitude"; vlon.units = "degrees_east"; vlon[:] = lon - kw = {} if nodata is None else {"fill_value": nodata} - dv = nc.createVariable(var, arr.dtype.str[1:], ("lat", "lon"), **kw) - if epsg and epsg != 4326: - crs = nc.createVariable("crs", "i4") - crs.grid_mapping_name = "latitude_longitude" - crs.spatial_epsg = int(epsg) - dv.grid_mapping = "crs" - dv[:] = arr - finally: - nc.close() - out = os.path.join(self.path, f"{stem}.nc") - shutil.copyfile(tmp.name, out) # FUSE-safe (no chmod, no rename) - written.append(out) - finally: - os.unlink(tmp.name) - return NetcdfCommitMessage(paths=written) - - def commit(self, messages): - return None - - def abort(self, messages): - for msg in messages: - if isinstance(msg, NetcdfCommitMessage): - for p in msg.paths: - try: - os.remove(p) - except OSError: - pass -``` - -- [ ] **Step 4: Wire `writer()` dispatch in `netcdf.py`** - -Add to `NetcdfGbxDataSource`: - -```python - def writer(self, schema, overwrite: bool): - mode = self._mode() - if mode == "raster": - from databricks.labs.gbx.ds._write_netcdf import NetcdfRasterGbxWriter - if not self.options.get("path"): - raise ValueError("netcdf_gbx writer requires an output path (.save(path)).") - return NetcdfRasterGbxWriter(self.options, schema, overwrite) - if mode == "vector": - from databricks.labs.gbx.ds._write_netcdf import NetcdfVectorGbxWriter - if not self.options.get("path"): - raise ValueError("netcdf_gbx writer requires an output path (.save(path)).") - return NetcdfVectorGbxWriter(self.options, schema, overwrite) - raise ValueError(f"netcdf_gbx: unknown mode={mode!r} (use 'raster' or 'vector').") -``` - -(Task 1 only exercises the raster branch; the vector import resolves in Task 2. If PySpark eagerly imports at `writer()` call time for vector, that's fine — Task 1 tests only call raster.) - -- [ ] **Step 5: Run raster round-trip to green** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_netcdf_writer.py -k roundtrip --log netcdf-writer.log` -Expected: PASS. - -- [ ] **Step 6: Add raster edge-case tests + run** - -Add: `test_raster_write_overwrite_clears_stale`, `test_raster_write_nameCol`, `test_raster_write_non4326_crs` (write a grid with EPSG:27700 source CRS → re-read → CRS preserved), `test_raster_write_nodata_preserved`. Run the raster subset to green. - -- [ ] **Step 7: Lint + commit** - -Run: `bash scripts/commands/gbx-lint-python.sh --fix` then Docker `--check` (confirm the new files clean; ignore the pre-existing `test_vector_raster_bridge.py`). - -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/_write_netcdf.py \ - python/geobrix/src/databricks/labs/gbx/ds/netcdf.py \ - python/geobrix/test/ds/test_netcdf_writer.py -git commit -F /tmp/task1-msg.txt # "feat(netcdf): light netcdf_gbx raster writer (grid tile -> CF grid .nc)" -``` - ---- - -### Task 2: Vector writer — `NetcdfVectorGbxWriter` (points → CF-DSG, one `.nc` per partition) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/_write_netcdf.py` (add `NetcdfVectorGbxWriter`) -- Test: `python/geobrix/test/ds/test_netcdf_writer.py` - -**Interfaces:** -- Consumes: the dynamic vector schema (attribute columns + `geom_0` WKB + `geom_0_srid` + `geom_0_srid_proj`); `shapely.from_wkb`; `netCDF4.Dataset`; `_netcdf.np_to_spark` (for type mapping reference — writer maps the other way, Spark col type → numpy). -- Produces: `NetcdfVectorGbxWriter(options, schema, overwrite)` — `write(iterator)` collects the partition's rows and writes ONE CF-DSG `.nc` (`featureType="point"`, `obs` dim, `latitude`/`longitude` coord vars, one data var per attribute with `coordinates="latitude longitude"`). Empty partition → empty commit message (no file). Filename = `nameCol` (first row) else `part-.nc`. - -- [ ] **Step 1: Write the failing vector round-trip test** - -```python -def test_vector_write_roundtrip(spark, tmp_path): - import shapely - from databricks.labs.gbx.ds.netcdf import NetcdfGbxDataSource - # build a points DataFrame matching the vector reader's output schema - from pyspark.sql.types import (StructType, StructField, FloatType, IntegerType, - BinaryType, StringType) - schema = StructType([ - StructField("ch4", FloatType(), True), - StructField("qa_value", IntegerType(), True), - StructField("geom_0", BinaryType(), True), - StructField("geom_0_srid", StringType(), True), - StructField("geom_0_srid_proj", StringType(), True), - ]) - pts = [(float(i), i % 2, - bytes(shapely.to_wkb(shapely.Point(10.0 + i * 0.1, 50.0 + i * 0.1))), - "4326", "EPSG:4326") for i in range(5)] - df = spark.createDataFrame(pts, schema) - spark.dataSource.register(NetcdfGbxDataSource) - outdir = tmp_path / "vout" - (df.write.format("netcdf_gbx").option("mode", "vector") - .mode("overwrite").save(str(outdir))) - re = (spark.read.format("netcdf_gbx").option("mode", "vector") - .option("variables", "ch4,qa_value").load(str(outdir)).orderBy("ch4").collect()) - assert len(re) == 5 - pt0 = shapely.from_wkb(bytes(re[0]["geom_0"])) - assert pt0.x == pytest.approx(10.0) and pt0.y == pytest.approx(50.0) - assert re[1]["qa_value"] == 1 -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_netcdf_writer.py -k vector_write --log netcdf-writer.log` -Expected: FAIL — `NetcdfVectorGbxWriter` not defined (ImportError in the vector `writer()` branch). - -- [ ] **Step 3: Implement `NetcdfVectorGbxWriter`** - -```python -class NetcdfVectorGbxWriter(DataSourceWriter): - def __init__(self, options: dict, schema: StructType, overwrite: bool): - from databricks.labs.gbx.ds._listing import to_local_path - names = [f.name for f in schema.fields] - if "geom_0" not in names or "geom_0_srid" not in names: - raise ValueError( - "netcdf_gbx vector writer requires the vector schema " - "(attributes + geom_0 + geom_0_srid[+ geom_0_srid_proj]); got " - f"{names}") - self.path = to_local_path(options.get("path")) - self.overwrite = overwrite - self.name_col = options.get("nameCol") - self.attr_cols = [n for n in names if not n.startswith("geom_0")] - self.dtypes = {f.name: f.dataType for f in schema.fields} - if overwrite and os.path.isdir(self.path): - for stale in glob.glob(os.path.join(self.path, "*.nc")): - try: - os.remove(stale) - except OSError: - pass - - def write(self, iterator: Iterator) -> WriterCommitMessage: - import shapely - import numpy as np - from netCDF4 import Dataset - from pyspark.sql.types import IntegerType, LongType, FloatType, DoubleType - - os.makedirs(self.path, exist_ok=True) - lons: list = []; lats: list = [] - attrs = {c: [] for c in self.attr_cols} - srids = set() - name = None - for row in iterator: - if name is None and self.name_col and row[self.name_col]: - name = os.path.basename(str(row[self.name_col])) - pt = shapely.from_wkb(bytes(row["geom_0"])) - lons.append(pt.x); lats.append(pt.y) - if row["geom_0_srid"] is not None: - srids.add(str(row["geom_0_srid"])) - for c in self.attr_cols: - attrs[c].append(row[c]) - if not lons: - return NetcdfCommitMessage(paths=[]) # empty partition -> no file - - def _np_dtype(col): - dt = self.dtypes[col] - if isinstance(dt, (IntegerType,)): return "i4" - if isinstance(dt, (LongType,)): return "i8" - if isinstance(dt, (FloatType,)): return "f4" - if isinstance(dt, (DoubleType,)): return "f8" - return "f8" - - stem = name or f"part-{uuid.uuid4().hex[:8]}" - tmp = tempfile.NamedTemporaryFile(suffix=".nc", delete=False); tmp.close() - try: - nc = Dataset(tmp.name, "w") - try: - nc.featureType = "point" - n = len(lons) - nc.createDimension("obs", n) - vlat = nc.createVariable("latitude", "f8", ("obs",)) - vlat.standard_name = "latitude"; vlat.units = "degrees_north"; vlat[:] = np.array(lats) - vlon = nc.createVariable("longitude", "f8", ("obs",)) - vlon.standard_name = "longitude"; vlon.units = "degrees_east"; vlon[:] = np.array(lons) - if len(srids) == 1 and next(iter(srids)) not in ("4326", None): - crs = nc.createVariable("crs", "i4"); crs.spatial_epsg = int(next(iter(srids))) - for c in self.attr_cols: - dv = nc.createVariable(c, _np_dtype(c), ("obs",)) - dv.coordinates = "latitude longitude" - dv[:] = np.array(attrs[c]) - finally: - nc.close() - out = os.path.join(self.path, f"{stem}.nc") - shutil.copyfile(tmp.name, out) - finally: - os.unlink(tmp.name) - return NetcdfCommitMessage(paths=[out]) - - def commit(self, messages): - return None - - def abort(self, messages): - for msg in messages: - if isinstance(msg, NetcdfCommitMessage): - for p in msg.paths: - try: - os.remove(p) - except OSError: - pass -``` - -- [ ] **Step 4: Run vector round-trip to green** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_netcdf_writer.py -k vector_write --log netcdf-writer.log` -Expected: PASS. - -- [ ] **Step 5: Add vector edge cases + full-suite run** - -Add: `test_vector_write_featuretype_and_obs` (open output with plain `netCDF4`, assert `featureType=="point"` + `obs` dim size), `test_vector_write_empty_partition_no_file`, `test_vector_write_nameCol`. Run the WHOLE `test_netcdf_writer.py` to green. - -- [ ] **Step 6: Serverless-safety guard + lint + commit** - -Run: `grep -nE "spark\\.conf\\.set|_jvm|\\.rdd|\\.cache\\(|\\.persist\\(" python/geobrix/src/databricks/labs/gbx/ds/_write_netcdf.py` — expect nothing. Lint (`--fix` then Docker `--check`). - -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/_write_netcdf.py \ - python/geobrix/test/ds/test_netcdf_writer.py -git commit -F /tmp/task2-msg.txt # "feat(netcdf): light netcdf_gbx vector writer (points -> CF-DSG .nc)" -``` - ---- - -### Task 3: Reader `_FillValue` cross-tier parity defense - -Extend the reader parity test so the "equivalent physical values" claim is verified on a fill cell; if heavy diverges, fix heavy to map the unscaled fill → NaN. - -**Files:** -- Modify: `python/geobrix/test/ds/test_netcdf_cross_tier.py` -- Possibly modify: `src/main/scala/.../rasterx/operations/WindowedExtract.scala` (only if heavy diverges — see Step 3) - -**Interfaces:** consumes both registered readers (needs the heavy JAR — this task's cluster run is folded into Task 6, but the fixture change + local-light assertions land here). - -- [ ] **Step 1: Add a `_FillValue` cell to the scaled-grid fixture** - -In `test_netcdf_cross_tier.py`'s `_write_scaled_grid`, set at least one pixel to the raw `_FillValue` (e.g. `v[0, 0] = -32768`). Add an assertion in `test_netcdf_gdal_applies_scale_matches_light` that the fill cell matches across tiers (light → NaN via mask_and_scale). Use `np.testing.assert_allclose(..., equal_nan=True)` so NaN==NaN passes and NaN!=number fails. - -- [ ] **Step 2: Run in Docker (needs heavy JAR)** - -This requires the JAR with Tasks 1+2 is NOT needed (this is a reader test) but DOES need the heavy `netcdf_gdal` JAR present. Run in the writer-bench cluster window (Task 6) OR locally if a JAR-backed session is available: -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_netcdf_cross_tier.py --with-integration --log netcdf-fillcell.log` -Expected: either PASS (heavy already maps fill→NaN — claim defended) or FAIL showing light=NaN vs heavy=decoded-sentinel. - -- [ ] **Step 3: If it FAILS — fix heavy to map unscaled fill → NaN** - -In the `applyScale` path (`WindowedExtract.fallback` with `-unscale`), ensure the source `_FillValue`/nodata is carried so decoded fill cells become NaN (matching light). The `gdal_translate -unscale` path preserves nodata; verify the output tile's nodata is set and the reader surfaces it as NaN. If GDAL's `-unscale` does not map the fill, add `-a_nodata` handling or post-process the decoded tile's nodata. Re-run Step 2 to green. (If this proves out-of-scope/complex, STOP and report — do NOT silently soften the doc claim; escalate the decision.) - -- [ ] **Step 4: Commit** - -```bash -git add python/geobrix/test/ds/test_netcdf_cross_tier.py \ - [src/main/scala/.../WindowedExtract.scala if changed] -git commit -F /tmp/task3-msg.txt # "test(netcdf): defend cross-tier parity on a _FillValue cell" -``` - ---- - -### Task 4: Writer benchmark harness (raster + vector legs) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/cluster.py` (`--benchmark-netcdf-writer`/`--netcdf-writer-only` cell) -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/readers.py` (if a `stage`/helper tweak is needed; `run_format_write` already exists) -- Modify: `notebooks/tests/push_and_run_bench_on_cluster.py` (parse the new flags, thread to the cell) -- Test: `python/geobrix/test/bench/` smoke tests (imports, skip-clean when corpus empty) - -**Interfaces:** -- Consumes: `readers.run_format_write(spark, input_path, out_path, run_id, warmup, measured, read_fmt=, write_fmt=, mode="overwrite", options=)` (existing). Produces: two writer-bench ResultRows (light-only — no heavy NetCDF writer): raster (`read_fmt="netcdf_gbx"` grid over `{CORPUS}/netcdf` → `write_fmt="netcdf_gbx"` to `{CORPUS}/netcdf-out`) and vector (`read_fmt="netcdf_gbx"` mode=vector over `{CORPUS}/netcdf-swath` → `write_fmt="netcdf_gbx"` mode=vector to `{CORPUS}/netcdf-swath-out`). - -- [ ] **Step 1: Write failing smoke test for the flag** - -In `python/geobrix/test/bench/`, assert the new `--netcdf-writer-only` flag parses and the cell is emitted when set (mirror the existing netcdf/pmtiles flag tests). - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/bench/ -k netcdf_writer --log bench-netcdf-writer.log` -Expected: FAIL — flag/cell not present. - -- [ ] **Step 3: Add the writer-bench flag + cell** - -In `cluster.py`, add `BENCHMARK_NETCDF_WRITER`/`NETCDF_WRITER_ONLY` (mirror `_CELL_NETCDF`) and a `_CELL_NETCDF_WRITER` string cell with two legs calling `run_format_write` (raster + vector), `mode="overwrite"`, light-only (comment: heavy has no netcdf writer). Guard: skip clean if the input corpus is empty; log row/granule counts. Add the flags to `push_and_run_bench_on_cluster.py` arg parsing + the reader-only exclusion set (so a writer-only run doesn't require the function-corpus scaffold — reuse the lazy `corpus.json` guard). - -- [ ] **Step 4: Run smoke tests to green + commit** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/bench/ --log bench-netcdf-writer.log` -Expected: PASS. - -```bash -git add python/geobrix/src/databricks/labs/gbx/bench/cluster.py \ - python/geobrix/src/databricks/labs/gbx/bench/readers.py \ - notebooks/tests/push_and_run_bench_on_cluster.py \ - python/geobrix/test/bench/ -git commit -F /tmp/task4-msg.txt # "bench(netcdf): light netcdf_gbx writer throughput legs (raster + vector)" -``` - ---- - -### Task 5: Docs - -**Files:** -- Modify: `docs/docs/readers/netcdf.mdx` (document the writer: both modes, options `mode`/`nameCol`/`varNameCol`, round-trip, vector one-file-per-partition, scale/offset + non-EPSG-CRS limitations) -- Modify: `docs/docs/api/benchmarking.mdx` (the writer-bench legs) -- Modify: `docs/docs/beta-release-notes.mdx` (new `netcdf_gbx` writer) - -- [ ] **Step 1: Write the docs** - -Document the writer accurately: `df.write.format("netcdf_gbx").save(path)` (raster default), `.option("mode","vector")` for DSG points. Note the two documented limitations (writes decoded physical values, no integer re-packing; non-EPSG CRS stored as WKT attr). USER-FACING VOICE — no internal vocabulary (no wave/subagent/phase/task/spec refs). Run `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/` → nothing new. - -- [ ] **Step 2: Commit** - -```bash -git add docs/docs/readers/netcdf.mdx docs/docs/api/benchmarking.mdx docs/docs/beta-release-notes.mdx -git commit -F /tmp/task5-msg.txt # "docs(netcdf): document the light netcdf_gbx writer + writer bench" -``` - ---- - -### Task 6: Wheel rebuild + at-scale writer bench (human-gated) - -**Files:** none (build + cluster run). - -- [ ] **Step 1: Rebuild + restage the wheel/JAR** - -The light change (Tasks 1-2) requires a wheel rebuild; Task 3 may touch the JAR. Dispatch: `set -a; source notebooks/tests/databricks_cluster_config.env; set +a` then `bash scripts/commands/gbx-data-push-wheel.sh` (rebuilds JAR + tests.jar + wheel, uploads); sync the wheel to `sample-data/` (per `bench-wheel-path-divergence`). - -- [ ] **Step 2: (Re)start cluster + run the writer bench** - -Start `0519-143423-0jwqt79u`, poll RUNNING + libs INSTALLED. The NASA-NEX raster corpus (`{CORPUS}/netcdf`) + S5P swath corpus (`{CORPUS}/netcdf-swath`) are already staged from the reader-bench cycle. Run `bash scripts/commands/gbx-bench-cluster.sh --netcdf-writer-only --row-counts 1000`. Confirm both writer legs converge; give the run's `summary.md` link. Record the raster + vector writer throughput in the ledger. - -- [ ] **Step 3: Run the `_FillValue` reader parity test on the warm cluster** - -With the cluster up + fresh JAR, run the Task-3 test with `--with-integration` to verify (or, if it fails, confirm the Task-3 heavy fix landed and re-verify). Record the parity result. - -- [ ] **Step 4: Stop the cluster** - -`databricks clusters delete 0519-143423-0jwqt79u --profile oauth-fe` once captured (`stop-clusters-you-start`). - ---- - -## Sequencing note - -Tasks 1→2 build the writer (Python; 1 wheel covers both). Task 3 (reader parity fixture) is independent test work (its cluster verification folds into Task 6). Task 4 (bench harness) is Python plumbing. Task 5 is docs. Task 6 is the human-gated wheel-rebuild + at-scale writer bench + the on-cluster `_FillValue` verification. Tasks 1–5 are local; only Task 6 needs the cluster. - -## Loose ends to surface at "done" (per report-loose-ends-after-spec-execution) - -- Scale/offset packing (writer emits decoded physical values, no integer re-compression) — documented limitation, possible follow-up. -- Non-EPSG CRS fidelity (WKT-in-attr fallback) — documented limitation. -- The `_FillValue` heavy fix (Task 3 Step 3) only materializes if the on-cluster test shows divergence — report which way it went. -- Wheel rebuild+restage is required after Tasks 1-2 (done in Task 6). diff --git a/docs/superpowers/plans/2026-07-28-netcdf-writer-singlefile.md b/docs/superpowers/plans/2026-07-28-netcdf-writer-singlefile.md deleted file mode 100644 index 88ca27395..000000000 --- a/docs/superpowers/plans/2026-07-28-netcdf-writer-singlefile.md +++ /dev/null @@ -1,300 +0,0 @@ -# NetCDF writer `singleFile` mode Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a symmetric opt-in `singleFile` option to BOTH light `netcdf_gbx` writers (default `false` = current sharded "parts" output, non-breaking): vector concatenates all points into one CF-DSG `.nc`; raster merges DISTINCT variables sharing one grid into one CF grid `.nc` (erroring clearly on incompatible grids / window-tile duplicates, redirecting to `gbx_rst_merge_agg`). Plus a parts-vs-single writer-bench variant and docs. - -**Architecture:** Mirror the proven `VectorGbxWriter` two-phase pattern in `ds/vector.py`: when `singleFile=true`, `write(iterator)` on each executor writes an Arrow-IPC (feather) **fragment** into a shared `_scratch` dir and returns the fragment path in the commit message; `commit(messages)` on the driver reads all fragments, merges to ONE `.nc` written to a driver-local temp (random-access for `netCDF4.Dataset`), then `shutil.copyfile` to the target (FUSE-safe). Default (`singleFile=false`) keeps today's per-partition / per-tile scatter untouched. - -**Tech Stack:** Python 3.12 / PySpark DataSource V2 / `netCDF4` / `pyarrow.feather` (fragments) / `rasterio` (tile decode) / `shapely` (point WKB) / `numpy`. Tests: pytest (local Spark) in Docker; cluster bench (human-gated). - -## Global Constraints - -- **Non-breaking:** `singleFile` defaults `false`. The existing parts-mode tests + output shape (`part-.nc` vector, one-`.nc`-per-tile raster) must stay byte-for-byte unchanged. Regression-gate them. -- **Serverless-safe:** NO `spark.conf.set`, `_jvm`, `.rdd`, `cache`, `persist` in the writer paths. -- **FUSE discipline** (`volumes-cleanpath-bare-not-file`): `_listing.to_local_path(path)` scheme-strip; `netCDF4.Dataset` writes to a worker/driver-local `tempfile` then `shutil.copyfile` (NOT `copy`/`copy2` — they `chmod`; NOT `os.rename`). Scratch fragments live under `_scratch.new_scratch_dir()` (dot-prefixed `.gbx_scratch/`, invisible to the recursive reader, age-GC'd). -- **Two-phase contract:** `write(iterator)` returns a commit message carrying the fragment path (extend `NetcdfCommitMessage` or add a fragment-carrying message); `commit(messages)` merges on the driver; `abort(messages)` deletes fragments + any partial output. In parts-mode, `commit` stays no-op and `write` still writes final `.nc` files directly (current behavior). -- **Raster merge scope:** merges DISTINCT variables on ONE shared grid. Same grid = identical `(width, height)` + CRS EPSG exactly + geotransform within a float rtol. Incompatible grids OR duplicate varname (same-variable window-tiles) → raise `ValueError` naming the conflict AND pointing at `gbx_rst_merge_agg` / `gbx_rst_merge` for upstream mosaicking. NEVER silently mosaic or pick one. -- **Mosaic is upstream, not in the writer:** `tiles → rst_merge_agg → write(singleFile)`. Docs must state this. -- **Docker + synchronous tests:** `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_netcdf_writer.py --log .log` — run SYNCHRONOUSLY to completion (seconds), quote the pytest result line; do NOT background-monitor and return early. Lint: `gbx-lint-python.sh --fix` then Docker `--check` (pre-existing `test_vector_raster_bridge.py` flag is unrelated). -- **Commit mechanics** (repo pre-commit guard false-positives on piping + leading-dash tokens): write each message to a temp file and `git commit -F /tmp/.txt`; no piping through `tail`/`2>&1`; no leading-dash tokens in the body; end with the `Co-authored-by: Isaac` trailer. No push, no account switch. -- **Wheel:** rebuild + restage after the light change (done in Task 5's cluster run). -- **No CI plumbing change:** `netcdf4`/`pyarrow` already deps; `test/ds/` + `test/bench/` already wired. - ---- - -### Task 1: Vector `singleFile` (concat points → one CF-DSG `.nc`) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/_write_netcdf.py` (`NetcdfVectorGbxWriter`: two-phase when `singleFile`) -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/netcdf.py` (thread `singleFile` into `writer()`) -- Test: `python/geobrix/test/ds/test_netcdf_writer.py` - -**Interfaces:** -- Consumes: `_scratch.new_scratch_dir(parent)`; `pyarrow.feather` (write/read fragment tables); `_listing.to_local_path`; `netCDF4.Dataset`; the existing `_resolve_single_file_output(path, file_name, ext)` from `ds/vector.py` (import it) for the single-file target name. -- Produces: `NetcdfVectorGbxWriter(options, schema, overwrite)` honoring `options["singleFile"]` (default `"false"`). When true: `write` → feather fragment (cols `longitude`,`latitude`, attrs, + a `geom_0_srid` scalar carried in metadata) in scratch, returns a fragment-carrying commit message; `commit` → one CF-DSG `.nc` (obs-dimension, streamed fragment-by-fragment), copied to the resolved single target. When false: unchanged (per-partition `part-*.nc`, `commit` no-op). - -- [ ] **Step 1: Write the failing vector singleFile test** - -Add to `test_netcdf_writer.py`. A multi-partition points DataFrame + `singleFile=true` must yield EXACTLY ONE `.nc`, round-tripping all points/attrs. - -```python -def test_vector_write_singlefile_one_nc(spark, tmp_path): - import shapely, os - from databricks.labs.gbx.ds.netcdf import NetcdfGbxDataSource - from pyspark.sql.types import (StructType, StructField, FloatType, IntegerType, - BinaryType, StringType) - schema = StructType([ - StructField("ch4", FloatType(), True), - StructField("qa_value", IntegerType(), True), - StructField("geom_0", BinaryType(), True), - StructField("geom_0_srid", StringType(), True), - StructField("geom_0_srid_proj", StringType(), True), - ]) - pts = [(float(i), i % 2, - bytes(shapely.to_wkb(shapely.Point(10.0 + i*0.1, 50.0 + i*0.1))), - "4326", "EPSG:4326") for i in range(12)] - df = spark.createDataFrame(pts, schema).repartition(4) # multiple partitions - spark.dataSource.register(NetcdfGbxDataSource) - out = tmp_path / "vout_single" - (df.write.format("netcdf_gbx").option("mode", "vector") - .option("singleFile", "true").mode("overwrite").save(str(out))) - ncs = [f for f in os.listdir(str(out)) if f.endswith(".nc")] - assert len(ncs) == 1, f"expected ONE .nc, got {ncs}" - re = (spark.read.format("netcdf_gbx").option("mode", "vector") - .option("variables", "ch4,qa_value").load(str(out)).orderBy("ch4").collect()) - assert len(re) == 12 - import pytest - pt0 = shapely.from_wkb(bytes(re[0]["geom_0"])) - assert pt0.x == pytest.approx(10.0) and pt0.y == pytest.approx(50.0) -``` - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_netcdf_writer.py -k singlefile --log nc-writer-sf.log` -Expected: FAIL — `singleFile` ignored, output is `part-*.nc` (multiple files), assert on `len(ncs)==1` fails. - -- [ ] **Step 3: Implement vector two-phase singleFile** - -In `NetcdfVectorGbxWriter.__init__`: read `self.single_file = str(options.get("singleFile", "false")).lower() == "true"`; when single, compute the scratch dir `self.scratch = _scratch.new_scratch_dir(to_local_path(path))`. In `write(iterator)`: -- If NOT single: current behavior (write `part-*.nc`), return `NetcdfCommitMessage(paths=[...])`. -- If single: buffer `(lons, lats, attrs, srids)` as today, but instead of a `.nc`, write a feather table (`pyarrow.Table` of `longitude`, `latitude`, each attr col; store the resolved single srid + attr dtypes in the table's schema metadata) into `self.scratch` (mkdir exist_ok). Return a message carrying the fragment path. Empty partition → message with empty frag. - -Extend the commit message to carry a fragment path — simplest: add a field, e.g. -```python -@dataclass -class NetcdfCommitMessage(WriterCommitMessage): - paths: List[str] = None # parts-mode: final files written - frag_path: str = "" # singleFile-mode: this partition's scratch fragment -``` -In `commit(messages)`: -- Parts-mode: no-op (as today). -- Single-mode: collect non-empty `frag_path`s; if none, write nothing. Else resolve the single target via `_resolve_single_file_output(self.path, self.name_col_value_or_None, "nc")`; write ONE CF-DSG `.nc` to a driver-local temp with an **unlimited `obs` dimension**, then **stream**: for each fragment, `feather.read_table`, append its rows to `obs` (grow `latitude`/`longitude`/attr vars via `var[start:start+k] = ...`). Reconcile srid across fragments (all-agree non-4326 → `crs` var). `shutil.copyfile` temp → target. Clean scratch. -`abort`: delete fragments + partial output + scratch. - -- [ ] **Step 4: Thread `singleFile` through `writer()`** - -`netcdf.py` `writer()` already passes `self.options` to the writer constructors — confirm `singleFile` is in `self.options` (it is, as a `.option`). No signature change needed; the writer reads it from options. - -- [ ] **Step 5: Run vector singleFile test + parts-mode regression to green** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_netcdf_writer.py --log nc-writer-sf.log` -Expected: the new singleFile test PASSES (one `.nc`, 12 points round-trip); ALL existing vector parts tests stay green (default still `part-*.nc`). - -- [ ] **Step 6: Lint + commit** - -Run `gbx-lint-python.sh --fix` then Docker `--check`. -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/_write_netcdf.py \ - python/geobrix/src/databricks/labs/gbx/ds/netcdf.py \ - python/geobrix/test/ds/test_netcdf_writer.py -git commit -F /tmp/sf-task1.txt # "feat(netcdf): vector writer singleFile mode (concat points -> one CF-DSG .nc)" -``` - ---- - -### Task 2: Raster `singleFile` (merge distinct same-grid variables → one CF `.nc`) - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/_write_netcdf.py` (`NetcdfRasterGbxWriter`: two-phase when `singleFile`) -- Test: `python/geobrix/test/ds/test_netcdf_writer.py` - -**Interfaces:** -- Consumes: same `_scratch`, feather, `to_local_path`, `_resolve_single_file_output`, `rasterio.MemoryFile` (decode tile), `netCDF4.Dataset`. -- Produces: `NetcdfRasterGbxWriter` honoring `singleFile`. Single-mode `write` → per-row fragment capturing `{varname, array (feather 2-D as a column or a small .npy sidecar), width, height, transform(6), crs_epsg, nodata}`; `commit` → grid-compat gate then ONE CF `.nc` with shared `lat`/`lon` + one data var per distinct varname. Incompatible grid or duplicate varname → `ValueError` with `rst_merge_agg` pointer. - -- [ ] **Step 1: Write failing raster singleFile tests (merge + error cases)** - -Add three tests: -```python -def test_raster_write_singlefile_multivar(spark, tmp_path): - # two DISTINCT vars (tas, pr) on the SAME grid -> one .nc with both, sharing lat/lon - ... # build two (source, tile) rows via netCDF4 fixtures with source NETCDF:"f":tas / :pr - (df.write.format("netcdf_gbx").option("singleFile","true").mode("overwrite").save(str(out))) - ncs = [f for f in os.listdir(str(out)) if f.endswith(".nc")] - assert len(ncs) == 1 - from netCDF4 import Dataset - with Dataset(os.path.join(str(out), ncs[0])) as nc: - assert "tas" in nc.variables and "pr" in nc.variables - assert nc.variables["tas"].dimensions == ("lat","lon") - -def test_raster_write_singlefile_incompatible_grid_errors(spark, tmp_path): - # two tiles, different grid sizes/CRS + singleFile -> ValueError mentioning rst_merge_agg - with pytest.raises(Exception) as e: - df.write.format("netcdf_gbx").option("singleFile","true").mode("overwrite").save(str(out)) - assert "rst_merge_agg" in str(e.value) - -def test_raster_write_singlefile_duplicate_var_errors(spark, tmp_path): - # two tiles, SAME varname + same grid dims (window-tiles) + singleFile -> ValueError -> rst_merge_agg - ... -``` - -(Building `(source, tile)` rows: read a netCDF4-written grid via `netcdf_gbx` raster to get real tiles, or construct GTiff tiles with `source = 'NETCDF:"f":tas'`. Reuse the Task-1-cycle raster fixture helpers.) - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_netcdf_writer.py -k "singlefile_multivar or singlefile_incompatible or singlefile_duplicate" --log nc-writer-rsf.log` -Expected: FAIL — singleFile ignored (multiple `.nc`), no error raised. - -- [ ] **Step 3: Implement raster two-phase singleFile** - -`write(iterator)` single-mode: per row, decode the tile (array, transform, crs epsg, nodata, varname — reuse the existing decode + `_var_from_source`), write a fragment (feather table with the flattened array + shape + a metadata blob for transform/crs/nodata/varname, or a compact container). Return frag-carrying message. -`commit` single-mode: read all fragments. **Grid-compat gate:** collect `(width,height,crs_epsg)` + transform per fragment; require identical `(width,height)`, identical `crs_epsg`, transforms equal within rtol (e.g. `1e-9`). If any differ → `raise ValueError(f"netcdf_gbx singleFile: tiles have incompatible grids ...; to mosaic window-tiles of one variable into a single grid, use gbx_rst_merge_agg / gbx_rst_merge before writing.")`. Detect duplicate varname across fragments → same message (window-tiles are a mosaic, not a multi-var merge). If compatible + distinct varnames: write ONE CF grid `.nc` (shared `lat`/`lon` from the common transform, one data var per varname with its `_FillValue`, shared `crs` grid_mapping for non-4326) to driver-local temp → `shutil.copyfile` to the resolved single target. `abort`: clean. - -- [ ] **Step 4: Run raster singleFile tests + parts regression to green** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_netcdf_writer.py --log nc-writer-rsf.log` -Expected: multivar merges to one `.nc`; incompatible + duplicate raise with the `rst_merge_agg` pointer; existing parts raster tests stay green. - -- [ ] **Step 5: Serverless-safety grep + lint + commit** - -`grep -nE "spark\\.conf\\.set|_jvm|\\.rdd|\\.cache\\(|\\.persist\\(" python/geobrix/src/databricks/labs/gbx/ds/_write_netcdf.py` → nothing. Lint. -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/_write_netcdf.py \ - python/geobrix/test/ds/test_netcdf_writer.py -git commit -F /tmp/sf-task2.txt # "feat(netcdf): raster writer singleFile merges same-grid vars (errors -> rst_merge_agg)" -``` - ---- - -### Task 2b: `merge` + `keepParts` + `fileName` + `partPrefix` - -Add the post-hoc directory-merge option and the filename controls, refactoring the singleFile merge cores into helpers reused by both `singleFile` (fragments) and `merge` (existing `.nc` files). - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/ds/_write_netcdf.py` (both writers) -- Test: `python/geobrix/test/ds/test_netcdf_writer.py` - -**Interfaces:** -- Consumes: the vector-concat and raster-grid-merge cores from Tasks 1–2 (factor them into module-level helpers taking an iterable of "point-batch" / "grid-fragment" inputs, so both a list of `_scratch` feather fragments AND a list of on-disk `.nc` files feed the same merge). `_resolve_single_file_output(path, fileName, "nc")`. -- Produces: `merge` (default false), `keepParts` (default false), `fileName` (single/merge output stem), `partPrefix` (default "part") honored in both writers. - -- [ ] **Step 1: Write failing tests** - -Add to `test_netcdf_writer.py` (vector + raster): -- `test_vector_merge_dir_no_rerun`: write parts-mode to a dir; then `.write.format("netcdf_gbx").option("mode","vector").option("merge","true").save()` passing a DIFFERENT tiny/empty DataFrame → assert ONE merged `.nc`, parts gone (default), re-read == the ORIGINAL parts' data (proves the DataFrame was ignored, the dir was merged). -- `test_merge_keepParts_true`: same but `keepParts=true` → merged file AND parts present. -- `test_merge_failure_preserves_parts`: force a merge failure with `keepParts=false` (e.g. raster incompatible grids in the dir, or monkeypatch the validate step) → assert parts STILL present + no valid-looking partial output. -- `test_merge_empty_dir_errors`: merge a dir with no `.nc` → clear `ValueError`. -- `test_partPrefix` (parts-mode filenames `myshard-*.nc`) and `test_fileName_singlefile` (singleFile output named as given). - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/ds/test_netcdf_writer.py -k "merge or partPrefix or fileName" --log nc-merge.log` -Expected: FAIL — options unrecognized. - -- [ ] **Step 3: Implement** - -- Refactor Tasks 1–2 singleFile commit bodies into shared helpers: `_merge_vector_points(inputs, out_temp, ...)` and `_merge_raster_grids(inputs, out_temp, ...)` where `inputs` is an iterable yielding point-batches / grid-fragments regardless of source (feather fragment OR `.nc` file). singleFile passes `_scratch` fragments; `merge` passes `glob(/*.nc)` (excluding `.gbx_scratch` + the resolved output name). -- `merge` in `write(iterator)`: return immediately WITHOUT iterating (no re-run); empty commit message. -- `merge` in `commit`: glob the dir; empty → `ValueError`. Merge via the shared core → driver-local temp. **DATA-SAFETY ORDER:** validate temp (reopen + element-count == summed inputs) → `shutil.copyfile` to target → verify target size == temp → only then, if not `keepParts`, delete the part files. Any failure → raise, leave parts intact. -- Constructor: when `merge=true`, DO NOT run the overwrite glob-delete. -- `fileName` → `_resolve_single_file_output`; `partPrefix` → parts-mode stem (`-.nc`, default "part"). - -- [ ] **Step 4: Run to green + Serverless grep + lint + commit** - -Run the full `test_netcdf_writer.py` (all singleFile + merge + parts tests green). Serverless grep. Lint. -```bash -git add python/geobrix/src/databricks/labs/gbx/ds/_write_netcdf.py \ - python/geobrix/test/ds/test_netcdf_writer.py -git commit -F /tmp/sf-task2b.txt # "feat(netcdf): writer merge/keepParts/fileName/partPrefix (data-safe dir merge)" -``` - ---- - -### Task 3: Writer-bench `singleFile` variant - -**Files:** -- Modify: `python/geobrix/src/databricks/labs/gbx/bench/cluster.py` (`_CELL_NETCDF_WRITER`: add a `singleFile` measured leg per mode) -- Test: `python/geobrix/test/bench/` smoke tests - -**Interfaces:** consumes `run_format_write(... options={... "singleFile":"true"})` (the single options dict flows `singleFile` to the writer). Produces two additional writer ResultRows (raster-single, vector-single) distinguished from the parts rows via a note/fn tag so they don't collide in the store. - -- [ ] **Step 1: Write failing smoke test** - -Assert the writer cell emits BOTH a parts leg and a singleFile leg per mode (grep the emitted cell for `"singleFile": "true"`), and rows are distinguishable. Mirror the existing writer-bench smoke tests. - -- [ ] **Step 2: Run to verify failure** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/bench/ -k netcdf_writer --log bench-sf.log` -Expected: FAIL — no singleFile leg emitted. - -- [ ] **Step 3: Add the singleFile legs to `_CELL_NETCDF_WRITER`** - -After each existing parts-mode `run_format_write` leg (raster, vector), add a parallel measured leg with `options={... "singleFile": "true"}` (raster keeps `filterRegex`; vector keeps `mode=vector`+`group=/PRODUCT`+`variables=...`+`singleFile`). Tag the row so parts vs single are distinct in the store (append a suffix to the note, or a distinct out-dir like `{CORPUS}/netcdf-out-single`). Keep the skip-clean + row-count>0 guards. Distinct out-dirs per leg so overwrite doesn't clobber. - -- [ ] **Step 4: Run smoke tests to green + commit** - -Run: `bash scripts/commands/gbx-test-python.sh --path python/geobrix/test/bench/ --log bench-sf.log` -```bash -git add python/geobrix/src/databricks/labs/gbx/bench/cluster.py python/geobrix/test/bench/ -git commit -F /tmp/sf-task3.txt # "bench(netcdf): parts-vs-single writer throughput legs" -``` - ---- - -### Task 4: Docs (`singleFile` + the mosaic pattern pointer) - -**Files:** -- Modify: `docs/docs/readers/netcdf.mdx` (writer section: `singleFile` on both modes; the `rst_merge_agg` mosaic pattern; memory tradeoff) -- Modify: `docs/docs/api/benchmarking.mdx` (parts-vs-single writer legs) -- Modify: `docs/docs/beta-release-notes.mdx` (the new `singleFile` option) - -- [ ] **Step 1: Write the docs** - -Document `.option("singleFile","true")` on both writer modes (default sharded parts; opt-in single). Vector = one CF-DSG `.nc` (all points). Raster = one CF `.nc` merging DISTINCT variables that share a grid. **Call out the mosaic pattern prominently:** to combine many spatial-window tiles of ONE variable into a single grid, use `gbx_rst_merge_agg` (or `gbx_rst_merge`) BEFORE the writer — show the `SELECT rst_merge_agg(tile) ... GROUP BY ...` → `.option("singleFile","true")` flow. Note the single-file memory tradeoff (driver-funneled; sharded parts safer at very large scale). USER-FACING VOICE — no internal vocabulary; run `grep -rn -iE "wave [0-9]+|wave-[0-9]+" docs/docs/` → nothing new. - -- [ ] **Step 2: Commit** - -```bash -git add docs/docs/readers/netcdf.mdx docs/docs/api/benchmarking.mdx docs/docs/beta-release-notes.mdx -git commit -F /tmp/sf-task4.txt # "docs(netcdf): document writer singleFile mode + rst_merge_agg mosaic pattern" -``` - ---- - -### Task 5: Wheel rebuild + at-scale parts-vs-single writer bench (human-gated) - -- [ ] **Step 1: Rebuild + stage the wheel** - -`set -a; source notebooks/tests/databricks_cluster_config.env; set +a` then `GBX_BUNDLE_SKIP_JAR_UPLOAD=1 bash scripts/commands/gbx-data-push-wheel.sh` (light-only change — JAR unchanged this cycle); sync the wheel to `sample-data/` (`bench-wheel-path-divergence`). - -- [ ] **Step 2: (Re)start cluster + run the parts-vs-single writer bench** - -Start `0519-143423-0jwqt79u`, poll RUNNING + libs INSTALLED. Corpora already staged (33 NASA-NEX grids at `{CORPUS}/netcdf`, 15 S5P swaths at `{CORPUS}/netcdf-swath`). Run `bash scripts/commands/gbx-bench-cluster.sh --netcdf-writer-only --row-counts 1000`. Confirm all four legs (raster parts/single, vector parts/single) report rows>0 (the 0-row guard fails loud otherwise). Verify vector-single produces ONE `.nc` in its out-dir. Give the run's `summary.md` link. Record parts-vs-single throughput in the ledger. - -- [ ] **Step 3: Stop the cluster** - -`databricks clusters delete 0519-143423-0jwqt79u --profile oauth-fe` after capture. - ---- - -## Sequencing note - -Tasks 1 (vector) → 2 (raster) build the feature (one wheel covers both). Task 3 (bench) + Task 4 (docs) are independent Python/docs. Task 5 is the human-gated wheel-rebuild + at-scale run. Tasks 1-4 are local; only Task 5 needs the cluster. - -## Loose ends to surface at "done" (per report-loose-ends-after-spec-execution) - -- Raster singleFile is a DISTINCT-variable merge, not a window-mosaic (mosaic → `rst_merge_agg` upstream; documented + error-guarded). -- Wheel rebuild+restage after Tasks 1-2 (done in Task 5). -- The 18 prior unpushed writer-cycle commits + these — a push (and the fixed CI `__all__` test) is still pending the user's go. diff --git a/docs/superpowers/plans/2026-07-28-rasterio-distributed-page.md b/docs/superpowers/plans/2026-07-28-rasterio-distributed-page.md deleted file mode 100644 index 10b07ee98..000000000 --- a/docs/superpowers/plans/2026-07-28-rasterio-distributed-page.md +++ /dev/null @@ -1,453 +0,0 @@ -# Rasterio Distributed Page Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Ship a `docs/docs/api/rasterio-distributed.mdx` positioning page ("RasterX's lightweight tier is distributed rasterio + a best-of-breed raster stack") with tested side-by-side examples, plus short blurb+link cross-references from the highest-traffic pages, and one attribution fix. - -**Architecture:** Docusaurus MDX page under Functions→RasterX. Flagship code snippets are imported (via the existing `CodeFromTest` component) from a new tested example module; the doc-test executes BOTH the single-node rasterio side and the distributed pyrx side and asserts they agree. Coverage matrix + honest three-bucket gaps are prose/tables. Cross-links are one-sentence additions at four existing sites. No product code changes. - -**Tech Stack:** Docusaurus MDX, `CodeFromTest` React component, `raw-loader`; pytest doc-tests running in the `geobrix-dev` Docker container (`gbx:test:python-docs`); pyrx (`databricks.labs.gbx.pyrx`, `databricks.labs.gbx.ds.register`), rasterio, numpy. - -## Global Constraints - -- Branch: `beta/0.4.0`. All commits local; **no push** this task set. -- Every new `docs/docs/**.mdx` MUST be wired into `docs/sidebars.js` in the same change (manual sidebar; unwired pages are orphaned / warn on build). -- Doc examples are the single source of truth and MUST be backed by executable tests with real assertions (no mocking Spark / GeoBrix / file I/O). Tests use tiny synthesized rasters. -- User-facing voice: no internal planning vocabulary (no "wave N"); justify by user utility, not Mosaic parity. Roadmap-forward claims say "for now" / "parity tracked", never a hard date/commitment. -- `docs/tests/python/api/` is the SQL/api doc-test suite — run via `gbx:test:python-docs --suite api` (excluded from the default python-docs run). New api tests must land there. -- Docs long-running work (Docker test, static build, browser screenshot) runs via `gbx:*` commands dispatched to a subagent; browser/dev-server checks use a NON-3000 port (e.g. `--port 3001`). -- Verify facts against code before asserting: terrain (`pyrx/core/terrain.py`) is pure NumPy (Horn 3×3, `np.pad(mode='edge')`), imports only numpy/pyproj/rasterio; only viewshed (`pyrx/core/analysis.py`) uses `xrspatial`. - ---- - -### Task 1: Tested example module + doc-test (flagship side-by-sides) - -Build the single source of truth for the three flagship snippets FIRST (TDD), so the page imports proven code. - -**Files:** -- Create: `docs/tests/python/api/rasterio_distributed_examples.py` -- Create (test): `docs/tests/python/api/test_rasterio_distributed_examples.py` - -**Interfaces:** -- Produces (consumed by Task 2 via `CodeFromTest` `functionName=`): string constants - `REGISTER`, `WARP_RASTERIO`, `WARP_PYRX`, `CLIP_RASTERIO`, `CLIP_PYRX`, - `NDVI_RASTERIO`, `NDVI_PYRX`. -- Produces (verifier fns, called by the test): `warp_both(spark, src_path)`, - `clip_both(spark, src_path, wkt)`, `ndvi_both(spark, src_path)` — each runs the - rasterio side and the pyrx side and returns `(rasterio_result, pyrx_result)` for the - test to compare. -- Consumes: `databricks.labs.gbx.ds.register.register`, `databricks.labs.gbx.pyrx` (as `rx`), - rasterio, numpy. - -- [ ] **Step 1: Write the example module with snippet constants + verifier functions** - -Create `docs/tests/python/api/rasterio_distributed_examples.py`: - -```python -"""'Rasterio, Distributed' page examples — single source of truth. - -Code shown in docs/docs/api/rasterio-distributed.mdx is imported from here. -Each flagship op is shown as a familiar single-node rasterio snippet next to the -distributed pyrx equivalent. The paired verifier functions run BOTH sides and the -test asserts they agree on tiny synthesized rasters — proving the equivalence the -page claims, not merely that pyrx runs. -""" - -REGISTER = """# Register the GeoBrix lightweight (pyrx) functions once per session -from databricks.labs.gbx.ds.register import register -import databricks.labs.gbx.pyrx as rx -rx.functions.register(spark) # registers rst_* as Spark UDFs / Column helpers""" - -WARP_RASTERIO = """# Single-node rasterio: reproject one file to EPSG:3857 -import rasterio -from rasterio.warp import calculate_default_transform, reproject, Resampling -with rasterio.open("in.tif") as src: - t, w, h = calculate_default_transform( - src.crs, "EPSG:3857", src.width, src.height, *src.bounds) - # ...write a reprojected file, one machine, one file at a time""" - -WARP_PYRX = """# GeoBrix pyrx: reproject a whole DataFrame of tiles, distributed -df2 = df.withColumn("tile", rx.rst_transform("tile", 3857)) -# rst_transform runs rasterio.warp on each tile in an Arrow UDF across the cluster""" - -CLIP_RASTERIO = """# Single-node rasterio: clip one raster to a geometry -import rasterio -from rasterio.mask import mask -with rasterio.open("in.tif") as src: - out, out_transform = mask(src, [geom], crop=True)""" - -CLIP_PYRX = """# GeoBrix pyrx: clip every tile to a geometry, distributed -df2 = df.withColumn("tile", rx.rst_clip("tile", "geom"))""" - -NDVI_RASTERIO = """# Single-node rasterio + NumPy: NDVI for one raster -import rasterio, numpy as np -with rasterio.open("in.tif") as src: - red = src.read(1).astype("float32"); nir = src.read(2).astype("float32") - ndvi = (nir - red) / (nir + red)""" - -NDVI_PYRX = """# GeoBrix pyrx: NDVI across a DataFrame of tiles, distributed -df2 = df.withColumn("tile", rx.rst_ndvi("tile", 2, 1)) # (nir_band, red_band)""" - - -def _register(spark): - from databricks.labs.gbx.ds.register import register - import databricks.labs.gbx.pyrx as rx - register(spark) - rx.functions.register(spark) - - -def _one_tile_df(spark, src_path): - """Load a single-file raster as a one-row (source, tile) DataFrame via the - lightweight raster_gbx reader.""" - return spark.read.format("raster_gbx").load(src_path) -``` - -NOTE for the implementer: the exact pyrx registration entry point and the exact -`rst_transform` / `rst_clip` / `rst_ndvi` Python-Column signatures MUST be confirmed -against `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` and the existing -tested examples in `docs/tests/python/api/rasterx_functions.py` before finalizing — -copy the real call form from there. Do not invent argument orders. - -- [ ] **Step 2: Add the three paired verifier functions to the same module** - -Append to `rasterio_distributed_examples.py` — each runs the rasterio side directly -and the pyrx side through Spark, returning both results as NumPy arrays for the test -to compare: - -```python -def warp_both(spark, src_path): - """Return (rasterio_crs, pyrx_crs) after reprojecting to EPSG:3857.""" - import rasterio - from rasterio.warp import calculate_default_transform - _register(spark) - import databricks.labs.gbx.pyrx as rx - with rasterio.open(src_path) as src: - rio_crs = rasterio.crs.CRS.from_epsg(3857) - df = _one_tile_df(spark, src_path).withColumn("tile", rx.rst_transform("tile", 3857)) - pyrx_srid = df.selectExpr("rst_srid(tile) AS srid").collect()[0]["srid"] - return (rio_crs.to_epsg(), pyrx_srid) - - -def ndvi_both(spark, src_path): - """Return (rasterio_ndvi_array, pyrx_ndvi_array) for band2=nir, band1=red.""" - import numpy as np - import rasterio - _register(spark) - import databricks.labs.gbx.pyrx as rx - with rasterio.open(src_path) as src: - red = src.read(1).astype("float32") - nir = src.read(2).astype("float32") - rio_ndvi = (nir - red) / (nir + red) - df = _one_tile_df(spark, src_path).withColumn("tile", rx.rst_ndvi("tile", 2, 1)) - # read the single result tile's band 1 back to a NumPy array - tile_bytes = bytes(df.selectExpr("tile.raster AS r").collect()[0]["r"]) - with rasterio.io.MemoryFile(tile_bytes) as mf, mf.open() as ds: - pyrx_ndvi = ds.read(1).astype("float32") - return (rio_ndvi, pyrx_ndvi) -``` - -NOTE: `clip_both` follows the same shape (rasterio `mask` vs `rx.rst_clip`); include it -only if `rst_clip`'s tested call form is confirmed. If confirming clip is costly, the -page can show the clip side-by-side as illustrative code while the TEST covers warp + -ndvi (the two with clean numeric equality). Prefer testing all three; drop clip from the -test (not the page) only if its equality is not cleanly assertable. - -- [ ] **Step 3: Write the failing test** - -Create `docs/tests/python/api/test_rasterio_distributed_examples.py`: - -```python -"""Executes the 'Rasterio, Distributed' doc examples and asserts the rasterio side -and the pyrx side AGREE on synthesized rasters (Docker; api suite).""" -import sys -from pathlib import Path - -import numpy as np -import rasterio -from rasterio.transform import from_bounds - -sys.path.insert(0, str(Path(__file__).parent)) -import rasterio_distributed_examples as ex # noqa: E402 - - -def _write_rgbnir(path, px=32): - # 2-band raster: band1=red, band2=nir, EPSG:4326 over a small AOI - red = np.linspace(0, 200, px * px, dtype="float32").reshape(px, px) - nir = np.linspace(50, 255, px * px, dtype="float32").reshape(px, px) - with rasterio.open( - path, "w", driver="GTiff", width=px, height=px, count=2, dtype="float32", - crs="EPSG:4326", transform=from_bounds(-122.5, 37.7, -122.4, 37.8, px, px), - ) as ds: - ds.write(red, 1) - ds.write(nir, 2) - - -def test_warp_agrees(spark, tmp_path): - p = str(tmp_path / "in.tif") - _write_rgbnir(p) - rio_epsg, pyrx_srid = ex.warp_both(spark, p) - assert rio_epsg == 3857 and pyrx_srid == 3857 - - -def test_ndvi_agrees(spark, tmp_path): - p = str(tmp_path / "in.tif") - _write_rgbnir(p) - rio_ndvi, pyrx_ndvi = ex.ndvi_both(spark, p) - assert rio_ndvi.shape == pyrx_ndvi.shape - assert np.allclose(rio_ndvi, pyrx_ndvi, rtol=1e-4, atol=1e-4, equal_nan=True) -``` - -- [ ] **Step 4: Run the test to verify it fails (then iterate to green)** - -Dispatch a subagent to run in Docker (do not run inline). Command: - -```bash -bash scripts/commands/gbx-test-python-docs.sh --suite api \ - --path api/test_rasterio_distributed_examples.py \ - --log rasterio-distributed-doctest.log -``` - -Expected first run: FAIL (e.g. wrong pyrx call form / registration). Iterate on the -example module until both tests PASS. The verifier is the definition of done — the -rasterio and pyrx results must actually agree, not just execute. - -- [ ] **Step 5: Commit** - -```bash -git add docs/tests/python/api/rasterio_distributed_examples.py \ - docs/tests/python/api/test_rasterio_distributed_examples.py -git commit -m "docs(test): tested rasterio-vs-pyrx equivalence examples for RasterX - -Both the single-node rasterio side and the distributed pyrx side are -executed and asserted to agree on synthesized rasters, backing the -'Rasterio, Distributed' page's side-by-side snippets. - -Co-authored-by: Isaac" -``` - ---- - -### Task 2: The `rasterio-distributed.mdx` page + sidebar wiring - -**Files:** -- Create: `docs/docs/api/rasterio-distributed.mdx` -- Modify: `docs/sidebars.js` (RasterX category `items`, currently `['api/h3-raster-tessellation']` at line ~109) - -**Interfaces:** -- Consumes: the snippet constants from Task 1 via `CodeFromTest` (`functionName="WARP_RASTERIO"`, etc.) and `raw-loader` import of `docs/tests/python/api/rasterio_distributed_examples.py`. -- Produces: doc id `api/rasterio-distributed` (referenced by Task 3 cross-links + sidebar). - -- [ ] **Step 1: Write the page** - -Create `docs/docs/api/rasterio-distributed.mdx`. Frontmatter: - -```mdx ---- -sidebar_position: 3 -sidebar_label: Rasterio, Distributed -title: Rasterio, Distributed ---- - -import CodeFromTest from '@site/src/components/CodeFromTest'; -import rioDist from '!!raw-loader!../../tests/python/api/rasterio_distributed_examples.py'; -``` - -Body sections (write real prose — no placeholders): -1. **Hook** — RasterX's lightweight (`pyrx`) tier is, in large part, `rasterio` plus a - best-of-breed Python raster stack (rio-tiler, rio-cogeo, scipy, xarray-spatial, - scikit-image, shapely, pyproj, h3/quadbin), run as Arrow UDFs / UDTFs across the - cluster — not a single-node `rasterio.open` loop, and not a reimplementation. No JAR, - no init script, no native GDAL install. -2. **Register** — ``. -3. **Side-by-sides** — three subsections (Reproject / Clip / NDVI), each with the rasterio - snippet then the pyrx snippet via `CodeFromTest` (`WARP_RASTERIO`+`WARP_PYRX`, - `CLIP_*`, `NDVI_*`). One sentence each: pyrx runs the same rasterio/NumPy compute per - tile, distributed. `testFile="docs/tests/python/api/test_rasterio_distributed_examples.py"`. -4. **Coverage matrix** — table: capability → pyrx function(s) → backing library, grouped - (I/O & metadata → rasterio; warp/reproject → rasterio; clip/mask → rasterio+shapely; - resample → rasterio; merge → rasterio+NumPy; COG → rio-cogeo; band math / indices → - NumPy+numexpr; terrain (Horn 3×3) → NumPy; rasterize/polygonize → rasterio+shapely; - focal → scipy.ndimage; proximity → scipy; contour → scikit-image; viewshed → - xarray-spatial; XYZ/tiling → rio-tiler+morecantile; grid aggregation → h3/quadbin). - Link to [RasterX Function Reference](./raster-functions) for runnable per-function code. -5. **Honest gaps (roadmap-forward, three buckets):** - - *Distributed in pyrx today* — the matrix above; every `rst_*` runs in both tiers. - - *Heavyweight-only for now (parity tracked)* — OGR vector readers (`*_ogr`), - `conforming` TIN mode (pyrx raises on it; `constrained` works in both), advanced - PMTiles DataSource writer options (the `gbx_pmtiles_agg` aggregate is in both tiers), - and SQL default-argument convenience. Link [Execution Tiers](./execution-tiers). - - *Known behavior divergences* — `rst_color_relief` (GDAL DEMProcessing vs NumPy - `np.interp`; no `default` keyword in pyrx), `rst_convolve`/`rst_derivedband` (edge: - GDAL halo vs NumPy `pad(mode='edge')`), `rst_resample` (NoData/edge boundary pixels), - `rst_contour` (`gdal.ContourGenerateEx` vs `skimage.find_contours`), `rst_viewshed` - (`gdal.ViewshedGenerate` vs `xrspatial.viewshed`). Link [Benchmarking](./benchmarking) - and [Performance](./performance). Also note rasterio's bundled GDAL has a narrower - driver set than the heavyweight custom build. -6. **How it's distributed** — brief: Arrow scalar UDFs (per-tile), grouped-aggregate Arrow - UDFs (merges), streaming UDTFs (fan-out); no driver-side `.rdd`/`_jvm`; DataSource V2 - readers/writers. Link [Performance](./performance). - -- [ ] **Step 2: Wire into the sidebar** - -In `docs/sidebars.js`, add `'api/rasterio-distributed'` to the RasterX category `items` -(the array currently holding `'api/h3-raster-tessellation'`): - -```js - { - type: 'category', - label: 'RasterX', - collapsed: true, - link: { type: 'doc', id: 'api/raster-functions' }, - items: [ - 'api/rasterio-distributed', - 'api/h3-raster-tessellation', - ], - }, -``` - -- [ ] **Step 3: Verify page compiles + links resolve (docs static build)** - -Dispatch a subagent (Docker): - -```bash -bash scripts/commands/gbx-docs-static-build.sh --skip-zip --log rio-dist-build.log -``` - -Expected: exit 0, and NO "Broken links" / "broken anchors" entries. The build is the -authoritative link check (double-hyphen anchors from `&` headings are correct — trust the -build, don't hand-fix). Confirm `api/rasterio-distributed` compiled and its links to -`./raster-functions`, `./execution-tiers`, `./benchmarking`, `./performance` resolve. - -- [ ] **Step 4: Commit** - -```bash -git add docs/docs/api/rasterio-distributed.mdx docs/sidebars.js -git commit -m "docs: add 'Rasterio, Distributed' RasterX positioning page - -RasterX's lightweight (pyrx) tier is distributed rasterio + a -best-of-breed Python raster stack: side-by-side rasterio-vs-pyrx -snippets (tested), a capability->function->library coverage matrix, -and an honest roadmap-forward gap list (heavyweight-only for now / -known behavior divergences). Wired into the RasterX sidebar. - -Co-authored-by: Isaac" -``` - ---- - -### Task 3: Cross-links + the terrain attribution fix - -Weave a one-sentence blurb + link into four high-traffic sites, and correct the two -`performance.mdx` lines that over-attribute terrain to xarray-spatial. Bundled into one -task/commit because each edit is a one-liner and they share a single verification (build). - -**Files:** -- Modify: `docs/src/pages/index.js` (RasterX `Feature` description, ~line 56) -- Modify: `docs/docs/intro.mdx` (tiers paragraph, line 11) -- Modify: `docs/docs/api/execution-tiers.mdx` (near the "Python-worker UDFs (rasterio + NumPy)" row, line 70) -- Modify: `docs/docs/api/raster-functions.mdx` (top, after the imports block, ~line 12+) -- Modify: `docs/docs/api/performance.mdx` (lines 56 and 388) - -**Interfaces:** -- Consumes: doc id `api/rasterio-distributed` from Task 2. - -- [ ] **Step 1: Homepage RasterX card** — append to the RasterX `Feature` `description` in `docs/src/pages/index.js`: - -Change the description string to end with: ` In large part, distributed rasterio + best-of-breed raster packages.` -(Keep the existing `link="/docs/api/raster-functions"`; the new page is reachable from the See-also added in Step 4. Do NOT restructure the `Feature` component to add a second link.) - -- [ ] **Step 2: intro.mdx** — in the tiers paragraph (line 11), after "...covers all of **RasterX**, all of **VectorX** ... and all of **GridX** ...", add a sentence before the existing "See [Choosing an Execution Tier]" link: - -`RasterX's lightweight tier is, in large part, [distributed rasterio](./api/rasterio-distributed) — the familiar rasterio/NumPy raster stack run as Arrow UDFs across the cluster.` - -- [ ] **Step 3: execution-tiers.mdx** — immediately after the Tradeoffs table (the block containing the "Python-worker UDFs (rasterio + NumPy)" and "rasterio's bundled build (narrower)" rows), add a sentence: - -`The lightweight raster tier is, in effect, [distributed rasterio](./rasterio-distributed): see how much of the rasterio/GDAL surface is already distributed — and what stays heavyweight-only for now.` - -- [ ] **Step 4: raster-functions.mdx** — add a short admonition/"See also" near the top (after the intro paragraph, before the first tabbed section): - -`:::tip Built on rasterio` newline `RasterX's lightweight tier is, in large part, **distributed rasterio** + a best-of-breed raster stack. See [Rasterio, Distributed](./rasterio-distributed) for the coverage matrix and honest gaps.` newline `:::` - -- [ ] **Step 5: performance.mdx terrain attribution fix (two lines)** - -Line 56 currently: `| **xarray-spatial** | Terrain analysis (slope, hillshade, aspect, tri, tpi, roughness, viewshed) |` -Change to attribute terrain to NumPy and scope xarray-spatial to viewshed only. Replace with two correct rows: -``` -| **NumPy** (Horn 3×3) | Terrain analysis (slope, aspect, hillshade, tri, tpi, roughness) | -| **xarray-spatial** | Viewshed | -``` -(Fold into the existing NumPy row if one already lists band math — implementer's judgment; the requirement is: terrain must be attributed to NumPy, and xarray-spatial scoped to viewshed only. Keep table columns consistent with the surrounding rows.) - -Line 388 currently: `` | `pyrx/core/terrain.py` | xarray-spatial | `rst_slope`, `rst_aspect`, `rst_hillshade`, `rst_tri`, `rst_tpi`, `rst_roughness`, `rst_color_relief`, `rst_viewshed` | `` -Change the backing-library column for `terrain.py` from `xarray-spatial` to `NumPy` (Horn 3×3). Note `rst_viewshed` actually lives in `analysis.py` (xarray-spatial); if this row lumps it under terrain.py, split it out or annotate — verify the module column against `pyrx/core/` before editing so the table is accurate. - -- [ ] **Step 6: Verify build clean (all links resolve, no regressions)** - -Dispatch a subagent (Docker): -```bash -bash scripts/commands/gbx-docs-static-build.sh --skip-zip --log rio-dist-links-build.log -``` -Expected: exit 0, no broken links/anchors. Confirm the four new `rasterio-distributed` -links resolve and the homepage/intro/execution-tiers/raster-functions pages still compile. - -- [ ] **Step 7: Commit** - -```bash -git add docs/src/pages/index.js docs/docs/intro.mdx docs/docs/api/execution-tiers.mdx \ - docs/docs/api/raster-functions.mdx docs/docs/api/performance.mdx -git commit -m "docs: link 'Rasterio, Distributed' from key pages; fix terrain attribution - -Blurb+link into the homepage RasterX card, intro, execution-tiers, and -the RasterX function reference. Correct performance.mdx: terrain -(slope/aspect/hillshade/tri/tpi/roughness) is pure NumPy (Horn 3x3), not -xarray-spatial; only viewshed uses xarray-spatial. - -Co-authored-by: Isaac" -``` - ---- - -### Task 4: Final verification (browser + full-suite sanity) - -**Files:** none (verification only). - -- [ ] **Step 1: Browser screenshot pass (non-3000 port)** - -Dispatch the web-devloop-tester subagent: start the docs dev server on `--port 3001` -(`bash scripts/commands/gbx-docs-dev.sh` defaults to 3000 — override to 3001, or -`cd docs && npm run start -- --port 3001`). Screenshot: -- `http://localhost:3001/geobrix/docs/api/rasterio-distributed` — hook, a side-by-side pair, the coverage matrix, and the gaps section render correctly; tier tabs (if any) behave. -- The homepage RasterX card shows the new blurb. -Confirm no console errors. Stop the dev server when done. (Port 3000 is reserved for the user.) - -- [ ] **Step 2: Report status; hold for push** - -Summarize: page + tests + cross-links + attribution fix landed on `beta/0.4.0` as 3 -commits; doc-test green; static build clean; browser verified. Do NOT push. Suggest -`/review` before any push, and `gh auth switch --user mjohns-databricks` when the user -approves a push. - ---- - -## Self-Review - -**Spec coverage:** -- New page under Functions→RasterX → Task 2 ✓ -- Hook / side-by-sides / coverage matrix / three-bucket gaps / mechanism → Task 2 body ✓ -- Flagship warp/clip/NDVI, both sides tested with equality assertion → Task 1 ✓ -- Cross-links (homepage, intro, execution-tiers, raster-functions) → Task 3 ✓ -- Terrain attribution fix (performance.mdx) → Task 3 Step 5 ✓ (two lines, verified against code) -- Sidebar wiring standing rule → Task 2 Step 2 ✓ -- Verification: doc-test green, static build clean, browser (non-3000) → Tasks 1/2/3/4 ✓ -- No push, own commits on beta/0.4.0 → commits per task + Task 4 Step 2 ✓ - -**Placeholder scan:** Snippet/verifier/test code is concrete. The two explicit "confirm -against `functions.py` / `rasterx_functions.py`" notes are deliberate guardrails (real -call signatures must come from code, not be invented) — not deferred work; Task 1 Step 4 -iterates to green, which forces resolution. - -**Type consistency:** `functionName` constants (`REGISTER`, `WARP_RASTERIO`, `WARP_PYRX`, -`CLIP_RASTERIO`, `CLIP_PYRX`, `NDVI_RASTERIO`, `NDVI_PYRX`) match between Task 1 (defined) -and Task 2 (consumed). Verifier fns `warp_both` / `ndvi_both` (+ optional `clip_both`) -match between the module and the test. Doc id `api/rasterio-distributed` consistent across -Tasks 2 and 3. diff --git a/docs/superpowers/specs/2026-05-28-rst-dtmfromgeoms-wireup-design.md b/docs/superpowers/specs/2026-05-28-rst-dtmfromgeoms-wireup-design.md deleted file mode 100644 index c69d0ee04..000000000 --- a/docs/superpowers/specs/2026-05-28-rst-dtmfromgeoms-wireup-design.md +++ /dev/null @@ -1,250 +0,0 @@ -# Design: Wire up and test `gbx_rst_dtmfromgeoms` - -**Date:** 2026-05-28 -**Status:** Approved (design); implementation pending -**Package:** RasterX (`com.databricks.labs.gbx.rasterx`) - -## Problem - -`gbx_rst_dtmfromgeoms` is ported from DBLabs Mosaic (`rst_dtmfromgeoms`). It builds a -Digital Terrain Model raster by interpolating elevation (Z) from Z-valued point -geometries and optional breakline geometries using a constrained Delaunay -triangulation (TIN) with barycentric Z-interpolation, bounded by the convex hull. - -The implementation exists (`RST_DTMFromGeoms.scala` + `InterpolateElevation.scala`) but -is **not production-wired**: - -- `rd.register(RST_DTMFromGeoms)` is commented out in `rasterx/functions.scala` (line ~112). -- Both files are excluded from scoverage (`pom.xml` lines 466, 508). -- `eval` uses the **wrong** `RST_ErrorHandler.safeEval` overload — the tile-array form - `safeEval(fn, rows: ArrayData, rasterType)`, passing `pointsArray` (geometries) as if - it were an array of raster tiles. On any error it would try to read point geometries as - tile structs. This is the `// TODO: this will need fixing` at ~line 109. -- `InterpolateElevation.pointGrid(origin, gridWidthX, gridWidthY, gridSizeX, gridSizeY)` is - called with cell-size and cell-count arguments in the wrong positions relative to the - `pointGrid(origin, xCells, yCells, xSize, ySize)` signature — a latent arg-order bug. -- `splitPointFinder` is accepted and parsed (`TriangulationSplitPointTypeEnum`) but never - passed to the triangulator — a dead parameter. -- There are no tests, no `registered_functions.txt` entry, and no `function-info.json` entry. - -**Coverage verdict (why we keep it, not remove it):** the gap is genuine. The closest -registered function, `gbx_rst_gridfrompoints(_agg)`, performs Inverse-Distance-Weighted -(IDW) interpolation — a non-local method with no breakline support and no convex-hull -bounding. TIN/Delaunay surface interpolation with breakline constraints is a distinct, -standard terrain-modeling capability that nothing else in RasterX provides. - -## Goals - -1. Make `gbx_rst_dtmfromgeoms` a registered, working, tested RasterX function. -2. Ship a streaming aggregator counterpart `gbx_rst_dtmfromgeoms_agg`, mirroring the - `rst_gridfrompoints` / `rst_gridfrompoints_agg` pairing. -3. Modernize the public signature to the RasterX house style (consistent with - `rst_gridfrompoints` / `rst_rasterize`), so it composes pixel-for-pixel with the other - vector→raster functions. -4. Both functions pass the `binding-parity` QC check (Scala name literal + Python binding + - function-info entry present for each). - -## Non-goals (YAGNI) - -- No resolution-argument variant and no `grid_mode` discriminator — the documented recipe - covers resolution-based usage (see API docs below). -- `splitPointFinder` is **not** reinstated. -- No changes to other RasterX functions. -- The aggregator streams **points only**; breaklines are a per-group constant array param - (not streamed). Rationale below. - -## Design decisions (and rationale) - -- **Modernize the signature** rather than preserve Mosaic's exactly. RasterX is a - *successor* to Mosaic raster (only GridX/BNG are mandated to preserve baseline behavior), - and the project is pre-1.0 beta that breaks APIs to stabilize (CLAUDE.md). The function - was never registered here, so there are no existing call sites to migrate. -- **Scheme A — bbox + pixel-count** for the grid spec (`xmin, ymin, xmax, ymax, width_px, - height_px, srid`), matching `rst_gridfrompoints` and `rst_rasterize`. This maximizes - cross-function consistency and gives free pixel-aligned composability (produce IDW and TIN - over an identical grid and overlay/diff them). It also avoids the float-resolution rounding - ambiguity of a resolution-first form. The resolution ergonomic ("I want 10 m cells") is - recovered via documentation (a one-line conversion), not a second API surface. -- **Provide a streaming aggregator (`_agg`).** Elevation/survey/LiDAR point data lives as one - row per point. The non-agg form needs all points pre-collected into an `ARRAY` column in a - single row; the aggregator instead accumulates points directly in a `TypedImperativeAggregate` - buffer with Spark partial aggregation (map-side `update` + `merge`), avoiding the giant - `collect_list` array-column materialization. The triangulation itself still holds all points - in memory at finalization, so the win is the delivery/collection path and `GROUP BY` - ergonomics, not the core algorithm footprint. -- **Aggregator streams points only; breaklines are a per-group constant array (Option 1).** A - UDAF aggregates one value per row; breaklines are inherently low-cardinality (a handful of - ridgelines/rivers per region) while points are high-cardinality (the thing worth streaming). - Passing breaklines as a group-stable constant array — evaluated against `InternalRow.empty` - in `eval()`, exactly as `RST_GridFromPointsAgg` handles `xmin`/`srid`/etc. — keeps the input - shape clean and the buffer small. The rejected alternative (a discriminator column so points - and lines both stream) forces users to `UNION` mixed geometry types with a boolean flag and - buys nothing given breakline cardinality. -- **Shared `execute` compute path.** Refactor the triangulate→interpolate→rasterize pipeline - into a pure `RST_DTMFromGeoms.execute(pointWkbs, breaklineWkbs, mergeTol, snapTol, xmin, ymin, - xmax, ymax, widthPx, heightPx, srid, noData): InternalRow`. The non-agg `eval` parses its - arrays and calls it; the aggregator's `eval()` reads the constant breaklines + params and - calls it with the buffer's accumulated points. Mirrors `RST_GridFromPoints.execute` shared by - both grid functions. - -## 1. Public API - -``` -gbx_rst_dtmfromgeoms( - points_geom ARRAY, -- Z-valued points (WKB or WKT) - breaklines_geom ARRAY, -- breakline LineStrings; pass empty array for none - merge_tolerance DOUBLE, -- Delaunay segment-merge tolerance - snap_tolerance DOUBLE, -- vertex-to-breakline snap tolerance - xmin DOUBLE, ymin DOUBLE, xmax DOUBLE, ymax DOUBLE, - width_px INT, height_px INT, - srid INT, - no_data DOUBLE -- optional, default -9999.0 -) -> tile -- single-band Float64 GTiff, width_px x height_px -``` - -- Output is a tile row `(index_id LONG, raster BINARY, metadata MAP)` holding - a single-band Float64 GTiff of exactly `width_px x height_px`. -- Builder accepts **11 args** (`no_data` defaulted to `-9999.0`) and **12 args** (explicit - `no_data`), mirroring `RST_GridFromPoints`' arg-count flexibility. -- **Resolution recipe (documented).** To get N-unit cells over a known extent: - `width_px = round((xmax - xmin) / N)`, `height_px = round((ymax - ymin) / N)`. - Example: a 1 km² extent in EPSG:27700 at 10 m cells ⇒ `width_px = height_px = 100`: - `gbx_rst_dtmfromgeoms(pts, lines, 0.0, 0.01, 530000, 180000, 531000, 181000, 100, 100, 27700)`. - This recipe MUST appear in the function description and the SQL doc example. - -### Aggregator form - -``` -gbx_rst_dtmfromgeoms_agg( - point_geom BINARY|STRING, -- AGGREGATED per row: one Z-valued point (WKB or WKT) - breaklines_geom ARRAY, -- per-group CONSTANT array of breakline LineStrings - merge_tolerance DOUBLE, - snap_tolerance DOUBLE, - xmin DOUBLE, ymin DOUBLE, xmax DOUBLE, ymax DOUBLE, - width_px INT, height_px INT, - srid INT, - no_data DOUBLE -- optional, default -9999.0 -) -> tile -- single-band Float64 GTiff, width_px × height_px -``` - -- `point_geom` is the only aggregated (per-row) input; every other argument is a per-group - constant (same value for all rows in the group; read once in `eval()`). -- Typical usage: `GROUP BY `, pass the per-row point column and per-group literal - extent/tolerance/breakline params. -- Produces the **same** DTM as the non-agg form over the same grid (verified by test). -- Builder accepts 11 args (`no_data` defaulted) and 12 args (explicit). - -## 2. Internals & bug fixes - -- **Error handling (TODO fix):** use `RST_ErrorHandler.safeEval(() => {...}, null, BinaryType, - conf)` — the no-raster-input overload used by `RST_GridFromPoints`. Wrap with - `Option(...).map(_.asInstanceOf[InternalRow]).orNull` per the sibling pattern. -- **PySpark support:** provide both an `Int`-args and a `Long`-args `eval` entry point (PySpark - passes Python ints as `Long`), each delegating to a shared private `doInvoke`. Replace the - current single packed-tuple `eval`. -- **Grid generation:** refactor `InterpolateElevation.pointGrid` (or add a bbox variant) to take - `(xmin, ymin, xmax, ymax, width_px, height_px, srid)` and emit cell-center points at - `x = xmin + (i + 0.5) * x_res`, `y = ymin + (j + 0.5) * y_res`, where - `x_res = (xmax - xmin) / width_px`, `y_res = (ymax - ymin) / height_px`. This removes the - latent arg-order bug. -- **TIN core unchanged:** keep the working constrained-Delaunay + barycentric Z-interpolation - in `InterpolateElevation` (`triangulate`, `interpolate`, `postProcessTriangulation`). -- **Rasterization (chosen approach — direct pixel-fill):** write the interpolated cell-center - Z values directly into a row-major Float64 pixel grid, `no_data` for cells outside the - triangulated hull (or with NaN Z), then emit a GTiff with geotransform - `(xmin, x_res, 0, ymax, 0, -y_res)` and the given `srid`. This is exact (the TIN already - produced Z at each cell center) and avoids a second rasterization pass. `RST_DTMFromGeoms` - will **no longer call** `GDALRasterize.executeRasterize`; the shared `GDALRasterize` util - itself is untouched (other functions may use it). -- **Validation:** `require()` guards — `width_px > 0`, `height_px > 0`, `xmax > xmin`, - `ymax > ymin`, points array non-empty — with `rst_dtmfromgeoms:`-prefixed messages. -- **NaN interpolation:** today `interpolate` throws if any cell's Z is NaN. For a grid that - extends beyond the convex hull this is expected for some cells. Change: cells with no - containing triangle (or NaN Z) become `no_data` rather than throwing. -- **Shared `execute`:** extract a pure - `RST_DTMFromGeoms.execute(pointWkbs: Seq[Array[Byte]], breaklineWkbs: Seq[Array[Byte]], - mergeTol, snapTol, xmin, ymin, xmax, ymax, widthPx, heightPx, srid, noData): InternalRow` - containing triangulate → interpolate → direct-fill rasterize. The non-agg `eval` and the - aggregator both call it. WKB/WKT decoding of input geometries happens before `execute` - (reusing the `geomsFromArrayData` WKB/WKT pattern from `RST_GridFromPoints`). -- **Aggregator (`RST_DTMFromGeomsAgg`):** a `TypedImperativeAggregate[DTMFromGeomsAcc]` mirroring - `RST_GridFromPointsAgg`: - - Buffer `DTMFromGeomsAcc` accumulates point WKB byte arrays only; `serialize`/`deserialize` - for partial aggregation across partitions; `merge` concatenates buffers. - - `update(buffer, row)`: evaluate `point_geom`, normalize WKT→WKB, append (skip nulls). - - `eval(buffer)`: evaluate the per-group constants (breaklines array, tolerances, bbox, - width_px, height_px, srid, no_data) against `InternalRow.empty` via Int/Long-tolerant - readers (mirror `evalDouble`/`evalInt`), decode the breakline array to WKBs, then call the - shared `RST_DTMFromGeoms.execute(...)` with `buffer.points`. - - `dataType` = the same tile `StructType` as the non-agg output. - - Companion overrides `name = "gbx_rst_dtmfromgeoms_agg"` and a `builder()` accepting 11/12 - args (defaulting `no_data`). - -## 3. Registration & metadata - -- Uncomment `rd.register(RST_DTMFromGeoms)` **and add** `rd.register(RST_DTMFromGeomsAgg)` in - `rasterx/functions.scala` (the `_agg` registration goes with the other aggregators). -- Remove the two scoverage `excludedFiles` entries (`pom.xml` lines 466, 508) covering - `RST_DTMFromGeoms.scala` and `InterpolateElevation.scala`. (`RST_DTMFromGeomsAgg` is a new - file, not excluded.) -- Add **both** `gbx_rst_dtmfromgeoms` and `gbx_rst_dtmfromgeoms_agg` to - `docs/tests-function-info/registered_functions.txt`. -- Add a `*_sql_example()` for **each** in `docs/tests/python/api/rasterx_functions_sql.py`, then - regenerate `function-info.json` via `gbx:docs:function-info`. No hand-edited `ExpressionInfo` - — usage/example flow from the doc-test single-source pipeline (matching `RST_GridFromPoints`, - which overrides only `name` and `builder`). - -## 4. Testing - -- **Scala unit test** (`src/test/scala/.../rasterx/`): construct Z-valued points sampling a - **known tilted plane** `z = a*x + b*y + c`. Because linear (barycentric) TIN interpolation of - a planar surface is exact, assert interpolated pixel values equal the plane within a small - tolerance. Assert out-of-hull cells equal `no_data`. Assert output is a valid single-band - Float64 GTiff of the requested dimensions. Include one case **with a breakline** to prove - constraints are honored. Mix in `SilenceProjError` if non-EPSG warnings appear; release GDAL - datasets in `try/finally`. -- **Scala aggregator test** (`src/test/scala/.../rasterx/`): feed the **same** known-plane - Z-valued points as a one-row-per-point DataFrame, `groupBy` a constant extent key, call - `gbx_rst_dtmfromgeoms_agg` with the breaklines as a literal array + the extent params, and - assert the resulting raster is **byte-for-byte (or pixel-for-pixel within tolerance) - equivalent** to the non-agg `gbx_rst_dtmfromgeoms` over the identical grid. Include the - breakline case. This is the key correctness guarantee: agg ≡ non-agg. -- **Python binding tests** (`python/geobrix/test/rasterx/`): `rst_dtmfromgeoms` and - `rst_dtmfromgeoms_agg` wrappers calling their respective `call_function(...)` with inline - points; assert a tile row is returned and the raster opens. The agg test uses a row-per-point - DataFrame + `groupBy`. -- **SQL doc tests** (`docs/tests/.../sql`): inline-constructed Z-valued points (deterministic, - real code, not mocked) for **both** functions; double as the `function-info` examples. The - `_agg` example demonstrates the `GROUP BY` row-per-point workflow. -- **binding-parity:** `bash scripts/commands/gbx-test-bindings.sh` passes with **both** - `gbx_rst_dtmfromgeoms` and `gbx_rst_dtmfromgeoms_agg` present in Scala (name literals), Python - (`functions.py`), and `function-info.json`. - -Test inputs are inline-constructed Z-valued geometries (deterministic), not sample-data files — -appropriate because the assertions need a known surface with a predictable interpolation result. - -## 5. Affected files - -| File | Change | -|---|---| -| `src/main/scala/.../rasterx/expressions/RST_DTMFromGeoms.scala` | Rework signature (bbox+pixels), Int+Long eval, safeEval fix, validation, drop splitPointFinder, extract shared `execute`, header comment | -| `src/main/scala/.../rasterx/expressions/RST_DTMFromGeomsAgg.scala` | **New** — `TypedImperativeAggregate` aggregator + `DTMFromGeomsAcc` buffer; delegates to `RST_DTMFromGeoms.execute` | -| `src/main/scala/.../rasterx/operations/InterpolateElevation.scala` | bbox-based `pointGrid`; out-of-hull/NaN → no_data instead of throw; header comment | -| `src/main/scala/.../rasterx/functions.scala` | Uncomment `rd.register(RST_DTMFromGeoms)`; add `rd.register(RST_DTMFromGeomsAgg)` | -| `pom.xml` | Remove 2 scoverage `excludedFiles` entries | -| `docs/tests-function-info/registered_functions.txt` | Add `gbx_rst_dtmfromgeoms` and `gbx_rst_dtmfromgeoms_agg` | -| `docs/tests/python/api/rasterx_functions_sql.py` | Add a `*_sql_example()` for each function | -| `src/main/resources/.../function-info.json` | Regenerated | -| `python/geobrix/src/databricks/labs/gbx/rasterx/functions.py` | Add `rst_dtmfromgeoms` and `rst_dtmfromgeoms_agg` wrappers | -| `src/test/scala/.../rasterx/` | New Scala tests (non-agg known-plane + breakline; agg ≡ non-agg) | -| `python/geobrix/test/rasterx/` | New Python binding tests (both functions) | -| `docs/tests/.../sql` | New SQL doc tests (both functions) | - -## Verification - -- `gbx:test:scala --suite '*RST_DTMFromGeoms*'` (or the rasterx suite) green — includes the - agg-equals-non-agg assertion. -- `gbx:test:python --path python/geobrix/test/rasterx/` green for both new tests. -- `gbx:test:bindings` green (parity for both functions). -- `gbx:test:function-info` green (every registered function has a non-empty example). -- Doc tests for the new SQL examples green (in Docker). diff --git a/docs/superpowers/specs/2026-05-30-execution-tiers-docs-design.md b/docs/superpowers/specs/2026-05-30-execution-tiers-docs-design.md deleted file mode 100644 index a19f80498..000000000 --- a/docs/superpowers/specs/2026-05-30-execution-tiers-docs-design.md +++ /dev/null @@ -1,125 +0,0 @@ -# Execution Tiers Docs — Design Spec - -**Date:** 2026-05-30 · **Branch:** `pyrx-0.4.0` · **Status:** approved design, pre-implementation. - -## Goal - -Restructure the GeoBrix documentation so the **lightweight (pyrx)** and **heavyweight (rasterx)** raster paths read as two **execution tiers of one API** — unified where they don't differ, clearly delineated where they do, with the lightweight tier elevated and discoverable. A reader should come away understanding the tradeoffs yet thinking of the two as *mostly compatible / swappable*. - -## Terminology (load-bearing) - -- Use **"Execution Tiers"** — the two tiers are **Lightweight** and **Heavyweight**. -- **Do NOT** use "runtime" / "runtimes" anywhere in this feature — it collides with **Databricks Runtime (DBR)** and confuses readers. The toggle, comparison page, badges, and identifiers all use *tier* language (`groupId="gbx-tier"`, URL `?tier=lightweight`). - -## Framing: everything here is WIP for 0.4.0 - -Nothing in the 0.4.0 docs is treated as "shipped." Consequences: -- **Do not document interim within-0.4.0-cycle changes.** Release notes, change logs, and similar describe only the **net 0.4.0 end-state vs 0.3.0** — e.g. "0.4.0 adds a lightweight execution tier (pyrx)" — never the step-by-step docs restructuring done this cycle. -- The existing `rasterx-functions.mdx` is **not** sacred shipped content; restructure freely. Client-side redirects for moved pages are a **nice-to-have** (avoid 404s for anyone holding 0.3.0 links), not a hard requirement. - -## Chosen approach - -Hybrid (Option C) with **full migration now**: build the cross-cutting framing layer *and* migrate the existing raster reference into the new structure in this project. - -## 1. Information architecture (sidebar) - -``` -Getting started - └─ Choosing an Execution Tier ← NEW: comparison / elevation page -Function Reference - ├─ Raster Functions ← NEW unified page: functions BOTH tiers provide (~52 today) - ├─ Raster Functions — Heavyweight only ← NEW: rasterx-only functions (~55 today) - ├─ GridX Functions (heavyweight; gains a Heavyweight tier badge) - ├─ VectorX Functions (heavyweight; gains a Heavyweight tier badge) - └─ PMTiles Functions (heavyweight; gains a Heavyweight tier badge) -``` - -`rasterx-functions.mdx` and `pyrx-functions.mdx` both retire: shared functions → **Raster Functions**; rasterx-only functions → **Heavyweight only**; pyrx overview/tradeoffs prose → **Choosing an Execution Tier**. - -## 2. The Execution Tier toggle (the "feels swappable" mechanism) - -Native Docusaurus synced tabs — **no custom React**: -```mdx - - … - … - -``` -Choosing a tier once flips every synced tab site-wide, persists in `localStorage`, and rides in the URL (`?tier=lightweight`) for shareable links. **Used only where the paths truly diverge:** installation, `register`/setup, the import line, and the few functions with per-tier behavior. Everything else (function purpose, parameters, tile struct) is authored once, outside tabs. - -## 3. "Choosing an Execution Tier" page (elevation) - -- What each tier is; the **one-line swap** — change only the import, keep the **same alias `rx`**, and all downstream code is byte-identical: - ```python - from databricks.labs.gbx.rasterx import functions as rx # Heavyweight tier - from databricks.labs.gbx.pyrx import functions as rx # Lightweight tier — SAME alias `rx` - # everything below is unchanged across tiers: - df.select(rx.rst_slope("tile", unit="degrees")) - ``` - Docs consistently alias **both** tiers as `rx` (never `prx`) — the identical alias is what makes the swap a single-line change. Same `rst_*` names; same `gbx_rst_*` SQL after explicit `register(spark)`. -- A **tradeoffs table**: install (init script + JAR vs `pip install geobrix[pyrx]`), native GDAL (system/PPA vs rasterio-bundled), ARM / serverless / shared-cluster / Lakeflow-SDP support, execution model & performance (JVM-native vs Python-worker UDFs), driver coverage, SQL default-argument behavior, function coverage, **and readers/writers (tier-specific format names — see §8, not a transparent swap).** -- "How to choose" guidance. -- **Elevated via BOTH** the navbar (a top-level link) **and** the docs homepage/landing (a card or callout), so the lightweight tier is discoverable, not buried. - -## 4. Unified "Raster Functions" page (both tiers) - -- Covers the functions available in **both** tiers (the pyrx-implemented set; grows as pyrx grows). -- Each entry: one shared signature + description; a **tier badge**; per-tier caveats as a short note (e.g. "Lightweight: NumPy reimplementation, not bit-identical to gdaldem"; "SQL: pass all arguments explicitly — no Python defaults"). -- Where a runnable example diverges, it's an Execution Tier **tab** pulling the heavyweight snippet from `docs/tests/python/api/rasterx_functions.py` and the lightweight snippet from `docs/tests/python/api/pyrx_functions.py` — preserving the "tests ARE the docs" single-sourcing. -- Top of page: a compact **availability matrix** (functions × Lightweight/Heavyweight, grouped by category, with footnoted caveats). - -## 5. "Raster Functions — Heavyweight only" page - -- The rasterx functions not (yet) in pyrx (~55: aggregators, H3/quadbin grid, tiling/generators, contour/proximity/viewshed, asformat/cog_convert/buildoverviews, etc.). -- Clearly labeled as **Heavyweight only**. As pyrx implements one, it graduates from this page into the unified Raster Functions page (a documented maintenance step). -- Heavyweight-specific concept prose (e.g. VRT Python pixel functions, the tile-payload invariant) lives here (or in a shared concepts area where genuinely shared). - -## 6. Tier badges - -A small reusable MDX/React component (e.g. ``, ``, ``) rendering compact colored pills, with minimal CSS. Applied to: -- function entries on the Raster Functions and Heavyweight-only pages, -- the availability matrix, -- the GridX / VectorX / PMTiles pages (a **Heavyweight** badge near the top so their tier is unambiguous). - -## 7. Release notes (0.4.0 end-state only) - -Add/adjust a single `docs/docs/beta-release-notes.mdx` entry framed as 0.4.0 vs 0.3.0: **"0.4.0 introduces a lightweight execution tier (pyrx) — the raster API on pure Python + rasterio, no JAR/native GDAL, for serverless/shared/ARM/SDP."** No entries about the interim docs restructuring. - -## 8. Readers & Writers (tier-specific, NOT a transparent swap) - -Unlike the function API (same `rst_*` names across tiers), readers and writers are intentionally **kept separate by name** — the GeoBrix wheel is installed and used *both* with and without the JAR, so the `format(...)` string must make the tier unambiguous: - -- **Heavyweight:** Scala DataSourceV2 readers/writers requiring the JAR — `spark.read.format("gtiff_gdal")`, `format("gdal")`, the OGR vector readers, the `gdal` writer, etc. -- **Lightweight:** `*_pyrx`-suffixed readers/writers built on the **PySpark Python Data Source API** (https://spark.apache.org/docs/latest/api/python/tutorial/sql/python_data_source.html) — e.g. `spark.read.format("gtiff_pyrx")` — which work with **no JAR and no init script**. - -Docs treatment: the Readers/Writers pages use the same Execution Tier **tabs**, but here the tab *content genuinely differs* (the format name itself changes per tier) — this is a clarity choice, not a swap. The "Choosing an Execution Tier" tradeoffs table notes that readers/writers are selected per tier by format name (`*_pyrx` vs `*_gdal`/OGR). - -**Current state vs direction (be honest in the docs):** the lightweight tier today ingests rasters via Spark's built-in `binaryFile` reader + `rx.rst_fromcontent(content, driver)`; the dedicated `*_pyrx` Python-Data-Source readers/writers are a **separate, forthcoming implementation** (not part of this docs project). The docs present the `binaryFile` + `rst_fromcontent` path as today's lightweight ingest and describe `*_pyrx` as the direction — without claiming `gtiff_pyrx` exists before it does. - -## Mechanics, constraints & risks - -- **Native tabs + a small badge component + (optional) `@docusaurus/plugin-client-redirects`** — no heavy custom code; no global navbar mode-switch (synced tabs deliver the persistent global feel at far lower risk). -- **Hard constraint — doc-coverage QC:** the `doc-coverage` check asserts every one of the 154 registered functions stays documented on a page. The migration must land each raster function on either *Raster Functions* or *Heavyweight only* (none dropped); this is the migration's acceptance test. Run `gbx:test:*-docs` / the doc-coverage check after migration. -- **internals-leak QC + docs voice:** no "wave N", no internal process vocabulary anywhere under `docs/docs/`. -- **Re-alias lightweight examples:** existing pyrx docs/examples (and `docs/tests/python/api/pyrx_functions.py`) currently import `as prx`; migrate them to `as rx` so the swap message holds (downstream code identical across tiers). -- **Biggest effort/risk:** migrating the comprehensive raster reference (107 raster fns) without losing content or coverage, and reconciling the two pages' doc-test imports. The implementation plan stages it: scaffolding (tier tabs, badge component, comparison page) → unified shared page → heavyweight-only page → GridX/VectorX/PMTiles badges → Readers/Writers tier tabs (distinct `*_pyrx`/`*_gdal` format names per §8) → re-alias lightweight examples to `rx` → release-note entry → retire old pages (+ optional redirects) → verify doc-coverage + lint + internals-leak. - -## Out of scope - -- Implementing additional pyrx functions (separate track). -- A custom global navbar mode-switch (synced tabs suffice). -- Restructuring GridX/VectorX/PMTiles content beyond adding a Heavyweight tier badge. -- Versioned-docs or multi-instance-docs machinery. - -## Acceptance criteria - -1. A reader can pick an Execution Tier once and have it persist across pages (synced tabs). -2. The lightweight tier is reachable from both the navbar and the homepage. -3. Every shared raster function appears once on **Raster Functions** with a tier badge; every heavyweight-only function appears on **Heavyweight only**; GridX/VectorX/PMTiles show a Heavyweight badge. -4. `doc-coverage`, lint, and internals-leak checks pass; no "runtime" terminology and no "wave N" appear in the new/changed docs. -5. Release notes describe only the 0.4.0 end-state (lightweight tier added), not interim churn. -6. Both tiers are aliased `rx` in all examples (never `prx`); the swap is demonstrably a single import-line change. -7. Readers/Writers are shown per tier with distinct format names (`*_pyrx` via the PySpark Python Data Source API vs `*_gdal`/OGR), not as a transparent swap; lightweight ingest today is documented as `binaryFile` + `rst_fromcontent`. - ---- -*Approved via brainstorming on 2026-05-30. Next: writing-plans for the staged implementation.* diff --git a/docs/superpowers/specs/2026-06-05-benchmark-suite-design.md b/docs/superpowers/specs/2026-06-05-benchmark-suite-design.md deleted file mode 100644 index bcbf20968..000000000 --- a/docs/superpowers/specs/2026-06-05-benchmark-suite-design.md +++ /dev/null @@ -1,230 +0,0 @@ -# GeoBrix benchmark suite — design spec - -**Date:** 2026-06-05 · **Branch:** `pyrx-0.4.0` · **Status:** design approved, pre-implementation. -**Goal:** a benchmark suite that compares the **heavyweight** (Scala/JNI RasterX, `gbx_rst_*`) and **lightweight** (pyrx, rasterio-backed) APIs on compute time, at various scales, runnable (1) locally — heavyweight in the `geobrix-dev` Docker container, lightweight in an isolated `uv` venv — and (2) on a Databricks cluster for same-hardware comparison. The suite **generates its own data** at scale (compute time matters more than pretty output), keeping each tile valid (controlled nodata, plus variety in SRID, pixel values, resolution, extent, bands). - -Related prior work: [`2026-05-29-rasterio-lightweight-api.md`](2026-05-29-rasterio-lightweight-api.md) (pyrx design + the pure-Python core), [`2026-06-05-bundling-gdal-natives-assessment.md`](2026-06-05-bundling-gdal-natives-assessment.md) (why pyrx stays pure-Python). - ---- - -## 1. Scope & key decisions - -- **Both timing models, reported separately:** - - **pure-core** — the raster algorithm only, no Spark. pyrx: call `pyrx.core.(ds)` on a rasterio dataset. heavyweight: construct the `rst_*` Catalyst expression and call `eval` on a single deserialized tile `InternalRow` in-JVM (no Spark scheduler). Asymmetric mechanism, but **both read the identical generated tile bytes**, so the algorithm comparison is fair. - - **spark-path** — the `rst_*` Column on a DataFrame of tiles, materialized: heavyweight JVM Catalyst expression vs pyrx pandas/Arrow UDF. Captures serialization + framework overhead. -- **Function coverage:** **all `rst_*` functions present in both APIs** (~100), driven by a registry, with `--functions`/`--category` filters to run a slice. -- **Scale = two independent ladders, not a full matrix:** - - pure-core: tile dimensions `256² → 512² → 1024² → 2048² → 4096²` × bands `{1, 4, 13}`. - - spark-path: row count `10 → 100 → 1k → 10k` at a fixed mid tile size (default `1024²`; cluster pushes higher via `--rows`). -- **Lightweight isolation:** a dedicated `uv` venv (no system-site-packages), locked from the `pyrx` optional-deps in `pyproject.toml`. This becomes the canonical runtime for pyrx **benchmarks and other pyrx tests** (existing `gbx:test:pyrx` is routed through it). -- **Compute-time first:** machine-readable results (JSONL local, Parquet/Delta on cluster); a comparison table + short markdown summary, **no charts by default**. -- **Honesty invariants** (encoded in the data, not just docs): same corpus/args/iteration-counts across runners; `where ∈ {docker, venv, cluster}` so laptop Docker-vs-venv numbers are never mistaken for same-hardware truth; `na_by_design` rows emitted (not dropped) so coverage gaps are legible; runners executed **sequentially** so they don't contend for CPU and skew timings. - -## 2. Architecture (Approach A — two native runners over a shared spec) - -``` - ┌───────────────────────────────────────────────┐ - │ shared spec (language-neutral) │ - │ • function registry: name, category, │ - │ input_shape, args, modes(pure-core| │ - │ spark-path|both), applies_to │ - │ • corpus manifest (corpus.json): │ - │ every tile's path + properties + scale pt │ - └───────────────────────────────────────────────┘ - ▲ ▲ - reads same │ │ reads same - corpus + spec │ │ corpus + spec - ┌───────────────────┴────────┐ ┌────────────┴───────────────────┐ - │ Scala runner (geobrix-dev │ │ Python runner (uv venv) │ - │ Docker, JVM/JNI) │ │ │ - │ pure-core: expression.eval│ │ pure-core: pyrx.core.fn(ds) │ - │ spark-path: local Spark │ │ spark-path: local PySpark+UDF │ - └───────────────┬────────────┘ └───────────────┬─────────────────┘ - │ result rows (common schema) │ - └──────────────┬───────────────────────┘ - ▼ - orchestrator (gbx:bench:*) → merge → comparison.csv + summary.md - │ - └── cluster phase: same runners, notebook job, - results → bench_results Delta table -``` - -The corpus is generated **once** and read by both runners — the same-bytes invariant is the foundation of fairness. No cross-runner coupling beyond the shared spec + result schema. - -## 3. Data generator - -**Module:** `python/geobrix/src/databricks/labs/gbx/bench/datagen.py` · **Command:** `gbx:bench:gen-data` (runs in the venv, rasterio-based). - -Deterministic and **seeded** — re-running with the same seed + generator version produces identical bytes. Emits a `corpus.json` manifest listing every tile's path, properties, and which scale point/axis it belongs to. Runners read the manifest and never re-derive. - -Per-tile validity knobs: - -| Property | Behavior | -|---|---| -| **NoData** | `--nodata-frac` accepts the full range (≈0.0–0.9) and a **list to sweep** (e.g. `0.02,0.25,0.5`), so the corpus spans minimal-nodata and nodata-heavy tiles. Placement modes: thin border / sparse mask / clustered holes. Default minimizes nodata; high fractions are allowed on purpose. | -| **SRID** | Cycled across EPSG:4326, 3857, 32618 (UTM 18N), 27700 (BNG); each tile gets a correct affine for that CRS's units. `--srids` to override. | -| **Pixel values** | Seeded-RNG generative patterns: gradients, noise, sinusoids, and **band-correlated spectral-like** values so NDVI/indices produce non-degenerate output. Range matched to dtype. | -| **Dtype** | Swept `uint8`, `int16`, `float32` (`--dtypes`). | -| **Resolution** | Varied per CRS, affine kept consistent with extent + dimensions. | -| **Extent** | Realistic per-CRS placements; deliberate overlap included for `merge`/`combineavg` aggregators. | -| **Bands** | `1 / 4 / 13` (`--bands`). | - -**Scale wiring:** `--tile-px 256,512,1024,2048,4096`, `--bands 1,4,13`, `--rows 10,100,1000,10000`, `--seed`, `--out`. - -**Validity gate (post-gen):** assert every tile opens, has the declared CRS/bands/dtype, and nodata fraction is within a configurable warn-threshold (warn, not hard-fail, so intentional high-nodata tiles pass). Fail fast before benchmarking on bad data. - -**Locations:** corpus under `sample-data/Volumes/.../bench-corpus/` locally; pushed/generated to a UC Volume for cluster runs. - -## 4. The two runners - -Both speak the shared spec, run **N warmup + M measured iterations** per (function × scale point), and report `median/min/p90` to absorb JIT and cache effects. Each function's registry `modes` field marks it `pure-core` / `spark-path` / `both`; aggregators (`_agg`) and multi-output tiling generators are `spark-path` only, and a skipped mode produces a `na_by_design` row recording why. - -**Python runner** — `python/geobrix/src/databricks/labs/gbx/bench/runner.py`, executed inside the `uv` venv: -- **pure-core:** `_serde.open_tile(bytes)` → `pyrx.core..(ds, **args)`. No Spark. -- **spark-path:** local `SparkSession`, DataFrame of N tiles, apply the `rst_*` Column wrapper (pandas/Arrow UDF), force materialization, time end-to-end. - -**Scala runner** — `src/main/scala/com/databricks/labs/gbx/bench/Runner.scala` (a `main`, invoked via Maven/`spark-submit` inside `geobrix-dev`): -- **pure-core:** deserialize the manifest tile to the heavyweight tile `InternalRow`, construct the `rst_*` expression, call `eval` directly — algorithm + GDAL-JNI time, no Spark scheduler. -- **spark-path:** local single-node `SparkSession`, DataFrame of N tiles, registered `gbx_rst_*` expression, materialize, time it. - -Each runner writes a JSONL/Parquet shard; no cross-runner coupling. - -## 5. Results schema & metrics - -One flat, language-neutral schema (JSONL local, Parquet/Delta cluster) so local and cluster rows stack into the same table. - -| Field | Meaning | -|---|---| -| `run_id` | one id per orchestrator invocation | -| `api` | `heavyweight` \| `lightweight` | -| `fn`, `category` | e.g. `rst_slope`, `terrain` | -| `mode` | `pure-core` \| `spark-path` | -| `tile_px`, `bands`, `dtype`, `srid` | input scale/variety coordinates | -| `rows` | tiles processed (1 for pure-core; ladder value for spark-path) | -| `nodata_frac` | actual nodata fraction of the input | -| `warmup_iters`, `measured_iters` | iteration counts | -| `median_ms`, `min_ms`, `p90_ms` | timing distribution | -| `throughput_mpix_s`, `throughput_rows_s` | derived | -| `peak_rss_mb` | coarse peak memory (best-effort per-process sampling) | -| `status` | `ok` \| `na_by_design` \| `error` | -| `note` | reason for `na`/`error` | -| `output_fingerprint` | JSON: a cheap, deterministic summary of the function's output, captured **outside** the timed loop (pure-core mode only). Scalar fns → the value(s); tile-returning fns → per-band `{shape, dtype, nodata_count, min, max, mean, std}`. Enables heavy-vs-light consistency comparison. Empty for spark-path rows. | -| `env_arch`, `env_cpu_model`, `env_cpu_count`, `env_os`, `env_gbx_version`, `env_gdal_version`, `env_runtime_version`, `env_where` | environment; `where ∈ docker\|venv\|cluster` | - -**Metrics philosophy:** primary = `median_ms` + `throughput_mpix_s` (median over min: GDAL block-cache/JIT make min optimistic; p90 retained for tail/variance). Default output is a **comparison table** joining heavyweight vs lightweight on `(fn, mode, tile_px, bands, rows)` with a `speedup = hw_median / lw_median` column → `comparison.csv` + a short `summary.md` (slowest fns, biggest divergences, error rows). A `--plot` flag (later) can render PNGs; off by default. Cross-run/scale/arch/version comparison is a `groupBy`/filter over accumulated Parquet/Delta — no bespoke diff. - -## 5b. Output consistency (heavy vs light agreement) - -Beyond compute time, the suite captures whether the two APIs produce **consistent output** on the **same input tile** (the seeded corpus guarantees identical bytes in). Because heavyweight (GDAL) and lightweight (numpy/rasterio) run different algorithms, "consistency" is defined as **numeric agreement**, not byte-equality. - -- **Capture (in the runner, pure-core mode only):** one **untimed** call per (function × input tile) produces the actual output; an `output_fingerprint` is computed and stored in the result row. Scalar fns (`width`, `srid`, `avg`…) store the value(s); tile-returning fns (`slope`, `ndvi`, `transform`…) store per-band `{shape, dtype, nodata_count, min, max, mean, std}`. The fingerprint is taken outside the timed iterations so it never affects timing. Spark-path rows leave it empty (consistency is an algorithm-output property, cleanest to capture from the pure-core path). -- **Default comparison (in `compare`, Plan 1b):** join heavyweight vs lightweight rows on `(fn, tile_px, bands, dtype, srid, nodata_frac)` and classify each as **exact-match** (scalars; integer/count stats), **within-tolerance** (float stats within configurable rel/abs tolerance), or **divergent** — reporting the largest stat delta. Output: a `consistency.csv` + a section in `summary.md` (per-fn agreement class, worst deltas, any divergences). -- **Deep per-pixel opt-in (`--deep-parity `):** for named functions, both APIs' raw outputs are aligned and compared per-pixel (max-abs-error, RMSE, %-pixels-within-tol). Heavier (needs grid alignment when CRS/resolution differ, e.g. after `transform`); used as a targeted parity audit, not the broad sweep. -- **Tolerances are explicit and recorded**, so "consistent" is a defined, reproducible claim — not a vibe. Divergences are surfaced, not hidden (same honesty principle as `na_by_design`). - -## 6. Orchestrator & commands - -Two new command categories under `scripts/commands/` (`.md`+`.sh` pairs, source `common.sh`, support `--help`/`--log`, fail-fast `check_docker` where needed). - -**Venv foundation (shared):** -- `gbx:venv:sync` — create/refresh an isolated `uv` venv at a fixed gitignored path (`.venv-pyrx/`), installing the `pyrx` optional-deps locked; assert no-system-site-packages. -- `common.sh` helper `run_in_pyrx_venv ""`. -- **Existing `gbx:test:pyrx` (and pyrx paths in `gbx:test:python`) routed through this venv** — fixing the command, not working around it, so lightweight tests are host-isolated everywhere. - -**Bench commands:** - -| Command | Where | Does | -|---|---|---| -| `gbx:bench:gen-data` | venv | datagen → corpus + `corpus.json` → validity gate. `--tile-px --bands --rows --nodata-frac --srids --dtypes --seed --out` | -| `gbx:bench:heavyweight` | Docker | Scala runner over the corpus. `--functions --category --mode --iters --warmup --out --log` | -| `gbx:bench:lightweight` | venv | Python runner over the **same** corpus, same option surface | -| `gbx:bench:compare` | host | merge result shards → `comparison.csv` + `summary.md` | -| `gbx:bench:all` | orchestrates | one-shot local entry point (below) | -| `gbx:bench:cluster` | Databricks | cluster phase (§7) | - -**`gbx:bench:all` flow:** `gbx:venv:sync` (idempotent) → ensure corpus exists at the shared path (generate if missing or `--regen`; the **one** path passed to both runners) → run **heavyweight, then lightweight, sequentially** (CPU-contention honesty rule) → `gbx:bench:compare`. - -**Result layout (gitignored, under `test-logs/`):** `test-logs/bench//{heavyweight.parquet, lightweight.parquet, comparison.csv, summary.md}`. - -**Usage note:** the heavyweight Docker/Maven leg is minutes-long; when driven via Claude it's dispatched as a Task subagent with periodic progress updates, but the commands themselves are plain shell for any contributor or CI. - -## 7. Cluster phase - -Reuses the **same two runners, the same corpus, the same schema**; only packaging, submission, and a results table are new. This is where the comparison is honest in absolute terms (same hardware). - -**Single `--cluster-id` with isolation flags:** -- Default `gbx:bench:cluster --cluster-id ` runs **both** APIs on that cluster (true same-hardware comparison on an x86 heavyweight-configured cluster). -- `--heavyweight-only` / `--lightweight-only` to isolate — e.g. operator installs the pyrx wheel on an **ARM cluster** and runs `--lightweight-only`. -- Pre-flight **verifies the target API imports** on the cluster and fails fast otherwise (clear "heavyweight not available here" rather than a crash). - -**ARM asymmetry is a headline result, not a forced comparison.** Heavyweight deploys as a CI-built, sha256-verified bundle whose init script **refuses `aarch64`** (x86-only by design); lightweight runs on Serverless/Standard/Lakeflow DLT **and ARM**. So: -- x86 cluster → both APIs run. -- ARM cluster → lightweight only; heavyweight recorded as `na_by_design`, note `"heavyweight: x86-only (init script refuses aarch64)"`. ARM-vs-x86 compares lightweight-on-ARM vs lightweight-on-x86 (with heavyweight-on-x86 as reference). - -**Operator owns artifact provisioning; the benchmark runs against what's installed and records it.** No artifact pushing or `--artifact-source` flag. The operator sets up the cluster per the installation docs (release bundle + init script + wheel library on x86 heavyweight; or just the `[pyrx]` wheel on any/ARM cluster). The benchmark **detects and records** `gbx_version`, `gdal_version`, `arch`, `runtime_version` from the live session, so "what was tested" reflects a real, operator-validated deployment. - -**Submission:** reuse the existing one-off notebook-job mechanism (`push_and_run_bundle_on_cluster.py`, `notebooks/tests/databricks_cluster_config.env` for host/token/cluster, `GBX_BUNDLE_VOLUME_*`). The notebook ensures the corpus exists on the Volume, runs the matching runner(s) — spark-path is the headline at scale; pure-core still runs on the driver — and appends rows to a **`bench_results` Delta table** keyed by `run_id + scale coords + env`. - -**Corpus on cluster:** generated once to a UC Volume (tiny seed in, big corpus out; `numpy` pinned in venv + cluster for reproducibility), then both runners read that one path. - -**Bench code delivery:** the Python bench module ships inside the installed wheel (importable once the operator's wheel is present). The Scala runner's on-cluster delivery (in the JAR vs notebook-attached) is settled in the plan. - -**Guardrails:** launching a cluster job is shared-state and costs money — `gbx:bench:cluster` requires an explicit `--cluster-id`, never auto-provisions or attaches init scripts, and confirms target + corpus path before submitting. When driven via Claude, confirm with the user before any cluster submission. - -## 8. File layout (new) - -``` -python/geobrix/src/databricks/labs/gbx/bench/ - __init__.py - spec.py # function registry + corpus manifest model - datagen.py # seeded corpus generator + validity gate - runner.py # Python runner (pure-core + spark-path) - results.py # result-row schema, JSONL/Parquet IO, compare/summary -src/main/scala/com/databricks/labs/gbx/bench/ - Runner.scala # Scala runner (eval pure-core + local Spark spark-path) -scripts/commands/ - gbx-venv-sync.{md,sh} - gbx-bench-gen-data.{md,sh} - gbx-bench-heavyweight.{md,sh} - gbx-bench-lightweight.{md,sh} - gbx-bench-compare.{md,sh} - gbx-bench-all.{md,sh} - gbx-bench-cluster.{md,sh} -python/geobrix/test/bench/ # unit tests for datagen, spec, results, compare -``` - -## 9. Out of scope (YAGNI) - -- Charts/dashboards (machine-readable + comparison table only; `--plot` is a later add). -- Auto-provisioning clusters or owning the heavyweight deploy chain (sha256 sidecar, init-script staging) — operator's job per install docs. -- Benchmarking functions that exist in only one API (parity set only). -- Vector (pyvx) / grid (pygx) benchmarks — raster (RasterX/pyrx) only for v1. -- Micro-profiling (flame graphs, per-line); this measures end-to-end op time. - -## 10. Open implementation details (for the plan) - -1. Scala runner on-cluster delivery: include the bench `Runner` in the assembly JAR vs ship as a notebook-attached jar (avoid bloating the production JAR — possibly a separate bench classifier artifact or a `src/test` main submitted explicitly). -2. Exact registry seed: enumerate the ~100 parity functions from `docs/tests-function-info/registered_functions.txt` with their `input_shape`/`args`/`modes` — generated semi-automatically, hand-verified. -3. `peak_rss_mb` sampling mechanism per runtime (psutil in venv; JVM `OperatingSystemMXBean`/RSS read in Scala). -4. Default warmup/measured iteration counts per mode (pure-core can afford more iters; spark-path fewer). -5. Whether `gbx:bench:all` should also accept a `--quick` profile (small ladder) for CI smoke vs the full sweep. - -## 11. Refinements surfaced during Plan 1a execution (carry into 1b/2) - -Building the local suite (Plan 1a, committed on `pyrx-0.4.0`) surfaced several real refinements — capture here so 1b/2 incorporate them rather than rediscover them: - -1. **NoData → fingerprint accuracy (consistency-critical).** pyrx terrain (e.g. `slope`) computes over NoData **sentinel** pixels rather than masking them, so a terrain output fingerprint on a nodata-bearing tile summarizes "slope-of-sentinel" values. The cross-API consistency compare (1b) must either **pre-mask NoData before fingerprinting** or **assert both APIs share identical NoData semantics** — otherwise the agreement check compares garbage on nodata tiles. (Also informs whether the deep-parity per-pixel mode masks nodata.) -2. **`min_bands` on FnSpec → `na_by_design`, not `error`.** Band-math fns (`ndvi`/`ndwi`/`nbr`) on 1-band tiles error because band 2 doesn't exist. Add a `min_bands` field to `FnSpec`; the runner skips tiles below it and records `na_by_design` (with note), keeping the `error` column meaningful (true breakage only). Applies to both runners. -3. **`peak_rss_mb` attribution.** Current value is process-wide `ru_maxrss` (monotonic high-water mark) — it can't attribute memory to a specific fn/tile and only ever rises. Either drop it from per-row output or measure a per-call delta (`tracemalloc` peak around the timed call). Decide in 1b. -4. **spark-path placeholder fields.** spark-path rows carry `srid=0` / `nodata_frac=0.0` as sentinels (the row pool mixes srids, nodata 0.0). Document these as sentinels (comment or a dedicated marker) so a downstream reader never treats `srid=0` as a real projection. -5. **`na_by_design` emission for skipped modes.** `run_pure_core` currently `continue`s past fns lacking `pure-core` mode (silent). Once spark-path-only fns (aggregators) enter the registry in 1b, emit `na_by_design` rows instead so coverage gaps stay legible (design §1 honesty rule). -6. **Coverage gaps to exercise before 1b sign-off:** a `--bands 1` sweep (exercises the band-math na/error path) and at least one `int16`/`uint8` terrain tile (1a e2e was float32-heavy). -7. **Lint gate not yet run.** Implementers couldn't run `flake8` in the venv; **`gbx:lint:python` (Docker) must run before any push** — it's the CI gate (isort/black/flake8). Manual scans were clean. -8. **Local spark-path is heap-bounded** (confirmed: large tiles OOM the local driver at default heap). `gbx:bench:lightweight` exposes `--driver-mem` (default 4g); the full row ladder (1000/10000 rows at large tile sizes) is **cluster-scope** (Plan 2), not laptop. Keep local `--row-counts`/tile sizes modest. -9. **Compare semantics (for Plan 1c) — confirmed during 1b fingerprint build.** The heavy (Scala/GDAL) and light (numpy) fingerprints are semantically aligned (schema, `[h,w]` shape, population std `÷N`, nodata-filtered stats, null-on-empty), but three representational divergences mean the `compare` step MUST: (a) **exclude `dtype` from the agreement gate** — GDAL emits `Float32`/`Byte`/`UInt16`, numpy emits `float32`/`uint8`/`int16`; treat dtype as informational/display only; (b) compare numeric band stats and scalar `value` with a **float tolerance** (parse-and-compare as numbers), never string/exact equality — a scalar can serialize `256` (int) on one side vs `256.0` (float) on the other; (c) compare **parsed JSON**, not raw strings (Python `sort_keys=True` vs Jackson insertion order differ). Also: terrain/band-math on nodata-bearing tiles will legitimately diverge (light computes over sentinels, heavy/GDAL may mask) — surface as a real consistency finding with a "likely nodata-handling" note (ref §11.1). **Confirmed in the 1b e2e:** even on **nodata-free** tiles, neighborhood ops diverge in `nodata_count` — heavyweight `rst_slope` marks the 1-px slope-kernel border as nodata (1020 px on a 256² tile) while lightweight does not (0); min/max/mean/std still agree to ~3 decimals. So the compare must (i) treat `nodata_count` as an expected divergence for neighborhood/terrain ops (gate agreement on the value stats with tolerance, report nodata_count delta separately as informational), and (ii) never expect fingerprints to be byte-equal across APIs — always parse-and-tolerance-compare. **Confirmed in the 1c e2e:** the value-stat impact of the border-nodata difference is **tile-size-dependent** — `rst_slope` is `within_tol` at 512² but `divergent` at 256² (max_rel_delta ~0.006), because the border is a larger fraction of a small tile, shifting min/max/mean/std past the 1e-3 tolerance. This is a real, expected divergence (don't loosen tolerance to hide it); the compare now tags such divergent cells with a "divergence likely nodata/border-handling (nodata_count differs by N)" note so the cause is self-evident in `summary.md`. Headline perf finding from the same run: lightweight `rst_ndvi` is ~900× faster than heavyweight (heavy shells out to a `gdal_calc` subprocess). -10. **Pure-core timing asymmetry (heavy vs light).** The Scala `HeavyRunner` pure-core timed body runs `BenchDispatch.pureCore` which *includes* building the output fingerprint string each iteration, whereas the Python runner times the core op only (fingerprint is captured once, untimed). Minor for expensive ops (terrain/warp dominate), but for cheap accessors the heavy median carries a small fingerprint-serialization tax the light side doesn't. If strict pure-core timing parity matters, split `BenchDispatch.pureCore` into "run op" vs "fingerprint output" so the timed closure excludes fingerprinting (mirrors Python). Low priority; note when interpreting accessor pure-core deltas in 1c. -11. **Heavyweight spark-path not yet exercised end-to-end (Plan 1c must).** Plan 1b validated the heavyweight runner's **pure-core** path e2e (8/8 ok, schema-parity confirmed) but only compiled `runSparkPath` — no real-data run. Plan 1c's `gbx:bench:all` (or a dedicated smoke) must run `gbx:bench:heavyweight --modes spark-path` on a real corpus before trusting those rows. (Low residual risk: the gdal_calc rootPath NPE that hit pure-core can't recur on spark-path, which goes through the production spectral-eval path that mkdir's independently; untested surface is the cache/warm-up/noop-sink timing harness + per-fn row emission.) Note also: heavyweight band-math (`rst_ndvi`/etc.) pure-core is ~30–70× slower than terrain because it shells out to a `gdal_calc` subprocess — a real perf signal to highlight in the compare. -12. **Spark-path JVM/Spark warm-up skews the first timed job.** Observed in a demo run: `rst_avg` measured 589 ms @ 2 rows but 138 ms @ 4 rows — the *first* spark job in the process pays one-time JVM/Spark spin-up that the per-`(fn,rows)` warmup loop doesn't absorb (warmup is per-call, but the process-level JIT/Spark init only happens once, on whichever `(fn,rows)` runs first). Fix in 1b: run a **dedicated throwaway spark warm-up job** (one trivial materialized job on the tile DataFrame) before the timing loop in `run_spark_path`, so steady-state timings aren't contaminated by interpreter/JVM init. Both runners' spark-path legs need this (heavyweight JVM JIT warm-up is the same hazard, arguably larger). Until fixed, spark-path absolute numbers at the smallest row count are unreliable — note it in `summary.md` or discard the first sample. - ---- -*Design approved 2026-06-05 via brainstorming; Plan 1a implemented + reviewed the same day (commits on `pyrx-0.4.0`, all 15 tasks, 26 bench tests green). Next: Plan 1b (Scala heavyweight runner + cross-API consistency compare + `gbx:bench:all`), then Plan 2 (cluster). Sources: installation.mdx (current deploy model), recon of existing test/data/cluster harness, pyrx core/_serde/_udf, gbx command conventions.* diff --git a/docs/superpowers/specs/2026-06-06-pyrx-nodata-edge-fix-design.md b/docs/superpowers/specs/2026-06-06-pyrx-nodata-edge-fix-design.md deleted file mode 100644 index bdcf9c401..000000000 --- a/docs/superpowers/specs/2026-06-06-pyrx-nodata-edge-fix-design.md +++ /dev/null @@ -1,103 +0,0 @@ -# pyrx NoData / edge-handling fix — design spec - -**Date:** 2026-06-06 · **Branch:** `pyrx-0.4.0` · **Status:** design approved, pre-plan. -**Goal:** Fix lightweight (pyrx) raster functions that diverge from the heavyweight (Scala/GDAL) on **NoData and kernel-edge handling**, so a `rx.rst_* → prx.rst_*` swap gives consistent results. Heavyweight is the reference (unchanged). - -**Evidence:** The benchmark consistency sweep (full pure-core, 19 ds-in fns × a nodata-free 256² tile + a 25%-nodata 512² tile) + a code read of both implementations. See [[pyrx-nodata-edge-divergences]] memory and the design spec §11 of the benchmark suite. Confirmed divergences, with the heavy target semantics quoted from `NDVI.compute`/`gdal_calc`, `GDALBlock`, and `RST_DEMProcessingHelper`. - -## 1. Scope & decisions - -- **Families fixed now (confirmed divergent on both sides):** - - **Terrain (6):** `slope`, `aspect`, `hillshade`, `tri`, `tpi`, `roughness`. - - **Band-math (4):** `ndvi`, `ndwi`, `nbr`, `mapalgebra` (and the shared `index`/spectral helpers `savi`/`evi` ride along since they use the same read+emit path). -- **Deferred (fast-follow):** focal (`filter`, `convolve`) — heavy behavior confirmed (mask-aware *skip* + window-shrink, **no** border ring), but not in the bench registry, so there's no lightweight consistency evidence yet. The follow-up first adds `filter`/`convolve` to the bench registry, then applies a mask-aware-skip fix (distinct from terrain's border-ring rule). -- **Unchanged:** all reductions/accessors (avg/min/max/median/pixelcount/numbands/width/height — already mask via `read_masks`), summary, histogram, contour (explicitly masks), clip/threshold/init_nodata/band/setsrid (manage nodata correctly), warp (transform/to_webmercator — exact). The heavyweight is **not** touched. -- **Terrain edge convention (decided):** **match GDAL exactly** — the undefined 1-px kernel border becomes NoData (as `gdal.DEMProcessing` does by default, no `-compute_edges`). This is a user-visible change (existing pyrx terrain calls currently return computed edge values); accepted for swap-consistency. -- **Justification:** swap-consistency + correctness ([[pyrx-robustness-over-checkbox]]), not Mosaic parity ([[justify-by-utility-not-mosaic]]). Computing slope/NDVI over a `-9999` sentinel produces meaningless output — the input-NoData masking is a genuine correctness fix; the terrain border-ring is the one convention change. -- **Declared-nodata contingency:** heavy masks input NoData **only when the input band has a *declared* NoData value** (it's `gdal_calc.py`/GDAL default behavior, not GeoBrix code). pyrx mirrors this automatically by keying off `ds.read_masks` (all-valid when no nodata is declared). - -## 2. Architecture (Approach A — shared masked-read core + explicit per-family edge rule) - -One new focused helper module isolates the single new concept (mask-aware reads + neighborhood propagation); each family stays thin and applies its *own* edge rule (the heavy semantics genuinely differ per family). The already-consistent families are untouched. - -**New: `python/geobrix/src/databricks/labs/gbx/pyrx/core/_nodata.py`** - -```python -read_masked(ds, band=1) -> (data: np.ndarray[float64], valid: np.ndarray[bool]) - # data = ds.read(band).astype("float64") - # valid = ds.read_masks(band) != 0 - # When the band has no declared nodata, read_masks() is all-255 → valid all-True - # → no masking. This mirrors heavy's "mask only when nodata is declared". - -nodata_value(ds, default=-9999.0) -> float - # ds.nodata if set, else `default` (the sentinel to write into output). - -propagate_invalid(valid: np.ndarray[bool], size=3) -> invalid: np.ndarray[bool] - # invalid where ANY pixel in the size×size window is invalid OR out-of-array: - # invalid = ~scipy.ndimage.binary_erosion(valid, np.ones((size, size)), border_value=0) - # border_value=0 (out-of-bounds treated invalid) yields BOTH the input-nodata - # propagation AND the 1-px border ring in ONE call — matching gdal.DEMProcessing. - -emit(template_ds, result: np.ndarray, nodata: float, invalid: np.ndarray[bool], - dtype: str) -> bytes - # result = result.copy(); result[invalid | ~isfinite(result)] = nodata - # write a GTiff from template_ds's transform/crs with the given dtype + profile - # nodata=nodata. Generalizes the current _emit/_emit_float32. -``` - -Dependencies: `scipy.ndimage` (already a pyrx dep, used by focal/proximity) and `ds.read_masks` (already used by `accessors._valid_values`). No new third-party deps. - -**Unit boundaries:** `propagate_invalid` is pure array→array (testable against hand-built masks); `read_masked` depends only on a rasterio dataset; `emit` is a write round-trip. Each is independently testable and has one responsibility. - -## 3. Terrain family (`pyrx/core/terrain.py`) - -The math (Horn/Wilson formulas, `_horn_gradients`/`_neighbors`) and the function signatures/args are **unchanged**. Only the read and emit change: - -1. `data, valid = read_masked(ds)` (was `ds.read(1).astype("float64")`). -2. Compute the op on `data` exactly as today — keep the internal `np.pad(..., mode="edge")` used to compute interior-adjacent gradients; the padded edge values are discarded by the mask in step 4. -3. `invalid = propagate_invalid(valid)` — the entire behavioral fix: input-NoData propagation through the 3×3 window **and** the 1-px border ring, in one erosion. -4. `emit(ds, result, nodata, invalid, dtype)`. - -Per-op nodata/dtype: -- `slope`/`aspect`/`tri`/`tpi`/`roughness` — Float32, `nodata = -9999.0` (matches GDAL DEMProcessing's Float32 nodata). -- `hillshade` — uint8, `nodata = 0` (matches GDAL hillshade; was `nodata=None`). - -Result: a nodata-free tile → today's interior values + a NoData border ring; a nodata-bearing tile additionally masks sentinel neighborhoods — both matching heavy. - -## 4. Band-math family (`pyrx/core/indices.py`, `pyrx/core/mapalgebra.py`) - -Per-pixel (no neighborhood) → uses `read_masked` but **not** `propagate_invalid`. - -**`indices.py`** (ndvi/ndwi/nbr/savi/evi + the `index` dispatcher): -1. For each contributing band: `data_i, valid_i = read_masked(ds, idx_i)`. -2. Compute the formula on `data_i` as today (`_normalized_diff`, EVI/SAVI exprs, the numexpr `index` registry). -3. `invalid = (~valid_a) | (~valid_b) | …` (OR of contributing bands' invalid; `emit` additionally folds in non-finite results). -4. `emit(ds, result, nodata=-9999.0, invalid, dtype="float32")`. - -**`mapalgebra.py`** (`rst_mapalgebra`): read inputs via `read_masked`, OR their `~valid` into the invalid mask, run numexpr, and `emit` with a declared nodata — fixing **two** gaps the investigation found: (a) it computed over sentinels, and (b) it never set an output nodata at all. - -Signatures, the `index` named-formula registry, and numexpr usage are unchanged. The declared-nodata contingency is automatic via `read_masked`. - -## 5. Testing & validation - -- **New helper tests** (`pyrx/test/pyrx/test_core_nodata.py`): `propagate_invalid` (single interior invalid → its 3×3 neighborhood invalid; border ring always invalid; all-valid interior loses only the ring); `read_masked` (declared-nodata tile → False at sentinels; no-declared-nodata tile → all-True); `emit` round-trip (nodata written, reopens correctly). -- **Per-family behavioral tests** (extend `test_core_*`): terrain (slope) — declared-nodata tile + planted sentinel → output NoData on the border ring *and* the sentinel's 3×3 neighborhood, interior values away from edges/nodata unchanged (regression-pinned); no-declared-nodata tile → only the border ring NoData; hillshade nodata=0. Band-math (ndvi) — planted sentinel → NoData at exactly those pixels (no spread); no-declared-nodata → no masking; `(B+A)=0` → NoData. -- **Update existing goldens** that legitimately change: pyrx terrain/band-math assertions checking edge/nodata values → updated to the new contract (interior-value assertions stay). Doc-test goldens for terrain/band-math examples → regenerated via the Docker doc-test path (`gbx:test:python-docs`), per-package, narrowed to changed nodes. -- **Cross-API acceptance gate:** re-run the consistency sweep (`gbx:bench:gen-data` nodata-free + nodata-heavy tiles → `gbx:bench:heavyweight` + `gbx:bench:lightweight --mode pure-core` → `gbx:bench:compare`). The sweep validates the **9 bench-registered fns**: 6 terrain (slope/aspect/hillshade/tri/tpi/roughness) + 3 band-math (ndvi/ndwi/nbr). **Done = those 9 cells flip `divergent` → `within_tol`/`exact`** on both tiles. The other touched band-math fns (`savi`, `evi`, `index`, `mapalgebra`) share the same `indices.py`/`mapalgebra.py` read+emit path but are **not in the bench registry**, so they're validated by **unit tests only** (their fix is the identical mechanism). Local unit tests (same masking contract as heavy) are the fast TDD loop; the sweep is the authoritative cross-API confirmation for the registered subset. -- **Residual caveat (pre-agreed):** if a terrain cell stays `divergent` after masking is verified correct, that's an interior-algorithm difference (pyrx Horn vs GDAL Horn rounding) — a separate finding, not chased here. -- **Execution:** unit tests in the `uv` venv (`gbx:test:pyrx`); doc-tests + the bench sweep in Docker. - -## 6. Out of scope (YAGNI / deferred) -- Focal (`filter`/`convolve`) — fast-follow: add to bench registry, then mask-aware-skip fix (different edge rule: window-shrink, no border ring). -- Any heavyweight change (heavy is the reference; e.g. the band-math `gdal_calc` subprocess slowness is a separate perf item, not behavioral). -- proximity/viewshed/derivedband/clip/threshold — already consistent or out of the NoData-divergence set. -- A `compute_edges` flag (rejected — strict swap contract; heavy has no such flag). -- Masked-array (`numpy.ma`/rioxarray) refactor across the read path (rejected — YAGNI, risks the stable families). - -## 7. Risks -- **Interior-algorithm residual:** masking may leave terrain `within_tol` but not `exact` (or, worst case, still `divergent` if pyrx Horn diverges from GDAL Horn beyond the nodata cause). Surfaced by the sweep; handled as a separate finding. -- **Golden churn:** terrain/band-math doc examples + unit goldens change (intended). Bounded to those two families; reductions/warp goldens untouched. -- **hillshade nodata=0 collision:** if a legitimate hillshade value is 0, it would read as NoData. GDAL has the same property (hillshade nodata=0 is GDAL's own default), so this matches heavy — acceptable. - ---- -*Design approved 2026-06-06 via brainstorming. Next: implementation plan (writing-plans). Sources: the benchmark consistency sweep (`test-logs/bench/sweep/`), code reads of pyrx `terrain.py`/`indices.py`/`mapalgebra.py`/`accessors.py` and heavyweight `NDVI.compute`/`RST_MapAlgebra`/`GDALBlock`/`RST_DEMProcessingHelper`.* diff --git a/docs/superpowers/specs/2026-06-06-terrain-crs-scale-alignment-design.md b/docs/superpowers/specs/2026-06-06-terrain-crs-scale-alignment-design.md deleted file mode 100644 index 139f4dd73..000000000 --- a/docs/superpowers/specs/2026-06-06-terrain-crs-scale-alignment-design.md +++ /dev/null @@ -1,93 +0,0 @@ -# Terrain CRS-scale alignment (GDAL-normal) — design spec - -**Date:** 2026-06-06 · **Branch:** `pyrx-0.4.0` · **Status:** design approved (substance + 2 key decisions), pre-plan. - -**Goal:** Make the gradient-based terrain functions — `slope`, `aspect`, `hillshade` — produce **GDAL-3.11-normal** output in *both* engines (heavyweight Scala/gdaldem and lightweight pyrx/numpy), so that (a) a `rx.rst_* → prx.rst_*` swap is consistent and (b) both match standalone `gdaldem`. The heavyweight is **not** a frozen reference here — these functions are all new and may change to align to GDAL-normal (user direction, 2026-06-06). - -## 1. Background / root cause - -A consistency sweep + code reads (see [[pyrx-nodata-edge-divergences]]) found `rst_hillshade` diverges heavy-vs-light (~0.99) and `rst_aspect` diverges on the geographic tile (~1e-3, was mis-attributed to float rounding). The unifying root cause: - -- **GDAL 3.11+** (heavy native libgdal **3.11.4**; light rasterio bundles **3.12.1**): *"if none of -scale, -xscale and -yscale are specified, and the CRS is a geographic or projected CRS, gdaldem will automatically determine the appropriate ratio from the units of the CRS."* -- The corpus tile is **EPSG:4326**, res **0.0001°**. On the heavy side: - - `RST_Hillshade`/`RST_Aspect` omit `-s` → GDAL auto-scales (degree→metre) → correct relief. - - `RST_Slope` **forces `-s 1.0`** → suppresses auto-scale → saturates (~90° everywhere). This is a wart. -- pyrx `slope`/`aspect`/`hillshade` hand-roll Horn on **raw pixel-size gradients** with **no CRS awareness** → on a degree grid the gradients are ~10⁵× too large. -- **Why the sweep looked the way it did:** `slope` *falsely agreed* (both engines degree-naive/saturated — heavy `-s 1.0`, pyrx scale=1.0); `hillshade` diverged (heavy auto-scales, pyrx doesn't); `aspect` diverged anisotropically on the geographic tile only. **TRI/TPI/Roughness are difference-based (no division by pixel size) → scale-invariant → already consistent** (out of scope). - -## 2. Target behavior — "GDAL normal" (GDAL 3.11 auto-scale) - -**Exact GDAL 3.11.4 formula** (verbatim from `apps/gdaldem_lib.cpp` lines 3652–3710, the `GDALDEMProcessing` defaulting block triggered when `std::isnan(psOptions->xscale)`): - -```c -xscale = 1; yscale = 1; -double zunit = 1; // from band GetUnitType(): "m"→1, "ft"→0.3048, - // "us-ft"→US-foot-conv, ""→1 (assume metre), else warn+1 -if (poSrcSRS && poSrcSRS->IsGeographic()) { - const double dfAngUnits = poSrcSRS->GetAngularUnits(); // degrees → π/180 ≈ 0.0174532925 - yscale = dfAngUnits * poSrcSRS->GetSemiMajor() / zunit; // WGS84: π/180·6378137 = 111319.4908 - const double dfMeanLat = (adfGT[3] + nYSize * adfGT[5] / 2) * dfAngUnits; // centre lat, RADIANS - // (warns if |meanLat| > 80°) - xscale = yscale * cos(dfMeanLat); // anisotropic -} else if (poSrcSRS && poSrcSRS->IsProjected()) { - xscale = poSrcSRS->GetLinearUnits() / zunit; // metre → 1.0; foot → 0.3048 - yscale = xscale; // isotropic -} -// else (no/unknown CRS): xscale = yscale = 1.0 -``` - -GDAL **applies** the scale as `gradient = (weighted Horn sum) / (pixel_size · scale)` (see `inv_ewres_xscale = 1/(adfGeoTransform[1]·xscale)`, `inv_nsres_yscale = 1/(adfGeoTransform[5]·yscale)` in the alg-data setup). Since pyrx's `_horn_gradients` already divides by `pixel_size`, pyrx applies the scale as an **extra divisor**: `dzdx /= xscale`, `dzdy /= yscale`. - -**pyrx reimplementation (numpy/rasterio + pyproj):** -- `dfAngUnits`: degrees → `π/180` (virtually all geographic CRS; can read from pyproj if non-degree). -- `GetSemiMajor()`: `pyproj.CRS(ds.crs).ellipsoid.semi_major_metre` (EPSG:4326 → 6378137.0). -- `dfMeanLat` (radians) = `(ds.transform.f + ds.height * ds.transform.e / 2) * (π/180)` — `transform.f` = y-origin (deg), `transform.e` = y pixel size (negative). Then `xscale = yscale * cos(dfMeanLat)`. -- Projected: `GetLinearUnits()` = `pyproj.CRS(ds.crs).axis_info[0].unit_conversion_factor` (metre → 1.0). -- `zunit` from band unit (rasterio `ds.units[band-1]`); default 1.0 (metre) when unset. - -(Anisotropy — `xscale ≠ yscale` on geographic — is why **aspect** shifts: `dzdx`/`dzdy` are divided by different scales. This is the real cause of the aspect "1e-3 rounding" residual.) - -## 3. Decisions (approved) - -1. **pyrx scale API:** auto-scale by default (CRS-derived, GDAL-normal). Expose optional `xscale: float | None = None`, `yscale: float | None = None` overrides on `slope`/`aspect`/`hillshade`, mirroring gdaldem's `-xscale`/`-yscale` (anisotropic-capable). **Drop** slope's current `scale: float = 1.0` parameter/default (degree-naive). When both overrides are `None` → auto-derive; when given → use as-is. -2. **Heavy `RST_Slope`:** **drop scale from the default registration.** The 1-arg `rst_slope(tile)` and 2-arg `rst_slope(tile, unit)` emit **no** `-s` (GDAL auto-scales). Only the explicit 3-arg `rst_slope(tile, unit, scale)` emits `-s `. The Scala `builder()` already branches on arity; adjust so the 1/2-arg paths construct without a scale literal (or with a sentinel that the helper omits). - -## 4. Architecture - -### 4.1 Light (pyrx) — `python/.../pyrx/core/terrain.py` -- **New shared helper** `_gdaldem_scale(ds) -> (xscale, yscale)`: replicate §2. Reads `ds.crs` (rasterio CRS): `is_geographic` → anisotropic lat formula (centre lat from `ds.transform`/`ds.bounds`, semi-major from the CRS via pyproj/`ds.crs` — pin the WGS84 `a` and confirm projected-units path against GDAL source); `is_projected` → linear-units scale; else `(1.0, 1.0)`. -- `_horn_gradients(ds)` stays as-is (raw per-pixel-size gradients). Each of `slope`/`aspect`/`hillshade`: - - resolve `(xs, ys)` = explicit overrides if both given, else `_gdaldem_scale(ds)`. - - divide: `dzdx /= xs`, `dzdy /= ys` (GDAL applies scale as a divisor of the gradient), then run the existing slope/aspect/hillshade math on the scaled gradients. -- `slope`: remove `scale` param, add `xscale`/`yscale`. `aspect`/`hillshade`: add `xscale`/`yscale`. Signatures otherwise unchanged. NoData/edge handling (the `_nodata.py` `read_masked`/`propagate_invalid`/`emit` path) unchanged. -- `tri`/`tpi`/`roughness`: **untouched** (scale-invariant). - -### 4.2 Heavy (Scala) — `RST_Slope.scala` -- `execute(ds, unit, scaleOpt)`: only append `Seq("-s", scale.toString)` when scale is explicitly provided. `builder()`: arity 1/2 → construct slope without forcing a scale (omit `-s`); arity 3 → pass the user scale. `functions.scala` `rst_slope(tileExpr)` / `rst_slope(tile, unit)` convenience overloads must NOT inject `lit(1.0)` as a scale that emits `-s`. -- aspect/hillshade/tri/tpi/roughness Scala: **no change** (already omit `-s`). Verify by re-reading. - -### 4.3 Bench — `python/.../bench/spec.py` + `src/test/.../bench/BenchDispatch.scala` -- Remove the `scale: 1.0` passed to pyrx `slope` and the `-s 1.0`/scale=1.0 on heavy slope (both were masking the divergence). Both engines now use auto-scale on slope, matching aspect/hillshade. -- Keep a projected-CRS tile in the corpus so the sweep exercises both auto-scale branches (geographic + projected). If the corpus lacks a metre-CRS DEM tile, add one in datagen. - -## 5. Testing & validation - -- **Unit (venv, pyrx):** new `test_gdaldem_scale_*` — geographic 4326 tile → `(xscale≈yscale·cos(lat), yscale≈111319.49)`; projected metre tile → `(1.0, 1.0)`; no-CRS → `(1.0, 1.0)`. Per-fn: `slope`/`aspect`/`hillshade` on a geographic tile now divide gradients by the auto-scale (assert the scaled-gradient result, not the raw); explicit `xscale`/`yscale` override path; metre tile unchanged. Update existing slope/aspect/hillshade goldens that change (compute new expected, don't weaken). TRI/TPI/roughness tests unchanged. -- **Heavy (Docker):** Scala suite for `RST_Slope` — 1/2-arg omit `-s` (auto-scale), 3-arg emits `-s`. Re-read aspect/hillshade tests still green. -- **Cross-API acceptance sweep (authoritative):** re-run pure-core sweep. **Done =** `slope`, `aspect`, `hillshade` all flip to `within_tol`/`exact` on the **geographic** tile AND stay `within_tol` on the **projected** tile; tri/tpi/roughness/band-math stay `within_tol`; `nodata_count_delta=0` throughout. (Formula is pinned exactly in §2; the sweep against real gdaldem 3.11.4 is the final confirmation.) -- **Doc tests:** regenerate any terrain doc-example goldens (Docker, per-package, narrowed to changed nodes). - -## 6. Out of scope -- TRI/TPI/Roughness (scale-invariant). -- Focal (`filter`/`convolve`) NoData fix (separate fast-follow, already tracked). -- High-latitude accuracy beyond GDAL's own `cos(lat)` approximation (GDAL itself documents this limitation). -- Any non-terrain function. - -## 7. Risks -- **Exact GDAL formula:** `meanLat` definition + projected linear-units path must match GDAL 3.11.4 precisely or the sweep won't hit `within_tol`. Mitigation: extract from in-container GDAL source; bench is the oracle; iterate. -- **Breaking change:** default `slope`/`hillshade`/`aspect` output changes on geographic rasters in BOTH engines (they become auto-scaled/correct). Beta, no aliases — document in `docs/docs/beta-release-notes.mdx`. -- **pyrx CRS access:** rasterio `ds.crs` semi-major axis — confirm pyproj is available in the pyrx venv (it ships with rasterio's deps) or derive `a` from the CRS WKT; fall back to WGS84 `a` for EPSG:4326. -- **Cross-language naming/parity:** removing slope's `scale` arg touches the Scala signature, `functions.py` binding, `registered_functions.txt`, and `function-info.json` — run `gbx:test:bindings`. - ---- -*Design approved 2026-06-06. Next: implementation plan (writing-plans). Sources: GDAL 3.11 gdaldem docs (auto-scale note), `gdaldem_lib.cpp`, root-cause investigation (`test-logs/bench/round2/`), code reads of pyrx `terrain.py` + heavy `RST_Slope`/`RST_Hillshade`/`RST_Aspect`/`RST_DEMProcessingHelper`.* diff --git a/docs/superpowers/specs/2026-06-07-benchmark-full-coverage-design.md b/docs/superpowers/specs/2026-06-07-benchmark-full-coverage-design.md deleted file mode 100644 index a4f50cbdc..000000000 --- a/docs/superpowers/specs/2026-06-07-benchmark-full-coverage-design.md +++ /dev/null @@ -1,86 +0,0 @@ -# Benchmark full coverage + core/full sets + deprecation scorecard — design spec - -**Date:** 2026-06-07 · **Branch:** `beta/0.4.0` · **Status:** design approved (scope + core-set + capture-motivation), pre-plan. - -**Goal:** Grow the heavy-vs-light benchmark from a 19-function representative set to **full coverage of the 107 registered `rst_*` functions**, while preserving a fast **`core`** set for routine runs. Make the suite the **evidence instrument** for a possible heavyweight-API deprecation. - -## 1. Motivation (internal) - -A surprising benchmark finding: lightweight (pyrx) performance is **at least comparable to, and in several cases far better than**, heavyweight (rasterx) — e.g. band-math is hundreds of times faster pure-core, terrain/reductions a few times faster, with **0 divergent** output. Combined with the fact that **pyrx already implements all 107 registered `rst_*` functions** (full functional parity — `registered_functions.txt` rst_ = 107, pyrx rst_ = 107, **0 heavy-only**), this opens the door to **eventually deprecating the heavyweight tier entirely**. - -That makes the benchmark a decision instrument, not just a comparison. To responsibly retire heavyweight we must show, across **every** function, that lightweight (a) produces consistent output and (b) performs acceptably. Any unbenchmarked function is an unanswered "is light safe here?". So **completeness is the objective**, and the suite must surface coverage/parity/performance as a **scorecard**. - -> **Voice boundary:** the deprecation framing lives in this internal (gitignored) doc only. User-facing docs (`docs/docs/`) and the generated `summary.md` stay neutral — they report coverage/parity/performance numbers; they do **not** say "we plan to delete heavyweight." The data supports the decision without the loaded framing. - -## 2. Scope - -- **This round (Phase 1):** the core/full selection mechanism + **bucket E (standard ds-in, ~51 functions)** + the scorecard/coverage section in `summary.md`. -- **Committed follow-on phases (not optional — required to complete the deprecation evidence base):** - - Phase 2 — **C: multi-input / constructors / tiling / readers** (14): `frombands`, `fromcontent`, `fromfile`, `combineavg`, `merge`, `maketiles`, `retile`, `tooverlappingtiles`, `xyzpyramid`, `tryopen`, `separatebands`, `buildoverviews`, `getsubdataset`, `subdatasets`. Needs multi-tile corpus + adapters. - - Phase 3 — **B: DGGS grid / vector-out** (13): `contour`, `polygonize`, `h3_tessellate`, `h3_rastertogrid{avg,count,max,median,min}`, `quadbin_rastertogrid{avg,count,max,median,min}`. Needs new fingerprint kind + consistency semantics for cell/geometry output. - - Phase 4 — **A: aggregators** (7, `*_agg`) + **D: geometry-in** (3): `rasterize`, `dtmfromgeoms`, `gridfrompoints`. Needs a group-by spark-path harness and a geometry corpus. -- The 107 = 19 (current) + 88 (missing). Buckets: E 51 · C 14 · B 13 · A 7 · D 3 = 88. - -## 3. Core vs full selection - -- Add `core: bool = False` to `FnSpec`. Mark the **current 19** as `core=True` (kept as-is per decision — they already span accessor / reduction / terrain / band-math / warp). -- Extend `spec.select(functions=None, categories=None, set="core")` with a `set` parameter: - - `set="core"` (default) → only `core=True` specs. - - `set="full"` → entire `REGISTRY`. - - `functions=`/`categories=` still filter within the chosen set (explicit `functions` overrides the set filter, as today). -- Thread `--set core|full` through the `gbx:bench:*` commands (`gen-data` picks tiles for whatever functions are selected; `lightweight`/`heavyweight`/`all`/`cluster` pass it to `select`). Default `core`. `full` is the on-demand / periodic-job run. - -## 4. Bucket E — standard ds-in (Phase 1) - -All take one tile in. Three sub-shapes, each a known pattern: - -1. **Scalar / list accessors** → `scalar` or `scalar_list` fingerprint (full consistency). E.g. `band`, `srid`, `type`, `format`, `memsize`, `isempty`, `rotation`, `scalex/y`, `skewx/y`, `upperleftx/y`, `pixelwidth/height`, `tilexyz`, `worldtorastercoord(+x/y)`, `rastertoworldcoord(+x/y)` (the coord ones take scalar args). Heavy: `BenchFingerprint.ofScalar/ofArray(RST_X.execute(ds[, args]))`. -2. **Map / struct outputs** → **pure-core timing only** (`modes=("pure-core",)`, no consistency assertion): `metadata`, `bandmetadata`, `georeference`, `boundingbox`, `summary`, `histogram`, `getnodata`. Their outputs (maps/structs/variable-length) have no clean cross-engine fingerprint; forcing one would be noise. Timed, not compared. Documented as such in the scorecard. -3. **Tile-out transforms** → `raster` fingerprint (full consistency), same pattern as terrain/warp: `clip`, `threshold`, `initnodata`, `setsrid`, `updatetype`, `fillnodata`, `filter`, `convolve`, `proximity`, `viewshed`, `color_relief`, `resample`, `resample_to_res`, `resample_to_size`, `derivedband`, `mapalgebra`, `index`, `evi`, `savi`, `cog_convert`, `asformat`, `sample`. Heavy: `fpDerived(RST_X.execute(ds, args))`. - -**Per-function cost (mechanical, mirrors existing 19):** -- Python `FnSpec` in `spec.py`: `name`, `sql_name`, `category`, `modes`, `args`, `core_fn=lambda ds,a: `, `col_fn=lambda t,a: prx.rst_x(t,...)`, `core=False`. -- Scala `BenchDispatch.scala`: a `pure-core` case `case "rst_x" => (RST_X.execute(ds, ))` and a `column` case `case "rst_x" => rst_x(tile, )`. **Heavy signatures are read from each `RST_X.scala`** (don't guess; the `execute`/registration shows the arg order). -- Fingerprint kind chosen by output type (scalar/scalar_list/raster). Map/struct → pure-core-only, fingerprint omitted/`na`. - -`min_bands` set where a function needs ≥2 bands (e.g. band-math, `band` index 2). - -## 5. Scorecard / coverage section (Phase 1) - -Add to `summarize_compare` (combined `summary.md`) an aggregate **Coverage & parity** block, computed from the compared cells + the registry + the canonical 107 list: - -- **Coverage:** `benchmarked N / 107 registered rst_ functions` (and which `set` produced this run). -- **Parity:** of the consistency-comparable cells, counts of `exact` / `within_tol` / `divergent` (and the divergent function names). -- **Performance:** count where lightweight ≥ heavyweight (speedup ≥ 1) vs where heavyweight wins. -- **Functional parity gap:** registered rst_ functions with no pyrx implementation — **currently 0** (state it explicitly; it's a positive, and the check guards against future regressions). -- **Not yet covered:** the explicit list/count of registered functions absent from this run's registry (the A–D buckets in Phase 1) — **no silent omission**; the reader sees exactly what remains. - -Framed neutrally (coverage/parity/performance), no deprecation language. - -## 6. Architecture / files (Phase 1) - -- `python/.../bench/spec.py` — `FnSpec.core` field; `select(set=...)`; ~51 new `FnSpec` entries; a `REGISTERED_RST` loader (read `docs/tests-function-info/registered_functions.txt`) for coverage math. -- `src/test/scala/.../bench/BenchDispatch.scala` — ~51 new `pure-core` + `column` cases (signatures from each `RST_*.scala`). New fingerprint helpers only if a new output shape needs one (reuse `ofScalar`/`ofArray`/`fpDerived`). -- `python/.../bench/compare.py` — the Coverage & parity block in `summarize_compare`; coverage math (registry vs 107). -- `scripts/commands/gbx-bench-*.sh` — `--set core|full` option (default core), passed to `select`. -- `python/.../bench/spec.py` `dump_functions_json` — include the `core` flag + `set` membership. - -## 7. Testing & validation - -- **Unit (venv):** `test_spec.py` — `select(set="core")` returns only the 19; `select(set="full")` returns all registered; `functions=`/`categories=` still filter; new FnSpecs are well-formed (have core_fn/col_fn, valid modes, fingerprint-appropriate). `test_compare.py` — the scorecard block renders coverage/parity/performance counts and the not-yet-covered list; `exact` cells unannotated; parity-gap=0 line present. -- **Scala (Docker):** the new `BenchDispatch` cases compile and dispatch (extend the bench Scala test to exercise a sample of new functions pure-core + column). -- **Cross-API acceptance (authoritative):** `gbx:bench:all --set full --modes pure-core` runs clean; the scorecard shows coverage 70/107 (Phase 1 = 19 + 51) and the A–D not-yet-covered list. Spot-check that newly-covered tile-out transforms are `within_tol` (and investigate any `divergent` as a real light-vs-heavy finding, not loosened tolerance). -- **`core` run unchanged:** `gbx:bench:all` (default core) still produces the 19-function comparison, fast. - -## 8. Out of scope (this round) -- Buckets A–D (committed to Phases 2–4). -- Any change to the consistency tolerance or `_close` semantics. -- Removing/altering heavyweight functions (deprecation is a future decision this evidence base informs — not an action here). - -## 9. Risks -- **Heavy signature drift:** each `RST_X.execute` arg order must be read from source, not guessed — a wrong arg silently benchmarks the wrong thing. Mitigation: implementers read each `RST_*.scala`; the acceptance sweep's consistency check catches gross mismatches. -- **Map/struct fingerprint temptation:** resist forcing consistency on metadata/struct outputs; pure-core-timing-only is correct, and the scorecard must label them so coverage isn't overstated as "parity-verified." -- **Scale:** ~51 functions × (FnSpec + 2 Scala cases) is large but mechanical — subagent-driven, grouped by sub-shape (scalar accessors / coord transforms / tile-out transforms / map-struct), one task per group + mechanism + scorecard. - ---- -*Design approved 2026-06-07. Next: implementation plan (writing-plans), Phase 1. Sources: `bench/spec.py` (registry + select), `BenchDispatch.scala` (heavy dispatch), `registered_functions.txt` (107 canonical), pyrx `functions.py` (107 parity).* diff --git a/docs/superpowers/specs/2026-06-07-benchmark-lifecycle-design.md b/docs/superpowers/specs/2026-06-07-benchmark-lifecycle-design.md deleted file mode 100644 index 49b4fb762..000000000 --- a/docs/superpowers/specs/2026-06-07-benchmark-lifecycle-design.md +++ /dev/null @@ -1,83 +0,0 @@ -# Benchmark results lifecycle — design spec - -**Date:** 2026-06-07 · **Branch:** `beta/0.4.0` · **Status:** design approved (forks settled), pre-plan. -**Goal:** Replace the pile of ad-hoc `test-logs/bench//` dirs with a **per-function authoritative store** keyed to the commit/state each function was validated at, a **change-aware** run command that benchmarks only functions affected by current changes, and a **cleanup** command. The scorecard then reflects the *current best-known* result per function with staleness flags. - -## 1. Motivation - -Today every bench invocation writes a new `/` dir; results accumulate, go stale, and there's no single source of truth for "what is `rst_slope`'s current heavy-vs-light result, and is it still valid?". For the heavyweight-deprecation evidence base ([[terrain-crs-scale-gdal-normal]]) we want, per function: the **latest authoritative** comparison + **provenance** (which commit/source-state it was validated at) + **staleness** (did its source change since?). And iterating on one function shouldn't require re-benchmarking all 70+ — only the **affected** functions. - -## 2. Authoritative store - -`test-logs/bench/authoritative/.json` — one record per function, **the latest authoritative result**, overwritten when that function is (re)validated. Schema: -```json -{ - "fn": "rst_slope", - "validated_commit": "' if working tree had uncommitted changes>", - "validated_at": "", - "sources_hash": "", - "corpus": {"tile_px": [256,512], "srids": [4326,32618], "bands": 2, "dtype": "float32", "nodata_frac": 0.0}, - "set": "core", - "cells": [ {tile_px, srid, mode, consistency, max_rel_delta, nodata_count_delta, speedup, hw_median_ms, lw_median_ms, hw_mpix_s, lw_mpix_s} , ... ], - "heavy_rows": [ ... ], - "light_rows": [ ... ] -} -``` -- **Latest-only:** re-validating a function overwrites its record (no history pile). Git history is the audit trail if needed. -- **Provenance + staleness:** `sources_hash` is the authority signal. A record is **stale** when the current hash of its `sources` ≠ stored `sources_hash` (works for committed *and* uncommitted changes; no git archaeology needed). `validated_commit` is informational. -- The store lives under the gitignored `test-logs/bench/` (not committed) — it's a local working cache, regenerable via the bench. (If we later want it shared/CI-persisted, that's a separate decision; out of scope now.) - -## 3. `sources` on `FnSpec` (the change→function map) - -Add `sources: tuple[str, ...] = ()` to `FnSpec` — repo-relative paths whose content defines the function's behavior on **both** engines: -- pyrx core module(s): e.g. `python/.../pyrx/core/focal.py`, plus shared `python/.../pyrx/core/_nodata.py` where used. -- heavy expression + shared helpers: e.g. `src/main/scala/.../expressions/RST_Filter.scala`, `.../operations/KernelFilter.scala`, `.../gdal/GDALBlock.scala`. -- (Deliberately **exclude** the bench harness files `spec.py`/`BenchDispatch.scala` — editing the registry shouldn't mark every function stale; harness changes are validated by the bench's own tests.) -- Shared files appear in many functions' `sources` → a change to `_nodata.py`/`GDALBlock.scala`/`PixelCombineRasters.scala` correctly marks **all** dependents affected (matching how those fan out — exactly the case convention-only inference gets wrong). - -Each FnSpec must declare its `sources`; a registry-completeness test asserts every registered function has a non-empty `sources` and that the listed paths exist. - -## 4. Commands - -- **`gbx:bench:changed [--base ] [--set core|full] [--all-affected]`** — the change-aware runner: - 1. Compute changed paths: working-tree changes vs `HEAD` (`git diff --name-only HEAD` + untracked), or vs `--base ` if given. - 2. Resolve **affected functions** = registered fns with any `sources` path in the changed set. - 3. Benchmark ONLY those (gen corpus if absent → heavyweight → lightweight → compare), and **write/overwrite their authoritative records** (stamping `validated_commit`/`sources_hash`/timestamp). - 4. Report: changed paths, affected functions (re)validated, and any **changed path mapped to no function** (unmapped — warn, so a forgotten `sources` entry surfaces). -- **`gbx:bench:seed [--set core|full]`** — one-shot: run the full (or core) set and populate the authoritative store for all those functions (bootstraps the store after the cleanup; also the "rebuild everything" escape hatch). -- **`gbx:bench:clean [--runs | --orphans | --all]`** — prune `test-logs/bench/`: - - `--runs` (default): delete ad-hoc `/` dirs, keep `authoritative/`. - - `--orphans`: also delete `authoritative/.json` for functions no longer in the registry. - - `--all`: wipe everything (including authoritative). -- **`gbx:bench:status [--stale-only]`** — render the scorecard **from the store**: per function, last consistency/speedup/max_rel_delta, `validated_commit`, and **STALE** flag (sources changed since). Plus the aggregate Coverage & parity block (N/107, parity counts, functional-parity-gap, not-yet-covered). This replaces "read the last run's summary.md". - -## 5. Pre-push staleness warning (cheap, non-blocking) - -A pre-push step (alongside the QC judge, or folded into a `gbx:*` pre-push helper) that — **without running any benchmark** — checks: for each function whose `sources` changed in the push range (or working tree), is its authoritative record missing or stale (`sources_hash` mismatch)? If so, **warn** (list the functions + suggest `gbx:bench:changed`). Never blocks (benchmarking is Docker-slow; the dev decides when to validate). This keeps the store honest without gating velocity. - -## 6. Scorecard migration - -`compare.py`'s Coverage & parity block (and the per-function table) now reads the **authoritative store** rather than a single run's cells: aggregate over `authoritative/*.json`, mark stale functions, compute coverage = `|store| / 107`, parity/perf counts from the stored cells, functional-parity-gap from `registered_rst()` vs pyrx-implemented, not-yet-covered = registered minus store. `gbx:bench:status` is the entry point; `summarize_compare` stays available for a single ad-hoc run. - -## 7. Sequencing - -Build this lifecycle infra **before resuming Phase 2 coverage**, so Phase 2's bucket-C functions declare `sources` from the start and land directly in the store. After this: re-seed the store (`gbx:bench:seed --set full`) to capture the current 70 functions' authoritative results, then resume Phase 2 (each new function benchmarked via `gbx:bench:changed` as it's added). - -## 8. Testing & validation -- **Unit (venv):** `sources` completeness (every fn has existing-path sources); change→function resolution (given a changed-paths set, the right functions are selected; a shared-file change selects all dependents); store read/write round-trip; staleness detection (hash mismatch → stale); `gbx:bench:status` renders coverage/parity from a synthetic store. -- **Integration:** `gbx:bench:changed` on a one-file edit (e.g. touch `focal.py`) selects exactly `rst_filter`/`rst_convolve`, benchmarks only those, writes 2 records; `gbx:bench:clean --runs` removes ad-hoc dirs but keeps `authoritative/`. -- **Bootstrap:** `gbx:bench:seed --set full` populates the store; `gbx:bench:status` shows 70/107 with no stale. - -## 9. Out of scope -- Committing/sharing the store (stays a local gitignored cache). -- Auto-benchmarking on push (warning only). -- Phase 2 coverage (resumes after). -- Historical result retention (latest-only; git is the audit trail). - -## 10. Risks -- **`sources` drift:** if a function gains a real dependency not listed in `sources`, change-aware runs miss it. Mitigation: the unmapped-changed-path warning in `gbx:bench:changed` + the completeness test; shared helpers explicitly listed. -- **Scripts can't call `Date.now()`** (workflow constraint) — `validated_at` is stamped by the shell command (bash `date`), passed into the Python writer; the Python/Scala bench code receives it as an arg. -- **Staleness via content hash** assumes `sources` fully capture behavior; the warning is advisory, and `gbx:bench:seed` is always available to rebuild. - ---- -*Design approved 2026-06-07 (forks: per-function records + provenance; explicit `sources`; opt-in `gbx:bench:changed` + cheap pre-push staleness warning; `gbx:bench:clean` + ran the stale-dir purge). Next: implementation plan (writing-plans), then re-seed + resume Phase 2. Sources: bench `spec.py`/`compare.py`/`runner.py`, `test-logs/bench/` layout.* diff --git a/docs/superpowers/specs/2026-06-07-benchmark-phase2-bucketC-design.md b/docs/superpowers/specs/2026-06-07-benchmark-phase2-bucketC-design.md deleted file mode 100644 index f73edd5d4..000000000 --- a/docs/superpowers/specs/2026-06-07-benchmark-phase2-bucketC-design.md +++ /dev/null @@ -1,53 +0,0 @@ -# Benchmark Phase 2 — bucket C (multi-input / constructors / tiling / readers) — design spec - -**Date:** 2026-06-07 · **Branch:** `beta/0.4.0` · **Status:** design (recommended basis), pre-plan. -**Goal:** Add the 14 bucket-C `rst_*` functions to the heavy-vs-light benchmark, taking coverage **70 → 84 / 107**. These are the "special-shaped" functions the harness deferred (multi-tile inputs, collection outputs, byte/path readers). Each registers `FnSpec.sources` and is validated via the new `gbx:bench:changed` into the authoritative store. - -## 1. The 14 functions, by shape (from binding signatures) - -| Group | Functions | In | Out | Harness need | -|---|---|---|---|---| -| **C1 readers / single-tile** | `fromcontent`(bytes,driver), `fromfile`(path,driver), `tryopen`(tile→bool), `buildoverviews`(tile,levels,resampling) | bytes / path / one tile | tile / bool | byte/path input adapter; `tryopen`→scalar, `fromcontent`/`fromfile`/`buildoverviews`→raster | -| **C2 subdataset metadata** | `subdatasets`(tile→MAP), `getsubdataset`(tile,name→tile) | tile | map / tile | GTiff has **no subdatasets** → timing-only | -| **C3 multi-tile IN** | `frombands`(ARRAY→tile), `combineavg`(ARRAY→tile), `merge`(ARRAY→tile) | **array of tiles** | tile | **array-input adapter** + synthesized multi-tile inputs | -| **C4 multi-tile OUT** | `maketiles`(tile,mb→ARRAY), `retile`(tile,w,h→ARRAY), `tooverlappingtiles`(tile,w,h,ov→ARRAY), `separatebands`(tile→ARRAY), `xyzpyramid`(tile,minz,maxz→ARRAY) | one tile | **array of tiles** | **collection fingerprint** | - -## 2. Decisions (recommended basis) - -- **C4 → collection fingerprint** (not timing-only). New fingerprint kind `raster_collection`: `{count: N, bands_total, agg: {min,max,mean,std} over all output tiles' band-0 (or all-band) pixels}`. Consistency compares `count` (exact) + the aggregate stats (tolerance) — a real swap-safety signal. Heavy `RST_*.execute` returns a tile array; pyrx returns a list of tile bytes — both reduce to the same collection fingerprint. -- **C3 → synthesize inputs from the corpus** (no new corpus fixture): `frombands` ← the 2 bands of the corpus tile split into 2 single-band tiles; `combineavg` ← 2–3 copies of the corpus tile (aligned); `merge` ← 2 variants with **offset extents** (shift the geotransform origin so the mosaic spans a union). Synthesize inside the bench adapter from the one corpus tile the runner already has. New **array-input adapter**: core_fn receives the synthesized list of open ds; col_fn builds an `ARRAY` Spark column; heavy dispatch builds the tile array. -- **C2 → timing-only** (`fingerprint=False`, pure-core): `subdatasets` returns an empty MAP on plain GTiff (times, not compared); `getsubdataset` has no subdataset to extract on GTiff — register **timing-only with a note**, and if it errors on the corpus, mark it `na`/skip with the reason surfaced (don't fake a subdataset). A dedicated NetCDF/HDF fixture is disproportionate for 2 functions (revisit only if subdataset coverage is later required). -- **C1 → fits the harness with a byte/path adapter:** `fromcontent`(corpus tile bytes + "GTiff"), `fromfile`(corpus tile path + "GTiff") → both read the same source → raster fingerprint should match. `tryopen`(valid tile bytes) → scalar bool (1.0 both). `buildoverviews`(tile, levels=[2,4], "average") → raster fingerprint of the **base band** (overviews are internal; base band unchanged → exact/within_tol). -- **Sequencing within Phase 2:** **C1 → C3 → C4** (C1 fits the harness fastest; C3 adds the array-input adapter; C4 adds the collection fingerprint). C2 timing-only throughout. Each sub-step lands its functions in the store via `gbx:bench:changed`. - -## 3. Architecture / harness additions - -- **Byte/path input adapter (C1):** the bench currently hands `core_fn` an open `ds` (a tile). For `fromcontent`/`fromfile`/`tryopen`, the input is bytes/path, not a derived tile. Extend `FnSpec` with an `input_kind` (default `"tile"`; new `"bytes"`, `"path"`) so the runner passes the corpus tile's raw bytes / file path instead of an opened ds. Heavy dispatch + col_fn mirror (heavy `RST_FromContent.execute(bytes, driver)` etc.). -- **Array-input adapter (C3):** `input_kind="tile_array"` → the runner synthesizes the input list (a small `_synthesize(ds, fn)` helper: split-bands / copies / offset-variants) and passes a list of ds to core_fn; col_fn wraps an `ARRAY` column; heavy builds the Scala tile array. Synthesis is deterministic from the corpus tile. -- **Collection-output fingerprint (C4):** add `BenchFingerprint.ofCollection(tiles)` (Scala) + the pyrx equivalent → serialize `{kind:"raster_collection", count, agg-stats}`. `compare.py` `compare_fingerprints` gains a `raster_collection` branch: `count` must match exactly (else divergent), agg-stats compared with the existing tolerance. `_STATS` reused. -- **`sources` for all 14** (per the lifecycle design): pyrx core module (`agg.py`/`tiling.py`/`ops.py`/`accessors.py`/`xyz.py`) + heavy `RST_.scala` (+ shared `PixelCombineRasters.scala` for frombands/combineavg/merge; tiling helpers as applicable). -- **Registry/scorecard:** all 14 `core=False`, added to `select(set="full")`; `gbx:bench:status` then shows **84/107**; the not-yet-covered list drops to the 23 bucket-B + bucket-A/D functions (Phases 3–4). - -## 4. Per-function fingerprint summary - -- **Compared (raster / scalar / collection):** `fromcontent`, `fromfile` (raster — re-read of same source), `tryopen` (scalar bool), `buildoverviews` (raster base band), `frombands`/`combineavg`/`merge` (raster — deterministic synthesized inputs), `maketiles`/`retile`/`tooverlappingtiles`/`separatebands`/`xyzpyramid` (raster_collection). -- **Timing-only (`fingerprint=False`):** `subdatasets`, `getsubdataset` (no subdatasets on the GTiff corpus). Any C-function whose synthesized input/op can't be made cross-engine-identical is downgraded to timing-only **with an explicit note**, not forced. - -## 5. Testing & validation - -- **Unit (venv):** registry well-formedness for the 14 (sources exist, modes/fingerprint valid); the synthesis helper (`_synthesize` produces the expected band-split/copies/offset-variants); the `raster_collection` fingerprint round-trips and `compare_fingerprints` flags a count mismatch as divergent + tolerates agg-stat noise; `tryopen` scalar path. -- **Cross-API acceptance (Docker, via the lifecycle commands):** `gbx:bench:changed` (or scoped `gbx:bench:all --functions `) over the 14 → write store records; `gbx:bench:status` shows **84/107**, the C functions' consistency (most within_tol; record + classify any divergent as a finding, e.g. `merge` mosaic edge or `combineavg` NoData — investigate, don't loosen). Run **backgrounded with ~30s status** (Docker). -- **Store integration:** each C function ends with an authoritative record + `sources`; `gbx:bench:status` stale=0 after validation. - -## 6. Sequencing & out of scope -- C1 (4) → C3 (3) → C4 (5); C2 (2) timing-only alongside. Then Phases 3 (bucket B: DGGS/vector-out, 13) and 4 (bucket A aggregators + D geometry-in, 10) reach 107/107. -- Out of scope: a NetCDF/HDF subdataset fixture (C2 stays timing-only); the `_agg` aggregators (Phase 4); DGGS/vector-out fingerprints (Phase 3). - -## 7. Risks -- **Collection fingerprint semantics:** tile ordering may differ heavy-vs-light → aggregate over the *whole collection* (order-independent stats), and compare `count` exactly + stats by tolerance. Don't rely on per-tile pairing. -- **C3 synthesis identity:** the synthesized inputs must be byte-identical across engines (same split/copies/offsets) or the comparison is meaningless — synthesize from the same corpus tile with deterministic transforms; if heavy can't receive the identical synthesized array cleanly, downgrade that fn to timing-only + note. -- **`merge`/`combineavg` may surface real divergences** (mosaic edge / NoData averaging) — those are findings for the scorecard (candidate consistency fixes), not tolerance loosening. -- **`getsubdataset` on GTiff** likely errors/empty — keep timing-only and surface the reason; do not fabricate a subdataset. - ---- -*Design (recommended basis) 2026-06-07. Next: implementation plan (writing-plans), sequenced C1→C3→C4 with C2 timing-only; validate via `gbx:bench:changed` into the store. Sources: bench `spec.py`/`compare.py`/`BenchDispatch.scala`/`store.py`, pyrx `functions.py` bindings, bucket-C investigation. See [[benchmark-lifecycle-design]] (the store/changed/status infra this plugs into).* diff --git a/docs/superpowers/specs/2026-06-07-pyrx-focal-nodata-edge-fix-design.md b/docs/superpowers/specs/2026-06-07-pyrx-focal-nodata-edge-fix-design.md deleted file mode 100644 index b35d6292d..000000000 --- a/docs/superpowers/specs/2026-06-07-pyrx-focal-nodata-edge-fix-design.md +++ /dev/null @@ -1,61 +0,0 @@ -# pyrx focal NoData + edge fix — design spec - -**Date:** 2026-06-07 · **Branch:** `beta/0.4.0` · **Status:** design approved, pre-plan. -**Goal:** Make pyrx focal ops (`filter` = min/max/median/mean; `convolve`) match the heavyweight (rasterx/`GDALBlock`) on **NoData handling** and **kernel-edge handling**, so `rx.rst_filter/convolve → prx.*` swaps are consistent. Heavyweight is the reference. - -## 1. Motivation - -The full-coverage benchmark left `rst_convolve` `divergent` (max_rel_delta ~0.012–0.057, `nodata_count_delta=0`). Root cause is **edge handling**, not NoData poisoning (the default corpus tiles are effectively nodata-free, so the NoData path isn't even exercised). `rst_filter` reads `within_tol` only because the corpus is nodata-free and median is edge-robust — on a NoData-bearing tile pyrx would poison while heavy skips. This is the deferred focal item from `2026-06-06-pyrx-nodata-edge-fix-design.md` §6, now root-caused. - -## 2. Heavy target semantics (verbatim from `GDALBlock.scala`) - -**`valuesAt(x,y,kw,kh)`** (backs avg/min/max/median/mode): iterate the kw×kh window; include a neighbor **only if** `xIndex/yIndex in [0,width/height)` **AND** `mask≠0` **AND** `value≠nodata`. Out-of-bounds neighbors are **skipped** → the window **shrinks at edges** (no padding, no reflect). Aggregate over the collected valid values: -- `avg` = `sum/length` (mean of valid); `min`/`max`/`median`/`mode` over valid. -- If **no** valid neighbor → output **nodata**. - -**`convolveAt(x,y,kernel)`**: `sum += value·kernel(i)(j)` **only** for in-bounds valid neighbors; out-of-bounds & invalid contribute **0**; weights are **not** renormalized → edge is effectively **zero-padded**. Kernel is applied **un-flipped** (`kernel(i)(j)` indexed in window reading order → correlation, not convolution). - -## 3. pyrx today (`pyrx/core/focal.py`) - -- `filt`: `ndimage.{minimum,maximum,median}_filter` / `uniform_filter` on raw arrays — default boundary (`reflect`); no mask. -- `convolve`: `ndimage.convolve(data, k, mode="nearest")` — edge **replication**; kernel **flipped** (scipy `convolve` flips); no mask. - -Divergences vs heavy: edge (reflect/nearest vs shrink/zero-pad), kernel orientation (flip vs none), and NoData (poisons vs skips). - -## 4. Fix — rewrite `focal.py` to match GDALBlock (reuse `_nodata.py`) - -Per band, `data, valid = read_masked(ds, band)` (`_nodata.read_masked` → float64 data + bool valid mask). Then: - -- **`convolve`** → `scipy.ndimage.correlate(data * valid, kernel, mode="constant", cval=0.0)`. - - `correlate` (not `convolve`) → un-flipped kernel, matching `convolveAt`'s `kernel(i)(j)`. - - `data*valid` zeros invalid pixels; `mode="constant", cval=0` zeros out-of-bounds → both contribute 0, no renormalization. Matches `convolveAt` exactly. - - Output dtype Float64 (unchanged). NoData: a pixel is output-nodata only where it has **no valid neighbor** — compute `valid_count = correlate(valid, ones_like(kernel!=0), mode="constant", cval=0)`; where `valid_count==0` → nodata. (Heavy `convolveAt` returns 0 for all-invalid, but its mask path marks it; emit nodata there for a clean fingerprint.) -- **`mean`** → `num = correlate(data*valid, box, mode="constant", cval=0)`, `cnt = correlate(valid, box, mode="constant", cval=0)`; `result = num/cnt` where `cnt>0`, else nodata. `box = np.ones((size,size))`. Window-shrinks at edges; skips NoData. Output Float32 (unchanged). -- **`min`/`max`/`median`/`mode`** → set invalid → `np.nan`; `scipy.ndimage.generic_filter(arr, func, size=size, mode="constant", cval=np.nan)` with NaN-aware `func` (`np.nanmin`/`np.nanmax`/`np.nanmedian`/a nan-aware mode). All-NaN window → nodata. Output dtype = input (unchanged) for min/max/median; mode = input dtype. - - `generic_filter` is slower than the built-ins but correct; acceptable for benchmark/typical tiles. (If a built-in fast path is needed later, optimize then — YAGNI now.) -- **Emit** via `_nodata.emit(ds, result, nodata, invalid, dtype)` (or the equivalent: set profile nodata + write), so the output declares NoData where all-invalid, consistent with terrain/band-math. - -Signatures unchanged: `filt(ds, kernel_size, operation)`, `convolve(ds, kernel)`. Bindings (`prx.rst_filter`/`rst_convolve`) and the bench FnSpecs need **no** change. - -## 5. Testing & validation - -- **Unit (venv, `test_core_focal.py`):** - - **Edge (nodata-free):** convolve/mean/median on a small tile → border pixels match the GDALBlock rule (zero-pad for convolve; shrink-renormalized mean; shrunk-window median). Pin against hand-computed expected for a 4×4 tile + 3×3 kernel. - - **NoData skip:** plant a NoData pixel; assert its neighbors' outputs exclude it (mean renormalizes over valid; convolve zero-contributes); a pixel whose entire window is NoData → output NoData. - - **Kernel orientation:** an **asymmetric** kernel convolve matches correlation (un-flipped), distinguishing the new behavior from the old `convolve` flip. - - Existing focal tests that assumed reflect/nearest edges get updated to the new (correct) GDALBlock contract — recompute expected, don't weaken. -- **Cross-API acceptance:** re-run the bench for `rst_convolve`/`rst_filter` (pure-core). `rst_convolve` must flip `divergent → within_tol`/`exact` on both tiles. Add a **NoData-bearing focal tile** to the validation (a small nodata sweep, or assert via the unit NoData test) so the NoData-skip path is actually exercised — currently the default sweep is nodata-free and would not catch poisoning. -- **No regression:** `rst_filter` stays `within_tol`/`exact`. - -## 6. Out of scope -- Other terrain/band-math (already fixed). -- A fast built-in path for min/max/median (use `generic_filter`; optimize only if a real perf need arises). -- `mode` filter is heavy-only in the bench sense (pyrx `filt` doesn't expose mode today) — keep pyrx's current operation set (min/max/median/mean); only align the shared ones. - -## 7. Risks -- **`generic_filter` perf**: slower for large tiles; acceptable for the corpus, and focal isn't a hot path. Flag if a sweep tile is huge. -- **NoData not exercised by default corpus**: must add a nodata tile/test or the fix's NoData half is unverified by the sweep — covered by the unit NoData tests as the authoritative check. -- **Output-nodata for convolve all-invalid**: heavy `convolveAt` returns 0 (mask handled separately); pyrx emitting nodata there is the cleaner/consistent choice and shouldn't disagree on valid interior pixels — verify the fingerprint (stats over valid pixels) still matches. - ---- -*Design approved 2026-06-07 via investigation (GDALBlock.scala read + bench root-cause). Next: implementation plan (writing-plans). Sources: `GDALBlock.scala` valuesAt/convolveAt, `KernelFilter.scala`, pyrx `focal.py`, bench `comparison.csv` (rst_convolve divergent). See [[pyrx-nodata-edge-divergences]].* diff --git a/docs/superpowers/specs/2026-06-08-benchmark-phase3-bucketB-design.md b/docs/superpowers/specs/2026-06-08-benchmark-phase3-bucketB-design.md deleted file mode 100644 index 59d275208..000000000 --- a/docs/superpowers/specs/2026-06-08-benchmark-phase3-bucketB-design.md +++ /dev/null @@ -1,61 +0,0 @@ -# Benchmark Phase 3 — bucket B (DGGS grid / vector-out) — design spec - -**Date:** 2026-06-08 · **Branch:** `beta/0.4.0` · **Status:** design (recommended basis approved), pre-plan. -**Goal:** Add the 13 bucket-B functions to the heavy-vs-light benchmark (coverage **84 → 97 / 107**) with two new fingerprint kinds for cell/geometry output, classify divergences per best-practice, and **perf-review** the bucket (flag light fns meaningfully slower than heavy → the perf backlog). Each declares `FnSpec.sources`; results land in the authoritative store. - -## 1. The 13 functions, by output shape - -| Group | Functions | Output | pyrx core | heavy | -|---|---|---|---|---| -| **B-grid (DGGS)** | `h3_tessellate`(tile,res), `h3_rastertogrid{avg,count,max,median,min}`(tile,res), `quadbin_rastertogrid{avg,count,max,median,min}`(tile,res) — 11 | array of **cells** `(cell_id[, value])` | `tessellate.tessellate_h3`, `gridagg.raster_to_grid` | `RST_H3_Tessellate`, `RST_H3_RasterToGrid*`, `RST_Quadbin_RasterToGrid*` | -| **B-vec (vector-out)** | `contour`(tile,levels,…), `polygonize`(tile,band,connectedness) — 2 | array of **geometry features** `(geom, level/value)` | `analysis.contour`, `features.polygonize` | `RST_Contour`, `RST_Polygonize` | - -All `input_kind="tile"` (single ds in); the novelty is **output** fingerprinting. - -## 2. New fingerprint kinds - -- **`dggs_grid`** (the 11 cell-output fns): `{"kind":"dggs_grid","count":N,"cells_hash":,"agg":{min,max,mean,std over cell values}}`. For `tessellate` / `*count`, the "value" is the per-cell count (or 1); the agg + count still apply. - - **Consistency = cell COUNT exact + value `agg` within tolerance** (the pass criterion). Additionally **report cell-set overlap** (Jaccard of the two `cells_hash` sets) as an informational note — exact cell-set equality across two H3/quadbin implementations on raster edges is unlikely, so it's reported, not gating. A `cells_hash` match → exact; count+agg match → within_tol; count mismatch → divergent. -- **`vector`** (contour, polygonize): `{"kind":"vector","count":N,"measure":,"attr_agg":{min,max,mean,std over the level/value attribute}}`. - - **Consistency = feature COUNT + `measure` within tolerance** + attr agg within tol. **NOT** exact geometry — GDAL vs pyrx contour/polygonize place vertices differently; count + aggregate length/area is the meaningful swap-safety signal (same as the `raster_collection` philosophy). - -Both kinds are order-independent (aggregate over the whole set). Add to Scala `BenchFingerprint` (`ofDggsGrid`, `ofVector`) + pyrx (`fingerprint_dggs_grid`, `fingerprint_vector`) + `compare.py` branches reusing `_close(rel_tol,abs_tol)`. - -## 3. H3 / quadbin parity (first plan step — gate before fingerprinting) - -pyrx uses the `h3` + `quadbin` Python libs; heavy uses the GridX H3/quadbin. **Verify they share the standard cell-id definitions** (same H3 index for a given lat/lon/res; same quadbin index) so `cell_id`s are directly comparable. Quick check: pick a known lat/lon/res, compute the cell id in both (a pyrx `_h3_cell`/`_quadbin_cell` call vs the heavy GridX call) and confirm equality. If they match → cell-set overlap is meaningful. If they diverge inherently (different cell conventions) → fall back to **count + value-agg only** (drop `cells_hash` from the pass) and record it as a finding (per best-practice: fix whichever tier is non-standard, or document). - -## 4. Per-function notes / known forks -- **rastertogrid sampling:** each engine maps pixels → cells (pixel centroid → H3/quadbin cell, then aggregate by the op). If both sample pixel centroids identically → same cells + same per-cell aggregates. Edge/resolution handling may differ → cell-set differences (reported via overlap). The **value agg** (e.g. mean of all cells' means) is the robust comparison. -- **`*count`:** value = per-cell pixel count; agg over those counts + total cell count. -- **`contour`:** fixed `levels` arg (e.g. evenly-spaced over the band's range); measure = total LineString length. -- **`polygonize`:** `band=1, connectedness=4` (or 8 — match heavy's default); measure = total polygon area; attr = the polygon value. -- **v0.3.0 status:** check each bucket-B fn's v0.3.0 membership; for any **heavy** behavior change document per [[commit-message-hygiene]]-adjacent release-notes discipline (light changes never need v0.3.0 docs — pyrx is v0.4.0-new). - -## 5. Perf-review (per the standing directive — [[perf-parity-light-vs-heavy]]) -As part of validation, after the bucket-B functions land in the store, **flag any light fn meaningfully slower than heavy** (light slower by ≥~1 ms AND ≥~1.5×) and add to the perf backlog (task #90). DGGS tessellation + contour/polygonize in pyrx may be slow vs heavy GDAL — capture it. This is a first-class output of the phase alongside consistency, not an afterthought. - -## 6. Architecture / files -- `bench/spec.py`: 13 FnSpecs (`core=False`, `sources`, args), grouped B-grid/B-vec; the runner fingerprints list-returning DGGS/vector core_fns via the new kinds. -- `bench/results.py`/`fingerprint.py`: `fingerprint_dggs_grid`, `fingerprint_vector` (pyrx side). -- `src/main/scala/.../bench/BenchFingerprint.scala`: `ofDggsGrid`, `ofVector`. -- `src/test/.../bench/BenchDispatch.scala`: 13 dispatch cases wrapping the heavy RST_* outputs in the new fingerprints; `BenchDispatchTest` size +13 → 97. -- `bench/compare.py`: `dggs_grid` + `vector` branches. - -## 7. Testing & validation -- **Unit (venv):** the two new fingerprint kinds round-trip + compare correctly (count mismatch → divergent; agg within tol → within_tol; cell-set overlap reported); H3/quadbin parity check; per-fn registry well-formedness. -- **Scala test-compile** for the dispatch cases. -- **Cross-API acceptance (Docker, controller-orchestrated, backgrounded + ~30s status):** `gbx:bench:changed`/scoped run → store; `gbx:bench:status` → **97/107**; classify each bucket-B fn's consistency (count/agg) — record divergences (likely cell-edge membership or geometry-count differences) per best-practice (fix the wrong tier, or document if a v0.3.0 heavy change). -- **Perf-review:** flag meaningfully-slower-light bucket-B fns → task #90. - -## 8. Sequencing & out of scope -- B-grid (11) then B-vec (2) — or together if the fingerprint infra lands first. Then Phase 4 (bucket A aggregators + D geometry-in) → 107/107. -- Out of scope: exact geometry/cell-set equality (use count + aggregate measures + overlap report); the heavy `RST_TileXYZ` RGBA-render follow-up; the perf optimizations themselves (queued, task #90 — Phase 3 only *flags* them). - -## 9. Risks -- **H3/quadbin cell-edge membership** differs between implementations → cell-set overlap <100% even when value-agg matches; handled by making value-agg+count the pass criterion, overlap informational. -- **contour/polygonize feature counts** may differ (GDAL vs pyrx algorithms produce different # of features at the same levels/connectedness) → a real divergence to classify (fix wrong tier or document), not auto-pass. -- **Perf:** DGGS tessellation / vectorization in pyrx may be slow → captured by the perf-review, not blocking coverage. - ---- -*Design 2026-06-08 (recommended basis). Next: implementation plan (writing-plans). Sources: pyrx `core/{gridagg,tessellate,features,analysis}.py` + bindings, heavy `RST_H3_*`/`RST_Quadbin_*`/`RST_Contour`/`RST_Polygonize`, bench `spec.py`/`compare.py`/`BenchFingerprint.scala`. Plugs into the store/changed/status lifecycle + [[perf-parity-light-vs-heavy]].* diff --git a/docs/superpowers/specs/2026-06-08-benchmark-phase4-bucketAD-design.md b/docs/superpowers/specs/2026-06-08-benchmark-phase4-bucketAD-design.md deleted file mode 100644 index fe40ec36c..000000000 --- a/docs/superpowers/specs/2026-06-08-benchmark-phase4-bucketAD-design.md +++ /dev/null @@ -1,72 +0,0 @@ -# Benchmark Phase 4 — buckets A (aggregators) + D (geometry-in) — design spec - -**Date:** 2026-06-08 · **Branch:** `beta/0.4.0` · **Status:** design (pre-plan). -**Goal:** Cover the final **10** registered `rst_` functions (coverage **97 → 107/107**) — the 7 `*_agg` aggregators via a **real Spark `groupBy().agg()` harness** (user-chosen, more faithful than pure-core for the deprecation decision) and the 3 geometry-in functions via a new geometry corpus — classify divergences per best-practice, and perf-review the bucket. - -## 1. The 10 functions - -| Bucket | Functions | Input | Output | Heavy | Light | -|---|---|---|---|---|---| -| **A — tile aggregators (4)** | `combineavg_agg`, `merge_agg`, `frombands_agg`, `derivedband_agg` | grouped tile rows | tile | `RST_*Agg` (`TypedImperativeAggregate`, `expressions/agg/`) | pandas_udf reducers in `.agg()` (`core/agg.py` + `functions.py:2632+`) | -| **A — geometry aggregators (3)** | `rasterize_agg`, `gridfrompoints_agg`, `dtmfromgeoms_agg` | grouped geometry rows | tile | `RST_*Agg` | pandas_udf reducers (`core/agg.py`,`core/tin.py`) | -| **D — geometry-in (3)** | `rasterize`, `gridfrompoints`, `dtmfromgeoms` | one geometry / geometry-array | tile | `RST_*` | `core/features.py`,`core/tin.py` | - -## 2. Key insight — leverage the EXISTING spark-path harness -- `bench/runner.py:307-468` `run_spark_path()` already: creates a `local[2]` arrow-enabled Spark session, loads a tile DataFrame on the `(cellid LONG, raster BINARY, metadata MAP)` struct schema, invokes `col_fn()` over N rows (10/100/1000/10000), times via `.write.format("noop").save()`. -- Heavy `HeavyRunner.runSparkPath(spark)` on `SilentSparkSession` (`HeavyBenchSuite extends SilentSparkSession`) registers SQL UDFs via `functions.register(spark)` and runs the columnar path. -- The synth recipes (`synth.py`: `frombands`/`combineavg`/`merge`) already produce deterministic multi-tile groups written-once / read-by-both. - -**So Phase 4 EXTENDS this, it doesn't rebuild it.** New work = (a) groupBy-aggregate invocation, (b) geometry corpus, (c) `geometry` input_kind, (d) 10 FnSpecs. - -## 3. Consistency vs perf — two signals, one harness -The bench's value is heavy-vs-light **consistency** AND **perf**. For aggregators both still apply (the recon's "no fingerprinting" note is rejected — consistency is the whole point): -- **Consistency** = aggregate a FIXED, deterministic group (small N) on both tiers → ONE output tile → fingerprint (raster kind) → exact/within_tol. The group must be identical + deterministic across tiers (reuse synth recipes for tile-aggs; the new geometry corpus for geometry-aggs). This answers "does light's aggregate match heavy's?" -- **Perf** = time the real `groupBy(key).agg(col_fn(...))` at scale (the existing N-row scaling + noop-write timing), on both tiers. This is the distributed-aggregation timing the user wants. - -Implementation: the spark-path aggregate mode runs `df.groupBy(key).agg(col_fn(...))`, (a) collects the single output tile for the fixed-group consistency fingerprint, (b) times the scaled groupBy for perf. - -## 4. New pieces (the actual work) - -### 4a. Geometry corpus (new — biggest piece) -`datagen.py`/`synth.py`: synthesize deterministic geometry sets from tile extents (seeded, written-once, read-by-both — same pattern as tiles): -- **boxes** (for `rasterize`/`rasterize_agg`): N axis-aligned boxes derived from a tile's bounds (shrunk/offset for variety), each with a burn value. WKB BINARY + value DOUBLE. -- **points** (for `gridfrompoints`/`gridfrompoints_agg`): N points scattered across a tile extent (seeded), value DOUBLE. -- **z-points** (for `dtmfromgeoms`/`dtmfromgeoms_agg`): N 3D points (Z sampled from the tile raster), breaklines = NULL/empty. -- All in the **output CRS** (geometry WKB carries no CRS; must match the target `srid`). Extent/size/srid args derived from the source tile. -- A `corpus.json`-style manifest entry so both tiers read identical geometry. - -### 4b. `input_kind="geometry"` -New input kind in `spec.py`/`runner.py`: feeds a geometry set (the corpus above) to a function. Bucket D uses it (single geometry / geometry-array → raster); geometry-aggregators use a grouped geometry DataFrame. - -### 4c. groupBy-aggregate invocation -- **Light** (`run_spark_path`): an aggregate branch — build the N-row DataFrame (tile rows from the synth group, or geometry rows from the geometry corpus) + a group key column, `df.groupBy(key).agg(col_fn(...).alias("out"))`, collect for fingerprint + time at scale. - - `frombands_agg`: add a `band_index` INT column (0,1,…) — heavy + light both sort ascending by it. - - `combineavg_agg`: aligned tiles (synth `combineavg` recipe). - - `merge_agg`: offset tiles (synth `merge` recipe). - - `derivedband_agg`: reuse the hardcoded `_DERIVEDBAND_PYFUNC` (mean-bands) from `spec.py`. - - `gridfrompoints_agg`: power=2.0, max_pts=12; `dtmfromgeoms_agg`: breaklines=NULL, tolerances=0.0. -- **Heavy** (`HeavyRunner.runSparkPath`/`BenchDispatch`): the matching `df.groupBy(key).agg(expr("gbx_rst_*_agg(...)"))` (the `TypedImperativeAggregate` runs as a real Catalyst aggregate). Same group key + N rows + columns. - -### 4d. 10 FnSpec entries (`spec.py`) -- 7 aggregators: `modes=("spark-path",)` (aggregate-mode), `core=False`, `sources` (pyrx core/agg.py|tin.py + functions.py binding + the Scala RST_*Agg + shared serde), fingerprint = raster (fixed-group output). -- 3 geometry-in: `input_kind="geometry"`, pure-core + spark-path, fingerprint = raster. -- Bump `BenchDispatchTest` size +10 → 107. - -## 5. v0.3.0 membership -Check each of the 10 against `git show v0.3.0:docs/tests-function-info/registered_functions.txt`. We are NOT changing heavy behavior (only benchmarking it), so no release-notes entries are expected — UNLESS a divergence forces a heavy fix (then document per the v0.3.0 rule; light fixes are v0.4.0-exempt). Several of these (TIN/IDW `dtmfromgeoms`/`gridfrompoints`, and likely `rasterize`) are v0.4.0-new. - -## 6. Perf-review (standing directive — [[perf-parity-light-vs-heavy]]) -Flag any of the 10 where light is meaningfully slower than heavy (≥~1ms AND ≥~1.5×). Distributed groupBy timing may surprise (pandas_udf serialization overhead vs Catalyst). Also run `gbx:perf:vectorscan` ([[pyrx-vectorization-standing-check]]) over any new pyrx core touched. Add findings to the perf backlog. - -## 7. Risks -- **Geometry corpus is net-new** — deterministic, CRS-correct, read-by-both. Highest-effort piece. -- **pandas_udf serialization** in light groupBy (Arrow tile-struct round-trip) — may dominate light timing; that's a real, fair signal to capture. -- **Consistency group must be byte-identical across tiers** — reuse the write-once/read-both synth path; extend it to geometry. -- **CRS alignment** geometry WKB ↔ target srid. -- **Aligned-tiles / band_index / breaklines** contract details (per §4c) — get them exact or consistency diverges spuriously (cf. the Phase-3 lesson: bench-arg mismatches masquerade as divergences). - -## 8. Validation (controller-orchestrated, backgrounded + 30s heartbeat) -Re-seed/scoped-bench the 10 → store; `gbx:bench:status` → **107/107**, stale=0; classify each fn's consistency (fix the wrong tier per best-practice, or document a v0.3.0 heavy change); perf-review; gates (`gbx:test:bindings`, `gbx:lint:python`, `gbx:lint:scalastyle`). **Re-bench is the verdict** (Phase-3 lesson: don't trust unit-test/hypothesis "done"). - -## 9. Sequencing -P4.1 geometry corpus + `geometry` input_kind → P4.2 bucket D (3 geometry-in) → P4.3 groupBy-aggregate harness + bucket A (7) → P4.4 validate (107/107) + perf-review. Out of scope: the queued tessellate mosaic-mode (#101) + viewshed reimpl (#106). diff --git a/docs/superpowers/specs/2026-06-11-light-pmtiles-writer-design.md b/docs/superpowers/specs/2026-06-11-light-pmtiles-writer-design.md deleted file mode 100644 index 1339456fa..000000000 --- a/docs/superpowers/specs/2026-06-11-light-pmtiles-writer-design.md +++ /dev/null @@ -1,296 +0,0 @@ -# Light Tiled-Output Framework + PMTiles Writer (`pmtiles_gbx`) Design - -**Date:** 2026-06-11 -**Branch:** `light-readers` -**Status:** Approved (design); ready for implementation plan -**Supersedes:** the earlier narrow single-archive `pmtiles_gbx` draft (this redesign -reframes it as the first backend on a shared, consolidating tiled-output framework, -per the spatial-sharding mental model `input/pmtiles/pmtiles_mental_model.md`). - -## Summary - -Build a **shared lightweight tiled-output framework** at -`databricks.labs.gbx.ds.tiles` that consolidates the cross-format pipeline -logic for distributed, spatially-sharded tile/pyramid outputs, and ship its -**first backend — PMTiles (`pmtiles_gbx`)** — as a pure-Python DataSource V2 -writer. The framework is deliberately factored so later backends (COG-by-quadbin -+ VRT, MVT-dir, MBTiles) are incremental additions, not rewrites. - -Mental model (full doc: `input/pmtiles/pmtiles_mental_model.md`; see -[[pmtiles-spatial-sharding-model]]): treat tiled outputs as **immutable, -spatially-indexed shards** — partition the world by a grid (quadtree) → each -worker owns a parent tile → emits one **bounded, non-overlapping** shard → deliver -a **catalog** over the shards (not necessarily one merged file). Buffering keeps -edges correct; raster adds bottom-up pyramids, global scaling, sparse-skip. - -**Writers consolidate the pipeline; the format is a pluggable backend.** Named, -documented writers (`pmtiles_gbx`, …) expose the behavior through `.option(...)` -knobs with sensible defaults (single-archive vs sharded, catalog type, tile -type/compression, metadata, mode) — not a parallel set of higher-level functions. - -## Why a framework (not a one-off writer) - -Per the function inventory (see the brainstorm analysis), PMTiles/COG/MVT/MBTiles -all share the same outer pipeline — **grid → shard → (pyramid) → assemble → -catalog** — and differ only in the per-shard container and catalog type. The -heavy tier reimplements this per format; the light tier should consolidate it once -and vary the leaf. The "consolidate logic in Writers/Readers" principle: a writer -does the spatial-sharding orchestration, not just "write exactly these bytes." - -Two backend *shapes* share the same grid/shard/catalog layer: -- **Tile-archive backends** (PMTiles, MBTiles, MVT-dir): input `(z,x,y,bytes)` → - assemble a container. -- **Raster-file backend** (COG, later): input raster tiles → one COG per cell. -The shared layer = grid + shard + catalog (+ pyramid helpers); the leaf = how one -shard becomes a file. - -## Function consolidation (retraction principle) - -As the framework absorbs pipeline logic, any `rst_*`/`st_*` function that is -*really* an I/O operation (writing a tiled/pyramid output, or reading a tiled -source) should **consolidate into the writer/reader, and the redundant function be -retracted** — one canonical path, no function-and-writer duplication. Beta + -no-aliases makes this clean. Decisions are made **per backend, not en masse**, and -each retraction is a coordinated **both-tier** change (remove from Scala -`override def name` + light `pyrx`, `registered_functions.txt`, -`function-info.json`, docs, and the function-count diagram) that must keep the QC -`binding-parity` / `doc-coverage` / `diagram-coverage` gates green. - -- **`rst_cog_convert` → the `cog_gbx` writer** — the clearest candidate; "convert - to COG" is a write. Retract when the COG-by-quadbin backend lands. -- **Tiling family** (`rst_xyzpyramid`, `rst_tilexyz`, `rst_maketiles`, - `rst_retile`, `rst_tooverlappingtiles`) — keep the genuine "produce tiles" - transforms, but fold pyramid/shard/overlap logic into the framework and retract - whatever becomes pure writer-internals; evaluate as each backend absorbs it. -- **This (PMTiles-first) phase retracts nothing** — it is net-new (no existing - light PMTiles function; heavy `pmtiles` stays). The principle is recorded here to - guide the COG and tiling phases. - -## Architecture - -New tier-neutral Python DataSource package `databricks.labs.gbx.ds` -(`python/geobrix/src/databricks/labs/gbx/ds/`) — a sibling of `pyrx`/`bench`/etc. -(every Python DataSource is inherently light; the `_gbx` format names + docs tier -labels carry the tier signal, so the path need not). With: - -### `ds/tiles/` — the shared framework - -- **`grid.py`** — grid-pluggable tile math. A `Grid` protocol: - `tile_bbox(z,x,y) -> (minlon,minlat,maxlon,maxlat)`, `parent(z,x,y, shard_zoom) - -> (sz,sx,sy)`, `tiles_for_bbox(bbox, zoom) -> Iterable[(z,x,y)]`, - `buffered_bbox(z,x,y, buffer) -> bbox`. Implementations: **`SlippyGrid`** - (web-mercator XYZ — the PMTiles-native grid) and **`QuadbinGrid`** (via the - `quadbin` pip package, already in `[light]` — for the COG path later). Pure - Python; fixes the heavy `rst_xyzpyramid` CRS-unit issue by being explicit about - units. The keystone reused by every tiled writer. -- **`shard.py`** — the generic **two-phase, entries-driven** orchestration: - `write()` (executor) streams tile bytes to a per-partition **indexed scratch** - (an append file + an `(tileid → offset, len)` entries index, like the heavy - writer) — *no shard assignment at write-time*. `commit()` (driver) reads only the - **entries metadata** (cheap; no tile bytes on the driver), assigns each tile to a - shard, then assembles each shard by streaming its tiles' bytes from the partition - scratch via offsets. Shard assignment is pluggable: - - **fixed** — `parent(z,x,y, shardZoom)` (default). - - **adaptive** (designed-in, near-term) — walk the quadtree, subdividing any cell - whose tile count exceeds `targetTilesPerShard` (sparse→shallow, dense→deep), - bounding per-shard size. Enabled purely by the entries-driven commit (no - re-read of bytes to count). - Tiles with `z < shardZoom` route to the **overview** shard (its own - `overview.pmtiles`). Backend-agnostic. -- **`catalog.py`** — `CatalogWriter` protocol `write(shard_entries, out_dir) -> - catalog_path`. Default impl **`STACManifestCatalog`** — a GeoJSON/STAC-style - manifest where each shard is an item with its bbox polygon, min/max zoom, and a - relative URL to the `.pmtiles` (best for spatial discovery; frontend- and - DE-friendly). Plus **`TileJSONCatalog`** (option). Designed so **`VRTCatalog`** - (GDAL virtual raster over COG shards — the DE "single Volume path", pure-Python - VRT XML from each shard's bounds/transform) and a "meta-PMTiles" catalog slot in - later. (Full STAC-spec compliance is a flagged future deepening — STAC is the - intended long-term direction.) -- **`backend.py`** — `TileArchiveBackend` protocol `assemble(sorted_tiles_iter, - header_info, out_path)`. First impl **`PMTilesBackend`** (uses the Protomaps - `pmtiles` library: `Writer(open(out,'wb')).write_tile(tileid, bytes)` + - `finalize(header, metadata)`; `pmtiles.tile.zxy_to_tileid`, `TileType`). Later - `MBTilesBackend` (sqlite), `MVTDirBackend`. (COG is the other backend shape, - added with `QuadbinGrid` + `VRTCatalog`.) -- **`_header.py`** — PMTiles `HeaderDict` assembly: `sniff_tile_type(bytes)` - (PNG/JPEG/WebP/MVT magic → `pmtiles.tile.TileType`), and bounds/center/min-max - zoom from a shard's z/x/y extent. Pure Python, unit-testable. - -### `ds/pmtiles.py` — the DataSource - -- **`PMTilesGbxDataSource`** (`DataSource`, `name()=="pmtiles_gbx"`) — `schema()` - enforces `(z:int,x:int,y:int,bytes:binary)`; `writer(schema, overwrite)` returns - the writer; `reader()` raises a clear "PMTiles is write-only here; read with the - `pmtiles` reader or `format('gdal')`" message. -- **`PMTilesGbxWriter`** (`DataSourceWriter`) — thin: reads options, drives - `tiles.shard` + `PMTilesBackend` (+ `TileJSONCatalog` when sharded). -- **Registration:** `gbx.ds.register.register(spark)` registers **all** light - Python DataSources. The shipped raster readers/writers (`raster_gbx`, - `gtiff_gbx`) are **migrated from `pyrx/ds/` into `gbx/ds/`** as a precursor step - (format names unchanged; imports/tests/docs/bench/Serverless-scan updated; - `pyrx.ds.register` → `gbx.ds.register`). So `gbx/ds/` becomes the single home for - raster, the tiles framework, and PMTiles — and the tier-neutral place future - `pyvx`/`pygx`-backed DataSources plug in. -- **Dependency:** add `pmtiles` to the **`[light]`** extra. - -## Write contract (`pmtiles_gbx`) - -- **Input schema (enforced):** `(z:int, x:int, y:int, bytes:binary)` (from - `st_asmvt`/`rst_xyzpyramid` upstream; tile-type sniffed). The writer packages + - shards + catalogs — it does **not** tile/pyramid (that stays upstream). -- **Options (named writer + knobs, sensible defaults):** - - `path` (required) — the `.save()` target. - - `mode` — `overwrite` (default, clears prior output + scratch); `append` - rejected (a finalized archive can't be appended to). - - `shardZoom` — **default `6` ⇒ SHARDED** (the real-world default): partition - tiles by their parent at `shardZoom`, emitting one `{z}/{x}/{y}.pmtiles` per - *populated* parent under `path/tileset/` + a catalog. Guidance: - `shardZoom ≈ maxZoom − 8` (Z6 suits max-zoom ~14); tune for ~100 MB–2 GB - shards. Tiles with `z < shardZoom` route to `overview.pmtiles` (see below). - - **`shardZoom = 0` ⇒ single archive** — one parent (the whole world) = one - `.pmtiles` at `path`, no catalog (the merge/simple case). - - `targetTilesPerShard` — **adaptive sharding** (designed-in, near-term; default - `None` = fixed `shardZoom`): when set, `commit()` walks the quadtree from - `shardZoom` and subdivides any cell whose tile count exceeds the target - (sparse Sahara → shallow Z4 shard, dense Manhattan → deep Z8 shard), bounding - per-shard size. Cheap because shard assignment runs off the **entries index**, - not the tile bytes (see flow). The catalog records each shard's actual zoom. - - `catalog` — `stac` (default, sharded — a GeoJSON/STAC-style manifest) | - `tilejson` | `none`. (Full STAC-spec compliance + a meta-PMTiles catalog are - future deepenings; STAC is the intended long-term direction.) - - `tileType` — default auto-sniff (PNG/JPEG/WebP/MVT; must agree within a shard); - override available. - - `tileCompression` — default none/passthrough. - - `metadata` — JSON string → archive metadata. -- **Sharded flow (default) — entries-driven:** - - `write(iterator)` (executor) appends every tile's bytes to a **per-partition - indexed scratch** (a bytes file + an `(z,x,y,tileid → offset, len)` entries - index) — *no shard assignment at write-time*. The `WriterCommitMessage` carries - only the entries-index location, not bytes. - - `commit(messages)` (driver) reads only the **entries metadata** (cheap; tile - bytes never land on the driver), assigns each tile to a shard — fixed - `grid.parent(z,x,y,shardZoom)` or adaptive `targetTilesPerShard` — and routes - `z < shardZoom` tiles to the overview. For each shard it streams that shard's - tiles' bytes from the partition scratch (via offsets), sorted by tileid, into - `PMTilesBackend.assemble`, then `STACManifestCatalog.write(...)` emits the - catalog. Shards are **bounded + non-overlapping** (disjoint tile sets). - - Because assembly is centralized in `commit()` (driver streams from scratch, - one writer per output file), no two tasks ever write the same archive — this - sidesteps the multi-worker overview-shard write race a naive design would hit. - - `abort` cleans scratch + partials. -- **Overview (`overview.pmtiles`):** v1 packages the input's `z < shardZoom` rows - into a separate `tileset/overview.pmtiles` (its own archive, browser-cached once). - Generating low zooms by **bottom-up downsample from the Z6 shards** (avoiding a - re-read of source) is deferred to the raster-pyramid phase, where resampling - semantics (incl. the 1-tile resampling buffer at shard edges) live. -- **Single-archive (`shardZoom=0`):** the same machinery with one implicit world - shard → per-partition scratch → driver sorts by tileid → `PMTilesBackend.assemble` - → one `.pmtiles`, no catalog. -- **Output layout (sharded):** - ``` - path/tileset/{z}/{x}/{y}.pmtiles # one per populated parent (Z≥shardZoom) - path/tileset/overview.pmtiles # Z>)`. Extras or missing both fail. -- **Writer `.option()`s:** `path` (required), `nameCol` (optional), `ext` - (default `"tif"`). **That is the entire writer-option surface.** -- **Encoding comes from `tile.metadata`, NOT writer options.** `GDAL_RowWriter` - calls `GDALTranslate.executeTranslate(localPath, ds, "gdal_translate", mtd)` - where `mtd` is the tile's metadata map. `OperatorOptions.appendOptions` then - reads from `mtd`: `format` (default `GTiff`), `compression` (default - `DEFLATE`), `blocksize` (default `512`, floored to mult-of-16, clamped ≥64 and - ≤min(w,h)), `zlevel` (DEFLATE, default `6`), `zstd_level` (ZSTD, default `9`), - plus a `PREDICTOR` chosen by dtype (3 for float, else 2). So the on-disk - format/compression are a property of the tile, set when it was read/produced. -- **Always re-encodes via `gdal_translate`** (never writes `tile.raster` - verbatim), then stamps `RASTERX_` for each metadata entry + `RASTERX_CELL` - = cellid, then `FlushCache`. -- **Filenames:** `nameCol` set → `{row[nameCol]}.{ext}`; else - `{MurmurHash3(tile)}_{pid}_{tid}.{ext}`. Flat directory under `path`, one file - per row. (`nameCol` must be an existing column — in practice overwrite - `source`, since the schema is fixed at 2 columns.) -- **Mode/commit:** append-only; writes directly to `path` via Hadoop copy from a - per-row local temp; batch `commit`/`abort` are no-ops; partial writes remain on - failure. No staging dir. -- **Format strings:** `gdal` + `gtiff_gdal` (same for read and write); - `gtiff_gdal` = `gdal` + `dsExtraMap(driver="GTiff")`. - -## Architecture & components - -Rework `pyrx/ds/writer.py` into a full writer and wire it to **both** light -formats (mirroring the heavy `gdal`/`gtiff_gdal` pair): - -- **`raster_gbx`** (`RasterGbxDataSource`, catch-all) gains `writer()` → the - catch-all writer; output driver derives from `tile.metadata` (default GTiff). -- **`gtiff_gbx`** (`GTiffGbxDataSource`) `writer()` presets `driver="GTiff"` (the - `dsExtraMap` mirror — same pattern as the reader). For the writer this means - GTiff is the assumed/forced output driver. -- **`pyrx/ds/writer.py`** — `RasterGbxWriter(DataSourceWriter)` + a picklable - `RasterCommitMessage`. Thin: schema check, filename derivation, mode handling, - delegating per-tile bytes to the helper. -- **`pyrx/ds/_write.py`** (new) — pure-Python per-tile byte production - (`tile_to_bytes(...)`): the hybrid verbatim/re-encode logic + `RASTERX_*` - stamping. Unit-testable without Spark (parallels `_encode.py` on the read side). - -## Write contract (light) - -- **Schema:** exact `(source, tile)` enforced up front — extras *or* missing both - fail (`assert_write_schema`, already present, kept). -- **Writer options:** `path` (req), `nameCol` (optional), `ext` (default `"tif"`) - — matching heavy. No format/compression writer options. -- **Output encoding from `tile.metadata`:** driver from `metadata["driver"]`/ - `metadata["format"]` (default `GTiff`); `compression`, `blocksize`, `zlevel`, - `zstd_level` read from metadata with the same defaults as heavy. `gtiff_gbx` - forces driver GTiff regardless. -- **Hybrid bytes (`_write.tile_to_bytes`):** - - Target driver **GTiff** (dominant case — `raster_gbx`/`gtiff_gbx` tiles are - already GTiff) → write `tile.raster` **verbatim**. Pixel-identical to heavy; - heavy's specific creation-options (TILED/BLOCKSIZE/PREDICTOR/ZLEVEL) differ, - but our contract is decoded-pixel parity, which holds. - - Target driver **non-GTiff** (e.g. `COG`, `PNG`) → **rasterio re-encode**: - decode `tile.raster`, write via the target driver applying the - metadata-derived `compress`/`blocksize`/`zlevel`/`zstd_level`, and stamp - `RASTERX_` (from `tile.metadata`) + `RASTERX_CELL` (from `cellid`) via - `dataset.update_tags`. - - **Verify-during-impl:** confirm rasterio's GTiff/COG creation-option names - (`COMPRESS`, `ZLEVEL`, `PREDICTOR`, `BLOCKXSIZE/BLOCKYSIZE`, `BIGTIFF`) match - what `appendOptions` emits, so re-encoded output is faithful. -- **Filenames:** `nameCol` set → `{row[nameCol]}.{ext}`; else an opaque unique - name `{content-hash}_{uuid}.{ext}`. **Verify-during-impl:** whether PySpark's - `DataSourceWriter` exposes a partition id (Scala uses `pid_tid`); if not, the - uuid keeps names collision-free across partitions. Documented as *not* - byte-identical to heavy's `MurmurHash3_pid_tid` — use `nameCol` for control. -- **Mode/commit:** flat dir, no staging. `append` adds files; `overwrite` clears - the output dir on the driver (in `writer()`/`__init__`, runs once) before tasks. - `commit` no-op; `abort` best-effort removes this run's files. (Heavy is - append-only no-op; light adds clean `overwrite` handling.) - -## Testing (TDD — tests are the contract) - -Unit (`_write.tile_to_bytes`, no Spark) + integration (local Spark, `ds/` -fixtures): - -- **Verbatim (GTiff):** `raster_gbx` read → `gtiff_gbx` write, no options → - output bytes **byte-identical** to input `tile.raster`; round-trip pixels equal. -- **Re-encode (non-GTiff):** a tile whose `metadata["driver"]="COG"` (or `format`) - → output bytes differ, decode to **same pixels** (within tol), and carry - `RASTERX_CELL` + `RASTERX_` tags. -- **`nameCol`:** `withColumn("source", …)` + `option("nameCol","source")` → - filenames are the column values (`{name}.{ext}`). -- **`ext`:** `option("ext","tiff")` → suffix honored. -- **Strict schema:** extra/missing column fails (kept). -- **Catch-all vs named:** `raster_gbx` writer honors `tile.metadata` driver; - `gtiff_gbx` forces GTiff. -- **Mode:** `overwrite` replaces (no stale accumulation — write-twice test); - `append` adds. -- **Serverless guard:** `_write.py` auto-covered by the path-based scan; extend - the explicit file list. -- **Light-vs-heavy round-trip parity (Docker/integration, skip-if-heavy- - unavailable):** `gtiff_gbx` write → re-read decodes to the same pixels as the - heavy `gtiff_gdal` write path. - -## Documentation (both tiers — closes the flagged gaps) - -Per the repo convention, **doc-tests are the documentation source** (code in -`docs/tests/python/...`, imported into `.mdx` via raw-loader; run in Docker). - -**Light tier (net-new):** -- Doc-tests `docs/tests/python/readers/raster_gbx_examples.py` and - `docs/tests/python/writers/raster_gbx_examples.py` — real reads/writes/round- - trips on sample data, both `raster_gbx` and `gtiff_gbx`, all options. -- `.mdx` pages `docs/docs/readers/raster_gbx.mdx` + `docs/docs/writers/raster_gbx.mdx` - importing that code; add both to `readers/overview.mdx` and `writers/overview.mdx`. -- Update `docs/docs/api/execution-tiers.mdx` — the line "native lightweight Python - Data Source readers are not yet available" is now stale; document - `raster_gbx`/`gtiff_gbx` read+write as available in the light tier. - -**Heavy tier (option audit + fill):** -- **Readers** (`readers/gdal.mdx`, `readers/gtiff.mdx`): ensure every option is - documented — `path`, `sizeInMB` (default 16, the tiling threshold), `filterRegex` - (default `.*`, recursive listing), `driver`. Verify the options table is complete. -- **Writers** (`writers/gdal.mdx`): document that `compression`, `blocksize`, - `zlevel`, `zstd_level`, `format`/`driver` are read from **`tile.metadata`** (not - writer options), with their defaults and how to influence them (upstream - transforms / `RST_AsFormat`), and add an explicit **`gtiff_gdal` writer** - example (today `gtiff_gdal` appears only as a read-back format). Keep the - existing accurate "driver comes from the tile" framing. - -## Performance (measured 2026-06-11, cluster, 1000 tiles, run `rw2-20260611`) - -| op | light | heavy | verdict | -|---|---|---|---| -| read | 20.43 s | 8.63 s | light ~2.4× slower (Python→JVM byte transfer is the wall; see reader spec) | -| **write** | **2.53 s** | 4.12 s | **light ~1.6× faster** | - -The **writer is a win for the light tier** — the inverse of the reader. The light -writer passes whole-file GTiff bytes through **verbatim**; the heavy writer -re-encodes every tile via `gdal_translate`, so it pays the ~per-tile encode cost -the light path skips. (Write timing reads the corpus once into a cached DataFrame, -then times only `.write…save()`, isolating write cost.) Heavy write must use -`mode("append")` — the GDAL writer is append-only and rejects `overwrite` -(`UNSUPPORTED_FEATURE` truncate). Writer bench: `bench.readers.run_format_write`, -wired into the cluster reader cell. - -## Out of scope (this spec) - -- Vector readers/writers (`vector_gbx`, `*_ogr`). -- Byte-identical filename parity with heavy's `MurmurHash3_pid_tid` (use - `nameCol` for controlled names). -- New write **modes** beyond `append`/`overwrite`. - -## Verify-during-impl checklist - -1. rasterio creation-option names vs `OperatorOptions.appendOptions` output - (`COMPRESS`, `PREDICTOR`, `ZLEVEL`, `ZSTD_LEVEL`, `BLOCKXSIZE/Y`, `BIGTIFF`, - `TILED`, COG `BLOCKSIZE`). -2. Whether PySpark `DataSourceWriter` exposes a partition id (else uuid uniquifier). -3. `tile.metadata` key used for driver: `"driver"` vs `"format"` (heavy reads - `format`; the reader emits both = "GTiff"). Honor `driver` then `format`. -4. rasterio `update_tags` lands `RASTERX_*` such that a re-read recovers them - (parity with heavy's `SetMetadataItem`). -5. Confirm `docs/docs/writers/gdal.mdx` "ext does not change format" framing stays - correct after documenting the tile-metadata encoding keys. diff --git a/docs/superpowers/specs/2026-06-11-light-readers-raster-design.md b/docs/superpowers/specs/2026-06-11-light-readers-raster-design.md deleted file mode 100644 index ad87caf96..000000000 --- a/docs/superpowers/specs/2026-06-11-light-readers-raster-design.md +++ /dev/null @@ -1,296 +0,0 @@ -# Light Readers — Raster (Python DataSource V2) Design - -**Date:** 2026-06-11 -**Branch:** `light-readers` -**Status:** Approved (design), revised post-recon; ready for implementation plan - -## Revision 2026-06-11 (post-recon) — parity contract corrected - -Reading the Scala `gdal` reader directly (`GDAL_Reader.scala:12-45`, -`RasterSerializationUtil.tileToRow`, `WindowedExtract.scala:108-119`, -`RasterDriver.writeToBytes`) overturned three assumptions baked into the -original parity contract below. The contract has been corrected throughout this -doc; this note records the change for review: - -1. **`tile.raster` is NOT raw file bytes — it is a re-encoded GTiff (DEFLATE) - tile.** The heavy reader splits each source raster into tiles and writes each - tile out via `RasterDriver.writeToBytes`, which coerces to GTiff/DEFLATE - regardless of source format. Two independent GDAL stacks (JVM bindings vs - rasterio's bundled libgdal) cannot produce byte-identical GTiffs, so - **byte-for-byte `tile.raster` equality is infeasible** and is replaced by - **decoded pixel-array parity within tolerance** (the model `bench/compare.py` - already uses: `REL_TOL`/`ABS_TOL = 1e-3`). -2. **One input file yields one row PER TILE, not one row per file.** The reader - splits via `BalancedSubdivision` (power-of-4 split sized by the `sizeInMB` - option, default 16). A sub-16MB raster produces exactly one tile (one row); - larger rasters produce N tiles. `cellid` is the literal **`-1L`**, not 0. -3. **`metadata` is an 11-key map**, not driver/width/height/count: - `path, sourcePath, driver, format, last_command, last_error, all_parents, - size, compression, isZipped, isSubset`. - -Also: reader options are `path` / `sizeInMB` (default `16`) / `filterRegex` -(default `.*`, recursive **regex** listing — not glob); the heavy reader is -**fail-fast with no `ignoreCorruptFiles`**, so the previously-proposed -`ignoreCorruptFiles` option is dropped to match heavy exactly. - -## Summary - -Provide pure-Python / PySpark raster readers and a writer that are **1:1 -swap-outs** for the current GDAL-backed Scala readers, built on the **Spark -DataSource V2** API exposed in PySpark 4.0 (`pyspark.sql.datasource.DataSource`). -This extends the lightweight `pyrx` family (currently function-only) with I/O, -so a user can change a single format string and get the same result through a -pure-Python path. - -Raster is the focus of this branch. Vector (`vector_gbx` + named readers, via -pyogrio) and `pygx` are explicitly out of scope and get their own spec; they are -named here only to validate the naming convention. - -### Naming convention (validated here, applied to vector later) - -| Tier | Catch-all raster | Named GeoTIFF | Catch-all vector | Named vector | -|---|---|---|---|---| -| Heavy (Scala, existing) | `gdal` | `gtiff_gdal` | `ogr` | `shapefile_ogr`, `geojson_ogr`, `gpkg_ogr`, `file_gdb_ogr` | -| Light (this work / future) | `raster_gbx` | `gtiff_gbx` | `vector_gbx` *(future)* | `shapefile_gbx`, `geojson_gbx`, `gpkg_gbx`, `file_gdb_gbx` *(future)* | - -The light tier uses a `*_gbx` form (catch-all `_gbx`, named -`_gbx`) rather than the heavy tier's `_gdal` / `_ogr` engine suffix. The -new names do not collide with the Scala-registered formats, so both tiers -coexist and a swap is a one-line `format(...)` change. - -## Why Python DataSource V2 (not a UDF transform) - -The reader is mandated to use DataSource V2. It is also the right choice for -Serverless: a Python DataSource is pure Python, with no `_jvm` / `sparkContext` -/ `.rdd` access, so it respects the Serverless constraint that the `pyrx` -product must never set Spark config or reach into the JVM. A `binaryFile + -pandas_udf` transform was rejected because it provides no -`spark.read.format("raster_gbx")` surface and therefore is not a reader; its -file-listing/glob semantics are still a useful reference for the driver-side -partition planning below. - -## Architecture - -New subpackage `python/geobrix/src/databricks/labs/gbx/pyrx/ds/`, mirroring the -Scala `rasterx/ds/` tree: - -- **`_base.py`** — shared scaffolding: `DataSourceReader` / `InputPartition` / - `PartitionReader` plumbing, path expansion (glob + recursive dir listing), - the `(source, tile)` schema sourced from `pyrx._serde.TILE_SCHEMA`, and the - rasterio open → metadata extraction → raw-bytes read. -- **`raster.py`** — `RasterGbxDataSource`, format `raster_gbx` (catch-all, - generic over any rasterio-readable driver). -- **`gtiff.py`** — `GTiffGbxDataSource`, format `gtiff_gbx`; extends the - catch-all and presets `driver="GTiff"` via an options-injection hook that - mirrors the Scala `dsExtraMap` pattern (`DataSourceExtras`). -- **`writer.py`** — `RasterGbxWriter`, the DSv2 write path; enforces the exact - `(source, tile)` schema like the GDAL writer. Write format: `gtiff_gbx`. -- **`register.py`** — `register(spark)` that calls `spark.dataSource.register(...)` - for each source, mirroring `functions.register` (the explicit, documented entry - point — call surface `pyrx.ds.register.register(spark)`). Also attempted - opportunistically on `pyrx.ds` import, guarded for no-active-session. (Not on - bare `pyrx` import — `pyrx/__init__` does not import `ds`, to avoid a circular - import during package init.) - -### Named-reader pattern (`dsExtraMap` mirror) - -The named GeoTIFF reader extends the catch-all and injects driver presets -through an options hook, exactly as the Scala `GTiff_DataSource` extends -`GDAL_DataSource` and overrides `dsExtraMap` to inject `driver -> "GTiff"`. The -catch-all stays clean and generic; named readers add presets only. - -## Parity contract (the swap-out guarantee) - -Resolved against the Scala `gdal` reader source (see Revision note); asserted by -a dedicated parity test. - -- **Schema:** identical `StructType` to the Scala reader — `source: string` + - `tile: struct{cellid: long, raster: binary, metadata: map}` — - sourced from `pyrx._serde.TILE_SCHEMA` (one definition, not a copy). -- **`tile.raster`:** a **re-encoded GTiff (DEFLATE) tile**, matching the heavy - reader's `RasterDriver.writeToBytes` behavior (always GTiff on the wire, - regardless of source format). Written via `rasterio` to an in-memory GTiff - with `compress="deflate"`. **Not byte-identical to heavy** (independent GDAL - stacks) — parity is asserted on the **decoded pixel array**, not the bytes. -- **Row cardinality + `cellid`:** one row **per tile**. The reader splits each - source raster into tiles using a port of `BalancedSubdivision`'s power-of-4 - split sized by `sizeInMB` (default 16): a sub-`sizeInMB` raster → 1 tile/row; - larger → N tiles/rows. Every emitted tile carries `cellid = -1` (the heavy - literal `-1L`). -- **`metadata`:** **key-set parity is enforced** over the 11 heavy keys — - `path, sourcePath, driver, format, last_command, last_error, all_parents, - size, compression, isZipped, isSubset`. Values are allowed to differ where - GDAL-Java and rasterio legitimately disagree (e.g. the `path` in-memory URI is - implementation-specific; `driver`/`format` = `"GTiff"`, `compression` = - `"DEFLATE"`, `isZipped`/`isSubset` = `"false"` are fixed). Light-vs-heavy - comparison testing surfaces any value gotchas. -- **`source`:** the resolved file path string from the recursive listing, - matching the heavy reader (`partition.filePath`). - -The parity test reads the same sample file through both `gdal` and `raster_gbx` -and asserts: schema equality, equal tile/row count, `cellid == -1` on every row, -metadata **key-set** equality, and **decoded pixel-array equality within -tolerance** (`REL_TOL`/`ABS_TOL = 1e-3`, decoding both tiers' `tile.raster` with -rasterio). That test is the operational definition of "1:1 swapout." - -## Distribution model - -The swap-viability crux: distribution must hold as well as the heavy reader's. - -- **`reader.partitions()` (driver):** recursively lists files under the input - `path` matching `filterRegex` (default `.*`), then emits **one - `InputPartition` per file** (carrying the file path + `sizeInMB`), so partition - count scales with the corpus and Spark spreads `read()` tasks across executors. - This mirrors the Scala reader's one-partition-per-file parallelism - (`GDAL_Batch.planInputPartitions`). -- **`read(partition)` (executor):** a Python task that opens the file with - rasterio, splits it into tiles (BalancedSubdivision port), and for each tile - windowed-reads + re-encodes a GTiff and yields a row. Reads are local to the - task; cross-file parallelism comes from partitioning, tile fan-out happens - within the task (same as heavy, where tiling is inside `GDAL_Reader`). -- Pure Python throughout — no `_jvm` / `sparkContext` / `.rdd` — so it holds on - Serverless. -- A **distribution smoke test** asserts partition count tracks file count and - that work spreads across more than one task (no accidental coalesce-to-1). - -## Writer - -DSv2 write lifecycle: - -- Enforce the exact `(source, tile)` schema up front — extras *or* missing - columns both fail (matches the GDAL writer's strict schema contract). -- `write(iterator)` runs on executors: for each row, write `tile.raster` bytes - to the resolved path (GeoTIFF for `gtiff_gbx`). -- `abort` cleans up partial outputs on task failure; `commit` finalizes. - -## Error handling - -- Path expansion failures (no match, unreadable dir) raise at `partitions()` on - the driver with a clear message — fail fast before launching tasks. -- Per-file open failures in `read(partition)`: **fail-fast** (surface the - offending path), matching the heavy reader, which propagates the - `RasterDriver.read` exception with no `ignoreCorruptFiles` escape hatch. No - such option is added (would diverge from heavy). -- rasterio resource hygiene: every `DatasetReader` / `MemoryFile` opened in a - `with` block so handles close per-file (the Python analogue of the - `releaseDataset` try/finally discipline). - -## Performance validation - -Extend the bench harness with a new reader mode, reusing `store` / `results` / -`compare` plumbing for one reporting surface. Two timing surfaces mirror the -function bench: - -- **pure-local:** single-file open + metadata + bytes; light (rasterio) vs heavy - (GDAL via the existing heavy path). Per-op cost, no Spark. -- **cluster spark-path:** N files distributed via `raster_gbx` vs `gdal`, - measuring wall-clock **and** parallel efficiency (per-partition timing - spread), per the parallel-efficiency method already used for functions. - -Scale policy: spark-path at the standard row/tile scale (1000), pure-local at 1 -file. Light meaningfully slower than heavy is a **deprecation blocker** — the -reader bench produces the same light-vs-heavy ratio the function bench does, and -emits a per-run summary link. - -### Cluster result (2026-06-11, run_id `readers-20260611`, 1000 tiles) - -Both readers timed on the cluster through the unified Delta/compare plumbing -(`fn="raster_read"`, `mode="spark-path"`): - -| reader | iter (1000 tiles) | per-tile | -|---|---|---| -| heavy `gdal` | 8.98 s | 0.00898 s | -| light `raster_gbx` | 24.13 s | 0.02413 s | - -**Light is ~2.7× slower than heavy `gdal`** (speedup 0.37). The heavy `gdal` -reader does **not** produce tiles in the local dev container (GDAL-init quirk) -but works on-cluster — hence the live comparison runs there. -Summary: `…/bench-out/readers-20260611/summary.md`. - -#### Optimization investigation (2026-06-11) - -1. **Profile (per whole-file tile):** the GTiff/DEFLATE **re-encode is ~95%** of - per-tile reader cost (52 ms of 55 ms; native GDAL, not Python). open/decode/ - transfer are negligible locally; raw file read is 0.7 ms. -2. **Whole-file pass-through** (committed): when a source splits to a single - whole-raster tile and is already GTiff (99.6% of the corpus), emit the - original bytes — **~48× faster per tile locally** (1.1 ms vs 55 ms), - parity-safe (decoded-pixel), and preserves colormaps/masks. -3. **Re-bench at scale (run `readers-passthru-20260611`):** light 24.13 → **20.33 s** - (heavy 8.27 s) — only **~16%** faster despite pass-through firing for 99.6% of - tiles. The at-scale bottleneck is the DataSource V2 Python→JVM binary transfer - (~12 ms/tile, [[pyrx-udf-boundary-tax]] in reader form), not the re-encode. - Pass-through is **kept** (free CPU win + colormap/mask fidelity). -4. **Arrow batch output — tried and reverted (neutral/negative).** Yielded - `pyarrow.RecordBatch` from `read()` instead of tuples, with a - `maxFilesPerPartition` knob to grow batches: - - `maxFilesPerPartition=16` (run `readers-arrow-20260611`): **28.20 s — worse.** - The cluster is 20× `rd-fleet.xlarge` (~80 cores); batching dropped 1000→~63 - partitions and starved the cores. Parallelism loss > any Arrow gain. - - `maxFilesPerPartition=1` (run `readers-arrow1-20260611`): **19.93 s** ≈ the - 20.33 s tuple path (within run-to-run noise; heavy swung 7.75–8.98 s across - runs). Arrow output itself is **neutral**. - - **Structural conclusion:** on an ~80-core cluster you can't get large Arrow - batches *and* adequate parallelism from 1000 files (good utilization needs - ~160–320 partitions → only ~3–6 files/partition → tiny batches). The residual - ~2.5× gap is the inherent cost of a Python reader handing ~4 GB of tile bytes - across the Python→JVM boundary (heavy stays in-JVM). Arrow was reverted to - keep the simpler tuple path. **Best light result: pass-through, ~20 s (~2.5× - heavy).** Further wins would need a fundamentally different transfer (e.g. a - JVM-side reader), out of scope for the pure-Python tier. - -## Testing (TDD — tests are the contract) - -- **Parity test:** same sample file through `gdal` and `raster_gbx` → schema eq, - byte-eq `tile.raster`, metadata key-set eq. -- **Round-trip test:** `raster_gbx` read → `gtiff_gbx` write → re-read; assert - tile integrity. -- **Distribution smoke test:** partition count tracks file count, work spreads - >1 task. -- **Named-reader test:** `gtiff_gbx` injects `driver="GTiff"` and reads a - GeoTIFF identically to the catch-all with explicit driver. -- **Serverless guard:** extend the existing no-Spark-config test to the new DS - modules. -- **Registration test:** `register(spark)` makes all formats resolvable via - `spark.read.format(...)`. -- Real sample data from `/Volumes/main/geobrix_samples/...`; doc tests run in - Docker per the doc-test convention. No mocking of Spark / rasterio / file I/O. - -## Packaging - -No new runtime dependency: rasterio is already in the `[pyrx]` extra, and Python -DataSource V2 ships in PySpark 4.0. pyogrio stays out (deferred to the vector -spec). - -## Out of scope (this branch) - -- Vector readers (`vector_gbx`, `shapefile_gbx`, `geojson_gbx`, `gpkg_gbx`, - `file_gdb_gbx`) via pyogrio — own spec. -- `pygx` grid I/O. -- These are named only to validate the `*_gbx` naming convention. - -### Known limitations (follow-up) - -- **Colormaps / per-band masks not propagated.** The tile re-encode carries band - data + nodata/dtype/crs/transform, but not source colormaps or per-band - masks/alpha. Sources relying on those will differ structurally from the heavy - reader. Tracked for a follow-up; the catch-all otherwise handles any - rasterio-readable driver. -- **On-disk size keys the split.** Tile count is derived from `os.path.getsize` - (matching heavy's `Files.size` for on-disk sources). Multi-file/VRT or remote - sources, where the heavy `memSize` differs from a single file size, may tile - differently — out of scope for the on-disk corpus this targets. - -## Verify-during-design checklist — RESOLVED (2026-06-11) - -1. **`cellid` literal** → `-1L` (`GDAL_Reader.scala:30`). Emit `-1`. -2. **`metadata` key set** → the 11 keys listed above - (`WindowedExtract.scala:108-119`). -3. **`source` path convention** → the listed file path, `partition.filePath` - (`GDAL_Reader.scala:34`). -4. **Corrupt-file behavior** → fail-fast, no option (`GDAL_Reader.scala:17`). -5. **`tile.raster` encoding** → re-encoded GTiff/DEFLATE - (`RasterDriver.writeToBytes`), not raw bytes — drove the byte→pixel parity - change. -6. **Tiling** → `BalancedSubdivision.splitRasterIter` power-of-4 split by - `sizeInMB` (default 16); port `getTileSize`/split-count for row-count parity. diff --git a/docs/superpowers/specs/2026-06-12-light-vector-readers-design.md b/docs/superpowers/specs/2026-06-12-light-vector-readers-design.md deleted file mode 100644 index 7f7f05e4c..000000000 --- a/docs/superpowers/specs/2026-06-12-light-vector-readers-design.md +++ /dev/null @@ -1,179 +0,0 @@ -# Light Vector Readers (`*_gbx`, pyogrio) — heavy-parity Design - -**Date:** 2026-06-12 -**Branch:** `light-readers` -**Status:** Approved (design); ready for implementation plan - -## Summary - -Bring the **lightweight** (pure-Python/PySpark, JAR-free, Serverless-safe) reader -tier to **parity** with the heavyweight tier for **vector** data. Heavy has five -OGR-backed vector readers; light has none. Build the five light equivalents as -**pyogrio**-backed PySpark DataSource V2 readers that emit the **exact same schema** -as the heavy readers, so swapping tiers is a one-line `format(...)` change (the same -guarantee the raster tier already provides). - -**Readers only.** The heavy tier has **no** vector *writers* in 0.4.0, so vector -write-parity is a no-op — out of scope. - -## Scope — the five readers - -| Light format (`_gbx`) | Heavy equivalent | OGR driver | -|---|---|---| -| `ogr_gbx` | `ogr` | auto-detect (or `driverName`) | -| `shapefile_gbx` | `shapefile_ogr` | `ESRI Shapefile` | -| `geojson_gbx` | `geojson_ogr` | `GeoJSON` / `GeoJSONSeq` | -| `gpkg_gbx` | `gpkg_ogr` | `GPKG` | -| `file_gdb_gbx` | `file_gdb_ogr` | `OpenFileGDB` | - -`ogr_gbx` is the generic core; the four named readers are **thin presets** that set -`driverName` (mirroring how `gtiff_gbx` presets the raster driver over `raster_gbx`). - -## Output schema — exact heavy parity - -Match the heavy OGR reader's schema column-for-column so downstream code is -identical (verified by a light-vs-heavy parity test): - -- Per geometry field `j`: - - `geom_j` — `binary` (WKB) when `asWKB=true` (default); `string` (WKT) when `asWKB=false`. - - `geom_j_srid` — `string` (authority code, e.g. `"4326"`; empty if unknown). - - `geom_j_srid_proj` — `string` (PROJ4 definition; empty if unavailable). -- One column per OGR attribute field, named as in the source, with Spark types - matching heavy's inference (integer/long, double, string, boolean, date/timestamp). -- **v1 supports a single geometry field (`geom_0`)** — the common case. Multi-geometry - sources (`geom_1`, `geom_2`, …) are a documented follow-up (see Out of scope). - -Implementation: `pyogrio.read_info(path, layer=…)` provides fields + geometry type -+ CRS for `schema()` without a full read; `pyogrio.read_dataframe` / `read_arrow` -provides rows; `shapely.to_wkb` encodes geometry; the CRS yields the SRID -(authority code) and PROJ4 string. Spark types are mapped from the OGR/arrow field -types to match heavy's column types — the **parity test gates any divergence**. - -## Options — heavy parity - -| Option | Default | Behavior | -|---|---|---| -| `driverName` | auto (from extension) | Explicit OGR driver (named readers preset it). | -| `asWKB` | `"true"` | Geometry as WKB binary (`true`) or WKT string (`false`). | -| `layerN` | `"0"` | Layer index for multi-layer sources. | -| `layerName` | `""` | Layer name (overrides `layerN`). | -| `chunkSize` | `"10000"` | Features per partition (parallel read). | - -## Parallelism / partitioning - -Partition a source into `chunkSize`-feature slices using pyogrio's `skip_features` -+ `max_features`, so large files read across Spark tasks (mirrors the heavy reader's -chunked read). Each `InputPartition` reads its `(skip, count)` slice via pyogrio and -yields rows. Layer count comes from `read_info`. This follows the raster reader's -"one slice per partition, picklable partition object, no Spark/JVM refs in `read()`" -pattern (Serverless-safe). - -Zipped sources in the corpus (`.shp.zip`, `.gdb.zip`) are read via OGR's -`/vsizip/…` virtual path (pyogrio supports it), matching heavy's behavior. - -## Architecture / files - -One new module `python/geobrix/src/databricks/labs/gbx/ds/vector.py` holds all five -readers + helpers (the four named readers are ~3-line presets, so a separate module -isn't warranted): -- `OgrGbxDataSource` (`name()`→`ogr_gbx`, write-less; `reader()`), - `OgrGbxReader(DataSourceReader)` (schema from `read_info`, `partitions()` = - chunk slices, `read(partition)` = pyogrio slice → WKB rows). -- `_vector_schema(...)` — the heavy-parity schema builder — plus inline helpers for - CRS→(srid, proj4) and arrow/OGR→Spark type mapping. -- `ShapefileGbxDataSource`, `GeoJSONGbxDataSource`, `GpkgGbxDataSource`, - `FileGdbGbxDataSource` — subclasses of `OgrGbxDataSource` overriding `name()` + - presetting `driverName` (and `multi=true` for GeoJSONSeq where heavy does). -- `register.py` — add the five sources to `_SOURCES`. - -Pure-Python; Serverless-safe (no `_jvm`/`.conf.set`/`.rdd`/`sparkContext`). The -existing Serverless guard (`test/pyrx/test_serverless_no_spark_config.py`, scans -`gbx.ds`) covers the new modules — extend its explicit file-list assertion. - -## Dependency - -Add `pyogrio` to: -- `pyproject.toml` `[light]` extra — as a **range** (e.g. `pyogrio>=0.8,<1`). -- the hash-pinned locks `requirements-pyrx-ci.txt` (light CI) and - `requirements-dev-container.txt` (doc-tests) — **pinned + hashed**, regenerated via - `uv pip compile --generate-hashes --index-url ` (same pattern as - `pmtiles`). pyogrio vendors its own `libgdal`; pin a version consistent with the - rasterio already in the stack. - -## Docs (slots into the tabbed structure) - -The five vector reader pages (`readers/ogr`, `readers/shapefile`, `readers/geojson`, -`readers/geopackage`, `readers/filegdb`) currently are heavyweight-only with a -`:::note No lightweight equivalent yet` admonition. For each: -- Add a **Lightweight tab** (first/default) documenting the `*_gbx` reader, using the - same `` convention; move the existing - heavy body into a **Heavyweight tab**. -- **Remove** the "no lightweight equivalent yet" note. -- New doc-test example file per reader under `docs/tests/python/readers/` (e.g. - `shapefile_gbx_examples.py`) exercising a real read against the corpus, imported - via raw-loader. Tests execute real reads with assertions (doc-tests-are-the-source). - -## Benchmark (per the bench-each-reader/writer requirement) - -For each of the five readers, a **light-vs-heavy** comparison (timing + parity), -mirroring the PMTiles writer bench: -- Reuse `bench/readers.py::run_format_read` (light `*_gbx` vs heavy `*_ogr`) for timing. -- Add a **parity** assertion: both tiers read the same source to the same row count - and equivalent geometry/attribute values (compare WKB-decoded geometries + - attribute columns). -- Wire a `--benchmark-vector` knob in `bench/cluster.py` + the launcher (a - `_CELL_VECTOR` cell), mirroring `--benchmark-readers` / `--benchmark-pmtiles`. -- Add a "Results — vector readers" section to `docs/docs/api/benchmarking.mdx`. - -## Testing (TDD) - -- **Schema-parity units** (local, no Spark needed for the schema builder): assert - `_vector_schema` produces the exact heavy column set/types for each corpus format. -- **Read round-trip** (local Spark): `ogr_gbx` + each named reader over the corpus — - correct row counts, valid WKB (round-trips through `shapely.from_wkb`), SRID/PROJ4 - populated, attribute columns present with expected types; `asWKB=false` yields WKT; - `layerN`/`layerName` select layers; `chunkSize` splits into multiple partitions and - the union equals the whole file. -- **Light-vs-heavy parity** (Docker/integration, skip-if-heavy-unavailable): same - source → light `*_gbx` vs heavy `*_ogr` produce identical schema + row set + - decoded geometries. This is the parity gate. -- **Serverless guard**: the new `vector.py` (+ named module) are in the scan and - contain no forbidden Spark-config/JVM calls. - -## Phasing - -This spec is **Phase 1 (vector readers)** of a three-phase track: -1. **Vector readers** (this spec) — its benchmark is *initial* (the limited sample-data - formats already on the Volume). -2. **Vector writers** (follow-on spec) — net-new light capability (heavy has none); - pyogrio can write, so writers double as a **corpus generator** for benchmarking - (write vector at scale / varied formats, read back). -3. **Scaled/final benchmarking** — light-vs-heavy reader numbers over writer-generated - data at scale. - -## Out of scope (this spec — see Phasing) - -- **Vector writers** — deferred to Phase 2 (data generation + net-new capability), not - part of this readers spec. Heavy has no vector writer to match anyway. -- **Multi-geometry-field** sources (`geom_1`, …) — v1 is single-geom parity. -- **GridX** readers/writers — separate tier, not part of the readers/writers section. -- A `pyvx` *functions* API (vector transforms) — this spec is DataSource readers only. - -## Verify-during-impl checklist - -1. `_vector_schema` matches the heavy OGR schema EXACTLY for each corpus format - (names + Spark types) — diff against a heavy read in the parity test; fix the - OGR/arrow→Spark type map until it matches (int width, date/timestamp, bool). -2. CRS→`srid` is the authority code STRING (e.g. `"4326"`), not an int; `proj4` - empty-string when unavailable (match heavy's nullability/empties). -3. `chunkSize` partitioning: `skip_features`/`max_features` slices are exhaustive + - non-overlapping; union row count == `read_info` feature count. -4. Zipped shapefile/filegdb (`.shp.zip`/`.gdb.zip`) read via `/vsizip/`; named - readers handle the corpus's zipped paths like heavy. -5. `asWKB=false` (WKT) path produces `geom_j` as `string`. -6. Serverless-safe: pyogrio/shapely calls only; no `_jvm`/`.conf.set`/`.rdd`. -7. Docs: each vector page's lightweight tab renders + its doc-test reads the corpus; - the "no lightweight equivalent yet" note is removed; tabs are lightweight-first. -8. Bench parity cell asserts light==heavy decoded rows; benchmarking.mdx section added. -9. pyogrio in `[light]` (range) + both locks (pinned+hashed); light CI + dev image - install it. diff --git a/docs/superpowers/specs/2026-06-12-light-vector-writers-design.md b/docs/superpowers/specs/2026-06-12-light-vector-writers-design.md deleted file mode 100644 index e2bca909c..000000000 --- a/docs/superpowers/specs/2026-06-12-light-vector-writers-design.md +++ /dev/null @@ -1,142 +0,0 @@ -# Light Vector Writers (`*_gbx`, pyogrio) — Phase 2 Design - -**Date:** 2026-06-12 -**Branch:** `light-readers` -**Status:** Approved (design); ready for implementation plan - -## Summary - -Phase 2 of the light vector track: **net-new** vector writers (the heavy tier has -none) as pyogrio-backed PySpark DataSource V2 writers, format names `*_gbx`. Input -is the **light vector reader's schema** (`attributes…, geom_0` WKB, `geom_0_srid`, -`geom_0_srid_proj`), so `read(ogr_gbx) → write(*_gbx)` round-trips. They also serve -as the **benchmark-corpus generator** for Phase 3 (write vector at scale / varied -formats, read back). - -Five writers, mirroring the readers: `ogr_gbx` (generic) + `shapefile_gbx`, -`geojson_gbx`, `gpkg_gbx`, `file_gdb_gbx`. - -## Output model — two-phase merge to one file - -Vector formats are single-file datasets, but DataSource V2 `write()` runs per -partition. So (mirroring the PMTiles writer): -- **`write()`** (executor): build a pyarrow table from the partition's rows - (attribute columns + the WKB geometry column) and `pyogrio.write_arrow(table, - fragment_path, driver=…, geometry_name=…, geometry_type=…, crs=…)` — one - **scratch fragment** per partition. -- **`commit()`** (driver): merge the fragments into ONE output file — - `pyogrio.write_arrow(..., append=True)` appending each fragment's table in turn - (or write the first, append the rest). Then clean scratch. -- **Shared filesystem is required** for the driver to read executor fragments — - the lesson from the PMTiles writer. Scratch lives under the output parent; on a - cluster that is a Volume/DBFS FUSE path, written **sequentially** (FUSE-safe; no - random writes / rename). `abort()` removes scratch + any partial output. - -## Geometry + CRS handling - -`pyogrio.write_arrow` needs `driver`, `geometry_name`, `geometry_type`, `crs`: -- **Geometry column:** the input's geom column is `geom_0` (WKB binary) — pass it as - the arrow geometry column. If the input geometry is WKT (`StringType`, from a - reader run with `asWKB=false`), convert WKT→WKB via `shapely` first. -- **`geometry_type`:** **inferred** from the first non-null geometry - (`shapely.from_wkb(g).geom_type` → `Point`/`LineString`/`Polygon`/…), with an - optional `geometryType` writer option to override (mixed/empty inputs). -- **`crs`:** from the `geom_0_srid` column (`"4326"` → `"EPSG:4326"`); fall back to - `geom_0_srid_proj` (PROJ4) when the authority code is `"0"`/empty. -- The `geom_0_srid` / `geom_0_srid_proj` columns are consumed for CRS, not written - as attributes (they're reader metadata, not OGR fields). - -## Options - -| Option | Default | Behavior | -|---|---|---| -| `driverName` | required for `ogr_gbx`; preset by named writers | OGR driver. | -| `mode` | `overwrite` | `overwrite` only; **`append` rejected** (output is one merged file — keep it simple). | -| `geometryType` | inferred | Override the inferred geometry type. | -| `layerName` | format default | Output layer name where the driver supports it. | - -## Architecture / files - -Extend `python/geobrix/src/databricks/labs/gbx/ds/vector.py`: -- `OgrGbxWriter(DataSourceWriter)` — `write(iterator)` → scratch fragment via - `write_arrow`; `commit(messages)` → merge fragments into the output file; - `abort(messages)` → cleanup. A `_VectorCommitMessage` carries the fragment path. -- `OgrGbxDataSource.writer(self, schema, overwrite)` returns `OgrGbxWriter` - (validates the input schema has a `geom_*` + `geom_*_srid` pair). The four named - `*GbxDataSource` subclasses already preset `_DRIVER`; their `.writer()` inherits. -- Helpers: `_geometry_type_of(wkb)` (shapely), `_srid_to_crs(srid, proj4)` (inverse - of the reader's `_crs_to_srid_proj`), reuse `_zip_vsi` for zipped targets. -- Pure-Python / Serverless-safe (pyogrio/pyproj/shapely lazy inside methods); the - Serverless guard already scans `vector.py`. - -## Docs - -New **lightweight-only** vector writer pages (heavy has no vector writer — a -`:::note` says so), mirroring the reader format set: a generic Vector writer page + -named Shapefile/GeoJSON/GeoPackage/GeoDatabase writer pages (or one Vector writer -page with the formats — match the readers' granularity). Add them under Writers → -Named/General in the sidebar. New doc-test example files exercising a real -write→read round-trip against the corpus. - -## Benchmark - -Fold into the existing `--benchmark-vector` bench cell (one vector cell does readers -+ writers): -- Vector **writer** timing is **light-only** (no heavy vector writer to compare) — - record the light write time per format. -- Add a **round-trip parity** check: write with `*_gbx` → read back with the `*_gbx` - reader → assert the read-back feature count + geometries match the input. (This is - the writer's correctness gate, replacing the light-vs-heavy parity used for - readers.) -- Benchmarking.mdx: extend the "Results — vector readers" section (or add a writers - subsection) with the writer timings + the round-trip note. - -## Corpus generator (Phase 3 enabler) - -A thin helper (a `gbx:data:*` command or a bench utility) that uses the writers to -generate **scaled / synthetic** vector data (N features, chosen geometry type, -chosen format) and stage it to the bench corpus — the input for Phase 3's -scaled/final benchmarking. Spec'd here as a deliverable; detailed scale knobs are -the Phase-3 plan's concern. - -## Testing (TDD) - -- **Round-trip parity** (local Spark) per format: build a small `(attrs…, geom_0 - WKB, geom_0_srid, geom_0_srid_proj)` DataFrame → `write.format("_gbx").save` → - read back with the `_gbx` reader → same feature count + geometries + - attributes. Covers `ogr_gbx` + the four named. -- **Two-phase merge** (local Spark): a multi-partition input writes one output file - whose feature count equals the union of all partitions (no lost/duplicated rows). -- **CRS + geometry_type**: srid round-trips (`4326` in → `4326` out); inferred - geometry_type matches; `geometryType` override honored. -- **`mode`**: `overwrite` replaces; `append` raises a clear error. -- **Serverless guard**: `vector.py` stays clean (no `_jvm`/`.conf.set`/`.rdd`). -- **Docker integration**: write→read round-trip against a real corpus file. - -## Out of scope (later / other phases) - -- **Multi-geometry-field** output (`geom_1`, …) — single `geom_0` in v1 (matches the reader). -- **Heavy vector writer** — none exists; nothing to match. -- **Phase 3 scaled benchmarking** — separate; this phase delivers the writer + the - corpus-generator primitive it needs. - -## Verify-during-impl checklist - -1. `write_arrow` fragment write: the input geom column (`geom_0` WKB) maps to the - arrow geometry column with the right `geometry_name`; attribute columns preserved; - `geom_0_srid`/`_srid_proj` consumed for CRS, NOT written as fields. -2. `commit()` merge: `append=True` across fragments yields one file with all features; - fragment order doesn't matter for vector (no tileid ordering); scratch cleaned. -3. Shared-FS: scratch under the output parent; sequential writes (FUSE-safe); on a - cluster the output is a Volume/DBFS path. No `os.rename` on FUSE. -4. `geometry_type` inference handles a layer of one type; `geometryType` override for - mixed; empty input → valid empty file or clear behavior. -5. CRS: `geom_0_srid="4326"` → `EPSG:4326` out; `"0"`/empty → fall back to proj4 or - write CRS-less (match what the reader produced). -6. Round-trip: `read(ogr_gbx) → write(_gbx) → read(_gbx)` is feature- and - geometry-stable for each corpus format. -7. Named writers preset the same `_DRIVER` as their reader counterpart (ESRI - Shapefile / GeoJSON / GPKG / OpenFileGDB). -8. Docs: lightweight-only writer pages render + the round-trip doc-test passes; the - "no heavyweight vector writer" note is present; internals-leak clean. -9. Bench: `--benchmark-vector` records writer timings + the round-trip parity gate. diff --git a/docs/superpowers/specs/2026-06-12-readers-writers-tabbed-tiers-design.md b/docs/superpowers/specs/2026-06-12-readers-writers-tabbed-tiers-design.md deleted file mode 100644 index e93808fb8..000000000 --- a/docs/superpowers/specs/2026-06-12-readers-writers-tabbed-tiers-design.md +++ /dev/null @@ -1,164 +0,0 @@ -# Readers & Writers: tabbed light/heavy tiers (consolidated by format) Design - -**Date:** 2026-06-12 -**Branch:** `light-readers` -**Status:** Approved (design); ready for implementation plan - -## Summary - -Consolidate the Readers & Writers docs **by format instead of by tier**. Today the -nav splits into `Lightweight` and `Heavyweight` subtrees, so a format with both -tiers (e.g. raster) appears as two separate pages (`raster_gbx` and `gdal`). As the -light and heavy tiers converge on a **1:1 reader/writer correspondence**, that split -duplicates structure and hides the equivalence. - -New model: **one page per format**, with common format-level intro text followed by -**``** to toggle between the **Lightweight** (default, first) and -**Heavyweight** tiers. Formats that exist in only one tier stay single pages with a -short note that the other tier has no equivalent yet — **no placeholder tabs**. - -This is a **docs-only** change (structure + prose). The executable doc-tests under -`docs/tests/` are unchanged; pages keep importing the same example files via -raw-loader. One genuine gap is filled: the heavy **`gtiff_gdal` writer** (a real, -registered DataSource — `src/main/scala/.../rasterx/ds/gtiff/GTiff_DataSource.scala`, -shortName `gtiff_gdal`, "read/write .tif") is currently undocumented; its write side -gets a Heavyweight tab on the GeoTIFF-writer page. - -## Goals / non-goals - -- **Goal:** group by format; light/heavy as synced tabs; light default + first - everywhere; document the equivalence (and where it isn't exact). -- **Goal:** fill the `gtiff_gdal`-writer doc gap. -- **Non-goal:** any change to the executable doc-tests' assertions or the example - `.py`/`.scala` code (only which snippet renders under which tab). -- **Non-goal:** a light vector tier — vector reader pages stay heavyweight-only + - note until `pyvx` lands. - -## Final navigation - -``` -Readers & Writers -├── Overview (readers/overview + writers/overview — consolidated) -├── Readers -│ ├── General -│ │ ├── Raster TABS: Lightweight (raster_gbx) | Heavyweight (gdal) -│ │ └── Vector heavyweight-only (ogr) + note -│ └── Named -│ ├── GeoTIFF TABS: Lightweight (gtiff_gbx) | Heavyweight (gtiff_gdal) -│ ├── Shapefile heavyweight-only + note -│ ├── GeoJSON heavyweight-only + note -│ ├── GeoPackage heavyweight-only + note -│ └── GeoDatabase heavyweight-only + note -└── Writers - ├── General - │ └── Raster TABS: Lightweight (raster_gbx) | Heavyweight (gdal) - └── Named - ├── GeoTIFF TABS: Lightweight (gtiff_gbx) | Heavyweight (gtiff_gdal)* - └── PMTiles TABS: Lightweight (pmtiles_gbx) | Heavyweight (pmtiles) - -* Heavyweight GeoTIFF-writer tab is NEW documentation (gtiff_gdal was undocumented). -``` - -## Page consolidation map - -Each tabbed page is **one new format-named `.mdx`** that absorbs the two tier pages. -Single-tier pages keep their existing id. - -| New page (doc id) | Lightweight tab source | Heavyweight tab source | Old pages replaced | -|---|---|---|---| -| `readers/raster` | `readers/raster_gbx.mdx` | `readers/gdal.mdx` | both | -| `readers/geotiff` | `readers/gtiff_gbx.mdx` | `readers/gtiff.mdx` | both | -| `readers/ogr` (vector) | — (note) | `readers/ogr.mdx` | unchanged id | -| `readers/shapefile` | — (note) | `readers/shapefile.mdx` | unchanged | -| `readers/geojson` | — (note) | `readers/geojson.mdx` | unchanged | -| `readers/geopackage` | — (note) | `readers/geopackage.mdx` | unchanged | -| `readers/filegdb` | — (note) | `readers/filegdb.mdx` | unchanged | -| `writers/raster` | `writers/raster_gbx.mdx` | `writers/gdal.mdx` | both | -| `writers/geotiff` | `writers/gtiff_gbx.mdx` | **new** (`gtiff_gdal`) | gtiff_gbx | -| `writers/pmtiles` | `writers/pmtiles_gbx.mdx` | `writers/pmtiles.mdx` | both | - -URL note: the merged pages get new format-based ids, so prior tier-specific URLs -(`readers/raster_gbx`, `readers/gdal`, …) change. **No redirects** — the docs are -beta and the organization is still WIP, so old URLs are allowed to break. - -## Page template - -**Tabbed page:** -1. `# ` H1 + `sidebar_label: ` (e.g. `Raster`, - `GeoTIFF`, `PMTiles`). -2. **Common intro** — what this reader/writer does at the format level, the shared - `(source, tile)` / input contract, links to the [one-line tier swap](execution-tiers). -3. **``**: - - `` — light usage, - options, `` example(s), perf pointer. - - `` — heavy usage, - options, example(s). -4. Where the tiers aren't feature-identical, a one-line caveat (e.g. General Raster: - "the heavy `gdal` reader supports many more GDAL drivers than the light path"). - -**Single-tier page:** the existing heavyweight content, plus an admonition: -``` -:::note Lightweight equivalent -This format does not have a lightweight reader yet; it is planned with the light -vector tier. Use the heavyweight reader below. -::: -``` - -## Tabs mechanism - -- Docusaurus theme `Tabs`/`TabItem` (import `@theme/Tabs`, `@theme/TabItem` in MDX). -- **`groupId="tier"`** on every `` so the tier choice **syncs across all pages - and persists** (localStorage). Tab **`value`** is the stable key `light`/`heavy`; - the **`label`** carries the format-specific name (`Lightweight · raster_gbx`). -- **Light is `default` and the first ``** on every page → light-first, - light-default everywhere; a returning reader who picked Heavyweight stays there. - -## Sidebar labels - -Pages become **format-named** (`sidebar_label`): `Raster`, `GeoTIFF`, `Vector`, -`Shapefile`, `GeoJSON`, `GeoPackage`, `GeoDatabase`, `PMTiles`. This **supersedes** -the per-tier sidebar labels added on 2026-06-11 (`Raster GBX`, `GDAL`, `GeoTIFF GDAL`, -…) — those tier+engine strings move into the **tab labels** (`Lightweight · raster_gbx` -/ `Heavyweight · gdal`). - -## Overview pages - -Consolidate `readers/overview` and `writers/overview` to drop the per-tier framing: -list the formats (General vs Named), explain the **light/heavy tab model** + the -one-line tier swap, and link each format page. Keep them as the two entry pages under -Readers & Writers. - -## New content: `gtiff_gdal` writer (fills a gap) - -The heavy GeoTIFF writer (`gtiff_gdal`) is registered but undocumented. The existing -`docs/tests/python/writers/gdal_examples.py` already exercises `format("gtiff_gdal")` -writes, so the GeoTIFF-writer Heavyweight tab renders that snippet (extract a -`gtiff_gdal`-focused example function if the current one is GDAL-generic). No new -runtime behavior — just documenting an existing writer. - -## Testing / validation - -- `gbx:test:python-docs --path readers/` and `--path writers/` stay green (the - imported example code is unchanged; only MDX structure changes). The new - `gtiff_gdal`-writer example must execute under the doc-test harness (Docker, heavy). -- Docusaurus build succeeds (`gbx:docs:start` / CI docs build): valid `Tabs`/`TabItem` - MDX, no broken sidebar ids, no dangling links to the removed tier pages. -- Internals-leak check stays clean (`grep -rn -iE "wave [0-9]+" docs/docs/`). - -## Verify-during-impl checklist - -1. `groupId="tier"` syncs across pages and persists; light is default+first on every - tabbed page (manually confirm in `gbx:docs:dev`). -2. Every internal link/sidebar id that pointed at a removed tier page is repointed - to the new format page (grep for `readers/raster_gbx`, `readers/gdal`, - `writers/gtiff_gbx`, etc. across `docs/`). -3. Single-tier pages render the note and NO empty tab. -4. The `gtiff_gdal` writer example actually runs in the doc-test harness. -5. `function-info`/binding parity unaffected (no function changes; pmtiles_gbx etc. - are formats, not registered functions). -6. No redirects from old ids (beta/WIP — old URLs may break). - -## Out of scope (later) - -- Light vector tier (`pyvx`) — when it lands, the vector reader pages gain a - Lightweight tab and drop the note. diff --git a/docs/superpowers/specs/2026-06-13-h3-raster-tessellation-modes-design.md b/docs/superpowers/specs/2026-06-13-h3-raster-tessellation-modes-design.md deleted file mode 100644 index beb760fb5..000000000 --- a/docs/superpowers/specs/2026-06-13-h3-raster-tessellation-modes-design.md +++ /dev/null @@ -1,212 +0,0 @@ -# H3 raster tessellation — pedigree, current behavior, and multi-mode design notes - -> **Status:** APPROVED design (2026-06-13). Sections 1–5 are the grounding background (pedigree + -> current behavior); sections 6–10 are the approved design. Much of §1–5 is institutional history -> (Mosaic → Databricks-native) — preserve it. Next step after this doc: writing-plans. - -## 1. Why this matters — lineage / pedigree - -GeoBrix's H3 raster functions descend from a technique **pioneered in DBLabs Mosaic** that has since -**inspired Databricks-native product functions**. The lineage: - -- **Mosaic** (vector + raster grid tessellation): - - vector `grid_tessellate` — https://databrickslabs.github.io/mosaic/api/spatial-indexing.html#grid-tessellate - - raster `rst_tessellate` — https://databrickslabs.github.io/mosaic/api/raster-functions.html#rst-tessellate - - raster `rst_rastertogridavg` (and `*count/max/min/median`) — https://databrickslabs.github.io/mosaic/api/raster-functions.html#rst-rastertogridavg -- **Databricks-native (product) H3 functions** that the Mosaic technique inspired: - - `h3_coverash3` — the **covering set** of a geometry: every H3 cell that *overlaps* it. - https://docs.databricks.com/aws/en/sql/language-manual/functions/h3_coverash3 - - `h3_tessellateaswkb` — **tessellation**: for each covering-set cell, the geometry **clipped to - the cell** (the intersection), returned as a WKB "chip". - https://docs.databricks.com/aws/en/sql/language-manual/functions/h3_tessellateaswkb -- **GeoBrix** deliberately did **NOT** re-port the H3 functions now built into the product (the - `h3_*` vector functions). It carries the **raster** H3 functions (`rst_h3_tessellate`, - `rst_h3_rastertogrid*`) and the discrete-grid families. Positioning: GeoBrix is an on-ramp that - offers the *same pioneered technique* (in **both** light and heavy tiers) and **complements** the - product's native `h3_*` functions rather than competing with them. - -**Implication for docs:** when we surface this, frame GeoBrix as the origin of (and complement to) -the native `h3_coverash3` / `h3_tessellateaswkb` technique — factual, lineage-grounded. - -## 2. The reference technique (raster H3), per MLJ - -For a raster, the canonical "tessellate-as-WKB" technique is: -- **(a)** Take the tile's extent as a **bbox polygon** ("bbox geom"). -- **(b)** Project bbox geom to **EPSG:4326** if needed (or require the caller to pass 4326). -- **(c)** Get the **COVERING SET** of bbox geom — every H3 cell that **overlaps** it. This is NOT - centroid-`polyfill` (the "is the cell's center inside?" half-in rule). -- **(d)** **Vector-intersect** each covering-set cell's hexagon with bbox geom → a per-cell WKB - **"chip"** (the clipped piece). Some functions keep the chip; others reduce it to a measure. - -H3 v4 primitives: -- **Covering set** = `polygonToCellsExperimental(poly, res, ContainmentOverlapping)` — exact overlap. -- **Centroid polyfill** = classic `polygonToCells` / `polyfill` — center-in-polygon containment. - -## 3. Current GeoBrix HEAVY behavior (grounded from code) - -(From a read of `src/main/scala/com/databricks/labs/gbx/rasterx/...`; file:line where cited.) - -### `gbx_rst_h3_tessellate` — `RST_H3_Tessellate` → `RasterTessellate.tessellateH3Iter` -- **(a) bbox geom: YES** — `BoundingBox.bbox(ds, GDAL.WSG84)` builds a 4-corner extent polygon. -- **(b) projection: reprojects to 4326 internally** (raster-CRS→4326 for the extent; hexagons - reprojected 4326→raster-CRS for clipping in `ClipToGeom`). -- **(c) cell selection: centroid-`polyfill` on `bbox.buffer(bufR)` — NOT a true covering set.** - `H3.polyfill(bbox.buffer(bufR), res)`. Uber `h3.polyfill` is centroid-containment; the bbox is - dilated by `getBufferRadius` (≈ one cell circumradius) to recover fringe cells. An *approximation* - of a covering set. -- **(d) per-cell output: hexagon-clipped WKB chip** — true H3 hexagon used as a `gdalwarp -cutline - … -crop_to_cutline` cutline with `CUTLINE_ALL_TOUCHED=TRUE`. **Keep-test = NoData-mask "any valid - pixel after the cutline"** (`RasterAccessors.isEmpty`), NOT a hexagon-area-coverage threshold. -- **Consequence (measured):** over-includes a **disjoint fringe** (~188–284 cells, *zero* geometric - overlap with the raster) — an artifact of buffer + bbox-snapped warp + nodata keep-test. Those - cells are beyond even the true covering set. - -### `gbx_rst_{h3,quadbin}_rastertogrid{avg,count,max,min,median}` (10) — diverges from the reference -- **(a) bbox geom: NO** — pure per-pixel walk, no polygon. -- **(b) projection: assumes/requires 4326, no reprojection** (geotransform fed straight to - `pointToCellID`; non-4326 silently wrong; quadbin docs say "callers reproject via RST_Transform"). -- **(c) cell selection: pixel-centroid point sampling** — each valid pixel's 0.5-offset center maps - to exactly ONE cell (`geoToH3` / `Quadbin.pointToCell`). Emergent set ("cells containing ≥1 valid - pixel centroid"). Neither covering set nor polyfill+buffer. **This is inherently - single-assignment per pixel.** -- **(d) per-cell output: scalar MEASURE, no clip** — valid pixels bucketed per cell, then `fAgg` - (avg/count/max/min/median). No area / partial-pixel weighting; a boundary cell gets whole pixels. - -### `gbx_rst_gridfrompoints(+agg)` — inverse direction (points→raster IDW), N/A to the lens. - -### Heavy inconsistencies to carry into the design -- **Cell selection differs across the family:** tessellate = polyfill-on-buffered-bbox - (geometry-driven, approximate covering); rastertogrid = pixel-centroid-to-cell (data-driven, - single-assignment). **Neither uses a true `all_touched`/overlap covering set.** -- **CRS handling differs:** tessellate reprojects internally; rastertogrid hard-assumes 4326. -- **`getBufferRadius` has only Polygon/MultiPolygon match arms — no default** (MatchError risk). -- The tessellate keep-test is nodata-mask, not hexagon coverage → the disjoint-fringe over-inclusion. - -## 4. Current GeoBrix LIGHT behavior - -- **`rst_h3_tessellate` (light)** — bbox→4326, `h3.h3shape_to_cells` (centroid polyfill) **+ a - one-ring `grid_disk` buffer + an `all_touched` pixel-coverage prune** → lands on the **true - all-touched / overlapping set** (verified `== oracle`: zero misses, zero extras, on both a 4326 - SRTM tile and a reprojected UTM tile). Emits a hexagon-clipped chip via `rasterio.mask`. - **Caveat:** the prune uses `all_touched=True` but the actual clip uses `all_touched=False` - (a touch-semantics asymmetry to reconcile). -- **`rst_*_rastertogrid*` (light)** — mirrors HEAVY exactly (pixel-centroid binning, scalar measure, - no clip, assumes 4326). Light and heavy agree with each other here; both differ from the - covering-set+clip reference (by design — this is the data-driven binning family). - -## 5. The tessellate divergence (root cause) - -Measured: light **11958** vs heavy **12242** cells on the same tile (~2.4%, heavy more). All the -heavy-extra cells are **fully disjoint** (zero overlap) — heavy's buffer + bbox-snapped warp + -nodata keep-test admits a fringe ring beyond the covering set. **Light is the correct all-touched -set.** This is a heavy correctness issue, *not* caused by the recent UDTF conversion (which only -changed the call form, not the cell math). - -## 6. Approved design — `rst_h3_tessellate` modes (light + heavy aligned) - -Scope (MLJ): `rst_h3_tessellate` ONLY — the one diverging function. `rst_h3_rastertogrid*` already -agrees light↔heavy and is out of scope. Two named modes, **identical in both tiers by construction** -(both tiers call the same H3 v4 primitive per mode); the alignment deletes both tiers' hand-rolled -approximations rather than patching them separately. - -### 6.1 The `mode` parameter -- Optional trailing **string** param `mode ∈ {"covering", "centroid"}`, **default `"covering"`** — - matching geobrix's string-enum convention for multi-choice params (`algorithm`, `operation`, - `split_point_finder`, `format`). A boolean (`useCentroid`) was rejected: modes may grow (e.g. - area-weighted) and a boolean can't extend without a breaking change. -- **Backward compatible.** SQL is positional-only: heavy `FunctionBuilder` registers arity **2** - (default `Literal("covering")`) **and 3**, so existing `(tile, resolution)` calls keep working; - `(tile, resolution, 'centroid')` selects the new mode. Python wrappers (light + heavy): - `mode: ColLike = "covering"` (positional or `mode=` kwarg; SQL positional only). -- **Validation** follows the rasterx pattern: Scala `require(AllowedSet.contains(...))` + Python - `ValueError` on a `{"covering","centroid"}` miss, message listing the valid values. - -### 6.2 `covering` mode (default) — the pioneered tessellate-as-WKB technique -- **Cell selection:** the **true covering set** of the tile's 4326 bbox — every cell that *overlaps* - the tile. - - **Light:** h3-py **4.4.2** native `polygon_to_cells_experimental(shape, res, contain='overlap')` - (`'overlap'` = covering; `'center'` = classic polyfill). - - **Heavy:** H3-Java is **pinned at 3.7.0** (``) — it - has **no v4 covering primitive**. So heavy **hand-rolls** the same set: keep polyfill + ring/buffer - candidate generation, then keep a cell iff its hexagon **geometrically overlaps the bbox** (a JTS - intersection test), replacing the current nodata keep-test. A **defensible cross-tier divergence in - *mechanism*** — the product ships the H3 3.7.x JAR while Python can ride the h3 4.x series — both - compute the identical overlapping set. -- **Per-cell output:** raster **clipped to the cell's hexagon** with **`all_touched=True`** (boundary - pixels included), applied consistently in any prune AND the clip (fixes light's prune-vs-clip - asymmetry; matches heavy's `CUTLINE_ALL_TOUCHED=TRUE`). One tile-struct chip per cell. -- **Semantics:** full coverage of the tile; border cells/pixels are **shared with neighboring tiles** - (overlap accepted — union across tiles reconstructs a full cell). -- Replaces heavy's polyfill-on-buffered-bbox + nodata keep-test (removing the disjoint-fringe - over-inclusion) AND light's seed+grid_disk+prune approximation → identical cells by construction. - -### 6.3 `centroid` mode (new, additive) — pixel-centroid single-assignment -- **Assignment:** each **pixel** → the single H3 cell whose hexagon contains the pixel's centroid - (per-pixel `pointToCellID`/`latlng_to_cell` — the **same selection `rst_h3_rastertogrid*` already - uses**). -- **Per-cell output:** one tile-struct chip per cell holding **only its assigned pixels** (others - nodata). The cell set emerges from the pixels (cells with ≥1 assigned pixel); no bbox/covering step. -- **Semantics:** a **partition** — every pixel assigned exactly once, **nothing dropped, no - double-count across tiles** (a pixel belongs to exactly one hexagon globally). The de-duped binning - case ("assign a set of rasters/tiles to H3 cells without double-counting"). -- This is **pixel**-centroid, NOT cell-centroid selection (which would drop border pixels — rejected). - -### 6.4 CRS -- Both modes, both tiers reproject the tile extent / pixel coords to **EPSG:4326** internally for the - H3 lookups (current tessellate behavior — kept). (`rastertogrid`'s hard-4326 assumption is separate - and out of scope.) - -### 6.5 Cross-tier alignment + "no harm" -- The two tiers compute the **same set by definition** (covering = the true overlapping set; centroid = - pixel-centroid assignment) — light via the h3-py v4 primitive, heavy via the 3.7.0-compatible - hand-rolled equivalent — and **parity is enforced by the per-mode tests** (§8), not by an identical - API call. Functionally equal, test-guaranteed. -- `covering` (default) is the **corrected** existing behavior — heavy drops its disjoint fringe, light - drops its approximation. The 0.4.0 H3 capabilities are unreleased WIP, so this is a fix, not a - back-compat break. `centroid` is purely **additive**. Existing capability is **fixed + extended**. - -## 7. Implementation scope - -- **Heavy (Scala, H3-Java 3.7.0 — NO v4 primitive):** `RST_H3_Tessellate` (+ `RasterTessellate` / `H3`) - — add `modeExpr`; **covering path** → hand-rolled true covering set: keep the polyfill + ring/buffer - candidate generation but **replace the nodata keep-test with a JTS hexagon∩bbox overlap test** (this - removes the disjoint-fringe over-inclusion and matches light's `contain='overlap'`); give - `getBufferRadius` a default match arm if the buffer path is retained; **centroid path** → per-pixel - `pointToCellID` assignment → per-cell chip; `FunctionBuilder` arity 2+3; Scala API + heavy Python - binding `mode="covering"`; validation. JAR rebuild + tessellate re-bench. -- **Light (pyrx):** `pyrx/core/tessellate.py` + the `rst_h3_tessellate` UDTF/wrapper — add `mode`; - covering path → the h3-py v4 overlapping-containment call (verify exact API: - `polygon_to_cells_experimental(..., contain='overlap')` or equivalent) replacing seed+grid_disk+prune; - centroid path → per-pixel `latlng_to_cell` → per-cell chip; fix the `all_touched` asymmetry; validation. -- `registered_functions.txt` name unchanged (`gbx_rst_h3_tessellate`); update the `function-info.json` - usage example + docstrings to show the `mode` arg. - -## 8. Testing - -- **Per-mode light-vs-heavy parity** on a border-containing tile (the regime that exposed the - divergence): for EACH mode, assert light and heavy produce the **same cell set** AND the **same - per-cell chip pixels** — passing by construction (same H3 primitive). Replaces the strict-equality - fan-out bench leg that the divergence tripped. -- **Covering:** cell set == the true overlapping set (vs a `ContainmentOverlapping` oracle); no - disjoint cells. -- **Centroid:** assert a **partition** — every input pixel appears in exactly one cell's chip; the - union of chips == all valid pixels; no pixel in two chips. -- Spark-free light core unit tests for both modes; the fan-out bench's `h3_tessellate` leg compares - per-mode (default covering). - -## 9. Docs — H3 explainer page (deliverable; outline to refine with MLJ) - -A dedicated H3 page explaining how H3 raster handling works. Draft outline (to refine before writing): -- **Lineage:** Mosaic → Databricks-native `h3_coverash3` / `h3_tessellateaswkb`; GeoBrix as - origin-of / complement-to the native technique. -- **The two tessellation modes** — `covering` (full coverage, shareable across tiles; the pioneered - chip technique) vs `centroid` (pixel-centroid single-assignment partition, de-duped) — with a - "when to use which" guide and a visual of the border behavior (overlap vs partition). -- **Relationship to `rst_h3_rastertogrid*`** — same centroid selection, measure vs chip output. -- **CRS expectations** (4326 internally for tessellate; the rastertogrid 4326 contract). -- **Cross-tier parity** (light ≡ heavy by construction). - -## 10. Status / next - -**Approved design (2026-06-13).** Next: **writing-plans** → subagent-driven implementation (heavy -Scala + JAR + light + per-mode parity tests + the explainer page). diff --git a/docs/superpowers/specs/2026-06-13-pyvx-mvt-light-tier-design.md b/docs/superpowers/specs/2026-06-13-pyvx-mvt-light-tier-design.md deleted file mode 100644 index e3de67497..000000000 --- a/docs/superpowers/specs/2026-06-13-pyvx-mvt-light-tier-design.md +++ /dev/null @@ -1,164 +0,0 @@ -# pyvx (light VectorX) — MVT tier design - -**Status:** approved design (brainstormed 2026-06-13). Next: writing-plans → implementation. -**Branch:** `pyvx-light` (from `beta/0.4.0`). -**Survey:** `prompts/features/2026-06-12-pyvx-light-tier-survey.md`. - -## Goal - -Deliver the **MVT** slice of a pure-Python / PySpark **light VectorX tier** (`pyvx`) — `st_asmvt` -(aggregator) and `st_asmvt_pyramid` (generator) — that is a drop-in swap for the heavyweight -`vectorx` MVT functions and runs where the heavyweight tier can't (Serverless, ARM, standard/shared, -Lakeflow — no JAR, no init script, no native GDAL). As part of the same effort, upgrade **both -tiers** to encode MVT feature attributes with **native protobuf value types** (not stringified). - -## Scope - -**In scope** -- `pyvx.st_asmvt` — light MVT aggregator. -- `pyvx.st_asmvt_pyramid` — light MVT pyramid generator. -- **Native attribute typing in both tiers** — light encoder + an upgrade to the heavyweight Scala - `MvtWriter` / `st_asmvt` / `st_asmvt_pyramid` paths (currently stringify all attributes). -- Light-vs-heavy parity + perf benches for both functions; the Benchmarking page **Vector** tab. -- `pyvx` docs page mirroring the readers/writers doc template. - -**Out of scope (separate later specs)** -- TIN / elevation generators (`st_triangulate`, `st_interpolateelevationbbox`, - `st_interpolateelevationgeom`) — the constrained-Delaunay / breakline trade-off is its own design. -- `st_legacyaswkb` (light) — deferred; revisit when the migration path is taken on. - -## Why native attributes (decision record) - -The MVT protobuf `Value` is a typed union (`string_value`, `int_value`, `uint_value`, `sint_value`, -`float_value`, `double_value`, `bool_value`). Stringifying everything (`pop="42"`) is valid MVT but: -(1) loses the format's typed-value efficiency (larger tiles), and (2) forces type-sensitive clients -(MapLibre/Mapbox GL data-driven styles, filters, numeric expressions) to add `to-number` casts. -Native typing (`pop=42`) is best practice. These MVT functions are **net-new, unreleased capabilities -since v0.3.0**, so changing the heavyweight behavior carries **no back-compat surface** — we make both -tiers native together, which keeps the swap invisible *and* the parity gate meaningful. - -## Architecture & package layout - -New light package `python/geobrix/src/databricks/labs/gbx/pyvx/`, mirroring `pyrx` / `vectorx`: - -- `functions.py` — Column-API wrappers with signatures **identical to heavy `vectorx.functions`** - (`st_asmvt(geom_wkb, attrs, layer_name)`, `st_asmvt_pyramid(geom_wkb, attrs, min_z, max_z, - layer_name=None, extent=None)`) + `register(spark)`. -- `_mvt.py` — pure-Python encode helpers (tile-local encode + per-tile clip/encode) over - `mapbox-vector-tile`. -- `_serde.py` — geometry WKB ↔ `shapely`; the attrs-struct → MVT-`Value` type mapping. -- `register(spark)` — **Serverless-safe wiring only**: `spark.udf.register(...)` for the aggregator, - `spark.udtf.register(...)` for the pyramid. **No `_jvm` / `spark.conf.set` / `sparkContext` / - `.rdd`** anywhere (hard Serverless constraint). A `test_serverless_no_spark_config`-style guard - asserts this, as for `pyrx`. - -Output tiles compose with the existing `gbx_pmtiles_agg` writer for end-to-end publishing. - -## Native attribute typing contract (both tiers) - -`attrs` is a Spark struct column; each field maps to the matching MVT `Value` field: - -| Spark field type | MVT `Value` field | -|---|---| -| `IntegerType` / `LongType` | `int_value` (signed → `sint_value`) | -| `FloatType` | `float_value` | -| `DoubleType` | `double_value` | -| `BooleanType` | `bool_value` | -| `StringType` | `string_value` | -| any other (date, timestamp, binary, decimal, array, struct, null) | `string_value` fallback | - -- Governs **both** the light encoder and the upgraded heavy `MvtWriter`, so the tiers emit - byte-equivalent typed tiles. -- **Heavy change (in scope):** `src/main/scala/com/databricks/labs/gbx/vectorx/mvt/MvtWriter.scala` - (and the `st_asmvt` aggregate + `st_asmvt_pyramid` generator paths) stop stringifying and read - typed struct fields; update the Scala MVT tests/docs that assert stringified output. -- **Parity** is measured at the **decoded-feature level** (geometry + typed properties), like the - PMTiles gate — not raw bytes (encoders differ in byte layout: key ordering, value dedup, - geometry quantization). - -## `st_asmvt` aggregator (grouped-agg pandas UDF) - -- **Call site, identical to heavy:** `df.groupBy("z","x","y").agg(pyvx.st_asmvt(geom_wkb, attrs, "layer"))`. -- **Input contract mirrors heavy:** `geom_wkb` per-row geometry in WKB, already in **tile-local - coordinates**; `attrs` per-row struct; `layer_name` constant. -- **Impl:** Arrow-backed grouped-agg pandas UDF — group columns arrive as pandas Series → decode WKB - via `shapely` → build features with native-typed properties → `mapbox-vector-tile` encodes one layer - (default extent 4096) → return MVT `BINARY`. One blob per group. Non-partial (whole group, one - post-shuffle stage), as with the `pyrx` `*_agg` functions; a group is one tile's features. - -## `st_asmvt_pyramid` generator (Python UDTF) - -- **Impl:** a Python UDTF whose `eval` **`yield`s rows incrementally** (avoids fan-out OOM — do not - build a large list) — one `(z, x, y, mvt_bytes)` per intersecting tile. Inputs EPSG:4326; for each - feature × zoom in `[min_z, max_z]`, compute intersecting tiles, clip (`shapely`), reproject to - tile-local extent (`pyproj` transformer built per-call/partition, not global), encode, yield. -- **Caps mirror heavy:** `max_z ≤ 20`; total tiles across the zoom range `≤ 10^6` — enforced, with a - clear raised error on breach. -- **Output schema matches heavy** (`z, x, y, mvt_bytes`) so it feeds `gbx_pmtiles_agg` identically. -- **Call site:** registered via `spark.udtf.register("pyvx_st_asmvt_pyramid", …)`, invoked as a - table/lateral function (`… FROM features, LATERAL pyvx_st_asmvt_pyramid(geom, attrs, min_z, max_z, - layer, extent)`). Differs from heavy's generator-in-`select` — documented; output + composition are - identical. -- **De-risk (first plan task):** verify the UDTF registers and runs on **Serverless + Spark Connect** - (and the bench cluster). If unsupported, fall back to **2A** — `pandas_udf(ArrayType(tile_struct))` - + caller `explode` — with the same output schema. The rest of the design is unchanged either way. - -## Error handling & edge cases - -- Empty/invalid geometry → zero rows / null (mirror heavy); a feature not intersecting a tile at a - zoom → no row for that tile. -- Unsupported attr field types → `string_value` fallback, never an error. -- Cap breach (`max_z`, 10^6 tiles) → explicit raised error, matching heavy. -- No process-global mutable state; UDF/UDTF are pure functions of inputs (Serverless-safe). - -## Testing & bench - -- **TDD, real data, no mocks:** encode known features → decode with `mapbox-vector-tile` → assert - geometry + native-typed properties. -- **Light-vs-heavy decoded-feature parity** tests per function (both tiers native): same input → - decode both outputs → features (geometry + typed props) match. -- **Heavy-side:** update the Scala MVT tests/docs to assert native-typed values (were stringified). -- **Bench:** extend the bench harness (like the vector readers/writers) — light-vs-heavy timing + - parity for `st_asmvt` and `st_asmvt_pyramid`; populate the Benchmarking page **Vector** tab. -- **Binding parity:** `gbx_st_*` already registered; keep `pyvx` bindings + `function-info` - consistent (`gbx:test:bindings`). - -## Dependencies - -- Add `mapbox-vector-tile` (pure Python, attribute-preserving) to the `light` extra. `shapely>=2.0` - and `pyproj` are already present. - -## Risks - -- **UDTF on Serverless/Connect** — **RESOLVED (2026-06-13): use approach 2B.** - Spike (Task 1) confirmed both conditions of the decision rule: - 1. **Local run passed** — trivial `Fan` UDTF registered via `spark.udtf.register`, executed via - `LATERAL`, yielded all expected rows under PySpark 4.0.0 / Python 3.12 (`PYSPARK_PYTHON` must - match driver version; worker picked up system Python 3.10 until env vars were set). - 2. **Platform support confirmed** — Databricks docs list Python UDTFs as Public Preview on - Serverless (DBR 14.3+) and supported over Databricks Connect / Spark Connect (16.4+); Unity - Catalog UDTFs supported from DBR 17.1+. Our target (DBR 17.3 LTS) clears both thresholds. - Sources: https://docs.databricks.com/aws/en/udf/python-udtf and - https://docs.databricks.com/aws/en/dev-tools/databricks-connect/python/udf - Task 5 will implement `st_asmvt_pyramid` as a Python UDTF (2B). The 2A fallback is retired. -- **Encoder byte differences** — handled by decoded-feature parity (not raw-byte) comparison. -- **Heavy MVT test churn** — switching heavy to native types will change existing Scala MVT - assertions; updating them is in scope. - -## Future exploration (note — not this phase) - -Once the `st_asmvt_pyramid` UDTF proves out **solid performance and confirmed Serverless-safety**, -audit other functions for the same incremental-`yield` UDTF pattern, where it would be a memory-safer -and closer-to-heavy alternative to the current `pandas_udf(ArrayType(...))` + `explode` (array-buffering) -generators: - -- **This vector phase (future specs):** the TIN/elevation generators (`st_triangulate`, - `st_interpolateelevationbbox`, `st_interpolateelevationgeom`) are heavyweight `CollectionGenerator`s → - prime UDTF candidates when those specs are taken on. -- **Existing raster (`pyrx`):** audit any light generator-style / fan-out functions currently realized - as `pandas_udf(ArrayType)` + `explode` (e.g. tiling/subdivision producers) — incremental-`yield` - UDTFs avoid buffering the whole fan-out array in the Python worker. -- **Reputational angle:** incremental-`yield`, Arrow-backed UDTFs are a documentable best practice for - memory-safe fan-out at scale — a point to elevate in the docs/benchmarking narrative once measured. - -Keep the focus on the MVT slice now; this is a tracked follow-up, not in-scope work. diff --git a/docs/superpowers/specs/2026-06-13-pyvx-vectorx-tin-legacy-light-tier-design.md b/docs/superpowers/specs/2026-06-13-pyvx-vectorx-tin-legacy-light-tier-design.md deleted file mode 100644 index 25424b2a7..000000000 --- a/docs/superpowers/specs/2026-06-13-pyvx-vectorx-tin-legacy-light-tier-design.md +++ /dev/null @@ -1,175 +0,0 @@ -# pyvx VectorX TIN + Legacy Light Tier — Design - -**Date:** 2026-06-13 -**Branch:** `pyvx-light` -**Status:** Approved design (pending user review) - -## Goal - -Bring the remaining heavy-tier VectorX `gbx_st_*` functions to light-tier parity in `databricks.labs.gbx.pyvx`, so the light tier is a genuine **exit from heavy** for surveying/DTM and Mosaic-migration workloads — not a partial port. This covers the **TIN block** (`st_triangulate`, `st_interpolateelevationbbox`, `st_interpolateelevationgeom`) and **legacy-geometry migration** (`st_legacyaswkb`). - -Because 0.4.0 VectorX is unreleased, greenfield WIP (no back-compat owed), the design also **modifies the heavy tier** where alignment makes the cross-tier swap seamless: a new `mode` parameter on the TIN functions, and two bug-fixes in `st_legacyaswkb` (preserve Z, preserve polygon holes). - -## Architecture - -Pure-Python/PySpark light tier, Serverless/Spark-Connect safe (only `udf`/`udtf` registration + Column expressions — never `spark.conf.set`, `_jvm`, `sparkContext`, or `.rdd`). Heavy compute primitives stay JVM/JTS; the light tier reimplements them on `scipy` + `shapely` + `numpy`. - -The triangulation engine is the crux: heavy uses JTS `ConformingDelaunayTriangulator` (a **conforming** Delaunay that inserts Steiner points to keep triangles Delaunay while honoring breakline constraints). No permissively-licensed Python library provides constrained/conforming Delaunay over sites + segments: - -- `triangle` (Shewchuk) — non-commercial license, disqualified for Databricks Labs. -- `shapely.constrained_delaunay_triangles` / `mapbox_earcut` — polygon-**interior** tessellators; empirically return *empty* for scattered point sets and ignore interior breakline segments (verified: shapely 2.1.2 / GEOS 3.13.1). Wrong tool for mass-point TIN. -- VTK — permissive and capable, but a ~100MB+ dependency that defeats a "light" tier. - -So the light tier uses **scipy `Delaunay` + a hand-rolled Sloan constraint-recovery step** (forced constraint edges via edge-flipping — a **constrained**, no-Steiner Delaunay). To keep the two tiers aligned by default, the **heavy tier gains the same constrained mode** and the conforming (Steiner) behavior becomes an explicit opt-in. - -## Tech Stack - -- Light: Python 3.12, `scipy.spatial.Delaunay`, `shapely` (2.x, WKB/EWKB/WKT I/O), `numpy`, PySpark `@udf`/`@udtf`. -- Heavy: Scala 2.13 / Spark 4.0 / JTS (existing); constrained path on JTS `QuadEdgeSubdivision` / `IncrementalDelaunayTriangulator`. -- New `[light]` dependency: **`scipy`** (BSD-3, permissive). `shapely`/`numpy` already present. - ---- - -## Component 1 — `mode` parameter (cross-tier TIN alignment) - -A new trailing `mode: String` parameter on `st_triangulate`, `st_interpolateelevationbbox`, and `st_interpolateelevationgeom`, in **both tiers**, following the established H3-modes playbook (backward-compatible arity — existing call arities keep working; `mode` defaults to `"constrained"`). - -| `mode` | Semantics | Light | Heavy | -|---|---|---|---| -| `"constrained"` **(default)** | Forced constraint edges, **no Steiner points** | ✅ scipy + Sloan | ✅ JTS QuadEdge + constraint recovery | -| `"conforming"` | JTS conforming Delaunay (**adds Steiner points**); `splitPointFinder` is meaningful | ❌ raises `NotImplementedError` (points to heavy) | ✅ `ConformingDelaunayTriangulator` (today's behavior) | - -Rationale: `"constrained"` is producible in **both** tiers, so the default swap is seamless. `"conforming"` is the richer JVM-only capability, now an explicit opt-in rather than a silent tier difference. - -`splitPointFinder` (`MIDPOINT`/`NONENCROACHING`) only affects Steiner placement, so it is meaningful **only** under `"conforming"`. Under `"constrained"` it is accepted for signature compatibility and documented as a no-op. - -Validation: unknown `mode` raises `IllegalArgumentException` (heavy) / `ValueError` (light) listing the valid values, mirroring `rst_h3_tessellate`. - ---- - -## Component 2 — Light TIN backend (`pyvx/_tin.py`) - -Pure-Python, Spark-free, the heavily-tested core. Geometry I/O via shapely. - -**Triangulation (`mode="constrained"`):** -1. Parse mass points (`ARRAY` → shapely geometries; collect XYZ vertices). Apply `mergeTolerance` vertex merge/dedup (snap near-coincident vertices). -2. `scipy.spatial.Delaunay(points_xy)` → base unconstrained triangulation. -3. **Sloan constraint recovery** for each breakline segment not already an edge: - - Robust `orient2d` predicate (determinant form). - - Walk the triangulation from segment start to end via `Delaunay.find_simplex` / `.neighbors` to find intersected edges. - - Iteratively flip intersected edges (only across convex quads) until the segment is an edge sequence. - - Termination guard (bounded flip count; raise on non-termination rather than loop). - - Handle degenerate/cocircular/near-collinear cases explicitly. -4. **Z-snap (`snapTolerance`)**: vertices within `snapTolerance` of a constraint line get Z overwritten by linear interpolation along that line (matches heavy `LengthIndexedLine` post-process). - -**Interpolation:** barycentric Z within the (constrained) TIN triangle containing each query point. Query points outside the convex hull → **no output row** (matches heavy's silent drop). NaN Z (degenerate triangle) → dropped. - -**Grid generation (replicate heavy exactly):** -- **bbox**: `xRes=(xmax-xmin)/widthPx`, `yRes=(ymax-ymin)/heightPx`; center `x=xmin+(i+0.5)*xRes`, `y=ymin+(j+0.5)*yRes`; **column-major** order (`i` over `[0,widthPx)` slowest, `j` over `[0,heightPx)` fastest); points carry the `srid` param. -- **geom**: `x=originX+(i+0.5)*cellSizeX`, `y=originY+(j+0.5)*cellSizeY`; `cellSizeY` may be negative (y-down raster convention); column-major; SRID taken from the `gridOrigin` geometry (EWKB/EWKT → non-zero, plain → 0). - -**Output encoding:** triangles → 2D OGC WKB (`shapely.to_wkb`, `output_dimension=2`). Elevation points → 3D `POINT Z` ISO WKB (`shapely.to_wkb`, `output_dimension=3`), SRID set on the geometry but **not** embedded (matches heavy `toWKB3` = ISO WKB-Z, not EWKB). - -`mode="conforming"` → `NotImplementedError` with guidance to use the heavy tier. - -## Component 3 — Light TIN functions (`pyvx/functions.py`, registered via `register(spark)`) - -All generators are `@udtf` (matching heavy's `CollectionGenerator` contract and the existing pyvx UDTF pattern; invoked via SQL `LATERAL`). The per-row input array **is** the bounded local point set, so no internal spatial grouping is needed. - -- `st_triangulate(points_geom, breaklines_geom, merge_tolerance, snap_tolerance, split_point_finder, mode="constrained")` → UDTF, output `STRUCT` (one row per triangle). -- `st_interpolateelevationbbox(points_geom, breaklines_geom, merge_tolerance, snap_tolerance, split_point_finder, xmin, ymin, xmax, ymax, width_px, height_px, srid, mode="constrained")` → UDTF, output `STRUCT`. -- `st_interpolateelevationgeom(points_geom, breaklines_geom, merge_tolerance, snap_tolerance, split_point_finder, grid_origin, grid_cols, grid_rows, cell_size_x, cell_size_y, mode="constrained")` → UDTF, output `STRUCT`. - -Null/empty `points` → empty iterator. Non-empty `breaklines` with `mode="conforming"` → `NotImplementedError`. - -## Component 4 — Legacy decode (`pyvx/_legacy.py` + `st_legacyaswkb`) - -Input legacy Mosaic struct: -``` -STRUCT< - typeId INT, -- 1 POINT, 2 MULTIPOINT, 3 LINESTRING, - -- 4 MULTILINESTRING, 5 POLYGON, 6 MULTIPOLYGON, - -- 7 LINEARRING; 8 GEOMETRYCOLLECTION → error - srid INT, - boundaries ARRAY>>, -- rings; each coord len 2 (XY) or 3 (XYZ) - holes ARRAY>>> -- interior rings per polygon -> -``` - -`_legacy.py` decodes the struct into a shapely geometry, then emits **ISO WKB preserving Z** (`shapely.to_wkb`, default `output_dimension=3` / `flavor="iso"` — Z written when present, 2D otherwise) and **preserving interior rings (holes)**. This fixes the two heavy bugs: `toWKB` (2D) dropped Z, and a `// TODO` silently dropped holes. - -Single function, `st_legacyaswkb`, a **scalar UDF**, both tiers: - -| Function | Output | Notes | -|---|---|---| -| `st_legacyaswkb` | ISO WKB, **Z preserved**, **holes preserved** | SRID **not** embedded — it lives in the source struct's `srid` field and the migrator applies it at ingestion (`ST_GeomFromWKB(wkb, srid)` / set CRS) | - -No EWKB variant: Z (the only extra dimension the legacy format carries) is preserved by plain ISO WKB; embedding SRID-in-bytes would be the sole reason for EWKB and is deferred as YAGNI (the SRID is available separately). M does not exist in the source format and is unsupported by GEOS — explicitly out of scope. - -`GEOMETRYCOLLECTION` (typeId 8) raises (matches heavy). Null input → null output. - -## Component 5 — Heavy-tier changes (Scala) - -- **`ST_Triangulate.scala`, `ST_InterpolateElevationBBox.scala`, `ST_InterpolateElevationGeom.scala`**: add the `mode` param (FunctionBuilder arity arms for backward-compatible defaulting to `"constrained"`); implement the `"constrained"` path via JTS `QuadEdgeSubdivision` / `IncrementalDelaunayTriangulator` + constraint recovery (no Steiner); keep `"conforming"` on `ConformingDelaunayTriangulator`. **Default behavior changes** to constrained (acceptable — unreleased WIP). -- **`InternalGeometry.scala`** (`jts/legacy`): fix the dropped-holes TODO for POLYGON/MULTIPOLYGON in `toJTS`. -- **`ST_LegacyAsWKB.scala`** (`jts/legacy`): switch the encoder from `JTS.toWKB` (2D) to `JTS.toWKB3` (Z-preserving ISO WKB), so heavy matches light. SRID still not embedded. - -## Data flow - -``` -mass points (ARRAY) ─┐ -breaklines (ARRAY) ─┼─→ _tin.triangulate(mode) ─→ constrained TIN -tolerances, mode ─┘ │ - ├─ st_triangulate: emit each triangle (2D WKB) -grid spec (bbox | origin) ────────────────────────────────┴─→ interpolate Z at cell centers - → emit in-hull POINT Z (3D WKB), drop outside-hull - -legacy struct ─→ _legacy.decode (Z + holes preserved) ─→ shapely geom ─→ st_legacyaswkb (ISO WKB, Z preserved) -``` - -## Error handling - -- Unknown `mode` → `ValueError`/`IllegalArgumentException` listing valid values. -- `mode="conforming"` in light → `NotImplementedError` pointing to heavy. -- Non-LineString breaklines for the interpolate functions → error (matches heavy's runtime type check). -- Sloan non-termination → raise (never silently return an unconstrained result). -- `GEOMETRYCOLLECTION` legacy input → raise. -- Empty/null points → empty iterator (generators) / null (scalar). - -## Testing strategy - -**Light unit (`_tin.py`, `_legacy.py` — Spark-free):** -- Delaunay correctness on scattered points; degenerate (collinear, cocircular, single point, duplicates), empty. -- Sloan recovery: every constraint segment is present as a triangle-edge sequence; flip termination; Z-snap correctness. -- Grid generation: exact center coordinates + column-major ordering for both bbox and geom specs; negative `cellSizeY`. -- Interpolation: known barycentric values; outside-hull drop; NaN drop. -- Legacy decode: each typeId; holes preserved; Z preserved (ISO WKB dim-3 round-trips XYZ); GEOMETRYCOLLECTION raises. - -**Cross-tier parity (JAR-gated, `test/pyvx/`):** -- **Legacy**: light↔heavy decoded-geometry equality for `st_legacyaswkb` incl. a **holed polygon** and a **Z-valued (XYZ)** geometry (both post-fix). -- **No-breakline TIN**: triangle-set / interpolated-surface equality within float tolerance (Delaunay ~unique). -- **With-breakline TIN** (`mode="constrained"` both tiers): assert (a) each constraint segment present as triangle edges, (b) interpolated Z at sample points within tolerance — **not** triangle-identity (Qhull vs JTS cocircular tie-breaks differ). -- **`mode="conforming"`**: heavy behavior test; light raises. - -**Heavy Scala:** constrained-vs-conforming mode tests; legacy holes-preserved + Z-preserved (`toWKB3`) tests. - -**Binding parity:** `st_legacyaswkb`, `st_triangulate`, `st_interpolateelevationbbox`, `st_interpolateelevationgeom` present in `registered_functions.txt`, Python `functions.py` (both tiers), and `function-info.json`. - -## Phasing - -1. **Legacy first** — `st_legacyaswkb` (Z-preserve + holes fixes), both tiers. Independent, lower-risk, establishes the non-MVT scalar-UDF pattern. -2. **TIN block second** — `_tin.py` backend (TDD the Sloan core hardest) → 3 light UDTFs → heavy `mode` (constrained path) → cross-tier parity tests. - -## Out of scope - -- M dimension (absent from the legacy format; unsupported by GEOS). -- `"conforming"` mode in the light tier (heavy-only opt-in by design). -- Byte-identical triangle parity with breaklines (different-but-valid triangulations; surface-closeness is the contract). -- Vector file readers/writers (a separate phase). - -## Documentation - -- pyvx VectorX functions page: add the 4 functions with light/heavy tabs, the `mode` param, and the constrained-vs-conforming + breakline divergence explainer (defensible-divergence framing, like the H3-Java 3.7.0 note). -- Legacy section: migration framing for Mosaic customers; `st_legacyaswkb` preserves Z + holes; SRID applied separately at ingestion; M out of scope. -- `function-info.json` examples for all 5. -- No internal vocabulary (no "wave N"); justify by user utility, not Mosaic parity. diff --git a/docs/superpowers/specs/2026-06-14-pmtiles-agg-light-tier-design.md b/docs/superpowers/specs/2026-06-14-pmtiles-agg-light-tier-design.md deleted file mode 100644 index 67afc5603..000000000 --- a/docs/superpowers/specs/2026-06-14-pmtiles-agg-light-tier-design.md +++ /dev/null @@ -1,210 +0,0 @@ -# Lightweight `gbx_pmtiles_agg` — Design - -**Date:** 2026-06-14 -**Branch:** `pygx-light` (add-on; not part of pygx Phase 1/2) -**Status:** Approved design — ready for implementation plan. - -## Goal - -Add a lightweight (pure-Python / Serverless-safe) implementation of `gbx_pmtiles_agg` -so the function — today heavyweight-only — works in the lightweight tier as a one-line -behavioural swap. A grouped aggregate that folds a group of `(bytes, z, x, y)` map tiles -into a single PMTiles v3 archive (BINARY), reusing the archive-assembly machinery already -built for the light `pmtiles_gbx` writer. - -## Background — the heavy contract (parity target) - -Heavy `gbx_pmtiles_agg` is a Scala `TypedImperativeAggregate` -(`src/main/scala/com/databricks/labs/gbx/pmtiles/PMTiles_Agg.scala`): - -- **Args:** `(bytes BINARY, z INT, x INT, y INT, metadata_json STRING = "{}")` — 4-arg and - 5-arg forms. `z/x/y` are coerced from `Long` (PySpark sends Python `int` as `LongType`). -- **Output:** `BINARY` — a PMTiles v3 archive. -- **Build (`PMTilesV3Encoder`):** accumulate `(z,x,y,bytes)` (100 MiB cap) → sort by Hilbert - TileID (`hilbertId(z,x,y)`) → SHA-256 dedup identical payloads → RLE root directory - (varint delta-encoded) → 127-byte header + root dir + metadata JSON + tile data. Internal - (directory) compression is **NONE**. No leaf directories (throws if the root directory - would exceed 16,257 bytes). Tile type auto-detected from the first non-null payload's magic - bytes (PNG / JPEG / WebP / else MVT). -- **Python binding** (`python/geobrix/src/databricks/labs/gbx/pmtiles/functions.py`): - `register(spark)` registers the JVM aggregate via the `register_ds` DataSource; - `pmtiles_agg(bytes_col, z, x, y, metadata_json=None)` returns - `f.call_function("gbx_pmtiles_agg", ...)`. - -`gbx_pmtiles_agg` is **already** in `docs/tests-function-info/registered_functions.txt` -(line 157). The light tier introduces **no new SQL name** — so binding-parity -(`gbx:test:bindings`) and `function-info.json` need no new entry. This is purely a second -implementation + register path behind the same name. - -## The reuse asset - -A complete light PMTiles archive assembler already exists under -`databricks.labs.gbx.ds.tiles` (it powers the `pmtiles_gbx` DataSource writer): - -- `ds/tiles/backend.py` — `PMTilesBackend.assemble(sorted_tiles, header_info, out_path)` - drives `pmtiles.writer.Writer` (`write_tile(tileid, data)` → `finalize(header_dict, metadata)`). -- `ds/tiles/_header.py` — `sniff_tile_type(data)`, `build_header_info(...)`, and - `HeaderInfo.header_dict()` (sets `internal_compression=Compression.GZIP`). -- `ds/tiles/shard.py` — `stream_sorted(entries)` sorts `(tileid, bytes)` ascending. -- `pmtiles.tile.zxy_to_tileid` gives the spec Hilbert TileID (same ordering as heavy's - `hilbertId`). -- `pmtiles>=3.4,<4` is already in the `[light]` extra. - -`pmtiles.writer.Writer` accepts any file-like object, so it writes to an `io.BytesIO` -for an in-memory BINARY result (no temp file on the worker for the *archive* — the lib -still uses a `tempfile` internally for its tile-data buffer before finalize, which is fine). - -## Architecture - -PMTiles is **format-agnostic** — it archives raster *or* vector tiles (PNG/JPEG/WebP/MVT) — -so the light implementation does **not** belong to RasterX/`pyrx` or VectorX/`pyvx`. - -1. **Tier-neutral home.** The light implementation lives in the existing - `databricks.labs.gbx.pmtiles` package — the same package as the heavy binding — in a new - submodule `_agg_light.py`. The user-facing Column wrapper `pmtiles_agg` stays in - `databricks.labs.gbx.pmtiles.functions` (one import path for both tiers; register is the - only difference). - -2. **Shared light register helper.** `register_pmtiles_agg(spark)` (in the light module) - does `spark.udf.register("gbx_pmtiles_agg", _pmtiles_agg_udf)`. It is called by **both** - `pyrx.functions.register` and `pyvx.functions.register`, so registering either lightweight - tier installs `gbx_pmtiles_agg`. It is also exposed standalone - (`from databricks.labs.gbx.pmtiles import register_pmtiles_agg`) for users who want only it. - `pygx` is **not** wired (grid cells, not tiles). The heavy `pmtiles.functions.register` - (JVM `register_ds` path) is untouched. - -3. **Idempotent.** Registering both pyrx and pyvx in one session re-registers the same name - harmlessly (`spark.udf.register` overwrites). No double-registration error. - -### Component / file map - -| File | Responsibility | -|---|---| -| `python/geobrix/src/databricks/labs/gbx/pmtiles/_agg_light.py` (new) | `_pmtiles_agg_udf` grouped-agg `pandas_udf` + `_assemble_archive(tiles, metadata)` BytesIO assembler + `register_pmtiles_agg(spark)` | -| `python/geobrix/src/databricks/labs/gbx/pmtiles/__init__.py` (modify) | re-export `register_pmtiles_agg` | -| `python/geobrix/src/databricks/labs/gbx/pmtiles/functions.py` (modify) | keep tier-neutral `pmtiles_agg` Column wrapper (see contingency below) | -| `python/geobrix/src/databricks/labs/gbx/pyrx/functions.py` (modify) | call `register_pmtiles_agg(spark)` at end of `register` | -| `python/geobrix/src/databricks/labs/gbx/pyvx/functions.py` (modify) | call `register_pmtiles_agg(spark)` at end of `register` | -| `python/geobrix/test/pmtiles/test_agg_light_core.py` (new) | Spark-free assembler unit tests | -| `python/geobrix/test/pmtiles/test_agg_light_udf.py` (new) | registered-UDF tests via spark fixture | -| `python/geobrix/test/pmtiles/test_parity_pmtiles_agg.py` (new) | JAR-gated cross-tier decoded parity | - -## The grouped-agg UDF - -Mirror the pyvx `_asmvt_udf` GROUPED_AGG pattern (`(pd.Series, ...) -> bytes`, detected by -PySpark as Series-to-Scalar): - -```python -@pandas_udf(BinaryType()) -def _pmtiles_agg_udf( - data: pd.Series, z: pd.Series, x: pd.Series, y: pd.Series, metadata_json: pd.Series -) -> Optional[bytes]: - return _assemble_archive(data, z, x, y, metadata_json) -``` - -`_assemble_archive`: -1. Drop rows where `data` is null (mirror heavy's non-null handling). If none remain → return `None`. -2. Resolve metadata: first non-null `metadata_json` in the group, parsed JSON; default `{}`. -3. Sniff tile type from the first non-null payload (`ds.tiles._header.sniff_tile_type`). -4. Build `(zxy_to_tileid(z, x, y), bytes)` tuples; sort ascending by tileid (== Hilbert order). -5. Build `HeaderInfo` via the existing `ds.tiles` helpers (zoom range, bbox, tile type, - `internal_compression=GZIP`). -6. Write to `io.BytesIO` through `pmtiles.writer.Writer` (the same call sequence - `PMTilesBackend.assemble` uses); return `buf.getvalue()`. - -**Memory cap.** Mirror heavy's 100 MiB accumulation cap with the same failure semantics: -raise a clear error if the group's total payload bytes exceed the cap, so the failure mode -matches across tiers. (A grouped aggregate materialises the whole group on one worker; the -cap is the guardrail.) - -**Dedup.** Heavy SHA-256-dedups identical payloads; this is an internal size optimisation that -does not change decoded output. The light tier relies on the `pmtiles` lib's directory RLE and -does not add explicit dedup — decoded-tile parity is unaffected. - -**Leaf directories (light advantage).** The `pmtiles` lib supports leaf directories -(`build_roots_leaves`/`optimize_directories`), so the light tier is **not** subject to heavy's -16,257-byte root-directory ceiling — it handles archives with more tiles than heavy can. Parity -is asserted only over the range where heavy succeeds; this is documented as a light-tier -capability, not a divergence to "fix." - -## Column wrapper (tier-neutral) - -The existing wrapper does `f.call_function("gbx_pmtiles_agg", ...)`. **Preferred:** keep it -unchanged — once the light UDF is registered under that SQL name, `call_function` invokes it in -a `groupBy().agg(...)` context for both tiers, so one wrapper serves both. - -**Contingency (decided by a TDD test in Task 1 of the plan):** if `call_function` does **not** -compose with a registered pandas grouped-agg UDF inside `.agg()`, fall back to the pyvx -`quadbin_cellunion_agg` pattern — `register_pmtiles_agg` stashes the udf object and the light -wrapper calls it directly (`_pmtiles_agg_udf(...)`), while heavy keeps `call_function`. The -wrapper picks the path based on which tier is registered. The plan must verify the wrapper works -in `.agg()` on both tiers before declaring done. - -## Parity contract - -**Decoded-tile parity, not byte-identical** — the established contract from the `pmtiles_gbx` -writer parity test (`test_pmtiles_parity.py`). Decode both archives with -`pmtiles.reader.Reader` and compare: -- the `{(z, x, y): bytes}` tile dictionaries are equal, and -- the metadata round-trips equal. - -Byte-level archives differ by design: heavy uses NONE internal (directory) compression, light -uses GZIP (the `pmtiles` lib default). Both are spec-valid and decode identically. - -## Serverless / Connect safety - -Light code uses only `pmtiles` + standard library + the `ds.tiles` helpers behind a -`spark.udf.register` + Column expressions. No `_jvm`, `sparkContext`, `.rdd`, or -`spark.conf.set`. Covered by the existing Serverless guard test pattern. - -## Testing (TDD) - -1. **Spark-free core** (`test_agg_light_core.py`): `_assemble_archive` on hand-built tile lists - — single tile, multiple tiles across zooms, MVT and PNG payloads, metadata round-trip, - null-payload handling, empty group → `None`, cap-exceeded raises. Decode with - `pmtiles.reader.Reader` and assert the tile dict + metadata. -2. **Registered UDF** (`test_agg_light_udf.py`, spark fixture, Docker): register via the light - helper; `df.groupBy(...).agg(pmtiles_agg(...))`; decode the result; assert wrapper works in - `.agg()` (the `call_function` contingency check); assert `pyrx.register` and `pyvx.register` - both install `gbx_pmtiles_agg`. -3. **Cross-tier parity** (`test_parity_pmtiles_agg.py`, JAR-gated): register light then heavy on - a shared corpus of MVT tiles (include a multi-zoom group and a group with a POLYGON-derived - MVT, not just points); assert decoded tile-dict + metadata equality. - -## Bindings & docs - -- **Bindings:** no new SQL name → `registered_functions.txt` and `function-info.json` - unchanged. (`gbx:test:bindings` already passes for `gbx_pmtiles_agg`.) Confirm the - function-info example still applies to both tiers. -- **Docs (per-function tiering):** `docs/docs/api/pmtiles-functions.mdx` is page-level - `` today. Change `gbx_pmtiles_agg` to ` ` with a - `:::note Lightweight tier` lib-attribution admonition (Powered by the **pmtiles** package), - matching the raster/quadbin convention; leave any genuinely heavy-only pmtiles entries as - `` (the page can no longer be page-level heavy). -- `docs/docs/api/execution-tiers.mdx`: move `gbx_pmtiles_agg` out of the heavy-only column. -- `docs/docs/api/performance.mdx` + `benchmarking.mdx`: add the pmtiles_agg light-vs-heavy - result (see bench) — per the standing "bench changes must update benchmarking.mdx" rule. -- Reflect light pmtiles_agg availability wherever the lightweight package set is enumerated - (the install/quick-start tier framing already covers RasterX/VectorX/GridX; pmtiles_agg rides - along via the pyrx/pyvx register and needs no separate landing-page bullet unless one exists). - -## Benchmark - -Add a `pmtiles_agg` light-vs-heavy leg to the bench harness (mirror the existing PMTiles -writer bench): a corpus of MVT tiles grouped into archives; measure grouped-agg wall time per -tier and assert decoded parity. Capture on the cluster, update `benchmarking.mdx`. - -## Out of scope - -- Spatial sharding / multi-archive catalog output (separate PMTiles sharding effort). -- A light PMTiles **reader** SQL function (readers live in the DataSource layer). -- `pygx` wiring (grid, not tiles). -- Any change to the heavy encoder or the heavy `register` path. - -## Open / risk items - -- **`call_function` vs grouped-agg pandas UDF in `.agg()`** — resolved by the Task-1 TDD test; - contingency documented above. -- **Tile-type heterogeneity** — heavy sniffs one type for the archive from the first payload; the - light tier follows the same single-type assumption. Mixed tile types in one group are not a - supported input on either tier. diff --git a/docs/superpowers/specs/2026-06-14-pygx-custom-gridding-light-tier-design.md b/docs/superpowers/specs/2026-06-14-pygx-custom-gridding-light-tier-design.md deleted file mode 100644 index 5be99808a..000000000 --- a/docs/superpowers/specs/2026-06-14-pygx-custom-gridding-light-tier-design.md +++ /dev/null @@ -1,171 +0,0 @@ -# pygx Light Custom-Gridding Tier — Design - -**Date:** 2026-06-14 -**Branch:** `pygx-light` -**Status:** **IMPLEMENTED 2026-06-15** (was APPROVED 2026-06-14). The lightweight `pygx` custom-grid tier shipped on `pygx-light`: all 7 `gbx_custom_*` functions registered in both tiers, cross-tier exact parity green (cell-id/set, no-SRID geometry, all-4-encodings, Y-NaN guard locked in both tiers), heavy `pointToCellID` NaN-Y typo fixed. This supersedes the "custom gridding is heavyweight-only" caveat across the docs — **GridX is now fully 1:1 light↔heavy**. The four open questions were resolved by the user (see **Resolved decisions** at the bottom); the cluster benchmark leg is staged and pending a coordinated run. - -## Goal / context - -Port the 7 heavyweight **custom-gridding** functions (`gbx_custom_*`) to the lightweight, pure-Python/PySpark `databricks.labs.gbx.pygx` tier so that **GridX reaches full 1:1 light↔heavy parity** — every GridX function (quadbin, BNG, **and custom**) available in both tiers with exact result parity. This removes the last GridX caveat: the "light vs heavy not 1:1 (custom gridding heavy-only)" note that pygx, the execution-tiers page, the gridx-functions reference, and the docs landing all currently carry. - -This is the **third and final pygx phase**, after quadbin (phase 1, done) and BNG (phase 2, done). It was originally declared **out of scope** in the pygx light-tier spec (`2026-06-14-pygx-light-tier-design.md`, "Out of scope" → custom gridding stays heavyweight-only). The user (2026-06-14) reversed that decision to close the parity gap **in this branch, after BNG**. BNG phase 2 is now complete, so this is the immediate next pygx work. - -**Net-new-in-0.4.0 status.** The `gbx_custom_*` family is net-new in 0.4.0 (it has no 0.3.0 predecessor), so there is no back-compat obligation; heavy custom-gridding behavior is the parity reference and stays fixed (except any validated bug fix — see open questions). No interim/planning vocabulary surfaces in user-facing docs. - -Goals carried from all prior lightweight work: (a) function parity, (b) exact result parity, (c) competitive performance, (d) explicit attention wherever light is slower than heavy, (e) coherence, (f) captured benchmarking, (g) consistent docs on every surface, (h) prominent-surface visibility. - -## What custom gridding *is* (heavy recon) - -Unlike quadbin (a fixed global CARTO grid) and BNG (a fixed British grid in EPSG:27700), a **custom grid is a user-defined, arbitrary regular rectangular grid** — the caller defines the grid's extent, root cell size, recursive split factor, and (optionally) a CRS. There is no global singleton: every operation is parameterized by a **grid-spec struct** the user builds first with `gbx_custom_grid`. - -Heavy source (read for this spec): -- `gridx/grid/CustomGridSystem.scala` — the canonical grid math (≈340 lines): cell-ID bit-packing, cell↔coordinate mapping, polyfill, kRing/kLoop, cell→geometry, centroid, distance. -- `gridx/grid/GridConf.scala` — the grid config case class + derived quantities (`bitsPerResolution`, `maxResolution`, `rootCellCountX/Y`). -- `gridx/custom/Custom_GridSpec.scala` — the 8-field STRUCT schema + `systemFromRow` decoder + Int/Long-tolerant coercion. -- `gridx/custom/Custom_*.scala` (7 expression classes) + `gridx/custom/functions.scala` (registration). - -### The grid model (from `GridConf` + `CustomGridSystem`) - -A grid is fully described by 8 integer parameters (the `gbx_custom_grid` STRUCT): - -| Field | Type | Meaning | Validation (heavy `Custom_Grid.eval`) | -|---|---|---|---| -| `bound_x_min` | LONG | grid extent min X (native CRS units) | — | -| `bound_x_max` | LONG | grid extent max X | `> bound_x_min` | -| `bound_y_min` | LONG | grid extent min Y | — | -| `bound_y_max` | LONG | grid extent max Y | `> bound_y_min` | -| `cell_splits` | INT | subdivisions per axis per resolution level | `>= 2` | -| `root_cell_size_x` | INT | root (resolution-0) cell width in CRS units | `> 0` | -| `root_cell_size_y` | INT | root cell height in CRS units | `> 0` | -| `srid` | INT | EPSG SRID of the grid CRS; **-1 == no CRS** | (defaulted to -1 when 7 args) | - -Derived (in `GridConf`): -- `resBits = 8` (top 8 bits of the cell ID hold the resolution), `idBits = 56` (low 56 bits hold the cell position). -- `subCellsCount = cell_splits²`; `bitsPerResolution = ceil(log2(subCellsCount))`; `maxResolution = min(20, floor(56 / bitsPerResolution))`. -- `rootCellCountX = ceil((bound_x_max - bound_x_min) / root_cell_size_x)`; `rootCellCountY` analogously. - -**Resolution** is an integer level `0..maxResolution` (0 = root cells of size `root_cell_size_{x,y}`; each level divides cell size by `cell_splits`). This is unlike BNG's ±1..±6 index/`resolutionMap` keys and unlike quadbin's 0..26 — custom resolution is a plain level index validated against `maxResolution`. - -### Cell-ID encoding (must port bit-exact) - -``` -cellId = (resolution.toLong << idBits) | cellPosition # idBits = 56 -cellPosition = cellPosY * totalCellsX(res) + cellPosX # row-major -totalCellsX(res) = rootCellCountX * cell_splits^res # (Y analogous) -getCellResolution(cellId) = (cellId >> 56).toInt -getCellPosition(cellId) = cellId & 0x00ffffffffffffffL -cellWidth(res) = root_cell_size_x / cell_splits^res # (height analogous) -``` - -Coordinate → cell position: `cellPosX = floor((x - bound_x_min) / cellWidth(res))` (Y analogous). Cell position → cell origin: `x = cellPosX * cellWidth + bound_x_min`. Cell → polygon: the axis-aligned rectangle `[x, x+cellWidth] × [y, y+cellHeight]` as a closed 5-point ring. Centroid: the geometric center (`x + cellWidth/2`, `y + cellHeight/2`). - -`pointToCellID` enforces bounds: `bound_x_min <= x < bound_x_max`, `bound_y_min <= y < bound_y_max`, `res <= maxResolution`, no-NaN — raising on violation (light must match these as `ValueError`). - -## The 7 functions — signatures, return types, light impl approach - -All cell IDs are **BIGINT** (`LongType`). Geometry outputs use `JTS.toWKB`/`JTS.toWKT` — **the 2D, no-SRID variant** (`JTS.scala:159`, *not* `toEWKB`). So custom geometry WKB carries **no SRID**, like BNG (and unlike quadbin's EWKB). The `srid` field in the grid spec is metadata only — it is **not** stamped into output geometries by the heavy code. - -| # | Function (SQL) | Heavy signature | Return type | Light shape | Light implementation | -|---|---|---|---|---|---| -| 1 | `gbx_custom_grid` | `(xMin, xMax, yMin, yMax, cellSplits, rootCellSizeX, rootCellSizeY[, srid])` — 7 or 8 INT/LONG args | `STRUCT` (8 fields, see schema above) | **plain `@udf` / direct Column** (foldable struct builder) | Build the identical 8-field struct with the **same validation** (`xMax>xMin`, `yMax>yMin`, `cell_splits>=2`, root sizes `>0`). No geometry, no cell math. See "grid-spec construction" below. | -| 2 | `gbx_custom_pointascell` | `(point: BINARY\|STRING, grid: STRUCT, resolution: INT\|LONG)` | `BIGINT` | **`pandas_udf`** (scalar, bounded; vectorizable) | parse_geom → coordinate → `point_to_cell_id(x, y, res, conf)` (bit-pack). NULL point/grid → NULL. | -| 3 | `gbx_custom_cellaswkb` | `(cell: BIGINT, grid: STRUCT)` | `BINARY` (WKB polygon, **no SRID**) | **`pandas_udf`** (bounded scalar) | `cell_id_to_polygon(cell, conf)` → shapely → `to_wkb()` (no SRID). | -| 4 | `gbx_custom_cellaswkt` | `(cell: BIGINT, grid: STRUCT)` | `STRING` (WKT polygon) | **`pandas_udf`** (bounded scalar) | same polygon → `to_wkt()`. | -| 5 | `gbx_custom_centroid` | `(cell: BIGINT, grid: STRUCT)` | `BINARY` (WKB point, **no SRID**) | **`pandas_udf`** (bounded scalar) | cell center point → shapely Point → `to_wkb()`. | -| 6 | `gbx_custom_polyfill` | `(geom: BINARY\|STRING, grid: STRUCT, resolution: INT\|LONG)` | `ARRAY` | **plain `@udf`** (variable-length array, row-by-row, scale-safe) | bbox of geom → candidate cell-center grid → keep centers with `geom.contains(Point)` → map to cell IDs. **Centroid-containment semantics** (exact port of `CustomGridSystem.polyfill`). NULL geom → NULL. | -| 7 | `gbx_custom_kring` | `(cell: BIGINT, grid: STRUCT, k: INT\|LONG)` | `ARRAY` | **plain `@udf`** (variable-length array, row-by-row) | decode cell → posX/posY → Chebyshev square `[posX±k]×[posY±k]` **clamped to `[0, totalCells]`** → map back to cell IDs (exact port of `CustomGridSystem.kRing`). | - -**No explode UDTFs and no aggregators in the custom family** — the heavy `gbx_custom_*` set has none (confirmed against `Custom_*.scala` + `registered_functions.txt`). So there is **no grouped-agg BINARY-vs-STRUCT deviation** to document for custom (unlike BNG/quadbin), and no `*_agg` / `*explode` light path. - -## Architecture - -Mirrors the established pygx shape (pure-Python/PySpark, **Serverless / Spark-Connect safe** — only `spark.udf.register` + Column expressions; never `spark.conf.set`, `_jvm`, `sparkContext`, or `.rdd`). Function names are identical to heavy (`custom_*`) and SQL names register under the same `gbx_custom_*` names, so swapping tiers is a one-line import change. - -**Files:** - -| File | Action | Responsibility | -|---|---|---| -| `pygx/_custom.py` | **new** | Pure-Python port of `CustomGridSystem`/`GridConf`: a `CustomGridConf` dataclass (8 fields + derived `maxResolution`, `rootCellCount{X,Y}`, `idBits=56`), and the grid-math functions `point_to_cell_id`, `cell_id_to_polygon`, `cell_id_to_centroid`, `polyfill`, `k_ring`, plus the bit-pack/unpack helpers. Spark-free, shapely for geometry. | -| `pygx/functions.py` | **extend** | Add the 7 `gbx_custom_*` UDF definitions, their `spark.udf.register(...)` calls in `register()`, and the 7 `custom_*` Column wrappers. Reuse the existing `_col` / `ColLike` helpers. | -| `pygx/_geom.py` | **reuse** | `parse_geom` for the `point`/`geom` inputs of `pointascell` and `polyfill` (WKB/EWKB/WKT/EWKT both tiers — cross-ST geom-input consistency). | -| `pygx/_serde.py` | **extend** | Add `CUSTOM_GRID_SCHEMA` (the 8-field STRUCT, matching `Custom_GridSpec.gridStructType` field names/types/nullability exactly: `bound_x_min/x_max/y_min/y_max` LONG non-null, `cell_splits`/`root_cell_size_x`/`root_cell_size_y`/`srid` INT non-null). | -| `pygx/_env.py` | **reuse / maybe extend** | No new dependency (shapely already required). Add an `assert_custom_available()` only if a guard is wanted for symmetry; custom needs nothing quadbin/BNG don't already pull in. | -| `python/geobrix/test/pygx/…` | **new tests** | Spark-free core unit tests → registered-fn tests → JAR-gated cross-tier exact parity (see Testing). | - -**No new dependencies.** Custom gridding is integer/coordinate arithmetic plus shapely (already in the `[light]` extra) for the rectangle/point geometry and `contains` test. The grid math is a direct port — no library equivalent exists or is needed. - -### Grid-spec construction (`gbx_custom_grid`) — the one structural difference from quadbin/BNG - -quadbin and BNG have no config struct; custom does. `gbx_custom_grid` is a **pure foldable struct builder** with the same 8-field schema and the same validation as `Custom_Grid.eval`. Two implementation options (decide in the plan): - -- **(A) Pure Column expression** — `custom_grid(...)` returns `f.struct(f.lit(...).alias("bound_x_min"), ...)` with the validation pushed into the consuming UDFs (they already decode the struct). Pro: no UDF, foldable, cheap, Serverless-trivial. Con: validation surfaces at consume time, not build time (heavy validates at build/eval time). -- **(B) Thin `@udf`** returning `CUSTOM_GRID_SCHEMA` that validates eagerly and raises on bad bounds/splits/sizes, matching heavy's eager `require(...)`. Pro: error parity (fails at `grid(...)` time). Con: a UDF call where a Column expression would do. - -**DECIDED (Q1): option B** — `gbx_custom_grid` is a thin validating `@udf` that eagerly validates (matching heavy's `require(xMax>xMin, ...)` error-at-build-time parity). The struct field names/types/nullability **must** match `CUSTOM_GRID_SCHEMA` so the same struct flows into both light and heavy consumers. - -The consuming light UDFs (`pointascell`, `cellaswkb`, `cellaswkt`, `centroid`, `polyfill`, `kring`) receive the grid spec as a **struct column** → arrives in the UDF as a Row / dict; reconstruct a `CustomGridConf` from its 8 fields (mirroring `Custom_GridSpec.systemFromRow`, Int/Long tolerant since PySpark may send Long for INT literals). - -## Impl-shape assignment (per the established pygx rule) - -The rule (documented in `functions.py`): **scalar/bounded-output → `pandas_udf`** (numpy-vectorized or batched-Arrow win); **variable-length array output → plain `@udf`** (row-by-row, OOM-safe at scale — a scalar `pandas_udf` would buffer a whole Arrow batch of arrays); **explode → `@udtf`** (none here); **grouped-agg → grouped-agg `pandas_udf` returning BINARY** (none here). - -- `gbx_custom_grid` → **plain `@udf` returning the struct** (option B) or a Column-expression builder (option A). Bounded fixed-size struct; no per-row geometry. -- `gbx_custom_pointascell` → **`pandas_udf`** → `LongType`. Bounded scalar; the bit-packing is vectorizable. -- `gbx_custom_cellaswkb` / `gbx_custom_cellaswkt` / `gbx_custom_centroid` → **`pandas_udf`** → `BinaryType` / `StringType` / `BinaryType`. One geometry per row (bounded); the win is the batched Arrow transfer. -- `gbx_custom_polyfill` → **plain `@udf`** → `ArrayType(LongType())`. Variable-length (a large bbox at fine resolution can emit many cells) → row-by-row for scale safety. -- `gbx_custom_kring` → **plain `@udf`** → `ArrayType(LongType())`. Variable-length `(2k+1)²` output → row-by-row. - -## Parity bar - -GridX custom gridding is **deterministic integer/coordinate math**, so the bar matches quadbin/BNG (stronger than pyvx's TIN): - -- **Cell IDs and cell sets: bit-exact.** `pointascell` produces the identical `Long` ID; `polyfill`/`kring` produce identical cell *sets* (no tolerance) — the bit-packing (`res << 56 | pos`), the `totalCellsX/Y` growth, the `floor`-based coordinate→position mapping, and the centroid-containment / Chebyshev-clamp semantics must be ported exactly from `CustomGridSystem`. -- **Geometry outputs (WKB/WKT): within ~1e-6.** `cellaswkb`/`cellaswkt`/`centroid` go through shapely (light) vs JTS (heavy); coordinates match to a relative/absolute tolerance of 1e-6, not byte-identical (ring orientation / coordinate formatting may differ harmlessly). -- **No SRID stamped.** Confirm light `to_wkb()` is called **without** `include_srid` (heavy uses `JTS.toWKB`, the 2D no-SRID variant) so the WKB byte layout class matches heavy — i.e. light must **not** stamp the grid's `srid` into the geometry. The `srid` field is grid metadata only. -- **Bounds/validation errors match.** Out-of-bounds coordinate, `res > maxResolution`, `cell_splits < 2`, `xMax <= xMin`, etc. raise in both tiers (light `ValueError` / `IllegalArgument`-equivalent message). - -## Serverless-safety - -`udf` + Column expressions only. No `spark.conf.set`, `_jvm`, `sparkContext`, `.rdd`, or repartition in the product path (those are bench-harness-only, as for the rest of pygx). The existing `test_serverless_no_spark_config.py` guard already covers `functions.py`; the custom additions live in the same module and inherit it. - -## Testing - -Mirror the quadbin/BNG test layering: - -1. **Spark-free core** (`test/pygx/test_custom_core.py` against `_custom.py`): bit-pack/unpack round-trips (`cell_id_to_(res,posX,posY)` ↔ `(res,posX,posY)_to_cell_id`) across resolutions; `point_to_cell_id` on known fixtures; `cell_id_to_polygon`/`centroid` coordinates; `polyfill` centroid-containment on a known polygon; `k_ring` Chebyshev + boundary clamp; the `GridConf`-derived `maxResolution`/`rootCellCount` formulas; validation raises. -2. **Registered-function tests** (Docker, Spark — `test/pygx/test_custom_functions.py`): each `gbx_custom_*` via the spark fixture, including `gbx_custom_grid` struct shape, NULL propagation, and a `pointascell → cellaswkb → polyfill → kring` round-trip on a small grid. -3. **Cross-tier exact parity** (JAR-gated — `test/pygx/test_parity_custom.py`, mirroring `test_parity_bng.py` / `test_parity_quadbin.py`): register light then heavy (same SQL name, last-wins), build the **same grid spec** in both tiers, assert exact cell-ID / cell-set equality for `pointascell`/`polyfill`/`kring`, and decoded-geometry equality within 1e-6 for `cellaswkb`/`cellaswkt`/`centroid`. Include edge cells (origin cell, max-corner cell), a multi-resolution grid (`cell_splits` 2 and 4), and a grid **with** and **without** a `srid`. -4. **Binding parity** (`gbx:test:bindings`): every `gbx_custom_*` present in `registered_functions.txt` (already is), Python `functions.py` (the new light wrappers), and `function-info.json`. - -## Bindings + docs surfaces to flip (heavy-only → both) - -- **`registered_functions.txt`** — already lists the 7 `gbx_custom_*` names (no change; binding parity already expects them). -- **`function-info.json`** / `gbx:docs:function-info` — examples already exist for the custom functions (heavy); confirm they still validate after the light registration (no placeholder/empty usage). -- **`docs/docs/api/gridx-functions.mdx`** — flip the custom-grid section's `` → `` (lines ~15, ~1027, and each `### gbx_custom_*` per-function badge); update the top-of-page "partially lightweight" sentence to say GridX is **fully** lightweight; add per-function lib-attribution notes (pure-Python port of `CustomGridSystem` + shapely geometry; no SRID stamped). -- **`docs/docs/api/execution-tiers.mdx`** — remove custom-grid from the heavyweight-only list (lines ~45, ~47); GridX becomes fully lightweight; update the "remaining heavyweight-only surfaces" sentence (custom-grid drops off; OGR readers / `conforming` triangulation / heavy `pmtiles` writer remain). -- **`docs/docs/api/performance.mdx`** — add `pygx/_custom.py` to the modules table; note custom gridding in the pygx perf narrative (cell-math competitive; geometry-returning crosses the WKB UDF boundary). -- **`docs/docs/api/benchmarking.mdx`** — fill the Grid tab with light-vs-heavy custom numbers + exact-parity verdict (per the "bench changes → update docs" rule). -- **`README.md`** — flip the GridX bullet to full lightweight availability (quadbin + BNG + custom). -- **Docs landing** (`docs/src/pages/index.js`) — GridX card / heavyweight-only line: custom no longer heavy-only. -- **`docs/docs/intro.mdx`** — if it enumerates GridX tier coverage, include custom. -- **The pygx light-tier spec** (`2026-06-14-pygx-light-tier-design.md`) — its "Out of scope" custom-gridding note is superseded by this spec; add a forward-reference there on completion (or note here that this spec supersedes that bullet). - -Voice: no internal/planning vocabulary; justify by user utility, not Mosaic parity; custom is net-new in 0.4.0 so no back-compat details surface publicly. - -## Bench - -Add a `--grid-custom-only` launcher leg mirroring `--grid-bng-only` / `--grid-quadbin-only` (`bench/cluster.py` ~line 198): generate a custom grid spec + a points corpus + geometries within the grid extent, run the 7 light-vs-heavy legs (timing + exact parity), and fill the `benchmarking.mdx` Grid tab. Pure cell-math (`pointascell`, `polyfill`, `kring`) should be competitive-to-faster (no JVM/JTS); the geometry-returning functions cross the WKB UDF boundary (the same ser/de tax measured for the rest of pygx) — quantify and note honestly. Terminate any cluster started for the run after capture; if reusing the standing bench cluster, suggest (don't auto) termination. - -## Resolved decisions (user, 2026-06-14) - -1. **`gbx_custom_grid` validation → option B (validating `@udf`).** Eager validation matching heavy's `require(...)` (error-at-build-time parity). Struct must match `CUSTOM_GRID_SCHEMA`. -2. **Geometry I/O → support `[E]WKB` + `[E]WKT` in BOTH tiers (like the other geometry functions).** The geom-accepting custom functions (`pointascell`, `polyfill`) accept all four input encodings: light via the shared `pygx/_geom.py` `parse_geom`; heavy via `JTS.fromWKB` (auto-detects/strips EWKB SRID) + `JTS.fromWKT` (strips `SRID=…;` EWKT) — **verify the heavy custom expressions use those decoders (per `dataType match { BinaryType => fromWKB; StringType => fromWKT }`); if any hardcodes a single format, extend it to all four** so both tiers match. **Output** geometry stays plain WKB **no-SRID** (`to_wkb()` / `JTS.toWKB`), matching heavy and the other coordinate grids (BNG) — the `srid` field remains metadata only; we do NOT stamp EWKB on output. -3. **Fix the heavy bug in BOTH tiers (0.4.0 is net-new).** `pointToCellID`'s `require(!x.isNaN && !x.isNaN, ...)` duplicate-`x` typo (Y NaN unguarded) is corrected to guard both X and Y in heavy **and** light, referenced in the commit/release notes. The `polyfill` `firstCellPos..lastCellPos + 1` over-scan is **intentional** (centroid-filtered) — ported as-is, not a fix. -4. **Resolution domain → exact parity.** Light computes `maxResolution = min(20, floor(56 / bitsPerResolution))` (cell_splits-dependent) identically to heavy and rejects `res > maxResolution` (no fixed cap). - -## Out of scope - -- The **`h3` GridX subpackage** (Databricks-native H3 covers hex; GeoBrix raster H3 is RasterX, already in pyrx). -- Any **heavy-tier behavior change** — heavy custom gridding is the parity reference and stays fixed, except a bug fix explicitly approved under open question Q3. -- **No new aggregators or explode UDTFs** — the heavy custom family has none; light adds none. -- **No EWKB/SRID stamping** on custom geometry OUTPUTS (no-SRID confirmed, decision 2). Note: this is output-only — INPUTS *do* accept `[E]WKB`/`[E]WKT` in both tiers (decision 2), which is input-encoding flexibility, not output stamping. diff --git a/docs/superpowers/specs/2026-06-14-pygx-light-tier-design.md b/docs/superpowers/specs/2026-06-14-pygx-light-tier-design.md deleted file mode 100644 index 8f176dd25..000000000 --- a/docs/superpowers/specs/2026-06-14-pygx-light-tier-design.md +++ /dev/null @@ -1,145 +0,0 @@ -# pygx Light GridX Tier — Design (quadbin + BNG) - -**Date:** 2026-06-14 -**Branch:** `pygx-light` -**Status:** Approved design (pending user review) - -## Goal - -Bring the heavy-tier **GridX** functions to a lightweight, pure-Python/PySpark tier `databricks.labs.gbx.pygx`, the next sibling after `pyrx` (RasterX, complete) and `pyvx` (VectorX, complete), so GridX is a genuine **exit from heavy** for discrete-global-grid work — full function parity, exact result parity, competitive performance. Two grid systems, designed together, **implemented phased**: - -- **Phase 1 — quadbin** (CARTO quadbin, 10 functions). Net-new in 0.4.0. -- **Phase 2 — BNG** (British National Grid, 23 functions). - -Goals carried from prior lightweight work: (a) function parity, (b) exact result parity, (c) performance wins, (d) special attention wherever light is slower than heavy, (e) coherence (functions genuinely useful as written), (f) captured benchmarking, (g) consistent docs, (h) prominent-surface visibility (README, docs landing, intro, execution-tiers, performance). - -## Architecture - -Pure-Python/PySpark, **Serverless/Spark-Connect safe**: only `spark.udf.register` / `spark.udtf.register` + Column expressions — never `spark.conf.set`, `_jvm`, `sparkContext`, or `.rdd`. Mirrors the pyrx/pyvx package shape. Function names are identical to the heavy tier (`bng_*`, `quadbin_*`) so swapping tiers is a one-line import change; SQL names (`gbx_quadbin_*`, `gbx_bng_*`) register pyspark-backed under the same names as heavy. - -**Files:** - -| File | Responsibility | Phase | -|---|---|---| -| `python/geobrix/src/databricks/labs/gbx/pygx/__init__.py` | package marker | 1 | -| `pygx/functions.py` | `register(spark)` + Column wrappers + UDF/UDTF classes (public surface) | 1 (extended in 2) | -| `pygx/_quadbin.py` | quadbin cell math (wraps the `quadbin` lib) + shapely geometry build | 1 | -| `pygx/_geom.py` | shared `parse_geom` (WKB/EWKB/WKT/EWKT) for geometry-input functions — mirrors the pyvx contract (cross-ST geom-input consistency) | 1 | -| `pygx/_serde.py` | output struct schemas (tessellate cell/geom structs) | 1 | -| `pygx/_bng.py` | pure-Python port of `BNG.scala` (codec + geometry + neighborhood + coverage). Split into `_bng_codec.py` + `_bng_geom.py` if it grows unwieldy | 2 | -| `pygx/_env.py` | dependency guard (`quadbin` + `shapely`) | 1 | -| `python/geobrix/test/pygx/…` | Spark-free core unit tests + registered-fn tests + JAR-gated cross-tier parity | 1, 2 | - -**No new dependencies.** `quadbin>=0.2,<0.3` and `shapely` are already in the `[light]` extra; BNG is pure Python (`numpy`/`shapely` already present). The `quadbin` lib is already used by `pyrx/core/gridagg.py` (a vectorized `point_to_cell` reference). - -## Parity bar (both phases) - -GridX is **deterministic** integer/coordinate math, so the bar is stronger than pyvx's TIN: - -- **Cell IDs and cell sets: bit-exact.** `pointascell`/`eastnorthasbng` produce identical IDs; `polyfill`/`kring`/`kloop`/`tessellate`/`geomkring`/`geomkloop` produce identical cell *sets* (no tolerance). -- **Geometry outputs (WKB): within ~1e-6.** `aswkb`/`aswkt`/`centroid`/`cellunion`/`cellintersection`/tessellate-chips go through shapely (light) vs JTS (heavy); coordinates match to a relative/absolute tolerance of 1e-6, not byte-identical. -- **BNG `tessellate`/`polyfill` held to exact cell-set parity.** Heavy uses a `buffer`-erode core/border split (0.1 m tolerance) + JTS `contains`/`intersects`; a shapely-vs-JTS boundary disagreement on a knife-edge cell is treated as a **bug to fix**, not a tolerated divergence. - ---- - -## Phase 1 — quadbin (10 functions) - -Heavy source: `src/main/scala/com/databricks/labs/gbx/gridx/quadbin/` + `gridx/grid/Quadbin.scala`. The `quadbin` lib does all cell math. - -| Function (SQL) | Shape | Output | Light implementation | -|---|---|---|---| -| `gbx_quadbin_pointascell` | scalar | BIGINT | `quadbin.point_to_cell(lon, lat, res)` (res ∈ [0,26]) | -| `gbx_quadbin_resolution` | scalar | INT | `quadbin.get_resolution(cell)` | -| `gbx_quadbin_kring` | scalar | ARRAY\ | `quadbin.k_ring(cell, k)` | -| `gbx_quadbin_distance` | scalar | INT | **custom** — the installed `quadbin` 0.2.x has no `cell_distance`; compute Chebyshev distance from `cell_to_tile(a)`/`cell_to_tile(b)` (`max(\|dx\|,\|dy\|)`), same-resolution-or-error, matching `Quadbin.scala` | -| `gbx_quadbin_polyfill` | scalar | ARRAY\ | **custom** — no `polyfill_bbox` in 0.2.x; enumerate the cells of the geometry's **bounding box** at the resolution (via `cell_to_tile`/`tile_to_cell` tile range, or `geometry_to_cells` on the bbox polygon), matching `Quadbin.scala`'s `getEnvelopeInternal` bbox semantics (res ∈ [0,20]). Hold to exact cell-set parity. | -| `gbx_quadbin_aswkb` | scalar | BINARY (EWKB polygon, SRID 4326) | cell → bbox (lib) → shapely polygon → `to_wkb(..., include_srid=True)` | -| `gbx_quadbin_centroid` | scalar | BINARY (EWKB point) | cell bbox center → shapely Point → EWKB | -| `gbx_quadbin_cellunion` | scalar | BINARY (EWKB MultiPolygon) | each cell → shapely polygon → `unary_union` → EWKB | -| `gbx_quadbin_tessellate` | scalar | ARRAY\\> | polyfill bbox → per-cell shapely intersection with the input geom | -| `gbx_quadbin_cellunion_agg` | aggregator | BINARY (EWKB MultiPolygon) | **grouped-agg `pandas_udf` returning BINARY directly** (atomic output — no struct-compose workaround); same union logic as `cellunion` | - -Geometry-input functions (`polyfill`, `tessellate`) accept WKB/EWKB/WKT/EWKT via `_geom.parse_geom`. - ---- - -## Phase 2 — BNG (23 functions) - -Heavy source: `src/main/scala/com/databricks/labs/gbx/gridx/bng/` + the **canonical algorithm** `gridx/grid/BNG.scala` (≈816 lines). **No PyPI BNG library exists** — `_bng.py` is a faithful pure-Python port of `BNG.scala`, validated bit-exact against it. Cell IDs are STRING (mirror heavy). All register under the `gbx_bng_*` prefix (incl. `gbx_bng_pointascell`). Resolution = integer index ±1..±6 (1=100km … 6=1m; negatives = quadrants) or a `resolutionMap` string key (`"1km"`, `"100m"`, …); **never** metres-as-Int. Coordinates are EPSG:27700 eastings/northings. - -Implemented in dependency order (each layer testable before the next): - -1. **Codec** — `encode`/`decode`/`format`/`parse`, `resolutionMap`/`sizeMap`, the `letterMap` 2-letter-prefix grid, and the ±quadrant logic. Functions: `bng_pointascell`, `bng_eastnorthasbng`, `bng_cellarea` (`(edgeSize/1000)²` km²), `bng_distance` (Manhattan), `bng_euclideandistance` (Chebyshev/max). -2. **Cell → geometry** — `cellIdToGeometry` (decode → x/y/edgeSize → shapely polygon), `cellIdToCenter`. Functions: `bng_aswkb` (WKB polygon, **no SRID** — heavy uses `toWKB` not `toEWKB`), `bng_aswkt`, `bng_centroid`, `bng_cellintersection`, `bng_cellunion`. -3. **Neighborhood (cell-centric)** — `kRing`/`kLoop` square-grid walks. Functions: `bng_kring`, `bng_kloop`, and the explode UDTFs `bng_kringexplode`, `bng_kloopexplode` (SQL-`LATERAL`-only in light). -4. **Coverage** — `bng_polyfill` (BFS flood-fill from boundary + centroid via `kLoop`, shapely `contains` per candidate centroid). -5. **Tessellation** — `bng_tessellate` / `bng_tessellateexplode` (buffer-erode core/border split, separate polyfill per region, per-border-cell shapely intersection, 0.1 m core tolerance) → `ARRAY>`. The hardest functions. -6. **Geometry-centric neighborhood** — `bng_geomkring`, `bng_geomkloop` and their explode UDTFs (depend on the tessellate/`lineFill`/`lineDecompose` machinery — BFS walk along LineStrings via `kRing`). -7. **Aggregators** — `bng_cellintersection_agg`, `bng_cellunion_agg` (grouped-agg `pandas_udf` returning BINARY directly). - -### BNG known-issue validation (Mosaic lineage) - -Two BNG bugs were flagged in upstream Mosaic and may have been inherited by the heavy port. Because pygx holds **exact cell-set / geometry parity** with heavy, a bug present in heavy would otherwise be *propagated* into light. Stance (same as the pyvx legacy holes/Z fixes): during Phase 2, **validate each against current `gridx/grid/BNG.scala`**; if still present, **fix in BOTH tiers** so parity holds on *correct* behavior, and **reference the Mosaic issue** in the commit message + the beta release notes. If already fixed or not applicable in the geobrix port, record that finding (no change). - -- **[mosaic#434](https://github.com/databrickslabs/mosaic/issues/434)** — **fixed upstream in [mosaic#580](https://github.com/databrickslabs/mosaic/pull/580)** (merged 2024-09-25, in `BNGIndexSystem.scala`). Affects `bng_aswkb` / `bng_aswkt` / `bng_centroid` / `bng_cellarea`: a **100km** grid reference whose two-letter code is `"NE"` / `"NW"` / `"SE"` / `"SW"` was misread as a *quadrant* (negative) resolution and given a half-dimension geometry. **Phase-2 task: VERIFY geobrix's `gridx/grid/BNG.scala` already carries the #580 quadrant-disambiguation fix** (geobrix may have been ported post-#580). If present → port the correct behavior, no heavy change. If the geobrix port predates/missed #580 → apply the equivalent fix in BOTH tiers, referencing #580 + #434. Either way, the cross-tier parity test **must** include these four 100km cells (expect full 100km × 100km = 10¹⁰ m² area). -- **[mosaic#423](https://github.com/databrickslabs/mosaic/issues/423)** — affects `bng_tessellate` / `bng_tessellateexplode`: when the input polygon is **aligned to the BNG grid** at the resolution, the per-border-cell intersection yields spurious **POINT/LINESTRING** chips at shared vertices/edges alongside the correct polygon chips. Tessellation must **filter intersection results to areal geometries** (Polygon/MultiPolygon) only. Parity test must include a grid-aligned polygon and assert no degenerate (non-areal) chips in either tier. - -## Data flow - -``` -quadbin: (lon,lat,res) ─pointascell→ BIGINT ─resolution/aswkb/centroid→ INT / EWKB - geom (WKB/EWKB/WKT/EWKT) ─polyfill→ ARRAY ; ─tessellate→ ARRAY> - ARRAY / grouped rows ─cellunion(_agg)→ EWKB MultiPolygon - -BNG: (point|east,north)+res ─encode→ STRING cellid ─cellIdToGeometry→ WKB / WKT / centroid - cellid ─kring/kloop(+explode)→ ARRAY / rows - geom+res ─polyfill→ ARRAY ; ─tessellate(+explode)→ STRUCT - geom+res+k ─geomkring/geomkloop(+explode)→ ARRAY / rows - cells ─cellintersection/cellunion(_agg)→ WKB -``` - -## Error handling - -- Resolution validation mirrors heavy: quadbin [0,26] (pointascell) / [0,20] (polyfill); BNG ±1..±6 or `resolutionMap` keys — reject metres-as-Int with a clear error. -- `quadbin_distance` / `bng` distances on different resolutions → error (mirror heavy). -- Unknown/malformed cell IDs → clear `ValueError`. -- Empty/null geometry inputs → empty array / null, matching heavy. -- Explode UDTFs (light) have no Python Column form → `NotImplementedError` pointing to SQL `LATERAL` (like pyvx pyramid). - -## Testing - -- **Spark-free core** (`_quadbin.py`, `_bng.py`): the BNG **codec round-trips** (encode↔decode↔format↔parse) and is checked **bit-exact against `BNG.scala`** across every resolution including ±quadrants; quadbin cell math via the lib. Geometry construction, kring/kloop walks, polyfill BFS, tessellate core/border split — unit-tested with known fixtures. -- **Registered-function tests** (Docker, Spark): each `gbx_*` UDF/UDTF/agg via the spark fixture. -- **Cross-tier parity** (JAR-gated, `test/pygx/test_parity_*`): exact cell-ID/set equality light-vs-heavy per function; geometry WKB decoded-equality within 1e-6. Register light, then heavy (same SQL name, last-wins), as in the pyvx parity tests. -- **Binding parity**: every `gbx_quadbin_*` / `gbx_bng_*` present in `registered_functions.txt`, Python `functions.py`, and `function-info.json`. - -## Performance (goals c, d) - -Pure cell-math functions (`pointascell`, `resolution`, `distance`, `kring`/`kloop`) should be **competitive-to-faster** than heavy (no JVM/JTS overhead). The geometry-returning functions (`aswkb`, `centroid`, `cellunion`, `tessellate`) cross the WKB UDF boundary — the same JVM↔Python ser/de tax measured for pyvx; the bench quantifies it, and any function where light is materially slower than heavy gets explicit attention (vectorization where possible) and an honest note in the docs (goal d). Cell IDs are scalar LONG/STRING — cheap across the boundary. - -## Benchmarking (goal f) - -Extend the `gbx:*` bench harness mirroring the vector-tin bench: `bench/corpus_*` generators (points + geometries + cell-id arrays), `bench/readers.py` `run_*` legs, a `bench/cluster.py` cell, and launcher flags `--grid-quadbin-only` / `--grid-bng-only`. Cluster light-vs-heavy timing + **exact-parity** verdicts. Fill the **`benchmarking.mdx` "Grid (soon)" tab** per phase. Terminate any cluster started for the run after capture. - -## Docs (goals g, h) — every surface, per phase - -- `docs/docs/api/gridx-functions.mdx` — flip each function's Tier badge heavy→both as it lands; per-function lib-attribution notes (quadbin lib / pure-Python BNG / shapely); document the BNG resolution-index + `resolutionMap` convention and the EPSG:27700 expectation. -- `docs/docs/api/execution-tiers.mdx` — move quadbin (phase 1) then BNG (phase 2) out of the "heavyweight-only" list and the `gbx_bng_*`/quadbin mentions; update the GridX framing. -- `docs/docs/api/performance.mdx` — add pygx to the execution-shape tabs, the modules table (`pygx/_quadbin.py`, `pygx/_bng.py`), the libraries table (`quadbin`), and the perf narrative. -- `docs/docs/api/benchmarking.mdx` — fill the Grid tab with light-vs-heavy numbers + exact-parity verdicts. -- `README.md` — flip the GridX bullet from "Heavyweight Scala tier (lightweight `pygx` planned)" to its availability (by phase). -- Docs landing (`docs/src/pages/index.js`) — GridX card + the heavyweight-only line. -- `docs/docs/intro.mdx` — note pygx alongside pyrx/pyvx. -- `function-info` examples for the registered functions. - -Voice: no internal/planning vocabulary; justify by user utility, not Mosaic parity. Quadbin is net-new in 0.4.0 — no interim/back-compat details surface publicly. - -## Phasing - -**One spec** (this document, both grid systems). **Two implementation plans**: the **quadbin plan executes first** and ships fully (10 functions + bench + all docs surfaces) before the **BNG plan** starts. Each phase flips its functions' Tier badges and updates every doc surface above on completion. - -## Out of scope - -- **Custom gridding (`gbx_custom_*`) — originally out of scope, SUPERSEDED.** This spec scoped pygx to quadbin + BNG only and declared the 7 custom user-defined grid functions heavyweight-only. The user reversed that on 2026-06-14, and custom gridding was subsequently ported to pygx (see `2026-06-14-pygx-custom-gridding-light-tier-design.md`, IMPLEMENTED 2026-06-15). All 7 `gbx_custom_*` now run in both tiers with exact parity, so **GridX is fully 1:1 light↔heavy**; this out-of-scope note is retained only for historical context. -- The `h3` GridX subpackage (Databricks-native H3 already covers hex; GeoBrix raster H3 is RasterX, already in pyrx). -- Any heavy-tier behavior change (GridX heavy is the parity reference; light conforms to it). diff --git a/docs/superpowers/specs/2026-06-20-stac-light-api-design.md b/docs/superpowers/specs/2026-06-20-stac-light-api-design.md deleted file mode 100644 index cde22449b..000000000 --- a/docs/superpowers/specs/2026-06-20-stac-light-api-design.md +++ /dev/null @@ -1,148 +0,0 @@ -# STAC Lightweight API — Design - -**Date:** 2026-06-20 -**Status:** Approved (design); pending implementation plan -**Scope:** A lightweight, Serverless-safe STAC client for GeoBrix — distributed **search**, resilient **download**, and **repair** — consolidating logic currently scattered across the EO-series `library.py` + `config_nb` helpers. - -## Motivation - -The EO-series example re-implements STAC query + download from per-notebook helpers (`library.py`: `get_items`, `get_assets`, `get_assets_for_cells`, `download_asset_v2`; `config_nb`: `download_band`, `update_assets`, `download_missing_assets`). This logic is: - -- **Scattered + copy-pasted** across `library.py` and `config_nb` — every notebook re-wires it. -- **Hardcoded to Planetary Computer** + `sign_inplace`. -- **Not Serverless-native** until recently (used `spark.conf.set`, now guarded). -- **Missing download resilience** until recently (size-only validity accepted throttled/truncated files). - -This design packages the now-proven, hardened logic behind one importable, catalog-agnostic, Serverless-safe surface so notebooks become a few calls instead of ~150 lines of helpers. - -This is **net-new lightweight capability** — the heavyweight tier has no STAC equivalent, so there is no cross-tier parity requirement. - -## Goals - -- A `StacClient` class (config held once) exposing `search`, `download`, `repair`. -- **Catalog-agnostic** (configurable catalog URL + pluggable signing), defaulting to Planetary Computer + `sign_inplace`. -- **Serverless-safe**: no `spark.conf.set`, no `.cache()`/`persist`; parallelism via `DataFrame.repartition(N)`. -- **Resilient download**: raise-on-HTTP-error, read-validation (open + decode a window), re-sign + retry/backoff, download-to-local → publish-to-Volume-only-when-valid. -- **Repair**: re-download only invalid rows (Delta MERGE). -- Distributed (Spark fan-out), idempotent (skip already-valid files). - -## Non-goals (YAGNI) - -Each is deliberately out of the initial build, with the reasoning and a revisit trigger. - -- **No async / concurrent-within-task client.** Parallelism comes from Spark (one task per AOI/asset via `repartition`), not `asyncio`/threads inside a UDF, which only adds failure modes (loop lifecycle, partial-await cleanup) when the executor already fans out. *Revisit if* a single task must issue many small requests where per-request latency dominates (not the case for whole-asset GeoTIFF fetches). - -- **No caching / memoization layer.** Search results and downloaded files are materialized by the *caller* (a Delta table, files on a Volume) — that already is the cache, inspectable and time-travel-friendly. *Revisit if* repeated identical searches in one session become a measured cost. - -- **`download` is a faithful fetch — no transformation, no non-raster validation.** `search` surfaces every asset's `href`/metadata, but `download` fetches bytes as the catalog serves them and read-validates assuming a raster (open + decode a window). It does **not** reproject / re-tile / COG-convert / CRS-reconcile (the raster tier's job: `pyrx` `rst_*`, `gtiff_gbx`, `rst_merge_agg`), and it does **not** download-validate non-raster assets (JSON, thumbnails, vector sidecars — discoverable via `search`, just not first-class download targets). Keeps provenance intact. *Revisit if* validated non-raster downloads are needed (per-type validation strategy). - -- **No catalog/item writing or publishing.** A read/consume client only (query a STAC API, fetch assets); creating items, writing a static catalog, or registering products is a separate feature. - -- **No AOI grid generation (`generate_cells`).** The client takes a geometry/GeoJSON column as input; tessellation lives in the `rst_h3_tessellate` UDTF / Databricks built-in H3. (The existing `library.py` helper is dead code using the heavyweight-only `rst_h3_tessellateexplode`.) - -- **No repair scheduling / orchestration / UI.** `repair` is a callable pass over invalid rows; deciding *when* to re-run it (a job schedule, loop-until-complete, alerting) is the caller's. *Revisit if* an "auto-complete until all valid" loop helper proves widely wanted. - -- **No secret/credential management beyond `sign`.** Auth is expressed only through `sign` (`'planetary_computer'` | `None` | callable); token storage/refresh and provider auth flows are the caller's. *Revisit if* a catalog needs request-time auth headers (extend the signing/transport hook). - -## Packaging - -- New top-level package `databricks.labs.gbx.stac` (parallel to `pyrx`, `pyvx`, `pygx`, `ds`). -- New **optional** extra `geobrix[stac]` → `pystac-client`, `planetary-computer` (`tenacity` is already pulled by `[light]`). Keeps `[light]` lean; STAC is opt-in (`pip install geobrix[light,stac]`). -- Pure-Python; runs on Serverless (environment version 5+, Python 3.12) and classic. - -## Module layout - -``` -gbx/stac/ - __init__.py # exports StacClient - client.py # StacClient: config + orchestration (search / download / repair) - _search.py # pandas-UDF internals: per-AOI catalog.search() + item/asset parsing - _download.py # resilient download UDF (raise-on-HTTP-error, read-validate, - # re-sign + retry/backoff, local-stage -> Volume publish) - _sign.py # signing strategies: 'planetary_computer' | None | callable(href)->href -``` - -## API surface - -```python -from databricks.labs.gbx.stac import StacClient - -client = StacClient( - catalog="https://planetarycomputer.microsoft.com/api/stac/v1", # default - sign="planetary_computer", # 'planetary_computer' | None | callable(href)->href -) - -# SEARCH — AOI rows (a GeoJSON-geometry column) -> one row per (aoi, item, asset). -assets_df = client.search( - df, - geojson_col="geojson", - collections=["sentinel-2-l2a"], - datetime="2022-06-01", - partitions=512, # repartition fan-out; no spark.conf -) -# columns: , item_id, date, item_bbox, asset_name, href, item_properties - -# DOWNLOAD — resilient, validated; one task per asset. -files_df = client.download( - assets_df, - out_dir, # a UC Volume path - asset_names=["B02", "B03", "B04", "B08"], # None = all assets present - name="{asset_name}_{item_id}.tif", # filename template - validate=True, # read-validation (open + decode a window) - max_tries=5, - partitions=None, # default: one task per asset (count-based) -) -# columns: item_id, asset_name, out_file_path, out_file_sz, is_out_file_valid - -# REPAIR — re-download only invalid rows (Delta MERGE); returns the repaired subset. -client.repair(table_or_df, where="is_out_file_valid = false") -``` - -## Behavior / data flow - -### search -1. `repartition(partitions)` the AOI DataFrame (Serverless-safe fan-out). -2. A `pandas_udf` opens the configured catalog (with the configured signer) and runs `catalog.search(collections=, intersects=, datetime=)` per row, with `tenacity` exponential retry; returns item JSONs. -3. Explode items → parse `item_id`, `date` (from `properties.datetime`), `item_bbox`, `item_properties`; explode assets → `asset_name`, `href`. -4. Returns one row per `(aoi, item, asset)`, carrying the input columns through. - -### download -1. Optionally filter to `asset_names`. -2. **Dedup to unique `(item_id, asset_name)`** — `search` emits one row per *(aoi, item, asset)*, so the same item reached via multiple AOIs/cells would otherwise be fetched repeatedly. (The href is re-signed per attempt from `item_id`+`asset_name`, so a stale search-time href is not relied on.) -4. `repartition` to one task per asset (or `partitions`). -5. Resilient download UDF per asset: - - **(re-)sign** the href each attempt (signed URLs expire). - - `requests.get(...).raise_for_status()` (HTTP throttle/expiry → tenacity backoff). - - download to **worker-local** disk. - - **read-validation**: `rasterio.open` + decode a window (rejects throttled error bodies and truncated files that a size check would accept). - - publish to the Volume with a **sequential copy** (FUSE-safe) only when valid. - - on failure: exponential backoff, re-sign, retry up to `max_tries`; then return `None`. -6. Compute `is_out_file_valid` from the validated outcome; return the files DataFrame. Idempotent: an existing file above the size floor is treated as already-published (only validated files are ever written). - -### repair -1. Read the band/files table (or DataFrame); filter to invalid (`is_out_file_valid` false/null) and optional `where`. -2. Re-run the resilient download on that subset. -3. `DeltaTable.merge` the repaired rows back (`out_file_path`, `out_file_sz`, `is_out_file_valid`, `last_update`). -4. Return the repaired subset. - -## Error handling - -- **Search:** per-AOI failures retry (tenacity); a permanently failing AOI yields an empty item list for that row rather than failing the job. -- **Download:** every failure mode (HTTP error, throttled body, truncation, expired URL, decode failure) funnels to `is_out_file_valid=False` and is retried in-UDF, then by `repair`. No partial/corrupt file is left on the Volume. -- **Serverless:** no `spark.conf.set` / `.cache()`; any incidental conf use goes through a guarded no-op helper. - -## Testing - -- **Unit (CI, no network):** `StacClient` accepts an injectable catalog opener (`_catalog_opener`) so tests use a fake catalog returning canned items. Cover: search parsing (items→assets→typed cols), download read-validation (good vs throttled/truncated bytes), retry/backoff (counts attempts), repair MERGE, and a Serverless-guard source check (no `spark.conf`/`.cache()` in the module). -- **Integration (marked, network):** `@pytest.mark.integration` — real Planetary Computer search + download of one small asset; excluded from CI (matches the existing integration-marker convention in `pyproject.toml`). - -## EO-series refactor (follow-on, same effort) - -Once built, refactor the EO-series to use the client: -- nb01 search → `client.search(cells_df, ...)` (replaces `get_assets_for_cells`). -- nb02 download → `client.download(...)` + `client.repair(...)` (replaces `download_band` / `download_missing_assets` orchestration). -- Remove the now-redundant `library.py`/`config_nb` STAC helpers (keep viz/plot helpers). - -## Rollout - -`stac` ships in the same wheel as `[light]` but behind the `[stac]` extra. The EO-series `config_nb` install line becomes `geobrix[light,stac]`. diff --git a/docs/superpowers/specs/2026-06-22-gbx-viz-escape-hatches-design.md b/docs/superpowers/specs/2026-06-22-gbx-viz-escape-hatches-design.md deleted file mode 100644 index 8435161f8..000000000 --- a/docs/superpowers/specs/2026-06-22-gbx-viz-escape-hatches-design.md +++ /dev/null @@ -1,215 +0,0 @@ -# gbx.viz + pyrx escape-hatches — Design - -**Date:** 2026-06-22 -**Branch:** `beta/0.4.0` -**Status:** design (pending user review) - -## Goal - -Promote the reusable EO-series notebook helpers into the package as first-class, -tested, documented APIs: a tier-agnostic visualization module `databricks.labs.gbx.viz` -(behind a new `[viz]` extra), plus two lightweight escape-hatches in `pyrx` for users -whose needs fall outside the canonical `rst_*` surface. - -## Motivation - -`notebooks/examples/eo-series/library.py` and `config_nb.ipynb` contain genuinely -useful, non-trivial helpers (a proper EO render pipeline; Spark→GeoDataFrame -adapters) that every notebook reimplements or `%run`-imports. They belong in the -package so users get them by `pip install` rather than copy-paste. A survey of both -files classified each helper as promote / leave-local (see "Helper disposition"). - -## Scope - -In scope: -- `gbx.viz`: `plot_raster`, `plot_file`, `as_gdf`, `cells_as_gdf` (+ private render helpers). -- `pyrx` escape-hatches: `tile_to_numpy`, `rst_apply`. -- New `[viz]` extra (matplotlib + geopandas + folium + mapclassify), pinned + hashed - in the lightweight CI lock; viz tests run in the light CI phase. -- Drop `generate_cells` (heavyweight-only, unused). - -Out of scope (separate follow-ups): -- A `rst_h3_tessellate_df` DataFrame wrapper hiding the LATERAL-UDTF + tile-rebuild - boilerplate — folded into the queued "UDTF generalization" audit. -- Migrating the eo-series notebooks to import from the package + re-executing them. -- `set_conf_safe` — stays notebook-local (the lightweight tier must never - `spark.conf.set`; shipping even a "safe" wrapper would signal otherwise). - -## Architecture - -Approach A: visualization is **tier-agnostic** (operates on raster bytes / Spark -DataFrames, independent of pyrx vs rasterx), so it lives at the top level -`databricks.labs.gbx.viz`. The escape-hatches operate on the pyrx tile struct and -rasterio, so they live in `pyrx`. - -``` -databricks/labs/gbx/ - viz/ - __init__.py # public: plot_raster, plot_file, as_gdf, cells_as_gdf - _raster.py # plot_raster/plot_file + _decimated_read/_percentile_stretch/ - # _needs_percentile_stretch/_render (private) - _vector.py # as_gdf, cells_as_gdf - _env.py # assert_viz_available() — lazy-dep guard (mirrors pyrx/_env.py) - pyrx/ - functions.py # + tile_to_numpy, rst_apply re-exported on the public surface - core/ - escape.py # tile_to_numpy, rst_apply implementations -``` - -All heavy `[viz]` deps are **lazy-imported inside functions**, guarded by -`viz/_env.py::assert_viz_available()`, which raises a clear -`pip install 'geobrix[viz]'` message when a dep is missing — mirroring -`pyrx/_env.py::assert_rasterio_available()`. Matplotlib is set to the `Agg` backend -when no display is available, so headless/cluster use never errors on a missing GUI. - -The tile struct is the canonical `struct>` (`pyrx/_serde.py::TILE_SCHEMA`); the escape-hatches -use the existing `_serde.open_tile(raster_bytes)` context manager. - -## Components - -### `gbx.viz._raster` - -```python -def plot_raster(raster_bytes, *, fig_w=10, fig_h=10, max_pixels=2000): ... -def plot_file(path, *, fig_w=10, fig_h=10, max_pixels=2000): ... -``` -- Pipeline (ported verbatim from `library.py`, kept as private helpers): - - `_decimated_read(src, max_pixels)` — bilinear downsample so `max(w,h) <= max_pixels`; - `masked=True` so nodata is honored; returns `(data, transform, scale)`. - - `_needs_percentile_stretch(data)` — True for integer dtypes whose max > 255. - - `_percentile_stretch(data, lo_pct=2, hi_pct=98)` — per-band 2–98th percentile - stretch to `[0,1]` float32, ignoring masked pixels; mask preserved. - - `_render(data, transform, *, title, fig_w, fig_h, scale)` — apply stretch when - needed; single-band → `viridis`, multi-band → RGB via `rasterio.plot.show`; title - suffixed with decimation factor when downsampled. -- `plot_raster` opens bytes via `rasterio.io.MemoryFile`; `plot_file` via `rasterio.open`. -- Both return `None` (side-effect: a matplotlib figure). Headless-safe via `Agg`. - -### `gbx.viz._vector` - -```python -def as_gdf(df, wkt_col="wkt", *, max_rows=10_000): ... -def cells_as_gdf(df, cell_col="cellid", extra_cols=(), *, max_rows=10_000): ... -``` -- `as_gdf`: Spark DataFrame with a WKT column → `geopandas.GeoDataFrame` (EPSG:4326). - Collects with `df.limit(max_rows + 1).toPandas()` (single collect); if the result - has `> max_rows` rows, truncate to `max_rows` and `warnings.warn` that output was - truncated for driver-side viz. `max_rows=None` opts out of the limit. Geometry built - with `geopandas.GeoSeries.from_wkt(..., crs=4326)`; non-geometry columns preserved. -- `cells_as_gdf`: H3 cell ids → boundary polygons via the **`h3` lib** (already a light - dep) computed on the collected pandas frame (portable, offline-testable — replaces - the notebook's Databricks-native `h3_boundaryaswkt`); carries `extra_cols`; delegates - to `as_gdf`. `max_rows` applied here (before the per-row boundary computation). - Note: gbx H3 `cellid` is a `bigint`; the h3 v4 API (`h3.cell_to_boundary`) takes a - string index, so convert per-cell with `h3.int_to_str(cellid)` before building the - boundary polygon (shapely `Polygon` from the lng/lat ring). - -### `pyrx` escape-hatches (`pyrx/core/escape.py`, re-exported in `functions.py`) - -```python -def tile_to_numpy(tile_or_bytes) -> "np.ndarray": ... -def rst_apply(tile_col, fn, returnType=DoubleType()) -> Column: ... -``` -- `tile_to_numpy`: accepts a tile struct (a `Row`/dict with a `raster` field) **or** - raw `bytes`/`bytearray`; reads all bands via `_serde.open_tile(...).read()` → ndarray. - The "drop to numpy" hatch (from `library.py::to_numpy_arr`, generalized to accept a - tile struct). No new deps (rasterio/numpy already light). -- `rst_apply`: returns a `Column`; builds a per-row scalar UDF that opens each tile's - `raster` bytes via `_serde.open_tile` and calls `fn(rasterio_dataset)`, returning one - value of `returnType` per row (default `DoubleType()`; any Spark `DataType` accepted). - Null/empty tile → null. Generalizes `library.py::rasterio_lambda`'s hardcoded - `DoubleType`. The documented "GeoBrix lacks function X — run your own rasterio per - tile" path. **Scalar return only** (raster→raster transforms remain the domain of - `rst_mapalgebra`/`rst_derivedband`). - -Both escape-hatches are **Python-API-only**: `tile_to_numpy` returns a host object -and `rst_apply` takes a Python callable, so neither is SQL-registerable. They are -exposed on the `pyrx.functions` import surface but are **not** added to the SQL -registry or `registered_functions.txt` — binding-parity and `function-info.json` -are unaffected (the QC `binding-parity` gate stays green). - -## Dependencies / `[viz]` extra - -`pyproject.toml`: -``` -viz = [ - "matplotlib>=3.7,<4", - "geopandas>=1.0,<2", - "folium>=0.16,<1", - "mapclassify>=2.6,<3", # geopandas .explore() choropleth classifier -] -``` -- geopandas 1.x uses `pyogrio` (already a light dep); `shapely`/`pyproj` already light. -- Versions match `requirements-dev-container.in` where present; pinned + hash-locked in - `requirements-pyrx-ci.{in,txt}` (regenerated with `--generate-hashes`). -- The `h3` lib is already in the light lock (used by `cells_as_gdf`). - -## Error handling - -- Missing `[viz]` deps → `assert_viz_available()` raises `ModuleNotFoundError`-style - message: `gbx.viz requires the [viz] extra: pip install 'geobrix[viz]'`. -- `plot_*`: an unreadable raster surfaces the underlying rasterio error (not swallowed). -- `as_gdf`/`cells_as_gdf`: missing `wkt_col`/`cell_col` → `KeyError`-style ValueError - naming the column; oversized DF → warn + truncate (not an error). -- `rst_apply`: null/empty tile → null row (no crash); a `fn` exception propagates as the - UDF's task failure (not silently swallowed — the escape-hatch is the user's code). - -## Testing - -New `python/geobrix/test/viz/` dir (real assertions; matplotlib `Agg`; no pixel compare): -- `_percentile_stretch`: known UInt16 array → output in `[0,1]`, masked pixels excluded - from percentile stats, mask preserved. -- `_decimated_read`: source larger than `max_pixels` → output `max(w,h) <= max_pixels` - and `scale > 1`; small source → untouched, `scale == 1`. -- `plot_raster`/`plot_file`: a synthesized GTiff renders without error and produces a - figure/axes (assert on the returned/active figure, not pixels). -- `as_gdf`: result CRS == EPSG:4326, geometries valid, non-geom columns preserved; - `> max_rows` input → exactly `max_rows` rows + a truncation warning; `max_rows=None` - → full collect. -- `cells_as_gdf`: a known H3 id → expected boundary polygon (h3 lib), `extra_cols` - carried through. - -New `python/geobrix/test/pyrx/test_escape.py`: -- `tile_to_numpy`: synthesized tile → expected shape/dtype; bytes-input and - struct-input agree. -- `rst_apply`: small DataFrame of tiles → expected scalar per row with a **non-default** - `returnType` (e.g. `IntegerType()`), proving the return-type generalization; null tile - → null. - -## CI / supply-chain - -Follows the maintained light-tier condition (`test/conftest.py` docstring): -1. Add `"viz"` to `_LIGHT_TEST_DIRS` in `python/geobrix/test/conftest.py` (heavy phase - skips it — no rasterio/geopandas there). -2. Add `test/viz` to the light pytest dir list in `.github/actions/pyrx_build/action.yml`. -3. Add the `[viz]` deps to `requirements-pyrx-ci.in`; regenerate `requirements-pyrx-ci.txt` - with `UV_INDEX_URL=https://pypi-proxy.dev.databricks.com/simple uv pip compile - --generate-hashes --python-version 3.12 -o requirements-pyrx-ci.txt requirements-pyrx-ci.in`. -4. Verify in a **clean venv built only from the lock** (`pip install --require-hashes -r - requirements-pyrx-ci.txt && pip install --no-deps .`) running the full light selection, - to catch any missing transitive dep before pushing (the dev container's pre-installed - extras mask gaps). - -## Docs - -- New `docs/docs/api/viz.mdx` documenting `plot_raster`/`plot_file`/`as_gdf`/`cells_as_gdf` - with the `[viz]` install note and runnable examples (doc-test-backed, per the repo's - single-source convention). -- Escape-hatches documented in `docs/docs/api/raster-functions.mdx` (an "Escape hatches" - section) — `tile_to_numpy` + `rst_apply` framed as the path for gaps in `rst_*` coverage. -- No internal/wave vocabulary in any `docs/docs/` page (QC `internals-leak` gate). - -## Helper disposition (full survey of library.py + config_nb) - -| Helper | Source | Disposition | -|---|---|---| -| `plot_raster`, `plot_file` (+ render pipeline) | library.py | → `gbx.viz._raster` | -| `to_numpy_arr` | library.py | → `pyrx` `tile_to_numpy` | -| `rasterio_lambda` | library.py | → `pyrx` `rst_apply` (generalized return type) | -| `as_gdf`, `cells_as_gdf` | config_nb | → `gbx.viz._vector` (h3-lib boundaries) | -| `generate_cells` | library.py | **dropped** (heavyweight-only, unused) | -| `set_conf_safe` | both | leave notebook-local | -| `file_size`, `timestamp_filename`, `get_now_formatted` | config_nb | leave local (trivial; validity already in STAC pkg) | -| `finalize_tiled_band_tbl`, `gen_tessellate_tiled_band` | config_nb | leave local (eo-series Delta-schema + Databricks-SQL-coupled ETL) | -| (LATERAL UDTF + tile-rebuild boilerplate inside `gen_tessellate_tiled_band`) | config_nb | follow-up: `rst_h3_tessellate_df` under the queued UDTF-generalization audit | diff --git a/docs/superpowers/specs/2026-06-22-register-only-light-tier-design.md b/docs/superpowers/specs/2026-06-22-register-only-light-tier-design.md deleted file mode 100644 index 1535d964e..000000000 --- a/docs/superpowers/specs/2026-06-22-register-only-light-tier-design.md +++ /dev/null @@ -1,120 +0,0 @@ -# `register(spark, only=[...])` — selective SQL registration (light tiers) - -**Date:** 2026-06-22 -**Status:** Approved (design) -**Scope:** Lightweight tiers only — the SQL-function registrars `pyrx`/`pygx`/`pyvx` **and** the light DataSource registrar `ds.register` (readers/writers). Heavyweight `only=` is a documented future follow-up (out of scope here). - -## Goal - -Add an optional `only` parameter to the lightweight `register()` functions so a session can register a **subset** of a tier's surface instead of the full set: - -```python -from databricks.labs.gbx.pyrx import functions as rx -rx.register(spark, only=["rst_slope", "gbx_rst_clip"]) # just these two SQL functions -rx.register(spark) # all (unchanged) - -from databricks.labs.gbx.ds import register as ds_register -ds_register.register(spark, only=["raster_gbx", "gtiff_gbx"]) # just these readers/writers -``` - -## Motivation - -`register()` today is all-or-nothing per package: it installs every `gbx_*` SQL name for that tier. The two tiers share SQL names, so cross-tier composition is last-registration-wins at whole-package granularity. `only=` gives finer control. The primary beneficiary is the lightweight tier: - -- **Light-only cluster:** register just the functions a session will actually use, rather than the whole surface. -- **Tier mixing:** with heavy installed and auto-registered, `light.register(spark, only=[...])` overrides just those names with the lightweight implementation, leaving the rest heavy. (The reverse — heavy re-registering a few over light — needs the deferred heavy `only=`.) - -## API - -Add `only: Optional[List[str]] = None` to `register()` in: -- `databricks.labs.gbx.pyrx.functions` (SQL functions — `gbx_rst_*`) -- `databricks.labs.gbx.pygx.functions` (SQL functions — `gbx_quadbin_*`, `gbx_bng_*`, `gbx_custom_*`) -- `databricks.labs.gbx.pyvx.functions` (SQL functions — `gbx_st_*`, `gbx_pmtiles_agg`) -- `databricks.labs.gbx.ds.register` (DataSource **readers/writers** — selected by **format name**: `raster_gbx`, `gtiff_gbx`, `pmtiles_gbx`, `vector_gbx`, `shapefile_gbx`, `geojson_gbx`, `geojsonl_gbx`, `gpkg_gbx`, `file_gdb_gbx`) - -The SQL-function `register()` and the DataSource `register()` are **separate entry points** today (the function registrars do not register readers/writers, and vice-versa); `only=` is added to both so the "register only what this session uses" story is uniform across the light tier. - -Semantics: -- `only=None` (default) → register **everything** — identical to today's behavior, in today's order. -- `only=[...]` → register **exactly** the named functions, nothing else. -- `only=[]` → register **nothing** (valid, no-op registration). Documented explicitly. - -### Name handling - -Accept **both** the SQL name and the short Python name, **case-insensitively**. Normalization: the input is `.lower()`-cased first (SQL names are all lowercase by convention, and the Scala classes are CamelCase — `RST_Slope`, `BNG_Polyfill` — so users naturally type mixed case), then `gbx_` is prepended if absent. This is uniform across every prefix (`rst_`, `bng_`, `quadbin_`, `custom_`, `st_`): - -| Input | Normalized | -|---|---| -| `rst_slope` | `gbx_rst_slope` | -| `gbx_rst_slope` | `gbx_rst_slope` | -| `RST_Slope` | `gbx_rst_slope` | -| `st_asmvt` | `gbx_st_asmvt` | -| `BNG_Polyfill` | `gbx_bng_polyfill` | -| `GBX_RST_Slope` | `gbx_rst_slope` | - -Lowercasing relaxes only the case dimension — a name that still doesn't match a registerable function after lowercasing is treated as unknown and raises (the typo guard below). Leading/trailing whitespace is stripped before normalizing. - -**DataSource (readers/writers) names** use a different convention — a `_gbx` **suffix** rather than a `gbx_` prefix (`raster_gbx`, `shapefile_gbx`, …). For `ds.register`, normalization strips + lowercases, then appends `_gbx` if absent: `raster` / `RASTER` / `raster_gbx` all resolve to `raster_gbx`. This is a second normalizer (`normalize_datasource_name`); the validation + close-match behavior is identical. - -### Validation - -Validate every normalized name against the package's full registerable set (the union of all groups). On any unrecognized name, raise `ValueError` that lists the unrecognized name(s) and, for each, up to 3 `difflib.get_close_matches` suggestions from the valid set. Rationale: a silently-unregistered function would otherwise surface much later at call time as `UNRESOLVED_ROUTINE` — fail fast at registration with an actionable message. - -Example message: -``` -register(only=...) got unknown function name(s): ['rst_slpe']. - rst_slpe -> did you mean: rst_slope? -Valid names are the gbx_* SQL names (or their short forms) for this tier. -``` - -## Mechanism — grouped registrar map - -Each light `register()` is currently a flat sequence of `spark.udf.register(name, udf)` / `spark.udtf.register(name, cls)` calls, with `pygx` and `pyvx` interleaving per-sub-module availability guards (`_env.assert_quadbin_available()`, `assert_bng_available()`, `assert_custom_available()`; `assert_mvt_available()`, `assert_legacy_available()`, `assert_tin_available()`). - -Refactor each `register()` to build an **ordered list of groups**, each a pair: - -``` -(assert_available_fn, { sql_name: register_fn(spark) -> None, ... }) -``` - -- `pyrx`: one group, guard `assert_rasterio_available`, mapping every scalar/agg UDF (derived from `SQL_REGISTRY`), every UDTF (the ~20 `spark.udtf.register` names), and `gbx_pmtiles_agg` (via `register_pmtiles_agg`) to its registration closure. -- `pygx`: three groups — `quadbin` (guard `assert_quadbin_available`), `bng` (guard `assert_bng_available`), `custom` (guard `assert_custom_available`) — each mapping its `gbx_quadbin_*` / `gbx_bng_*` / `gbx_custom_*` names. -- `pyvx`: groups for `mvt` (`assert_mvt_available`: `gbx_st_asmvt`, `gbx_st_asmvt_pyramid`), `legacy` (`assert_legacy_available`: `gbx_st_legacyaswkb`), `tin` (`assert_tin_available`: `gbx_st_triangulate`, `gbx_st_interpolateelevationbbox`, `gbx_st_interpolateelevationgeom`), and `pmtiles` (no guard / its own: `gbx_pmtiles_agg`). - -`register(spark, only=None)` algorithm: -1. `spark = spark or SparkSession.builder.getOrCreate()`. -2. If `only` is not None: normalize + validate (raise on unknown) → `wanted: set[str]`. -3. For each group in order: - - `selected = {n: fn for n, fn in group.entries.items() if only is None or n in wanted}` - - if `selected` is non-empty: call the group's `assert_available_fn()`, then call each `fn(spark)` in the group's defined order. - -Key property: with `only=None`, every group runs its guard and registers every name **in the same order as today** — a behavior-preserving refactor. A guard for a sub-module with no selected functions is **not** invoked (so `pygx` `only=['gbx_quadbin_polyfill']` never asserts bng/custom availability). - -A small shared helper module (`databricks/labs/gbx/_register.py`) holds `normalize_name(name)`, `normalize_datasource_name(name)`, `resolve_only(names, valid, normalizer=normalize_name)` (validation + close-match error), and `run_groups(groups, spark, only)` so the packages don't duplicate the logic. The grouped registrar structures stay per-package (they reference package-local UDF/UDTF objects). - -### Readers/writers (`ds.register`) - -`ds.register.register` is a flat list of 9 `spark.dataSource.register(source)` calls — no availability guards, no `run_groups` needed. It builds a `{format_name: source_class}` map from the existing `_SOURCES` tuple via each class's `name()` classmethod, then: `only=None` → register all (today's behavior); `only=[...]` → `resolve_only(only, names, normalizer=normalize_datasource_name)` then register just the selected source classes. Unknown format name raises `ValueError` (same as SQL). - -## Testing (TDD) - -Per package (`pyrx`, `pygx`, `pyvx`), using the existing Python test session fixture: -1. `only` with a subset registers exactly those functions (each present in `spark.catalog`/callable) and at least one omitted function is **absent**. -2. Both name forms resolve: `only=['rst_slope']` and `only=['gbx_rst_slope']` register the same function. -3. Unknown name raises `ValueError` whose message contains the offending name. -4. `only=None` registers the full set (count/spot-check parity with the pre-refactor behavior). -5. A UDTF is selectable by name (e.g. `pyrx` `only=['gbx_rst_retile']`; `pyvx` `only=['gbx_st_asmvt_pyramid']`), and the pmtiles agg is selectable (`only=['gbx_pmtiles_agg']`). -6. `pygx` `only=['gbx_quadbin_polyfill']` does **not** raise from the bng/custom availability guards (sub-module isolation). Where practical, assert the guard isn't tripped (e.g. monkeypatch the bng/custom `assert_*` to raise and confirm it is not called). -7. `only=[]` registers nothing and does not error. - -For `ds.register`: `only` with a subset registers exactly those formats (e.g. `spark.read.format("raster_gbx")` resolves) and an omitted format does not; format name accepted with and without the `_gbx` suffix (`raster` ≡ `raster_gbx`); unknown format raises `ValueError`; `only=None` registers all 9. - -Tests are pure-Python (no Docker/JAR) and run via `gbx:test:python --path python/geobrix/test//`. - -## Out of scope (future follow-up) - -Heavyweight `only=`. It is feasible but requires a JAR change + cluster re-validation: an optional `only: Set[String]` on `RegistryDelegate` (skip `register(companion)` when `only` is non-empty and the companion name is absent), threaded from a new `RegisterBatch` `"only"` option through each Scala package `functions.register(spark, only)` signature, with the heavy Python `register(only=...)` passing `.option("only", ",".join(...))`. Tracked separately. - -## Docs - -Add a short **"Registering a subset"** subsection to `docs/docs/api/execution-tiers.mdx`: the `only=` signature for both SQL functions (`.functions.register`) and readers/writers (`ds.register.register`, by format name), the both-name-forms note, the light-over-heavy mixing pattern, and a note that heavy `only=` is not yet available (use whole-tier registration order for heavy). diff --git a/docs/superpowers/specs/2026-06-23-h3-cell-rasterizer-design.md b/docs/superpowers/specs/2026-06-23-h3-cell-rasterizer-design.md deleted file mode 100644 index 9f1d50bbb..000000000 --- a/docs/superpowers/specs/2026-06-23-h3-cell-rasterizer-design.md +++ /dev/null @@ -1,283 +0,0 @@ -# H3 cell rasterizer (`rst_h3_rasterize_agg` + `rst_h3_gridspec`) — Design - -**Date:** 2026-06-23 -**Branch:** `beta/0.4.0` -**Status:** design (pending user review) - -## Goal - -Add a **DGGS-cell rasterizer** to RasterX: rasterize a set of H3 cell ids (each -carrying a value) into a raster tile — the inverse of the existing -`rst_h3_rastertogrid*`. Scope is **step (2)** of the customer pipeline only; step -(1) polyfill is done upstream, and step (3) band-stacking reuses the existing -`rst_frombands_agg`. - -## Motivation - -Customer (telco) pipeline: transmitter coverage multipolygons over ~80 -signal-strength thresholds → `h3_polyfill` at res 12 (done) → **rasterize the -cellids grouped by `(TxId, SourceYear, SourceMonth, Threshold)`** → stack the -per-threshold rasters as bands. GeoBrix has rich raster→DGGS support -(`rst_h3_rastertogrid*`, `rst_h3_tessellate`) and vector→raster (`rst_rasterize`, -`rst_rasterize_agg`), but **no cell→raster primitive in either tier** — confirmed -gap. This feature fills it, symmetric with the raster→DGGS path. - -## Scope - -In scope: -- `rst_h3_rasterize_agg` — grouped aggregator: H3 cells (+ optional value) → tile. -- `rst_h3_gridspec` — grouped helper: H3 cells → the complete shared output grid - (snapped origin + pixel size + dims + srid) so per-threshold bands stack aligned. -- Both tiers: the rasterizer is a heavyweight Scala UDAF + a lightweight pyrx grouped - `pandas_udf`; the grid-spec helper is a scalar bbox + native `min/max` + snap in both tiers. -- Validation (CI synthetic round-trip + a committed FCC fixture) and a - DEM-isoband notebook example. - -Out of scope (separate follow-ups, after H3 proves out): -- `rst_quadbin_rasterize_agg` / `rst_quadbin_gridspec`. -- `rst_bng_rasterize_agg` / `rst_bng_gridspec`. -- A scalar (non-agg) or scalar-over-array form — agg-only for now. -- A boundary-polygon-burn mode (centroid-only for now; could be a future `mode=`). -- Step (1) polyfill and step (3) `rst_frombands_agg` stacking (already exist). - -## Algorithm: pixel-centroid burn (inverse of `rst_h3_rastertogrid`) - -For each output pixel, take its **center**, convert to lon/lat (unproject if the -output CRS is projected), call `h3.latlng_to_cell(lat, lon, resolution)`, and burn -the cell's value if that cell is in the group's set; otherwise NoData (-9999.0). - -- This is the exact inverse of `rst_h3_rastertogrid*` (which sends each pixel - centroid *to* a cell), so a `rastertogrid → rasterize` round-trip is near-lossless - at a matching grid/resolution. -- H3 hexagons tile without gaps/overlap, so every pixel center maps to exactly one - cell — a clean partition, no edge double-counting. -- **Resolution is inferred from the cells** (`h3.get_resolution`); all cells in a - group must share one resolution (validate; error if mixed). This makes the - function resolution-agnostic (res-8 FCC data and res-12 customer data exercise the - same code path). -- **Implementation note (perf):** do not H3-index every pixel in a large grid - blindly. Build the `cell→value` map once per group; for the output grid compute - pixel-center lon/lat via the affine transform, then index per pixel (the existing - `gridagg._h3_cells` scalar-loop pattern). Bound the work to the cells' bbox. - The heavy tier mirrors `RST_H3_RasterToGrid`'s centroid math - (`xGeo = gt[0] + (px+0.5)*gt[1] + (py+0.5)*gt[2]`). - -## Output grid (extent, pixel size, CRS) - -- **Default — fully auto:** extent = union bounding box of the group's cell - boundaries; pixel size derived from the H3 resolution (so each cell maps to ≥1 - pixel — e.g. pixel ≈ average hexagon edge length at that resolution). One-call - convenience. -- **Overrides (full control):** caller may supply any of `pixel_size`, or an explicit - `xmin, ymin, xmax, ymax, width, height` (like `rst_rasterize`). For aligned band - stacking, feed every threshold the *same* grid from `rst_h3_gridspec` (see below). -- **CRS:** default **EPSG:4326** (H3 is natively WGS84; simplest, no reprojection). - Optional projected `srid` → pixels are uniform **meters** (square on the ground); - pixel centers are unprojected to lon/lat for the H3 lookup. A 4326 result reprojects - to any CRS post-hoc via the existing `rst_transform(tile, target_srid)` (both tiers). - Note: a fixed-degree pixel in 4326 is uniform in degrees, not meters (longitude - degrees shrink with `cos(lat)`); use a projected `srid` when metric uniformity matters. - -## Functions - -### `rst_h3_rasterize_agg` - -Grouped aggregator. One H3 cell id per row; accumulate per group; burn on eval. - -**Signature (SQL / both tiers):** -``` -rst_h3_rasterize_agg( - cellid BIGINT, -- H3 cell id (one per row) - value DOUBLE, -- optional; omitted/null -> 1.0 (presence mask) - srid INT, -- optional; default 4326 - pixel_size DOUBLE, -- optional; default auto from H3 resolution - xmin DOUBLE, ymin DOUBLE, xmax DOUBLE, ymax DOUBLE, -- optional explicit extent - width INT, height INT, -- optional explicit dims (alt to pixel_size) - mode STRING, kring_pad INT -- optional auto-extent controls (default 'centroids', 1) -) [GROUP BY ...] -> tile -``` -- **Grid:** when an explicit extent (`xmin…height`) is supplied (e.g. from - `rst_h3_gridspec` for aligned stacking), it is used as-is. Otherwise the extent is - derived per `mode` + `kring_pad` exactly as in `rst_h3_gridspec`, on the same snapped - global lattice — so a standalone call and a per-cell-then-merge call agree. -- **Value:** burns the per-cell `value`; default `1.0` over covered cells with - NoData elsewhere (presence mask) when `value` is omitted/null. Cells of one - resolution don't overlap, so there is no within-group burn conflict; if two rows - carry the same cell with different values, last-wins after a deterministic sort - (mirrors `rst_rasterize_agg`'s canonical ordering). -- **Heavy:** UDAF modeled on `RST_RasterizeAgg` (accumulate cell ids + values, cap - buffer at the same 200 MiB guard, build the grid via - `VectorRasterBridge.buildEmptyRaster`, burn by centroid, return the - `STRUCT` tile with `cellid=0`). -- **Light:** new grouped `pandas_udf` in pyrx. Per the established light-agg - convention, the **light SQL aggregate returns `BINARY`** (raster bytes), and the - Python wrapper `rx.rst_h3_rasterize_agg(...)` composes the full tile struct - (BINARY → `rst_fromcontent`-style wrap). Document the SQL return-type deviation as - an orange `:::warning` (both return types + the PySpark `StructType`-in-grouped-agg - reason + the GROUP-BY recovery example), exactly as the other `rst_*_agg` light - functions do. - -### `rst_h3_gridspec` (shared-grid / canvas definer) - -Defines the **complete shared output grid** over a cell set — the "canvas" every -band paints onto — so all thresholds of a transmitter rasterize to a -**byte-identical transform** and the per-threshold bands stack cleanly via -`rst_frombands_agg`. It returns not just a bounding box but the full affine grid -(origin + pixel size + dims + srid); its job is to *negotiate one shared reference -frame* for the bands, not to process pixel data. - -**Return — the full grid spec:** -``` -STRUCT -``` - -**Bounds mode + padding.** -- `mode='centroids'` (default) — bounds = bbox of the cell **centroids**, snapped - outward to the pixel lattice via `floor(xmin/ps)*ps` and `ceil(ymax/ps)*ps` so - every centroid falls within a full pixel. Compact; ≈1 pixel per cell at the - default pixel size. -- `mode='spatial_envelope'` — bounds = OGC **envelope** of the cell geometries - (full hexagon footprints). Use with sub-cell pixels when you want complete cell - shapes rendered. -- `kring_pad` (int, default `1`) — expand the cell set by N H3 rings (`h3.grid_disk`) - **before** computing bounds, in *both* modes. The pad cells are **NoData** — they - only enlarge the canvas. Purpose: (a) preserve each cell's full *value* footprint - instead of chopping it at its centroid; (b) provide a margin for the per-cell - independent-rasterize→merge pattern (below); (c) de-degenerate the single-cell - case (a lone cell's centroid is a point → `kring_pad=1` uses its 6 neighbors' - centroids). `kring_pad=0` → tight bounds (centroids: 1 px / chop at centroid; - spatial_envelope: the tight single-hexagon envelope). For large sets the ring - expansion may be approximated by padding the bbox by `N × cell_spacing` rather - than materializing every kring cell. - -**Grid alignment / snapping (prevents half-pixel shifts when stacking):** -- Compute the bounds per `mode` + `kring_pad` above (in `srid`). -- Choose `pixel_size` (auto from the H3 resolution — e.g. a clean fraction of the - cell edge so each cell maps to a consistent integer pixel count — or caller-given). -- **Snap the origin to the pixel grid:** `xmin = floor(bounds.xmin / pixel_size) * - pixel_size`, `ymax = ceil(bounds.ymax / pixel_size) * pixel_size`; then - `width = ceil((bounds.xmax - xmin)/pixel_size)`, `height = ceil((ymax - bounds.ymin)/pixel_size)`, - and `xmax = xmin + width*pixel_size`, `ymin = ymax - height*pixel_size`. -- Result: a deterministic, pixel-aligned grid on a **global lattice** (origin is a - multiple of `pixel_size`, independent of which cells are present). Two consequences: - every threshold of a Tx handed the *same* struct shares one origin/resolution/extent - (clean band stacking), **and** cells rasterized *independently* land on the same - lattice — so per-cell tiles abut/overlap exactly and `rst_merge_agg` mosaics them - losslessly (see the merge pattern below). - -**Implementation (both tiers, no custom UDAF):** a *scalar* per-cell bbox plus -native Spark `min/max` group aggregates, then the snap arithmetic. This avoids the -grouped-`pandas_udf` `StructType`-return limitation (see -[[light-agg-struct-return-convention]]) and is identical in both tiers. -- Scalar per-cell bbox `h3_cell_bbox(cellid, srid)` (centroid point in `mode='centroids'`, - hexagon envelope in `mode='spatial_envelope'`) — a scalar UDF from - `h3.cell_to_boundary` / `cell_to_latlng` / `H3.cellIdToGeometry`, reprojected to `srid`; - `kring_pad` applied via `h3.grid_disk`. -- Union bounds = `groupBy(...).agg(F.min(...), F.max(...))` in native Spark SQL; the - snap step produces the final grid struct. -- Exposed as a **Python/DataFrame helper** `rx.rst_h3_gridspec(df, cell_col="cellid", - *group_cols, srid=4326, pixel_size=None, mode="centroids", kring_pad=1)` returning the - grouped DataFrame with a `grid STRUCT<...>` column; SQL users build the same recipe - from the scalar `gbx_h3_cell_bbox`. (Final name is a plan-time decision; the `_agg` - suffix is intentionally NOT used — this is a spatial reduction / grid definer, not a - Spark aggregator UDAF.) -- Usage for stacking: compute the grid over `(TxId, SourceYear, SourceMonth)` - (union across all thresholds) → join the `grid` struct back onto each - `(…, Threshold)` group → pass `grid.xmin/ymin/xmax/ymax/width/height/srid` into - `rst_h3_rasterize_agg`. All thresholds then share one canvas. - -**Per-cell independent rasterize → merge (first-class pattern).** Because the snapped -grid lives on a global lattice, you can rasterize each cell (or small cell group) -*independently* — `rst_h3_rasterize_agg` with `kring_pad ≥ 1` so each cell's full -footprint plus a margin is captured — and then mosaic the resulting tiles with the -existing `rst_merge_agg`. The per-cell tiles are lattice-aligned, so the merge is -lossless and order-independent. This is the embarrassingly-parallel alternative to -rasterizing a whole group against one big canvas. - -## Data flow (customer + example) - -``` -cells(Threshold, TxId, Year, Month, cellid_12) - │ -- per-transmitter common grid (union across thresholds) - ├─► rx.rst_h3_gridspec(cells, "cellid", "TxId","Year","Month") -> grid(shared canvas struct) - │ - └─► join grid ─► groupBy(TxId,Year,Month,Threshold) - .agg(rst_h3_rasterize_agg(cellid, lit(1), - srid=grid.srid, xmin=grid.xmin, ..., width=grid.width, height=grid.height)) - AS band_tile -- one aligned band per threshold - │ - └─► groupBy(TxId,Year,Month) - .agg(rst_frombands_agg(band_tile, Threshold_rank)) -- stack bands - AS coverage_stack -``` - -## Error handling - -- Mixed resolutions within a group → clear error naming the offending resolutions. -- Empty group → null tile (no rows to burn). -- Invalid/0 cell id → skipped with a warning (don't fail the whole group). -- Auto extent on a single cell → that cell's bbox (degenerate but valid). -- Projected `srid` with an out-of-range cell (antimeridian/pole) → fall back to the - cell boundary's lon/lat bbox; document the limitation. - -## Testing - -Real assertions on real data; matplotlib `Agg`; no mocking of h3/rasterio. - -**CI (committed, no external download, exact oracle):** -- **Round-trip vs the inverse:** sample DEM `srtm_n40w073.tif` → - `rst_h3_rastertogridavg(res)` → `(cellid, measure)` → - `rst_h3_rasterize_agg(cellid, measure, )` → assert the burned values - match the source's centroid-sampled values within tolerance (toraster is the - centroid inverse of rastertogrid). -- **Partition property:** `h3.polyfill` a synthetic polygon at a resolution → - `rst_h3_rasterize_agg` → assert every burned pixel's centroid re-indexes - (`latlng_to_cell`) to a cell *in* the set, and every NoData pixel's centroid does - *not* — proving the clean partition. -- **Extent helper:** `rst_h3_gridspec` over a known cell set equals the union of - `h3.cell_to_boundary` bboxes (4326), and in a projected `srid` equals the - reprojected bbox. -- **Presence-mask default / value passthrough:** omitted `value` → covered=1.0, - elsewhere NoData; explicit `value` → that value burned. -- **Tier parity:** heavy and light produce the same cell-set → same covered-pixel - set (JAR-gated parity test, per repo convention). -- **Binding parity / light-agg convention:** light SQL returns BINARY, Python wrapper - returns the tile struct; add the three bindings (Scala `override def name`, - Python `functions.py`, `function-info.json`) + `registered_functions.txt`. - -**Realistic fixture (committed, public FCC data):** -- Curate a small subset of `bdc_12_UnlicensedFixedWireless_fixed_broadband_*.csv` - (already res-8 H3 via `h3_res8_id`; one provider, one FL county via `block_geoid` - prefix `12086` = Miami-Dade, a few `max_advertised_download_speed` tiers) → a - few-hundred-cell CSV committed under the test fixtures (FCC data is open/public). - Maps to the customer flow: `provider_id`≈`TxId`, speed tier≈`Threshold`, - `h3_res8_id`≈the polyfilled cellid. Rasterize per (provider, speed-tier), stack the - tiers; assert covered cells match the fixture. This validates step (2) on real, - pre-celled coverage data (no polyfill needed). - -## Notebook example (full flow from polygons — DEM isobands) - -A standalone example notebook demonstrating the **complete** flow (for context; -the *function* scope remains step 2): -- Start from the sample DEM `srtm_n40w073.tif`; quantize into N filled elevation - isobands (e.g. every 100 m) via `rasterio.features.shapes` on the banded array → - multipolygons over a range of thresholds (elevation bands stand in for signal - thresholds). -- `h3_polyfill` each band → cells with `(band_level, cellid)`. -- `rst_h3_gridspec` for a common grid → `rst_h3_rasterize_agg` per band → - `rst_frombands_agg` stack. -- Render with `gbx.viz.plot_raster`; the stacked bands visibly reconstruct the - terrain — demonstrating *and* validating the rasterize→stack flow with **no - external data**. -- Telco-authentic variant (optional, external): FCC **Mobile** coverage polygons - (per provider/technology) stacked by technology tier. Dataset choice deferred to - the notebook build; DEM isobands is the headline. - -## Docs - -- New entries on `docs/docs/api/raster-functions.mdx`: `rst_h3_rasterize_agg` - (Aggregator section, ``, with the light-tier BINARY `:::warning`) - and `rst_h3_gridspec`. Cross-reference `rst_h3_rastertogrid*` as the inverse and - `rst_frombands_agg` for stacking. No internal/"wave" vocabulary (QC gate). -- Notebook example page under `docs/docs/notebooks/` when the notebook lands. diff --git a/docs/superpowers/specs/2026-06-24-vizx-static-map-design.md b/docs/superpowers/specs/2026-06-24-vizx-static-map-design.md deleted file mode 100644 index 9bcecdbaa..000000000 --- a/docs/superpowers/specs/2026-06-24-vizx-static-map-design.md +++ /dev/null @@ -1,227 +0,0 @@ -# VizX static-map helper (`plot_static`) — Design - -**Date:** 2026-06-24 -**Branch:** off `beta/0.4.0` (single PR) -**Module:** `databricks.labs.gbx.vizx` (Python-only, tier-agnostic, `[vizx]` extra) - -## Goal - -Add one public helper, `plot_static`, that renders Spark- or GeoPandas-derived -vector geometries (or DGGS cells) over a tiled basemap as a **static** -matplotlib figure — a prettier alternative to plain `GeoDataFrame.plot()` (no -map context) and to folium `.explore()` (interactive, but renders a blank -*"Make this Notebook Trusted"* placeholder on GitHub and the docs site). - -The name `plot_static` is deliberate: it signals the non-interactive -counterpart to `.explore()`, and it accepts **both** a GeoDataFrame and a Spark -DataFrame. - -## Why this works on GitHub (the caching question) - -GitHub never executes a notebook — it renders the **saved output cells** -committed in the `.ipynb`. When an executed notebook calls `plot_static`, -contextily fetches basemap tiles *at execution time* (on a cluster/laptop with -egress) and matplotlib rasterizes the basemap **into the output PNG**, which is -embedded in the notebook and committed. GitHub displays the baked pixels with -**zero network at render time**. This is the same model the existing static -`.plot()` cells use; we are only adding a basemap layer to the baked image. - -Consequence: the basemap "just works" on GitHub as long as whoever *executes* -the notebook has egress. No basemap tiles are committed to the repo. - -## Non-goals (v1) - -- No interactive output (that remains `.explore()`). -- No committed/offline tile cache for no-egress executors (e.g. Docker - doc-tests). The fallback below covers no-egress by rendering without a - basemap. -- `grid_system` values `'quadbin'`, `'bng'`, `'custom'` are **forward-declared - but not implemented** in v1 (fast-follow). `'custom'` is the trickiest: - custom grids need their own cell→boundary resolver, currently heavy-only. - - **Fast-follow note (quadbin / bng):** no pure-Python cell→boundary port is - needed — the light tier already ships driver-side scalar `_aswkb` impls that - mirror what `_h3_boundary` does for h3: - - quadbin: `databricks.labs.gbx.pygx._quadbin.as_wkb(cell: int) -> bytes` - - bng: `databricks.labs.gbx.pygx._bng.cell_aswkb(cell_id: int) -> bytes` - - So each resolver is one `_GRID_DISPATCH` entry that, per collected cell id, - calls the scalar `_aswkb` → WKB `bytes` → `parse_geom` (shapely). This runs - **driver-side after the collect** (exactly like the h3 path), so it is - unit-testable in the dev container with **no Spark-runtime / SQL-registration - dependency**. (The columnar `quadbin_aswkb` / `bng_aswkb` SQL functions remain - available as an in-Spark coercion alternative, but the scalar impls are - preferred here for the same reason h3 uses the `h3` lib directly.) - - `'custom'` stays the trickiest: its scalar impl is - `pygx._custom.cell_aswkb(conf: CustomGridConf, cell_id: int)` — it needs the - grid configuration threaded through, so it requires either an extra param or - resolving the conf from the frame, not just a cell-id column. - -## Public API - -```python -plot_static( - data, # Spark DataFrame OR geopandas.GeoDataFrame - *, - geom_col=None, # geometry/cell column; auto-detected if None - grid_system=None, # None | 'h3' | 'quadbin' | 'bng' | 'custom' (v1: None, 'h3') - column=None, # attribute column → choropleth; None → single style - cmap="viridis", - legend=True, - basemap=True, # contextily tiles; graceful fallback if unreachable - basemap_source=None, # contextily provider; default CartoDB.Positron - alpha=0.8, - edgecolor="face", - markersize=None, # point layers - title=None, - fig_w=10, fig_h=10, - max_rows=10_000, # driver-collect guard (same convention as adapters) - srid=None, # CRS override for bare WKT/WKB (default: assume 4326) - ax=None, # overlay onto an existing Axes -) -> "matplotlib.axes.Axes" -``` - -- **Returns** the `Axes` and renders inline. `pyplot.show()` is called **only - when `plot_static` created the figure** (i.e. `ax is None`). Passing the - returned `ax` back in composes overlays (cells choropleth → grid boundary → - points) on one basemap. This replaces the notebooks' - `gdf.plot(...)` + `grid_gdf.boundary.plot(ax=ax)` pair. -- Requires the `[vizx]` extra; guarded by `assert_viz_available()` (matplotlib + - geopandas) as the other plotters are. - -## Architecture / components - -New file `python/geobrix/src/databricks/labs/gbx/vizx/_static_map.py` holding -`plot_static` plus private helpers. Exported from `vizx/__init__.py`. - -### 1. Input resolution → GeoDataFrame - -Private `_resolve_gdf(data, geom_col, grid_system, max_rows, srid)` returns an -EPSG:4326 GeoDataFrame: - -- **`data` is a GeoDataFrame** → use as-is (its own CRS; reprojected later). -- **`data` is a Spark DataFrame:** - - Resolve `geom_col`: explicit arg, else auto-detect — a native - `GEOMETRY`/`GEOGRAPHY`-typed column first, else a column named - `wkt`/`geometry`/`geom`, else (when `grid_system` is set) the lone - remaining candidate. Ambiguous/none → `ValueError`. - - **`grid_system is None`** (geometry) — branch on - `df.schema[geom_col].dataType`: - - native `GEOMETRY`/`GEOGRAPHY` → coerce **in Spark** with - `expr("st_asbinary()")` to WKB bytes; resolve SRID via - `expr("st_srid()")` (`GEOGRAPHY` ⇒ 4326); collect → `parse_geom`. - - `BinaryType` → collect bytes → `parse_geom` (WKB/EWKB). - - `StringType` → collect strings → `parse_geom` (WKT/EWKT, EWKT - `SRID=...;` prefix honored). - - any other type → `ValueError` naming the dtype and suggesting - `st_asbinary` / `st_astext`. - - **`grid_system` set** (cells) — dispatch table - `{'h3': _h3_boundaries, 'quadbin': _nyi, 'bng': _nyi, 'custom': _nyi}`: - - `'h3'` (v1): each cell id may be a **string** h3 index or a **long** - bigint; longs are converted via `h3.int_to_str`, then - `h3.cell_to_boundary` → shapely `Polygon` (lng, lat order). CRS 4326. - - `'quadbin'`/`'bng'`/`'custom'`: `NotImplementedError( - "grid_system='' is a planned fast-follow; not supported yet")`. - - **Collect guard**: same truncate-and-warn at `max_rows` as - `as_gdf`/`cells_as_gdf` (`max_rows=None` opts out). - -Reuses the shared decoder `databricks.labs.gbx._geom.parse_geom`, so -`plot_static` accepts exactly the same geometry encodings as every other -`gbx_st_*` function (geometry-input-consistency rule). The `grid_system` -dispatch table is the single seam where quadbin/bng/custom slot in later -without reshaping the API. - -### 2. CRS + render - -- When `basemap=True`, reproject the resolved gdf to **EPSG:3857** (contextily - requires Web Mercator). -- Draw: - `gdf.plot(column=column, cmap=cmap, legend=legend, alpha=alpha, - edgecolor=edgecolor, markersize=markersize, ax=ax)`. -- Axis ticks off; set `title` if given. - -### 3. Basemap (with graceful fallback) - -- Lazy `import contextily as cx` **inside** the `basemap` branch. -- `cx.add_basemap(ax, source=basemap_source or cx.providers.CartoDB.Positron, - crs=gdf.crs)`. -- Wrap in `try/except Exception`: on **any** failure (no egress, HTTP error, - contextily not installed) → `warnings.warn(...)` and continue **without** the - basemap. The figure still renders and the PNG still bakes — so a locked-down - cluster degrades to a basemap-less map rather than erroring. -- `basemap=False` skips tiles entirely (deterministic, no network). -- Default provider: **CartoDB.Positron** (light, clean under viridis). - -## Dependency - -`contextily` is the new package. Two layers, per the project's supply-chain -pinning rule (execution environments are exact-version **and** hash-pinned; -range pins are only for the published library extra): - -- **Published `[vizx]` extra** (`python/geobrix/pyproject.toml`): add a range - pin `contextily>=1.5,<2` (library convention, alongside - matplotlib/geopandas/folium/mapclassify). -- **Execution-env lock files** (local **and** GitHub CI): regenerate with - `contextily` and **all its transitive deps** exact-version + `--hash=sha256` - pinned (`--require-hashes` style), the same way every other package in these - files is pinned: - - `python/geobrix/requirements-pyrx-ci.txt` — the light-tier GitHub CI lane - that runs the vizx tests (already carries geopandas/matplotlib/folium/ - mapclassify; `contextily` must be added so the basemap-fallback test can - import and monkeypatch it). - - `python/geobrix/requirements-dev-container.txt` — the local dev container. - - Regenerate via the repo's pip-compile path (do not hand-edit hashes), then - verify the lock installs cleanly in a fresh venv (a dir added to CI needs - *all* its non-stubbed third-party imports present in the lock, not just - what the full dev extras happen to provide). - -`contextily` is lazy-imported only in the basemap branch; its absence at -runtime is still handled by the warn-and-fallback path (not a hard error). -`assert_viz_available` continues to guard matplotlib + geopandas only. - -## Testing (TDD) - -`python/geobrix/test/vizx/test_static_map.py` — runs in the dev container with -no Databricks runtime and no network: - -1. WKT-string Spark DF → expected polygon geometry; returns an `Axes`; one - figure produced. -2. WKB-binary Spark DF → geometry identical to the WKT case (round-trip via - `parse_geom`). -3. `grid_system='h3'` with **string** ids and with **long** ids → both produce - identical cell-boundary polygons. -4. `column=` → choropleth path renders with a legend. -5. `ax=` overlay → a second `plot_static` call draws onto the same `Axes` and - returns that same object. -6. `max_rows` smaller than input → truncation `UserWarning`. -7. Unknown column dtype → `ValueError`; `grid_system` in - `{'quadbin','bng','custom'}` → `NotImplementedError`. -8. **Basemap fallback**: monkeypatch `contextily.add_basemap` to raise → - assert a warning is emitted and a figure is still produced. - -Real tile fetching is **not** exercised in CI (no egress); it is validated -manually by running an example notebook. The native `GEOMETRY`/`GEOGRAPHY` -coercion path (`st_asbinary`/`st_srid`) only runs where those built-ins exist -(Databricks/Serverless); unit tests cover the string/binary/h3 routing -directly. Headless rendering uses the `Agg` backend already established in the -vizx test suite. - -## Docs + notebook adoption - -- New `plot_static` section in `docs/docs/api/vizx.mdx`: signature, the - supported-encodings table (WKT, EWKT, WKB, EWKB, native GEOMETRY, native - GEOGRAPHY, H3 cell ids), `grid_system` (with the fast-follow note), the - basemap graceful-fallback note, and an overlay example. -- Adopt in the **h3-rasterize** notebook (replace the `.plot()` + boundary pair - with `plot_static(..., grid_system='h3', column='band_level')` over a - basemap) and **eo-series 01 / 03** cell maps, so the committed output PNGs - gain a basemap. xView object footprints optional. - -## Delivery - -- Lands on **PR #45** (`refactor/vizx-rebrand`), per standing guidance to route - this batch of vizx work through that PR. Commit locally as work progresses; - **push only on the user's go** (each push triggers a CI build). -- After merge, fast-follow tasks: implement `grid_system` `'quadbin'`, `'bng'`, - and `'custom'` cell→boundary resolvers in the dispatch table. diff --git a/docs/superpowers/specs/2026-06-25-geojson-streaming-write-design.md b/docs/superpowers/specs/2026-06-25-geojson-streaming-write-design.md deleted file mode 100644 index f79abc820..000000000 --- a/docs/superpowers/specs/2026-06-25-geojson-streaming-write-design.md +++ /dev/null @@ -1,131 +0,0 @@ -# Streaming single-file vector writes (bounded driver memory) — Design - -**Date:** 2026-06-25 -**Branch:** `beta/0.4.0` -**File:** `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (`VectorGbxWriter.commit` / `_write_streaming`) - -> **Scope note (generalized in implementation):** this began as a GeoJSON-only -> streaming write, but during implementation it was generalized to **all the -> pyogrio single-file writers — `geojson`, `shapefile`, and `gpkg`** (they all -> benefit identically from bounded-memory streaming; `_should_stream` covers the -> three). GeoJSON/Shapefile geometry is structural; GPKG renames its geometry -> column per batch to the format default (`geom`). `file_gdb_gbx` (native osgeo -> path, pyogrio bundled GDAL is read-only for it) is **not** streamed — a -> heavy-tier follow-up. The sections below are written GeoJSON-first; read -> "GeoJSON" as "any structural-geom streamed driver", plus the GPKG rename. - -## Purpose - -`geojson_gbx` (the single merged-file GeoJSON writer) assembles the whole output -on the driver: today `commit` reads every partition fragment into memory and -`pa.concat_tables(...)` them into one table, then `pyogrio.write_arrow` serializes -it. For large inputs (e.g. 782k features) that peak OOMs / times out the -single-node driver, surfacing on Serverless as a masked -`CancelledKeyException`. - -Make the GeoJSON write **stream**: feed `pyogrio.write_arrow` a single Arrow -**`RecordBatchReader`** that yields batches from the fragment `.arrow` files one -at a time, so neither a concat nor the final write materializes the whole -dataset. This bounds driver memory to ~one record batch and is a single O(n) -pass — which also avoids the GeoJSON quadratic-append problem (the reason we -concat today). - -## Scope (GeoJSON only) - -Only the **`GeoJSON`** driver changes. The other writers are deliberately left -as-is because they're limited by something streaming can't help, or already -stream: - -- **`shapefile_gbx`** — stays **concat**. A streamed/batched shapefile write - reintroduces silent `.dbf` field-width truncation (GDAL fixes string widths - from the first batch); concat sizes fields to the global max. Making shapefile - streaming safe would require a separate **field-width pre-scan** pass (scan all - fragments for max widths, create the layer, then stream-append) — a future - enhancement, not this spec. (Shapefile also has a 2 GB-per-file cap, but that is - a separate hard limit, not the memory concern this spec addresses.) -- **`gpkg_gbx` / `file_gdb_gbx`** — already append per fragment (bounded memory, - no 2 GB cap). -- **`geojsonl_gbx`** — already shards per partition (no driver merge); remains - the most scalable large-data path. This change does NOT make single-file - GeoJSON match geojsonl's throughput — it is still single-node, just - memory-bounded. - -## Mechanism - -In `VectorGbxWriter.commit`, dispatch by driver into three paths: -`GeoJSON` → stream (new), `ESRI Shapefile` → concat (unchanged), -`GPKG`/`OpenFileGDB` → per-fragment append (unchanged). (`_should_concat` -becomes `ESRI Shapefile` only; GeoJSON moves to the new streaming branch.) - -The GeoJSON streaming branch: - -1. **Infer geometry type + CRS cheaply.** Read only the **first non-empty - fragment** (one partition — bounded) and derive `geom_type` + `crs` from its - first non-null feature via the existing `_infer_geom_crs` logic. Do not - materialize all fragments for inference. -2. **Build a chaining `RecordBatchReader`.** Open each fragment `.arrow` (Arrow - IPC file) and iterate its record batches; for each batch drop the meta columns - (`srid`/`proj`, via the same column set as `_drop_meta_cols`) and yield it. - Wrap the generator with - `pyarrow.RecordBatchReader.from_batches(stream_schema, gen)`, where - `stream_schema` is the fragment schema minus the meta columns. All fragments - share one explicit schema (set at executor write time by - `_writer_arrow_table`), so batches are schema-consistent. -3. **Single streaming write to driver-local temp.** - `pyogrio.write_arrow(reader, local_out, driver="GeoJSON", - geometry_name=self.geom_col, geometry_type=geom_type, crs=crs)` — one call; - GDAL pulls batches and appends features incrementally to the local file. - (GeoJSON geometry is structural, so the per-format output-geom rename does not - apply.) -4. **Copy to the Volume.** Byte-copy the finished local `.geojson` to the target - with the existing FUSE-safe `_copy_file_to_fuse`. - -## Volume / object-storage constraint - -GDAL needs random-access (seeking) to assemble the file, which a UC Volume -(cloud object storage, no random access) does not provide — so the assembly -stays on **driver-local temp** (random-access OK), exactly as today, and only the -**finished** file is copied to the Volume with a single sequential byte copy. -The streaming change is purely how the local temp file is assembled; the -Volume-side I/O (one byte copy) is unchanged. - -## Memory profile - -Peak ≈ one record batch being written + one fragment's batch being read + GDAL's -internal buffers — instead of "all fragments + the concatenated table." The -inference step reads one fragment (one partition), not the whole dataset. - -## Fallback - -If the bundled pyogrio/GDAL `write_arrow` does not accept a streaming -`RecordBatchReader` (only a materialized table), fall back to the current -concat-then-write path for GeoJSON. A TDD step confirms which the bundled -GDAL supports; the streaming path is gated on that capability. - -## Testing - -- **Round-trip correctness (primary):** a multi-partition `geojson_gbx` write - streams to one file and round-trips via `geojson_gbx` to the same row count, - geometry types, and attribute values (incl. an all-null attribute column, to - confirm the streamed schema typing still holds per batch). -- **Reader chaining (unit):** the batch generator chains multiple fragment - `.arrow` files and drops the meta (`srid`/`proj`) columns — assert the yielded - schema excludes them and the row count equals the sum across fragments. -- **Dispatch policy (unit):** GeoJSON takes the streaming path; Shapefile still - concats; GPKG/FileGDB still append per fragment. -- **CRS/geom-type inference:** derived from the first fragment matches the value - the old full-scan produced for the same data. -- Deterministic memory assertions are impractical (the DataSource `commit` runs - out-of-process and AQE coalesces tiny frames), so correctness + the - dispatch/structure unit tests are the guards, not a memory-spy. - -## Out of scope - -- Shapefile / GPKG / FileGDB / GeoJSONL behavior (unchanged). -- Lifting the shapefile 2 GB cap (impossible) — `geojsonl`/`gpkg` remain the - large-data recommendation, now documented in the writers overview. - -## Delivery - -Light tier only (no heavy `geojson` writer). Commits to `beta/0.4.0` (PR #46); -pushed on the user's go. diff --git a/docs/superpowers/specs/2026-06-25-shapefile-zip-option-design.md b/docs/superpowers/specs/2026-06-25-shapefile-zip-option-design.md deleted file mode 100644 index 247b08417..000000000 --- a/docs/superpowers/specs/2026-06-25-shapefile-zip-option-design.md +++ /dev/null @@ -1,89 +0,0 @@ -# Shapefile writer `zip` option — Design - -**Date:** 2026-06-25 -**Branch:** `beta/0.4.0` -**File:** `python/geobrix/src/databricks/labs/gbx/ds/vector.py` (`VectorGbxWriter`) - -## Purpose - -A shapefile is a multi-file bundle (`.shp/.shx/.dbf/.prj/.cpg`), and -`shapefile_gbx` currently writes it as a **directory** of those sidecars. Add a -`zip` write option so the bundle can be emitted as a **single, portable -`.shp.zip` file** instead. The readers already open a zipped shapefile, so the -output round-trips with no reader change. - -## Option - -- **`zip`** — boolean, default **`False`** (off). Parsed case-insensitively like - the other write options (`driverName`, `layerName`, `maxRecordsPerFile`, the - `geomCol`/`sridCol`/`projCol` set). -- **Shapefile only.** It is ignored / not offered for the other writers: - `gpkg`/`geojson` are already single files; `geojsonl` is a directory of shards - (zipping would defeat its splittable/parallel-read design); `file_gdb` is a - directory but goes through the native `osgeo` path and is a **separate - follow-up spec** (TBD from the lessons here). - -## Behavior - -- **`zip=False` (default):** unchanged — a **directory** holding the - `.shp/.shx/.dbf/.prj/.cpg` bundle. -- **`zip=True`:** a **single `.shp.zip` file** containing those sidecars at the - zip root, so a reader opens it via `/vsizip/.shp.zip` (GDAL finds the - `.shp` inside). Round-trips through `shapefile_gbx` unchanged — the reader maps - a `.zip`-suffixed path to `/vsizip/` and the ESRI reader already accepts - `.shp` / `.shz` / `.zip`, so `.shp.zip` is recognized. - -## Output naming - -`.save(path)` with `zip=True` produces `.shp.zip`: -- `.save(".../roads")` → `.../roads.shp.zip` -- `.save(".../roads.shp")` → `.../roads.shp.zip` -- `.save(".../roads.shp.zip")` → used as-is - -So `zip=True` yields one file, not a directory — consistent with the -"What `.save()` produces" table on the writers overview. - -## Implementation - -`VectorGbxWriter.commit` already assembles the shapefile bundle on local disk and -then copies the result to the Volume target. The zip option plugs in there: - -- **Preferred:** write the shapefile directly to a local path ending - `.shp.zip`. GDAL's ESRI Shapefile driver (3.1+) writes a single compressed - shapefile for a `.shp.zip` path, so `pyogrio.write_arrow(local_out, driver="ESRI - Shapefile", ...)` with `local_out = ".shp.zip"` produces the single file; - the existing copy step then byte-copies that one file to the Volume (FUSE-safe, - via `_copy_file_to_fuse`). -- **Fallback** (if the bundled GDAL build does not write `.shp.zip`): write the - directory bundle as today, then zip its sidecar files **flat at the zip root** - into `.shp.zip` and copy that single file. The TDD step determines which - path the bundled GDAL supports. - -The final Volume target is `.shp.zip` (per the naming rule). Only the -`ESRI Shapefile` driver consults `zip`; other drivers ignore it. - -## Testing - -- `zip=True` produces a **single `.shp.zip` file** at the target (not a - directory). -- The `.shp.zip` contains `.shp/.shx/.dbf/.prj` at the archive root. -- A `zip=True` write **round-trips** via `shapefile_gbx` to the same row count - and geometry. -- `zip=False` still yields the directory bundle (no regression). -- Output naming: `.save("roads")` + `zip=True` → `roads.shp.zip`. - -## Docs (required deliverable) - -- Update the **"What `.save(path)` produces"** table in - `docs/docs/writers/overview.mdx` to note that `shapefile_gbx` with `zip=True` - writes a single `.shp.zip` file (vs. the default directory bundle). -- Add a one-line `zip` option note on the shapefile writer page - (`docs/docs/writers/shapefile.mdx`). - -These land **with** the implementation so the docs match shipped behavior. - -## Delivery - -Commits to `beta/0.4.0` (flows into PR #46); pushed on the user's go. Light tier -only (the heavy tier has no shapefile writer). A `file_gdb` zip option is a -separate future spec. diff --git a/docs/superpowers/specs/2026-06-25-vector-writer-column-options-design.md b/docs/superpowers/specs/2026-06-25-vector-writer-column-options-design.md deleted file mode 100644 index 2581df8a1..000000000 --- a/docs/superpowers/specs/2026-06-25-vector-writer-column-options-design.md +++ /dev/null @@ -1,174 +0,0 @@ -# Vector writer column options (`geomCol` / `sridCol` / `projCol`) — Design - -**Date:** 2026-06-25 -**Branch:** `beta/0.4.0` -**Files:** -- Light: `python/geobrix/src/databricks/labs/gbx/ds/vector.py` -- Heavy: `src/main/scala/com/databricks/labs/gbx/vectorx/ds/geojsonl/` (`GeoJSONL_DataSource.resolveRoles`, `GeoJSONL_RowWriter`) - -## Purpose - -Let the vector writers be pointed at a DataFrame's existing geometry / SRID / -proj columns **by name**, so a user does not have to rename their columns to the -`X` / `X_srid` / `X_srid_proj` convention the writers auto-derive today. Pure -input-side convenience — the on-disk output is unchanged. - -## Tier scope (which writers exist) - -The options are added to **every vector writer that actually writes** — in both -tiers — with identical names and semantics: - -- **Light tier — all five writers:** `geojson_gbx`, `geojsonl_gbx`, `gpkg_gbx`, - `shapefile_gbx`, `file_gdb_gbx` (shared `_writer_col_roles`). -- **Heavy tier — `geojsonl` only.** The heavy `geojsonl` writer is the only - heavy vector *writer*; the other heavy OGR formats (`shapefile_ogr`, - `gpkg_ogr`, `geojson_ogr`, `file_gdb_ogr`) are **read-only** (no write path), - so there is nothing to add the options to there. (Heavy `geojsonl` already - derives roles by the same convention via `GeoJSONL_DataSource.resolveRoles` - and parses case-insensitive options in `GeoJSONL_RowWriter`.) - -So `geojsonl` gains the options in **both** tiers (parity); the other four -formats gain them in the light tier, which is where they write. - -## Options - -Three new write options (camelCase, parsed case-insensitively like the existing -`driverName` / `layerName` / `geometryType` / `maxRecordsPerFile`): - -| Option | Role | Required | -|---|---|---| -| `geomCol` | geometry column (Binary WKB **or** String WKT) | geometry must resolve | -| `sridCol` | CRS authority-code column (String; `"0"` = unknown) | srid must resolve | -| `projCol` | PROJ4 fallback column (String) | optional | - -## Resolution - -For each role: use the option if given; otherwise fall back to its -default/convention name **if that column is present**. - -- **geom** = `geomCol`, else the auto-derived geom (the `X` paired with the lone - `X_srid` column, as today). If `geomCol` is given, the srid/proj defaults are - derived from it (`_srid`, `_srid_proj`). -- **srid** = `sridCol`, else `_srid` if present. -- **proj** = `projCol`, else `_srid_proj` if present. - -**Required-ness** (the `if present at all` rule, made precise): -- **geom — required.** Cannot write without geometry; clear error if it does not - resolve to an existing column. -- **srid — required.** Must resolve via `sridCol` or the `_srid` default; - clear error otherwise, naming `sridCol`. (Matches today's behavior — the - current `_writer_col_roles` already requires a `*_srid` column — and the - `REQUIRED` annotation in the request. CRS-less output is therefore not a - supported mode; use `"0"` for an unknown CRS.) -- **proj — optional.** Absent is fine; it is only consulted when srid is `"0"`. - -So a frame with arbitrary names works with explicit options -(`geomCol="the_geom", sridCol="epsg", projCol="proj4"`), and a conventional -frame still needs zero options. - -## Column semantics (unchanged from today) - -- **geom encoding** is inferred from the resolved column's Spark type: - `BinaryType` → WKB; `StringType` → WKT (WKT is converted to WKB internally - before encoding). -- **CRS** comes from srid + proj via `_srid_to_crs`: `"EPSG:"` when srid is - not `"0"`, else the proj4 string, else CRS-less. -- **srid / proj are CRS metadata** — they are dropped before the OGR write and - are never emitted as attribute fields. Every other (non-geom) column is an - attribute. - -## Output geometry name - -The geometry is written under the **format's conventional name**, not the input -geom column name, so output files are clean regardless of the input column -name: - -| Driver | Output geometry name | -|---|---| -| `GeoJSON` / `GeoJSONSeq` | structural GeoJSON `geometry` member — no named field (N/A) | -| `ESRI Shapefile` | the shape record — no named field (N/A) | -| `GPKG` | `geom` | -| `OpenFileGDB` (FileGDB) | `SHAPE` | - -On read-back, `geojson_gbx` / the other `*_gbx` readers reconstruct the -`geom_0` (+ `_srid` / `_srid_proj`) schema as today. - -## Per-writer distinctives - -The **column-role options behave identically** across all five writers. The only -per-format differences are independent of these options and already exist: - -- **`shapefile_gbx`** — the `.dbf` format truncates attribute field names to 10 - characters (GDAL behavior); not affected by these options. -- **`file_gdb_gbx`** — writing requires the native GDAL/`osgeo` bindings - (pyogrio's bundled GDAL ships a read-only `OpenFileGDB` driver). The column - options resolve the same way; the geometry name default is `SHAPE`. -- **`geojson_gbx`** merges all partitions into one FeatureCollection on the - driver; **`geojsonl_gbx`** writes one shard per partition. Both consume the - same resolved roles. - -## Implementation sketch — light tier - -- Generalize `_writer_col_roles(schema)` → - `_writer_col_roles(schema, geom_col=None, srid_col=None, proj_col=None)`: - resolve each role per the rules above; raise clear errors when geom or srid - cannot resolve. -- Both `VectorGbxWriter.__init__` and `GeoJSONLGbxWriter.__init__` read - `geomCol` / `sridCol` / `projCol` from their (already-lowercased) options dict - and pass them into `_writer_col_roles`. -- Output geometry name: a small per-driver default map; pass it as pyogrio's - `geometry_name` (Arrow + classic paths) and as the osgeo FileGDB geometry - field name. For structural-geometry drivers the value is inert. -- The all-null typing helper `_writer_arrow_table` is unaffected (it already - keys off the resolved `geom_col`). - -## Implementation sketch — heavy tier (`geojsonl` only) - -- Generalize `GeoJSONL_DataSource.resolveRoles(schema)` → - `resolveRoles(schema, geomCol=None, sridCol=None, projCol=None)` with the same - resolution rules (option → else convention default if present; geom & srid - required; proj optional). `resolveRoles` is the shared role-derivation and is - called from two places — `GeoJSONL_Table.newWriteBuilder` and the - `GeoJSONL_RowWriter` constructor — so both must thread the options through (the - `WriteBuilder`/`BatchWrite`/`DataWriterFactory` chain already carries the - options map to the `RowWriter`). -- `GeoJSONL_RowWriter` already lowercases options (`ciOptions`); read - `geomcol` / `sridcol` / `projcol` alongside the existing `maxrecordsperfile` / - `geometrytype` / `layername`. -- Geometry encoding (WKB/WKT auto-detect from the resolved column's Spark type) - and the SRID/PROJ4 → `SpatialReference` mapping are unchanged. Output geometry - is structural for GeoJSONSeq, so no output-name change applies (heavy - `geojsonl` keeps `layerName`-or-geom-col for the internal layer name). - -## Cross-tier parity - -Identical option names (`geomCol` / `sridCol` / `projCol`), identical resolution -rules and required-ness, and identical CRS handling. A `geojsonl` write with the -same options + frame produces the same output on either tier (the existing heavy -↔ light geojsonl round-trip continues to hold). - -## Testing - -**Light:** -- Resolution: explicit options override; defaults used when omitted; arbitrary - names (`geomCol="the_geom", sridCol="epsg"`) round-trip. -- geom required → error when it cannot resolve; srid required → error when it - cannot resolve (no option, no `_srid`). -- proj optional → CRS from srid alone; proj4 fallback exercised when srid is - `"0"`. -- Output geometry name per format (e.g. a GPKG write produces a `geom` geometry - column); geojsonl round-trips via `geojson_gbx`. -- Uniform behavior across the writers (the shared `_writer_col_roles` covers all - five). - -**Heavy** (Scala, `GeoJSONLWriterTest`): -- `resolveRoles` unit cases: explicit overrides; convention defaults when - omitted; geom/srid required errors. -- A `geojsonl` write driven by `.option("geomCol", ...)` etc. on a frame with - non-convention column names, round-tripped via the `geojson_ogr` reader. - -## Delivery - -Light commits + the heavy (Scala) change land on `beta/0.4.0` (flows into the -open PR #46); pushed on the user's go. Heavy work builds + tests in the -`geobrix-dev` Docker container (Maven); a JAR rebuild is needed for cluster use. diff --git a/docs/superpowers/specs/2026-06-26-heavy-volumes-coherent-access-design.md b/docs/superpowers/specs/2026-06-26-heavy-volumes-coherent-access-design.md deleted file mode 100644 index 143b30f0c..000000000 --- a/docs/superpowers/specs/2026-06-26-heavy-volumes-coherent-access-design.md +++ /dev/null @@ -1,103 +0,0 @@ -# Coherent UC Volume (`/Volumes`) access across the heavyweight tier — Design - -**Date:** 2026-06-26 -**Branch:** `beta/0.4.0` -**Status:** approved (design agreed in session; user approved proceed + revive `rst_fromfile`). - -## Problem - -UC Volumes (`/Volumes/...`) + Unity Catalog are a primary customer selling point, but the -0.4.0 heavyweight tier has **inconsistent** `/Volumes` support, and the documented rationale -for one decision (`rst_fromfile` removed from heavy) is **factually wrong**. We want one -coherent story: every heavy reader and writer reads/writes `/Volumes`, and our docs state the -*accurate* constraint. - -## Root-cause model (empirically established on cluster 0519) - -The credential issue is **driver-side raw Hadoop FS metadata** on `/Volumes`. The UC FUSE mount -is credential-gated; the Spark **analyzer thread** (DSV2 `inferSchema`) AND the **read-planning** -driver thread (`Batch.planInputPartitions`) lack the credential, so raw Hadoop FS -`getFileStatus`/`listStatus`/`listFiles` on `/Volumes` throws `FileNotFoundException` (it -delegates to `RawLocalFileSystem`'s POSIX stat wrapped in `WSFSCredentialForwardingHelper`, -which has no token on those threads). - -**What reliably works on `/Volumes` (proven by probes):** -- **Spark FileIndex listing** — `spark.read.format("binaryFile").load(dir).inputFiles` — on the driver (forwards the credential). `binaryFile` *content* reads do NOT (use POSIX instead). -- **POSIX `java.io.File` / `Files.readAllBytes`** — on the driver-REPL thread AND executors. -- **Executor reads** — `NodeFileManager.readRemote` (Hadoop `file:` FS) and POSIX both work in Spark *tasks* (the executor task carries the credential): proven by `geojson_ogr` reading a 782,054-row `/Volumes` dataset and an executor reading a 6.9 MB raster. -- **Scheme:** `file:/Volumes/...` is correct (bare `/Volumes` and `dbfs:/Volumes` both fail `INVALID_DBFS_MOUNT`). - -The credential-aware **toolkit** (already added in commit `ce61e84`): -- `HadoopUtils.listDataFilesSpark(spark, path)` — FileIndex listing (`.gdb`/`.zip` returned as-is). -- `HadoopUtils.stageHeadForSchemaSpark(spark, head, candidates)` — POSIX read (+ executor fallback) of the schema file/sidecars to a local temp. - -## Per-component status + remediation - -| Component | Status | Action | -|---|---|---| -| OGR readers `inferSchema`/planning (geojson/shapefile/gpkg/ogr) | fixed (ce61e84) | none | -| OGR/GDAL executor reads (`readRemote`→local) | works | none | -| `geojsonl` writer on `/Volumes` | works (proven) | none | -| **`GDAL_Batch.planInputPartitions`** (`gdal`/`gtiff_gdal` readers) | broken — `listAllHadoopFiles` (raw Hadoop FS) | **Task 1** | -| **OGR `.gdb`/`.gdb.zip`/`.zip` `inferSchema`** (`file_gdb_ogr`) | broken — `NodeFileManager.readRemote` on driver | **Task 2** | -| **`rst_fromfile`** removed from heavy | rationale false | **Task 3** (revive) | -| `pmtiles` / `gdal` writer driver-side commit on `/Volumes` | untested | **Task 4** (probe/harden) | -| docs/comments stating the wrong `rst_fromfile` reason | wrong | **Task 5** | - -### Task 1 — `GDAL_Batch` listing -Replace `HadoopUtils.listAllHadoopFiles(inPath, hConf, regex)` (raw Hadoop FS recursive list) in -`GDAL_Batch.planInputPartitions` with a credential-aware FileIndex listing. Add -`HadoopUtils.listDataFilesSparkRecursive(spark, path, regexFilter)` (FileIndex via `binaryFile` -with `recursiveFileLookup=true`, applying the same regex/empty-file filter `listAllHadoopFiles` -applied), and use it. Keep `listAllHadoopFiles` for non-`/Volumes` callers if simpler, but route -the GDAL reader through the credential-aware path. Validate `gtiff_gdal`/`gdal` read a `/Volumes` -raster dir + single file. - -### Task 2 — OGR `.gdb`/`.zip` schema inference -In `OGR_DataSource.inferSchema`, the `isGdbLike` branch currently calls -`NodeFileManager.readRemote(headPath)` (driver raw Hadoop FS). Replace with credential-aware -staging: for a `.gdb` **directory**, stage its whole tree to a local temp via POSIX/executor -(extend `stageHeadForSchemaSpark` to handle a directory dataset, or add -`stageDatasetDirForSchemaSpark`), then OGR-open the local copy. Validate `file_gdb_ogr` reads a -`/Volumes` `.gdb` (and `.gdb.zip`). - -### Task 3 — Revive `rst_fromfile` in the heavyweight tier -Add the heavyweight Scala expression `RST_FromFile` (in `rasterx/expressions/constructor/`), -mirroring the light pyrx semantics + the existing `RST_FromContent` tile-construction. It reads -per-row **on executors** via the proven path: `NodeFileManager.readRemote(path)` (or -`RasterDriver.read`, which stages `/Volumes`→`/tmp` then `gdal.Open`s the local copy) → build the -tile struct. Register it in `rasterx/functions.scala` (remove the "NOT registered" block). -**Binding parity (enforced):** add the Scala `override def name` literal, the -`registered_functions.txt` entry, the `function-info.json` example, and confirm the Python -binding (it already exists as the pyrx UDF name — ensure the heavy registration coexists so SQL -`gbx_rst_fromfile` resolves to the JVM expression on heavy clusters). Add a Scala expression test -(read a LOCAL test raster by path → identical decoded pixels as `rst_fromcontent`). Validate on -0519: `gbx_rst_fromfile` on a `/Volumes` raster path returns a tile. - -### Task 4 — Writer driver-side commit on `/Volumes` -Probe on 0519: do `pmtiles` and `gdal`/`gtiff_gdal` writers write to a `/Volumes` target -end-to-end (esp. `PMTiles_BatchWrite.commit`, which reads scratch files + writes the final -`.pmtiles` via driver `file:` FS)? If a driver-side raw Hadoop FS metadata/read on `/Volumes` -fails, harden it the same way (FileIndex listing / POSIX). `geojsonl` already works, so the -write-commit driver context may be fine — confirm, don't assume. - -### Task 5 — Docs + comments correction -Correct every place stating "the executor JVM cannot read `/Volumes`" to the accurate model: -*driver-side raw Hadoop FS metadata on `/Volumes` lacks the FUSE credential; use Spark FileIndex -listing + POSIX/executor reads.* Files: `rasterx/functions.scala`, `util/HadoopUtils.scala` (the -`withFileSystem` NOTE), `docs/docs/api/raster-functions.mdx`, `docs/docs/beta-release-notes.mdx`, -the issue-34 references, and the test comments (`udfs.scala`, `BenchDispatch.scala`, -`ConstructorExpressionsTest.scala`). State that `rst_fromfile` IS available in both tiers. - -## Validation -Each task validated on cluster 0519 (warm) via a notebook job reading/writing the relevant -`/Volumes` corpus (rasters under `…/data/out/netcdf-gtiff`, `.gdb.zip` under `…/data/gdb`, -geojsonl under `…/data/out/geojsonl`). JAR built+staged via `gbx:data:push-jar` to the -init-script volume; cold-restart 0519 to load. - -## Out of scope -- Light tier (pyrx) — already FUSE-native; unchanged. -- Non-`/Volumes` paths (DBFS/Workspace/local) — unchanged behavior. - -## Delivery -Commits to `beta/0.4.0` (flows into PR #46). Validated on 0519 before the batch is pushed. diff --git a/docs/superpowers/specs/2026-06-26-vizx-plot-interactive-design.md b/docs/superpowers/specs/2026-06-26-vizx-plot-interactive-design.md deleted file mode 100644 index bd293499b..000000000 --- a/docs/superpowers/specs/2026-06-26-vizx-plot-interactive-design.md +++ /dev/null @@ -1,125 +0,0 @@ -# vizx.plot_interactive — design - -**Goal:** Add `plot_interactive`, the interactive (folium) twin of `plot_static`, to `gbx.vizx`. -It must be **scale-safe** (folium hangs on millions of vertices) and **Databricks-safe** (folium -does not auto-render; it needs `displayHTML`). Promotes the validated helper from the XPlore -customer notebook into the library. - -## Why -Today vizx has only static rendering (`plot_static`) and gdf adapters; docs tell users to call -raw geopandas `.explore()`, which (a) **hangs at scale** (3.4M-vertex coverage footprints → -20-min blank) and (b) **does not render in Databricks** (must go through `displayHTML`). Every -notebook hits this. - -## API -```python -plot_interactive( - data, *, column=None, mode="auto", - grid_system=None, geom_col=None, - max_vertices=60_000, # auto's detailed->fast crossover + "detailed may be slow" trigger - max_px=1400, # fast/overlay raster resolution - opacity=0.65, - debug_level=1, # 0 silent · 1 key decisions+warnings · 2+ verbose internals - **explore_kw, -) -``` - -`data` is a GeoDataFrame **or** a Spark DataFrame. For a Spark DataFrame, convert to a gdf the -same way `plot_static` does: `grid_system` set → `cells_as_gdf`; else geometry column via -`geom_col`/auto-detect → `as_gdf`. (Reuse `_static_map`'s detection where practical.) - -## Modes (intent-oriented) -- **`auto`** (default): use **detailed** if total vertex count `<= max_vertices`, else **fast**. - Announce the chosen path + reason at `debug_level >= 1`. -- **`detailed`**: geopandas `.explore()` — full vector, hover tooltips/popups. If vertex count - `> max_vertices`, emit (at `debug_level >= 1`) `"detailed mode: {n:,} vertices > {max_vertices:,} - — may be slow to render. (set debug_level=0 to silence)"` and **proceed** (honor the choice). -- **`fast`**: raster image overlay — rasterize polygons to a PNG (`max_px` resolution) and lay it - on a folium `ImageOverlay`. Complete (every polygon burned, nothing dropped), scales to millions - of vertices, but a flat image (no per-feature hover). The proven `_raster_overlay` below. - -Invalid `mode` → `ValueError` listing the valid modes. - -## Tunable thresholds (conservative Serverless defaults; raise to tune) -- `max_vertices` (default 60_000) — auto crossover + detailed-slow trigger. -- `max_px` (default 1400) — overlay raster resolution (higher = sharper, slower/bigger PNG). - -## debug_level (default 1) -- **0** — silent. -- **1** — key decisions only: auto's chosen path + why; the detailed-over-threshold warning. Every - level-1 message ends with `" (set debug_level=0 to silence)"`. -- **2+** — verbose internals: vertex counts always, raster px used, render timing. - -Example level-1 lines: -- `auto -> fast (image overlay): 3,377,195 vertices > max_vertices=60,000; per-feature hover unavailable at this scale. (set debug_level=0 to silence)` -- `detailed mode: 3,377,195 vertices > 60,000 — may be slow to render. (set debug_level=0 to silence)` - -## Databricks vs Jupyter rendering -Build the folium map `m`, then render it as the function's **last statement**: -```python -try: - html = m._repr_html_() -except Exception: - html = m.get_root().render() -try: - displayHTML(html) # Databricks: render via side effect; function returns None -except NameError: - return m # plain Jupyter: return the map so it auto-renders -``` - -## Proven `_raster_overlay` (basis — adapt into the module) -rasterize polygons (numeric column → value; categorical → integer codes; none → 1..n) to a -`max_px`-wide grid in EPSG:4326, viridis-colormap to an RGBA PNG (NoData transparent), folium -`ImageOverlay` over the bounds, `fit_bounds`. (Verbatim logic validated on Serverless: 3.4M-vertex -rings → ~29s, 10 KB HTML, renders.) - -## Tests (TDD, `python/geobrix/test/vizx/test_interactive.py`) -Mock `displayHTML` (inject into builtins / module globals) and assert it is called in Databricks -mode and not in Jupyter mode (NameError path returns the map). Use a small GeoDataFrame fixture. -- `mode="fast"` → `_raster_overlay` path; output is a folium Map with an ImageOverlay; complete. -- `mode="detailed"` small gdf → `.explore()` path (can mock `gdf.explore`). -- `mode="auto"` picks detailed under threshold, fast over it (drive with `max_vertices`). -- `detailed` over threshold emits the warning at `debug_level=1`, silent at `0` (capture stdout). -- `debug_level=0/1/2` output gating. -- invalid `mode` → ValueError. -- Spark DataFrame input path (can be a light unit with a tiny spark df, or mock the adapter). -- categorical vs numeric `column` in `_raster_overlay`. - -## Wiring & follow-ups -- Export `plot_interactive` from `vizx/__init__.py` and add to `__all__`. -- (Follow-up, separate change) update docs/example that recommend raw `.explore()` → - `vizx.plot_interactive`; retrofit the XPlore notebook to import it instead of the inline helper. - ---- - -## Addendum — representative sampling (`sample_seed`) + fast-path truncation warning - -**Motivation:** the only place rows are dropped is the Spark→gdf collection cap (`max_rows`), -currently `.limit(N)` = **first N**, which is partition-order arbitrary (spatially biased) and -unstable across runs. Add an opt-in **reproducible sample**. - -**`sample_seed` (new param, default `None`):** -- `None` → first `max_rows` via `.limit` (current behaviour; deterministic, cheapest; **non-breaking**). -- `` → reproducible sample via **`pyspark.sql.DataFrame.sample`** (NOT the pandas-on-Spark - API): compute `frac = min(1.0, (max_rows * 1.3) / df.count())`, then - `df.sample(withReplacement=False, fraction=frac, seed=sample_seed).limit(max_rows)`. SQL-native, - reproducible by `seed`. Note `.sample` is Bernoulli/**approximate-N**, so the 1.3× headroom + the - trailing `.limit(max_rows)` yield up to `max_rows` rows; same seed → same sample. Costs one extra - `count()` job (only on the opt-in sampling path). - -**Where it lives:** add `sample_seed=None` to the collection adapters **`as_gdf`** and -**`cells_as_gdf`** (they own the `max_rows` cap), and thread it through `plot_static` and -`plot_interactive`. One implementation, both plotters benefit. Default `None` keeps all existing -behaviour/tests intact. - -**Fast-path truncation warning:** in `plot_interactive`, when the collected gdf was actually -truncated (collected row count `== max_rows` and the source had more), emit at `debug_level >= 1`: -`"fast: showing {max_rows:,} of {n:,} geometries — pre-aggregate (st_union_agg / rst_h3_rasterize_agg) for complete coverage. (set debug_level=0 to silence)"`. -Rationale: a *sampled* "complete-coverage" raster is a contradiction; for true completeness the -caller pre-aggregates so the gdf is few-rows-many-vertices and the cap never fires (the validated -XPlore path: 8 dissolved footprints). - -**Added tests:** `sample_seed=None` → first-N (unchanged); `sample_seed=int` → reproducible -(same seed twice → identical rows; different seed → different); adapter-level sampling in -`as_gdf`/`cells_as_gdf`; the fast-path truncation warning fires only when truncated and is gated by -`debug_level`. diff --git a/docs/superpowers/specs/2026-06-26-writer-filename-naming-design.md b/docs/superpowers/specs/2026-06-26-writer-filename-naming-design.md deleted file mode 100644 index b5989b38c..000000000 --- a/docs/superpowers/specs/2026-06-26-writer-filename-naming-design.md +++ /dev/null @@ -1,95 +0,0 @@ -# `fileName` Option + Adaptive Output Naming for Single-File Writers — Design - -**Date:** 2026-06-26 -**Status:** Approved in brainstorm → spec for review -**Tiers:** light (Python/PySpark `*_gbx`) and heavy (Scala/JVM DataSource V2), with **one shared contract**. - ---- - -## 1. Problem - -GeoBrix writers that emit a **single file or single-unit archive** (a `.gpkg`, a `.geojson`, a zipped shapefile `.shp.zip`, a FileGDB `.gdb`/`.gdb.zip`, and later a `.pmtiles`) have no consistent, intuitive output-naming behavior: - -- A user calling `.save("/out/shapefile-heavy")` (a stem) expects the writer to produce `/out/shapefile-heavy.shp.zip` — it does not auto-complete the extension. -- A user pointing `.save()` at an **existing directory** expects a sensible default name, not an error or a confusing failure. -- There is no way to **name** the output unit explicitly. -- These behaviors differ (or are absent) across writers and across tiers. - -Most Spark writers emit a *directory of shards*; single-file/single-unit writers are the exception and need a more adaptive naming model. (The original trigger: a user ran `df.write.format("shapefile_ogr")…` — a **read-only** heavy format — and got a cryptic `stageHeadForSchemaSpark` failure instead of a clear "use the light writer" message. See §6.) - -## 2. Goal - -A single, tier-agnostic **output-naming contract** for single-file/single-unit writers: - -1. A `.option("fileName", "")` that, when present, names the output unit (extension auto-completed) and creates parent directories as needed. -2. Adaptive defaults when `fileName` is absent: complete the extension on a stem path, or derive a name when given an existing directory. -3. **Identical semantics in both tiers** wherever such a writer exists. - -## 3. The contract - -Inputs: the `.save(path)` argument, an optional `.option("fileName", name)`, and the writer's **canonical extension** `EXT` (per §4). - -**Resolution rules** (evaluated in order): - -1. **`fileName` provided** → treat `path` as the **parent directory**. Create it (and parents) if missing. Output = `path / complete(fileName)`. -2. **`fileName` absent AND `path` is an existing directory** → output = `path / complete(basename(path))` — a unit named after the directory, written **under** it. -3. **`fileName` absent AND `path` does not exist / is file-like** → output = `complete(path)`; create `path`'s parent directory if missing. (`path`'s last segment is the target name.) - -**`complete(name)`** (extension completion): -- If `name` already ends with `EXT` (case-insensitive) → use as-is. -- Else append the missing part(s). For multi-part extensions this is incremental: `roads` → `roads.shp.zip`; `roads.shp` → `roads.shp.zip`; `roads.shp.zip` → unchanged. - -**Validation:** if `name` ends with a **different recognized geo extension** (e.g. `.gpkg` passed to the shapefile writer, or `.geojson` to the gpkg writer), raise a clear error naming the expected `EXT` — rather than silently appending and producing `roads.gpkg.shp.zip`. - -## 4. Canonical extensions (`EXT`) per writer - -| Writer (light `_gbx` / heavy) | `EXT` | Notes | -|---|---|---| -| `gpkg_gbx` | `.gpkg` | single file | -| `geojson_gbx` | `.geojson` | single file | -| `shapefile_gbx` (`zip=true`) | `.shp.zip` | single archive; **non-zip shapefile stays a directory bundle — not in scope** | -| `file_gdb_gbx` | `.gdb` (or `.gdb.zip` when `zip=true`) | `.gdb` is a directory treated as a named unit | -| PMTiles (light + heavy) | `.pmtiles` | single archive — **same contract, applied when PMTiles writers adopt it** | - -Sharded/directory writers (`geojsonl_gbx`, `geojsonl_ogr`, the raster `gdal`/`gtiff_gdal` tile-dir writers) are **out of scope** — `fileName` is a single-unit concept; per-tile naming there is the existing `nameCol` option. - -## 5. Applicability across tiers (the "consistency" answer) - -- **Light single-file vector writers** (`gpkg_gbx`, `geojson_gbx`, `shapefile_gbx`+zip, `file_gdb_gbx`): implement the contract **now**. -- **PMTiles** (exists in both tiers, single archive): the **same contract** governs `fileName`/naming; applied when the PMTiles writers take this treatment (tracked with the PMTiles writer work, not implemented here, but the contract is fixed now so both tiers match). -- **Heavy single-file *vector* writers do not exist** (heavy OGR is read-only). For those formats, "both tiers" is satisfied by: the **light** writer does the naming, and the **heavy** read-only format returns a clear error directing to the light writer (§6) — not by adding heavy vector writers (explicitly out of scope, §8). - -## 6. Heavy read-only-format clear error (folded in) - -`df.write.format("shapefile_ogr"|"gpkg_ogr"|"file_gdb_ogr"|"geojson_ogr"|"ogr")` currently fails confusingly: `OGR_DataSource.supportsExternalMetadata=true` makes Spark call `inferSchema` on the write path, which reads the nonexistent target → cryptic `stageHeadForSchemaSpark` `NoSuchFileException`/"Is a directory". - -**Fix:** these read-only formats should reject a write attempt with a clear, actionable message, e.g. *"`shapefile_ogr` is a read-only reader; write with the light `shapefile_gbx` writer (or `geojsonl_ogr` for sharded GeoJSONL)."* Reads are unaffected. (Exact mechanism — overriding the write path / capability surface so Spark raises before `inferSchema` — to be finalized in the plan.) - -## 7. Implementation - -- **Light:** one shared helper `_resolve_single_file_output(path: str, file_name: str | None, ext: str) -> str` in `python/geobrix/src/databricks/labs/gbx/ds/vector.py`, applying §3 exactly. Each of the four single-file writers calls it to compute its output target; parent-dir creation centralized there. Pure-function core (path-string logic) is unit-testable without Spark; the FS touch (exists-dir check, mkdirs) is the only IO. -- **Heavy:** a Scala mirror of the same 3-case logic for any heavy single-file writer (PMTiles), kept behaviorally identical to the Python helper — same rule order, same `complete()` semantics — so cross-tier output names match for the same inputs. -- **DRY + reuse:** the helper is the single source of naming truth; designed for reuse by PMTiles (both tiers) so the contract is defined once. - -## 8. Out of scope - -- **Adding heavy single-file vector writers** (`shapefile_ogr`/`gpkg_ogr`/`file_gdb_ogr`/`geojson_ogr` write). Heavy vector stays read-only; the clear error (§6) is the UX there. -- **Single-`.shp`-path *reader* sidecar staging** (the `.shx`-not-found error) — a related but separate **reader** fix, tracked independently (light + heavy shapefile readers). -- **Implementing PMTiles `fileName`** — the contract is fixed here; the application lands with the PMTiles writer work. -- Sharded/directory writers' naming. - -## 9. Testing - -- **Unit (light):** `_resolve_single_file_output` across the full matrix — 3 path cases × `fileName` present/absent × each `EXT` (incl. multi-part `.shp.zip`, `.gdb.zip`); the wrong-extension validation error; parent-dir creation. -- **Round-trip (light):** for each of the four writers, write with (a) a stem path, (b) an existing dir, (c) an explicit `fileName`, on a UC Volume; assert the resolved output path matches the contract and the result reads back. -- **Heavy clear-error:** asserting each read-only OGR format raises the actionable message on `.save()` (reads still succeed). -- **Cross-tier (when PMTiles adopts):** identical inputs → identical resolved names in light vs heavy. - -## 10. Success criteria - -- `.save("/out/roads")` on any single-file `_gbx` writer produces `/out/roads.`. -- `.save("/out/existing_dir")` produces `/out/existing_dir/.`. -- `.option("fileName","roads")` produces `/roads.`, creating dirs as needed. -- Wrong-extension `fileName` fails with a clear message. -- Writing a read-only heavy OGR format fails with a clear "use the `_gbx` writer" message. -- The naming contract is defined once and reused (light now; heavy/PMTiles to the same contract). diff --git a/docs/superpowers/specs/2026-06-27-helios-tiling-series-overture-design.md b/docs/superpowers/specs/2026-06-27-helios-tiling-series-overture-design.md deleted file mode 100644 index 3e980974f..000000000 --- a/docs/superpowers/specs/2026-06-27-helios-tiling-series-overture-design.md +++ /dev/null @@ -1,334 +0,0 @@ -# Design: PMTiles Multi-Tiling Series + Overture Data Source ("Project Helios") - -**Date:** 2026-06-27 -**Branch target:** `beta/0.4.0` (new feature branch off it; PR into `beta/0.4.0`) -**Status:** Approved design — ready for implementation planning. - -## Summary - -A "twofer" deliverable for GeoBrix 0.4.0, decomposed into **three sequenced sub-projects**: - -1. **Overture data source** (`gbx.sample.overture`) — an API-level, distributed, AOI-driven - downloader for Overture Maps GeoParquet (all themes/types), via Overture's STAC catalog, - into a Unity Catalog Volume — with an optional metadata Delta table that catalogs each - asset's Volumes path (`source`/`path` column) for queryable, re-runnable, reader-ready output. -2. **VizX viewers** — net-new public `gbx.vizx` functions `plot_pmtiles` (and `plot_cog`) that - render a PMTiles archive / COG inline in a Databricks notebook, plus a small reusable - Python PMTiles inspector in `gbx.pmtiles`. -3. **The notebook series** (`notebooks/examples/helios/`) — a `config_nb` spine + three focused - notebooks demonstrating vector (MVT), raster (XYZ), and elevation (COG + STAC) tiling, all - written as PMTiles, over a **San Francisco** AOI with a solar site-selection meta-narrative. - -The notebooks consume sub-projects 1 and 2 plus already-shipped tiling primitives, so the build -order is **SP1 → SP2 → SP3**. - -## Goals - -- Show GeoBrix's distributed tiling story end-to-end: ingest → tile (MVT / XYZ / COG) → - package as PMTiles → inspect/visualize, all on Databricks (Serverless-safe lightweight tier - by default, heavyweight switchable). -- Give Overture Maps a first-class, data-source-specific API (all themes), since it is the - most popular open vector source. Other sources (NAIP, USGS 3DEP) stay as notebook helpers. -- Provide an in-notebook PMTiles viewer (and COG viewer) in VizX so the series can *show* the - output, not just write it. -- Keep the series DRY via a `config_nb`, promoting only genuinely reusable helpers into the - light API. - -## Non-goals - -- No Spark-side PMTiles *read* (still unsupported; the Python inspector reads on the driver). -- No new heavyweight Scala expressions are required; the series uses existing functions - (`gbx_st_asmvt`, `st_asmvt_pyramid`, `gbx_rst_to_webmercator`, `gbx_rst_xyzpyramid`, - `gbx_rst_cog_convert`, terrain, `gbx_pmtiles_agg`, `.write.format("pmtiles")`). -- NAIP / USGS 3DEP do **not** get module-level APIs — notebook/`config_nb` helpers only. -- No tile server / external hosting; the viewer renders entirely in-notebook. - ---- - -## Sub-project 1 — Overture data source (`gbx.sample.overture`) - -A data-source-specific extension of the existing `sample` package (sibling to `_bundle.py`), -shipped in the WHL. Mirrors `gbx.stac.StacClient`'s shape, distribution strategy, and -test-injection seams. - -### Public surface - -```python -class OvertureClient: - def __init__(self, catalog="https://stac.overturemaps.org/catalog.json", - release=None, _catalog_opener=None, _get_fn=None): ... - - def discover(self, bbox, themes=None, release=None) -> "DataFrame": - """One row per intersecting GeoParquet asset for the AOI. - Columns: theme, type, href, asset_bbox, release. - themes=None => ALL themes/types.""" - - def download(self, assets_df, out_dir, *, bbox=None, table=None, - validate=True, max_tries=5, partitions=None) -> "DataFrame": - """Distributed download of discovered assets to out_dir (a Volume). - Serverless-safe repartition(N, col); idempotent skip; parquet-readable - validation. `bbox` (the AOI) drives the distributed-read predicate - pushdown on the performant default path; `download_overture_aoi` passes - it through. Each asset is written to a unique per-asset subdirectory - under out_dir/// (a stable href-derived token) so sharded - types do not clobber one another and the (theme, type, source) MERGE key - stays 1:1 with assets. - - Returns a metadata DataFrame: theme, type, source (the Volumes path of - the downloaded asset; also aliased as `path`), out_file_sz, - is_out_file_valid, last_update, plus carried discovery columns - (asset_bbox, release, href). - - table= also persists/UPSERTs that metadata to a Delta - table (idempotent MERGE keyed by (theme, type, source)), so the catalog - of downloaded assets is queryable and re-runnable.""" - - def read(self, source, theme=None, type=None, bbox=None) -> "DataFrame": - """Load downloaded GeoParquet back into Spark, optional bbox-struct AOI - filter (the overture.py loader pattern). `source` may be a Volume - directory OR a metadata Delta table / DataFrame whose `source`/`path` - column points at the per-asset Volumes paths.""" - -# convenience one-shot -def download_overture_aoi(bbox, out_dir, themes=None, release=None, - table=None) -> "DataFrame": ... -``` - -### Behavior / design notes - -- **Discovery:** Overture's STAC is a **static catalog** (`catalog.json` traversal → collections - → items), *not* a searchable STAC API. So discovery cannot reuse `StacClient` directly; it - traverses the catalog and filters items client-side by axis-aligned bbox intersection. - An `overturemaps` CLI fast-path is used when the CLI is installed; otherwise the - pystac/static-traversal fallback runs. (Both paths are present in the reference - `stac_download.py`; this generalizes them to all themes and to distributed download.) -- **Themes/types covered (all):** `addresses`, `base/*` (infrastructure, land, land_cover, - land_use, water, bathymetry), `buildings/{building,building_part}`, - `divisions/{division,division_area,division_boundary}`, `places/place`, - `transportation/{connector,segment}`. `themes=None` selects everything; a list narrows it. -- **Release handling:** `release=None` resolves the latest Overture release from the catalog; - an explicit string pins it. -- **Distribution (Serverless is the first-class target, not Classic):** the distribution - strategy is designed for Serverless and must drive aggressive parallelism, modeled on the - h3-rasterize example (fan the work into many fine-grained, balanced units rather than tuning - cluster knobs). Concretely: - - **Performant default — distributed read + AOI rewrite.** STAC resolves the release and the - set of Overture GeoParquet paths for the bbox; the heavy I/O is then a *distributed Spark - read* of those parquet files over the cloud path (`s3://overturemaps-us-west-2/...` / - `abfs://...`) with **`bbox`-struct predicate pushdown** so only AOI rows are read, written - distributed to the Volume (+ the metadata Delta table). The bytes move on workers, in - parallel — the AOI subset, not whole continental files. - - **Fallback — asset-level parallel download.** When only an `https` href is available (no - direct cloud read), fan whole-file downloads out with `repartition(N, F.col())`. - - **Avoid the file-count bottleneck.** A bbox may intersect only a few very large parquet - files; to keep parallelism high (one balanced unit per core, h3-rasterize style), fan finer - than file granularity — by parquet **row-group / byte-range** — rather than letting a - handful of files cap task count. - - **Serverless constraints (hard):** only `repartition(N, column)` for parallelism — never - number-only `repartition(N)` (AQE-coalesced to serial); **no** `spark.conf` / cache / - persist / checkpoint / `.rdd` / `sparkContext`. Verify partitions are not coalesced - (`getNumPartitions`) when iterating the plan locally. `CREATE TEMP TABLE` materialization - (used to pin a distributed result) is Serverless / DBR 18.1+ only. - - **No driver bottleneck:** catalog traversal/discovery is driver-side but lightweight - (metadata only); all asset I/O is distributed. -- **Validation:** `validate=True` means the downloaded parquet opens (pyarrow/geopandas), - not rasterio-decodable. Idempotent skip when the target exists and is valid. -- **Output targets (both supported):** (1) asset **files on a UC Volume** under `out_dir`; - (2) an optional **metadata Delta table** (`table=...`) with one row per asset and a `source` - column (aliased `path`) holding that asset's Volumes path. The table is UPSERTed via Delta - `MERGE` keyed by `(theme, type, source)` so re-runs are idempotent and a `repair()`-style - re-download of invalid rows works (the StacClient/eo-series pattern). The `source`/`path` - naming matches what downstream GeoBrix readers and `read()` consume, so the metadata table - can directly drive distributed reads. -- **Testability:** `_catalog_opener` and `_get_fn` injection seams (exactly like `StacClient`) - so unit tests run offline on the driver with a fake catalog and fake fetcher — no network. - -### Files - -- `python/geobrix/src/databricks/labs/gbx/sample/overture.py` (public `OvertureClient` + - `download_overture_aoi`). -- `python/geobrix/src/databricks/labs/gbx/sample/_overture_discover.py` (catalog traversal / - CLI fast-path / bbox intersect — kept separate so it is unit-testable in isolation). -- Re-export from `sample/__init__.py`. -- Tests: `python/geobrix/test/sample/test_overture.py` (offline, injected opener + fetcher). - -### Dependencies / CI lock - -- New runtime dep: `pystac` (catalog traversal). Parquet read via existing `geopandas`/`pyarrow`. - `overturemaps` CLI is optional (fast-path only). -- Follow the light-CI-lock checklist: add deps to `requirements-pyrx-ci.in` **and** - `requirements-dev-container.in`, then recompile the hashed `.txt` files; register the new - `test/sample/` directory in **both** the light-test dir list and the `pyrx_build` dir list. - ---- - -## Sub-project 2 — VizX viewers (`gbx.vizx`) - -Net-new public functions exported from `vizx/__init__.py`, behind the existing `[vizx]` extra. - -### Public surface - -```python -def plot_pmtiles(path_or_bytes, *, max_embed_mb=64, fallback=True, - style=None, **map_kwargs): - """Render a .pmtiles archive inline. Interactive MapLibre GL JS + pmtiles.js - (archive base64-embedded as an in-browser FileSource) via displayHTML. - Auto-detects vector (MVT -> vector layer) vs raster (PNG/JPG/WebP -> raster - layer) from the archive header. Above max_embed_mb (or fallback path), - renders a Python-side static image instead.""" - -def plot_cog(path, *, band=None, **kw): - """Render a Cloud-Optimized GeoTIFF: rasterio overview read -> plot_raster; - optionally also added as a raster source in the interactive map.""" -``` - -### Design notes - -- **Interactive path:** build a MapLibre GL JS HTML page (libraries from CDN: maplibre-gl + - pmtiles), register the `pmtiles://` protocol, and feed the archive bytes as a base64 - `FileSource` (`new pmtiles.PMTiles(new pmtiles.FileSource(...))`) so it streams entirely - in-browser — no HTTP server, no range requests against remote storage. Render through the - existing `_notebook_display_html()` channel (IPython `user_ns['displayHTML']`) from - `_interactive.py`; reuse its fallback chain. -- **Size guard + static fallback:** archives larger than `max_embed_mb` (base64 bloats ~33%) - would hang the notebook; instead decode tiles with the Python `pmtiles` reader and composite - — raster PMTiles → reuse `plot_raster`; vector PMTiles → decode MVT to geometries → reuse - `plot_static`. Mirrors `plot_interactive`'s scale-safe philosophy. -- **Static rendering uses `contextily` basemaps (continued).** The vector static fallback and - `plot_cog` lay their layers over a `contextily` basemap, consistent with the existing - `plot_static` (`basemap=True`, `basemap_source=...`). `contextily` is already a `[vizx]` - dependency — no new dep — so the static path stays visually consistent with the rest of VizX. -- **Vector vs raster detection:** read the PMTiles header `tile_type` (PNG/JPEG/WebP/MVT) via - the inspector (below). - -### Light-API promotion: PMTiles inspector - -`gbx.pmtiles.pmtiles_info(path) -> dict` (header: tile_type, min/max zoom, bounds, tile count, -tilejson-ish metadata). Spark-side read is unsupported, so a driver-side inspector is broadly -useful and is needed by both the viewer and the static fallback. Implemented with the existing -`pmtiles` PyPI dependency. - -### Files - -- `python/geobrix/src/databricks/labs/gbx/vizx/_pmtiles.py` (`plot_pmtiles`). -- `python/geobrix/src/databricks/labs/gbx/vizx/_cog.py` (`plot_cog`). -- `python/geobrix/src/databricks/labs/gbx/pmtiles/_inspect.py` (`pmtiles_info`); re-export from - `pmtiles/__init__.py`. -- Export `plot_pmtiles`, `plot_cog` from `vizx/__init__.py` (+ `__all__`). -- Tests: `test/vizx/test_pmtiles.py`, `test/vizx/test_cog.py` (assert HTML structure for the - interactive path, image output for the fallback, header parsing for the inspector). - -### Dependencies / CI lock - -- No new Python deps for the interactive path (CDN JS); `pmtiles` + `rasterio` already present. -- `plot_cog` uses `rasterio` (present) over a `contextily` basemap (present); `rio-tiler` - optional for nicer overview selection. -- CI-lock: `vizx` test dir already registered; only add new deps if `plot_cog` adopts - `rio-tiler`. - ---- - -## Sub-project 3 — The notebook series (`notebooks/examples/helios/`) - -Mirrors the eo-series layout (`config_nb.ipynb` + numbered notebooks + `README.md`) plus a -docs page and sidebar entry. - -### Layout - -``` -notebooks/examples/helios/ - config_nb.ipynb # shared spine, %run by each notebook - 01. Vector Engine (MVT).ipynb - 02. Visual Basemap (XYZ).ipynb - 03. Analytical Core (COG + STAC).ipynb - README.md -docs/docs/notebooks/helios.mdx # docs page; add to docs/sidebars.js -``` - -### `config_nb.ipynb` (the spine) - -`%pip` install (light tier by default), imports, `OvertureClient` + `StacClient` setup, the -**San Francisco AOI bbox constant**, `ETL_DIR` Volume config, tier switch (light/heavy), -rebuild flags, and **series-only helpers**: solar-slope/aspect scoring, demo plot wrappers, -Delta table finalizers. Anything that proves generally reusable is promoted to the light API -(SP1/SP2) instead of living here. - -### Notebook arcs (one SF AOI, solar site-selection narrative) - -- **NB01 — Vector Engine (MVT):** Overture **buildings** for SF (`OvertureClient`) → - `gbx_st_asmvt` / `st_asmvt_pyramid` → `gbx_pmtiles_agg` → vector PMTiles in a Volume → - `plot_pmtiles`. Narrative: roof footprints as candidate solar surfaces. -- **NB02 — Visual Basemap (XYZ raster):** NAIP imagery (notebook helper download) → - `gbx_rst_to_webmercator` → `gbx_rst_xyzpyramid` → raster PMTiles → `plot_pmtiles`. - Narrative: aerial site context. -- **NB03 — Analytical Core (COG + STAC + hillshade):** USGS 3DEP DEM (notebook helper) → - `gbx_rst_cog_convert` → COGs in a Volume → STAC-catalog the COGs into a Delta table → - hillshade/slope → hillshade PMTiles → `plot_cog` + `plot_pmtiles`. Narrative: roof - slope/aspect for solar yield. - -Each notebook includes meta-narrative markdown, a catchy data→tile→PMTiles flow diagram, and -ample plotting sections. - ---- - -## Cross-cutting concerns - -### Testing (TDD) - -- SP1 and SP2 are built test-first with offline injection seams (no network in unit tests). -- Doc/example code executes real assertions on real sample data per repo convention; notebooks - are validated in Docker (`gbx:test:notebooks`). Doc tests are the documentation source. -- The viewer's interactive path is asserted by HTML structure (script tags, embedded source, - protocol registration); the fallback path by produced image; the inspector by parsed header. - -### Sequencing & plans - -- Build order: **SP1 → SP2 → SP3**. Each sub-project gets its own implementation plan from the - `writing-plans` skill (three plans under `docs/superpowers/plans/`). - -### Bench - -- The Overture path is a downloader (light-only); add a light bench only if a reader/writer - surfaces that warrants the "bench each reader/writer" convention. No heavy comparison expected. - -### Performance methodology & knowledge capture (standing practice for all tiling work) - -Every tiling operation exercised by this work is a chance to improve the underlying function. -Whenever a tiling-path improvement is found (light/Serverless is the target), it MUST be: - -1. **Propagated by assessment, not assumed.** Evaluate and record the gain's applicability to - (a) other *similar* light-tier functions, and (b) the *same and similar* heavy-tier - functions. Record the verdict even when "not applicable" (and why). -2. **Captured as a reusable pattern in two homes** so we stop rediscovering best practices: - - **Developer-facing engineering corpus** — `docs/superpowers/performance/` (version-controlled, - in-repo). One file per pattern: problem → symptom/signature → the fix → applicability - matrix (light-similar / heavy-same+similar) → evidence/bench numbers → canonical code refs. - - **Agent memory** — a thin pointer memory (slug + one-line) that `[[links]]` to the canonical - corpus file, so future sessions recall fast without bloating `MEMORY.md`. -3. **Kept distinct from the user-facing doc.** `docs/docs/api/performance.mdx` stays user-facing - (execution shapes + function classification + where the light tier wins). The corpus is the - internal "how/why we got the gain" engineering record; the two cross-reference but do not merge. - -The `docs/superpowers/performance/` corpus is built **incrementally within the Helios plans** (no -separate precursor): the `README.md` index and each pattern file are created the first time a gain -is validated, with its paired thin pointer memory added alongside. The three-layer structure is — -(1) user-facing `docs/docs/api/performance.mdx`, (2) developer corpus `docs/superpowers/performance/`, -(3) agent-memory pointers — and each SP plan carries a "capture validated gains" step. - -### Docs voice - -- All user-facing docs (README, `helios.mdx`) avoid internal planning vocabulary (no wave - numbers / dispatch references); QC `internals-leak` check enforces this. - -## Open items deferred to planning - -- Exact SF AOI bbox extent (small enough for demo-friendly data volumes; reuses the - h3-rasterize SF area where possible). -- Whether `plot_cog` also injects the COG as a raster layer in the interactive map, or stays - static-only (decide during SP2 implementation). -- CDN pin vs vendored copy for maplibre-gl / pmtiles JS (pin a specific version for - reproducibility). -- Confirm Serverless can read Overture's public cloud paths directly (`s3://overturemaps-us-west-2` - / `abfs://...overturemapswestus2...`) — including any requester-pays / credential config — since - the performant distributed-read default depends on it; otherwise the asset-level HTTP-href - download fallback becomes the primary path. Validate during SP1. diff --git a/docs/superpowers/specs/2026-06-28-pmtiles-vector-merge-design.md b/docs/superpowers/specs/2026-06-28-pmtiles-vector-merge-design.md deleted file mode 100644 index e3a9e5baa..000000000 --- a/docs/superpowers/specs/2026-06-28-pmtiles-vector-merge-design.md +++ /dev/null @@ -1,102 +0,0 @@ -# Design: `gbx_pmtiles_agg` merges multi-feature vector tiles - -**Date:** 2026-06-28 -**Branch:** `beta/0.4.0` -**Status:** Proposed — awaiting approval before planning. - -## Problem - -`gbx_pmtiles_agg` (both tiers) deduplicates tiles by `(z, x, y)` **first-write-wins**: -- light `pmtiles/_agg_light.py`: `if tileid in seen: continue` (docstring: "duplicate (z,x,y) keep the first") — first-wins, structurally valid but drops vector features. -- heavy `PMTilesV3Encoder.scala:83-105`: **no tileId dedup at all** — `PMTilesAcc.add` appends unconditionally and the encoder writes a directory entry per tuple, so duplicate `(z,x,y)` produce **two directory entries → a structurally malformed PMTiles archive** (the spec requires ≤1 entry per tileId; content-hash dedup only collapses identical bytes, not differing blobs at the same tileId). This affects **raster too**, not just vector — a distinct, worse failure mode than light's first-wins. (Found by the cross-aggregator audit, which also confirmed the drop-on-collision flaw is UNIQUE to `gbx_pmtiles_agg` — all other `_agg`s combine correctly.) - -`gbx_st_asmvt_pyramid` is a generator that emits **one single-feature MVT blob per `(feature, z, x, y)`**. So the documented composition — release notes: *"`st_asmvt_pyramid` … composes with `gbx_pmtiles_agg` for end-to-end vector publishing pipelines"* — **drops all but the first feature in each tile** for dense data (e.g. a buildings basemap: thousands of features per tile collapse to one). First-wins is correct for **raster** (one tile = one image) but wrong for **vector** (one tile = many features). The Helios NB01 buildings basemap is the first end-to-end exercise of this path and exposed it. - -## Goal - -Make `gbx_pmtiles_agg` produce correct multi-feature **vector** tiles, so the -`st_asmvt_pyramid → gbx_pmtiles_agg` pipeline works for real data, at parity across both tiers. - -## Design - -When packing a group of tiles, partition by tile id `(z, x, y)`. For each tile id with -multiple blobs: - -- **Vector (MVT) tile type:** **merge** the blobs into one MVT tile — decode each blob, union - its features into the combined tile **keyed by layer name** (a feature from layer `buildings` - joins the merged `buildings` layer), then re-encode one MVT at the **same extent**. Geometry - stays in tile-local `[0, extent]` integer space — **no reprojection** (the blobs are already - tile-local for that exact `(z,x,y)`; decode→encode round-trips the local coords). Attributes - are preserved per feature. -- **Raster (PNG/JPEG/WebP/etc.) tile type:** emit **exactly one entry per tile id** (first-wins — - images can't be meaningfully merged; one tile = one image). For the LIGHT tier this is unchanged. - For the HEAVY tier this is a **fix**: today it writes a directory entry per tuple (duplicate - `(z,x,y)` → malformed archive); grouping by tile id and emitting one blob per group makes the - heavy raster archive structurally valid. So the both-tiers "group by tile id, one output blob per - group" design fixes BOTH the vector feature-drop AND the heavy raster malformed-duplicate-entry bug. - -Tile type is auto-detected from the first non-null payload's magic bytes, as today. The -vector-vs-raster branch keys off that detected type. - -### Light tier (`pmtiles/_agg_light.py`) -Group payloads by tileid. For vector: `mapbox_vector_tile.decode` each blob → accumulate -features per layer → `mapbox_vector_tile.encode` (or the existing `_mvt.encode_layer`) once per -tileid at the standard extent. For raster: first non-null. Then write one tile per tileid in -Hilbert order (unchanged). `mapbox_vector_tile` is already a dependency — **no new dep**. - -### Heavy tier (`PMTilesAcc` / `PMTiles_Agg.scala`) -Same grouping + branch, using the JVM MVT codec already in the heavyweight tier (the -`vectorx`/`mvt` encoder behind `gbx_st_asmvt`). For vector tile ids with >1 blob, decode + union -features per layer + re-encode; raster keeps first. Preserve the existing serialize/deserialize -merge-phase and the partition size cap. - -### Parity -Both tiers must produce equivalent merged tiles. Parity test MUST use a **POLYGON** multi-feature -case (points-only gives a false pass — see the MVT tile-local contract): build two single-feature -MVT blobs for the same `(z,x,y)`, pack via each tier, decode the packed tile, assert **both** -features are present with their attributes. - -## Decisions (sensible defaults; flag if you disagree) - -1. **Per-tile size cap on merge:** merge all features; rely on the **existing partition/buffer - cap** (the 100 MiB guard). Per-tile feature **simplification/dropping** (RDP, drop-by-zoom — - the PMTiles mental-model's "pressure point") is **out of scope** here and tracked as a future - enhancement. A merged tile that's individually huge is allowed (the partition cap still guards - OOM); we may add a per-tile soft-warn but not dropping. -2. **Duplicate features:** union without dedup (the pyramid won't emit the same feature twice for - one tile; we don't pay to detect duplicates). -3. **Layer handling:** union by layer **name**; multiple layers preserved. -4. **Raster unchanged:** first-wins; existing raster PMTiles tests stay green. - -## Testing - -- Light unit: pack 2+ single-feature MVT blobs for one `(z,x,y)` → decode packed tile → assert N - features. A raster test confirms first-wins is unchanged. -- Heavy unit: same, JVM side. -- **Light-vs-heavy parity** (POLYGON multi-feature) per the convention. -- Existing `gbx_pmtiles_agg` raster + the PMTiles writer tests stay green (regression). -- **Heavy raster duplicate-tile regression (new):** pack a group with two RASTER blobs sharing one - `(z,x,y)` → the archive has **exactly one** directory entry for that tile id and reads back as a - valid PMTiles archive (locks the malformed-duplicate-entry fix the audit surfaced). Add the - equivalent light assertion if not already covered by the existing `duplicate_tileid_dropped` test. -- A focused re-run proving the `st_asmvt_pyramid → groupBy? no → pmtiles_agg` path now preserves - features (NB01 will consume this). - -## Docs / bench - -- `docs/docs/api/pmtiles-functions.mdx`: document that `gbx_pmtiles_agg` **merges** multi-feature - vector tiles per `(z,x,y)` and **first-wins** for raster. (User-facing voice; no internal vocab.) -- If merge changes timings materially, note it in `benchmarking.mdx` per the bench-doc rule. -- Capture any validated perf characteristic in the `docs/superpowers/performance/` corpus if warranted. - -## Out of scope - -- Per-tile feature simplification / drop-by-zoom (RDP) — future. -- Changing `st_asmvt_pyramid` itself (it stays a per-feature generator; the merge happens in the agg). - -## Sequencing - -PV lands before NB01 is finished. After PV merges + reviews green, NB01's pipeline -(`st_asmvt_pyramid → gbx_pmtiles_agg`) is correct; then fix NB01's residual items -(`dbutils.fs.mkdirs` → `os.makedirs`, remove the misleading "packs them correctly" comment, the -deg² unit note, the tier-split LATERAL syntax note) and accept it. diff --git a/docs/superpowers/specs/2026-06-28-vizx-multilayer-viewer-design.md b/docs/superpowers/specs/2026-06-28-vizx-multilayer-viewer-design.md deleted file mode 100644 index e6eeb7f6a..000000000 --- a/docs/superpowers/specs/2026-06-28-vizx-multilayer-viewer-design.md +++ /dev/null @@ -1,446 +0,0 @@ -# Design: VizX multi-layer viewers (unified vector / raster / grid) - -**Date:** 2026-06-28 -**Branch:** `beta/0.4.0` -**Status:** Proposed — awaiting user review before planning. - -## Problem - -VizX today renders one thing at a time. An audit of the viewers found: - -- `plot_static` (matplotlib) already composes layers via `ax=` chaining and reprojects every - layer to EPSG:3857 — but only for vector/grid; `plot_cog` makes its own figure (no `ax=`). -- `plot_interactive` (folium) renders one geometry set; no overlay parameter. -- `plot_pmtiles` (MapLibre GL) embeds exactly **one** base64 archive → one source → one layer. -- `plot_cog` renders one COG over a contextily basemap. - -So the capability the Helios series is built toward — **buildings + NAIP + hillshade overlaid in -one interactive map** — does not exist. Worse, the Helios prose *overstates* it: cells call -`show_pmtiles(one_archive)` and render layers separately, while the prose implies a combined -overlay. That factual gap must be corrected regardless of what we build. - -We want in-notebook rendering that is (a) **unified** across vector, raster, and grid, and -(b) **simple for users** even if complex underneath — a "halo" surface the library is remembered -for. - -## Goals - -- One coherent **`Layer`** abstraction (vector / raster / grid / pmtiles) consumed by two - renderers: `plot_static` (matplotlib) and `plot_interactive` (MapLibre GL). -- Any reasonable **combination and number** of layers in one view. -- **Consolidate interactive rendering on MapLibre GL**; retire folium (one interactive engine - that does GeoJSON + raster image + raster/vector tiles + PMTiles). -- A server-less notebook experience that still works on **Serverless**, classic clusters, and as - a **static thumbnail on GitHub** (committed `.ipynb`). -- Honest, actionable behavior at scale: a defined `>64 MB` strategy, budget-bounded - simplification, and graceful fallback — **no silent degradation**. - -## Non-goals (Phase-2 roadmap, not built here) - -- A Databricks **App** with a FastAPI tile server (the unconditional dynamic / indefinite-size - single-archive path). -- DuckDB as a second spatial engine. GeoBrix's own pyrx/pyvx + tippecanoe + rasterio cover the - preparation needs; DuckDB is not introduced. -- Any heavy-tier (Scala) or new-Spark-function changes. This is Python `gbx.vizx` only. - ---- - -## Core concept: the `Layer` model - -A lightweight dataclass declared with typed constructor helpers (discoverable params per type), -so users write intent, not plumbing: - -```python -vector_layer(data, *, geom_col=None, column=None, cmap="viridis", - fill=True, color=None, width=None, opacity=0.8, label=None) -raster_layer(data, *, band=None, cmap="viridis", opacity=1.0, label=None) # COG path | array | tile struct -grid_layer(data, *, grid_system, cellid_col=None, column=None, cmap="viridis", - opacity=0.7, label=None) # H3 / BNG / quadbin -pmtiles_layer(path_or_bytes_or_url, *, style=None, simplify=None, label=None) # already-tiled archive -``` - -### Column-naming convention (settled) - -One canonical name per concept across the whole VizX surface (beta = no aliases): - -- **`geom_col`** — vector geometry column. -- **`cellid_col`** — DGGS cell-id column. `None` → auto-detect via the existing - `_CELL_COL_CANDIDATES = ("cellid","cell","cell_id","h3","quadbin","bng","index")` - (matches `cells_as_gdf(cell_col="cellid")`). -- **`column`** — the value to color / symbolize by (choropleth). `plot_static` already uses - `column`; we keep it (no rename to `value_col`). - -`data` may be a Spark DataFrame, a pandas/GeoDataFrame, a Volume path, an array, or bytes, -depending on layer type — each constructor documents what it accepts. - ---- - -## The two renderers - -Both accept a single `Layer`, a list of `Layer`s, or (back-compat) a bare DataFrame/path that is -wrapped as one layer. Layers draw in list order (first = bottom). - -```python -plot_static(layers, *, basemap=True, basemap_source=None, title=None, - fig_w=10, fig_h=10, ax=None, ...) -plot_interactive(layers, *, basemap="carto-positron", simplify_tiles_spec=None, - max_embed_mb=64, fallback=True, ...) -``` - -- **`plot_static`** — matplotlib. Each layer reprojected to 3857 and drawn on one `Axes`. - Requires giving **`plot_cog` an `ax=` parameter** so COGs compose with vector/grid; raster - layers draw via `imshow`/`rasterio.show` onto the shared axes. Always works (incl. GitHub). -- **`plot_interactive`** — one self-contained **MapLibre GL** HTML page (see below). Folium is - retired and dropped from the `[vizx]` extra. - -### Back-compat wrappers (kept) - -`plot_pmtiles`, `plot_cog`, `plot_raster`, `plot_file`, `plot_mask_layers` remain as single-layer -convenience wrappers that build the appropriate `Layer` and delegate. The `show_*` config_nb -wrappers are updated to the new entry points. The single-layer call stays the common case. - ---- - -## MapLibre compositor internals - -One HTML page, N sources + N layers, server-less. Per-layer adapter → MapLibre source(s)+layer(s): - -| Layer type | MapLibre representation | -|---|---| -| vector / grid (tiny) | inline **GeoJSON** source + fill/line/circle layers | -| vector / grid (larger) | **tippecanoe → PMTiles** (zoom-aware LOD), embedded or simplified | -| raster (COG / array) | georeferenced **image source** (4-corner coords), decimated to `max_px` | -| pmtiles | `pmtiles://` protocol source (base64 `FileSource`, or URL `FetchSource`) | - -Default basemap is **CARTO Positron** (hosted raster style), configurable, with a `none` option — -it must work under **normal Serverless conditions** (browser has outbound internet). The current -single-archive `_pmtiles.py` HTML builder generalizes into this multi-source builder; -`_interactive.py` (folium) is replaced by it. - -**contextily keeps its place — on the static path.** The interactive engine (MapLibre) uses hosted -basemap tiles; the **static** surface (`plot_static`, `plot_cog`, and the static fallback) stays -matplotlib, and that is where contextily provides the basemap. Complementary, not redundant. - ---- - -## Preparation, the `>64 MB` strategy, and Volume access - -`displayHTML` output runs in a sandboxed iframe served from `databricksusercontent.com` — a -**different origin** from the workspace API. Findings that constrain the design: - -- **Empirically confirmed (2026-06-28, CLI probe of the AWS workspace Files API):** the gateway - returns `Access-Control-Allow-Origin: *` and allows the `Authorization` header cross-origin, **but - its `Access-Control-Allow-Headers` does NOT include `Range`.** pmtiles.js range reads send a - `Range` header, so the browser **preflight fails on that header** and the request is blocked. So - the block is not the origin — it is specifically the `Range` request header not being permitted - cross-origin. The same wall blocks `driver-proxy`. **pmtiles.js cannot range-read a Volume via the - Files API from the notebook iframe.** -- Copying a Volume archive to a **driver temp path does not help** — the browser cannot read a - driver-local file; bytes still have to be either embedded in the page or served over a - CORS-reachable URL the driver can't provide to the iframe. -- Given the Files-API result, the **only** remaining notebook route to indefinite single-archive - size is a **presigned object-store URL** (range-read straight from S3/ADLS/GCS, bypassing the - gateway so the *bucket's* CORS governs, not the API gateway's), gated on that bucket's CORS - allowing the iframe origin **and the `Range` header** — **conditional and spike-gated** (often not - controllable for UC-managed storage; minting a presigned URL for a managed Volume file is itself - uncertain). **Spike A resolved this: for a MANAGED volume no presigned URL is mintable → App-only.** - An EXTERNAL volume with customer-configured bucket CORS could enable the rung — parked, out of - scope now. -- The durable answer for indefinite single-archive streaming is the **Phase-2 App** (same-origin). - -### Budget is measured on the *prepared artifact* bytes, not raw input - -A 500 MB DataFrame can prepare to a 2 MB GeoJSON / a small tippecanoe PMTiles; a 4 GB DEM -decimates to a small PNG. The ladder for a layer (per layer; the page total is the sum): - -```text -1. layer has an explicit CORS-reachable http(s) URL -> FetchSource (range) [indefinite size] -2. prepared bytes <= max_embed_mb -> embed (FileSource / inline) [<= ~64 MB] -3. simplify_tiles_spec present -> simplify to <= budget, embed [NEW] -4. else -> static composite fallback [always works] -``` - -- A **finished PMTiles archive is never silently shrunk** — only a streamable URL or the static - fallback get it past the budget. -- **Every reduction warns, loudly and actionably** (no silent caps). -- Raster is bounded by decimation at prep, so it rarely drives the budget. - -### Two complementary scaling axes for "indefinite size", server-less - -- **Sharding (spatial axis, already in NB04):** many bounded per-shard archives + `mosaic.json`; - embed/preview a shard, assemble the mosaic client-side. -- **Simplification (zoom/precision axis, this spec):** one archive simplified across zoom levels - to a byte budget (`simplify_tiles()`). - -They compose (shard *and* simplify). - ---- - -## `simplify_tiles()` and `simplify_tiles_spec` - -One engine, one spec, two materialization modes. The spec is plain JSON/dict (consistent with -`grid_conf`), so it is the single source of truth and serializes cleanly. - -**Two flavors of the engine (decided): split by input rather than overload one function with two -modes.** Combining "re-tile from source" and "trim an existing archive" in one signature is -confusing, so: - -```python -simplify_tiles_from_source(source, *, spec=, out_path=None) -> bytes | path # re-tile (tippecanoe / distributed) -simplify_tiles_from_archive(pmtiles_path, *, spec=, out_path=None) -> bytes | path # tile-join down-zoom/trim -plot_interactive(layers, *, simplify_tiles_spec=, ...) # inline, picks the right flavor by layer input -``` - -`plot_interactive` routes to the right flavor automatically based on whether the layer carries -source data or an existing archive path. Both flavors consume the **same** `spec`. - -### Spec schema - -```json -{ - "budget_mb": 64, // total embed ceiling for the simplified archive - "min_z": 0, - "max_z": 10, // overview ceiling — and the zoom cut-over seam (one knob, not two) - "tolerance": "auto", // geometry simplification; "auto" derives per-zoom, or a number - "drop_densest": true, // shed least-important features when a tile exceeds budget - "cluster_distance": null, // optional point clustering (tippecanoe --cluster-distance) - "keep_attrs": null, // null = all; a list prunes attributes (a big size lever) - "raster_max_px": 1024, // overview downsample ceiling for raster layers - "effort": "fast" // "fast" (inline default) | "full" (durable default) -} -``` - -`max_z` defines both the simplified-overview ceiling and the zoom cut-over seam; there is no -separate `overview_max_z`. - -### Default vs per-layer override - -`simplify_tiles_spec` on `plot_interactive` is the **default policy**; a `Layer` may carry its own -`simplify=` to override (e.g. `vector_layer(df, simplify={...})`). A layer with no spec that is -under budget embeds as-is. - -### Ephemeral vs durable (the tension, resolved by the two entry points) - -| | (a) Ephemeral "just let me see it" | (b) Durable "prepare once, reuse" | -|---|---|---| -| Entry | `plot_interactive(layers, simplify_tiles_spec=…)` | `simplify_tiles(source, spec, out_path="/Volumes/…/overview.pmtiles")` | -| Output | driver temp, session-cached by `hash(source, spec)`, GC'd | persistent PMTiles on a Volume | -| Defaults | `effort: "fast"` (favor latency) | `effort: "full"` (favor fidelity) | -| Reuse | this cell / session | many cells / sessions / notebooks / Phase-2 App / external | -| Identity | transient | an ETL artifact — catalog-able, versionable | - -The same spec drives both, so a preview you liked is **promoted to durable** by handing that exact -spec to `simplify_tiles(out_path=…)`; the result is then a `pmtiles_layer(path)` both compositors -consume with zero re-prep. - -**Guardrails so (a) never masquerades as (b):** -- `plot_interactive`'s inline simplify is best-effort, **cached by `hash(source, spec)`**, and - **transient**, and it **warns** when it finds itself doing heavy simplification repeatedly, - nudging the user to `simplify_tiles(out_path=…)`. - -### Engine policy (where tippecanoe lands) - -tippecanoe ships **pip-installable manylinux wheels** that bundle the binary — gold-standard -budget-aware vector simplification (`--maximum-tile-bytes`, `--drop-densest-as-needed`, -`--cluster-distance`, `--accumulate-attribute`) with PMTiles output. Its expanded role here: - -- **Default vector tiler in `prepare()`** (not just simplification): tiny vector → inline GeoJSON; - larger → tippecanoe → PMTiles with zoom-aware LOD. Simplification is just tippecanoe with a - budget — not a separate code path. -- **`tile-join`** powers `simplify_tiles()` when the input is an **existing PMTiles** (Volume path): - down-zoom + budget-trim an already-built archive into a 0–`max_z` overview without re-tiling - from source. -- **Clustering** (`--cluster-distance`) for dense point/grid layers. - -Policy by scale and type: - -- **Vector, moderate (driver-local):** tippecanoe. -- **Vector, very large:** GeoBrix **distributed** tiling (pyvx) — fan out, then aggregate a bounded - overview to the driver. (tippecanoe is single-node and would bottleneck/OOM.) -- **Raster:** rasterio overview downsampling (no tippecanoe). - -**Positioning guardrail:** tippecanoe is *VizX plumbing for viz simplification/overviews*. GeoBrix's -own distributed tiling (`gbx_st_asmvt_pyramid`, `gbx_pmtiles_agg`, the Helios pipelines) **remains -the product tiling story** for at-scale, full-fidelity, published deliverables. The two are -complementary, not competing, and the doc/notebooks frame them that way. - -Caveats: single-node ceiling (driver), vector-only, **exact-pin + hash** in the `[vizx]` extra, -and **verify the cp312/manylinux wheel on the real Serverless env** (a one-line spike). - ---- - -## Zoom cut-over (contingent on Phase 1.5) - -Low zooms (0–`max_z`) are cheap to embed (few tiles, aggressive simplification); the budget -pressure is at high zoom (full detail). So: - -- Embed the simplified `min_z..max_z` overview as a `maxzoom=max_z` PMTiles source. -- Stream `max_z+1..N` detail dynamically (a dynamic source with `minzoom=max_z+1`), the - `moveend`/`zoomend` hook firing only at `z > max_z`. - -This **only lights up if the Phase-1.5 spike succeeds**. If it fails, the map is interactive up to -`max_z` (overview embedded) and sends users to static / sharding for detail — still coherent. - ---- - -## Phase 1.5 spike (gating): in-notebook dynamic loading - -The only notebook-native bidirectional JS↔Python channel that sidesteps CORS is **ipywidgets / -[AnyWidget](https://anywidget.dev/)** — its comm rides the kernel socket, not a cross-origin HTTP -request. The dynamic loop would be: MapLibre `moveend` → `model.send({bbox, zoom})` → a driver-side -callback prepares viewport data → trait update → JS `model.on("change")` refreshes the map source. - -**Spike question:** does the AnyWidget/ipywidgets comm work on **Serverless** (the strategic -target)? Solid on recent classic DBR; Serverless support must be proven before we bank on it. - -- **Pass →** build the dynamic high-zoom cut-over onto the embedded overview. -- **Fail →** document honestly; dynamic deferred to the Phase-2 App; Phase-1 stays embed + static - + cell-driven re-render (a Python-side helper that re-prepares for a new bbox and re-displays). - -Caveats even on pass: AnyWidget does not render in a committed `.ipynb` on GitHub (needs a live -kernel) — the static category-2 fallback covers that surface; and it is a meaningfully larger build. - ---- - -## Phasing - -- **Phase 1 (build):** `Layer` model + constructors; `plot_static(layers)` (incl. `plot_cog ax=`); - `plot_interactive(layers)` on MapLibre (folium retired); the `>64 MB` ladder; `simplify_tiles()` - + `simplify_tiles_spec` (ephemeral/durable modes, tippecanoe/distributed/rasterio engine policy); - the repo-wide notebook/doc audit + Helios NB02/NB03 rewiring + prose fix. -- **Phase 1.5 (gating spike, run BEFORE planning):** AnyWidget comm on Serverless (Spike B) and the - presigned-URL CORS/range path (Spike A). **Spike B pass → the dynamic zoom cut-over is built as an - immediate follow-on within this effort** (not deferred to Phase 2). Spike B fail → documented; - cut-over degrades to overview-only-interactive + cell-driven re-render. Spike A pass → wire the - presigned-URL helper into ladder step 1 for indefinite-size Volume archives; fail → indefinite - single-archive is Phase-2 only. -- **Phase 2 (roadmap only):** Databricks App tile server for unconditional dynamic / indefinite - single-archive streaming. - -> **Spikes run before the plan** (per direction): the CORS Files-API probe is **done** (Range header -> blocked — see above). Spike A (presigned-URL) and Spike B (AnyWidget) are browser/notebook-frontend -> behaviors that require a human run on a live Serverless notebook; harnesses are provided. Their -> results finalize whether the cut-over and the indefinite-size ladder rung are in or out before -> task-by-task planning begins. - ---- - -## Scope - -**In:** the `Layer` model + constructors; `plot_static(layers)` (+ `plot_cog ax=`); MapLibre -`plot_interactive(layers)` replacing folium; the `>64 MB` ladder; `simplify_tiles()` + -`simplify_tiles_spec`; docs (`vizx.mdx`) + tests; a **repo-wide audit of every notebook + doc that -calls the VizX functions** (`plot_interactive`/`plot_static`/`plot_pmtiles`/`plot_cog`/`plot_raster` -+ `show_*`) so the folium retirement and the `plot_interactive` signature change don't strand -`eo-series`, `h3-rasterize`, `xview`, or Helios; and the **Helios NB02/NB03 rewiring + prose fix**. - -**Out:** the Phase-2 App, FastAPI tile server, DuckDB, on-the-fly server tiling; heavy-tier or -new-Spark-function changes. - ---- - -## Documentation (red-carpet — this is a halo surface) - -Rendering data "that doesn't quite fit in a notebook cell" is something many users struggle to -noodle out on their own. That makes this a **halo capability**, and the docs get first-class -treatment, not a reference-table afterthought: - -- **A dedicated narrative page** (`docs/docs/api/vizx-layers.mdx` or a prominent expansion of - `vizx.mdx`) that *teaches the problem and the ladder*: "I have more tile data than a notebook cell - can hold — here's how to see it anyway." Walk the decision tree (embed → simplify → URL → shard → - static) in plain language with runnable examples for each rung. -- **Multi-layer worked examples** for both renderers (static composite and interactive MapLibre), - using real sample data — vector + raster + grid in one map. -- **The ephemeral-vs-durable story made explicit**: when to `plot_interactive(simplify_tiles_spec=…)` - vs. when to materialize with `simplify_tiles_from_source(out_path=…)` and reuse — with a - copy-paste "promote your preview to a durable artifact" snippet. -- **Honest scale guidance**: the `>64 MB` behavior, the sharding alternative (link to Helios NB04), - and the Phase-2 App as the path for truly indefinite single-archive interactivity — so users are - never surprised by a fallback. -- **Visual-first**: screenshots/GIFs of multi-layer output; a decision-tree diagram (a new - `resources/images/diagrams/vizx/` asset following the established diagram-generator pattern). -- Doc code is **executable doc-tests** (the repo's single-source rule), so every example is proven. - -The audit-and-migrate of existing notebooks/docs (below) is part of this: every VizX usage across -the docs site reads consistently against the new surface. - -## Dependencies & supply chain - -- **tippecanoe** (PyPI manylinux wheel), **anywidget** (Phase-1.5 spike only), MapLibre GL JS + - pmtiles.js (vendored/CDN as today). folium **removed** from `[vizx]`. -- All execution-env packages **exact-version + hash-pinned** (`--require-hashes`), per the repo's - supply-chain rule. Add to the `[vizx]` extra and the CI lock. -- The MapLibre GL JS + pmtiles.js the viewer injects must be **vendored or loaded with Subresource - Integrity** (`integrity="sha384-…" crossorigin="anonymous"`), not bare CDN `