databrickslabs · sllynn · Jul 7, 2026 · Jul 9, 2026 · Jul 9, 2026
@@ -1,4 +1,6 @@
 /.idea/
+/.metals/
+/.bloop/
 /scripts/docker/m2/
 /python/geobrix/.venv/
 /python/geobrix/dist/
@@ -48,4 +50,6 @@ __pycache__/
 !/.claude/qc-judge/
 /input/
 .venv-pyrx/
+.venv-host-ci/
+.venv-host-pyrx/
 test-logs/bench/
@@ -57,6 +57,19 @@ Use `gbx:docker:start` / `gbx:docker:exec` rather than `docker run` directly. Th
 
 Default Maven profile is **`skipScoverage`** for fast compile/test (`mvn clean package -DskipTests`). Coverage commands explicitly trigger the `standard` profile.
 
+### Building/testing on the host (arca, outside the container)
+
+The container is still the canonical path, but the project also builds and tests directly on an arca host (Ubuntu 24.04 noble, x86_64) via a `$HOME`-local GDAL install — useful when Docker isn't wanted. This host setup lives in the **`geobrix-arca` plugin** (marketplace `experimental/general`), not in this repo — install it with `isaac plugin add geobrix-arca@experimental`. Its two skills:
+
+- **`geobrix-gdal-env`** — provisions native GDAL into `$HOME`. Its `setup-host-gdal.sh` adds the fingerprint-pinned UbuntuGIS PPA, downloads the pinned `.debs` (GDAL `3.11.4+dfsg-1~noble0`, matching `scripts/build-gdal-artifacts.sh`), extracts them under `$HOME/.local/gdal` (no root, no `/usr` writes), reuses the committed JNI (`scripts/gdal311/libgdalalljni.so` — a plain committed file, **not** LFS), and bridges the two `/usr` paths the Scala hardcodes. **The LFS platform tarball is unreachable from arca** (`lfs.github.com` is firewall-blocked), so the PPA-download path is the one that works here — not extracting the committed tarball.
+  - **Each fresh session:** `setup-host-gdal.sh --bridges-only` recreates the two root-owned `/usr` symlinks (they don't persist; `$HOME/.local/gdal` does).
+  - **In every build/test shell:** `source ~/.local/geobrix-gdal-env.sh`. It pins **Java 17** (arca defaults to 21; pom targets `release=17`), and sets `LD_LIBRARY_PATH` (so the JNI resolves `libgdal.so.37` + deps), `PROJ_DATA`/`GDAL_DATA`, and `PYTHONPATH` (extracted `osgeo`/`numpy` — required or GDAL's embedded-Python VRT pixel functions in RST_MapAlgebra/CombineAvg/DerivedBand/vegetation-indices silently no-op and the output raster is null).
+- **`geobrix-metals-nav`** — fixes Cursor/Metals code navigation on arca (the managed Metals is wired to the monorepo Bazel BSP, so geobrix imports underline as unresolved). Generates an analysis-only Bloop config (Scala 2.13.18 to match the mtags arca's Metals ships; the pom/build/JAR stay 2.13.16) and steers Metals to it.
+
+Then build with `mvn clean package -DskipTests -PskipScoverage`, or run suites with `mvn test -PskipScoverage -DskipTests=false -Dsuites='...'`. For `/Volumes` suites, symlink `/Volumes` → `sample-data/Volumes` (mirrors the container mount); the full sample bundles still require `gbx:data:download`.
+
+**Caveat:** the `gbx:test:*` commands are hardwired to `docker exec geobrix-dev` and do **not** run on the host — invoke `mvn` directly, or extend those commands with a `--host` mode.
+
 ## Commands (the `gbx:*` palette)
 
 The repo has **50 `gbx:*` commands** in `scripts/commands/` (each is a `.md` registration + a `.sh` implementation). They handle Docker setup, env vars, log paths (`--log filename` → `test-logs/filename`), and profile selection. Originally registered for Cursor's command palette (hence the `.md` files), they're now invoked directly from any shell or via the Task tool.

@@ -186,6 +186,121 @@ validate_set() {
     esac
 }
 
+# Assert the arca host GDAL environment is active (LD_LIBRARY_PATH points at the $HOME-local
+# GDAL install and gdalinfo is on PATH). The --host test paths need native GDAL in the forked
+# Spark JVM; that is provided by sourcing ~/.local/geobrix-gdal-env.sh (geobrix-arca plugin),
+# NOT by this repo. We only assert it — we never source it (it's user/plugin-owned).
+# Returns non-zero with a remediation message if the env is not active.
+require_host_gdal_env() {
+    if ! command -v gdalinfo >/dev/null 2>&1 || [[ "${LD_LIBRARY_PATH:-}" != *".local/gdal"* ]]; then
+        echo -e "${RED}❌ Host GDAL environment not active.${NC}" >&2
+        echo -e "${YELLOW}   --host mode needs native GDAL on the arca host. Source the env first:${NC}" >&2
+        echo -e "${YELLOW}     source ~/.local/geobrix-gdal-env.sh${NC}" >&2
+        echo -e "${YELLOW}   (provisioned by the geobrix-arca plugin's geobrix-gdal-env skill).${NC}" >&2
+        return 1
+    fi
+    return 0
+}
+
+# Ensure a host test venv exists, built from one of CI's exact hash-pinned locks via uv, with the
+# geobrix package installed editable+no-deps. Mirrors CI's TWO-environment split (CI never runs all
+# Python tests in one env) — pick the venv by the "kind" arg:
+#
+#   ci    -> $PROJECT_ROOT/.venv-host-ci   from requirements-ci.txt  (~27 pkgs: pyspark, py4j,
+#            numpy, pytest; NO rasterio/pandas/pdal). Matches CI's python_build "heavy" job, which
+#            runs `pytest test -m "not integration" --ignore=test/pyrx --ignore=test/pyvx`.
+#   pyrx  -> $PROJECT_ROOT/.venv-host-pyrx from requirements-pyrx-ci.txt (~104 pkgs: rasterio,
+#            shapely, pandas, pyarrow, h3, mapbox-vector-tile, vizx stack). Matches CI's pyrx_build
+#            "light" job (test/pyrx test/ds test/pyvx test/pygx test/pmtiles_light test/stac
+#            test/vizx test/sample) and is also the right env for the doc-tests (rasterio/pandas).
+#
+# Both locks are pure wheels on arca (neither contains pdal, which is source-only and needs native
+# PDAL the container builds but arca lacks — so no package filtering is required, unlike the
+# container image lock). uv is required (stdlib `python3 -m venv` yields a pip-less venv on arca —
+# ensurepip is absent). The index is taken from ambient PIP_INDEX_URL/UV_INDEX_URL — never hardcoded.
+#
+# gdal/osgeo are NOT in either lock (CI installs gdal[numpy] from the apt-matched sdist); on the host
+# they are provided on PYTHONPATH by the sourced arca env, additive to the venv at runtime.
+#
+# A stamp holds the sha256 of the source lock so re-runs skip the install unless the lock changed.
+# Set GBX_REBUILD_VENV=1 to force a rebuild. Echoes the venv bin dir; callers use
+# "$(ensure_host_test_venv <kind>)/python -m pytest".
+ensure_host_test_venv() {
+    local kind="${1:-pyrx}"
+    local venv_dir reqs
+    case "$kind" in
+        ci)   venv_dir="${PROJECT_ROOT}/.venv-host-ci";   reqs="${PROJECT_ROOT}/python/geobrix/requirements-ci.txt" ;;
+        pyrx) venv_dir="${PROJECT_ROOT}/.venv-host-pyrx"; reqs="${PROJECT_ROOT}/python/geobrix/requirements-pyrx-ci.txt" ;;
+        *)    echo -e "${RED}❌ ensure_host_test_venv: unknown kind '$kind' (expected ci|pyrx)${NC}" >&2; return 1 ;;
+    esac
+    local stamp="${venv_dir}/.gbx-reqs-stamp"
+    local py_version="${GBX_HOST_PY_VERSION:-3.12}"
+
+    if ! command -v uv >/dev/null 2>&1; then
+        echo -e "${RED}❌ uv not found on PATH — required to build the host test venv.${NC}" >&2
+        echo -e "${YELLOW}   Install uv: https://docs.astral.sh/uv/ (or use the geobrix-arca plugin).${NC}" >&2
+        return 1
+    fi
+    if [ ! -f "$reqs" ]; then
+        echo -e "${RED}❌ Pinned requirements not found: $reqs${NC}" >&2
+        return 1
+    fi
+
+    # Stamp on the source lock's hash: if the committed lock changes, the venv rebuilds.
+    local want_hash cur_hash
+    want_hash="$(sha256sum "$reqs" | awk '{print $1}')"
+    cur_hash="$(cat "$stamp" 2>/dev/null || true)"
+
+    if [ "${GBX_REBUILD_VENV:-0}" = "1" ] || [ ! -x "${venv_dir}/bin/python" ] || [ "$want_hash" != "$cur_hash" ]; then
+        echo -e "${CYAN}🐍 Building host test venv (${kind}) at ${YELLOW}${venv_dir}${CYAN} from $(basename "$reqs")...${NC}" >&2
+        [ "${GBX_REBUILD_VENV:-0}" = "1" ] && rm -rf "$venv_dir"
+        uv venv "$venv_dir" --python "$py_version" >&2 \
+            || { echo -e "${RED}❌ uv venv failed${NC}" >&2; return 1; }
+        uv pip install --python "${venv_dir}/bin/python" --require-hashes -r "$reqs" >&2 \
+            || { echo -e "${RED}❌ uv pip install ($kind lock) failed — check PIP_INDEX_URL/proxy coverage${NC}" >&2; return 1; }
+        uv pip install --python "${venv_dir}/bin/python" --no-deps -e "${PROJECT_ROOT}/python/geobrix" >&2 \
+            || { echo -e "${RED}❌ editable geobrix install failed${NC}" >&2; return 1; }
+        printf '%s\n' "$want_hash" > "$stamp"
+        echo -e "${GREEN}✅ Host test venv (${kind}) ready.${NC}" >&2
+    fi
+
+    echo "${venv_dir}/bin"
+}
+
+# Prepare the process environment so a host --host pytest run drives Spark + rasterio correctly.
+# Call with the venv bin dir (from ensure_host_test_venv) BEFORE launching pytest. Two exports the
+# forked Spark JVM's Python workers and rasterio need:
+#   - PYSPARK_PYTHON / PYSPARK_DRIVER_PYTHON = the venv interpreter, so Spark workers use the venv
+#     (pandas/pyarrow for Arrow UDFs live there, not in system python3). These MUST be real exports
+#     in the current shell — a `VAR=x eval "..."` command-prefix does NOT propagate to the python
+#     grandchild, so workers would silently fall back to system python3 (ModuleNotFound: pandas).
+#   - unset PROJ_DATA / PROJ_LIB so the venv rasterio uses its own bundled proj.db (layout >=6);
+#     the arca env points these at the older $HOME GDAL proj.db (layout 3), which rasterio rejects
+#     (CRSError "... another PROJ installation"). unset (not empty-string) is required — an empty
+#     PROJ_DATA is a search path of "", not a fallback to bundled data. The JVM GDAL sets PROJ_LIB
+#     internally via SetConfigOption (the /usr/share/proj bridge), so the heavy tier is unaffected.
+# Usage:  activate_host_python_env "$VENV_BIN"
+activate_host_python_env() {
+    local venv_bin="$1"
+    export PYSPARK_PYTHON="${venv_bin}/python"
+    export PYSPARK_DRIVER_PYTHON="${venv_bin}/python"
+    unset PROJ_DATA PROJ_LIB
+}
+
+# The light-tier test dirs (single source of truth: python/geobrix/test/conftest.py _LIGHT_TEST_DIRS).
+# Their modules import light-only deps (rasterio/shapely/pandas/h3/…) at collection time, so they run
+# only in the pyrx venv; the ci venv's conftest collect_ignore skips them (rasterio absent). Echoed
+# space-separated. Falls back to a hardcoded list only if the conftest can't be parsed.
+host_light_test_dirs() {
+    local conftest="${PROJECT_ROOT}/python/geobrix/test/conftest.py"
+    local dirs
+    dirs="$(awk '/_LIGHT_TEST_DIRS *= *\[/{f=1;next} f&&/\]/{f=0} f{gsub(/[",[:space:]]/,""); if($0!="") print}' "$conftest" 2>/dev/null | tr '\n' ' ')"
+    if [ -z "${dirs// /}" ]; then
+        dirs="bench ds pyrx pyvx pygx pmtiles_light stac vizx sample"
+    fi
+    echo "$dirs"
+}
+
 # Run a command inside the isolated pyrx venv (host, no Docker).
 # Usage: run_in_pyrx_venv "<command string>"
 # Requires gbx:venv:sync to have been run (venv at $PROJECT_ROOT/.venv-pyrx).
@@ -209,4 +324,6 @@ run_in_pyrx_venv() {
 export RED GREEN YELLOW BLUE CYAN NC DOCKER_MAVEN_ENV
 export -f check_docker resolve_log_path setup_log_file show_banner show_separator \
           print_report_link open_report generate_timestamp warn_if_jar_stale \
-          print_banner print_separator setup_log run_in_pyrx_venv validate_set 2>/dev/null || true
+          print_banner print_separator setup_log run_in_pyrx_venv validate_set \
+          require_host_gdal_env ensure_host_test_venv activate_host_python_env \
+          host_light_test_dirs 2>/dev/null || true
@@ -20,6 +20,8 @@ bash scripts/commands/gbx-test-docs.sh [OPTIONS]
 
 **Common**
 
+- `--host` – Run on the host (arca), not the Docker container. Passed through to each child suite (python-docs, sql-docs, scala-docs). Requires `source ~/.local/geobrix-gdal-env.sh` first (provisioned by the `geobrix-arca` plugin). See "Host mode" below.
+- `--rebuild-venv` – (with `--host`) force-rebuild the host test venv; forwarded to the venv-based child suites (python-docs, sql-docs).
 - `--log <path>` – Log file (filename → `test-logs/<name>`).
 - `--markers <markers>` – Pytest markers for Python (e.g. `"not slow"`).
 - `--include-integration` – Include Python integration tests (excluded by default).
@@ -30,6 +32,10 @@ bash scripts/commands/gbx-test-docs.sh [OPTIONS]
 - `--no-sample-data-root` – Do **not** set `GBX_SAMPLE_DATA_ROOT` (use your env or path_config default; e.g. full bundle).
 - `--help` – Help and examples.
 
+## Host mode (arca, no Docker)
+
+With `--host` the orchestrator runs on the host instead of `docker exec geobrix-dev`, forwarding `--host` to each child suite (python-docs, sql-docs, scala-docs). Prerequisites: `source ~/.local/geobrix-gdal-env.sh` (native GDAL + Java 17 + PYTHONPATH) and `uv` on PATH, with `PIP_INDEX_URL` pointing at the internal pip proxy. The venv-based child suites build a host test venv from the exact CI-pinned locks via `uv` on first run — `.venv-host-pyrx` from `python/geobrix/requirements-pyrx-ci.txt` (light-tier deps); neither CI lock contains `pdal` (source-only, unbuildable on arca); `--rebuild-venv` is forwarded to them. The scala-docs child runs `mvn` directly and needs only the sourced GDAL env. See the `geobrix-arca` plugin for the full setup.
+
 **Sample data (default):** The command sets `GBX_SAMPLE_DATA_ROOT=/Volumes/main/default/test-data` inside the container so doc tests use the minimal bundle (host path `sample-data/Volumes/main/default/test-data`). This is required for running docs unit tests on remote/CI. Use `--no-sample-data-root` to leave it unset (e.g. to use a full bundle or your own env).
 
 ## Examples
@@ -38,6 +44,10 @@ bash scripts/commands/gbx-test-docs.sh [OPTIONS]
 # Full run with build
 bash scripts/commands/gbx-test-docs.sh
 
+# On the arca host (no Docker) — source the GDAL env first
+source ~/.local/geobrix-gdal-env.sh
+bash scripts/commands/gbx-test-docs.sh --host
+
 # Fast run (skip build), with log. Uses in-repo minimal bundle; no download.
 bash scripts/commands/gbx-test-docs.sh --skip-build --log docs.log
 

@@ -21,6 +21,7 @@ show_help() {
     echo -e "  ${GREEN}--test <nodeid>${NC}         Python: single test node id"
     echo ""
     echo -e "${CYAN}Common options:${NC}"
+    echo -e "  ${GREEN}--host${NC}                 Run on the host (arca), not Docker. Requires ${YELLOW}source ~/.local/geobrix-gdal-env.sh${NC} first."
     echo -e "  ${GREEN}--log <path>${NC}           Write output to log (filename → test-logs/<name>)"
     echo -e "  ${GREEN}--markers <marker>${NC}     Pytest markers for Python (e.g. \"not slow\")"
     echo -e "  ${GREEN}--include-integration${NC}  Include Python integration tests (excluded by default)"
@@ -47,13 +48,22 @@ SCALA_SUITE="tests.docs.scala.*"
 PYTHON_ONLY=false
 SCALA_ONLY=false
 SET_SAMPLE_DATA_ROOT=true
+USE_HOST=false
 # Pass-through for Python phase (only one of these set)
 SUITE_VAL=""
 PATH_VAL=""
 TEST_VAL=""
 
 while [[ $# -gt 0 ]]; do
     case $1 in
+        --host)
+            USE_HOST=true
+            shift
+            ;;
+        --rebuild-venv)
+            export GBX_REBUILD_VENV=1
+            shift
+            ;;
         --suite)
             case "$2" in
                 quickstart|api|readers|rasterx|advanced|setup)
@@ -128,13 +138,18 @@ done
 cd "$PROJECT_ROOT"
 
 show_banner "📚 GeoBrix: All Documentation Tests"
-check_docker
 setup_log_file "$LOG_PATH"
 
-mkdir -p "$PROJECT_ROOT/sample-data/Volumes/main/default/geobrix_samples"
-if ! docker exec geobrix-dev test -d /Volumes 2>/dev/null; then
-    echo -e "${RED}❌ /Volumes not found. Start with: ./scripts/docker/start_docker_with_volumes.sh${NC}"
-    exit 1
+if [ "$USE_HOST" = true ]; then
+    require_host_gdal_env || exit 1
+    echo -e "${CYAN}🖥️  Host mode (arca) — passing --host to child suites${NC}"
+else
+    check_docker
+    mkdir -p "$PROJECT_ROOT/sample-data/Volumes/main/default/geobrix_samples"
+    if ! docker exec geobrix-dev test -d /Volumes 2>/dev/null; then
+        echo -e "${RED}❌ /Volumes not found. Start with: ./scripts/docker/start_docker_with_volumes.sh${NC}"
+        exit 1
+    fi
 fi
 
 [ "$SKIP_BUILD" = true ] && echo -e "${CYAN}⏭️  Skipping build (--skip-build)${NC}"
@@ -146,6 +161,7 @@ TOTAL_EXIT=0
 PYTHON_ARR=()
 SQL_ARR=()
 SCALA_ARR=()
+[ "$USE_HOST" = true ] && { PYTHON_ARR+=(--host); SQL_ARR+=(--host); SCALA_ARR+=(--host); }
 [ "$SKIP_BUILD" = true ] && { PYTHON_ARR+=(--skip-build); SQL_ARR+=(--skip-build); SCALA_ARR+=(--skip-build); }
 [ "$SET_SAMPLE_DATA_ROOT" = false ] && { PYTHON_ARR+=(--no-sample-data-root); SQL_ARR+=(--no-sample-data-root); SCALA_ARR+=(--no-sample-data-root); }
 [ -n "$MARKERS_VAL" ] && [ "$INCLUDE_INTEGRATION" = false ] && { PYTHON_ARR+=(--markers "$MARKERS_VAL"); SQL_ARR+=(--markers "$MARKERS_VAL"); }

@@ -11,9 +11,15 @@ bash scripts/commands/gbx-test-function-info.sh [OPTIONS]
 ## Options
 
 - `--skip-generate` - Skip the generator; run only pytest in `docs/tests-function-info/`
+- `--host` - Run on the host (arca), not the Docker container. Requires `source ~/.local/geobrix-gdal-env.sh` first (provisioned by the `geobrix-arca` plugin); builds/reuses `.venv-host-pyrx` from the pinned CI lock and runs against a host-built JAR. See "Host mode" below.
+- `--rebuild-venv` - (with `--host`) force-rebuild the host test venv.
 - `--log <path>` - Write output to log file
 - `--help` - Display help
 
+## Host mode (arca, no Docker)
+
+With `--host` the command runs directly on the host instead of `docker exec geobrix-dev`. Prerequisites: `source ~/.local/geobrix-gdal-env.sh` (native GDAL + Java 17 + PYTHONPATH) and `uv` on PATH, with `PIP_INDEX_URL` pointing at the internal pip proxy. The pytest registers functions via the built JAR; no sample data is needed. The first run builds a host test venv from the exact CI-pinned lock via `uv` — `.venv-host-pyrx` from `python/geobrix/requirements-pyrx-ci.txt` (the light-tier deps: rasterio/pandas/h3/vizx). Neither CI lock contains `pdal` (source-only, unbuildable on arca and not needed here). See the `geobrix-arca` plugin for the full setup.
+
 ## Default behavior (inside Docker)
 
 1. **Generate**: `python3 docs/scripts/generate-function-info.py` in container  
@@ -29,6 +35,10 @@ bash scripts/commands/gbx-test-function-info.sh [OPTIONS]
 # Full run: generate then test
 gbx:test:function-info
 
+# On the arca host (no Docker) — source the GDAL env first
+source ~/.local/geobrix-gdal-env.sh
+gbx:test:function-info --host
+
 # Only run tests (do not regenerate JSON)
 gbx:test:function-info --skip-generate