release: GeoBrix 0.5.0 — virtual tiles, v2 tile struct, CRS families by mjohns-databricks · Pull Request #72 · databrickslabs/geobrix

mjohns-databricks · 2026-08-13T14:34:58Z

GeoBrix 0.5.0 — release PR (`beta/0.5.0` → `main`)

Promotes the 0.5.0 line to main (~496 commits). Versions are bumped (pom.xml + __version__ = 0.5.0). Full change list: docs/docs/release-notes.mdx → "What's new in v0.5.0".

Highlights

Virtual tiles — bytes-free windowed reads for large rasters. Lightweight cog_gbx/raster_gbx/gtiff_gbx readers can emit virtual tiles (source path + pixel window, no bytes); pixels are read lazily one window at a time. A virtual row is ~100 B vs 148–527 KB materialized (~1,400–5,000× smaller), dissolving Serverless OOM on large-raster fan-out. Lightweight raster readers now default to virtual (breaking behavior change).
v2 8-field tile struct — cellid, raster (nullable), path, window, clip_polygon, clip_crs, crs, metadata. Both tiers read v1 and v2 and always emit v2.
Every lightweight rst_* is virtual-tile-aware via one shared open path (header-only accessors; reference/passthrough ops stay virtual; pixel ops materialize only their window). New force-output params: virtualize_dir / virtualize_prefix / materialize.
COG preparation lane — new file_gbx (path lister) + cog_gbx writer (master-COG prep) + cog_gbx reader (COG-aware windowed read: tileSize/overlapPercent/clipPolygons/windows). Reader splitStrategy now defaults to none.
CRS families (both tiers) — gbx_st_{crs,setcrs,transformcrs} (VectorX) and gbx_rst_{crs,setcrs,transformcrs} (RasterX): authority-string CRS handling that survives non-EPSG round-trips.
Plus per-function 4-tab docs examples, Scala-derived DESCRIBE FUNCTION signatures, and many fixes.

Pre-merge gate (requested)

Complete the eo-series' use of virtual tiles (lightweight tier / Serverless): notebooks showcase virtual tiles (narrative + functions used), the eo-series doc page (docs/docs/notebooks/eo-series.mdx) and the eo-series README.md reflect the support, validated on Serverless.

Opened as draft until the gate above is met. beta/0.5.0 CI is the dev-branch signal (not a merge gate); the main build runs on merge.

This pull request and its description were written by Isaac.

PR #72's build failed on `black --check`. CI lints as `isort && black && flake8 src test`, so black's failure short-circuited before flake8 — hiding pre-existing flake8 debt that surfaces once black passes. This clears both in one pass: - black-reformat 7 drifted files (bench/{runner,spec}.py, test/bench/{cluster, results,spec}, test/pyrx/{core_agg,v2_tile_output_invariant}). - flake8: drop 3 unused imports (F401 in pyrx/_serde.py, test/rasterx/ test_udtf_error_row.py, test/vectorx/test_crs.py) and mark the bench helper run_spark_path with # noqa: C901 (matches the _render precedent). No functional change; none of these files were part of the vizx/eo-series work — this is latent tree-wide lint debt that was masked by the black short-circuit. Co-authored-by: Isaac

… updated bindings

…ons + bindings

- Task 2b: COG multi-window corpus generator (datagen extension, local TDD). - Task 2c: COG multi-window FILE-on/off bench (cluster, manual) — the scenario FILE byte-range wins sharpest (narrow window into a large COG). - Task 6: expand docs to large-rasters + benchmarking pages (FILE-capability callout) and add a stock-Spark-reader underscore-file note to raster.mdx (binaryFile skips _-prefixed files; pathGlobFilter does not bypass; driver-side-listing workaround). Doc-only; no reader-default change. - Final gate: Gate A (listing rake, 10K) + Gate B (FILE capability, COG mw). Co-authored-by: Isaac

GDAL_RowWriter derived the default tile filename from a signed MurmurHash3.seqHash via `.toString.replace("-", "_")`. Negative hashes (~half of all values) became `_`-leading names, which stock Spark file readers (binaryFile, text, parquet, ...) silently skip via Hadoop's hidden-file filter — a customer lost ~25% of files reading GeoBrix raster output. `pathGlobFilter` does not bypass that filter. Use `Integer.toUnsignedString(hash)` so the name is always digit-leading (never `_`/`-`/`.`) and collision-free over the full 32-bit space (no Math.abs -N/+N collision). GeoBrix's own readers list via os.walk and were unaffected; the light Python writer (sha1 hex) was already clean. Co-authored-by: Isaac

… bug GDAL_RowWriter previously used MurmurHash3.seqHash(...).toString.replace("-","_") which produced "_"-leading names for ~half of tiles (signed Int negatives). Hadoop's hidden-file filter silently skipped those files, losing ~50 % of output. Fix (666ca77): Integer.toUnsignedString() always produces digit-leading names. New test in GDAL_DataSourceTest writes tiles with no nameCol and asserts every output .tif filename begins with a digit. Co-authored-by: Isaac

…d_budget helpers Approach 1 foundations: helpers that will let partitions() skip os.walk + per-file rasterio.open when a manifest or tilesTable is provided.

Approach 1: .option("manifest", path) reads pre-computed tile rows from JSON/Parquet; .option("tilesTable", name) reads from a Spark table. Both bypass os.walk + per-file rasterio.open when window/dims are supplied. Flat-column layout (col_off/row_off/width/height top-level fields) handled for tilesTable query results. Mutual exclusion validated in __init__. All three readers benefit via shared partitions() in the base class.

Add _tile_producing_udf_file factory and _uf_* 2-arg FILE-aware UDFs for all Group 2 single-input tile-producing ops: - _uf_initnodata (pending-instruction virtual path + FILE for materialized) - _uf_clip (C1-guard exercise) - _uf_resample, _uf_resample_to_size, _uf_resample_to_res - _uf_update_type, _uf_threshold (op, value) - _uf_transform (identity short-circuit preserved), _uf_to_webmercator - _uf_transformcrs (FILE degrades via C1 warp guard) - _uf_slope, _uf_aspect, _uf_hillshade - _uf_setsrid, _uf_setcrs (pending-instruction virtual path) Public bindings updated to call _uf_*(tc, file_ref_arg(tc), ...) on the non-force-output path. SQL registry single-arg entries unchanged. 19 new tests added; 19 pass, 0 regressions.

Skip rasterio.open at partitions() for the common emit_virtual=True, no-split, no-AOI case. window=None signals read() to resolve dims lazily from the executor-side rasterio.open that was already required there. Over N files this drops planning from N header opens to 0. tileSize/AOI paths are unchanged (header read at plan preserved for correctness). Update test_raster_manifest path-only test to use emit_virtual=False (materialized) since virtual path-only rows now go lazy.

Adds generate_cog_multiwindow_corpus() + --cog-multiwindow CLI mode. Writes K large COGs (driver=COG, internally tiled) + a manifest JSON of M (path,window) rows per COG. Manifest format matches _read_manifest_rows so the corpus feeds run_virtual_tile_pixel_read in the FILE-capability bench.

Add performance guidance for large tile counts (manifest/tilesTable/fewer larger COGs). Add FILE byte-range callout to large-rasters, virtual-tiles, and benchmarking pages for the large-COG multi-window scenario. Add stock-Spark-reader underscore-file filter note to raster.mdx (DOC-ONLY).

…ge/_uf_combineavg/_uf_mapalgebra + _merge_agg_file/_combineavg_agg_file/_frombands_agg_file

…nvariance Add run_gpkg_chunksize_sweep to bench/readers.py (Task 7). Sweeps chunkSize (default 1k/10k/100k) by calling run_gpkg_file_read once per value, returning one ResultRow per chunkSize. Each row records chunk_size + input_partitions so the fanout-invariance claim is checkable at a glance: partition count is file-count bound and invariant to chunkSize. Test in test_gpkg_chunksize_sweep.py asserts both the one-row-per-chunkSize contract and the fanout-invariance property (len({input_partitions}) <= 1 across ok rows). Also fix test_layout_sweep.py isort/black cycling conflict: move the mid-function comment above the rasterio imports to after all imports, eliminating the blank line that caused isort to fail while black restored it. Docker gbx:lint:python --check now passes clean (513/513 files unchanged). Co-authored-by: Isaac

…rnal) Add a trailing `file_mode="na"` kwarg to `run_grouped_file`; thread it into `_ok_row`/`_err_row` (new `file_mode` keyword param, default "na") so every emitted ResultRow carries the storage class. Default "na" preserves all existing callers unchanged. Tested by new test_grouped_file_label.py (6 tests: 4 unit _ok_row/_err_row + 2 end-to-end via corpus). Co-authored-by: Isaac

…ot grouped-read) Implements run_layout_scan_comparison in bench/readers.py (Task 9 of Phase-2 named-format FILE access bench). Per-layout: read_file_table → time df.count() (category "layout-scan"); optionally time df.repartition(n,"path").count() (category "layout-shuffle-input"). No grouped_tile_map invocation — the grouped read self-amortizes regardless of layout; this leg measures the layout's real benefit on scan/pruning/shuffle axes. TDD: 5/5 new tests pass, 419/419 bench suite clean. Co-authored-by: Isaac

…ls + flags Adds 4 new Phase-2 benchmark legs to build_bench_notebook (bench/cluster.py): _CELL_FILE_MATRIX: sweeps file_mode ∈ {fuse,external,managed} for GeoTIFF + GeoPackage reads. Isolation: each mode builds a fresh DataFrame with no shared warm state (per-mode source path + loop iteration). GeoTIFF reuses {CORPUS}/rows (10k tiles, exceeds cluster slots). GeoPackage stages {GPKG_CORPUS}/copies_80/ via stage_gpkg_bench_corpus (80-copy bracket ≈ cluster slots). external/managed yield na_by_design on FUSE-only tiers; managed without FILE_FILESPACE falls back to Volume path (access='managed'+location → ValueError → na_by_design). _CELL_GPKG_CHUNKSIZE: sweeps chunkSize (1k/10k/100k) in fuse mode over the staged 80-copy GeoPackage corpus. Records input_partitions per row; cell checks fanout-invariance (partition count must be stable across chunk sizes). _CELL_LAYOUT_SWEEP: sweeps GeoTIFF (tile_df from raster_gbx) + GeoPackage write layouts (order/cluster/plain). Each layout writes to its OWN isolated target_prefix_{layout} so no layout inherits a prior write's on-disk grouping. 'cluster' layout runs OPTIMIZE after write. FILE_FILESPACE selects external FILE tables vs fuse Volume paths. _CELL_LAYOUT_SCAN: reads FILE tables written by the sweep via read_file_table, times df.count() (layout-scan) + repartition shuffle input per layout. Skips cleanly when FILE_FILESPACE is not set. Each cell: light-only, Connect-safe, _sink'd immediately (serialized). All 4 flags default OFF (existing build_bench_notebook callers unaffected, confirmed by 419+ passing prior tests). New flags: file_matrix/file_matrix_only, gpkg_chunksize/gpkg_chunksize_only, layout_sweep/layout_sweep_only, layout_scan/layout_scan_only, file_filespace, gpkg_corpus. Corresponding preamble config vars: BENCHMARK_FILE_MATRIX, FILE_MATRIX_ONLY, etc. CLI: gbx-bench-cluster.sh + .md + push_and_run_bench_on_cluster.py updated with --file-matrix, --gpkg-chunksize, --layout-sweep, --layout-scan (+{-only}), --file-filespace, --gpkg-corpus. Each *_only sets modes=spark-path and is included in the _only_run pool-size-check bypass. TDD: test_cluster_notebook_file_cells.py (26 tests) verifies flag→cell emission, *_only suppression of fn-bench sections, default-OFF absence, isolation patterns (per-mode loop/source), fanout-invariance check, tables_by_layout usage, and all-four-cells-together. 445 total bench tests pass, lint clean. Co-authored-by: Isaac

…ayouts FIX 1 (readers.py): the _shuffle closure in run_layout_scan_comparison was repartitioning by bare "path" but read_file_table returns a DataFrame with a 'tile' struct column — there is no top-level 'path' column. On-cluster this silently random-repartitions (all layouts show equal shuffle time, defeating the comparison) or raises AnalysisException. Fixed to `_F.col("tile.path")`. Two new tests lock this: a source-inspection guard asserting "tile.path" in the closure and NOT the bare string, and an execution test asserting the closure returns rows without raising. FIX 2 (cluster.py): _CELL_LAYOUT_SCAN scanned only bench_layout_gtiff_* tables while _CELL_LAYOUT_SWEEP also writes bench_layout_gpkg_* tables. The scan now calls run_layout_scan_comparison for both formats via separate _tables_by_layout_gtiff / _tables_by_layout_gpkg dicts. New notebook test asserts all six table paths (3 layouts × 2 formats) appear in the cell. 448 passed (bench/ suite); lint clean. Co-authored-by: Isaac

Adds the GBX Common Functions reference page that catalogs the shared file-access base (file_gbx) — generic session-ful functions (gbx_file_read, gbx_file_write), format-specific decoders (rst_fromfile, vector_file_read), and the session-free core floor (list_local_files, to_local_path). Documents the FUSE-vs-FILE and generic-vs-format-specific boundaries with the no-gating rule. - docs/docs/common-functions.mdx: new page wired into Readers & Writers sidebar - docs/sidebars.js: 'common-functions' added as first item in the category - resources/images/generators/gbx-common-functions.py: house-style SVG generator - resources/images/diagrams/rasterx/gbx-common-functions.{svg,png}: generated glyph - docs/tests/python/readers/common_functions_examples.py: doc-test source - docs/tests/python/readers/test_common_functions_examples.py: passing doc-test - docs/docs/readers/raster.mdx + vector.mdx + writers/vector.mdx: used-vs-excluded matrices - docs/docs/api/tile-structure.mdx: cross-ref from rst_fromfile/rst_fromcontent section The compose example grounds gbx_file_read on its real [path, size, file] return shape — NOT content/bytes. rst_fromfile(files["path"]) is the canonical raster decode pattern. Two doc-tests pass: test_gbx_file_read_then_decode (Spark + sample data) and test_list_local_files_example (tmp_path, no session needed). Docs build succeeds; voice gate prints clean. Co-authored-by: Isaac

The wheel version is a fixed 0.5.0 across rebuilds, so a plain '%pip install "geobrix[light-dbr19] @ file://<wheel>"' treats an already-installed 0.5.0 (from a prior run on a warm cluster) as satisfied and silently runs STALE geobrix code -- a freshly added bench helper is then missing at runtime (AttributeError). The serverless branch already force-reinstalls; the classic branch did not. Stays a single %pip cell (one kernel restart). Test locks it. Co-authored-by: Isaac

…+ SVG newline Three review-requested fixes: - common-functions.mdx: rewrite "Session-free core" intro to be accurate — enumerate_files accepts spark= and issues Spark SQL on FILE-capable runtimes (not FUSE-only); minimum viable mode is FUSE, graceful degradation when no session. Restrict "every DataSource reader consumes" to list_local_files only. Update the enumerate_files table row to match. - readers/raster.mdx: fix the enumerate_files matrix row — change "Used by the function layer" (non-binary, wrong) to "Not in the DataSource" with the correct explanation that matches vector.mdx. - gbx-common-functions.py generator: add trailing newline to SVG write; regenerate gbx-common-functions.svg so the file ends with \n. Co-authored-by: Isaac

…writer pages Adds the "Common functions: used vs excluded" 3-row table to every reader and writer page not already covered in Task 9. Raster readers carry the enumerate_files + gbx_file_read rows (FUSE-only DataSource, function-layer for FILE); vector readers carry gbx_file_read/vector_file_read + vector_file_write; raster writers carry gbx_file_write/gbx_file_read; vector writers name the exact format-specific DataSource (`geojson_gbx`, `gpkg_gbx`, etc.) and route FILE via vector_file_write/vector_file_read. Cross-link to ../common-functions. Build verified (MDX valid, links resolve). Voice grep clean (no wave-number leaks). Co-authored-by: Isaac

…ile_read pyogrio.read_dataframe requires geopandas, which the light tier does not ship. On-cluster (DBR19, geobrix[light-dbr19]) vector_file_read failed with ImportError: geopandas is required to use pyogrio.read_dataframe(). Refactor the _map closure to use pyogrio.read_arrow, which returns (meta, arrow_table) with the geometry column as raw WKB bytes — no geopandas needed. The geometry column name comes from meta["geometry_name"], falling back to "wkb_geometry" for GeoJSON drivers that return an empty geometry_name. WKT mode (as_wkb=False) uses shapely.from_wkb, which is a light-tier dep. Output contract (source, geometry) is unchanged. Adds static + behavioral tests: static asserts read_dataframe is absent from the source; behavioral tests use pyogrio.write_arrow fixtures (no geopandas) to verify WKB and WKT mode on .gpkg files. Co-authored-by: Isaac

…not crash On-cluster, run_gpkg_file_read(file_mode="managed", source=<Volume dir>) crashed the Spark job with PARSE_SYNTAX_ERROR: read_file_table issued SHOW TBLPROPERTIES with a bare path instead of a table name. Add an early guard in the else/managed branch: if source starts with "/" or "dbfs:" (a path, not a table name), return a clean na_by_design ResultRow immediately. A Volume path can never yield a MANAGED FILE reference — that is minted on write. This mirrors the behaviour of vector_file_read (raises ValueError for access=managed on any location source) and the raster gbx_file_read gate. Table-name sources (no leading slash) continue to read_file_table unchanged. Adds test: test_gpkg_file_read_managed_directory_path_is_na_by_design — previously failed with ParseException; now returns status="na_by_design". Also includes black reformat of bench/cluster.py (pre-existing drift). Co-authored-by: Isaac

The FILE-write layout sweep read the full 10k-tile pool and wrote it across 3 layouts x (warmup+measured) -- ~60k large-blob writes + OPTIMIZE -- which dominated the leg wall-clock (~50 min) for no added signal beyond demonstrating write + the layout effect. Cap the write source to 1000 tiles (still >> cluster slots, so writer fanout stays saturated) and repartition(160) so .limit() cannot collapse the write to a single partition. The GeoPackage sweep already used a single 100k-feature .gpkg. Read/fanout legs keep the full 10k pool. Test locks the cap. Co-authored-by: Isaac

…he closure The classic %pip cell used a full '--force-reinstall "geobrix[light-dbr19] @ file://<whl>"', which reinstalled the ENTIRE dependency closure every run: slow (re-fetches DBR-provided numpy/pandas/rasterio from the mirror) and it re-resolved transitive deps -- bumping rio-tiler's unpinned cachetools past pyiceberg's <7 cap (dependency-conflict warning). Switch to the two-step the serverless path already uses: (1) pip install --force-reinstall --no-deps <whl> refreshes ONLY the geobrix code (fast, fixed-0.5.0 stale-code problem solved, nothing else touched); (2) a NON-forced deps install adds only missing deps and leaves satisfied ones (incl DBR cachetools<7) alone. One restartPython(). Co-authored-by: Isaac

The 'Runtime support vs. benchmark environment' note omitted DBR 19 LTS (now supported for the lightweight tier per the installation page) and claimed the per-result stamps were all DBR 17.3 LTS, but they span 17.3 LTS and 18.x. Reworded to list all supported runtimes, state the actual stamp span, and note FILE-capability benchmarks run on a FILE-enabled runtime. Co-authored-by: Isaac

…des' The page had two '## Results' headings (#results and #results-1), which is confusing. The first is actually the measurement-methodology section (pure-core vs spark-path models + the consistency check), not results. Rename it 'Measurement modes' and cross-link to the real Results tables. Side effect (a fix): the 'At a glance' link to #results now resolves to the result tables instead of this methodology section. Co-authored-by: Isaac

_enumerate_fuse was calling os.stat(fp) on every file even when the caller (list_local_files) only needed the paths, not sizes. Over a FUSE Volume mount with 10k files this cost ~165 s per listing invocation (measured: raster_gbx.load(dir).limit(1).collect() = 165 s vs single rasterio.open + read = 0.7 s). Fix: - Add need_size: bool = True to _enumerate_fuse. When False, skip all per-file stat syscalls (size = None). Only list_local_files opts out; enumerate_files keeps need_size=True (default) for sized scheduling. - Switch os.walk + os.listdir + os.stat to os.scandir via a recursive _walk() inner function. DirEntry.is_file() / .is_dir() use the readdir-cached type (no extra syscall). When need_size=True, entry.stat().st_size is wrapped in _retry_transient for FUSE eventual-consistency tolerance. - list_local_files passes need_size=False, eliminating all 10k stats from the reader's plan-time listing path. Tests: 8 new tests in test_file_gbx_enum_core.py (TDD — 5 failed before fix, all 13 pass after). Full ds/ suite: 629 passed, 3 pre-existing failures unchanged. Co-authored-by: Isaac

…tive .limit() .limit(N) does NOT bound a raster_gbx read: Spark 4's Python DataSource API has no limit pushdown, so .load(10k-dir).limit(1000) still opens EVERY tile in the directory on each action (read() opens each virtual tile for metadata). The write leg has ~5 actions (measure-parallelism, count, probe-write, measured-write, read-back), so an unbounded, un-cached source re-scanned all 10k tiles ~5x per layout (~88s each) -- dominating wall-clock AND inflating the write timing (invalid numbers). Fix: .cache() the source so the actions share one metadata read, and size the SOURCE DIR to the write demo (point GBX_BENCH_CORPUS at a ~1k-tile pool) rather than a limit that cannot bound the read. Co-authored-by: Isaac

Every silent bench loop now emits a flush=True start line before the measured work and an end line after (status, rows, elapsed s), so an operator watching a running cluster Job sees forward motion rather than minutes of silence. Prefixes added: [gtiff-read] run_gtiff_file_read (before/after time_iters) [gtiff-write] run_gtiff_file_write (before/after time_iters) [gpkg-read] run_gpkg_file_read (before/after time_iters) [gpkg-write] run_gpkg_file_write (before/after time_iters, both branches) [chunksize] run_gpkg_chunksize_sweep per chunk_size [layout-scan] run_layout_scan_comparison per layout [write-sweep] run_file_write_layout_sweep per layout (N of total) [grouped] run_grouped_file per mode (+ existing _print_progress per fn) Cell-level banners added to the notebook cell strings in cluster.py: "=== FILE-access matrix starting ===" + [read-matrix] per mode "=== GPKG chunkSize sweep starting ===" "=== FILE write layout sweep starting ===" "=== layout scan comparison starting ===" "=== grouped FILE-amortization starting ===" All print calls use flush=True (notebook Job stdout is buffered without it). No timing logic, ResultRow fields, or control flow changed — prints are purely additive. Test: test_progress_prints.py (13 tests) asserts the expected prefixes appear in captured stdout for GPKG fuse-mode legs exercisable on local[2], and uses source inspection for the raster legs and cell strings. All 462 pre-existing bench tests still pass; 1 pre-existing failure in test_cluster.py::test_build_bench_notebook_cells is unrelated (pip install format mismatch, predates this change). Also include a user-authored docs/docs/readers/overview.mdx note about DataSource limit/filter pushdown absence. Co-authored-by: Isaac

…anaged _CELL_LAYOUT_SWEEP now loops over write modes instead of picking a single mode at build time. When FILE_FILESPACE is provisioned three modes run: fuse → layouts (order,cluster,plain), Volume path targets external → layout (order,) only, FILE EXTERNAL table targets managed → layout (order,) only, FILE MANAGED table targets Total: 5 write legs per format (gtiff+gpkg), 10 legs per run. Per-leg isolation: each (mode,layout,format) writes its own distinct target — fuse uses OUT + "/bench_layout_<fmt>", external/managed use catalog.schema table names derived from TABLE at runtime (_TABLE_SCHEMA). No two legs share a path or table, so no layout benefits from a prior write's on-disk grouping. Corrects the pre-existing bug where external mode was passed a Volume path instead of a table name as its target (the write function distinguishes by mode: fuse → .save(path), external/managed → write_file_table(table_name)). Adds 10 new tests: - 8 source-inspection tests in test_cluster_notebook_file_cells.py verifying mode loop structure, per-mode targets, _TABLE_SCHEMA derivation, layouts tuple per mode, filespace routing, and progress print format. - 2 local[2] tests in test_layout_sweep.py verifying managed/external modes return na_by_design (not a crash) when FILE tables are unavailable. Co-authored-by: Isaac

Two corrections to the write-mode sweep (follow-up to 7d44ee8): 1. External mode hosts the 3-layout sweep (order/cluster/plain). The fuse DataSource writer ignores the layout arg so sweeping fuse × 3 layouts is redundant and writes Volume dirs the scan cannot read via read_file_table. External FILE tables carry the meaningful layout bit (Delta order / cluster+OPTIMIZE / plain), making them the correct host for the layout-dimension leg. fuse and managed remain order-only (mode-comparison baseline). Net: fuse×1 + external×3 + managed×1 = 5 legs per format, 10 total. 2. _CELL_LAYOUT_SCAN now references the correct external tables: _scan_schema.bench_layout_gtiff_external_order/cluster/plain _scan_schema.bench_layout_gpkg_external_order/cluster/plain file_mode changed from "managed" to "external" (matching the tables actually written). _scan_schema derived same way as _TABLE_SCHEMA in the sweep cell. The if-not-FILE_FILESPACE skip guard is preserved. Tests updated: sweep tests now assert external has 3 layouts; scan tests assert the external_order/cluster/plain table names and file_mode="external". 473 passed, 1 pre-existing failure (test_build_bench_notebook_cells). Co-authored-by: Isaac

…ree listing perf The raster reader's directory enumeration had a regression (_enumerate_fuse did os.stat per file, ~157 s at 10k files). After the fix (stat-free os.scandir/glob), partitions() plans 10k files in ~1.5 s — on par with the original os.walk baseline. raster.mdx: replace the alarmist "Loading many small files" tip (which led with a ~37× table and "most effective remedy" framing) with a one-line pointer heading that keeps the #loading-many-small-files anchor for backward compat. Move the manifest/tilesTable content into a new "## Advanced: pre-computed tile inputs" section near the end of the page. New lead: default listing is fast (~1.5 s at 10k); manifest/tilesTable is optional for pre-computed windows or very large tile counts. benchmarking.mdx: update "Reader plan-time listing" — default row label now "Default directory listing (stat-free)", value updated to ~1.5 s, narrative lead changed from "dominant planning cost" to "fast — stat-free walk", tip reframed as optional (not urgent). ~37× ratio retained as accurate context. Co-authored-by: Isaac

…nstall test_build_bench_notebook_cells asserted the old single-%pip URL form ('geobrix[light-dbr19] @ file://<whl>'), which fix 09e0649 replaced with the two-step subprocess install (force-reinstall --no-deps the wheel, then a non-forced deps install). Assert the two-step markers instead (wheel path + --force-reinstall --no-deps + geobrix[light-dbr19]). This was a missed test in that fix -- test_cluster_notebook_file_cells.py was updated but this one in test_cluster.py was not. Co-authored-by: Isaac

…trip read Two fixes for the FILE-write bench matrix: 1. Filespace for external: vector_file_write() requires a filespace (staging dir) for external FILE writes, unlike raster which ignores it. Pass FILE_FILESPACE for both "external" and "managed" modes. 2. Readback validation: a vector FILE write stores the whole .gpkg as ONE FILE reference (not per-feature rows like raster). Validate the round-trip: read the FILE table, then read features back via vector_file_read(), and compare feature count to source. Leave raster readback unchanged. Both fixes are light-tier only, Connect-safe, and preserve na_by_design on FUSE-only tiers. Co-authored-by: Isaac

- Bottom padding: taller GENERIC/FORMAT-SPECIFIC bands + content shifted up (smaller top offset) so the WRITE box, callout, and write-side cards no longer spill past the band edge. - Arrows: end each vertical connector in the clear gap ABOVE the lower band's pill+descriptor row so no arrow crosses the descriptor text (fixes the middle box2->box3 arrow cutting 'enumerate_files'); box1->box2 left arrow moved onto the rst_fromfile column. - gbx_file_write: 'auto' moved next to 'file_mode='. - Subtitle shortened so it no longer runs under the top-right FUSE legend. Co-authored-by: Isaac

The overview had two adjacent :::tip blocks (virtual-tiles-by-default and scales-beyond-a-single-node). Combine into one shorter tip covering both: distributed DataSource V2 readers + virtual tiles by default. Co-authored-by: Isaac

- benchmarking 'At a glance' → #virtual-tile-read-performance retargeted to ../api/performance#virtual-tile-read-performance (the section lives on the performance page). - benchmarking Pure-core heading given an explicit {#pure-core} id (its auto-id included the '(local, 1024²)' suffix, so #pure-core 404'd). - raster-sampling Interpolation heading given {#interpolation-bilinear-and-cubic}. Not introduced by recent work (At-a-glance links to headings whose auto-ids differ); surfaced by the docs build's broken-anchor check. Co-authored-by: Isaac

mjohns-databricks temporarily deployed to runtime August 13, 2026 14:35 — with GitHub Actions Inactive

mjohns-databricks had a problem deploying to runtime August 13, 2026 15:02 — with GitHub Actions Failure

mjohns-databricks temporarily deployed to runtime August 13, 2026 20:45 — with GitHub Actions Inactive

mjohns-databricks temporarily deployed to runtime August 13, 2026 20:46 — with GitHub Actions Inactive

mjohns-databricks temporarily deployed to runtime August 13, 2026 21:12 — with GitHub Actions Inactive

mjohns-databricks had a problem deploying to runtime August 13, 2026 21:21 — with GitHub Actions Failure

mjohns-databricks temporarily deployed to runtime August 13, 2026 22:26 — with GitHub Actions Inactive

mjohns-databricks temporarily deployed to runtime August 13, 2026 22:54 — with GitHub Actions Inactive

mjohns-databricks temporarily deployed to runtime August 13, 2026 22:55 — with GitHub Actions Inactive

mjohns-databricks added 13 commits August 14, 2026 09:36

feat(pyrx): FILE-aware _uf_avg/min/max/median/pixelcount 2-arg UDFs +…

527ef05

… updated bindings

feat(pyrx): FILE-aware _uf_type/_uf_getnodata header-accessor singlet…

5086320

…ons + bindings

feat(pyrx): FILE-aware histogram + coord-fn UDFs; Group 1 complete

b5ad37a

feat(ds): add _read_manifest_rows/_partitions_from_tile_rows/_resolve…

255c667

…d_budget helpers Approach 1 foundations: helpers that will let partitions() skip os.walk + per-file rasterio.open when a manifest or tilesTable is provided.

feat(pyrx): Group 3 FILE-aware array/agg UDFs — _uf_frombands/_uf_mer…

cb93bc6

…ge/_uf_combineavg/_uf_mapalgebra + _merge_agg_file/_combineavg_agg_file/_frombands_agg_file

mjohns-databricks added 26 commits August 21, 2026 08:16

mjohns-databricks had a problem deploying to runtime August 21, 2026 21:01 — with GitHub Actions Failure

mjohns-databricks deployed to runtime August 21, 2026 21:01 — with GitHub Actions Active

mjohns-databricks had a problem deploying to runtime August 21, 2026 21:01 — with GitHub Actions Failure

Navigation Menu

Uh oh!

release: GeoBrix 0.5.0 — virtual tiles, v2 tile struct, CRS families - #72

release: GeoBrix 0.5.0 — virtual tiles, v2 tile struct, CRS families#72
mjohns-databricks wants to merge 771 commits into
mainfrom
beta/0.5.0

mjohns-databricks commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

mjohns-databricks commented Aug 13, 2026

GeoBrix 0.5.0 — release PR (beta/0.5.0 → main)

Highlights

Pre-merge gate (requested)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

GeoBrix 0.5.0 — release PR (`beta/0.5.0` → `main`)