Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
771 commits
Select commit Hold shift + click to select a range
527ef05
feat(pyrx): FILE-aware _uf_avg/min/max/median/pixelcount 2-arg UDFs +…
mjohns-databricks Aug 14, 2026
5086320
feat(pyrx): FILE-aware _uf_type/_uf_getnodata header-accessor singlet…
mjohns-databricks Aug 14, 2026
b5ad37a
feat(pyrx): FILE-aware histogram + coord-fn UDFs; Group 1 complete
mjohns-databricks Aug 14, 2026
a3adf2a
docs(plan): add COG multi-window bench + underscore-file doc to Thread B
mjohns-databricks Aug 14, 2026
666ca77
fix(rasterx): stop naming heavy-writer tiles with a leading underscore
mjohns-databricks Aug 14, 2026
fd35d8c
test(rasterx): regression for signed-hash leading-underscore filename…
mjohns-databricks Aug 14, 2026
255c667
feat(ds): add _read_manifest_rows/_partitions_from_tile_rows/_resolve…
mjohns-databricks Aug 14, 2026
09c36f9
feat(ds): wire manifest/tilesTable options into RasterGbxReader
mjohns-databricks Aug 14, 2026
71204de
feat(pyrx): Group 2 FILE-aware UDFs — tile-producing ops
mjohns-databricks Aug 14, 2026
c33b4d8
feat(ds): lazy window planning for virtual-tile passthrough (Approach 3)
mjohns-databricks Aug 14, 2026
e6182ea
feat(bench): cog-multiwindow corpus generator for FILE byte-range bench
mjohns-databricks Aug 14, 2026
3568b48
docs(readers): many-small-files note + COG multi-window FILE callout
mjohns-databricks Aug 14, 2026
cb93bc6
feat(pyrx): Group 3 FILE-aware array/agg UDFs — _uf_frombands/_uf_mer…
mjohns-databricks Aug 14, 2026
afc80db
fix(bench): post-final-review fixes F1-F6 — manifest param, doc ancho…
mjohns-databricks Aug 14, 2026
c2f5d1a
fix(pyrx): final-review fix wave A1-A7 for FILE-extend light-tier
mjohns-databricks Aug 14, 2026
3448dae
feat(bench): parallel row-pool writes via --jobs N (ThreadPoolExecutor)
mjohns-databricks Aug 14, 2026
dc12232
fix(bench): write COG via local temp file to avoid FUSE seek error
mjohns-databricks Aug 14, 2026
c3c1da0
feat(bench): --disable-file FILE-off A/B leg + plan_s results field
mjohns-databricks Aug 14, 2026
56e715c
perf(pyrx): eliminate double ZSTD encode in _mapalgebra_bytes for vir…
mjohns-databricks Aug 14, 2026
a97af2b
fix(bench): tile_array/agg inputs use path-based virtual tiles
mjohns-databricks Aug 14, 2026
413fc53
fix(bench): wire real FILE path + absolute manifest paths in virtual-…
mjohns-databricks Aug 14, 2026
9402a12
docs(spec): design for DBR-runtime-matched light-tier pip extras
mjohns-databricks Aug 14, 2026
75fb8f2
docs(spec): lead light-extras spec with Serverless-default positioning
mjohns-databricks Aug 14, 2026
89475c5
docs(spec): mandate Serverless-default framing on the installation page
mjohns-databricks Aug 14, 2026
77362a2
docs(spec): split Serverless/DBR mashups into separate statements
mjohns-databricks Aug 14, 2026
35851ce
docs(spec): add light_all / light_dbrNN_all umbrella extras (spec §4d)
mjohns-databricks Aug 14, 2026
9c9a106
docs(plan): light-tier DBR-runtime-matched extras (Option B)
mjohns-databricks Aug 14, 2026
78ef6de
plan(light-extras): ground-check fixes + _all umbrella tasks + review…
mjohns-databricks Aug 14, 2026
867c1e8
chore(packaging): Option B extras — light-base + light_dbr19 variant
mjohns-databricks Aug 14, 2026
454ba9e
chore(ci): rename pyrx-ci lock → light-ci; add light-dbr19-ci lock
mjohns-databricks Aug 14, 2026
20be0a6
chore(ci): fix abbreviated pyrx-ci ref + dbr19 lock runtime note
mjohns-databricks Aug 14, 2026
8bd1221
chore(packaging): add light_all + light_dbr19_all umbrella extras + l…
mjohns-databricks Aug 14, 2026
083e610
docs(installation): lead with Serverless-default framing + DBR-19 com…
mjohns-databricks Aug 14, 2026
7f1a003
fix(ds): lazy pmtiles import so raster paths don't require it
mjohns-databricks Aug 14, 2026
6b20b3a
fix(bench): route spark-path legs through SPARK_WARMUP/SPARK_MEASURED…
mjohns-databricks Aug 14, 2026
66fe835
fix(bench): dogfood bench installs geobrix[light-dbr19] (retire slim-…
mjohns-databricks Aug 14, 2026
18e60d6
docs(virtual-tiles): link FILE-type + file-UDFs docs, honest framing
mjohns-databricks Aug 14, 2026
60b4cbb
feat(bench): single large-COG + 2D-window corpus mode for FILE byte-r…
mjohns-databricks Aug 14, 2026
da33030
docs(bench): publish 37x manifest-listing result + one-iter FILE how-to
mjohns-databricks Aug 14, 2026
4af430b
docs(readers): clarify tilesTable is a tile index (paths), not tiles
mjohns-databricks Aug 14, 2026
980d8c0
feat(bench): --existing-cluster-id to attach runs to a warm cluster
mjohns-databricks Aug 15, 2026
92c1956
docs(file): correct FILE framing — measured slower than FUSE, not a p…
mjohns-databricks Aug 15, 2026
7cef885
docs(bench): add 384px + 2.5GB FILE-vs-FUSE rows (FUSE wins all sizes)
mjohns-databricks Aug 15, 2026
c894735
docs(bench): add real NAIP + Sentinel-2 FILE rows + block-size nuance
mjohns-databricks Aug 15, 2026
4eada85
feat(bench): memory-efficient streamed large-raster generator (COG + …
mjohns-databricks Aug 15, 2026
a022ef4
fix(pyrx): probe-then-stage _stage_local_if_needed (avoid uncondition…
mjohns-databricks Aug 15, 2026
04d340c
test(pyrx): reset _FILE_SUPPORT_CACHE per test to fix order-dependent…
mjohns-databricks Aug 15, 2026
18db3e1
fix(sample-data): minimal-bundle generator guarantees valid pixels
mjohns-databricks Aug 16, 2026
800f777
fix(ds): bound whole-RAM fallback in vector writer via lazy streaming
mjohns-databricks Aug 16, 2026
04eb7b6
fix(ds): pmtiles sharded write uses save path as root, not nested
mjohns-databricks Aug 16, 2026
176e1b6
test(docs): writers gdal tests inherit JAR-attached conftest spark
mjohns-databricks Aug 16, 2026
3a54738
feat(rasterx): log when ReTile yields zero non-empty tiles
mjohns-databricks Aug 16, 2026
935f43b
docs(intro): add Serverless env column to runtimes matrix (match README)
mjohns-databricks Aug 16, 2026
a6eebd0
test(bench): sync test_cluster with current notebook builder + schema
mjohns-databricks Aug 16, 2026
34112f7
fix(sample-data): copy SRTM tiles into minimal bundle (fix flat DEM)
mjohns-databricks Aug 16, 2026
392904f
docs: correct README function badges + point Readers&Writers links to…
mjohns-databricks Aug 16, 2026
e4e2a73
docs(plan): VectorX tabbed function-examples implementation plan
mjohns-databricks Aug 16, 2026
a6dbf35
docs(plan): VectorX tab + output consistency gate + representation co…
mjohns-databricks Aug 16, 2026
a646ab0
feat(docs/vectorx): T1 foundation — fixtures, conventions, generator …
mjohns-databricks Aug 16, 2026
6d36e97
docs(vectorx): tabbed examples for CRS family (st_crs / st_setcrs / s…
mjohns-databricks Aug 16, 2026
365f1ba
fix(docs): align CRS SQL examples with one-example-per-function rule
mjohns-databricks Aug 16, 2026
d5d92fd
docs(vectorx): tabbed examples for MVT family (st_asmvt / st_asmvt_py…
mjohns-databricks Aug 16, 2026
e5fc6e7
docs(perf): note the light-tier benefit of chaining functions in one …
mjohns-databricks Aug 16, 2026
ca699ba
fix(docs/vectorx): correct MVT SQL wrapper + light pyramid MVT verifi…
mjohns-databricks Aug 16, 2026
bbe6636
feat(gbx): add --filter (pytest -k) to test:python-docs for changed-f…
mjohns-databricks Aug 16, 2026
55cf0ab
docs(vectorx): tabbed 4-tab examples for TIN/elevation family (T4)
mjohns-databricks Aug 16, 2026
023a6f6
docs(vectorx): add conforming-mode :::note + align testFile for TIN tabs
mjohns-databricks Aug 16, 2026
c4dbb2f
docs(vectorx): 4-tab FunctionExamples for st_legacyaswkb (T5)
mjohns-databricks Aug 16, 2026
f085b57
docs(vectorx): alphabetize TIN functions within Triangulation section
mjohns-databricks Aug 16, 2026
a3955d9
docs(vectorx): final-review fix wave — 6 correctness/consistency nits
mjohns-databricks Aug 16, 2026
8c189b0
docs(vectorx): fix Setup fixture-source sentence (all views are inlin…
mjohns-databricks Aug 16, 2026
7221c3c
test(ds): update pmtiles naming tests to non-nested sharded output
mjohns-databricks Aug 16, 2026
16a1c19
docs(plan): GridX tabbed function-examples implementation plan
mjohns-databricks Aug 17, 2026
f2e3780
docs(gridx): per-function 4-tab examples for all 40 GridX functions
mjohns-databricks Aug 17, 2026
739ea40
docs(rasterx): tessellate SQL examples — heavy gen-in-SELECT + light …
mjohns-databricks Aug 17, 2026
6924c42
style(python): normalize black + isort formatting (lint drift)
mjohns-databricks Aug 17, 2026
33e5e7f
fix(lint): run gbx:lint:python --fix in Docker (py3.12) to match the …
mjohns-databricks Aug 17, 2026
f048fb2
fix(lint): resolve F821 for optional-dep type hints via TYPE_CHECKING…
mjohns-databricks Aug 17, 2026
3d5ec2e
fix(consistency): CRS-2/CRS-3/CON-4/CON-5 small-fixes sweep
mjohns-databricks Aug 17, 2026
47c9a47
docs(spec): FILE-for-raster reader/writer/grouped-execution design
mjohns-databricks Aug 17, 2026
e3cf0df
docs(plan): FILE-for-raster light-tier implementation plan
mjohns-databricks Aug 17, 2026
09b2f5d
docs(plan): byte-budgeted LRU + keep dogfood cluster warm (plan refin…
mjohns-databricks Aug 17, 2026
5750d96
feat(pyrx): add path_mode field to tile v2 (materialized/external/man…
mjohns-databricks Aug 17, 2026
91a1032
fix(pyrx): guard windowless virtual-tile reads with descriptive Value…
mjohns-databricks Aug 17, 2026
56fa494
feat(pyrx): self-describing FILE-table TBLPROPERTIES (writer-version …
mjohns-databricks Aug 17, 2026
64be720
feat(pyrx): read_file_table projects plain cols (Serverless-GC-safe)
mjohns-databricks Aug 17, 2026
ae07164
fix(pyrx): absent-field type safety + empty-col guard + shape anchor
mjohns-databricks Aug 17, 2026
40a8c8e
feat(pyrx): write_file_table (typed FILE column + FILE-aligned INSERT)
mjohns-databricks Aug 17, 2026
316572f
fix(pyrx): qualify tile.raster/tile.path in write_file_table INSERT
mjohns-databricks Aug 17, 2026
b118bea
feat(pyrx): align_partitions helper (hash+sort by tile.path)
mjohns-databricks Aug 17, 2026
6837f6b
feat(pyrx): per-partition byte-budgeted open-resource LRU
mjohns-databricks Aug 17, 2026
92205c8
feat(pyrx): partition-scoped mapInPandas grouped executor + LRU
mjohns-databricks Aug 17, 2026
06419d5
feat(pyrx): rst_memsize_grouped via grouped executor (proof of fast p…
mjohns-databricks Aug 17, 2026
84fbb1c
fix(pyrx): add virtual-tile equivalence proof + fix rst_memsize_group…
mjohns-databricks Aug 17, 2026
1a72631
fix(pyrx): dereference .dataType before .simpleString() in tile-struc…
mjohns-databricks Aug 17, 2026
e3b13af
feat(pyrx): wire FILE stream fast path in grouped_exec (Task 9b)
mjohns-databricks Aug 18, 2026
ff4e594
style(pyrx): black/isort normalize FILE-for-raster branch files
mjohns-databricks Aug 18, 2026
f8cbba1
fix(pyrx): add path_mode as 9th field in ds/raster.py _v2_tile_row
mjohns-databricks Aug 18, 2026
53e270c
fix(pyrx): final code-review fix wave for FILE-for-raster light tier
mjohns-databricks Aug 18, 2026
11afb51
fix(pyrx): gate FILE fast path on _file_ref column presence, not work…
mjohns-databricks Aug 18, 2026
fc533d1
fix(pyrx): coerce str column-name to Column before ["path"] in file_r…
mjohns-databricks Aug 18, 2026
40015ea
docs(file): FILE-for-raster increment-2 design spec
mjohns-databricks Aug 18, 2026
4e6de9d
docs(file): confirm open/FUSE memory behavior is governance-independent
mjohns-databricks Aug 18, 2026
61c9095
docs(file): FILE-for-raster increment-2 implementation plan
mjohns-databricks Aug 18, 2026
38661a6
fix(rasterx): add path_mode 9th field to heavy v2 tile schema
mjohns-databricks Aug 18, 2026
7acdc5d
feat(pyrx): reader-2b managed-uri resolution in read_file_table
mjohns-databricks Aug 18, 2026
be67c94
feat(pyrx): header-vs-pixel view contract in grouped_tile_map (M2)
mjohns-databricks Aug 18, 2026
57a2432
fix(pyrx): memory-aware LRU weigher + size-adaptive open + stage guard
mjohns-databricks Aug 18, 2026
8f3a37b
test(pyrx): add staged-temp eviction coverage for _make_opener closer
mjohns-databricks Aug 18, 2026
c504efc
feat(pyrx): rst_clip_grouped pixel op via grouped executor
mjohns-databricks Aug 18, 2026
c896bec
feat(pyrx): _grouped variants for the pixel-op family
mjohns-databricks Aug 18, 2026
d45d573
feat(pyrx): FILE-aware 1:n generator UDTFs (trailing file_ref=None)
mjohns-databricks Aug 18, 2026
b25c6ed
fix(pyrx): restore size_bytes for materialized tiles in MakeTiles UDTF
mjohns-databricks Aug 18, 2026
5b90e63
fix(pyrx): aggregators open FILE tiles as external (drop try_to_file …
mjohns-databricks Aug 18, 2026
d7e0e3e
refactor(pyrx): remove orphaned _merge/_combineavg_agg_file_udf + the…
mjohns-databricks Aug 18, 2026
6fe7db4
fix(pyrx): replace colliding out_col field in out_schema instead of a…
mjohns-databricks Aug 18, 2026
bfe0a50
fix(pyrx): broaden _describe_cols FILE type detection for DBR-19 qual…
mjohns-databricks Aug 18, 2026
b84d867
docs(file): virtual-tiles + FILE performance section and tile-structu…
mjohns-databricks Aug 18, 2026
5d5e822
docs(file): add RAM constraint + size-adaptive guidance to virtual-ti…
mjohns-databricks Aug 18, 2026
a8441c8
fix(pyrx): final-fix wave: C901, stream-close, weigher, 9-field, docs…
mjohns-databricks Aug 18, 2026
35758c3
docs(tile-structure): remove stale Field Details section; relocate ra…
mjohns-databricks Aug 18, 2026
a28104d
docs(diagram): add MANAGED FILE callouts to virtual-tiles lifecycle
mjohns-databricks Aug 18, 2026
d82f9f0
test(pyrx): migrate two stale _make_opener tests to _OpenerContext API
mjohns-databricks Aug 18, 2026
42b68b7
docs(rasterx): label FILE chips MANAGED | EXTERNAL, pad fuller
mjohns-databricks Aug 18, 2026
4e664dd
docs(plans): scrub personal email from committed plan docs
mjohns-databricks Aug 19, 2026
56dd6f4
test(ds): index tile metadata by name, not position
mjohns-databricks Aug 19, 2026
be732a8
docs(tests): realign SQL example output tables (D5)
mjohns-databricks Aug 19, 2026
6bff5ec
test(rasterx): add path_mode to heavy v2 parity field list
mjohns-databricks Aug 19, 2026
fc5ac21
docs(spec): v2 tile path_mode-after-path reorder design
mjohns-databricks Aug 19, 2026
7b87c08
chore: gitignore docs/superpowers internal planning tree
mjohns-databricks Aug 19, 2026
515cc33
chore: consolidate prompts, input, planning under .superpowers
mjohns-databricks Aug 19, 2026
cb12fa9
feat(rasterx): add V2Tile name-based row builder + accessors
mjohns-databricks Aug 19, 2026
950d962
refactor(rasterx): route tile serde through V2Tile by name
mjohns-databricks Aug 19, 2026
fb98050
test(rasterx): update stale 8-field tile assertions to 9
mjohns-databricks Aug 19, 2026
67fd757
refactor(rasterx): build tiles via V2Tile in constructors/readers
mjohns-databricks Aug 19, 2026
a82207a
refactor(rasterx): route tile aggregators through V2Tile
mjohns-databricks Aug 19, 2026
697cff6
refactor(rasterx): route generator/grid tiles through V2Tile
mjohns-databricks Aug 19, 2026
a3af75b
style(pyrx): black/isort normalize two test files (Docker CI drift)
mjohns-databricks Aug 19, 2026
9d5ff3e
feat(rasterx)!: move path_mode after path, metadata last
mjohns-databricks Aug 19, 2026
04e2e00
refactor(rasterx): drop dead 8-field v2 tile handling
mjohns-databricks Aug 19, 2026
789d4b5
docs(rasterx): tile-structure reflects path_mode-after-path order
mjohns-databricks Aug 19, 2026
8b23f0e
fix(pyrx): read_file_table struct follows V2_TILE_SCHEMA order
mjohns-databricks Aug 19, 2026
2addef4
test(rasterx): update stale tile-order assertions post-reorder
mjohns-databricks Aug 19, 2026
4d1fe3d
docs(rasterx): refresh Working-with-Tiles for v2 tile model
mjohns-databricks Aug 19, 2026
cab2bed
docs(rasterx): correct stale field-order comments post-reorder
mjohns-databricks Aug 19, 2026
e00c746
feat(bench): add Serverless submission (env-v6, max_retries=0)
mjohns-databricks Aug 19, 2026
6becee2
fix(bench): guard AQE config readback for serverless/Connect
mjohns-databricks Aug 19, 2026
6489d3f
fix(bench): make light spark-path Connect-safe for serverless
mjohns-databricks Aug 19, 2026
6b7fe64
test(bench): pin Connect-safe spark-path helpers (both branches)
mjohns-databricks Aug 19, 2026
d540c14
feat(bench): grouped FILE-amortization leg module (multiwindow, 3 modes)
mjohns-databricks Aug 19, 2026
186b3db
test(bench): local verify for grouped FILE-amortization leg
mjohns-databricks Aug 19, 2026
f24ccf6
feat(bench): wire grouped-file leg into cluster submission harness
mjohns-databricks Aug 19, 2026
83c3822
fix(bench): skip AQE-disable on Serverless/Connect instead of attempt…
mjohns-databricks Aug 19, 2026
2fa3448
docs(perf): hoist virtual-tile/FILE story + lead benchmarking with ta…
mjohns-databricks Aug 19, 2026
6ecb7d8
docs(bench): align intro with reorganized order (results-first)
mjohns-databricks Aug 19, 2026
d6f0f9f
docs(perf): frame stream/FUSE as auto-selected + document LRU auto-cache
mjohns-databricks Aug 19, 2026
69ffc3d
fix(pyrx): Connect-aware LRU/stream sizing so serverless tasks don't OOM
mjohns-databricks Aug 20, 2026
cae42e7
feat(pyrx): add metadata-key size helpers for tile size metadata
mjohns-databricks Aug 20, 2026
26019a3
feat(pyrx): grouped_exec consumer prefers metadata tile size
mjohns-databricks Aug 20, 2026
3bf83f1
feat(pyrx): raster reader stamps tile size metadata at emission
mjohns-databricks Aug 20, 2026
bc2ced8
style(pyrx): isort cleanup grouped_exec imports
mjohns-databricks Aug 20, 2026
1554d9b
style(pyrx): isort/black cleanup for metadata size tests
mjohns-databricks Aug 20, 2026
3737b2c
refactor(pyrx): rename tile_size metadata key to tile_byte_size
mjohns-databricks Aug 20, 2026
1ed68a9
feat(pyrx): stamp tile size metadata on function outputs (Phase 2)
mjohns-databricks Aug 20, 2026
58311a6
test(pyrx): fix two stale tests broken by tile-size metadata changes
mjohns-databricks Aug 20, 2026
3fc9e44
fix(docs): clean build-static-zip before cp so static-zip build doesn…
mjohns-databricks Aug 20, 2026
51d0d7f
feat(ds): file_gbx base skeleton + capability-tier + no-gating resolver
mjohns-databricks Aug 20, 2026
268aa3c
style(ds): correct resolve_access type hints (Task 1 review fixup)
mjohns-databricks Aug 20, 2026
e84b60a
feat(ds): file_gbx enumeration (read_files/list_files/os.walk) with _…
mjohns-databricks Aug 20, 2026
3b85e81
fix(ds): file_gbx enumeration skip parity + cross-tier test + SQL pat…
mjohns-databricks Aug 20, 2026
472cfcc
refactor(ds): move FILE read engine into file_gbx.open_for_read behin…
mjohns-databricks Aug 20, 2026
e88b1b5
fix(ds): complete Task 3 review fixes
mjohns-databricks Aug 20, 2026
f7ff19e
feat(ds): add file_gbx.open_for_write + ingest_files write committer
mjohns-databricks Aug 20, 2026
2b22536
fix(ds): file_gbx review fixes — filespace guard, fuse+cluster warn, …
mjohns-databricks Aug 20, 2026
d4987ad
refactor(ds): point grouped raster read path at file_gbx.open_for_read
mjohns-databricks Aug 20, 2026
23bb366
feat(ds): vector reader gains file_gbx FILE read + per-source open am…
mjohns-databricks Aug 20, 2026
a8a8919
fix(ds): address Task 6 review — uuid4 stage name, lock docstring, mo…
mjohns-databricks Aug 20, 2026
efcb774
feat(ds): vector writer native FILE write (MANAGED/EXTERNAL) via open…
mjohns-databricks Aug 20, 2026
dfecf5c
fix(ds): vector writer FILE gate: reject OpenFileGDB+zip=false; drop …
mjohns-databricks Aug 20, 2026
0fbe93c
feat(ds): reader layout conventions — path-ordered emit + repartition…
mjohns-databricks Aug 20, 2026
494436e
fix(ds): remove inert numPartitions option; add explicit sort + real …
mjohns-databricks Aug 20, 2026
2b84020
feat(ds): enumerate_files positive path_glob_filter/extensions select…
mjohns-databricks Aug 20, 2026
de4f5aa
fix(ds): widen LIKE→RLIKE routing to cover _ and % in path_glob_filter
mjohns-databricks Aug 20, 2026
17d13e3
docs: file_gbx explainer — readers-writers section, diagram, cross-re…
mjohns-databricks Aug 20, 2026
e8bc0c3
docs(file_gbx): replace mermaid with inline SVG diagram; add cross-re…
mjohns-databricks Aug 20, 2026
f43f15c
docs: file_gbx flow diagram in house glyph style; drop mermaid plugin
mjohns-databricks Aug 20, 2026
67e3ad2
docs: dedupe file_gbx cross-ref admonitions to one per reader/writer …
mjohns-databricks Aug 20, 2026
38b7e5d
docs: correct file_gbx no-gating param name and open_for_read framing
mjohns-databricks Aug 20, 2026
b2d10d7
test(bench): file_gbx read+write×mode QA harness
mjohns-databricks Aug 20, 2026
bb2cf3c
fix(bench): file_gbx_qa syntax + signature fixes from on-cluster run
mjohns-databricks Aug 20, 2026
5782fdf
fix(bench): file_gbx_qa harness fixes for dogfood re-run
mjohns-databricks Aug 20, 2026
a027c69
docs: note rst_sample as the RS_VALUE / ArcGIS Sample equivalent
mjohns-databricks Aug 20, 2026
5435815
docs: note rst_resample family as the ArcGIS Resample equivalent
mjohns-databricks Aug 20, 2026
a09e887
docs: add RasterX Raster Sampling page (RS_VALUE workflows)
mjohns-databricks Aug 20, 2026
f2d87f6
docs: fix st_intersects usage in Raster Sampling page
mjohns-databricks Aug 20, 2026
4855505
fix(ds): file_access_tier must treat PATH_NOT_FOUND as FILE-available…
mjohns-databricks Aug 20, 2026
26a416d
fix(ds): driver-side vector capability probe, read_files _metadata.fi…
mjohns-databricks Aug 20, 2026
7edfe79
style(bench): clear flake8 in file_gbx_qa; drop stale test patch
mjohns-databricks Aug 20, 2026
a1f4c39
fix(ds): None-safe raster sort, fuse-CTAS path guard, vector reader v…
mjohns-databricks Aug 20, 2026
5e6052f
feat(file_gbx): shared _accept_basename + session-free list_local_fil…
mjohns-databricks Aug 20, 2026
df5b7eb
feat(file_gbx): generic session-ful gbx_file_read (location + table)
mjohns-databricks Aug 20, 2026
9e2f3db
fix(file_gbx): route list_files tier to binaryFile; strengthen fuse-t…
mjohns-databricks Aug 20, 2026
663e184
feat(file_gbx): generic session-ful gbx_file_write composing open_for…
mjohns-databricks Aug 20, 2026
aa95b38
feat(ds): raster/vector readers enumerate via file_gbx core; drop vec…
mjohns-databricks Aug 20, 2026
4af1c92
fix(ds): prune hidden dirs in _enumerate_fuse; replace resolve_local_…
mjohns-databricks Aug 20, 2026
70def9a
refactor(pyrx): write_file_table composes file_gbx._validate_layout; …
mjohns-databricks Aug 20, 2026
8e45fd2
test(pyrx): lock write_file_table layout I1 invariant before DROP
mjohns-databricks Aug 20, 2026
98dcd2f
feat(pyvx): vector_file_read — function-layer FILE read (parity with …
mjohns-databricks Aug 20, 2026
3e64cfc
fix(pyvx): diagnosable FILE fallback + as_wkb=False coverage
mjohns-databricks Aug 20, 2026
2506e47
feat(pyvx): vector_file_write; vector DataSource writer is FUSE-only …
mjohns-databricks Aug 20, 2026
7c270d5
fix(pyvx): task-7 fix round 1 — external filespace/byte asserts, driv…
mjohns-databricks Aug 20, 2026
94c20e3
refactor(pyrx): rst_fromfile composes file_gbx to_local_path + stagin…
mjohns-databricks Aug 20, 2026
3045a43
fix(pyrx): rewrite gbx_file_read to return [path,size,file] reference…
mjohns-databricks Aug 20, 2026
65f8b9f
docs(pyrx): correct gbx_file_read FUSE-schema comment (no cross-tier …
mjohns-databricks Aug 20, 2026
bf98f2f
fix(file_gbx): replace LIKE '_%' wildcard with startswith in enumerat…
mjohns-databricks Aug 20, 2026
5637cee
fix(pyvx): vector_file_read(access="managed") obeys table-only MANAGE…
mjohns-databricks Aug 21, 2026
feb38ab
fix(ds): vector_gbx reader access managed/external -> clear error at …
mjohns-databricks Aug 21, 2026
60077e1
feat(bench): add file_mode/layout/parallelism/chunk_size to ResultRow…
mjohns-databricks Aug 21, 2026
374e1a1
feat(bench): Connect-safe measure_parallelism (partitions + slots)
mjohns-databricks Aug 21, 2026
01ca4f9
feat(bench): GeoTIFF FILE read+write legs (managed/external/fuse) + _…
mjohns-databricks Aug 21, 2026
3eabdb6
test(ds): lock UNRESOLVABLE_TABLE_VALUED_FUNCTION in _is_routine_unav…
mjohns-databricks Aug 21, 2026
78cad31
feat(bench): GeoPackage FILE read+write legs (managed/external/fuse)
mjohns-databricks Aug 21, 2026
4f4b4ac
feat(bench): FILE-write layout sweep (order/cluster+OPTIMIZE/plain)
mjohns-databricks Aug 21, 2026
87a3e7a
feat(bench): stage_gpkg_bench_corpus (few files x >=100k rows, swept …
mjohns-databricks Aug 21, 2026
777d00a
feat(bench): GeoPackage chunkSize × LRU amortization sweep + fanout-i…
mjohns-databricks Aug 21, 2026
b6218cf
feat(bench): label run_grouped_file rows with file_mode (managed/exte…
mjohns-databricks Aug 21, 2026
d4f9ea1
feat(bench): writer-layout effects leg (scan/pruning/shuffle-input, n…
mjohns-databricks Aug 21, 2026
70960c1
feat(bench): wire FILE matrix / chunkSize / layout sweep / payoff cel…
mjohns-databricks Aug 21, 2026
690fdb2
fix(bench): repartition by tile.path subfield; scan both gtiff+gpkg l…
mjohns-databricks Aug 21, 2026
cfd37bb
docs: GBX Common Functions page + glyph + per-reader/writer matrices
mjohns-databricks Aug 21, 2026
673beb8
fix(bench): force-reinstall geobrix in classic %pip cell
mjohns-databricks Aug 21, 2026
47a7b50
docs(common-functions): fix enumerate_files accuracy + raster matrix …
mjohns-databricks Aug 21, 2026
98bfd10
docs: Common-options used-vs-excluded matrix on all remaining reader/…
mjohns-databricks Aug 21, 2026
e827fc2
fix(pyvx): replace pyogrio.read_dataframe with read_arrow in vector_f…
mjohns-databricks Aug 21, 2026
fb05314
fix(bench): run_gpkg_file_read managed+dir path returns na_by_design …
mjohns-databricks Aug 21, 2026
80ae2dc
perf(bench): cap GeoTIFF write-sweep source to 1000 tiles
mjohns-databricks Aug 21, 2026
09e0649
fix(bench): classic install force-reinstalls geobrix CODE only, not t…
mjohns-databricks Aug 21, 2026
ae124cf
docs(benchmarking): correct runtime-support note
mjohns-databricks Aug 21, 2026
f607b55
docs(benchmarking): rename first 'Results' heading to 'Measurement mo…
mjohns-databricks Aug 21, 2026
784cad6
perf(pyrx): eliminate per-file stat in list_local_files FUSE enumeration
mjohns-databricks Aug 21, 2026
6c0ea15
fix(bench): write-sweep caches a bounded source instead of an ineffec…
mjohns-databricks Aug 21, 2026
a22a0c6
feat(bench): add progress prints to all long-running bench legs
mjohns-databricks Aug 21, 2026
7d44ee8
feat(bench): extend layout-sweep to sweep write modes fuse/external/m…
mjohns-databricks Aug 21, 2026
53e680c
fix(bench): layout dimension on external, scan reads external tables
mjohns-databricks Aug 21, 2026
50b9ef7
docs(readers): demote manifest/tilesTable to optional, reflect stat-f…
mjohns-databricks Aug 21, 2026
efc8562
test(bench): update install-cell assertion for the two-step classic i…
mjohns-databricks Aug 21, 2026
e44d2c5
fix(bench): pass filespace for gpkg external writes + validate round-…
mjohns-databricks Aug 21, 2026
255f747
docs(diagram): fix GBX Common Functions diagram layout
mjohns-databricks Aug 21, 2026
88bdde3
docs(readers): combine the two green tips on the overview into one
mjohns-databricks Aug 21, 2026
77c7346
docs: fix 3 pre-existing broken anchors surfaced by the docs build
mjohns-databricks Aug 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
20 changes: 19 additions & 1 deletion .claude/qc-judge/config.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"release_notes_path": "docs/docs/beta-release-notes.mdx",
"release_notes_path": "docs/docs/release-notes.mdx",
"internals_extra_patterns": [
"wave\\s*\\d+",
"@databricks\\.com",
Expand Down Expand Up @@ -30,6 +30,24 @@
"expect_exit": 0,
"timeout_seconds": 30
},
"param-name-parity": {
"name": "Parameter name consistency and arity parity across tiers",
"type": "command",
"severity": "warn",
"enabled": true,
"cmd": "[ -f docs/scripts/check-param-names.py ] || { echo 'param-names script absent; skip'; exit 0; }; python3 docs/scripts/check-param-names.py",
"expect_exit": 0,
"timeout_seconds": 30
},
"docs-examples": {
"name": "Docs example output tables well-formed + all declared tabs complete",
"type": "command",
"severity": "warn",
"enabled": true,
"cmd": "[ -f docs/scripts/check-docs-examples.py ] || { echo 'absent; skip'; exit 0; }; python3 docs/scripts/check-docs-examples.py",
"expect_exit": 0,
"timeout_seconds": 30
},
"doc-coverage": {
"name": "Every registered function documented + no placeholder example outputs",
"type": "command",
Expand Down
32 changes: 0 additions & 32 deletions .github/CODECOV.md

This file was deleted.

6 changes: 3 additions & 3 deletions .github/actions/pyrx_build/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,10 @@ runs:
# Hash-pinned supply chain (matches requirements-ci.txt / dev-container
# discipline; see docs/docs/security.mdx): every wheel is sha256-verified,
# so a compromised mirror serving same-version-different-bytes fails closed.
# Regenerate via: uv pip compile --generate-hashes python/geobrix/requirements-pyrx-ci.in
# Regenerate via: uv pip compile --generate-hashes python/geobrix/requirements-light-ci.in
# No GDAL exception here — rasterio's bundled-GDAL binary wheel is pinned
# in the lock too (the lightweight API installs no native GDAL, no JAR).
pip install --require-hashes -r python/geobrix/requirements-pyrx-ci.txt
pip install --require-hashes -r python/geobrix/requirements-light-ci.txt
# Install the in-repo package itself with --no-deps; every dependency is
# already in the hash-pinned closure above.
pip install --no-deps python/geobrix
Expand All @@ -68,4 +68,4 @@ runs:
# bench (bench-harness spec/compare/fingerprint/runner unit tests -- guards
# the light bench FnSpec registry + cross-tier count invariants).
# See test/conftest.py for the maintained condition.
pytest test/pyrx test/ds test/pyvx test/pygx test/pmtiles_light test/stac test/earthdata test/vizx test/sample test/bench -m "not integration" -v
pytest test/pyrx test/ds test/pyvx test/pygx test/pmtiles_light test/stac test/earthdata test/vizx test/sample test/bench test/core -m "not integration" -v
6 changes: 3 additions & 3 deletions .github/actions/python_build/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ inputs:
description: 'Run Python lint (isort, black, flake8) and fail build on errors? true or false. Typically true only for PRs targeting main.'
default: 'false'
enable_coverage:
description: 'Run pytest with coverage (--cov) and emit coverage.xml for Codecov? true or false.'
description: 'Run pytest with coverage (--cov) and emit coverage.xml? true or false.'
default: 'false'
runs:
using: "composite"
Expand Down Expand Up @@ -117,14 +117,14 @@ runs:
# Heavyweight must not test lightweight. The lightweight tiers (pyrx, pyvx,
# pygx, ds, pmtiles_light, stac, earthdata, vizx, sample) need
# rasterio/shapely/pandas/pyarrow etc. that are NOT in requirements-ci.txt;
# they run in their own 'pyrx' CI job against requirements-pyrx-ci.txt.
# they run in their own 'pyrx' CI job against requirements-light-ci.txt.
# test/conftest.py collect_ignore already skips these when rasterio is
# absent, but we ALSO pass --ignore explicitly so a light dir can never run
# in the heavy phase even if the env drifts (gains rasterio) or a caller
# invokes a light dir directly. This list MUST stay in sync with
# _LIGHT_TEST_DIRS in test/conftest.py.
# pytest-cov is already locked in requirements-ci.txt; no extra install needed.
LIGHT_IGNORES="--ignore=test/pyrx --ignore=test/pyvx --ignore=test/pygx --ignore=test/ds --ignore=test/pmtiles_light --ignore=test/stac --ignore=test/earthdata --ignore=test/vizx --ignore=test/sample --ignore=test/bench"
LIGHT_IGNORES="--ignore=test/pyrx --ignore=test/pyvx --ignore=test/pygx --ignore=test/ds --ignore=test/pmtiles_light --ignore=test/stac --ignore=test/earthdata --ignore=test/vizx --ignore=test/sample --ignore=test/bench --ignore=test/core"
if [ "${{ inputs.enable_coverage }}" = "true" ]; then
pytest -m "not integration" $LIGHT_IGNORES --cov=src --cov-report=xml --cov-report=term --cov-config=pyproject.toml -q
else
Expand Down
4 changes: 2 additions & 2 deletions .github/docs/ci-runner-and-cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ runs-on:

Currently used by: `build_main` build, `build_python` build, `build_scala`
build, `build_scala_by_package` test-package, `codecov-scala-parallel`
coverage-package, `codecov-upload` coverage, `codeql-analysis` analyze.
coverage-package, `codeql-analysis` analyze.

### Light jobs — `databrickslabs-protected-runner-group` / `linux-ubuntu-latest`

Expand All @@ -33,7 +33,7 @@ runs-on:
labels: linux-ubuntu-latest
```

Currently used by: `build_main` update-doc-inventory + codecov,
Currently used by: `build_main` update-doc-inventory,
`codecov-scala-parallel` merge-and-upload, `verify-maven-pgp` verify,
`deploy-docs` build + deploy.

Expand Down
40 changes: 1 addition & 39 deletions .github/workflows/build_main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ jobs:
- name: build scala
uses: ./.github/actions/scala_build
with:
# Disable scoverage in main build to keep CI ~10–15 min (full Scala coverage is ~30+ min). Run "Upload coverage to Codecov" workflow manually for Scala+Python coverage.
# Disable scoverage in main build to keep CI ~10–15 min (full Scala coverage is ~30+ min). Run coverage locally via the gbx:coverage:* commands.
enable_coverage: "false"
fail_on_scalastyle: ${{ github.event_name == 'pull_request' && github.base_ref == 'main' }}
- name: build python
Expand All @@ -130,44 +130,6 @@ jobs:
run: |
mkdir -p "$GBX_SAMPLE_DATA_ROOT"
sudo -E mvn -C -q test -Dsuites='tests.docs.scala.*,docs.tests.scala.*' -DfailIfNoTests=false -Dscoverage.skip
- name: Stage coverage for upload job
run: |
mkdir -p coverage-reports
cp -f target/scoverage.xml coverage-reports/ 2>/dev/null || true
cp -f target/scoverage-report/scoverage.xml coverage-reports/scoverage-report.xml 2>/dev/null || true
cp -f python/geobrix/coverage.xml coverage-reports/ 2>/dev/null || true
if [ -z "$(ls -A coverage-reports 2>/dev/null)" ]; then
echo "No coverage files found. Expected Scala: target/scoverage.xml or target/scoverage-report/scoverage.xml; Python: python/geobrix/coverage.xml"
exit 1
fi
ls -la coverage-reports/
- name: Upload coverage artifacts
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
with:
name: coverage-reports
path: coverage-reports
if-no-files-found: warn
- name: upload artifacts
uses: ./.github/actions/upload_artifacts

codecov:
name: Upload to Codecov
runs-on:
group: databrickslabs-protected-runner-group
labels: linux-ubuntu-latest
needs: build
if: always() && needs.build.result == 'success'
permissions:
contents: read
steps:
- name: Download coverage artifacts
uses: actions/download-artifact@70fc10c6e5e1ce46ad2ea6f2b72d43f7d47b13c3 # v8.0.0
with:
name: coverage-reports
path: coverage-reports
- name: Upload to Codecov
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2
with:
token: ${{ secrets.CODECOV_TOKEN }}
directory: coverage-reports
fail_ci_if_error: false
6 changes: 0 additions & 6 deletions .github/workflows/build_scala.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,3 @@ jobs:
enable_coverage: "true"
- name: upload artifacts
uses: ./.github/actions/upload_artifacts
- name: Publish test coverage to Codecov
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: target/scoverage.xml,target/scoverage-report/scoverage.xml
fail_ci_if_error: false
14 changes: 7 additions & 7 deletions .github/workflows/codecov-scala-parallel.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Optional: Run Scala coverage by package in parallel, merge, then upload to Codecov.
# Optional: Run Scala coverage by package in parallel and merge into one report.
# Faster wall-clock time than a single job running all tests (~10–15 min vs ~30–45 min).
# Trigger: workflow_dispatch only (or add schedule/on push if desired).
# The merged scoverage.xml is published as a workflow artifact (scoverage-merged).
#
# Requires: CODECOV_TOKEN secret.
# Merge script: scripts/ci/merge_scoverage.py
# Design: .github/docs/scala-coverage-parallel-design.md

Expand Down Expand Up @@ -108,9 +108,9 @@ jobs:
echo "=============================================================================="
fi

- name: Upload to Codecov
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2
- name: Upload merged coverage artifact
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: merged/scoverage.xml
fail_ci_if_error: false
name: scoverage-merged
path: merged/scoverage.xml
if-no-files-found: warn
75 changes: 0 additions & 75 deletions .github/workflows/codecov-upload.yml

This file was deleted.

14 changes: 14 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -53,3 +53,17 @@ test-logs/bench/
# Lakeflow DAB local override files (real per-workspace values; template committed as *.override.yml.example)
*.override.yml
.databricks/

# Agent/session scratch + local tooling (never committed). All internal planning
# (specs, plans, prompts, input, SDD ledgers) is consolidated under /.superpowers/;
# the public representation of decisions lives in docs/docs/. The legacy root
# locations (/prompts/, /input/, /docs/superpowers/) stay ignored as guards.
/.superpowers/
/docs/superpowers/
/.isaac/
/.tmp
/scratchpad/
# pytest cache (appears in repo root and under docs/, python/, notebooks/, etc.)
.pytest_cache/
# Locally-built docs bundle (only the LFS platform tarball in resources/static/ is committed)
/resources/static/geobrix-docs-*.zip
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
## v0.2.0

Version bump with notable fixes and improvements. See [Beta Release Notes](docs/docs/beta-release-notes.mdx) for API and naming changes.
Version bump with notable fixes and improvements. See [Release Notes](docs/docs/release-notes.mdx) for API and naming changes.

### Notable changes
- BNG aggregators (`bng_cellunion_agg`, `bng_cellintersection_agg`): fixed shared aggregation buffer bug (fresh buffer per partition); chip field resolution by type/name in union agg.
- Reader renames and other API changes documented in Beta Release Notes.
- Reader renames and other API changes documented in Release Notes.

---

Expand Down
Loading
Loading