Sudhanva's submission PR - #5
Open
nsudhanva wants to merge 20 commits into
Open
Conversation
- pyproject.toml + uv.lock: cross-platform uv env (macOS + Linux/CUDA), natten/matepoint gated to linux via markers with shi-labs TLS bypass - notebooks/explore_dataset.ipynb: dataset exploration - notebooks/inference.ipynb: reconstructed input pipeline + model inference Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- coastline overlay from land_mask (no cartopy) - robust percentile color limits (no outlier blowout) - add 500 hPa height + 250 hPa jet, MSLP isobars - fix precip: exponentiate log-space 142_lsp back to mm - label 6h increment honestly (ERA5 fcst - GFS analysis) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- gfs.py: byte-range GFS f000 fetch from noaa-gfs-bdp-pds via .idx - preprocess.py: GFS -> both encoders (level mismatch via interp_levels), normalize - inference/validate/outputs/s3io/metrics/run_cycle: full cycle + CloudWatch - Dockerfile, README, requirements; validated fetch+preprocess locally (5568 gpm z500, N(0,1) normed) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- API variable names/units: temperature_2m, dewpoint_2m, pressure_msl,
wind_{u,v,speed}_{10m,100m}, total_cloud_cover, total_precipitation_6h,
and upper-level geopotential/temperature/wind_u/wind_v/specific_humidity
- (time, level[25 hPa], lat, lon) dims, CF-1.8, zlib-compressed
- run_cycle passes full output array + mesh to the writer
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- .dockerignore keeps weights/data/.venv out of the build context - CI builds+pushes the pipeline image to ECR on native amd64 (manual trigger) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ook cron - statemachine.json: EventBridge->StepFunctions->SageMaker Processing (retry, catch->SNS) - cron-onstart.sh: 6-hourly cron on the notebook GPU as the interim 24h runner - infra/README: deploy steps for both paths Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Decoder returns (1,720,1440,157); netCDF writer expected (720,1440,157). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- global attrs can't hold the nested validation dict -> json.dumps it - exp() of log-space 6h precip could overflow -> clip to [0,2000] mm Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
exp() of log-space precip could overflow; clip to [0,2000] mm for the validation/plot fields too (netCDF was already clipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Avoids apt/dpkg (flaky under emulation), ships torch 2.5.1 + CUDA 12.1, then pip-installs natten + matepoint + pipeline deps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- validate: report + fail on negative humidity, dewpoint>temp; finite precip metric - outputs: clamp specific_humidity>=0 and dewpoint<=temperature in the netCDF - cron-onstart: reinstall uv (does not survive notebook stop/start); emit cycle_success=0 on bootstrap-level failure - smoke_fetch: exit non-zero on FAIL - add server/infra/dash.json (was referenced but missing) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ntainer self-test, honest docs - validate: finiteness over ALL 157 channels (catches NaN in any channel); precip validated on the SAME 6h-total quantity that's written, raw finiteness + capped-frac reported; precipitation flagged EXPERIMENTAL (precip_reliable), not vacuously passed - outputs: single-source total_precip_6h_mm() (raw for validate, clamped for output at a credible 1000mm cap); overview plots the SAVED 6h total at a fixed scale - run_cycle: --selftest (imports model + builds meshes, no GPU/weights); fetch weights from S3 so the container is self-sufficient; corrected emitted metric keys - CI: run the container --selftest after push (demonstrates the image imports/builds) - tests/test_validate.py: exercises the thresholds (NaN-in-unselected, Td>T, precip overflow) - READMEs (root/server/infra): design-vs-live, quota reality, notebook runs source not the container, fixed docker command (S3 weights + creds) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…E runner Can't finish+test SF/EB (Processing-job quota in manual review), and the notebook cron already satisfies automation + 24h+. So: delete statemachine.json, reframe infra around the cron, ensure crond is started, wire SNS alerts. SF/EB kept only as a described "99.9% uptime" future, not as dead IaC. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y shi-labs) - shi-labs.com (expired cert) times out in CI -> build from an S3-vendored natten wheel - workflow fetches the wheel before docker build; CI role granted S3 read - selftest tolerates natten import failing on GPU-less CI runners (reports, no hard fail) - .gitignore the 451MB wheel; document the local fetch step Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…entials v6, setup-buildx v4, build-push v7) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… drift - run_cycle/s3io: invalid forecasts go to a quarantine/ prefix and never update latest.json; SNS alert fires on invalid output (not just on crash) - validate: precip_reliable stays False until the log inverse transform is verified (PRECIP_TRANSFORM_VERIFIED=False); also report precip_capped_mass_frac (the capped tail can dominate the mean even at <1% capped cells) - docs/readme: remove "Step Functions IaC committed" (it was deleted), cron not SF in run_cycle docstring, precip cap 1000mm (was 2000), dashboard drops removed metric Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ocessing Processing-job quota approved, so the pipeline now runs the ECR container on an ephemeral g5 GPU on a 6-hourly schedule (verified: standalone Processing job produced a valid forecast in 45.6s). Retires the notebook cron (kept for reference, notebook stopped). Re-adds statemachine.json and documents the deployed resources. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Processing) architecture
…cle on error
EventBridge-started executions are named <uuid>_<uuid>, so States.Format('wm3-sfn-{}', Execution.Name)
produced an 81-char name with an underscore, which SageMaker rejects. Split on '_' and take the
first UUID for a valid 44-char name. Also route Alert -> Fail so a failed cycle reports FAILED
instead of a misleading SUCCEEDED.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please read: https://windborne.sudhanva.me for more details. Ignore this PR