[MAJOR] CLI updates, adding stage3 and stack functions, and reports

[eleanorfrajka]

Summary

• Restored sbe-asc + sbe-ascii support in Stage 1 after conflict-resolution losses: both file types now in SUPPORTED_FILE_TYPES; _normalize_sbe_ascii date-normalisation method restored; mooring_name subdirectory restored in raw file path; hab support restored in _add_instrument_metadata; nortek-ascii
  restored in header-path condition; serial filter (instrument_list) loop restored in both Stage 1 and Stage 2
• Suppressed seasenselib/pycnv INFO logs from stdout; redirected to log file only
• Standardised terminal output format across stage1 / stage2 / stage3 / stack: --> Processing {type}: {relative_path} then Creating output file: {relative_path}
• New: stage3.py — salinity derivation via gsw.SP_from_C; pressure interpolation for instruments without a sensor (flag 8); QARTOD gross-range and spike QC on T, C, S, P; tilt/roll QC for Aquadopps (flag 3 at ≥20°, flag 4 at ≥30°)
• New: mooring_level.py — oceanarray stack and oceanarray grid commands; stack resamples all instruments to common time axis (default 60 s), grid linearly interpolates onto regular pressure grid
• New: report.py — self-contained HTML mooring recovery report plus per-instrument sub-pages (timeseries, T-S diagram, QC histograms, flag breakdowns)
• New: parameters.py — package-default QC thresholds (gross-range, spike, tilt)
• New: validation.py — YAML and file format validation
• CLI (cli.py): added validate, plot, stack, grid entry points
• Time gridding: changed default file suffix from _use to _stage2
• Clock correction semantics: clock_offset = total correction at deployment; clock_drift_seconds = total correction at recovery; Stage 2 ramps linearly between the two (previously they were additive independent shifts)
• Report cruise/ship fields: now reads deployment_cruise/recovery_cruise (shows "deploy / recover" if they differ); location priority: seabed_ → deployment_ → planned_ → latitude
• Docs: updated clock_offset.rst (three-option YAML, new sign-convention table), processing_framework.rst (tilt QC, grid limitation re flagged data), roadmap.rst (per-instrument report pages marked implemented), README.md (current CLI, YAML example trimmed to oceanarray-only fields, QC override scoping)

Checklist:

• I have followed the coding conventions.
• [~] I have updated or added tests to cover my changes.
• I have updated the documentation if needed.
• I have run pytest to check that all tests pass.
• I have run pre-commit run --all-files to lint and format the code.

[Copilot]

Pull request overview

This PR is a major update to OceanArray’s CLI and processing workflow, expanding the pipeline beyond Stage 2 by adding Stage 3 QC, mooring-level stack/grid processing, YAML validation, and updated documentation while standardizing the stage filename suffixes (e.g., _stage2.nc instead of _use.nc).

Changes:

• Standardize stage file naming (_stage1.nc, _stage2.nc, _stage3.nc) and update tests/notebooks/docs accordingly.
• Add new capabilities: YAML validation (validation.py), mooring-level stack/grid (mooring_level.py), CLI entrypoints (cli.py), and shared defaults (parameters.py).
• Enhance Stage 2 clock-correction semantics (offset + drift ramp) and add plotting/report-related utilities.

Reviewed changes

Copilot reviewed 39 out of 103 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tests/test_time_gridding.py	Updates expected Stage 2 filename suffix to _stage2.nc.
tests/test_stage2.py	Updates Stage 1/2 filenames and adds new unit tests for clock parsing/drift resolution.
tests/test_stage1.py	Updates supported file type expectations and Stage 1 output naming/layout expectations.
tests/legacy/test_rodb.py	Minor string formatting tweak in legacy test fixture content.
requirements.txt	Removes seasenselib from install requirements and replaces with install note.
README.md	Overhauls docs to reflect new CLI commands, stages, stack/grid, report, and QC configuration.
pyproject.toml	Adds oceanarray console script entry point.
oceanarray/validation.py	Introduces YAML validation (instruments, file types, clock fields).
oceanarray/time_gridding.py	Switches default input suffix to _stage2 and updates docstrings/output formatting.
oceanarray/stage2.py	Implements new clock drift semantics, history annotations, cleanup rules, and standardized logging output.
oceanarray/readers.py	Refactors imports and adds Nortek CSV parsing/loading helpers.
oceanarray/plotters.py	Adds downsampling + new raw plotting helpers and a mooring overview plotter.
oceanarray/parameters.py	Adds package defaults for QC flags, thresholds, plotting constants, and instrument abbreviations.
oceanarray/oceanarray.mplstyle	Adjusts default DPI and style formatting.
oceanarray/mooring_level.py	Adds stacking (time resample) and gridding (pressure interpolation) commands.
oceanarray/clock_offset.py	Minor formatting change in one print statement.
oceanarray/cli.py	Adds CLI subcommands: process/plot/stack/grid/report/validate and standardized console output.
notebooks/legacy/demo_mooring_rdb.ipynb	Formatting-only notebook cleanup.
notebooks/legacy/demo_instrument_rdb.ipynb	Formatting-only notebook cleanup.
notebooks/legacy/demo_batch_instrument.ipynb	Formatting-only notebook cleanup.
notebooks/demo_step1.ipynb	Formatting-only notebook cleanup and minor style changes.
notebooks/demo_stage2.ipynb	Formatting-only notebook cleanup and minor style changes.
notebooks/demo_stage1.ipynb	Formatting-only notebook cleanup and minor style changes.
notebooks/demo_mooring.ipynb	Formatting-only notebook cleanup and minor style changes.
notebooks/demo_instrument.ipynb	Formatting-only notebook cleanup and minor style changes.
notebooks/demo_clock_offset.ipynb	Formatting-only notebook cleanup and minor style changes.
notebooks/demo_climatology.ipynb	Formatting-only notebook cleanup and minor style changes.
docs/source/roadmap.rst	Updates roadmap to reflect implemented Stage 3, stack/grid, reports, validation, and revised priorities.
docs/source/processing_framework.rst	Updates pipeline description to include Stage 3 and stack/grid behavior and limitations.
docs/source/conf.py	Updates Sphinx logo asset name.
docs/source/clock_offset.rst	Updates sign conventions and YAML clock correction semantics and examples.
data/wb2_9_201114info.dat	Removes tracked legacy data file (now ignored).
data/wb1_12_2015_6123_head10.use	Removes tracked legacy data file (now ignored).
data/wb1_12_2015_005.microcat.txt	Removes tracked legacy data file (now ignored).
data/test_mooring.yaml	Removes tracked legacy test YAML (now ignored).
data/moor/proc/dsE_1_2018/dsE_1_2018.mooring.yaml	Removes tracked example mooring YAML (now ignored).
.gitignore	Ignores data/ and outputs/ directories and keeps legacy patterns.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.