feat: upgrade version to 0.16.12 - #25
Open
chun-wei-yu-cardinalblue wants to merge 381 commits into
Open
Conversation
* Create scanner for removing comments and continuations * When reading the requirements txt, handle line continuations and remove comments * Fix fast_generator test * Update pkg/requirements/requirements_test.go Co-authored-by: Aron Carroll <aron@replicate.com> Signed-off-by: Will Sackfield <will.sackfield@gmail.com> * Fix config test * Include beginning of line comment testing * Handle CRLF line breaks * Move no whitespace string logic to scanner * Restore removing whitespace strings above scanner * Scanner exits when the token is nil on return --------- Signed-off-by: Will Sackfield <will.sackfield@gmail.com> Co-authored-by: Aron Carroll <aron@replicate.com>
* Allow "openable" outputs as path-like uploadables so that more types than `cog.types.Path` will be treated as path-like uploadables. Connected to PLAT-1107 * Drop import of unused `Path` type * Go back to using `isinstance` but based on `pathlib.Path`
…eplicate#2233) * uv for dependencies * Create scripts/setup to bootstrap local env * Remove invalid ruff lint ignore rules These are no longer available and caused warnings in logs and in the editor. Remove 'em. More info: https://docs.astral.sh/ruff/rules/#flake8-2020-ytt * Remove deprecated ruff VSCode config VSCode is using the new native ruff server by default and these config options are no longer valid. Removing to stop warnings. * Commit pending `go mod tidy` changes Appears `go.{mod,sum}` had changes at some point that weren't committed, and running any go command fixed them. This just commits changes from `go mod tidy` so they stop cluttering my unstaged files. * Update setup instructions in CONTRIBUTING.md * Add bin dir to gitignore This gives an option to use dev builds in this repo without conflicting with a system installed cog.
Update init.go
* Allows fast to be defined as a build property in cog.yaml to prevent the constant use of —x-fast flags on the CLI.
* Add local image flag for fast context loading * Add a new experimental flag —x-localimage for use with —x-fastpush * When activated, the flag uses the .cog tmp directory as an rsync for the source code without weights * It separates the different build steps into different build contexts * On run/predict/train/serve it loads the weights in as volume mounts * This allows it to not reload the context each time a small file is changed for weights * Fix no tmpMount usage * Potential fix for code scanning alert no. 24: Potentially unsafe quoting Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: Will Sackfield <will.sackfield@gmail.com> * Fix lint * Use . in non local image for copy * Add back weights copy exclusion from non local images * Send standard build directory if not local image * Fix calculating rel dir on relative dir * Build then predict in int test * Add fast and local image to predict test * Use the image name from the config in predict * Only if we use build fast * Fix cog predict build on fast build * Feed docker image in directly * Use projectDir to fill in weights path * Put debug in the right position * Remove -t in cog predict * Capture output of test and check return code * Use absolute path for weights manifest * Use hard links rather than file copies * Mirror the local directories permissions * Follow symlink to target * Make sure directory permissions are aligned * Fix fast push * Synchronise directories between different golang modules * Use MkdirAll * Use a set for weight path check --------- Signed-off-by: Will Sackfield <sackfield@replicate.com> Signed-off-by: Will Sackfield <will.sackfield@gmail.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
to see if it's an off by one issue that's preventing extension packages from directly depending on `cog`.
* zsh system packages were overwriting the /bin dir * Follow the symlink on ubuntu instead
* Prevent Optional types from being required * Add integration test for optional required * Fix lint * Fix pydantic v2 required
* Add local whl install integration test * Bind the CWD to /src in requirements * This was mistakenly binding to the src context which is not applicable in a non local image build
* Add zsh integration test * Extract then copy the apt package * Instead of extracting to root, extract to tmp then copy over to root * This preserves symlinked directories * Use rsync to copy * Do not clobber symlinked dirs * Use --keep-directory-symlink instead
* Add integration test to verify overrides * Add support for python_overrides in cog.yaml * Allows users to force specific versions of packages even if pips dependency resolution rejects them * Add test for determining the version of numpy * Fix test --------- Signed-off-by: Will Sackfield <sackfield@replicate.com>
* Add integration test for JSON in cog predict * Test that we can use cog predict on a typeddict * Support complex types in cog predict * Check the OpenAPI schema * If it isn’t a string, attempt a JSON parse * Fix stdout check * Allow for schema retrieval errors * Only serialise if the type is an object --------- Signed-off-by: Will Sackfield <sackfield@replicate.com>
Docker desktop w/ containerd enabled is creating provenance attestations with each build. When this happens, each image is pushed with 2 additional manifests, one is the provenance manifest containing build facts, and the other an index pointing to the actual image and the provenance manifest. While useful in some contexts, cog is not one of them atm. The registry is ignoring them, but this PR disables creation entirely.
* Add install requires packaging int test * Log the build process * Include CUDA in test * Make cuda a string * Add torch * Use cuda 12.1 * Downgrade to torch 2.5.0
* The writes get discarded by Docker, this allows editable pip installs
* Add integration test for string list prediction * Add a test for inputting a single value to a list input * Handle arrays when processing cog predict * Fix hello world assertion
* Add granite integration test * Verify the behaviour of pydantic clobbering with the IBM granite project * Use the stated pydantic tolerances * We now support >=2 in pydantic so mirror the tolerances we set in the toml * Fix test to focus on granite-project --------- Signed-off-by: Will Sackfield <sackfield@replicate.com>
Fixes replicate#2275 There is a bug where an existing .python-version file will be copied from an existing project into the Docker container and conflict with the Python version set in the cog.yaml. We now include the .python-version in the .dockerignore file so that it will not be copied into the container at build time.
Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.74.1 to 1.79.2. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@service/s3/v1.74.1...service/s3/v1.79.2) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.79.2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
* Upgrade go to 1.24.2 * remove toolchain directive no longer needed in go 1.24+ * port tools to go tools, fixes import issues * Update CONTRIBUTING.md * add TODO for moving golangci-lint to a binary install later * add gotest.tools/gotestsum to tools * remove tools.go now that it's in go.mod
Bumps [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) from 5.2.1 to 5.2.2. - [Release notes](https://github.com/golang-jwt/jwt/releases) - [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md) - [Commits](golang-jwt/jwt@v5.2.1...v5.2.2) --- updated-dependencies: - dependency-name: github.com/golang-jwt/jwt/v5 dependency-version: 5.2.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github.com/go-git/go-git/v5](https://github.com/go-git/go-git) from 5.12.0 to 5.13.0. - [Release notes](https://github.com/go-git/go-git/releases) - [Commits](go-git/go-git@v5.12.0...v5.13.0) --- updated-dependencies: - dependency-name: github.com/go-git/go-git/v5 dependency-version: 5.13.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.5.0 to 4.5.2. - [Release notes](https://github.com/golang-jwt/jwt/releases) - [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md) - [Commits](golang-jwt/jwt@v4.5.0...v4.5.2) --- updated-dependencies: - dependency-name: github.com/golang-jwt/jwt/v4 dependency-version: 4.5.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
* propagate context.Context to build functions * add context.Context to docker & openapi helpers
* Add build logging * Add R8_COGLOG_DISABLE to disable logging * Add push logging * Fix tests * Fix lint
…licate#2288) * DockerCommand.exec accepts io.Writer DockerCommand.exec will write the underlying command stdout and stderr to the provided io.Writer. This allows callers to either capture the entire output or allow long running commands to stream output to a writer that isn't os.Stdout or os.Stderr * upgrade docker/docker github.com/docker/docker needed an upgrade, but this wasn't compatible with a dependency of goreleaser, so I removed it from go tools * Move several docker CLI helpers to the docker interface and implementation Several helper functions in the `docker` package (`ImageInspect`, `ImageExists`, `ContainerInspect`, `ContainerLogsFollow`, and `Pull`) invisibly shelled out to the docker CLI, often with slightly different logic than was the `DockerCommand` struct was doing. This PR moves them to functions on the `DockerCommand` struct as well as adds them to the `command.Command` interface so they can be swapped out with the upcoming engine client. * add docker client integration tests test harness for running integration tests against an actual docker daemon and local registry * port docker.Stop to command.ContainerStop * force alpine test fixture to platform/amd64 * handle Pull errors from both dockerd & containerd different errors are returned when working with a docker registry and OCI registry. This maps both of them to the not found error * fix tests & lint * Pull returns the inspect response and has a force param There's a few places in the code where we check for a local image, and if not found begin a pull, then carry on. I had a bug that allowed pull failures to be swallowed and stale inspect responses to be passed to predict+train. This fixes the issue by ensuring that Pull returns the inspect response of the image that was just pulled, or to skip pulling if the image already exists using a force flag.
* Add cog migrate command * Attempts to rewrite python packages to python requirements * Removes run commands that are known to be safe * Add fast: true to config * Rewrites python code to deal with Optionals properly * Fix accept flag * Fix wrong variable assigned to accept * Fix integration test
* Correct long line (with continuation) handling Signed-off-by: Dan Fairs <dan@condensereality.com> * Remove a redundant line Signed-off-by: Dan Fairs <dan@condensereality.com> --------- Signed-off-by: Dan Fairs <dan@condensereality.com> Co-authored-by: Dan Fairs <dan@condensereality.com>
* Add —config to allow the user to specify the config * —config allows the user to tell cog which file to look at * This is useful when they have multiple configs for different environments * Fix lint * Inject configFilename into migrator * Change —config to be -f to conform to docker --------- Signed-off-by: Will Sackfield <sackfield@replicate.com>
) * fix: make TestPredictionPathUploadIterator deterministic This test was flaky in CI because it assumed webhooks would arrive in strict chronological order with specific statuses at specific indices. However, webhooks are delivered asynchronously and can arrive out of order, especially in CI environments with varying performance. The test would fail with: Expected: "processing" Actual: "succeeded" This happened when the prediction completed faster than expected and the "succeeded" webhook arrived before some "processing" webhooks. Changes: - Collect all webhooks until the final "succeeded" webhook arrives - Validate the collected set of webhooks rather than assuming order - Check for presence of expected states (initial processing, outputs) rather than asserting on specific indices - Make upload validation order-independent by collecting all uploads first, then verifying their contents exist - Remove unused strconv import The test now passes consistently across multiple runs and different environments. * chore: try to debug test flakiness Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * chore: dont use quiet in IT harness Signed-off-by: Mark Phelps <mphelps@cloudflare.com> --------- Signed-off-by: Mark Phelps <mphelps@cloudflare.com> Co-authored-by: Mark Phelps <mphelps@cloudflare.com>
…ied (replicate#2619) Co-authored-by: SharkFishDeveloper <shahzebakhtart66@gmail.com> Co-authored-by: Mark Phelps <209477+markphelps@users.noreply.github.com>
Add internal architecture docs covering Cog's core concepts: - Model source (cog.yaml, Predictor, types, weights) - Schema (OpenAPI generation and consumers) - Prediction API (endpoints, envelope format) - Container runtime (two-process architecture, IPC) - Build system (config to image pipeline) - CLI (commands and host-container interaction) These docs establish shared vocabulary for maintainers and provide mental models for understanding Cog's design. Also removes outdated ARCHITECTURE.md which contained deprecated coglet/monobase content.
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 6 to 7. - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v6...v7) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mark Phelps <209477+markphelps@users.noreply.github.com>
…eplicate#2632) * Port concurrent predictions test to Go and consolidate curl command - Add TestConcurrentPredictions in integration-tests/concurrent/ package Tests parallel HTTP requests with graceful shutdown behavior - Consolidate curl/retry-curl into single curl command with built-in retry logic (10 attempts, 500ms delay) - Update 4 txtar tests to use simplified curl command - Remove Python test_concurrent_predictions and async-sleep-project fixture - Update README with Test Formats section explaining when Go tests are used instead of txtar (concurrent requests, PTY interaction) * Port interactive TTY test to Go and remove Python integration tests (replicate#2634) * Port interactive TTY test to Go - Add TestInteractiveTTY and TestInteractiveTTYEchoCommand in integration-tests/pty/ package using github.com/creack/pty - Tests verify cog run works with interactive PTY sessions - Remove Python test_run_shell_with_with_interactive_tty - Update README to reflect PTY tests are now implemented * Remove Python integration tests - all ported to Go - Remove test-integration/ directory entirely - Remove tox integration environment - Update Makefile to consolidate test-integration-go into test-integration - Update CI workflow to remove Python integration test jobs - Update CONTRIBUTING.md, AGENTS.md, and README documentation - Update pyproject.toml to remove test-integration lint exclusions All integration tests now live in integration-tests/ using Go testscript. * Add CODEOWNERS to auto-request review from @replicate/cog --------- Co-authored-by: Mark Phelps <mphelps@cloudflare.com> * Replace COG_TEST_FAST with t.Short() for test skipping Replace custom COG_TEST_FAST environment variable with Go's standard testing.Short() mechanism across all tests. Changes: - Use t.Short() in concurrent_test.go instead of COG_TEST_FAST check - Remove custom [fast] condition from suite_test.go (testscript has built-in [short] support) - Replace [fast] with [short] in all .txtar test files - Update documentation in CONTRIBUTING.md, README.md, and design docs This aligns with Go testing conventions where -short flag is the standard way to skip slow tests. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Replace COG_TEST_FAST with t.Short() for test skipping Replace custom COG_TEST_FAST environment variable with Go's standard testing.Short() mechanism across all tests. Changes: - Use t.Short() in concurrent_test.go instead of COG_TEST_FAST check - Remove custom [fast] condition from suite_test.go (testscript has built-in [short] support) - Replace [fast] with [short] in all .txtar test files - Update documentation in CONTRIBUTING.md, README.md, and design docs This aligns with Go testing conventions where -short flag is the standard way to skip slow tests. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Simplify and modernize integration test code - Use http.StatusOK instead of magic number 200 - Use io.ReadAll instead of manual buffer loop in harness - Use strings.NewReader instead of bytes.NewBufferString - Use testing.Short() instead of COG_TEST_FAST env var in pty tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: quiet buildkit_progress to see failing tests Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * fix: increase timeout and add polling for TestInteractiveTTY The test was failing in CI because the 5-second timeout wasn't long enough for Docker image builds when the cache is cold. In CI environments, pulling base images and building can take 20-60 seconds. Changes: - Add waitForOutput() helper that polls for expected output pattern - Increase timeout from 5s to 60s for bash prompt detection - Return early when bash prompt is detected (no need to wait full 60s) - Add explicit check for bash prompt pattern ':/src#' This makes the test: - Fast locally (returns in ~5s with warm cache) - Resilient in CI (up to 60s for cold cache builds) - More robust (waits for actual prompt, not arbitrary timeout) * fix: use dynamic port allocation in concurrent predictions test Fixed flaky test caused by hardcoded port 5555. In CI environments, this port could be in use by stale servers or other processes, causing predictions to go to the wrong server and return empty output. Now uses dynamic port allocation via net.Listen to get an available port, matching the pattern used in harness/harness.go. * ci: add Go module caching for integration tests Configure actions/setup-go to cache both root and integration-tests go.sum files to speed up dependency downloads in CI. * test: skip concurrent predictions test for coglet-alpha coglet-alpha has a known bug with concurrent async predictions that causes predictions to return empty output. Since this is in the released version, skip this test when COG_WHEEL=coglet-alpha. --------- Signed-off-by: Mark Phelps <mphelps@cloudflare.com> Signed-off-by: Mark Phelps <209477+markphelps@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
* chore: update error messages Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * test: add integration test for torch version required error message Adds a testscript-based integration test that validates the error message when torch is specified without a version and GPU is enabled. This ensures the fix from PR replicate#2619 continues to work correctly. --------- Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
* cog-dataclass: add SDK core types (Phase 1) Implements the user-facing API for the new pydantic-less Cog SDK: - types.py: Path, Secret, File, URLFile, URLPath, iterators - predictor.py: BasePredictor - model.py: BaseModel with __init_subclass__ auto-dataclass - input.py: Input(), FieldInfo with mutable default handling - coder.py + coders/: Coder system for custom type encoding 50 passing tests covering all core functionality. * cog-dataclass: add internal machinery (Phase 2) Add ADT types, inspector, and schema generation: - _adt.py: PrimitiveType, FieldType, Repetition, OutputKind, InputField, OutputType, PredictorInfo - _inspector.py: create_predictor(), check_input(), type introspection - _schemas.py: OpenAPI JSON schema generation - test_adt.py, test_inspector.py: 105 tests passing * cog-dataclass: add server adaptation (Phase 3) Server files adapted from cog-pydantic: - eventtypes.py: attrs -> dataclasses - scope.py: attrs.frozen -> @DataClass(frozen=True), evolve -> replace - helpers.py: removed pydantic OpenAPI fixup code - webhook.py: removed PYDANTIC_V2 branches - runner.py: @define -> @DataClass, removed pydantic model_dump - worker.py: attrs -> dataclasses, removed PYDANTIC_V2 blocks - http.py: removed PYDANTIC_V2/ValidationError/unwrap_pydantic_serialization_iterators New supporting modules: - schema.py: PredictionRequest/Response as dataclasses with .dict() method - config.py: simplified Config class for reading cog.yaml - json.py, files.py, errors.py, logging.py, mode.py: copied/adapted predictor.py additions: - load_predictor_from_ref, get_predict, get_train - has_setup_weights, extract_setup_weights, wait_for_env 105 tests passing. * Add COG_WHEEL=cog-dataclass support Adds 'cog-dataclass' as a magic keyword for wheel selection, enabling the pydantic-less cog SDK to be used via: COG_WHEEL=cog-dataclass cog build Changes: - pkg/wheels/wheels.go: Add WheelSourceCogDataclass, ReadCogDataclassWheel(), update ParseCogWheel() and go:generate directives - pkg/dockerfile/standard_generator.go: Add installEmbeddedCogDataclassWheel() which installs the wheel WITHOUT pydantic dependency - cog-dataclass/pyproject.toml: Rename package to 'cog-dataclass' so wheel is named cog_dataclass-*.whl (avoids conflict with cog-*.whl) * cog-dataclass: Phase 4 - Integration & wiring Server now fully functional: - python -m cog.server.http starts and runs predictions - OpenAPI schema properly generated with Input/Output types - Predictor loading via spec_from_file_location (like original cog) Changes: - predictor.py: Fix load_predictor_from_ref to use importlib.util - schema.py: Add missing fields to PredictionResponse (context, webhook, etc) - http.py: Remove response_model params (dataclasses work differently), inject Input/Output schemas into OpenAPI via custom_openapi() - _inspector.py: Handle raw default values in _create_input_field Tested end-to-end: setup runs, predictions work, OpenAPI reflects inputs. * cog-dataclass: Phase 5 compatibility + fix Go schema panic Python changes: - Add command modules (openapi_schema, call_graph) for build validation - Fix OpenAPI schema to include PredictionResponse and update response refs - Handle raw default values in _inspector._create_input_field() Go changes: - Fix nil pointer panic in predict.go when schema missing output definition - Add predict_test.go with tests for malformed schema handling Tested: cog build and cog predict work with COG_WHEEL=cog-dataclass * cog-dataclass: add input validation with check_input() - Call _inspector.check_input() to validate and normalize inputs before prediction - Validates constraints: ge/le, min_length/max_length, regex, choices - Returns proper 422 validation error format for Go CLI compatibility - Defaults now properly extracted from FieldInfo objects * cog-dataclass: wire up integration tests CI changes: - Add cog-dataclass to integration test matrix - Propagate COG_WHEEL env var to cog serve in test harness - Add cog_dataclass condition for skipping tests - Fix server cleanup to kill Docker containers by port cog-dataclass fixes: - Add input validation with check_input() and proper error format - Fix evolve_scope to handle 'tag' -> '_tag' mapping (attrs compat) - Handle string-to-int coercion in normalize() - Convert URL/data URI strings to URLPath in worker Skip Python <3.10 and monobase tests for cog-dataclass * integration-tests: skip monobase tests for cog-dataclass * cog-dataclass: fix training support - Handle function trainers (not just class-based) in load_predictor_from_ref - Add module to sys.modules for pickle compatibility - Fix get_predict/get_train to handle function predictors - Add /trainings endpoint schema to OpenAPI * cog-dataclass: add File type, fix validation, add pydantic dep - Add File type support to _adt.py for list[File] inputs - Fix Type import in types.py for pydantic schema support - Re-raise errors during build (is_build=True) so validation errors like invalid defaults are properly reported - Add pydantic>=2.7.0 dependency (required by fastapi) - Fix error messages for ge/le constraint violations - Skip predict_pipeline test (base image pydantic v1 issue) * cog-dataclass: support future_annotations and graceful import failures - Use typing.get_type_hints() to resolve string annotations from 'from __future__ import annotations' - Remove the explicit check that rejected future annotations - Handle import errors gracefully during build - continue with placeholder schemas instead of failing (allows builds when predictor has missing dependencies like replicate) - Distinguish between import errors (graceful fallback) and validation errors (fail build) during schema generation * integration-tests: skip monobase tests for cog-dataclass These tests use fast: true which requires monobase. Skip for cog-dataclass until monobase support is added. * integration-tests: skip complex_output for cog-dataclass This test uses pydantic.BaseModel as output type which is not supported by cog-dataclass (intentionally pydantic-free). * Correct CI issue Build the cog-dataclass wheel * cog-dataclass: address code review feedback Address valid Copilot review comments: - Fix resource leak in suppress_output.py (use nested context managers) - Add security clarification comments to noqa: S310 suppressions - Remove unused imports (sys, Type, BasePredictor, Any, Field, PredictorNotSet) False positives not addressed: - Mixed import styles are intentional where both module and names are used - Type errors for Input(default_factory=...) and predict() signatures are incorrect - "Unused" test class variables are testing exception-raising behavior * cog-dataclass: fix input validation error format and security issue Address CodeQL security warning about information exposure through exceptions: 1. Changed all ValueError messages in check_input() to use "{field}: {message}" format 2. Removed user input values (v!r) from all error messages 3. Added wrapper around normalize() to reformat errors consistently 4. Updated normalize() to not include user input values Error messages now look like: - "prompt: Value must be >= 1" - "text: Length must be <= 100" - "format: Value must be one of ['json', 'yaml']" Instead of the old format that exposed user values: - "invalid input value: prompt='<user input>' fails constraint >= 1" This: - Preserves API compatibility (errors still structured as FastAPI/Pydantic format) - Fixes the parsing bug in http.py (messages now match "{field}: {msg}" pattern) - Prevents information leakage through user input echo - Maintains good UX with clear, actionable error messages * Skip ITs that coglet alpha does not support Coglet Alpha is now being tested properly across the board. A number of these tests were not properly run for coglet alpha due to missing propagation of the `COG_WHEEL` env to the build process.
* chore: update ci to run dataclass python tests Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * chore: update ci to build wheels correctly Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * chore: install correct python versions Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * fix: use command substitution for wheel glob pattern in CI uv pip install doesn't handle glob patterns with extras correctly. Use command substitution to expand the glob first, then append [dev]. * fix: correct Path/Secret normalization and error messages in cog-dataclass - Fix PrimitiveType.normalize() to convert strings to Path and Secret objects - Update validation error messages to match expected test format: - Use 'fails constraint' instead of 'Value must be' - Use 'does not match' instead of 'Value must match/be one of' - Use 'missing required input field' instead of 'Field required' - Format numeric constraints without decimal points when integers All 105 tests now passing. * chore: rm coglet tests Signed-off-by: Mark Phelps <mphelps@cloudflare.com> --------- Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
* Ignore local agent and mise configuration files * Add Rust tooling tasks to mise configuration Tasks for coglet development: - build:rust / build:rust:release - Build Rust crates - build:coglet - Build Python wheel with maturin (dev) - build:coglet:wheel* - Build release wheels for various platforms - test:rust - Run Rust tests with nextest - test:coglet-python - Run Python integration tests - fmt:rust / fmt:rust:check - Format Rust code - clippy - Run lints - deny - Check licenses/advisories Tools added: cargo-binstall, cargo-deny, cargo-insta, cargo-nextest, maturin, ruff, ty * Add Cargo workspace for Rust crates Workspace structure: - crates/Cargo.toml - workspace root with shared dependencies - crates/coglet/ - main Rust library (empty scaffold) - crates/deny.toml - cargo-deny license/advisory config - crates/.gitignore - ignore build artifacts Shared workspace dependencies defined for async runtime (tokio), HTTP (axum, reqwest), serialization (serde), and error handling. * Add coglet dependencies to Cargo.toml Runtime: tokio, tokio-util (codec), futures, async-trait Serialization: serde, serde_json HTTP: axum (server), reqwest + ureq (webhooks) Utils: uuid, chrono, thiserror, anyhow, dashmap, tracing Unix: nix (signal handling) Dev: insta, wiremock, tower, http-body-util * Add health and version foundation types Health enum: Unknown, Starting, Ready, Busy, SetupFailed, Defunct SetupStatus enum with SetupResult for tracking setup phase VersionInfo for runtime version reporting Includes serde serialization with appropriate casing and tests. * Add IPC bridge protocol and codec Protocol types for parent-worker communication: - ControlRequest/Response for control channel (cancel, shutdown, ready) - SlotRequest/Response for per-slot data channel (predict, logs, output) - SlotId using UUID for unique slot identification - SlotOutcome for type-safe completion handling JsonCodec wraps LengthDelimitedCodec with serde_json serialization. * Add Unix socket transport for slot IPC NamedSocketTransport: filesystem sockets (all Unix platforms) AbstractSocketTransport: Linux abstract namespace (no filesystem) Adds ControlRequest::Init variant for worker initialization with transport info, predictor ref, slot count, and async/train flags. * Add Prediction state tracking and PredictionOutput PredictionStatus enum with terminal state detection. PredictionOutput for single values or streamed chunks. Prediction struct tracks lifecycle: status, logs, outputs, timing. Completion notification via tokio Notify for async waiting. CancellationToken re-exported from tokio_util for cancel propagation. * Add predictor traits and prediction lifecycle types PredictionResult aggregates output, timing, and logs. PredictionGuard provides RAII timing with cancellation support. PredictionMetrics for timing data collection. PredictionError for typed prediction failures. PredictFn/AsyncPredictFn type aliases for predictor function signatures. Also fixes PredictionOutput to use untagged serde serialization. * Add permit pool with typestate for slot management PermitInUse/PermitIdle/PermitPoisoned enforce valid state transitions at compile time - poisoned permits cannot become idle. PermitPool manages slot permits with RAII semantics: - Idle permits return to pool on drop - Poisoned permits are orphaned, reducing capacity AnyPermit enum for dynamic state storage. * Add PredictionSlot combining Prediction and Permit PredictionSlot holds both together with separate concerns: - Prediction behind Mutex for concurrent updates - Permit for RAII pool return on drop Slot transitions: into_idle() returns permit, into_poisoned() orphans it. Drop handler fails non-terminal predictions if slot leaked. * Add webhook sender with throttling and retry WebhookEventType enum for filtering (start, output, logs, completed). WebhookConfig with throttle interval, retry settings, status codes. TraceContext for W3C distributed tracing headers. WebhookSender implements: - send(): fire-and-forget for non-terminal events with throttling - send_terminal(): async with exponential backoff retries - send_terminal_sync(): blocking version for Drop contexts (uses ureq) Bearer auth via WEBHOOK_AUTH_TOKEN env var. * Add PredictionSupervisor for lifecycle management PredictionState snapshot for API responses with to_response() builder. PredictionHandle for waiting, state queries, and cancellation. SyncPredictionGuard for cancel-on-drop behavior in sync predictions. DashMap provides lock-free concurrent access - no deadlock risks. Terminal status triggers webhook send via spawned task. * Add Orchestrator for worker subprocess lifecycle * Add PredictionService for transport-agnostic prediction lifecycle * Add HTTP transport layer with axum server and routes * Add worker subprocess types for PyO3 bindings * Add coglet-python crate with PyO3 bindings * Add Python input/output processing infrastructure - input.rs: Runtime detection (Pydantic vs Coglet), URLPath download with parallel ThreadPoolExecutor, PreparedInput RAII for temp file cleanup - output.rs: Output processing using cog.json.make_encodeable() and upload_files() for base64 data URL conversion - audit.rs: TeeWriter for protecting sys.stdout/stderr with audit hooks that wrap user replacements while preserving slot routing - log_writer.rs: Full SlotLogWriter with ContextVar-based prediction routing, line buffering, setup log sender for health-check logs - cancel.rs: SIGUSR1 signal handling with CancelationException, CancelableGuard RAII for sync prediction cancellation * Add full predictor and worker bridge implementation - predictor.rs: Complete PythonPredictor with runtime detection, async/sync support, generator handling, input preparation, output processing, and schema generation - worker_bridge.rs: Full PredictHandler with asyncio event loop, per-slot cancellation tracking, log context management, SIGUSR1 signal cancellation for sync, future.cancel() for async * Add type stubs and Python tests - coglet.pyi: Type stubs for active(), serve(), _run_worker(), _is_cancelable() - tests/test_coglet.py: Integration tests for sync/async/generator predictors, health check, cancellation endpoints * Add setup log hook and audit helper exports to coglet module - Setup log hook for routing setup logs via SlotLogWriter - Export audit helpers (_is_slot_log_writer, _is_tee_writer, etc.) - Export SlotLogWriter and TeeWriter classes for isinstance checks * Upgrade PyO3 from 0.24 to 0.27 - Use abi3-py38 feature for stable ABI compatibility - Replace Python::with_gil with Python::attach (0.27 API) - Replace py.allow_threads with py.detach (0.27 API) - Add auto-initialize feature for standalone testing * Install log writers and audit hook in worker startup Add missing install_slot_log_writers() and install_audit_hook() calls in _run_worker() to enable stdout/stderr routing and protection. Remove module-level #![allow(dead_code)] since code is now integrated. * Add SetupError type for typed setup failures Replace Result<(), String> in PredictHandler::setup() with typed SetupError enum. Variants distinguish load vs setup vs internal errors, enabling future error code mapping without changing behavior. * Add architecture documentation for coglet crates - crates/README.md: Overall architecture, prediction flow, startup sequence, bridge protocol, directory structure - crates/coglet/README.md: Detailed coglet internals, components, health states, behaviors (shutdown, cancellation, slot poisoning) - crates/coglet-python/README.md: Python bindings details, active() flag, single async loop, stdout/stderr routing, audit hook, cancel * Remove dead predict_fn code path from service Production always uses orchestrator. The predict_fn/async_predict_fn path was only used in tests and never ran in production. - Remove PredictFn, AsyncPredictFn, PredictFuture types - Remove predict_via_function method - Remove legacy_pool field - Remove PredictionService::new(pool) constructor - Enforce READY requires orchestrator (silently ignores otherwise) - Update tests to not create impossible states * Extract Orchestrator trait for testable service layer Introduce Orchestrator trait with register_prediction() method: - Enables mocking orchestrator in service unit tests - OrchestratorHandle implements trait, keeps cancel/shutdown as inherent methods - Service now takes Arc<dyn Orchestrator> instead of Arc<OrchestratorHandle> - Export trait from coglet crate for downstream use * Add comprehensive tests using MockOrchestrator - Add MockOrchestrator in both service and routes test modules - Service tests: orchestrator integration, prediction flow, capacity - HTTP routes tests: full request/response cycle with mock predictor - Fix race in predict() by checking terminal before waiting on notify - 15 new tests covering prediction lifecycle end-to-end * Remove panics from permit/slot.rs state transitions - into_idle() now returns Option<IdleToken> instead of panicking - into_poisoned() now returns bool instead of panicking - Invalid state transitions log errors and use debug_assert for dev builds - Callers updated to handle new return types gracefully * Remove panics from orchestrator.rs - stdin/stdout take() now returns OrchestratorError instead of panicking - Prediction mutex locks use try_lock_prediction() helper that: - On poison: fails the prediction, logs error, returns None - Caller removes poisoned predictions from tracking - Further data from poisoned predictions is ignored * Remove panics from service.rs prediction locks - Add try_lock_prediction() helper that fails prediction on mutex poison - predict() returns PredictionError on mutex poison instead of panicking * Remove panics from coglet-python setup and mutex locks - worker_bridge: init_async_loop() and PythonPredictHandler::new() return Result - input: detect_runtime() returns Result, PreparedInput uses Py<PyDict> - log_writer: mutex locks use expect() with clear messages - predictor: handle detect_runtime Result propagation Worker mutex poisoning now panics with clear message - orchestrator handles worker exit and fails predictions appropriately. * Remove panics from webhook, routes, and server modules - webhook: WebhookSender::new() returns Result for HTTP client creation - webhook: Mutex locks use into_inner() to recover from poison - routes: Document timestamp unwrap safety (can't fail after UNIX_EPOCH) - server: Improve signal handler expect messages for clarity Callers handle webhook creation failure by logging and continuing without webhooks, which is acceptable graceful degradation. * Replace remaining panics with PyErr in coglet-python - input.rs: get_item().unwrap() -> ok_or_else(PyKeyError) for missing keys - log_writer.rs: OnceLock.get().unwrap() -> ok_or_else(PyRuntimeError) Both are now proper Python exceptions rather than Rust panics. * Add Rust CI workflow and stub generation - Add .github/workflows/rust.yaml for Rust CI (fmt, clippy, nextest) - Add mise tasks: stubs:generate, stubs:check, stubs:typecheck - Add mise tasks: ci:rust, ci:coglet for local CI simulation - Add scripts/generate_stubs.py to auto-generate coglet.pyi from module - Update coglet.pyi with all module exports (classes + internal functions) CI runs: - Pure Rust checks (fmt, clippy, tests) without Python - coglet-python checks on Python 3.9/3.12/3.13 (build, stub check, typecheck) * Apply cargo fmt formatting * Fix Rust CI: add cargo-binstall, uv sync for Python - Install cargo-binstall before mise so mise uses binstall for cargo tools - Add uv sync to setup Python environment before maturin build - Order: Rust -> cargo-binstall -> rust-cache -> mise -> uv sync * Fix CI: resolve checkout and virtualenv issues - Add #[allow(dead_code)] on AbstractSocketTransport.prefix (kept for debugging) - Add 'uv venv' step before maturin build in CI - Move virtualenv PATH setup to individual steps after checkout to prevent "Unable to locate executable file: tar" error during actions/checkout@v4 * Replace boolean flags with Rust enums (replicate#2643) * refactor: replace boolean flags with Rust enums Replace boolean soup with strongly-typed enums across four areas: - PredictorKind enum (Class/StandaloneFunction) replacing 5 boolean flags in predictor.rs (is_async, is_async_gen, has_train, is_train_async, is_standalone_function) - PredictionOutcome enum (Success/Failed/Cancelled) replacing success bool and error string matching in worker.rs - HandlerMode enum (Predict/Train) replacing is_train boolean in worker_bridge.rs - SlotState enum (Idle/SyncPrediction/AsyncPrediction) replacing 4 fields with state machine in worker_bridge.rs Benefits: - Invalid states impossible at compile time - Eliminates string matching for cancellation - Clearer semantics and self-documenting code - Easier to extend with new variants Addresses PR replicate#2641 review feedback. * fix: apply clippy suggestions Use derive(Default) with #[default] attribute instead of manual impl. * style: apply rustfmt formatting * test: update tests to use PredictionOutcome enum Replace direct field access (.success, .error) with pattern matching on the PredictionOutcome enum. * chore: rm deadcode, simplify start_sync_prediction logic Signed-off-by: Mark Phelps <mphelps@cloudflare.com> --------- Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * Add Rust coglet integration to CI tests (replicate#2644) * cog-dataclass: fix Rust server predictor configuration Fix http.py to use config.get_predictor_ref() instead of non-existent config.predictor attribute. This properly loads the predictor reference from cog.yaml for the Rust coglet server. Changes: - Use config.get_predictor_ref(args.mode) to get predictor ref - Add helpful error handling with example if predictor not configured - Exit cleanly with error code 1 instead of crashing * Enable Rust coglet server in integration tests Wire up the Rust coglet server (coglet-python) for integration testing alongside both cog and cog-dataclass Python SDKs. CI changes: - Build Rust coglet wheel (ABI3) in build-python job - Update test-integration matrix to 5 combinations: * cog (Python server) * cog-dataclass (Python server) * coglet-alpha (old Go server) * cog-rust (cog + Rust server) * cog-dataclass-rust (cog-dataclass + Rust server) - Set COG_WHEEL and COGLET_RUST_WHEEL appropriately per runtime Integration test harness: - Propagate COGLET_RUST_WHEEL environment variable to tests - Add conditions: cog_rust, cog_dataclass_rust, coglet_rust - Allow tests to conditionally run based on server implementation Dockerfile generation: - Prevent installing Rust coglet when using coglet-alpha - Add clear warning when both are specified - Do not install Rust coglet for monobase/fast builds Rust coglet: - Add SdkImplementation enum (Pydantic, Dataclass, Unknown) - Detect SDK type by checking for PYDANTIC_V2 attribute - Log SDK implementation at startup for visibility This allows us to test the Rust server with both SDK implementations and ensure compatibility across the matrix. * Clarify SDK detection: PYDANTIC_V2 exists in both v1 and v2 Add comment explaining that PYDANTIC_V2 attribute exists in both pydantic v1 (set to False) and pydantic v2 (set to True), but does not exist at all in dataclass-based cog. This means the presence of the attribute (regardless of value) indicates pydantic-based cog, while absence indicates dataclass-based. * Exclude Rust coglet wheel from Go binary embedding Only embed the Python coglet wheel (py3-none-any) in the Go binary. The Rust coglet wheel (cp38-abi3-*) is distributed separately and installed via COGLET_RUST_WHEEL environment variable at runtime. This prevents the "expected exactly 1 coglet wheel embedded, found 2" panic when both wheels are present in dist/. * Fix type checking errors in cog http.py for Rust coglet integration - Replace config.predictor with config.get_predictor_ref(args.mode) - Add proper error handling for missing predictor configuration - Add type: ignore comments for coglet attributes (conditional import) This matches the fixes already applied to cog-dataclass and resolves the pyright type checking failures in CI lint job. * Remove embedded Python coglet wheel to avoid package name conflicts The Python coglet wheel (py3-none-any) conflicts with the Rust coglet wheel (cp38-abi3) as both have package name "coglet". Removed Python coglet wheel completely: - Remove Python coglet wheel build from CI - Remove coglet wheel from Go binary embedding (only cog + cog-dataclass) - Remove installEmbeddedCogletWheel() function - Remove ReadCogletWheel() function - Remove WheelSourceCogletEmbedded switch case (falls through to error) Users should now use: - COG_WHEEL=cog or cog-dataclass (Python servers) - COG_WHEEL=coglet-alpha (Go implementation, downloaded from URL) - COGLET_RUST_WHEEL=path (Rust server, installed alongside cog/cog-dataclass) If someone sets COG_WHEEL=coglet, they'll get "unknown wheel source: coglet" error instead of a panic. * Remove coglet wheel test references - Delete TestReadCogletWheel (function no longer exists) - Delete TestCOGWheelEnvCoglet (embedded coglet removed) - Delete TestCOGWheelCogletWithPython39Succeeds (embedded coglet removed) - Remove WheelSourceCogletEmbedded from TestWheelSourceString - Remove "coglet keyword" and "coglet uppercase" test cases - Remove "env coglet overrides" test case Tests now only cover: cog, cog-dataclass, coglet-alpha, file paths, and URLs. * Fix Path handling for data URLs in cog-dataclass - Use Path.validate() instead of Path() constructor to properly convert data URLs to URLPath objects - Fix PredictionRequest.dict() to avoid asdict() serialization issues with URLPath objects This fixes path_input and path_list_input integration tests where data URLs were being passed as raw strings instead of being converted to file paths. * cog-dataclass: allow non-BasePredictor classes and add tests - Accept predictor classes without BasePredictor inheritance - Keep setup optional; validate if present - Add unit tests for permissive create_predictor behavior - Add integration tests for class/function predictors (skip coglet_alpha) * coglet: detect non-pydantic runtime via cog-dataclass inspector - Detect cog-dataclass predictors via cog._inspector.create_predictor - Use non-pydantic input processor with URLPath downloads - Generate schemas via cog-dataclass _schemas for non-pydantic runtime - Remove legacy coglet runtime detection path * cog-dataclass: accept local paths in Path.validate Keep URL inputs as URLPath while restoring local path normalization. * coglet: fix spurious error log during normal shutdown Send ControlRequest::Shutdown to workers before exiting to avoid "Control channel closed (parent died?)" error message during normal shutdown. Changes: - Add shutdown() to Orchestrator trait - Implement shutdown() in OrchestratorHandle trait impl - Add PredictionService.shutdown() to call orchestrator.shutdown() - Call service.shutdown() in HTTP server after graceful shutdown - Update MockOrchestrator test doubles with no-op shutdown() Before: Workers saw channel close and logged spurious error After: Workers receive shutdown message and exit cleanly * coglet: use manylinux2014 for Python 3.8+ compatibility Change from auto-detected manylinux_2_39 to manylinux2014 (glibc 2.17) to ensure the coglet wheel installs on Python 3.8 base images. * Linting fixes * Use rustls with native OS certificate stores Configure both HTTP clients to use operating system certificate stores: - reqwest: Use rustls-tls-native-roots feature (rustls-native-certs) - ureq: Use rustls + platform-verifier with RootCerts::PlatformVerifier This allows custom CA certificates to be supplied via the OS certificate store (e.g., /etc/ssl/certs) instead of being hardcoded at compile time. Trade-offs: - Requires ca-certificates package in base images - Explicit failure if certs missing (better than stale bundled certs) - No diamond dependency issues (avoided native-tls) - No unstable features (avoided rustls-no-provider) Note: webpki-roots remains in dependency tree via ureq's rustls feature but is not used at runtime due to explicit PlatformVerifier configuration. * Allow CDLA-Permissive-2.0 license in cargo-deny Add Community Data License Agreement Permissive 2.0 to the allow list for webpki-roots certificate data. This is a permissive license compatible with Apache-2.0, used for Mozilla's CA certificate bundle. * Fix coglet-python test task to install dependencies Use mise dir field and uv --with flags to install test dependencies (pytest, requests) before running tests. * CI: Add coglet-python tests and ensure rust+binstall before mise - Add test-coglet-python job to run Python binding tests - Install Rust toolchain and cargo-binstall before mise-action in all jobs to avoid compiling cargo tools from source (uses binstall instead) - Ensures correct order: rust -> binstall -> mise -> rust-cache * Fix coglet-python tests with dynamic port discovery and cog.yaml - Add cog.yaml files to test fixtures (required by both cog and cog-dataclass) - Implement dynamic port assignment using port 0 for tests - Add background thread to capture stderr logs for port discovery - Log actual bound port after TcpListener::bind() in coglet server - Update mise task to install dependencies in proper venv - All 12 coglet-python tests now passing * Require Python 3.10+ for coglet (matching cog-dataclass) - Update coglet-python requires-python from >=3.8 to >=3.10 - Change PyO3 ABI3 target from py38 to py310 - Update CI wheel paths from cp38-abi3 to cp310-abi3 - Matches cog-dataclass minimum Python version requirement * Skip integration tests with Python <3.10 or monobase for coglet (Rust) Add [coglet_rust] skip to integration tests that coglet cannot run: Python <3.10 (9 tests): - python37_deprecated.txtar (3.7) - optional_input.txtar (3.8) - setup_subprocess_multiprocessing.txtar (3.8) - apt_packages.txtar (3.9) - bad_dockerignore.txtar (3.9) - torch_baseimage_fallback.txtar (3.9) - torch_baseimage_no_cog_base.txtar (3.9) - torch_baseimage_precompile.txtar (3.9) - torch_cuda_baseimage.txtar (3.9) Monobase/fast build (7 tests): - build_base_image_sha.txtar - build_cog_version_match.txtar - build_localimage.txtar - build_python313_base_image.txtar - predict_localimage.txtar - fast_build.txtar - run_fast_build.txtar coglet (Rust) requires Python >=3.10 and does not support monobase. * CI: Use zig cross-compilation for Linux coglet wheel build Use build:coglet:wheel:linux instead of build:coglet:wheel to ensure manylinux2014 (GLIBC 2.17) compatibility. Building on modern Linux systems picks up too-recent GLIBC symbols (2.18-2.39) which breaks manylinux compliance. Zig cross-compilation targets the correct GLIBC version. * fixup! CI: Add coglet-python tests and ensure rust+binstall before mise Use astral-sh/setup-uv for proper venv management * Fix Secret serialization and skip pydantic2_output for cog-dataclass - Fix Secret.json_encode() to unwrap Secret objects before JSON serialization This fixes 'TypeError: Object of type Secret is not JSON serializable' when generating OpenAPI schemas with Secret input fields that have defaults - Skip pydantic2_output test for cog-dataclass/coglet_rust since it explicitly tests Pydantic 2 BaseModel features * Fix OpenAPI schema generation for training mode in coglet - Make cog-dataclass to_json_schema() mode-aware: - Accept Mode parameter (PREDICT or TRAIN) - Generate appropriate routes (/predictions or /trainings) - Use correct schema names (PredictionRequest/Response or TrainingRequest/Response) - Update coglet-python to pass HandlerMode through schema generation: - Pass mode from worker_bridge to predictor.schema() - Convert Rust HandlerMode enum to Python Mode enum - Pass mode to both pydantic (schema_via_fastapi) and cog-dataclass (to_json_schema) This fixes the "missing output definition for /trainings" error when using coglet with training endpoints, ensuring training has its own distinct Input/Output schemas based on the train() method signature. * Fix validation error messages to match pydantic format - Change 'missing required input field' to 'Field required' in cog-dataclass - Revert string_predictor.txtar test to expect 's: Field required' - Fix mise.toml to use 'python -m pytest' to avoid hardcoded shebang issues This ensures cog-dataclass validation errors match pydantic's format, making the integration test pass. * Fix subprocess output corrupting control channel in coglet When Python's setup() spawns subprocesses (subprocess.Popen), they inherit fd 1 (stdout) which is the control channel pipe to the orchestrator. Subprocess output writes directly into the pipe, corrupting JSON framing with raw text. This caused "frame size too big" errors when the bash subprocess in setup_subprocess_simple.txtar wrote 100 lines during setup. Solution: File descriptor redirection early in worker startup - Move control channel to high-numbered fds (99-101) - Replace fd 1/2 with capture pipes - Spawn threads to read pipes and route output through log system - Subprocess logs flow to coglet::user target (like orphan logs) - Log forwarder runs for entire worker lifetime Changes: - Add crates/coglet/src/fd_redirect.rs with fd manipulation - Update worker.rs to use redirected fds for control channel - Add RUST_LOG propagation in CLI commands (serve, predict, run) - Add RUST_LOG support in integration test harness - Add frame size and log forwarding trace logging for debugging - Route control channel logs to coglet::user target Safety contracts documented for all unsafe blocks. Passes clippy with no warnings. Fixes setup_subprocess_simple integration test. * Fix validation error format to show simple field messages Standardize validation error messages across cog-dataclass and coglet: - cog-dataclass: Change "missing required input field" to "Field required" - coglet: Parse Pydantic ValidationError to extract "field: message" format - Both now output simple "field: message" instead of verbose error details Fixes integration test string_predictor validation error matching. * Fix TrainingOutput serialization for Pydantic v2 models make_encodeable() now checks for model_dump() (Pydantic v2) before falling back to dict() (Pydantic v1). This fixes train() functions that return Pydantic BaseModel instances like TrainingOutput. Fixes train_basic integration test. * Fix validation error formatting in coglet to show simple messages Parse Pydantic ValidationError to extract simple "field: message" format instead of verbose error details. For ValueError (cog-dataclass), the message is already in the correct format. Includes clippy fixes for collapsible if statements. Fixes string_predictor integration test validation error matching. * Fix capture thread feedback loop and update to bounded channels Capture thread improvements: - Remove all tracing calls from capture threads to prevent feedback loop (stderr is redirected to capture pipe, so tracing would capture itself) - Update comments to reflect bounded channel (500 messages) instead of unbounded - Align safety comment with header documentation about tokio runtime threads Log writer updates: - Change SetupLogSender to use bounded Sender instead of UnboundedSender - Use blocking_send() for backpressure handling * Update cog-dataclass test to expect new validation error format Change test to expect "Field required" instead of "missing required input field" to match the updated error message format. * Fix blocking panic and add dropped log tracking Critical fixes: 1. Replace blocking_send() with try_send() in ControlChannelLogSender to prevent "Cannot block the current thread" panic when called from Python code on tokio runtime threads 2. Add DroppedLogs control message and periodic reporting (every 5s) to track logs dropped due to backpressure 3. Rename SetupLogSender -> ControlChannelLogSender with accurate documentation: - Lives for entire worker lifetime, not just setup - Handles both Python setup logs and subprocess output - Cleanup is no-op (sender persists) 4. Log dropped logs as warnings in orchestrator with human-readable intervals Fixes integration test panics during setup phase. * Ship worker tracing logs over IPC with preserved targets Add custom tracing layer for worker subprocess that ships structured tracing events over IPC to orchestrator, preserving component targets. Changes: - Add WorkerLog control message with target/level/message fields - Implement WorkerTracingLayer that intercepts tracing events before formatting - Initialize custom layer in worker with IPC channel sender - Orchestrator re-emits logs with original target using low-level tracing API - Optional fd 101 direct logging via RUST_WORKER_DIRECT_LOG=1 for debugging - Increased channel depth 500 → 5000 for trace-level logging support This replaces the previous approach where tracing went through capture pipes with ANSI codes, losing component information and appearing as generic 'coglet::setup' source. Note: Setuplog accumulation not yet implemented - logs are shipped and re-emitted but not added to SetupResult. Will be addressed in follow-up. * Prevent feedback loop in codec logging Filter out coglet::bridge::codec logs from IPC shipping to prevent infinite loop where encoding a WorkerLog message triggers codec trace logging, which creates another WorkerLog, which triggers more encoding, etc. Changes: - WorkerTracingLayer skips shipping logs with codec target - Added SAFETY comment in codec.rs explaining the feedback loop risk - Codec logs still visible via RUST_WORKER_DIRECT_LOG=1 for debugging * Accumulate coglet server setup logs via tracing layer Adds SetupLogAccumulator tracing layer that captures all logs from the coglet server process during the setup phase, from initial startup through worker Ready. Key implementation details: - Tracing layer installed before any logs emitted (captures "coglet <version>" log) - Accumulates orchestrator logs + re-emitted worker logs shipped over IPC - Critical ordering: accumulate → Ready received → drain → flip Ready state - Performance optimization: tx.is_closed() check prevents expensive ops after Ready - Channel semantics: unbounded for setup phase, drainable even after close - Log format: lines joined with \n separator, trailing newline at end The accumulated logs are returned in SetupResult.logs and exposed via /health-check endpoint for debugging setup failures. * Ensure 'Server ready' log is accumulated before stopping Keep the setup log receiver alive through the 'Server ready' log, then do a final drain to capture it before setting the setup result. The 'Setup complete, now accepting requests' log happens after the receiver is dropped, so it correctly does NOT appear in setup logs. Test verifies both behaviors: - 'Server ready' IS in setup logs - 'Setup complete' is NOT in setup logs * Stabilize coglet bindings runtime detection Split coglet bindings CI runs by runtime and align predictor ref parsing with cog loader semantics. - Parse predictor refs to use file stem instead of full dotted path (e.g., 'path/to/predict.py:Predictor' -> module='predict') - Split CI matrix to test both cog-dataclass and cog runtimes separately - Add mise tasks for testing each runtime independently - Ensures proper module import behavior matching Python's loader * Normalize coglet output serialization Handle dataclass BaseModel outputs and unwrap pydantic iterators before JSON encoding, plus rust-only setup log test guard. - Add dataclass conversion in make_encodeable() for BaseModel outputs - Unwrap pydantic serialization iterators in input processing and output serialization - Guard setup_worker_tracing_logs.txtar test for rust-only coglet - Add type safety check for numpy isinstance calls --------- Signed-off-by: Mark Phelps <mphelps@cloudflare.com> Co-authored-by: Mark Phelps <209477+markphelps@users.noreply.github.com>
* docs: reorganize architecture docs for FFI and legacy runtimes Restructure architecture documentation to clearly separate the two runtime implementations (legacy Python/FastAPI and experimental Rust/PyO3 FFI). Changes: - Move existing runtime docs to architecture/legacy/ - Create new architecture/ffi/ with comprehensive FFI documentation - Add README files for both runtime implementations - Update architecture/00-overview.md to explain both runtimes - Document FFI components: PredictionService, Supervisor, PermitPool - Document IPC protocol: control channel and slot channels - Document health state machine and prediction flows - Add migration notes and performance comparisons - Update docs/http.md and README.md to mention USE_COGLET option The FFI runtime provides significant improvements: faster HTTP layer, better worker isolation, slot-based concurrency, and automatic cancellation on connection drops. * docs: fix FFI architecture docs to match crate READMEs - Fix cancellation propagation to show sync (SIGUSR1 + CancelableGuard) vs async (future.cancel()) mechanisms instead of incorrect cancel_token - Fix control channel protocol messages and fields - Fix file structure (worker.rs not worker/ subdirectory) - Remove incorrect cancel_token from component ownership * chore: add rust to mise * chore: default profile mise? Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * docs: fix health-check response documentation per PR review - Health-check endpoint always returns HTTP 200 with status in JSON body - Clarify that 503/409 responses only apply to prediction endpoints - Mark worker restart as future potential behavior (not yet implemented) - Update mermaid diagram notes to be more accurate * chore: update AGENTS.md Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * chore: remove dupl rust install in ci Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * fix: explicitly install rustfmt and clippy components in mise The 'default' profile doesn't reliably include rustfmt in CI, causing 'cargo-fmt is not installed' errors. Explicitly specifying the components ensures they are always available. --------- Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
* chore: rip coglet Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * chore: update Makefile for Rust coglet, remove old Go coglet artifacts - Replace Go coglet targets with Rust coglet targets (cargo test/clippy/fmt) - Add Rust coglet to main fmt and lint targets - Remove coglet/ exclusion from test-go (no longer needed) - Delete script/build-coglet-server (Rust coglet builds via maturin) * chore: rm rust cache action for now * chore: try to bust mise cache Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * chore: one more mise cache buster Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * fix: handle variable-length git short hashes in version comparison test Git short commit hash length can vary (7-9+ chars) depending on core.abbrev settings and automatic expansion to avoid ambiguity. The version comparison test now checks that one hash is a prefix of the other, rather than requiring an exact match, while still verifying both refer to the same commit. --------- Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
* Allow user to define healthcheck function * Add integration tests to verify behavior * python 3.8 compatability fix * Accept suggestions from comments * Fix tests * test: cover healthcheck unhealthy statuses --------- Co-authored-by: Nikhil Sinha <nsinha@cloudflare.com> Co-authored-by: morgan fainberg <mfainberg@cloudflare.com>
* Use recently built image instead of latest for src * Recalculate hash to prove it is correct * Fix mock * Update mocks * Only do it if there's a r8.im model * Suggestions from comments --------- Co-authored-by: Nikhil Sinha <nsinha@cloudflare.com> Co-authored-by: Mark Phelps <209477+markphelps@users.noreply.github.com> Co-authored-by: mfainberg-cf <mfainberg@cloudflare.com>
Co-authored-by: SharkFishDeveloper <shahzebakhtart66@gmail.com> Co-authored-by: Mark Phelps <209477+markphelps@users.noreply.github.com>
* Remove reference to deleted script now that we don't need a separate step to build embedded binaries for the wheel. * Also skip wheel build for coglet since it is no longer here to build
* chore: add useragent support to cog-dataclass Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * chore: fix rust workflow temporarily to not use mise cache Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * chore: rm version.py Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * fix: remove cog-dataclass src before tests to use installed wheel This ensures tests import from the installed wheel which has _version.py baked in, rather than the source directory where _version.py doesn't exist (it's gitignored and generated by setuptools-scm during build). Matches the pattern used by the main test-python job. --------- Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
* Update FROM line to work with orbstack as well * Update image name to not have tag * Use tmp image instead * Remove now-unused error * Update mocks * Conditionally do tmp image stuff --------- Co-authored-by: Nikhil Sinha <nsinha@cloudflare.com>
* Fix integration tests * Comment update --------- Co-authored-by: Nikhil Sinha <nsinha@cloudflare.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…icate#2665) When pushing to r8.im/*, commit 2ad2c7f introduced temporary image naming to fix OrbStack compatibility, but missed updating three function calls that run containers to extract metadata: - GenerateOpenAPISchema - GeneratePipFreeze - GenerateModelDependencies These functions were still using the original imageName (e.g., r8.im/replicate/model-test) instead of tmpImageId (cog-tmp:<hash>), causing 'No such image' errors during the 'Validating model schema' step. This fix ensures all three functions use the temporary image ID that was actually built by Docker.
* Fix synchronous healthcheck by managing via a separate channel & event loop. This also removes `BasePredictor.healthcheck` since this makes detection of a user-provided healthcheck method weirder and less reliable than it needs to be, consolidates healthcheck code more into a single class, removes now-unused `event_type` from `Done`, and attempts to generally minimize the diff of total healthcheck code since v0.16.9. * Shore up integration tests around healthcheck --------- Co-authored-by: Nikhil Sinha <nsinha@cloudflare.com>
Sync the fork with replicate/cog v0.16.12 (previous baseline: v0.11.6),
preserving every CardinalBlue customization.
Conflicts resolved (11 files, 33 hunks), preserving fork behavior:
* http.py -- kept the Vertex-AI-style `instances`/`predictions` batch API on
top of upstream's Config/Mode refactor and lifespan handler. Dropped
upstream's /trainings routes, PUT /predictions/{id} and
POST /predictions/{id}/cancel again; kept /health/ready and /health/live.
Adopted upstream's index_document, listing only routes this fork exposes.
Adapted to new APIs: runner.predict() now takes `is_train`, and
_log_invalid_output() now takes `mode`.
* worker.py -- re-applied the Sentry capture_exception branches on top of
upstream's async worker rewrite, keeping the deliberate ordering where
PredictorInputError is NOT reported to Sentry.
* predictor.py -- kept the NOT_ALLOWED_INPUT_TYPES deny-list and nested
BaseModel recursion, while adopting upstream's is_union/is_optional helpers.
* json.py -- kept file upload disabled against upstream's new os.PathLike
branch.
* eventtypes.py -- kept error_type/http_status_code on Done alongside
upstream's new Envelope.
* types.py -- kept the __url__-first ordering fix (py3.13 __repr__ crash)
while taking upstream's `filename` parameter; de-duplicated upstream's
repeated init block.
* pyproject.toml / tox.ini -- kept requires-python >=3.9 and the py39-py313
tox matrix.
Tests: ported upstream's new tests to the batch API where they apply and
dropped the ones that exercise removed features. 242 passed, 1 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Upstream v0.16.10 added a branch at the top of `__main__` that tried
`import coglet` and, on success, handed the entire process to
`coglet.serve()` and exited, falling back to the Python server only on
ImportError. It arrived in the v0.16.12 merge through a non-conflicting
region.
The Rust server implements upstream's API, not this fork's. If `coglet` ever
became importable -- COGLET_RUST_WHEEL set at build time, a base image that
ships it, or a model requirement named coglet -- the server would start
normally and serve a completely different API: no instances/predictions batch
shape, no Sentry reporting, no /health/ready or /health/live, and no
400-vs-500 error semantics. The only signal would be an info-level
"Using Rust coglet server" log line.
Remove the branch entirely so the fork always serves its own Python app.
The Python argument parsing below it was already self-contained, so this is a
straight deletion with a comment recording why.
Verified by booting `python -m cog.server.http`: GET / advertises only the
routes this fork exposes, /health/ready and /health/live respond,
POST /predictions returns {"predictions": [...]} for a 3-instance batch, and
/trainings, PUT /predictions/{id} and POST /predictions/{id}/cancel all 404.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things the v0.16.12 merge left to finish: teaching upstream's new Pydantic
v1 nullable pass about this fork's models, and expressing every customization
the way cb-main expressed it.
update_nullable_optional is new in v0.16.12 -- it did not exist at the
v0.11.6 baseline -- and is the Pydantic v1 counterpart of
update_openapi_schema_for_pydantic_2. It looks for a body model with an
`input` property and a response model with an `output` property. This fork
serves `instances: List[Input]` and `predictions: List[Output]`, so both
lookups missed and the function returned having done nothing. Under Pydantic
v1 the generated OpenAPI therefore lost every `nullable: true` marker on
optional Input fields and on a nullable Output, which matters for consumers
that generate clients from the spec. It now reads our field names, which is
three string swaps against upstream:
- get_annotations(model).get("input")
+ get_annotations(model).get("instances")
- schema_node["properties"]["input"]["$ref"]
+ schema_node["properties"]["instances"]["items"]["$ref"]
- get_annotations(response_model).get("output")
+ get_annotations(response_model).get("predictions")
That is the same surgical edit PR #24 made to _flatten_selected_allof_refs.
Only our own shape is handled: the `input`/`output` fallbacks cannot be
reached, since /predictions is the only route with a pydantic body or response
model and the routes that would have used them are removed here. The
`*_model_union` locals become `*_model_list`, because `get_args(...)[0]` now
unwraps a List rather than an Optional. Covered by test_openapi_specification
and test_openapi_specification_with_optional_output_type, which previously
failed under Pydantic v1.
_flatten_selected_allof_refs keeps cb-main's four-line adaptation rather than
being deleted as inert. Against upstream that is +54/-10 instead of +49/-27,
and removing less upstream code is what matters for future syncs. Restoring
upstream's version verbatim is not an option despite being a zero diff: its
second block overwrites the /predictions requestBody with a dangling $ref to
PredictionRequest, which does not exist in this fork's components. Its first
block still reads properties["instances"] on the response model, where the
field is "predictions", so it raises KeyError into the surrounding pass and
does nothing -- left exactly as cb-main wrote it, since pydantic already emits
the correct ref for predictions: List[Output].
For everything else the rule is to follow `git diff 8333a83 18bbf28`: where
cb-main deleted, delete; where it commented out, comment out. cb-main deleted
whole features -- the /trainings family, PUT /predictions/{id}, the cancel
route -- and commented out inline logic inside functions it kept: the prefer
header, respond_async, task_kwargs, the input cleanup callback, the
upload_files call, the Path/IOBase branches in json.upload_files, the
path/image params in openapi_complex_input, and the bodies of tests for
features it dropped. All of the latter stay commented.
* Drop the [cb] paragraphs restating what an adjacent commented-out block
already said; they were redundant and had started to drift. Keep the one
note that earns its place, on why the Rust coglet delegation is absent.
* Commented-out code tracks upstream rather than freezing at v0.11.6: the
disabled branch in json.upload_files reads os.PathLike/open(obj) after
upstream's rename, the commented runner.predict alternative carries the new
is_train argument, and the commented test_input_with_unsupported_type body
matches upstream's reformatted assertion.
* Restore the commented CogFile/CogPath/CogSecret entries in
ALLOWED_INPUT_TYPES, which cb-main left as markers of what it moved to
NOT_ALLOWED_INPUT_TYPES.
* GET / uses upstream's index_document with only the two entries for routes
this fork does not expose commented out -- the same rule applied to
upstream's new code. It replaces cb-main's older inline dict, which predates
index_document existing. readiness_url and liveness_url stay out: they are
this fork's own routes, not upstream entries, so listing them would be an
addition rather than a removal.
Verified the full OpenAPI document for 10 fixtures is byte identical to before
these changes. 242 passed / 1 skipped on Pydantic v2, 240 passed / 4 skipped
on v1. ruff check and ruff format --check both clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chun-wei-yu-cardinalblue
force-pushed
the
feat_upgrade_version_to_0.16.12
branch
from
September 2, 2026 09:07
2b98e50 to
fd2976d
Compare
chun-wei-yu-cardinalblue
changed the title
chun-wei-yu-cardinalblue
marked this pull request as ready for review
September 3, 2026 09:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrade the fork to upstream v0.16.12
Ports every CardinalBlue customization from the v0.11.6 baseline onto upstream v0.16.12, preserving behavior exactly.
8333a83)18bbf28)ba5b1a9)fd2976d)How this was reviewed
The core of the review is a manual, line-by-line eyeball comparison of these two diffs:
The first is the original customization.
The second is the same customization re-expressed on the new baseline.
Both were read side by side, and the second was required to match the first file by file
Verification per customization
Prerequesite
Install hop_cog_0.16.12
Run
hop buildusinghop_cog_0.16.12Run it locally
1. Batch
instances/predictionsAPI2. Replicate-platform features removed
3. Custom error semantics (400 vs 500 +
error_type)Tests:
tests/server/test_http_output.py.4. Sentry integration
5. Kubernetes health probes
6. Complex input types
Tests:
tests/server/test_http_input.py.7. OpenAPI schema
additionalProperties: truein openapi.jsonNotable merge decisions
The Rust coglet delegation is removed (
5fd72cd).