status: preserve clean-status proofs across mixed Git writers - #21
Open
ttaylorr-oai wants to merge 145 commits into
Open
status: preserve clean-status proofs across mixed Git writers#21ttaylorr-oai wants to merge 145 commits into
ttaylorr-oai wants to merge 145 commits into
Conversation
GitHub exposes manual dispatch only for workflows present on the default branch. The controller itself must remain on the orphan meta branch so master stays identical to upstream. Add the fixed caller as a dedicated active topic. It delegates to the meta-pinned reusable workflow without putting controller code on codex.
The reusable controller reads its publication key from the codex-publish environment. The default-branch trampoline therefore needs only read access to Actions and repository contents; it no longer forwards repository secrets. Keep this exact workflow in the automation topic so the Actions page can dispatch the controller pinned to meta.
Codex rebuilds already call a trusted workflow on meta, but their dispatch-only trampoline cannot run pull request or merge-queue checks. Topics could therefore reach codex without verifying their review, branch ownership, or published base. Run the pinned admission workflow for pull requests targeting codex and for merge-group checks. Keep rebuild preparation limited to explicit workflow dispatch, and grant the admission job only read access.
The default-branch trampoline only listened for pull requests against codex and ran one admission job for every event. A preview pull request would therefore have no required check, and merge groups could not distinguish the production and preview lanes. Listen for both generated outputs, keep the existing production job and check context unchanged, and add a target-specific preview job that calls the trusted meta admission workflow with read-only permissions.
The release workflow cross-compiles Linux arm64 on an x64 runner and skips the smoke test for arm64 POSIX bundles. That prevents the workflow from executing the Linux artifact it just produced. Run Linux arm64 on GitHub's arm64 runner and install native development packages rather than configuring a foreign dpkg architecture. All matrix entries can then run the existing distribution smoke test.
Codex consumes Git release artifacts built with the Makefile's default -O2 flags. The release job compiles each artifact without link-time optimization. Add a release-only config.mak.openai and copy it into Git's ignored config.mak slot before building. Use thin LTO for Clang targets and automatic LTO for GCC targets, then check GIT-CFLAGS records the selected flag in every distribution job. Keeping the setting in config.mak.openai avoids carrying release-only policy in the upstream Makefile.
LTO can optimize across translation units, but the release job has no execution profile for the status, diff, clone, fetch, and repack paths Codex invokes frequently. Git's built-in profile target runs the 1,048-script test suite serially. That is too expensive for every release target and weights test-harness paths more heavily than the local workload. Extend config.mak.openai with GCC and LLVM profile modes. Gate GIT-CFLAGS on an instrumented build, run a short offline trainer, merge LLVM raw profiles when needed, and rebuild with profile-use flags. Each matrix entry runs on its target architecture, so it can execute the instrumented binary. Check that final GIT-CFLAGS includes a profile-use flag and increase the timeout for the second compilation pass. The focused trainer took about 30 seconds locally; the full macOS build/install validation completed with thin LTO and LLVM profile-use enabled.
Integrate the current tb/codex/automation topic into the internally distributed codex branch. Codex-Integration: tb/codex/automation@17738e2cc87ba67ed36cd1ffde983d43e01a5f41
Integrate the current tb/codex/geometric-maintenance-promisor topic into the internally distributed codex branch. Codex-Integration: tb/codex/geometric-maintenance-promisor@dc2fffc37cead551f8036c9ecab5e52a4cbee37b
Integrate the current tb/codex/release topic into the internally distributed codex branch. Codex-Integration: tb/codex/release@ba107e0ae8c7142238bb612e530d51d42f0280d3
Integrate the current dr/codex/dugite topic into the internally distributed codex branch. Codex-Integration: dr/codex/dugite@988cecced01f69765d599a2d6c023406af98fa1b
Integrate the current tb/codex/lto-pgo topic into the internally distributed codex branch. Codex-Integration: tb/codex/lto-pgo@88fcb4ac12c583bedf010e97ebf83cec240e3120
Clearing CE_FSMONITOR_VALID is not enough to make a provider event authoritative. With core.trustctime disabled, core.checkStat set to minimal, and a restored modification time, stat matching can still accept changed file contents. The same stale match can affect diff, apply, checkout, and unpack-trees. Mark a reported entry with the in-memory CE_CONTENT_CHECK_REQUIRED flag, clear CE_UPTODATE, and discard its cached stat data. Route diff, apply, checkout, and unpack-trees comparisons through ie_match_stat_with_content_check(), which calls ie_modified() only for marked non-gitlinks. Other direct ie_match_stat() callers retain their existing paths. Marking an entry up to date clears the transient flag. Ordinary entries, gitlinks, and unmarked zero-stat entries retain their existing stat behavior. Add hook regressions for restored timestamps, diff and status, indexed apply, checkout, case-insensitive unpacking, unchanged reset, and ordinary zero-stat behavior in t/t7519-status-fsmonitor.sh. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A filesystem-monitor provider can know that its event history is incomplete without being able to identify every affected path. Treating such a response as an ordinary path leaves tracked entries, cached attributes, and untracked-cache state falsely valid. Reserve // as a provider-only global invalidation record. It cannot collide with a worktree-relative path. When the client receives it, discard cached attribute stacks and untracked-cache state, invalidate every tracked entry, and mark the fsmonitor extension changed. Recognize the existing trivial response only when a complete record consists of a single slash and NUL, newline, or carriage-return terminators. This prevents the new double-slash record from being discarded as a trivial response while preserving existing hook forms. Add a hook regression in t/t7519-status-fsmonitor.sh that changes a tracked file, restores its timestamp, emits the global marker, and requires status to report the change. Global invalidation intentionally scans the tracked index. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An implicitly started fsmonitor daemon inherits its caller's repository environment and current directory. In a linked worktree, inherited Git directory, worktree, common-directory, prefix, and index settings can make the child discover a different repository than the worktree whose status requested the daemon. Resolve the requested worktree to its canonical path, start the child from that directory, and remove repository-addressing variables from its environment. Keep the existing daemon start command and return an error if the worktree cannot be resolved. Add a macOS regression that implicitly starts fsmonitor from a linked worktree and checks the daemon child's working directory in Trace2. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A pathname monitor cannot establish that every name for a multiply-linked regular file lies inside its watch cone. Persisting CE_FSMONITOR_VALID after checking the tracked name can therefore hide a later write through an unmonitored hardlink. Use fsmonitor_stat_can_be_valid() to exclude regular files with more than one link from persistent fsmonitor validity when the platform reports real link counts. Apply that decision where index refresh, threaded preload, and diff-files first consume an actual stat. Preserve CE_UPTODATE for the current process and retain existing persistent validity for single-link and nonregular entries. Windows and Cygwin synthesize their link counts, so preserve their existing fsmonitor behavior without claiming the hardlink guarantee there. Add a hardlink regression in t/t7519-status-fsmonitor.sh on platforms with trustworthy stat metadata. It keeps a tracked hardlink outside the fsmonitor-valid bitmap and checks that a write through an alias outside the worktree appears in status. The deliberate cost is another stat in a subsequent process. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Implicit fsmonitor startup resolves a Git command through the execution path and invokes its start subcommand. An overridden execution path can therefore select a different Git than the dispatcher that initiated the query, while adding another launcher between the client and daemon. Retain the absolute executable path during dispatcher initialization and expose it only for a real Git dispatcher. Start that executable directly with fsmonitor--daemon run --detach, then wait until its IPC socket is listening before accepting startup. Respect the configured startup timeout, defaulting to 60 seconds, and retain Git-command lookup when an authoritative dispatcher path is unavailable. The canonical worktree and sanitized environment established by S03/P01 remain in place. Update existing startup Trace2 checks for the direct invocation and add a macOS regression with a fake Git on the execution path to verify that the original executable is used. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Darwin FSEvents identifies the pathname associated with a hardlink event, not every name referring to the same inode. Invalidating only that pathname can leave another tracked hardlink trusted after its contents change. Classify the event's absolute path before handling its hardlink flags. For worktree events, enqueue the provider-wide marker introduced by S04/P02 so clients content-check the tracked set. Leave gitdir events in the existing cookie and gitdir handling; otherwise reads of hardlinked object files could repeatedly trigger global invalidation. Add a MACOS,HARDLINKS daemon regression that rejects a marker for a gitdir hardlink, then verifies the marker and correct status for a changed worktree hardlink with its timestamp restored. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The fsmonitor.startTimeout setting controls how long a client waits for daemon startup; the daemon's run subcommand does not consume it. Nevertheless, daemon configuration parsing validates that setting for every subcommand. A malformed value can consequently kill an implicitly started daemon before it opens its IPC socket. Pass a run-specific configuration flag into the callback and skip startup-timeout parsing only for run. Continue parsing other daemon settings normally, and preserve strict timeout validation for the explicit start subcommand. Add a macOS regression that verifies implicit status still starts the daemon with a malformed timeout while explicit daemon start rejects the same configuration. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Changing a .gitattributes file can change how tracked content is converted without changing the tracked file's stat data. Invalidating the attribute-file path alone therefore leaves cached conversion state and affected fsmonitor-valid tracked entries falsely reusable. Recognize an exact .gitattributes basename in the refresh callback. Discard cached attribute stacks globally and strongly invalidate only tracked entries beneath that file's parent directory. A root attribute file invalidates all tracked entries; tracked entries in sibling directories remain valid after a nested attribute-file event. Mark the fsmonitor extension changed only when an entry is invalidated. Add Clar unit coverage for unrelated paths, nested-directory scope, root-directory scope, cleared validity, zeroed stat data, and the content-check marker. Register u-fsmonitor-attributes in both Makefile and t/meson.build so the suite is included in both build systems. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An fsmonitor socket is selected through the Git directory, so separate worktree paths can reach the same daemon when they share that directory. A client in the second worktree can then consume change history from a daemon that watches the first, incorrectly treating changed files in its own worktree as clean. Hash the canonical worktree path together with its device and inode, plus birth time and generation on Apple platforms. Cache the resulting 64-character SHA-256 identity in the daemon and attach it to every client query. Check the identity before interpreting the requested token; reject missing or mismatched bindings with a cookie-synchronized trivial response that forces the ordinary refresh path. The protocol change must also tolerate a daemon left running by an older Git. Such a daemon treats a bound query as an opaque token and can return a plausible trivial response. After that exact response, query an unbound capability command. If the daemon does not advertise query-v1, serialize replacement through a per-socket restart lock, stop it, and start the invoking Git executable before retrying the bound query. Keep quit, flush, and capability control commands unbound. Bound daemon lifecycle retries, and fail the query instead of trusting history when the root cannot be identified or an incompatible daemon cannot be replaced. Regression tests cover shared-gitdir worktree aliases, replacement of a legacy daemon, and acceptance of a daemon that advertises a capability superset. The replacement test also verifies that the next status neither refreshes tracked entries nor starts another daemon. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A provider can report a directory move or modification without naming a changed .gitattributes file beneath it. Existing directory handling invalidates tracked entries in the reported cone but can leave cached attribute stacks describing the old conversion rules. Discard cached attribute stacks only after directory handling matches at least one tracked index entry. Record semantic/attributes-cone with the number of matched entries. An unmatched directory keeps its existing case-correction and untracked-path fallback without speculatively flushing attribute state. Extend t/helper/test-read-cache.c to cache an old attribute, process a directory event, and require the new attribute value. Add hook regressions in t/t7519-status-fsmonitor.sh for both an indexed cone and an unmatched directory, including their distinct Trace2 behavior. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
load_index_extensions() enters its extension loop only when a complete eight-byte header fits before the trailing checksum. It nevertheless trusts the declared payload size. An oversized payload can send an extension parser beyond the mapped extension area, while an incomplete trailing header is silently ignored. Compute the checksum boundary once and require the initial offset, each complete header, and each declared payload to fit within it. Advance only by checked header and payload sizes, reject partial trailing headers, and report framing failures as index file corruption. Add a PERL_TEST_HELPERS regression test that overwrites an FSMN payload length with 0xffffffff and checks that porcelain-v2 status fails with the existing corruption diagnostic. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
read_one() allocates a subtree array even for leaf cache-tree nodes. It also inserts each serialized child through cache_tree_sub(), which searches children that the writer already emits in increasing order. Allocate a child array only for non-leaf nodes and append increasing child names directly. Retain subtree_nr + 2 pointer slots for each non-leaf, but allocate them without zeroing because only populated slots are inspected. Keep cache_tree_sub() as the compatibility fallback for older, unsorted input. Existing t/t0090-cache-tree.sh tests exercise ordinary cache-tree decoding. This change adds no dedicated unsorted-input regression or isolated benchmark. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
add_patterns() rejects pattern files larger than 100 MiB only after allocating and reading their complete contents. An oversized filesystem input can therefore exhaust the memory the limit is meant to protect, or terminate Git when GIT_ALLOC_LIMIT rejects the allocation. Check the size obtained from fstat() before allocating a filesystem pattern buffer. Preserve the existing warning, close the descriptor, and return the existing failure result. Keep the later size check for index-backed fallback data, whose size is unavailable before it is read. Strengthen the existing EXPENSIVE regression by reading its 101 MiB .gitignore under GIT_ALLOC_LIMIT=1m. The old ordering dies in xmallocz(); the early rejection preserves the expected warning without attempting the oversized allocation. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The index extension worker decodes TREE and UNTR serially even though their parsers read the same immutable mapping and publish to different index_state fields. An unconditional additional worker would consume cache-entry workers and interfere with split-index assembly. Use the bounded framing from S02/P01 to select exactly one TREE and one UNTR extension. Require extension-offset metadata and at least four index workers; start an additional TREE worker only when both payloads reach 1 MiB. Leave at least two cache-entry workers available and join the TREE worker before unmapping the index. Keep LINK, duplicate or missing extensions, insufficient workers, small payloads, and auxiliary-worker creation failures on the existing serial path. Malformed framing still reports index file corruption. Allow GIT_TEST_PARALLEL_INDEX_EXTENSIONS to bypass only the payload threshold. Add a PTHREADS, UNTRACKED_CACHE, and SHA1 regression that compares parallel and serial status, cache-tree, and untracked-cache results and checks the extension/parallel/tree-untracked Trace2 marker. The regression unsets GIT_TEST_SPLIT_INDEX because split indexes intentionally remain on the serial path. The eligible path adds one auxiliary worker and its stack. The benchmark covers the complete series, not this patch in isolation. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
ttaylorr-oai
force-pushed
the
tb/codex/status-preview-unstable
branch
2 times, most recently
from
August 6, 2026 20:03
f26a55f to
c1113e5
Compare
ttaylorr-oai
force-pushed
the
tb/codex/status-preview-unstable
branch
from
August 11, 2026 04:27
6c58b4d to
52558de
Compare
dreynaud-oai
approved these changes
Aug 11, 2026
A pathspec prevents status from closing its fsmonitor token, so every scoped invocation invalidates the attribute manifest and rewrites the index. Repeated commands like "git status -- api" consequently rescan the entire worktree metadata. Allow scoped status to close and checkpoint global semantic history. Validate the complete untracked cache when establishing that proof, but discard its unfiltered results and collect the requested pathspec separately. Keep clean-worktree sidecars restricted to root-wide queries so dirt outside the selected paths cannot be hidden.
ttaylorr-oai
force-pushed
the
tb/codex/status-preview-unstable
branch
3 times, most recently
from
August 11, 2026 17:59
bba14fa to
edf431f
Compare
dreynaud-oai
approved these changes
Aug 11, 2026
ttaylorr-oai
force-pushed
the
tb/codex/status-preview-unstable
branch
3 times, most recently
from
August 11, 2026 20:15
43ac706 to
ecd090d
Compare
A root-wide clean proof also proves that a literal scoped status is clean, but scoped status must not create a new root-wide proof. Reuse an existing clean sidecar and render the current branch and HEAD state. Dirty scoped status bypasses the untracked cache because directory traversal disables it for non-empty pathspecs. Reuse the selected cached subtree after checking its builtin fsmonitor token, directory flags, exclusion identities, and expanded index. An fsmonitor event below a directory containing tracked entries cannot make that directory an untracked collapsed parent. Keep its cached ancestors valid and mark their recursive proofs stale. For an ordinary file event, retain the authenticated directory contents and reconcile only that path against its ignore rules instead of reopening the entire directory. Recompute proofs only along the affected ancestor path. Accept a legacy exclude identity containing the parser's synthetic newline when the actual file still matches the indexed blob. Otherwise the next root status invalidates and reopens its entire cached tree. Preserve ordinary invalidation for index additions and removals. Fall back to ordinary traversal for directories, changed exclusions, complex pathspecs, sparse indexes, and unsupported directories.
Bound fsmonitor queries prevent a shared Git directory from borrowing events from another worktree. An older daemon cannot interpret those queries, though, and replacing it discards the index token and all existing event history. Concurrent legacy clients can also recreate the socket before the replacement observes the original daemon exit. Authenticate a legacy Unix-socket peer against its effective user and watched worktree before replaying the existing token. Verify the daemon's open root on macOS and its root inotify watch on Linux. Cache successful checks under the canonical root, peer identity and start time, and socket generation so large Linux watch lists are read once. Track socket generations when an incompatible daemon must be replaced. The untracked-cache identity also changed between versions, causing add_untracked_cache() to discard the legacy directory tree before the daemon can be authenticated. Retain only the matching older identity until a nontrivial response proves the daemon watches this worktree; then upgrade the identity, preserve invalid directory frontiers, and close the existing forward-baseline proof. Keep ordinary invalidation for unmatched roots, missing tokens, weak stat settings, and failed authentication. Linux system Git without daemon support instead writes the placeholder token "builtin:fake" while retaining the legacy directory tree. That token is not a usable event boundary. When an authenticated daemon returns a full invalidation for it, preserve the old cache identity and validate tracked entries and directory timestamps normally. Avoid semantic fast paths, private FSUC/FSCF index extensions, and optional rewrites of an otherwise unchanged shared index during this fallback. Continue accepting unbound token requests from older clients once a bound-aware daemon is running. Mixed-version clients can therefore share one correctly identified daemon without restart loops or whole-worktree cache rebuilds.
The APFS bulk-preload race tests pause status until the test driver writes a byte to a resume FIFO. The child currently publishes its ready file before it opens the FIFO. If it is descheduled between those operations, the parent can observe readiness, write and close its descriptor, and discard the byte before a reader exists. Status then blocks forever in strbuf_read_file(), leaving a macOS CI job apparently hung. Open the resume FIFO first and read from that descriptor after publishing readiness. The parent opens the FIFO read/write before starting status, so the child open cannot block. Readiness now proves a reader is attached, and the resume byte cannot be lost. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Exercise pathspec-scoped status with selected untracked files and dirt outside the selected directory. Ensure a repeat query retains semantic history without rescanning metadata, rewriting the index, or creating a root-wide clean proof.
A clean path observed by status can remain uppercase in a subsequent ls-files invocation when external history avoids rewriting the main index. A late directory event can produce the same representation. Accept either fsmonitor marker while continuing to verify case-alias events and the final modified-path results. This removes a macOS CI failure without forcing an otherwise unnecessary index rewrite.
Exercise repeated tracked-directory queries with root and nested working directories after builtin fsmonitor proves the selected untracked-cache subtree is closed. Require selected files to remain visible, outside files to stay hidden, and the index to stay untouched. Create and remove an untracked child after the initial root-wide cache population. Verify each scoped query reports the correct result, visits exactly one path, opens no directory, and leaves the subsequent ordinary root status clean. Include tracked root and scoped ignore files and reject a subsequent root-wide ignore invalidation. Spell the adjacent scoped-history assertion as the lint-approved negated test_grep invocation.
ttaylorr-oai
force-pushed
the
tb/codex/status-preview-unstable
branch
from
August 11, 2026 20:47
ecd090d to
dc5c415
Compare
dreynaud-oai
approved these changes
Aug 11, 2026
ttaylorr-oai
force-pushed
the
codex-unstable
branch
from
August 11, 2026 22:24
7c7960a to
ed4c48a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The APFS bulk-preload test barrier published its READY file before it
opened the resume FIFO. If the child was descheduled at that point, the
parent could observe READY, write and close the FIFO, and lose the byte
before any reader existed. The child then blocked forever while macOS CI
appeared hung.
Open the resume FIFO before publishing READY. The parent already holds
the FIFO open read/write before it starts status, so the child open does
not block; READY now proves that a reader owns the resume descriptor.
The previous topic tip is already published as
v2.55.0-openai.619.gb264c6f26648, so this correction is additive on topof that released history rather than rewriting it. Its resulting tree is
byte-identical to the separately audited owner-amended replay.
Validation:
preload_index_bulk_darwinunit suite (6/6)122/122 ordered parents