rdma: expose live RC sessions on the admin server - #2365
Open
potatogim wants to merge 8 commits into
Open
Conversation
The hipobj-rc-v2 data plane started with its session, queue pair, staging and timeout limits hardcoded at the rcserver.Init call site, so operators could not size the RC plane for their hardware the way they can for the cuObject backend. Add one flag per limit plus the READY admission slot count, all defaulting to the values the gateway passes today, and validate them through a new rdmamode.V2ValidationError consulted only when the RC data plane is enabled, mirroring the stale-value handling of the v1 settings. Counts are parsed as uint64 and range-checked against the uint32 narrowing at the DeviceOpts boundary, and the timeouts carry an upper bound that keeps the nowMs + timeout deadline arithmetic in the C core from wrapping.
Wire C-side diagnostics (session reap, READY data phase outcome, init failures) through a sink callback so the gateway can surface them next to its own logs instead of losing them in stderr noise. The sink is a plain C function pointer installed once after init and valid until destroy: the Go side registers a fixed cgo trampoline (closures cannot cross the boundary), copies the message immediately per the lifetime contract, and never runs under the session map lock. Error-level lines keep the existing stderr output; --debug enables the level-2 diagnostic stream.
The verbs loader and device enumeration failures returned RC_E_INTERNAL without any stderr trace, which made a VM or container without RDMA indistinguishable from a genuine library problem. Print the failing step so operators can tell the two apart at startup.
ibv_poll_cq, ibv_post_send, and ibv_post_recv stopped being exported library symbols in modern rdma-core: verbs.h ships them as static inline wrappers that dispatch through cq->context->ops. dlsym therefore returned null for them and the loader rejected perfectly usable libraries, failing RC server init with a bare RC_E_INTERNAL on hosts with rdma-core 61+. Open the first device briefly, read the three function pointers from its context ops table, and close it again. The check now only requires symbols that actually exist in the library, and the failure mode for an ops-less provider is explicit.
The Prepare and ReadyTransfer wrappers embed string views built from Go heap strings inside request structs passed to C by pointer. The cgo pointer check rejects such requests when the string data is an unpinned Go heap pointer, so any live PREPARE or READY call with header-derived strings panicked at the call boundary and the route returned a 500. Constant strings passed the check because their data lives in read-only static storage, which is why standalone callers kept working while the gateway did not. Pin the string bytes with runtime.Pinner for the duration of the cgo call and drop the now redundant KeepAlive calls in those two wrappers. The other string-taking wrappers pass rc_str_in by value and are unaffected. Also add a deviceless cgo boundary regression test that calls the real Prepare wrapper with heap-backed interior-pointer strings and an invalid opcode, so C returns from argument validation before the server handle is touched.
The session id only existed as the sessions map key; the session record itself kept an empty id string, so the terminal reap record logged an empty id for every expired, cancelled, or destroyed session. Copy the id into the record at creation time so teardown logs identify the session they describe.
Expose rc_server_sessions_snapshot, which copies every live session into fixed rc_session_snapshot records under the map lock and invokes the callback once per record outside the lock. Each session records a monotonic creation timestamp, because the prepare/ready deadlines move as the session progresses and cannot serve as an age reference. The state byte combines the session state machine value with a reap-pending marker, so callers can distinguish sessions that are about to be reaped from healthy ones. Records whose op or target does not fit the fixed fields are skipped rather than truncated.
Collaborator
|
This won't automatically run in continuous integration without approval. A member of the Versity organization must allow it. |
Add a SessionsSnapshot view over the new C ABI entry point and serve it from the admin server as GET /rc-sessions. The admin server gains a WithAdminRoute option so an embedding binary can register extra admin routes that run with the same signature verification and admin checks as the built-in endpoints; vgwrdma registers the snapshot there when the RC feature is enabled. The route replies with the usual XML error surface so unsigned or non-admin requests get a 403 rather than a generic 500. Stub builds return a not supported error, keeping the build matrix unchanged.
potatogim
force-pushed
the
rc-parity-pr3
branch
from
September 7, 2026 07:29
a971cc1 to
6eb574d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #2361.
Builds on #2363.
Adds live RC session observability:
GET /rc-sessionson the admin server returns a point-in-time JSON snapshot of every session in the RDMA control-plane map.The C ABI gains
rc_server_sessions_snapshot(), which copies each live session into a fixedrc_session_snapshotrecord under the map lock and invokes the callback once per record outside the lock, so the callback cannot race the reaper moving or erasing entries. Each record carries the session id, op, target, state machine value combined with a reap-pending marker (CANCEL and expiry only set that marker, they do not move the state), age in milliseconds, and staged bytes. Age comes from a new monotonic creation timestamp on the session, because the prepare/ready deadlines move as the session progresses and cannot serve as an age reference. Records whose op or target does not fit the fixed fields are skipped rather than truncated.On the Go side an
RCSvc.SessionsSnapshot()wrapper marshals the records through the same fixed-trampoline pattern as the log sink, and the admin server gains aWithAdminRouteoption so an embedding binary can register extra admin routes.vgwrdmaregisters the snapshot route when--rdma-rc-enableis set, running it behind the same signature verification and admin checks as the built-in admin endpoints, so unsigned or non-admin requests get the usual 403 XML error rather than a generic 500. Stub builds return a not supported error, keeping the build matrix unchanged.Verified on a host with rdma-core 61 and a passthrough mlx5_0 VF: archive rebuild, full vgwrdma link, stub and unit tests, and an end-to-end run where a signed
POST /.hipobj-rc/preparecreates a session and the snapshot endpoint immediately returns it (id,op=GET, target,state=prepared, staged bytes), while the same request without a signature is rejected with 403.