Skip to content

feat(audit): 请求审计中新增不同账号调度详情、完整HTTP请求/响应报文、默认7天全量日志记录等完整诊断生命周期重要功能 - #983

Merged
chenyme merged 5 commits into
chenyme:mainfrom
jeikl:feat/request-audit-payloads-and-retention
Aug 20, 2026

Conversation

@jeikl

@jeikl jeikl commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

📌 概述 (Overview)

完善网关请求审计体系,提供客户端完整请求报文、服务端响应报文(含 SSE 流式自动重组)的持久化与可视化诊断能力,修复上游流式中断时的尝试诊断记录,并新增跨数据库兼容的日志保留策略与独立设置面板。


image image image

✨ 主要功能与变更 (Features & Improvements)

1. 客户端完整 HTTP 请求报文捕获与查看

  • 报文完整捕获:完整抓取客户端发起的 HTTP 请求行、所有自定义 Header / User-Agent 及完整 Body(支持文本与多模态数据)。
  • Markdown / JSON 可视化:审计详情中新增「请求报文」面板,支持原生 HTTP 报文高亮显示,并提供一键「复制 JSON」与「复制完整 HTTP 报文」功能。
  • 放开成功请求限制:移除了此前 200 成功状态码禁止点击查看详情的限制,方便排查正常调用的账号与报文行为。

2. 服务端响应报文捕获与 SSE 流式自动组包

  • SSE 流式聚合重构:针对 Chat Completions、Anthropic Messages 及 Responses API 的流式输出,在流结束时自动聚合 Delta Chunks 并重构成标准的合法 JSON 响应报文;非流式直接捕获响应 JSON。
  • 响应报文面板:新增「响应报文 (Response Body)」选项卡,展示模型完整输出、Token 统计与大小,支持复制原始响应与 Markdown 代码块。

3. 修复上游流式中断诊断回溯

  • 修复了当上游先返回 200 OK 响应头但随后中途断开(如 upstream_stream_interrupted / upstream_stream_incomplete)时,网关未记录尝试诊断导致后台显示“在发送到上游前失败”的问题。
  • 异常结束时自动补录 response_stream 诊断快照,准确记录所调用的账号、节点、状态码及底层网络 Transport 错误。

4. 独立「日志设置」面板与定时自动清理

  • 将请求审计配置从“运行策略”中拆分,设立独立的 「日志设置 (Audit & Logs)」 面板。
  • 日志保留天数 (retentionDays):支持自由配置日志保留时长(默认 7 天,设为 0 永久保留)。
  • 报文采集开关:提供 recordRequestBody(记录请求体)与 recordResponseBody(记录响应体)独立开关。
  • 自动清理机制:后端引入定时任务,跨 SQLite、MySQL、PostgreSQL 数据库均使用标准时间戳安全清理过期审计与诊断数据。

🔍 影响与兼容性 (Compatibility)

  • 数据库兼容:通过 GORM AutoMigrate 自动迁移 response_body 字段,跨 SQLite / MySQL / PostgreSQL 存储源无缝兼容。
  • 性能友好:保持了原有的 16384 队列缓冲与 256 批量写入机制,SSE 累加器限制最大 512KB 防御内存溢出,不影响并发吞吐。
  • 全量测试通过:后端全部单元测试(go test ./...)与前端静态类型编译(npm run build)均 100% 通过。

@jeikl jeikl changed the title feat(audit): 支持请求/响应全量报文审计、SSE流式聚合重构与可配置日志生命周期 feat(audit): 请求审计中新增不同账号调度详情、完整HTTP请求/响应报文、默认7天全量日志记录等完整诊断生命周期重要功能 Aug 20, 2026
@jeikl

jeikl commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

@chenyme 能合并一下吗 这个功能非常重要 我每天都在使用...

@chenyme

chenyme commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Summary

Improve the request audit system with privacy-safe request metadata, per-account scheduling diagnostics, bounded failure details, configurable retention, and a redesigned admin audit view.

The implementation deliberately avoids persisting user prompts and successful model output. It focuses on the information operators need to diagnose routing, account selection, upstream failures, and interrupted streams without turning the audit database into a full traffic archive.

What changed

Request metadata

Request audits now record bounded and sanitized metadata:

  • HTTP method and path
  • Client IP
  • Protocol and routing information
  • Usage, timing, status, and error metadata
  • Request headers with sensitive values redacted

Query strings are stripped before persistence.

Headers related to authorization, cookies, credentials, API keys, tokens, sessions, signatures, and passwords are stored as [REDACTED].

Per-attempt upstream diagnostics

Each failed scheduling or upstream attempt can now be inspected independently, including:

  • Attempt number
  • Selected account and egress node
  • Failure stage and source
  • HTTP method and sanitized upstream URL
  • Upstream status and duration
  • Transport error and bounded error chain
  • Sanitized upstream headers
  • Failed response excerpt

Failures that occur after an upstream 200 OK stream has started are recorded as response_stream attempts instead of incorrectly appearing as failures before upstream dispatch.

Diagnostic storage is bounded to:

  • 64 KiB per attempt
  • 256 KiB per request
  • 2 KiB for individual error text
  • 4 KiB for serialized diagnostic headers
  • 8 error-chain frames

Retention and cleanup

Add an independent audit retention policy:

  • Default retention: 7 days
  • Configurable range: 0–365 days
  • 0 means permanent retention
  • Cleanup runs periodically using the latest configuration
  • Expired attempts and audit rows are deleted in batches of 1,000

The cleanup implementation works across SQLite, MySQL, and PostgreSQL.

Admin UI

Redesign the audit detail dialog around three focused views:

  • Request overview
  • Request information
  • Upstream diagnostics

Additional UI improvements include:

  • Compact request metadata header
  • Consistent typography, color, spacing, and alignment
  • Simplified request path and header presentation
  • Fewer unnecessary borders and badges
  • Compact attempt navigation with status shown on the right
  • Overview and error-chain tabs aligned to the right
  • Correct fallback status for the final transport attempt
  • Responsive behavior for desktop and narrow viewports
  • Consistent dialog margins and internal gutters

Data policy

Stored

  • Request and routing metadata
  • Sanitized request headers
  • Account scheduling attempts
  • Usage and timing information
  • Failed upstream response excerpts
  • Transport and stream failure diagnostics

Not stored

  • User prompts, messages, or input payloads
  • Images, audio, files, base64 data, or other multimodal content
  • Successful response bodies
  • Successful SSE output
  • Query strings
  • Authentication tokens, cookies, credentials, or secrets

What was removed and why

Full request-body capture

The original implementation persisted complete client request bodies.

This was removed because request bodies may contain prompts, conversation history, private files, images, audio, credentials, and large base64 payloads. Persisting them would introduce substantial privacy, security, compliance, and database-growth risks.

Request metadata and sanitized headers are sufficient for routing and connectivity diagnosis.

Successful response-body and SSE persistence

Successful response bodies and reconstructed SSE output are no longer stored.

They duplicate content already delivered to the client, can be extremely large, and would make normal successful traffic the main source of database growth. Failure diagnostics provide more operational value at a much lower storage cost.

SSE response reconstruction

The protocol-specific SSE accumulator and response reconstruction logic were removed after successful response persistence was dropped.

Keeping this code would add memory usage and maintenance complexity while risking incorrectly reconstructed Responses, Chat Completions, and Anthropic Messages payloads.

Body-recording settings

The following settings and UI controls were removed:

  • recordRequestBody
  • recordResponseBody

Optional body recording would create ambiguous security expectations and could be accidentally enabled in production. The audit policy is now explicit and consistent: metadata for all requests, bounded response excerpts for failures only.

Query-string persistence

Only the request path is retained.

Query strings may contain signed URLs, temporary tokens, credentials, or user-provided data. They are unnecessary for identifying the API operation being audited.

Request/response payload tabs

The full payload tabs and copy actions were removed because the backend no longer exposes or persists those bodies. Leaving them in the interface would imply that sensitive payload capture is supported.

Unrelated repository changes

The following unrelated changes from the original branch were removed to keep the PR focused:

  • frontend/package-lock.json: the frontend uses pnpm; retaining a second lockfile could cause dependency drift.
  • start.bat and stop.bat: unrelated operational scripts outside the audit feature scope.
  • Vite host: "::": unrelated to auditing and would unnecessarily broaden development-server network exposure.
  • Unused boolPointer helper: became dead code after removing the body-recording settings.

Compatibility

  • Existing audit rows remain readable.
  • New metadata fields migrate with empty defaults.
  • Successful request and response bodies remain unpersisted.
  • Failed diagnostics are sanitized and size-limited before persistence.
  • Setting retention to 0 is now preserved correctly instead of falling back to the seven-day default.

Validation

  • go test ./... -count=1
  • go vet ./...
  • Targeted audit, settings, persistence, and HTTP handler tests
  • cd frontend && pnpm lint
  • cd frontend && pnpm test -- --run
  • cd frontend && pnpm build
  • git diff --check
  • Manual UI verification at desktop and narrow viewport widths

@chenyme
chenyme merged commit 7d908f6 into chenyme:main Aug 20, 2026
9 checks passed
mcheiyue added a commit to mcheiyue/grok2api that referenced this pull request Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants