app/vlinsert: add -insert.maxLogsPerSecond and -insert.maxBytesPerSecond command-line flags

[YdiBik]

This is a reworked version of #1710.

Adds -insert.maxLogsPerSecond and -insert.maxBytesPerSecond for limiting the global data
ingestion rate, as agreed in #887. Both are disabled by default.

The limits are applied on the common ingestion path shared by all the data ingestion protocols,
so they are global rather than per-protocol. -insert.maxBytesPerSecond is applied to the
estimated JSON size of the ingested entries - the same value exposed via vl_bytes_ingested_total.
The throttling is done with the existing lib/ratelimiter, the same way -remoteWrite.rateLimit
does it in vlagent. The rate limiters are unblocked before the http server is stopped, so throttled
in-flight requests don't delay the graceful shutdown.

The ingestion is throttled rather than rejected with 429, since LogMessageProcessor.AddRow()
doesn't return an error and protocols which don't run on top of HTTP, such as syslog, can't report
the rate limit back to the client. Rejecting would need a non-blocking budget check in
lib/ratelimiter.

Logs collected by vlagent from files and from Kubernetes pods don't go through the data ingestion
protocols, so they aren't affected by these flags - this is documented next to them.

An integration test in apptest verifies that the ingestion is throttled with each of the limits.
make check-all, make test-full and make apptest pass.

Fixes #887

Checklist

The following checks are mandatory:

• My change adheres to VictoriaMetrics contributing guidelines.
• My change adheres to VictoriaMetrics development goals.

Benchmark on Intel i9-10900K, -benchtime=3s -count=10, benchstat:

                                     │      sec/op      │
BenchmarkAddRow-20                          71.26n ± 3%
RateLimiters/disabled-20                    2.654n ± 1%
RateLimiters/logs-limit-20                  10.97n ± 1%
RateLimiters/bytes-limit-20                 11.17n ± 0%
RateLimiters/both-limits-20                 20.21n ± 0%
RateLimiters/both-limits-parallel-20        138.3n ± 8%

0 B/op and 0 allocs/op in every configuration. The rate limiters are registered once per flush,
which holds up to 20972 entries, while AddRow is called per entry, so the overhead is amortized.
With the limits disabled - the default - the call is a nil check.

[cubic-dev-ai]

2 issues found across 10 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="app/vlinsert/insertutil/common_params.go">

<violation number="1" location="app/vlinsert/insertutil/common_params.go:322">
P2: `flushLocked()` calls `Register` while holding `lmp.mu`, but `ratelimiter.RateLimiter.Register` can block for a long time and holds both `lmp.mu` and the package-global rate limiter mutex for its whole duration (the `for rl.budget <= 0` loop runs under `rl.mu.Lock()`). The budget is replenished by only `perSecondLimit` per ~1s blocking iteration, so one flush batch bigger than the limit accumulates a debt repaid at `limit` per second and the single `Register` call blocks for roughly `debt/limit` seconds. During that block every other processor registering on the same global limiter is serialized and stalls, and in-flight HTTP ingest handlers block. This also delays graceful shutdown: in `victoria-logs/main.go`, `httpserver.Stop()` (which waits for in-flight handlers) runs before `vlinsert.Stop()` -> `StopRateLimiters()`, so a throttled request blocked in `Register` is not unblocked early via the closed `stopCh` and shutdown waits the full throttle duration.</violation>
</file>

<file name="docs/victorialogs/CHANGELOG.md">

<violation number="1" location="docs/victorialogs/CHANGELOG.md:29">
P2: Custom agent: **Changelog Review Agent**

This entry violates the changelog scope: it names the `data ingestion` capability instead of an affected service, and its behavior applies only to users who opt into disabled-by-default flags. Identify the affected service and establish majority-wide impact, or remove this entry.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: flushLocked() calls Register while holding lmp.mu, but ratelimiter.RateLimiter.Register can block for a long time and holds both lmp.mu and the package-global rate limiter mutex for its whole duration (the for rl.budget <= 0 loop runs under rl.mu.Lock()). The budget is replenished by only perSecondLimit per ~1s blocking iteration, so one flush batch bigger than the limit accumulates a debt repaid at limit per second and the single Register call blocks for roughly debt/limit seconds. During that block every other processor registering on the same global limiter is serialized and stalls, and in-flight HTTP ingest handlers block. This also delays graceful shutdown: in victoria-logs/main.go, httpserver.Stop() (which waits for in-flight handlers) runs before vlinsert.Stop() -> StopRateLimiters(), so a throttled request blocked in Register is not unblocked early via the closed stopCh and shutdown waits the full throttle duration.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At app/vlinsert/insertutil/common_params.go, line 322:

<comment>`flushLocked()` calls `Register` while holding `lmp.mu`, but `ratelimiter.RateLimiter.Register` can block for a long time and holds both `lmp.mu` and the package-global rate limiter mutex for its whole duration (the `for rl.budget <= 0` loop runs under `rl.mu.Lock()`). The budget is replenished by only `perSecondLimit` per ~1s blocking iteration, so one flush batch bigger than the limit accumulates a debt repaid at `limit` per second and the single `Register` call blocks for roughly `debt/limit` seconds. During that block every other processor registering on the same global limiter is serialized and stalls, and in-flight HTTP ingest handlers block. This also delays graceful shutdown: in `victoria-logs/main.go`, `httpserver.Stop()` (which waits for in-flight handlers) runs before `vlinsert.Stop()` -> `StopRateLimiters()`, so a throttled request blocked in `Register` is not unblocked early via the closed `stopCh` and shutdown waits the full throttle duration.</comment>

<file context>
@@ -317,6 +317,11 @@ func (lmp *logMessageProcessor) AddInsertRow(r *logstorage.InsertRow) {
 func (lmp *logMessageProcessor) flushLocked() {
+	// Throttle the ingestion if the limits set via -insert.maxLogsPerSecond or -insert.maxBytesPerSecond are exceeded.
+	// This is the common path for all the data ingestion protocols, so the limits are global.
+	logsRateLimiter.Register(lmp.unflushedRows)
+	bytesRateLimiter.Register(lmp.unflushedBytes)
+
</file context>

[cubic-dev-ai]

P2: Custom agent: Changelog Review Agent

This entry violates the changelog scope: it names the data ingestion capability instead of an affected service, and its behavior applies only to users who opt into disabled-by-default flags. Identify the affected service and establish majority-wide impact, or remove this entry.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At docs/victorialogs/CHANGELOG.md, line 29:

<comment>This entry violates the changelog scope: it names the `data ingestion` capability instead of an affected service, and its behavior applies only to users who opt into disabled-by-default flags. Identify the affected service and establish majority-wide impact, or remove this entry.</comment>

<file context>
@@ -26,6 +26,7 @@ according to the following docs:
 * SECURITY: upgrade Go builder from Go1.26.5 to Go1.26.6. See [the list of issues addressed in Go1.26.6](https://github.com/golang/go/issues?q=milestone%3AGo1.26.6%20label%3ACherryPickApproved).
 * SECURITY: [deletion API](https://docs.victoriametrics.com/victorialogs/#how-to-delete-logs): restrict the `/delete/run_task` endpoint to the `POST` method only in order to prevent some [SSRF](https://en.wikipedia.org/wiki/Server-side_request_forgery)-based log deletion attacks. See [#1635](https://github.com/VictoriaMetrics/VictoriaLogs/issues/1635).
 
+* FEATURE: [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/): add `-insert.maxLogsPerSecond` and `-insert.maxBytesPerSecond` command-line flags for limiting the global data ingestion rate across all the supported data ingestion protocols. Both limits are disabled by default. The ingestion is throttled when any of the configured limits is exceeded. See [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/#rate-limiting) and [#887](https://github.com/VictoriaMetrics/VictoriaLogs/issues/887).
 * FEATURE: [cluster version](https://docs.victoriametrics.com/victorialogs/cluster/): optimize queries, which return the limited number of log entries with the biggest timestamps on the selected time range. [Web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui) usually executes such queries. See [#1602](https://github.com/VictoriaMetrics/VictoriaLogs/issues/1602).
 * FEATURE: [dashboards/cluster](https://grafana.com/grafana/dashboards/23274), [dashboards/single](https://grafana.com/grafana/dashboards/22084), and [dashboards/vlagent](https://grafana.com/grafana/dashboards/24513): add `Fsync avg duration` panel to the Troubleshooting section of the single-node, cluster, and vlagent dashboards. This panel shows average `fsync` latency to help identify slow storage persistence. See [VictoriaMetrics#10432](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10432).
</file context>

[cubic-dev-ai]

All reported issues were addressed across 3 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 2 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 1 file (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 3 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="docs/victorialogs/data-ingestion/README.md">

<violation number="1" location="docs/victorialogs/data-ingestion/README.md:385">
P2: Custom agent: **Technical Writer Review Agent**

With concurrent throttled flushes, `rate()` aggregates per-request waiting time and can exceed one; it does not represent a bounded fraction of wall-clock time. Document the metric as aggregate waiting time instead.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Custom agent: Technical Writer Review Agent

With concurrent throttled flushes, rate() aggregates per-request waiting time and can exceed one; it does not represent a bounded fraction of wall-clock time. Document the metric as aggregate waiting time instead.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At docs/victorialogs/data-ingestion/README.md, line 385:

<comment>With concurrent throttled flushes, `rate()` aggregates per-request waiting time and can exceed one; it does not represent a bounded fraction of wall-clock time. Document the metric as aggregate waiting time instead.</comment>

<file context>
@@ -370,8 +378,13 @@ The `-insert.maxBytesPerSecond` limit is applied to the estimated JSON size of t
+and the entries processed with `debug=1`, which aren't stored. This is the same set of entries which is accounted
+at the `vl_rows_ingested_total` and `vl_bytes_ingested_total` metrics.
+
+The `vl_insert_rate_limit_reached_total` metric is incremented approximately once per second spent waiting
+for the budget replenishment, so `rate()` over this metric shows the fraction of time the ingestion is throttled.
+The `vl_insert_rate_limit` metric exposes the configured limits. These metrics can be used for alerting
</file context>

Suggested change

	The `vl_insert_rate_limit_reached_total` metric is incremented approximately once per second spent waiting
	for the budget replenishment, so `rate()` over this metric shows the fraction of time the ingestion is throttled.
	The `vl_insert_rate_limit_reached_total` metric is incremented approximately once per second spent waiting across ingestion requests. Consequently, `rate()` shows aggregate waiting time and may exceed one with concurrent throttled requests.