Use response payload budget instead of per-field entry limits for gRPC responses

[kkovaacs]

Background

In #2153, we added support for GetAccount to return all storage maps. During review, we noted that the current API limit model is still too rigid: some response paths cap individual fields by number of elements, even though tonic enforces an overall 4 MiB gRPC message size limit.

For example, storage map details currently cap individual maps at 1000 entries. That means a response with 10,000 entries from one map may be rejected/truncated, while a response with 1000 entries from each of 10 maps may be allowed, even though both have similar payload-size implications.

The review suggestion was to move away from individual field limits where the real constraint is response size.

Current behavior

• GetAccount has special response budgeting for AllStorageMaps, but individual storage maps still use AccountStorageMapDetails::MAX_RETURN_ENTRIES = 1000.
• AccountVaultDetails also uses MAX_RETURN_ENTRIES = 1000.
• Shared RPC limits in crates/utils/src/limiter.rs use element-count limits for several endpoints, with comments tying those limits to the 4 MiB payload budget.
• Some endpoints already use payload-size-aware pagination, e.g. SyncNotes / SyncTransactions, so behavior is inconsistent across APIs.

Problem

Per-field element caps are not a good proxy for the actual gRPC response size. They can reject valid responses that fit under 4 MiB and allow combinations that only happen to stay safe because each individual field is capped.

This makes the API less flexible than necessary and spreads response-size reasoning across endpoint-specific constants.

Proposed direction

Refactor response-size enforcement so that endpoints whose main constraint is tonic's 4 MiB response limit apply an overall encoded-size budget instead of arbitrary per-field entry limits.

For GetAccount, this likely means:

• apply a single response payload budget across account code, vault details, storage header, and storage map details;
• allow any individual storage map to exceed 1000 entries if the final encoded response remains within the budget;
• return existing too_many_entries / too_many_assets markers only when including that data would exceed the response budget;
• keep stricter request/work limits where they protect query fan-out, proof generation, or database load rather than response size.

Acceptance criteria

• GetAccount enforces the 4 MiB response payload budget at the response level, not via a hard 1000-entry cap per storage map.
• Existing request fan-out limits, such as explicit storage map key proof requests, remain in place where needed.
• Other RPC/store endpoints with element-count limits are audited and categorized as either:
  • true request/work limits, or
  • response-size limits that should become payload-budget based.
• GetLimits documentation/API output is updated if any exposed limit semantics change.
• Tests cover cases where:
  • a single large field fits under the response budget and is returned;
  • a combination of fields exceeds the response budget and is truncated/paginated/marked consistently;
  • generated responses stay below tonic's 4 MiB limit.