bug: native HKDF allocates output before validating the RFC 5869 length limit

[harrshita123]

Summary

The native FFI implementation of HkdfSecretKey.deriveBits allocates the requested output buffer before validating HKDF's maximum output length. An oversized request can therefore trigger an arbitrarily large native allocation attempt before it is rejected.

RFC 5869 limits HKDF output to 255 * HashLen bytes. The implementation already translates BoringSSL's HKDF_R_OUTPUT_TOO_LARGE error into a useful OperationError, but BoringSSL is invoked only after the output buffer has been allocated.

Reproduction

import 'package:webcrypto/webcrypto.dart';

Future<void> main() async {
  final key = await HkdfSecretKey.importRawKey([1, 2, 3, 4]);

  for (final length in <int>[
    (255 * 32 + 1) * 8,
    9223372036854775800,
  ]) {
    try {
      await key.deriveBits(length, Hash.sha256, const [], const []);
    } catch (error) {
      print('$length: $error');
    }
  }
}

Native output:

65288: Length specified for HkdfSecretKey.deriveBits is too long
9223372036854775800: error:0e000041:common libcrypto routines:OPENSSL_internal:malloc failure

The first value is one byte beyond SHA-256 HKDF's maximum and reaches BoringSSL's intended output-length check. The second value fails earlier while attempting to allocate the output buffer.

Expected behavior

Every output length greater than 255 * HashLen bytes should be rejected before allocating output memory, with the existing deterministic "too long" OperationError.

RFC 5869 Section 2.3 defines L <= 255 * HashLen:

https://www.rfc-editor.org/rfc/rfc5869#section-2.3

The Web Cryptography HKDF derive-bits algorithm delegates to that HKDF extract-and-expand operation:

https://www.w3.org/TR/WebCryptoAPI/#hkdf-operations

Actual behavior

The FFI implementation currently performs these operations in this order:

final lengthInBytes = length ~/ 8;

return _Scope.async((scope) async {
  final out = scope<ffi.Uint8>(lengthInBytes);
  final r = ssl.HKDF(
    out,
    lengthInBytes,
  );

Consequently, the requested native allocation occurs before ssl.HKDF can report HKDF_R_OUTPUT_TOO_LARGE.

Impact

Invalid input that should be rejected cheaply can instead cause very large native allocation attempts. Depending on allocator and process memory conditions, this can produce inconsistent errors, memory pressure, or process termination.

This is especially relevant when the requested derivation length is influenced by external configuration or input.

Suggested fix

• Determine the selected hash's output size before entering the allocation scope.
• Reject lengthInBytes > 255 * hashLengthInBytes with the existing "too long" OperationError.
• Keep the BoringSSL HKDF_R_OUTPUT_TOO_LARGE handling as a defensive fallback.
• Add regression tests covering:
  • the exact maximum for SHA-1, SHA-256, SHA-384, and SHA-512;
  • one byte beyond each maximum; and
  • a very large valid positive Dart integer, confirming rejection occurs without an allocation failure.