Skip to content

Conversation

@JornEngelbart
Copy link

@JornEngelbart JornEngelbart commented Oct 31, 2025

Fix Content-Range header handling in resumable uploads

Problem

The resumable upload implementation was not correctly handling the Content-Range header when uploading chunks. The code was simply appending all chunks sequentially (obj.Content = append(obj.Content, content...)), which caused incorrect behavior when:

  1. Overlapping ranges: Clients may re-upload the same byte range (e.g., for retry scenarios)
  2. Out-of-order chunks: Though rare, chunks might arrive out of sequence
  3. Non-sequential uploads: Some clients upload chunks that don't start at position 0

This resulted in:

  • Content being placed at incorrect byte positions
  • Duplicate content when ranges overlapped
  • Incorrect file sizes and corruption

Solution

The fix properly parses and respects the Content-Range header to place content at the exact byte positions specified:

Key Changes

  1. Parse Content-Range before appending: The code now parses the Content-Range header first to determine where content should be placed

  2. Place content at specified positions: Instead of appending, content is copied to the exact byte range specified:

    // For known ranges (e.g., "bytes 0-262143/377464")
    copy(obj.Content[contentStart:contentStart+len(content)], content)
  3. Handle overlapping ranges: When the same range is uploaded multiple times, the content correctly overwrites the previous content at those positions

  4. Extend buffer as needed: The buffer is automatically extended with zero-filled bytes when a range requires a larger buffer size

  5. Validate content length: Added validation to ensure the actual content length matches the Content-Range header specification

Example Scenario

Previously, if a client sent:

  • Request 1: Content-Range: bytes 0-262143/377464 (262144 bytes)
  • Request 2: Content-Range: bytes 1-262144/377464 (262144 bytes, overlapping)
  • Request 3: Content-Range: bytes 1-377463/377464 (377463 bytes, overlapping)

The old code would append all chunks sequentially, resulting in an incorrect file.

The new code correctly:

  • Places first chunk at positions 0-262143
  • Places second chunk at positions 1-262144 (overwriting positions 1-262143 and adding new byte at 262144)
  • Places third chunk at positions 1-377463 (overwriting and extending as needed)
  • Final result: Correct file with all bytes in the right positions

Testing

Added comprehensive test coverage in TestResumableUploadWithContentRange that verifies:

  • Overlapping range uploads (same range uploaded twice)
  • Sequential non-overlapping range uploads
  • Proper content placement at specified byte positions
  • Checksum validation of final content

Impact

This fix ensures that fake-gcs-server correctly handles resumable uploads with Content-Range headers, making it compatible with clients that:

  • Retry uploads with overlapping ranges
  • Upload chunks in non-sequential order
  • Use the Content-Range header to specify exact byte positions

The fix is backward compatible and maintains the existing behavior for uploads without Content-Range headers (falls back to append behavior).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant