Enable AWS client to copy objects >5GB in size

[james-rms]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I'm trying to copy objects within a bucket as part of my data ingestion workflow. However, when thse objects exceed 5GB in size, this fails with an error. There's a built-in limit in the S3 API for the maximum size of a copy operation.

Describe the solution you'd like

I've outlined the solution i'd prefer in #561 - essentially the copy method should detect large copies and transparently switch to a multipart workflow.

Describe alternatives you've considered

Alternatively, we could also expose this API as part of the multipart-upload API.

Additional context

[james-rms]

Continuing discussion from @tustvold 's previous comment on #561 :

I think this probably warrants a higher level ticket to discuss how we should support this, as a start it would be good to understand how other stores, i.e. GCS and Azure handle this, so that we can develop an abstraction that makes sense.

GCS has no per-operation limit on copies within a bucket. However the rewrite API (which is what they call this) may not complete in a single request, and may return a rewriteToken which has to be repeatedly provided in requests until the response returns done: true.

Azure appears to have no corresponding multi-part copy operation, see quote:

The Copy Blob operation always copies the entire source blob or file. Copying a range of bytes or set of blocks is not supported.

Back to the comment:

In particular I wonder if adding this functionality would make more sense as part of the multipart upload functionality? This of course depends on what other stores support.

Two reasons steer me away from providing this API as an option for multipart uploads:

1. This API is somewhat unique to AWS, as demonstrated above
2. the action the user's trying to complete is a a copy, not an upload. If multipart copies are considered multipart uploads, then shouldn't single-request copy be an option on the put_opts method?

[tustvold]

Makes sense