-
Notifications
You must be signed in to change notification settings - Fork 255
Description
I have a weird issue where we see that writing files s3Proxy is also adding chunk-signatures and checksums to the file content when using either Apache Camel or S3Client (which the Camel connector is based on). Using the cli is totally fine. The same code writes files to S3 and localstack as I would expect.
We would prefer s3 proxy as we typically use Azure but we see the same with the filesystem configuration.
File content: "fileContent-none"
CLI
Command: docker run --rm -ti -e AWS_ACCESS_KEY_ID=dummy -e AWS_SECRET_ACCESS_KEY=dummy -e AWS_DEFAULT_REGION=eu-central-1 -it -v .:/aws amazon/aws-cli s3 cp /aws/test.txt s3://etl-export-container/ --endpoint-url http://host.docker.internal:4560
Content in file: "fileContent-none"
Camel/S3Client
Code: PutObjectRequest request2 = PutObjectRequest.builder()
.bucket(BLOB_CONTAINER_NAME)
.key(filename + "s3client")
.contentType("text/plain; charset=UTF-8")
.build();
s3Client.putObject(request2, RequestBody.fromString(fileContent));
Content in file:
"10;chunk-signature=460c0b1b7dfc968c3c1bfbdff125747c5afb0afa0ea3129f63b9a0dcf1205bc2
fileContent-none
0;chunk-signature=65183ac24cbe0eed3b56470c462bae1f70e04b464da0caed2a4df18d44e26bdb
x-amz-checksum-crc32:VdfJwQ==
x-amz-trailer-signature:80866cca3e043de1df9ee970273eec29cb69be8da7cd9f6c16c3fef2cb77cd25
"
We deploy via docker with locally:
Image: andrewgaul/s3proxy:sha-6355f7826b8e25b55067b9713d852c4d8e41e0ca
S3PROXY_ENDPOINT: "http://0.0.0.0:4560"
S3PROXY_AUTHORIZATION: "none"
S3PROXY_IGNORE_UNKNOWN_HEADERS: "true"
LOG_LEVEL: "info"
JCLOUDS_PROVIDER: "azureblob-sdk"
#JCLOUDS_PROVIDER: "filesystem-nio2"
JCLOUDS_IDENTITY: "devstoreaccount1"
JCLOUDS_CREDENTIAL: "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
JCLOUDS_ENDPOINT: "http://azurite:10000/devstoreaccount1"
JCLOUDS_AZUREBLOB_AUTH: "azureKey"
JCLOUDS_FILESYSTEM_BASEDIR: "/data"
That I can see there should be no reason for the extra data.
Any help is appreciated.