Skip to content

Using Apache Camel or S3Client is writing additional content compared to CLI to storage #922

@ZeroWiggliness

Description

@ZeroWiggliness

I have a weird issue where we see that writing files s3Proxy is also adding chunk-signatures and checksums to the file content when using either Apache Camel or S3Client (which the Camel connector is based on). Using the cli is totally fine. The same code writes files to S3 and localstack as I would expect.

We would prefer s3 proxy as we typically use Azure but we see the same with the filesystem configuration.

File content: "fileContent-none"

CLI

Command: docker run --rm -ti -e AWS_ACCESS_KEY_ID=dummy -e AWS_SECRET_ACCESS_KEY=dummy -e AWS_DEFAULT_REGION=eu-central-1 -it -v .:/aws amazon/aws-cli s3 cp /aws/test.txt s3://etl-export-container/ --endpoint-url http://host.docker.internal:4560
Content in file: "fileContent-none"

Camel/S3Client

Code: PutObjectRequest request2 = PutObjectRequest.builder()
.bucket(BLOB_CONTAINER_NAME)
.key(filename + "s3client")
.contentType("text/plain; charset=UTF-8")
.build();

s3Client.putObject(request2, RequestBody.fromString(fileContent));

Content in file:
"10;chunk-signature=460c0b1b7dfc968c3c1bfbdff125747c5afb0afa0ea3129f63b9a0dcf1205bc2
fileContent-none
0;chunk-signature=65183ac24cbe0eed3b56470c462bae1f70e04b464da0caed2a4df18d44e26bdb
x-amz-checksum-crc32:VdfJwQ==
x-amz-trailer-signature:80866cca3e043de1df9ee970273eec29cb69be8da7cd9f6c16c3fef2cb77cd25
"

We deploy via docker with locally:
Image: andrewgaul/s3proxy:sha-6355f7826b8e25b55067b9713d852c4d8e41e0ca

  S3PROXY_ENDPOINT: "http://0.0.0.0:4560"
  S3PROXY_AUTHORIZATION: "none"
  S3PROXY_IGNORE_UNKNOWN_HEADERS: "true"
  LOG_LEVEL: "info"
  JCLOUDS_PROVIDER: "azureblob-sdk"
  #JCLOUDS_PROVIDER: "filesystem-nio2"
  JCLOUDS_IDENTITY: "devstoreaccount1"
  JCLOUDS_CREDENTIAL: "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
  JCLOUDS_ENDPOINT: "http://azurite:10000/devstoreaccount1"
  JCLOUDS_AZUREBLOB_AUTH: "azureKey"
  JCLOUDS_FILESYSTEM_BASEDIR: "/data"

That I can see there should be no reason for the extra data.

Any help is appreciated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions