Skip to content

fix: use the system CA bundle for the aws cli in the stats image - #65

Open
handecelikkanat wants to merge 1 commit into
masterfrom
fix/docker-aws-ca-bundle
Open

handecelikkanat wants to merge 1 commit into
masterfrom
fix/docker-aws-ca-bundle

Conversation

@handecelikkanat

Copy link
Copy Markdown
Contributor

Use the system CA bundle for the aws cli in the stats image

Debian's awscli package ships its own copy of the CA bundle
(awscli/botocore/cacert.pem), which only changes when Debian updates the
awscli package itself. This sets AWS_CA_BUNDLE, so the aws cli, and
botocore via s3fs, use the system bundle /etc/ssl/certs/ca-certificates.crt
instead. The system bundle receives CA additions and removals through
Debian's ca-certificates package.

Tested with:

docker build -f stats.Dockerfile -t cc-crawl-statistics-stats .

# prints /etc/ssl/certs/ca-certificates.crt
docker run --rm cc-crawl-statistics-stats sh -c 'echo "$AWS_CA_BUNDLE"'

# lists the first entries of an old crawl
docker run --rm -v ~/.aws:/root/.aws:ro cc-crawl-statistics-stats \
    sh -c 'aws s3 ls s3://commoncrawl/crawl-data/CC-MAIN-2018-17/ | sed -n 1,3p'

# fails with a certificate error, so the aws cli follows AWS_CA_BUNDLE
docker run --rm -v ~/.aws:/root/.aws:ro -e AWS_CA_BUNDLE=/dev/null -e AWS_MAX_ATTEMPTS=1 \
    cc-crawl-statistics-stats aws --debug s3 ls s3://commoncrawl/crawl-data/CC-MAIN-2018-17/ 2>&1 | tail -20

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant