access public S3 storage anonymously

[jiron12]

How to access parquet in public S3 storage anonymously with Trino? Was that removed?

Old thread showing how it was once possible: #15259

[jiron12]

Ok, to maybe add some context:

my hive connector config, hive_aws.properties:

connector.name=hive
hive.metastore.uri=thrift://hive-metastore:9083
fs.native-s3.enabled=true
s3.endpoint=https://s3.amazonaws.com
s3.region=us-east-1

(I understand I simply have to provide no credentials for anonymous S3 access?)

Then I want to create my Table

CREATE TABLE hive_aws.default.noaa_ghcn_2024 (
    id VARCHAR,
    date varchar,
    data_value INT,
    m_flag VARCHAR,
    q_flag VARCHAR,
    s_flag VARCHAR,
    obs_time VARCHAR,
    year int,
  	element VARCHAR
)
WITH (
    format = 'PARQUET',
    external_location = 's3://noaa-ghcn-pds/parquet/by_year/', -- I have also tried s3://noaa-ghcn-pds.s3.amazonaws.com/parquet/by_year/ and not providing s3.endpoint=https://s3.amazonaws.com hive_aws.properties
    partitioned_by = ARRAY['YEAR', 'ELEMENT']
)

Whatever I try, I get this error about credentials:

io.trino.spi.TrinoException: External location is not a valid file system URI: s3://noaa-ghcn-pds.s3.amazonaws.com/parquet/by_year/
	at io.trino.plugin.hive.HiveMetadata.checkExternalPathAndCreateIfNotExists(HiveMetadata.java:1429)
	at io.trino.plugin.hive.HiveMetadata.createTable(HiveMetadata.java:1109)
	at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.createTable(ClassLoaderSafeConnectorMetadata.java:461)
	at io.trino.tracing.TracingConnectorMetadata.createTable(TracingConnectorMetadata.java:388)
	at io.trino.metadata.MetadataManager.createTable(MetadataManager.java:878)
	at io.trino.tracing.TracingMetadata.createTable(TracingMetadata.java:436)
	at io.trino.execution.CreateTableTask.internalExecute(CreateTableTask.java:306)
	at io.trino.execution.CreateTableTask.execute(CreateTableTask.java:128)
	at io.trino.execution.CreateTableTask.execute(CreateTableTask.java:94)
	at io.trino.execution.DataDefinitionExecution.start(DataDefinitionExecution.java:152)
	at io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:284)
	at io.trino.dispatcher.LocalDispatchQuery.startExecution(LocalDispatchQuery.java:150)
	at io.trino.dispatcher.LocalDispatchQuery.lambda$waitForMinimumWorkers$1(LocalDispatchQuery.java:134)
	at io.airlift.concurrent.MoreFutures.lambda$addSuccessCallback$0(MoreFutures.java:570)
	at io.airlift.concurrent.MoreFutures$3.onSuccess(MoreFutures.java:545)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
	at io.trino.$gen.Trino_478____20251201_172151_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
	at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: io.trino.filesystem.TrinoFileSystemException: Failed to list location: s3://noaa-ghcn-pds.s3.amazonaws.com/parquet/by_year/
	at io.trino.filesystem.s3.S3FileSystem.listObjects(S3FileSystem.java:270)
	at io.trino.filesystem.s3.S3FileSystem.listFiles(S3FileSystem.java:240)
	at io.trino.filesystem.s3.S3FileSystem.directoryExists(S3FileSystem.java:279)
	at io.trino.filesystem.switching.SwitchingFileSystem.directoryExists(SwitchingFileSystem.java:123)
	at io.trino.filesystem.tracing.TracingFileSystem.lambda$directoryExists$0(TracingFileSystem.java:129)
	at io.trino.filesystem.tracing.Tracing.withTracing(Tracing.java:51)
	at io.trino.filesystem.tracing.TracingFileSystem.directoryExists(TracingFileSystem.java:129)
	at io.trino.plugin.hive.HiveMetadata.checkExternalPathAndCreateIfNotExists(HiveMetadata.java:1419)
	... 19 more
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])): Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
	at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:130)
	at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
	at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:134)
	at software.amazon.awssdk.auth.credentials.AwsCredentialsProvider.resolveIdentity(AwsCredentialsProvider.java:54)
	at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.lambda$trySelectAuthScheme$6(S3AuthSchemeInterceptor.java:169)
	at software.amazon.awssdk.core.internal.util.MetricUtils.reportDuration(MetricUtils.java:81)
	at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.trySelectAuthScheme(S3AuthSchemeInterceptor.java:169)
	at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.selectAuthScheme(S3AuthSchemeInterceptor.java:87)
	at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.beforeExecution(S3AuthSchemeInterceptor.java:67)
	at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.lambda$beforeExecution$1(ExecutionInterceptorChain.java:59)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
	at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.beforeExecution(ExecutionInterceptorChain.java:59)
	at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.runInitialInterceptors(AwsExecutionContextBuilder.java:319)
	at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:155)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:67)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:76)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
	at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:8724)
	at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:154)
	at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:145)
	at software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58)
	at software.amazon.awssdk.core.pagination.sync.PaginatedItemsIterable$ItemsIterator.<init>(PaginatedItemsIterable.java:58)
	at software.amazon.awssdk.core.pagination.sync.PaginatedItemsIterable.iterator(PaginatedItemsIterable.java:48)
	at java.base/java.lang.Iterable.spliterator(Iterable.java:101)
	at software.amazon.awssdk.core.pagination.sync.SdkIterable.stream(SdkIterable.java:34)
	at io.trino.filesystem.s3.S3FileSystem.listObjects(S3FileSystem.java:263)
	... 26 more

Just in case it is not clear, what I am basically after is what in duckdb goes like this (of course without partitions):

SELECT *
FROM read_parquet('https://noaa-ghcn-pds.s3.amazonaws.com/parquet/by_year/YEAR%3D2024/ELEMENT%3DADPT/c156ff7bec274f90b378a666356649c0_0.snappy.parquet')

Am I doing something wrong here? Any help very much apprechiated!

[jiron12]

I also tried by setting:

s3.signer-type=Aws4UnsignedPayloadSigner

but I get the same error when trying to create the table.

[jiron12]

When I read the error message seems to indicate that providers are looked at for the AwsCredentialsProviderChain in this order

1. SystemPropertyCredentialsProvider
2. EnvironmentVariableCredentialsProvider
3. WebIdentityTokenCredentialsProvider
4. ProfileCredentialsProvider
5. ContainerCredentialsProvider
6. InstanceProfileCredentialsProvider

Am I correct to assume that as the AnonymousCredentialsProvider is nowhere in this chain anonymous access to S3 Storage is not possible anymore with Trino?

If that is the case: Could it be added to the end of that chain?

[jiron12]

Tried the following:

s3.credentials-provider=software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider

Looking at TrinoS3FileSystem.java I would have tought this could work but when starting Trino I get:

2025-12-02T14:26:47.990Z        ERROR   main    io.trino.server.Server  Configuration errors:

1) Error: Configuration property 's3.credentials-provider' was not used

1 error
io.airlift.bootstrap.ApplicationConfigurationException: Configuration errors:

1) Error: Configuration property 's3.credentials-provider' was not used

1 error
        at io.airlift.bootstrap.Bootstrap.configure(Bootstrap.java:302)
        at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:331)
        at io.trino.plugin.hive.HiveConnectorFactory.createConnector(HiveConnectorFactory.java:109)
        at io.trino.plugin.hive.HiveConnectorFactory.create(HiveConnectorFactory.java:74)
        at io.trino.connector.DefaultCatalogFactory.createConnector(DefaultCatalogFactory.java:199)
        at io.trino.connector.DefaultCatalogFactory.createCatalog(DefaultCatalogFactory.java:124)
        at io.trino.connector.LazyCatalogFactory.createCatalog(LazyCatalogFactory.java:44)
        at io.trino.connector.StaticCatalogManager.lambda$loadInitialCatalogs$1(StaticCatalogManager.java:160)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:545)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
        at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:187)
        at io.trino.util.Executors.executeUntilFailure(Executors.java:46)
        at io.trino.connector.StaticCatalogManager.loadInitialCatalogs(StaticCatalogManager.java:154)
        at io.trino.server.Server.doStart(Server.java:140)
        at io.trino.server.Server.lambda$start$0(Server.java:79)
        at io.trino.$gen.Trino_478____20251202_142639_1.run(Unknown Source)
        at io.trino.server.Server.start(Server.java:79)
        at io.trino.server.TrinoServer.main(TrinoServer.java:37)