Support for S3 "Compatible" Storage - Minio, etc

Real AWS S3 supports accessing buckets via either virtual or path style URLs:

virtual-style URLs with bucket name as part of the hostname:

path-style URLs with the bucket name as part of the path:

It seems like SnapLogic’s various S3 snaps and other S3 capabilities (file writer, etc.) use the AWS SDK under the covers, and always uses path-style URLs that actual AWS supports just fine…

However, implementing DNS resolution solutions for virtual-style URLs with other S3 Compatible on-prem storage solutions (like Minio) is obviously difficult, and path-style URLs are preferred. There doesn’t seem to be any options in SnapLogic for configuring which type of URL to use.

From SnapLogic’s S3 documentation, I can pull files using expressions like below:

  • s3:///[S3 bucket name]@[region name][.amazonaws.com]/[path]

For example, I can successfully interact with my actual Amazon S3 bucket via SnapLogic with any of the expressions below:

  • s3:///bucketname/path/filename
  • s3:///bucketname@us-west-2/path/filename
  • s3:///bucketname@s3.amazonaws.com/path/filename
  • s3:///bucketname@s3.amazonaws.com:443/path/filename
  • s3:///bucketname@s3-us-west-2.amazonaws.com/path/filename
  • s3:///bucketname@s3-us-west-2.amazonaws.com:443/path/filename

So it seems there is an opportunity to override the hostnames, ports, etc. for potentially pointing to something else, even on-prem (like Minio). But there doesn’t seem to be any way to tell SnapLogic to put the bucket name in the path rather than in the hostname, which would make it much easier to interact with other S3 “Compatible” types of storage.

AWS plans to remove support for the path-style URLs next year, see https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

The SnapLogic implementation does use the AWS Java SDK and the access style is not currently configurable.