A remote endpoint for AWS S3 storage requiring credentials. More...

#include <remote_handle.hpp>

Inheritance diagram for kvikio::S3Endpoint:

Public Member Functions
	S3Endpoint (std::string url, std::optional< std::string > aws_region=std::nullopt, std::optional< std::string > aws_access_key=std::nullopt, std::optional< std::string > aws_secret_access_key=std::nullopt, std::optional< std::string > aws_session_token=std::nullopt)
	Create a S3 endpoint from a url. More...

	S3Endpoint (std::pair< std::string, std::string > bucket_and_object_names, std::optional< std::string > aws_region=std::nullopt, std::optional< std::string > aws_access_key=std::nullopt, std::optional< std::string > aws_secret_access_key=std::nullopt, std::optional< std::string > aws_endpoint_url=std::nullopt, std::optional< std::string > aws_session_token=std::nullopt)
	Create a S3 endpoint from a bucket and object name. More...

void	setopt (CurlHandle &curl) override
	Set needed connection options on a curl handle. More...

std::string	str () const override
	Get a description of this remote point instance. More...

std::size_t	get_file_size () override
	Get the size of the remote file. More...

void	setup_range_request (CurlHandle &curl, std::size_t file_offset, std::size_t size) override
	Set up the range request in order to read part of a file given the file offset and read size.

Public Member Functions inherited from kvikio::RemoteEndpoint
RemoteEndpointType	remote_endpoint_type () const noexcept
	Get the type of the remote file. More...

Static Public Member Functions
static std::string	url_from_bucket_and_object (std::string bucket_name, std::string object_name, std::optional< std::string > aws_region, std::optional< std::string > aws_endpoint_url)
	Get url from a AWS S3 bucket and object name. More...

static std::pair< std::string, std::string >	parse_s3_url (std::string const &s3_url)
	Given an url like "s3://<bucket>/<object>", return the name of the bucket and object. More...

static bool	is_url_valid (std::string const &url) noexcept
	Whether the given URL is valid for S3 endpoints (excluding presigned URL). More...

Additional Inherited Members
Protected Member Functions inherited from kvikio::RemoteEndpoint
	RemoteEndpoint (RemoteEndpointType remote_endpoint_type)

Protected Attributes inherited from kvikio::RemoteEndpoint
RemoteEndpointType	_remote_endpoint_type {RemoteEndpointType::AUTO}

Detailed Description

A remote endpoint for AWS S3 storage requiring credentials.

This endpoint is for accessing private S3 objects using AWS credentials (access key, secret key, region and optional session token).

Definition at line 152 of file remote_handle.hpp.

Constructor & Destructor Documentation

◆ S3Endpoint() [1/2]

kvikio::S3Endpoint::S3Endpoint	(	std::string	url,
		std::optional< std::string >	aws_region = `std::nullopt`,
		std::optional< std::string >	aws_access_key = `std::nullopt`,
		std::optional< std::string >	aws_secret_access_key = `std::nullopt`,
		std::optional< std::string >	aws_session_token = `std::nullopt`
	)

Create a S3 endpoint from a url.

Parameters

url	The full http url to the S3 file. NB: this should be an url starting with "http://" or "https://". If you have an S3 url of the form "s3://<bucket>/<object>", please use `S3Endpoint::parse_s3_url()` and `S3Endpoint::url_from_bucket_and_object() to convert it.
aws_region	The AWS region, such as "us-east-1", to use. If nullopt, the value of the `AWS_DEFAULT_REGION` environment variable is used.
aws_access_key	The AWS access key to use. If nullopt, the value of the `AWS_ACCESS_KEY_ID` environment variable is used.
aws_secret_access_key	The AWS secret access key to use. If nullopt, the value of the `AWS_SECRET_ACCESS_KEY` environment variable is used.
aws_session_token	The AWS session token to use. If nullopt, the value of the `AWS_SESSION_TOKEN` environment variable is used.

◆ S3Endpoint() [2/2]

kvikio::S3Endpoint::S3Endpoint	(	std::pair< std::string, std::string >	bucket_and_object_names,
		std::optional< std::string >	aws_region = `std::nullopt`,
		std::optional< std::string >	aws_access_key = `std::nullopt`,
		std::optional< std::string >	aws_secret_access_key = `std::nullopt`,
		std::optional< std::string >	aws_endpoint_url = `std::nullopt`,
		std::optional< std::string >	aws_session_token = `std::nullopt`
	)

Create a S3 endpoint from a bucket and object name.

Parameters

bucket_and_object_names	The bucket and object names of the S3 bucket.
aws_region	The AWS region, such as "us-east-1", to use. If nullopt, the value of the `AWS_DEFAULT_REGION` environment variable is used.
aws_access_key	The AWS access key to use. If nullopt, the value of the `AWS_ACCESS_KEY_ID` environment variable is used.
aws_secret_access_key	The AWS secret access key to use. If nullopt, the value of the `AWS_SECRET_ACCESS_KEY` environment variable is used.
aws_endpoint_url	Overwrite the endpoint url (including the protocol part) by using the scheme: "<aws_endpoint_url>/<bucket_name>/<object_name>". If nullopt, the value of the `AWS_ENDPOINT_URL` environment variable is used. If this is also not set, the regular AWS url scheme is used: "https://<bucket_name>.s3.<region>.amazonaws.com/<object_name>".
aws_session_token	The AWS session token to use. If nullopt, the value of the `AWS_SESSION_TOKEN` environment variable is used.

Member Function Documentation

◆ get_file_size()

std::size_t kvikio::S3Endpoint::get_file_size ( )

overridevirtual

Get the size of the remote file.

Returns: The file size

Implements kvikio::RemoteEndpoint.

◆ is_url_valid()

static bool kvikio::S3Endpoint::is_url_valid ( std::string const & url )

staticnoexcept

Whether the given URL is valid for S3 endpoints (excluding presigned URL).

Parameters

url A URL.

Returns: Boolean answer.

◆ parse_s3_url()

static std::pair<std::string, std::string> kvikio::S3Endpoint::parse_s3_url ( std::string const & s3_url )

static

Given an url like "s3://<bucket>/<object>", return the name of the bucket and object.

Exceptions

std::invalid_argument if url is ill-formed or is missing the bucket or object name.

Parameters

s3_url S3 url.

Returns: Pair of strings: [bucket-name, object-name].

◆ setopt()

void kvikio::S3Endpoint::setopt ( CurlHandle & curl )

overridevirtual

Set needed connection options on a curl handle.

Subsequently, a call to curl.perform() should connect to the endpoint.

Parameters

curl	The curl handle.

Implements kvikio::RemoteEndpoint.

◆ str()

std::string kvikio::S3Endpoint::str ( ) const

overridevirtual

Get a description of this remote point instance.

Returns: A string description.

Implements kvikio::RemoteEndpoint.

◆ url_from_bucket_and_object()

static std::string kvikio::S3Endpoint::url_from_bucket_and_object	(	std::string	bucket_name,
		std::string	object_name,
		std::optional< std::string >	aws_region,
		std::optional< std::string >	aws_endpoint_url
	)

static

Get url from a AWS S3 bucket and object name.

Exceptions

std::invalid_argument if no region is specified and no default region is specified in the environment.

Parameters

bucket_name	The name of the S3 bucket.
object_name	The name of the S3 object.
aws_region	The AWS region, such as "us-east-1", to use. If nullopt, the value of the `AWS_DEFAULT_REGION` environment variable is used.
aws_endpoint_url	Overwrite the endpoint url (including the protocol part) by using the scheme: "<aws_endpoint_url>/<bucket_name>/<object_name>". If nullopt, the value of the `AWS_ENDPOINT_URL` environment variable is used. If this is also not set, the regular AWS url scheme is used: "https://<bucket_name>.s3.<region>.amazonaws.com/<object_name>".

The documentation for this class was generated from the following file:

remote_handle.hpp

Public Member Functions

Static Public Member Functions

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ S3Endpoint() [1/2]

◆ S3Endpoint() [2/2]

Member Function Documentation

◆ get_file_size()

◆ is_url_valid()

◆ parse_s3_url()

◆ setopt()

◆ str()

◆ url_from_bucket_and_object()