A remote endpoint using AWS's S3 protocol. More...

#include <remote_handle.hpp>

Inheritance diagram for kvikio::S3Endpoint:

Public Member Functions
	S3Endpoint (std::string url, std::optional< std::string > aws_region=std::nullopt, std::optional< std::string > aws_access_key=std::nullopt, std::optional< std::string > aws_secret_access_key=std::nullopt, std::optional< std::string > aws_session_token=std::nullopt)
	Create a S3 endpoint from a url. More...

	S3Endpoint (std::pair< std::string, std::string > bucket_and_object_names, std::optional< std::string > aws_region=std::nullopt, std::optional< std::string > aws_access_key=std::nullopt, std::optional< std::string > aws_secret_access_key=std::nullopt, std::optional< std::string > aws_endpoint_url=std::nullopt, std::optional< std::string > aws_session_token=std::nullopt)
	Create a S3 endpoint from a bucket and object name. More...

void	setopt (CurlHandle &curl) override
	Set needed connection options on a curl handle. More...

std::string	str () const override
	Get a description of this remote point instance. More...

std::size_t	get_file_size () override
	Get the size of the remote file. More...

void	setup_range_request (CurlHandle &curl, std::size_t file_offset, std::size_t size) override
	Set up the range request in order to read part of a file given the file offset and read size.

Public Member Functions inherited from kvikio::RemoteEndpoint
RemoteEndpointType	remote_endpoint_type () const noexcept
	Get the type of the remote file. More...

Static Public Member Functions
static std::string	url_from_bucket_and_object (std::string bucket_name, std::string object_name, std::optional< std::string > aws_region, std::optional< std::string > aws_endpoint_url)
	Get url from a AWS S3 bucket and object name. More...

static std::pair< std::string, std::string >	parse_s3_url (std::string const &s3_url)
	Given an url like "s3://<bucket>/<object>", return the name of the bucket and object. More...

static bool	is_url_valid (std::string const &url) noexcept
	Whether the given URL is valid for S3 endpoints (excluding presigned URL). More...

Additional Inherited Members
Protected Member Functions inherited from kvikio::RemoteEndpoint
	RemoteEndpoint (RemoteEndpointType remote_endpoint_type)

Protected Attributes inherited from kvikio::RemoteEndpoint
RemoteEndpointType	_remote_endpoint_type {RemoteEndpointType::AUTO}

Detailed Description

A remote endpoint using AWS's S3 protocol.

Definition at line 143 of file remote_handle.hpp.

Constructor & Destructor Documentation

◆ S3Endpoint() [1/2]

kvikio::S3Endpoint::S3Endpoint	(	std::string	url,
		std::optional< std::string >	aws_region = `std::nullopt`,
		std::optional< std::string >	aws_access_key = `std::nullopt`,
		std::optional< std::string >	aws_secret_access_key = `std::nullopt`,
		std::optional< std::string >	aws_session_token = `std::nullopt`
	)

Create a S3 endpoint from a url.

Parameters

url	The full http url to the S3 file. NB: this should be an url starting with "http://" or "https://". If you have an S3 url of the form "s3://<bucket>/<object>", please use `S3Endpoint::parse_s3_url()` and `S3Endpoint::url_from_bucket_and_object() to convert it.
aws_region	The AWS region, such as "us-east-1", to use. If nullopt, the value of the `AWS_DEFAULT_REGION` environment variable is used.
aws_access_key	The AWS access key to use. If nullopt, the value of the `AWS_ACCESS_KEY_ID` environment variable is used.
aws_secret_access_key	The AWS secret access key to use. If nullopt, the value of the `AWS_SECRET_ACCESS_KEY` environment variable is used.
aws_session_token	The AWS session token to use. If nullopt, the value of the `AWS_SESSION_TOKEN` environment variable is used.

◆ S3Endpoint() [2/2]

kvikio::S3Endpoint::S3Endpoint	(	std::pair< std::string, std::string >	bucket_and_object_names,
		std::optional< std::string >	aws_region = `std::nullopt`,
		std::optional< std::string >	aws_access_key = `std::nullopt`,
		std::optional< std::string >	aws_secret_access_key = `std::nullopt`,
		std::optional< std::string >	aws_endpoint_url = `std::nullopt`,
		std::optional< std::string >	aws_session_token = `std::nullopt`
	)

Create a S3 endpoint from a bucket and object name.

Parameters

bucket_and_object_names	The bucket and object names of the S3 bucket.
aws_region	The AWS region, such as "us-east-1", to use. If nullopt, the value of the `AWS_DEFAULT_REGION` environment variable is used.
aws_access_key	The AWS access key to use. If nullopt, the value of the `AWS_ACCESS_KEY_ID` environment variable is used.
aws_secret_access_key	The AWS secret access key to use. If nullopt, the value of the `AWS_SECRET_ACCESS_KEY` environment variable is used.
aws_endpoint_url	Overwrite the endpoint url (including the protocol part) by using the scheme: "<aws_endpoint_url>/<bucket_name>/<object_name>". If nullopt, the value of the `AWS_ENDPOINT_URL` environment variable is used. If this is also not set, the regular AWS url scheme is used: "https://<bucket_name>.s3.<region>.amazonaws.com/<object_name>".
aws_session_token	The AWS session token to use. If nullopt, the value of the `AWS_SESSION_TOKEN` environment variable is used.

Member Function Documentation

◆ get_file_size()

std::size_t kvikio::S3Endpoint::get_file_size ( )

overridevirtual

Get the size of the remote file.

Returns: The file size

Implements kvikio::RemoteEndpoint.

◆ is_url_valid()

static bool kvikio::S3Endpoint::is_url_valid ( std::string const & url )

staticnoexcept

Whether the given URL is valid for S3 endpoints (excluding presigned URL).

Parameters

url A URL.

Returns: Boolean answer.

◆ parse_s3_url()

static std::pair<std::string, std::string> kvikio::S3Endpoint::parse_s3_url ( std::string const & s3_url )

static

Given an url like "s3://<bucket>/<object>", return the name of the bucket and object.

Exceptions

std::invalid_argument if url is ill-formed or is missing the bucket or object name.

Parameters

s3_url S3 url.

Returns: Pair of strings: [bucket-name, object-name].

◆ setopt()

void kvikio::S3Endpoint::setopt ( CurlHandle & curl )

overridevirtual

Set needed connection options on a curl handle.

Subsequently, a call to curl.perform() should connect to the endpoint.

Parameters

curl	The curl handle.

Implements kvikio::RemoteEndpoint.

◆ str()

std::string kvikio::S3Endpoint::str ( ) const

overridevirtual

Get a description of this remote point instance.

Returns: A string description.

Implements kvikio::RemoteEndpoint.

◆ url_from_bucket_and_object()

static std::string kvikio::S3Endpoint::url_from_bucket_and_object	(	std::string	bucket_name,
		std::string	object_name,
		std::optional< std::string >	aws_region,
		std::optional< std::string >	aws_endpoint_url
	)

static

Get url from a AWS S3 bucket and object name.

Exceptions

std::invalid_argument if no region is specified and no default region is specified in the environment.

Parameters

bucket_name	The name of the S3 bucket.
object_name	The name of the S3 object.
aws_region	The AWS region, such as "us-east-1", to use. If nullopt, the value of the `AWS_DEFAULT_REGION` environment variable is used.
aws_endpoint_url	Overwrite the endpoint url (including the protocol part) by using the scheme: "<aws_endpoint_url>/<bucket_name>/<object_name>". If nullopt, the value of the `AWS_ENDPOINT_URL` environment variable is used. If this is also not set, the regular AWS url scheme is used: "https://<bucket_name>.s3.<region>.amazonaws.com/<object_name>".

The documentation for this class was generated from the following file:

remote_handle.hpp

Public Member Functions

Static Public Member Functions

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ S3Endpoint() [1/2]

◆ S3Endpoint() [2/2]

Member Function Documentation

◆ get_file_size()

◆ is_url_valid()

◆ parse_s3_url()

◆ setopt()

◆ str()

◆ url_from_bucket_and_object()