academic_observatory_workflows.s5cmd

Module Contents

Classes

S5CmdCpConfig	Configuration for S5Cmd cp command
S5Cmd

class academic_observatory_workflows.s5cmd.S5CmdCpConfig(flatten_dir: bool = False, no_overwrite: bool = False, overwrite_if_size: bool = False, overwrite_if_newer: bool = False)

Configuration for S5Cmd cp command

Parameters:

• flatten_dir – Whether to flatten the directory structure

• no_overwrite – Whether to not overwrite files if they already exist

• overwrite_if_size – Whether to overwrite files only if source size differs

• overwrite_if_newer – Whether to overwrite files only if source is newer

__str__()

Return str(self).

class academic_observatory_workflows.s5cmd.S5Cmd(access_credentials: Tuple[str, str], out_stream: None | int | TextIO = PIPE)

_uri(uri: str)

_initialise_command(uri: str)

Initializes the command for the given bucket URI. :param uri: The URI being accessed. :return: The initialized command.

_bucket_credentials()

download_from_bucket(uris: List[str] | str, local_path: str, cp_config: S5CmdCpConfig = None) → Tuple[bytes, bytes, int]

Downloads file(s) from a bucket using s5cmd and a supplied list of URIs.

Parameters:

• uris – The URI or list of URIs to download.

• local_path – The local path to download to.

• cp_config – Configuration for the s5cmd cp command.

Returns:

A tuple of (stdout, stderr, s5cmd exit code).

upload_to_bucket(files: List[str] | str, bucket_uri: str, cp_config: S5CmdCpConfig = None) → Tuple[bytes, bytes, int]

Uploads file(s) to a bucket using s5cmd and a supplied list of local file paths.

Parameters:

• files – The file(s) to upload.

• cp_config – Configuration for the s5cmd cp command.

Bucket_uri:

The bucket URI to upload to.

Returns:

A tuple of (stdout, stderr, s5cmd exit code).