academic_observatory_workflows.s5cmd
====================================

.. py:module:: academic_observatory_workflows.s5cmd


Classes
-------

.. autoapisummary::

   academic_observatory_workflows.s5cmd.S5CmdCpConfig
   academic_observatory_workflows.s5cmd.S5Cmd


Module Contents
---------------

.. py:class:: S5CmdCpConfig(flatten_dir: bool = False, no_overwrite: bool = False, overwrite_if_size: bool = False, overwrite_if_newer: bool = False)

   Configuration for S5Cmd cp command

   :param flatten_dir: Whether to flatten the directory structure
   :param no_overwrite: Whether to not overwrite files if they already exist
   :param overwrite_if_size: Whether to overwrite files only if source size differs
   :param overwrite_if_newer: Whether to overwrite files only if source is newer


   .. py:attribute:: flatten_dir
      :value: False



   .. py:attribute:: no_overwrite
      :value: False



   .. py:attribute:: overwrite_if_size
      :value: False



   .. py:attribute:: overwrite_if_newer
      :value: False



   .. py:method:: __str__()


.. py:class:: S5Cmd(access_credentials: Tuple[str, str], out_stream: Union[None, int, TextIO] = PIPE)

   .. py:attribute:: access_credentials


   .. py:attribute:: out_stream
      :value: -1



   .. py:attribute:: uri_identifier_regex
      :value: '^[a-zA-Z0-9_]{2}://'



   .. py:method:: _uri(uri: str)


   .. py:method:: _initialise_command(uri: str)

      Initializes the command for the given bucket URI.
      :param uri: The URI being accessed.
      :return: The initialized command.



   .. py:method:: _bucket_credentials()


   .. py:method:: download_from_bucket(uris: Union[List[str], str], local_path: str, cp_config: S5CmdCpConfig = None) -> Tuple[bytes, bytes, int]

      Downloads file(s) from a bucket using s5cmd and a supplied list of URIs.

      :param uris: The URI or list of URIs to download.
      :param local_path: The local path to download to.
      :param cp_config: Configuration for the s5cmd cp command.
      :return: A tuple of (stdout, stderr, s5cmd exit code).



   .. py:method:: upload_to_bucket(files: Union[List[str], str], bucket_uri: str, cp_config: S5CmdCpConfig = None) -> Tuple[bytes, bytes, int]

      Uploads file(s) to a bucket using s5cmd and a supplied list of local file paths.

      :param files: The file(s) to upload.
      :bucket_uri: The bucket URI to upload to.
      :param cp_config: Configuration for the s5cmd cp command.
      :return: A tuple of (stdout, stderr, s5cmd exit code).