API Reference

This API reference describes the components of the AQIS staging library.

cli.py

Overview

cli.py is the CLI entry point for the streaming workflow. It:

  • Loads config from INI
  • Retrieves secrets (Infisical optional)
  • Creates source/destination adaptors
  • Builds a file manifest
  • Streams files between backends in parallel

Supported backends: FS, S3, iRODS, SCP.
Transfers use chunked streaming with size validation and optional full checksums.

Constants

ADAPTOR_MAP maps backend names to adaptor classes (FS, S3, IRODS, SCP).

ManifestType = (relative_path, size_bytes): mapping of file path to the size of the file. This is precalculated and allows for faster check-summing and skipping of existing files.

_stream_chunk_size default: 64 MiB

Core Functions and Classes

StreamingFileRunner

Class that is responsible for the work of copying one file. stream_single_file(self, source_file_str: ManifestType, position: int = 1) executes the actual streaming copying, with source_file_str as the path of the source file. position is used for progress tracking with tqdm (the location of the progress bar in the terminal output).

init_worker(...)

def init_worker(
    source_path: str,
    upload_dest_base: str,
    source_adaptor_cls: Type[AdaptorClass],
    dest_adaptor_cls: Type[AdaptorClass],
    stream_chunk_size: int,
    checksum_strategy: str,
    checksum_algorithm: str,
)

Initializes worker processes with adaptors, paths, chunk size, and checksum settings. This is a helper function to create a new StreamingFileRunner for each worker process.

process_item(item: ManifestType, free_indices: Queue)

Streams one file using a worker-local runner and manages progress slots.

dest_path_check(source_path: str, dest_path: str, single_file: bool) -> str

Adjusts destination path and creates missing folders at destination for directory transfers.

load_configuration(args)

Loads [env] config and secrets (Infisical or [creds] fallback).

main()

Coordinates the workflow:

  1. Parse args
  2. Load config
  3. Validate backends
  4. Build manifest
  5. Start worker pool
  6. Stream files in parallel
  7. Validate size and checksums
  8. Report results

Required environment variables

  • SOURCE_TYPE, DEST_TYPE
  • SOURCE_PATH, DEST_PATH

Optional environment variables

  • CONCURRENCY, STREAM_CHUNK_SIZE_MB
  • CHECKSUM_STRATEGY (SIZE, FULL, NONE)

Parses command-line arguments for the staging workflow.

Supported Arguments

Argument Required Default Description
--config No config.ini Main workflow INI file
--infisical-src-config No None Optional Infisical config for source secrets
--infisical-dst-config No None Optional Infisical config for destination secrets

Returns

An argparse.Namespace containing the parsed command-line arguments.

Example

python -m staging_workflow.cli \
  --config configs/workflows/s3_to_irods.ini \
  --infisical-src-config configs/infisical/src.ini \
  --infisical-dst-config configs/infisical/dst.ini

load_configuration()

def load_configuration(args: argparse.Namespace) -> None:

Loads workflow configuration into environment variables.

The function always loads the [env] section from the main configuration file. It then either loads secrets from Infisical or falls back to the [creds] section of the main configuration file.

Name Type Description
args argparse.Namespace Parsed command-line arguments

Behaviour

The configuration loading order is:

  • Load [env] from args.config.
  • If Infisical source or destination configs are provided:
  • Load secrets from Infisical.
  • Do not load [creds] from the main config.
  • If no Infisical config is provided:
  • Load [creds] from the main config if it exists.

Example

args = parse_args()
load_configuration(args)

main()

def main():

Main entry point for the streaming staging workflow.

This function coordinates the complete transfer lifecycle.

Workflow Steps

  1. Parse command-line arguments.
  2. Load configuration and secrets.
  3. Read required environment variables.
  4. Validate source and destination backend types.
  5. Create source adaptor for manifest generation.
  6. Run source compliance checks.
  7. Determine destination path layout.
  8. Create a pool of worker processes.
  9. Stream files in parallel.
  10. Validate file sizes and optional checksums.
  11. Report bandwidth and transfer result.

Required Environment Variables

Variable Description
SOURCE_TYPE Source backend type: FS, S3, IRODS, or SCP
DEST_TYPE Destination backend type: FS, S3, IRODS, or SCP
SOURCE_PATH Source path to transfer
DEST_PATH Destination path

Optional Environment Variables

Variable Default Description
MAX_FILE_SIZE_MB Unlimited Maximum allowed file size in MiB
CONCURRENCY 10 Number of parallel worker processes
STREAM_CHUNK_SIZE_MB 64 Streaming chunk size in MiB
CHECKSUM_STRATEGY SIZE Validation strategy
CHECKSUM_ALGORITHM md5 Hashing algorithm for full checksums

Checksum Strategies

Strategy Description
SIZE Validate expected byte count and destination size
FULL Hash source stream and compare with destination checksum

Note

The manifest compliance check currently uses:

IntegrityManager("SIZE", max_file_size_mb)

This means the manifest is built using size-based validation and file size limits. Full checksum validation is handled later during streaming by StreamingFileRunner.


Classes

ProgressReader

Wraps a stream to:

  • Track progress
  • Count bytes
  • Update checksum during reads

StreamingFileRunner

Handles single-file transfers.

Key responsibilities:

  • Resolve source/destination paths
  • Stream data in chunks
  • Track bytes and progress
  • Validate size and optional checksum

Checksum modes:

  • SIZE: byte count only
  • FULL: hash comparison (e.g. md5)
  • NONE: no checks performed

Execution Model

  • Uses ProcessPoolExecutor
  • One task per file
  • Workers stream files independently

Error Handling

  • Failures are tracked per file
  • Workflow continues on errors
  • Final report lists failed items

Storage Adaptors

StorageAdaptor (base class)

Defines interface for all backends:

Required

  • list_objects(path): Get a list of all files/objects in a path
  • open_read(path): Open a single source object/file as a readable binary stream.
  • open_write() or upload_stream(): Open a single destination object/file as a writable binary stream. Some backends do not support this, so use upload_stream instead.
  • get_size(path): Get the size of a local or remote file in bytes.

Optional

  • prepare_for_stream_write(): Checks if a file already exists at the destination and can be skipped.
  • get_remote_checksum(): Backend native checksum (not md5), e.g. ETag for S3.

Backend Implementations

FSAdaptor

  • Local filesystem
  • Full read/write support
  • Recursive listing

S3Adaptor

  • Uses boto3
  • open_write() unsupported, uses upload_stream() instead
  • Supports multipart upload

IRODSAdaptor

  • Uses iRODS session
  • Supports read/write, listing, checksum
  • Auto-creates collections

SCPAdaptor

  • Uses Paramiko (SSH/SFTP)
  • Supports remote read/write and listing
  • Handles file skip/resume