API Reference
This API reference describes the components of the AQIS staging library.
cli.py
Overview
cli.py is the CLI entry point for the streaming workflow. It:
- Loads config from INI
- Retrieves secrets (Infisical optional)
- Creates source/destination adaptors
- Builds a file manifest
- Streams files between backends in parallel
Supported backends: FS, S3, iRODS, SCP.
Transfers use chunked streaming with size validation and optional full checksums.
Constants
ADAPTOR_MAP maps backend names to adaptor classes (FS, S3, IRODS, SCP).
ManifestType = (relative_path, size_bytes): mapping of file path to the size of the file.
This is precalculated and allows for faster check-summing and skipping of existing files.
_stream_chunk_size default: 64 MiB
Core Functions and Classes
StreamingFileRunner
Class that is responsible for the work of copying one file.
stream_single_file(self, source_file_str: ManifestType, position: int = 1) executes the actual streaming copying, with source_file_str as the path of the source file.
position is used for progress tracking with tqdm (the location of the progress bar in the terminal output).
init_worker(...)
def init_worker(
source_path: str,
upload_dest_base: str,
source_adaptor_cls: Type[AdaptorClass],
dest_adaptor_cls: Type[AdaptorClass],
stream_chunk_size: int,
checksum_strategy: str,
checksum_algorithm: str,
)
Initializes worker processes with adaptors, paths, chunk size, and checksum settings. This is a helper function to create a new StreamingFileRunner for each worker process.
process_item(item: ManifestType, free_indices: Queue)
Streams one file using a worker-local runner and manages progress slots.
dest_path_check(source_path: str, dest_path: str, single_file: bool) -> str
Adjusts destination path and creates missing folders at destination for directory transfers.
load_configuration(args)
Loads [env] config and secrets (Infisical or [creds] fallback).
main()
Coordinates the workflow:
- Parse args
- Load config
- Validate backends
- Build manifest
- Start worker pool
- Stream files in parallel
- Validate size and checksums
- Report results
Required environment variables
SOURCE_TYPE,DEST_TYPESOURCE_PATH,DEST_PATH
Optional environment variables
CONCURRENCY,STREAM_CHUNK_SIZE_MBCHECKSUM_STRATEGY(SIZE,FULL,NONE)
Parses command-line arguments for the staging workflow.
Supported Arguments
| Argument | Required | Default | Description |
|---|---|---|---|
--config |
No | config.ini |
Main workflow INI file |
--infisical-src-config |
No | None |
Optional Infisical config for source secrets |
--infisical-dst-config |
No | None |
Optional Infisical config for destination secrets |
Returns
An argparse.Namespace containing the parsed command-line arguments.
Example
python -m staging_workflow.cli \
--config configs/workflows/s3_to_irods.ini \
--infisical-src-config configs/infisical/src.ini \
--infisical-dst-config configs/infisical/dst.ini
load_configuration()
def load_configuration(args: argparse.Namespace) -> None:
Loads workflow configuration into environment variables.
The function always loads the [env] section from the main configuration file. It then either loads secrets from Infisical or falls back to the [creds] section of the main configuration file.
| Name | Type | Description |
|---|---|---|
args |
argparse.Namespace |
Parsed command-line arguments |
Behaviour
The configuration loading order is:
- Load
[env]from args.config. - If Infisical source or destination configs are provided:
- Load secrets from Infisical.
- Do not load [creds] from the main config.
- If no Infisical config is provided:
- Load
[creds]from the main config if it exists.
Example
args = parse_args()
load_configuration(args)
main()
def main():
Main entry point for the streaming staging workflow.
This function coordinates the complete transfer lifecycle.
Workflow Steps
- Parse command-line arguments.
- Load configuration and secrets.
- Read required environment variables.
- Validate source and destination backend types.
- Create source adaptor for manifest generation.
- Run source compliance checks.
- Determine destination path layout.
- Create a pool of worker processes.
- Stream files in parallel.
- Validate file sizes and optional checksums.
- Report bandwidth and transfer result.
Required Environment Variables
| Variable | Description |
|---|---|
SOURCE_TYPE |
Source backend type: FS, S3, IRODS, or SCP |
DEST_TYPE |
Destination backend type: FS, S3, IRODS, or SCP |
SOURCE_PATH |
Source path to transfer |
DEST_PATH |
Destination path |
Optional Environment Variables
| Variable | Default | Description |
|---|---|---|
MAX_FILE_SIZE_MB |
Unlimited | Maximum allowed file size in MiB |
CONCURRENCY |
10 |
Number of parallel worker processes |
STREAM_CHUNK_SIZE_MB |
64 |
Streaming chunk size in MiB |
CHECKSUM_STRATEGY |
SIZE |
Validation strategy |
CHECKSUM_ALGORITHM |
md5 |
Hashing algorithm for full checksums |
Checksum Strategies
| Strategy | Description |
|---|---|
SIZE |
Validate expected byte count and destination size |
FULL |
Hash source stream and compare with destination checksum |
Note
The manifest compliance check currently uses:
IntegrityManager("SIZE", max_file_size_mb)
This means the manifest is built using size-based validation and file size limits. Full checksum validation is handled later during streaming by StreamingFileRunner.
Classes
ProgressReader
Wraps a stream to:
- Track progress
- Count bytes
- Update checksum during reads
StreamingFileRunner
Handles single-file transfers.
Key responsibilities:
- Resolve source/destination paths
- Stream data in chunks
- Track bytes and progress
- Validate size and optional checksum
Checksum modes:
SIZE: byte count onlyFULL: hash comparison (e.g. md5)NONE: no checks performed
Execution Model
- Uses
ProcessPoolExecutor - One task per file
- Workers stream files independently
Error Handling
- Failures are tracked per file
- Workflow continues on errors
- Final report lists failed items
Storage Adaptors
StorageAdaptor (base class)
Defines interface for all backends:
Required
list_objects(path): Get a list of all files/objects in a pathopen_read(path): Open a single source object/file as a readable binary stream.open_write()orupload_stream(): Open a single destination object/file as a writable binary stream. Some backends do not support this, so useupload_streaminstead.get_size(path): Get the size of a local or remote file in bytes.
Optional
prepare_for_stream_write(): Checks if a file already exists at the destination and can be skipped.get_remote_checksum(): Backend native checksum (not md5), e.g. ETag for S3.
Backend Implementations
FSAdaptor
- Local filesystem
- Full read/write support
- Recursive listing
S3Adaptor
- Uses boto3
open_write()unsupported, usesupload_stream()instead- Supports multipart upload
IRODSAdaptor
- Uses iRODS session
- Supports read/write, listing, checksum
- Auto-creates collections
SCPAdaptor
- Uses Paramiko (SSH/SFTP)
- Supports remote read/write and listing
- Handles file skip/resume