Getting Started

Below, we cover the following:

How to install and configure the AQIS staging library

For installation we recommend a virtual environment; please make sure Python 3.12 or newer are available:

Make a virtual environment:

python -m venv ~/aqis_venv
cd ~/aqis_venv
source ~/aqis_venv/bin/activate

Clone the repo:

git clone https://opencode.it4i.eu/exa4mind-private/platform/aqis/staging-library.git
cd ~/staging-library

Install the required packages:

pip install -e .

How to use the AQIS staging library

Basic usage

python cli.py --config config.ini

Command-line options

python cli.py \
  --config config.ini \
  --infisical-src-config src_infisical.ini \
  --infisical-dst-config dst_infisical.ini
Option Description
--config Main workflow INI file. Defaults to config.ini.
--infisical-src-config Optional Infisical config for source credentials.
--infisical-dst-config Optional Infisical config for destination credentials.

Example config.ini

[env]
SOURCE_TYPE=S3
DEST_TYPE=IRODS

SOURCE_PATH=/bucket-name/path/to/source
DEST_PATH=/tempZone/home/rods/staging_output

CHECKSUM_STRATEGY=FULL
MAX_FILE_SIZE_MB=

CONCURRENCY=10

[creds]
SRC_S3_ENDPOINT_URL=https://minio.example.org
SRC_S3_ACCESS_KEY_ID=your-access-key
SRC_S3_SECRET_ACCESS_KEY=your-secret-key
SRC_S3_REGION=us-east-1
SRC_S3_BUCKET=bucket-name

DST_IRODS_HOST=irods.example.org
DST_IRODS_PORT=1247
DST_IRODS_USER=rods
DST_IRODS_PASSWORD=your-password
DST_IRODS_ZONE=tempZone

Required Environment Variables

These must be set either in [env] or directly in the shell:

SOURCE_TYPE
DEST_TYPE
SOURCE_PATH
DEST_PATH

Supported values for SOURCE_TYPE and DEST_TYPE are:

FS
S3
IRODS
SCP

Example:

SOURCE_TYPE=S3
DEST_TYPE=FS
SOURCE_PATH=/my-bucket/data
DEST_PATH=/tmp/output

Checksum Strategies

CHECKSUM_STRATEGY=FULL
Strategy Meaning
FULL Performs full checksum/integrity verification.
SIZE Verifies transferred files by size only.
NONE Skips integrity verification.

Use FULL when correctness matters most. Use SIZE for faster validation when checksums are expensive.

File Size Limit

MAX_FILE_SIZE_MB=5000

If set, files larger than this limit are excluded during compliance checking.

To allow unlimited file size:

MAX_FILE_SIZE_MB=

Parallel Transfer Settings

CONCURRENCY=10
Variable Description
CONCURRENCY Number of parallel worker processes. Defaults to 10.

Resume Behaviour

The workflow enables resume mode automatically inside each worker.

Before downloading a file, it checks whether a file with the same name and size already exists at the destination. If the destination file exists and the size matches, the transfer is skipped.

This avoids re-uploading files that were already transferred successfully.

Infisical Secrets Configuration

The workflow can load credentials from Infisical instead of reading them directly from the main workflow INI file. This is handled by the InfisicalObj helper class.

When an Infisical config is provided with:

python cli.py \
  --config workflow.ini \
  --infisical-src-config src_infisical.ini \
  --infisical-dst-config dst_infisical.ini

the workflow loads the normal [env] section from workflow.ini, then fetches credentials from Infisical and sets them as environment variables.

In this mode, the [creds] section in the main workflow config is not loaded.

Example Infisical Config

[conn]
infisical_url=https://infisical.example.org
authorization=your-infisical-token
environment=dev
workspace_id=your-workspace-id
workspace_slug=your-workspace-slug
secret_path=/STAGING/S3

[secrets]
secrets_list=SRC_S3_ENDPOINT_URL,SRC_S3_ACCESS_KEY_ID,SRC_S3_SECRET_ACCESS_KEY,SRC_S3_REGION,SRC_S3_BUCKET

A short example workflow using the AQIS staging library

Transfer from local filesystem to S3:

# fs_to_s3.ini

[env]
SOURCE_TYPE = FS
DEST_TYPE = S3
SOURCE_PATH = /data/input
DEST_PATH = s3://my-bucket/output
CHECKSUM_STRATEGY = FULL

[creds]
DST_S3_ENDPOINT_URL=https://s3.example.org
DST_S3_ACCESS_KEY_ID=your_access_key_id
DST_S3_SECRET_ACCESS_KEY=your_secret_access_key
DST_S3_REGION=your-region-1
DST_S3_BUCKET=your-bucket-name

Run with:

python cli.py --config fs_to_s3.ini

Output

The workflow logs:

Staging Workflow Started
Transfer Mode: FS -> S3
Source: /source/path
Dest: /destination/path
Checksum Strategy = FULL
MaxFileSize = Unlimited MB
Concurrency = 10
StreamChunkSize = 64 MB
ChecksumStrategy = FULL
ChecksumAlgorithm = md5

Progress is shown with:

Files being processed: 100%|██████████| ...

At the end, it reports:

SUCCESS: Workflow finished successfully.

Or, if failures occurred:

FAILURE: 3 files did not transfer successfully.
Total size not transferred: ...
Files not transferred = [...]