ID-to-Dataset Resolution

EXA4MIND aims at facilitating FAIR Research Data Management and Data Publication from all the various backends in a generalized EXA4MIND/EDD instance. To achieve this, it provides a mechanism for resolving persistent identifiers (PIDs) to datasets or dataset subsets, enabling consistent and reproducible data access in line with FAIR principles.

Overview

The ID-to-Dataset Resolution submodule introduces the FAIR Data Subsetting API, which enables:

  • Assignment of persistent identifiers (PIDs) to specific data products or subsets
  • Mapping of PIDs to backend systems and executable queries
  • Retrieval of data products via a standardized API interface

Each data product is associated with a UUID, which encapsulates:

  • The target backend system
  • The query (or object path) used to retrieve the data
  • The required access credentials

Credentials for internal databases are securely managed using Infisical, as described in the AQIS Engine documentation.

The API exposes RESTful endpoints for retrieving:

  • Structured data in JSON format
  • Files in their original format

A modular adapter-based architecture enables support for heterogeneous data backends. Based on the UUID, the API dynamically selects the appropriate backend adapter to execute the request. This design ensures extensibility, allowing new backend systems to be integrated with minimal effort.

The following backend systems are currently supported:

  • iRODS
  • MariaDB
  • Milvus
  • MongoDB
  • MySQL
  • OpenSearch
  • PostgreSQL
  • S3-compatible storage systems (e.g., MinIO, AWS S3)

FAIR Data Subsetting API

Features

  • A modular architecture ensures scalability and ease of integration for additional backend systems.
  • Secure credential handling via Infisical ensures that sensitive connection details are never exposed in the API layer.
  • The system supports both structured queries and object-based retrieval, making it suitable for a wide range of data types.

Requirements

To deploy and operate the FAIR Data Subsetting API, the following components are required:

  • Infisical Secrets Manager for secure credential storage
  • SQLite database for internal storage of:
  • UUID-to-query mappings
  • Backend configuration metadata

How It Works

Data Retrieval Workflow

  1. A client sends a GET request via:

    • /api/download?uuid=<UUID> → returns data in JSON format
    • /api/download_file?uuid=<UUID> → returns data in file format

  2. The API resolves the provided UUID by querying the internal SQLite database.

  3. The corresponding backend system and query definition are identified.
  4. The appropriate backend adapter is dynamically loaded.
  5. Required credentials are securely retrieved from Infisical.
  6. The query (or object retrieval request) is executed against the backend.
  7. The result is returned to the client in the requested format.

Creating New Mappings

New UUID mappings can be created by authorized users (e.g., data owners or administrators) via:

  • The frontend GUI
  • create_mapping endpoint

This operation requires appropriate access to the internal SQLite database.

Installation Guide

After cloning this repository, all the required packages can be installed with pip using the following commands in the root directory:

python3 -m venv venv
source venv/bin/activate
pip install --editable .

NOTE: --editable is required, otherwise changes to the code will not be picked up in the virtual environment.

Infisical Setup

Next, an instance of Infisical Secrets Manager needs to be deployed where all the datastore credentials will be securely stored. A guide can be found on this page. Furthermore, there is an official installation guide provided by Infisical.

The cred_manager.py needs an environment (.env) file with the following parameters (.env file should be created in the utils directory):

TOKEN="<infisical-service-token>" # starts with `st.`
HOST="<infisical-host>" # needs to be full url. e.g. http://localhost:80/
PROJECT_ID="<infisical-project-id>" # a UUID

NOTE: The infisical token with limited scopes can be generated following the steps defined on the Infisical API documentation page.

SQLite Setup

A SQLite database is needed for the FAIR API, from which the corresponding backend and query can be retrieved by each adapter. The details of the table name and the rows needed in this internal database is given below.

CREATE TABLE IF NOT EXISTS uuid_mapping (
    uuid TEXT PRIMARY KEY,
    db_server TEXT,
    db_user TEXT, -- Path to the secret in Infisical. For eg. f"{base_folder}/postgres/DB_USER"
    db_pass TEXT, -- Path to the secret in Infisical. For eg. f"{base_folder}/postgres/DB_PASS"
    db_type TEXT,
    db_details TEXT,
    query_str TEXT
)

This should be executed to create a database file with the name internal_metadata.db. Example: sqlite3 internal_metadata.db < sqlite-create.sql

NOTE: All the fields above are mandatory for the adaptors to work.

Once everything is set up, the FAIR data Subsetting API can be run using the command:

uvicorn main:app --reload

The OpenAPI front end of the application can be found on under http://127.0.0.1:8000/docs#/

The front end GUI to fetch data or create UUIDs from queries can be found the root url: http://127.0.0.1:8000/