OpenSearch Metadata Exporter

The extractor presented here works as an ETL (Extract, Transform, Load) CLI tool designed to stream bulk metadata from an OpenSearch index and transform it into flattened YAML files in DataCite schema. The OpenSearch metadata extractor is created to support the FAIRification datastore tool for EXA4MIND. This tool helps in adding support of Elastic based (OpenSearch) indexes to be used as the source of metadata extraction.

This exporter tool was designed to add support of Elasticsearch based indexes to our datastore module.

Features

• Granular Splitting: Intelligently parses nested datasets arrays and transforms datacite objects into independent YAML files.
• Flattened Attributes: Removes redundant top-level keys (e.g., datacite) for a clean, 1:1 attribute mapping in the resulting YAML.
• Structured directories: Creates a folder hierarchy that can be crawled by the datastore for seamless publishing of metadata to the FAIR data portal (B2SHARE, InvenioRDM etc.).

---

Output Directory Structure

The tool converts large JSON blobs from OpenSearch into the following organized, file-system hierarchy:

output_dir/
├── metadata.yaml                # root metadata file (To be used for fixed attributes, can be empty)
├── dataset_1/
│   └── metadata_datacite.yaml   # flattened datacite attributes
├── dataset_2/
│   └── metadata_datacite.yaml
└── ...

How to Use It

Follow these steps to extract your OpenSearch indexed metadata into structured YAML chunks.

1: Initial Setup

Ensure you have uv installed on your machine. Then, initialize the project:

# Creates the virtual environment and installs dependencies
make install

2. Usage

Once the virtual environment is set up using the above command, run the following command to use the venv:

source .venv/bin/activate

Finally, the tool can now directly be run using the os_exporter command.

os-exporter --host localhost --port 9200 --index your_index_name --user os_user --password os_pass --out ./my_metadata

Arguments

Argument	Description	Default Value	Required?
--host	OpenSearch Host	-	Y
--port	penSearch port	9200	N
--index	Index name	-	Y
--user	OpenSearch User	"admin"	N
--password	OpenSearch password	"admin"	N
--out	Output directory path	"./metadata_os_export"	N