Metadata Uploader (datastore)

The datastore is an Extract-Transform-Load (ETL) pipeline built for publishing datasets to EUDAT-B2SHARE or custom InvenioRDM based portals. It streamlines the process of extract and transforming metadata into the supported formats of the target system and uploads teh data in a FAIR compliant manner. It is part of our larger FAIRification pipeline which aims to streamline the data publishing workflow.

NOTE: The uploader tool is called datastore as it pushes metadata from a data store to a publication platform.

Overview

It pushes metadata from YAML files into these repository frameworks, where the user is supposed to provide a structure with:

one global metadata file (containing entries for all datasets to be published)
local metadata files in subdirectories (one for each dataset=subdirectory to be published)

An example structure for publishing is provided in tests/testdata directory.

Installation

Clone the repository:

git clone <REPO-URL>
cd b2share-datastore

Installation via uv

NOTE: It's recommended to use uv as it manages setting up of a virtual environment, installation of the correct Python interpreter and handling of dependencies all automatically.

uv tool install .

NOTE: You can find installation instructions for uv itself in their official documentation.

Installation with pip

If you prefer, you can of course also just use pip for installation. Depending on your operating system, you are required to create a virtual environment. This can be done with

python -m venv <ENVIRONMENT-NAME>

where you can choose the environment's name yourself. Depending on your Python installation it is necessary to change python to python3 in the above command. You can activate the environment with

source <ENVIRONMANT-NAME>/bin/activate

when you are using bash/zsh. Or via

# cmd.exe
.\<ENVIRONMANT-NAME>\Scripts\activate.bat
# PowerShell
.\<ENVIRONMANT-NAME>\Scripts\Activate.ps1

on windows. You can find the usage with other shells and more information about virtual environments in the official documentation.

You can then install datastore by executing

pip install .

in the project root, i.e., the directory where the pyproject.toml file is located. Depending on your Python installation it is necessary to change pip to pip3 in the above command.

NOTE: When you installed inside a virtual environment, the datastore command will only be available when this virtual environment is active.

Usage

After installation you have access to the datastore command. For publishing to e.g., a B2Share test instance at https://trng-b2share.eudat.eu/, you can simply run:

datastore \
  --target b2sharev3
  --publish-host "https://trng-b2share.eudat.eu" \
  --publish-auth <AUTH-TOKEN> \
  ./lrz_metadata_global.yml

assuming your metadata files are names exa4mind_metadata.yml or metadata.yml and your global/shared metadata file is exa4mind_metadata_global.yml, located in the current working directory. To change the pattern for metadata files to search for, you can specify it with the -m/--metadata-pattern option. In case your target instance is InvenioRDM use invenio as argument for -t/--target. Otherwise use either b2sharev2 or b2sharev3, respectively.

By default, the B2Share datastore crawls from the directory where the global metadata file is located with a depth of 4. This can be changed with the -c/--crawl-root and -M/--max-depth options, respectively.

It's also possible to simply print the converted B2Share metadata with the -d/--dry-run option, create converted files with the --convert option or just validate your metadata with the --validate option. Below, you can see an overview of all options with descriptions and their corresponding environment variables which can used for configuration instead.

Option (short, long)	Argument	Description	Default Value	Corresponding Environment Variable
-c , --crawl-root	CRAWL_ROOT (path)	Directory where crawling for metadata files start	Directory of main_metadata_file	DATASTORE_SOURCE_FS_PATH
-m , --metadata-pattern	METADATA_PATTERN (string \| regex-pattern)	Pattern to identify metadata files	.(exa4mind)?metadata.ya?ml	DATASTORE_METADATA_FILE_PATTERN
-M , --max-depth	MAX_DEPTH (integer number)	Maximum crawling depth from CRAWL_ROOT	4	DATASTORE_SOURCE_FS_CRAWL_DEPTH
-p , --publish-host	PUBLISH_HOST (string)	Name of the host to publish to	None	DATASTORE_SITE_HOSTNAME
-a , --publish-auth	PUBLISH_AUTH (string)	Authentication token ot the publish host	None	DATASTORE_AUTH_TOKEN
-d , --dry-run	-- (boolean flag)	Run conversion without publishing and print metadata in B2Share format	False	-
-t, --target	DATA-REPOSITORY (invenio \|b2sharev2 \| b2sharev3)	Target data respository to publish to.	invenio	-
--validate	-- (boolean flag)	Run validation only	False	-
--convert	-- (boolean flag)	Run conversion only	False	-
-v , --verbose	-- (count flag)	Increase verbosity (up to -vvv)	0	DATASTORE_LOGGING_LEVEL
-h , --help	-- (boolean flag)	Show help message and exit	False	-

Metadata schema

For our metadata schema, we follow the one from DataCite, though in YAML format. This allows us to nicely separate the DataCite schema (keys and values that describe data) from additional metadata that is needed for the publication workflow via the usage of YAML document delimiters (---). A resulting metadata file would look like the following (note the usage of --- and ...).

---
# Metadata about the dataset strictly following DataCite schema, e.g.
identifiers:
  - identifier: '12345'
    identifierType: Local accession number
creators:
  - name: ExampleFamilyName, ExampleGivenName
    nameType: Personal
    givenName: ExampleGivenName
    familyName: ExampleFamilyName
    nameIdentifiers:
      - nameIdentifier: https://orcid.org/0000-0001-5727-2427
        nameIdentifierScheme: ORCID
        schemeUri: https://orcid.org/
    affiliation:
      - name: ExampleAffiliation
        affiliationIdentifier: https://ror.org/04wxnsj81
        affiliationIdentifierScheme: ROR
        schemeUri: https://ror.org/
# etc.
---
# Metadata needed for publication workflow to B2SHARE (not needed for InvenioRDM)
community:           # Community ID, e.g. e9b9792e-79fb-4b07-b6b4-b9c2bd06d095
community_extension: # Any community specific key-value pairs
open_access:         # `true` or `false`
schemaVersion: "b2share"
...

Dependencies and Licenses

Dependency	License
iso639-lang	MIT License (MIT)
jsonschema	MIT License (MIT)
pydantic	MIT License (MIT)
python-dateutil	Apache Software License, BSD License (Dual License)
requests	Apache Software License (Apache-2.0)
ruamel-yaml	MIT License (MIT)