Dataspace Connectors and Adaptors

This submodule aims at providing connectivity to international and, in particular, European data ecosystems. In the first release, we focus exclusively on connection to Data Spaces according to the Dataspace Protocol by IDSA and Eclipse Foundation.

EDC-based Dataspace Connector for Publishing Data from S3 Backends

We have built a demonstrator setup which showcases file transfer between a provider and a consumer using the Eclipse Dataspace Connector (EDC). The transfer uses Amazon S3-compatible storage backends (e.g., MinIO).

This illustrates how data from an S3 deployment within EXA4MIND Platform can be shared with a data space. The functionality can accordingly be added to any EXA4MIND/EDD Platform instance. The current setup is a starting point for integrating additional policies, authentication, or catalog services.

What is a Dataspace Connector

The Dataspace Connector is a secure gateway for sharing data and services within a dataspace. It allows data owners to define and enforce usage policies, ensuring that data is accessed only under agreed conditions. The connector also records data provenance, making it possible to track where data originated and how it has been used. This provides both control and transparency while enabling seamless data exchange between participants.

Scope of Demonstrator Setup

In this demonstrator, two connectors are deployed—one acting as the provider and one as the consumer. Data is transferred between two S3-compatible storage systems, demonstrating how a dataspace can expose datasets to external participants via a shared data catalog.

The files and the setup instructions are provided with our stack of software presented as part of the EXA4MIND project and is available on our GitLab platform to test out the functionality.

Disclaimer

The demonstrator setup per se is intended for testing and further development towards production scenarios as required. It is derived from the official Eclipse EDC sample.

Prerequisites

  • Docker & Docker Compose
  • Java 17+
  • cURL
  • MinIO (with test buckets)
  • jq

Setup Instructions

General Information

The configuration for this demo is controlled by config.toml. It contains most of the information required by the system to be able to function.

To generate the files required to the test the flow, run python3 generate_from_templates.py.

1. MinIO Setup

For this demo, two instances of S3 backends are encouraged. We chose MinIO, but an AWS S3 backend should also work. The two instances will be linked to the consumer and provider each. The transfer of data will be from provider bucket to the consumer bucket.

For one or both MinIO instances, we need to provide the corresponding bucket access keys to a vault.properties` file. (For both consumer and provider in their respective instances).

Development MinIO

For quick test instance of MinIO, a docker compose file is provided, which starts a MinIO system.

⚠️ It is a test docker container with default credentials and no extra security.

Both "backends" are served by one MinIO instance. After starting the MinIO instance, create

  • an access key
  • and 2 buckets (for the consumer and the provider) in the GUI

Then upload a file to the provider bucket that can be used. A region can also be set, but does not need to be.

2. Vault Configuration

This setup uses a simple filesystem vault for credentials, which can be set up using simple keystore.jks file using the setup instructions.

3. Container Setup

The containers are configured from the config.toml used earlier, which creates a docker compose file. It is possible to run both containers on one machine with the following:

docker compose up -d

To run the consumer and provider containers on separate instances, run the following on the provider VM:

docker compose up -d provider

and

docker compose up -d consumer

on the consumer VM.

Once the images are built and running, we can test the flow of transfer files between the provider and consumer using simple curl commands. For detailed information on this, please refer to our README file in the module repository.