ETL installation¶
The ETL component reads clinical and imaging files from a configured directory, extracts structured metadata, and writes it to the database. It is reached over an HTTP API and may also be run as a command-line tool for bulk loading.
Responsible team: Simavi.
Supported formats¶
| Category | Formats |
|---|---|
| Imaging | DICOM |
| Clinical records | FHIR resources, HL7 messages |
| Genomic | VCF variant files, tabular genomic data |
| Signals | HDF5, EDF, WFDB waveform records |
| Images | PNG |
| Tabular | CSV, including PTB-XL and OMOP person data |
Where a mapping is defined, the service also transforms data into the OMOP Common Data Model.
Before you begin¶
- The Database component installed and verified, or the shared database confirmed reachable
- Database credentials for the environment
- Port 8002 free
- The data directory containing the files to be loaded
1. Prepare the configuration¶
The service is configured by environment variables. Create a .env file in the
installation directory with the values for your deployment profile.
For a central deployment, substituting the data host address from Hosts and ports:
DB_NAME=cvdlink
DB_USER=postgres
DB_PASSWORD=<database password>
DB_HOST=<data host address>
DB_PORT=5432
UVICORN_HOST=0.0.0.0
UVICORN_PORT=8002
RUN_CLI=0
OFFLINE_MODE=false
KEYCLOAK_BASE_URL=https://cvdlink.simavi.ro/auth
KEYCLOAK_REALM=cvdlink
REQUIRED_ROLE=api-call
For a standalone deployment:
DB_NAME=cvdlink
DB_USER=postgres
DB_PASSWORD=<database password>
DB_HOST=host.docker.internal
DB_PORT=5444
UVICORN_HOST=0.0.0.0
UVICORN_PORT=8002
RUN_CLI=0
OFFLINE_MODE=true
Restrict access to the file, which contains the database password:
chmod 600 .env
Configuration reference¶
| Variable | Purpose | Default |
|---|---|---|
DB_HOST |
Database host | host.docker.internal |
DB_PORT |
Database port | See note below |
DB_NAME |
Database name | cvdlink |
DB_USER |
Database user | postgres |
DB_PASSWORD |
Database password | None, must be set |
UVICORN_PORT |
Port the service listens on | 8002 |
RUN_CLI |
Run the bulk loader at container start | 1 |
OFFLINE_MODE |
Disable token authentication | false |
KEYCLOAK_BASE_URL |
Keycloak base URL | https://cvdlink.simavi.ro/auth |
KEYCLOAK_REALM |
Keycloak realm | cvdlink |
REQUIRED_ROLE |
Role required to call the API | api-call |
CVDLINKBK_CORS |
Permitted origins, comma separated | * |
The image carries a default DB_PORT of 5492, which is not used by any current
deployment. Always set DB_PORT explicitly.
Set RUN_CLI=0 for a service installation. With RUN_CLI=1 the container runs the
bulk loader once at start before the API becomes available, which is appropriate only
for a deliberate bulk load.
2. Build the image¶
docker compose -f docker-compose-onlyserver8002.yml build
Note that the file docker-compose.yml in the same directory configures the
command-line variant and publishes no ports. Use
docker-compose-onlyserver8002.yml for a service installation.
For installations without build capability, load a supplied image archive instead:
docker load -i cvdlink-etl.tar
3. Start the service¶
docker compose -f docker-compose-onlyserver8002.yml up -d
4. Verify the installation¶
Confirm the container is running¶
docker ps
The output must include a container named cvdlink-app with a status beginning Up.
Confirm the service responds¶
curl http://localhost:8002/cvdlinkbk/health
A successful response confirms the service has started and is accepting requests. This endpoint requires no authentication.
Confirm the interactive documentation is available¶
Open the following address in a browser:
http://<application host>:8002/docs
The page lists the available operations and their parameters.
Confirm the database connection¶
docker logs cvdlink-app | grep -iE "error|refused|could not connect"
No output is expected. A connection error indicates that DB_HOST, DB_PORT, or the
credentials are incorrect, or that the database is not reachable from the container.
API¶
The service exposes two endpoints.
| Method | Path | Authentication | Purpose |
|---|---|---|---|
| GET | /cvdlinkbk/health |
None | Service health check |
| POST | /cvdlinkbk/api |
Bearer token | Run an extraction operation |
In a central deployment, requests to /cvdlinkbk/api require a bearer token issued by
Keycloak, and the token must carry the role named in REQUIRED_ROLE. In a standalone
deployment with OFFLINE_MODE=true, token validation is not applied.
The operation to run is selected by a parameter in the request body. The available
operations are listed in the interactive documentation at /docs.
Bulk loading from the command line¶
For loading a large volume of existing files, the command-line variant is more appropriate than repeated API calls. Run it interactively:
docker compose run --rm cli
The loader prompts for the operation to perform. It uses the same database configuration as the service.
Routine operations¶
| Operation | Command |
|---|---|
| Restart | docker compose -f docker-compose-onlyserver8002.yml restart |
| Stop | docker compose -f docker-compose-onlyserver8002.yml down |
| View logs | docker logs -f cvdlink-app |
| Rebuild after a code change | docker compose -f docker-compose-onlyserver8002.yml up -d --build |
Troubleshooting¶
The container starts and then exits¶
Inspect the logs:
docker logs cvdlink-app
The most common cause is RUN_CLI=1, which runs the bulk loader at start; if the loader
fails, the container exits before the API starts. Set RUN_CLI=0 for a service
installation.
The database cannot be reached from the container¶
host.docker.internal resolves to the host machine on Windows and macOS. On Linux it
is not available by default; use the host address or attach the container to the same
Docker network as the database and use the service name postgres.
Requests to the API are rejected as unauthorised¶
In a central deployment, confirm that the bearer token has not expired and that it
carries the role named in REQUIRED_ROLE. Confirm that KEYCLOAK_REALM and
KEYCLOAK_BASE_URL match the realm that issued the token.
Files are not found by the loader¶
The data directory must be mounted into the container and the paths given to the loader must be those inside the container, not on the host. Confirm the mapping:
docker inspect cvdlink-app --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}'
Next steps¶
Continue to Metadata installation.