Troubleshooting¶

This page covers problems affecting the system as a whole. Problems specific to one component are covered on that component's installation page.

How to approach a failure¶

Work from the bottom of the stack upwards. Most failures in the Metadata or ETL components are caused by the database being unavailable rather than by a fault in the component reporting the error.

1. Confirm the database is running and accepting connections.
2. Confirm the component can reach the database from inside its container.
3. Inspect the logs of the component reporting the problem.
4. Confirm the configuration values the component was started with.

To inspect the configuration a running container actually received:

docker inspect <container name> --format '{{range .Config.Env}}{{println .}}{{end}}'

This shows the values in effect, which may differ from the configuration file if the container was not recreated after a change.

A container exits immediately after starting¶

Inspect the logs, including those from the failed run:

docker logs <container name>

Common causes, in order of likelihood:

• A required configuration value is missing or empty
• The database is not reachable, so a startup check fails
• A port is already in use on the host
• A mounted path does not exist on the host

A port is already in use¶

Docker reports that it cannot bind the port. Identify the process holding it.

On Linux:

sudo lsof -i:<port>

On Windows:

netstat -ano | findstr :<port>

Either stop that process, or change the published port in the compose file. Where a port is changed, every component configured to reach that service must be updated to match.

A component cannot reach the database¶

Test from inside the container rather than from the host, as the container has its own network view:

docker exec <container name> getent hosts <database host>

Where the name does not resolve, the address or the network membership is wrong.

Note that host.docker.internal resolves on Windows and macOS but not on Linux by default. On Linux, either use the host address or attach both containers to the same Docker network and address the database by its service name.

Confirm the containers share a network:

docker network inspect cvdlink-net

Connections are refused from another machine¶

Confirm the container publishes the port:

docker port <container name>

A container port that is exposed but not published is reachable only from other containers. Where the port is published and connections are still refused, the host firewall is responsible.

Authentication fails in a central deployment¶

Confirm that the Keycloak realm is reachable from the container, that the client identifier matches a client configured in the realm, and that the realm permits the redirect URI of this installation.

Confirm the system clock is accurate. Token validation is time sensitive, and a clock differing by more than a short interval causes valid tokens to be rejected:

timedatectl status

A user cannot access an administrative page¶

In a central deployment, access depends on roles issued by Keycloak. Confirm the account holds the required role in the realm; a role added after the user signed in takes effect only after a new sign-in.

In a standalone deployment, role checks are not applied and every authenticated user has access to all pages. Where a user cannot reach a page in such an installation, the cause is not the role configuration.

Federation shows no peer sites¶

Confirm that the message broker is reachable from the Metadata container, that the P2P node is running, and that the node identifier of this site differs from that of every other site.

Confirm that the broker address in the configuration matches the current data host, as recorded in Hosts and ports.

Uploaded files are not visible to the application¶

Confirm the data directory is mounted and writable from inside the container:

docker exec <container name> ls -la /home/simavi/data

Where the directory is owned by another user, correct its ownership on the host. Paths given to the application must be those inside the container, not the host paths.

The system is slow¶

Check resource usage:

docker stats --no-stream

A container consistently at its memory limit requires either a higher limit or more memory on the host. Where the database is the constraint, confirm the host has sufficient free memory and that the data directory is not on a volume that has run out of space:

df -h

Disk space has run out¶

Identify what is consuming space:

docker system df

Unused images and stopped containers can be removed safely:

docker image prune
docker container prune

Do not use docker system prune --volumes, which removes volumes and may destroy database contents. The database data directory is a host path and is not affected by these commands, but other components may hold state in volumes.

Collecting information for a support request¶

Where a problem cannot be resolved locally, collect the following before requesting assistance.

docker --version
docker compose version
docker ps -a
docker compose logs --tail 200 > logs.txt

Include the deployment profile, the component affected, the verification step that fails, and the configuration in use with passwords and secret keys removed.