Troubleshooting

Below are some troubleshooting operations performed during development/testing of the platform that may be useful for the future.

Checking Logs

  • To check the logs, locate the Syslog server
  • This can be done by determining the address used by any of the containers for Syslog logging (check the environment they are run with)
  • Once located, determine the volume that is mounted to /var/log/gavip within the container
  • In ESAC early deployment, this is /home/gavipadmin/gavip/gavip_logs
  • The logs are within this directory. Alternatively, running ‘docker logs -f <container name>’ also works in most cases

Images not appearing

  • If AVI builds (aka AVI Docker images) do not appear, check that the build was succesful.
  • If so, ensure that the AVI image was pushed to the designated docker registry (this is a central location for AVI servers to pull the images from)
  • If it was, attempt to pull it using the docker client and docker credentials as required
  • Usually, the image either wasn’t pushed, or the docker client is not logged, or the docker client cannot succesfully pull the image due to disk issues.
  • If the disk is out of space, increase the volume of the disk, or remove old images (this step can be automated each night via Cron, when an AVI is deployed, the server will pull the image anyway - it just adds time).

Requests failing

HTTP requests to the platform can fail if deployment was performed incorrectly, check the following

  • Ensure that the containers are running (‘docker ps’ or ‘docker service ls’ if using swarm)
  • If not, check the log from the container as it ended, or check the reason for the service not being deployed.
  • If it is running, but not responding, check the Apache and Gavip logs
  • Sometimes this is easier within the portal_web container itself.

Sometimes requests will fail to an AVI

  • Login to the AVI host to ensure the container is running (docker ps)
  • Or login to the Portal host and ensure the service availability is ‘1/1’ (docker service ls)
  • If not, check the service status using (docker service ps <service name>) or docker logs
  • The portal will also record these errors in deployment
  • If the container is running, but requests are failing, check that the AVI name resolves within the portal_web container (it proxies requests on the docker network to the AVI)
  • Enter the container using (docker exec -it <portal_web ID> bash), then run ‘wget <AVI container name>’ you should see that the internal container IP (10.*.*.*) is used
  • If not, restart the docker service on each server to attempt to reset the internal network.