Accessing Application Log Data
This document is intended to assist engineers in accessing application and system logs stored in a centralized Elasticsearch cluster.
To access Kibana, follow the link below and authenticate with your GitHub credentials:
Any member of the ministryofjustice GitHub organization can access Kibana, which will provide access to logs for all teams. Information written to log files can be of a sensitive nature, so keep track of your sensitive data, and it is important to keep it out of the logs.
Examples of sensitive data which should not be logged include:
- Personally identifying information (PII) such as names, email/postal addresses etc.
- User-generated data (users will put PII and other sensitive information into text boxes, regardless of how many times you tell them not to)
- Financial Data (like credit card numbers)
- Passwords, API keys or any other secrets
As a quick example, we will filter down to the logs of a particular environment.
1) On the Kibana dashboard, select the ‘Discover’ tab.
2) Ensure the index selected is
kubernetes_cluster* (which is the default).
3) Select ‘Add a filter +’
kubernetes.namespace_name, with operator
is and the value equal to your environment name.
The log entries will contain any data that your pods wrote to STDOUT/STDERR.
Accessing Ingress logs
A special case is the Nginx Ingress which serves as reverse proxy for all the other applications in the cluster. Its logs originate in the
ingress-controllers namespace and are indexed separately. To search:
1) On the same Kibana ‘Discover’ tab, change the index pattern to ‘index-json*’ in the drop-down on the left
2) Filter using the prefix
nginx-ingress, the full list of indexed fields is defined in terraform
Accessing namespace events
Messages generated by events which are not part of the applications themselves (e.g. deployment misconfiguration, pods failing on startup due to lack of resources, actions forbidden by security restrictions) are trapped by a shared
Eventrouter which annotates and stores them in the
logging namespace. To view such events:
1) On the same Kibana ‘Discover’ tab, use the index pattern ‘logstash*’
2) Search for
kubernetes.namespace_name:"logging" AND kubernetes.labels.app:"eventrouter"
3) Filter by additional strings in the
More in-depth guides on using Kibana can be found below: