Accessing Application Log Data
Overview
This document is intended to assist engineers in accessing application and system logs stored in a centralized Elasticsearch cluster.
Accessing Kibana
The Cloud Platform collects, indexes and presents your application and system log data enabling you to query using Kibana’s standard query language (based on Lucene query syntax).
To access Kibana, follow the link below and authenticate with your GitHub credentials:
https://kibana.cloud-platform.service.justice.gov.uk/_plugin/kibana
Sensitive data
Any member of the ministryofjustice GitHub organization can access Kibana, which will provide access to logs for all teams. Information written to log files can be of a sensitive nature, so keep track of your sensitive data, and it is important to keep it out of the logs.
Examples of sensitive data which should not be logged include:
- Personally identifying information (PII) such as names, email/postal addresses etc.
- User-generated data (users will put PII and other sensitive information into text boxes, regardless of how many times you tell them not to)
- Financial Data (like credit card numbers)
- Passwords, API keys or any other secrets
Using Kibana
As a quick example, we will filter down to the logs of a particular environment.
1) On the Kibana dashboard, select the ‘Discover’ tab.
2) Ensure the index selected is live_kubernetes_cluster*.
3) Select ‘Add a filter +’
4) Filter kubernetes.namespace_name, with operator is and the value equal to your environment name.
The log entries will contain any data that your pods wrote to STDOUT/STDERR.
Accessing Ingress logs
A special case is the Nginx Ingress which serves as reverse proxy for all the other applications in the cluster. Its logs originate in the ingress-controllers namespace and are indexed separately. To search:
1) On the same Kibana ‘Discover’ tab, change the index pattern to live_kubernetes_ingress* in the drop-down on the left
2) Filter using the prefix nginx-ingress, the full list of indexed fields is defined in terraform
Accessing namespace events
Messages generated by events which are not part of the applications themselves (e.g. deployment misconfiguration, pods failing on startup due to lack of resources, actions forbidden by security restrictions) are trapped by a shared Eventrouter which annotates and stores them in the logging namespace. To view such events:
1) On the same Kibana ‘Discover’ tab, use the index pattern live_eventrouter*
2) Search for kubernetes.namespace_name:"logging" AND kubernetes.labels.app:"eventrouter"
3) Filter by additional strings in the log field
More in-depth guides on using Kibana can be found below:
https://www.elastic.co/guide/en/kibana/6.3/search.html
https://www.elastic.co/guide/en/beats/packetbeat/current/kibana-queries-filters.html