Skip to main content

Accessing Application Log Data

Overview

This document is intended to assist engineers in accessing application and system logs stored in a centralized Elasticsearch cluster.

Accessing Kibana

The Cloud Platform collects, indexes and presents your application and system log data enabling you to query using Kibana’s standard query language (based on Lucene query syntax).

To access Kibana, follow the link below and authenticate with your GitHub credentials:

https://kibana.cloud-platform.service.justice.gov.uk/_plugin/kibana

Sensitive data

Any member of the ministryofjustice GitHub organization can access Kibana, which will provide access to logs for all teams. Information written to log files can be of a sensitive nature, so keep track of your sensitive data, and it is important to keep it out of the logs.

Examples of sensitive data which should not be logged include:

  • Personally identifying information (PII) such as names, email/postal addresses etc.
  • User-generated data (users will put PII and other sensitive information into text boxes, regardless of how many times you tell them not to)
  • Financial Data (like credit card numbers)
  • Passwords, API keys or any other secrets

Using Kibana

As a quick example, we will filter down to the logs of a particular environment.

1) On the Kibana dashboard, select the ‘Discover’ tab.

2) Ensure the index selected is kubernetes_cluster* (which is the default).

3) Select ‘Add a filter +’

4) Filter kubernetes.namespace_name, with operator is and the value equal to your environment name.

The log entries will contain any data that your pods wrote to STDOUT/STDERR.

Accessing Ingress logs

A special case is the Nginx Ingress which serves as reverse proxy for all the other applications in the cluster. Its logs originate in the ingress-controllers namespace and are indexed separately. To search:

1) On the same Kibana ‘Discover’ tab, change the index pattern to ‘index-json*’ in the drop-down on the left

2) Filter using the prefix nginx-ingress, the full list of indexed fields is defined in terraform

Accessing namespace events

Messages generated by events which are not part of the applications themselves (e.g. deployment misconfiguration, pods failing on startup due to lack of resources, actions forbidden by security restrictions) are trapped by a shared Eventrouter which annotates and stores them in the logging namespace. To view such events:

1) On the same Kibana ‘Discover’ tab, use the index pattern ‘logstash*’

2) Search for kubernetes.namespace_name:"logging" AND kubernetes.labels.app:"eventrouter"

3) Filter by additional strings in the log field

More in-depth guides on using Kibana can be found below:

https://www.elastic.co/guide/en/kibana/6.3/search.html

https://www.elastic.co/guide/en/beats/packetbeat/current/kibana-queries-filters.html

This page was last reviewed on 6 April 2021. It needs to be reviewed again on 6 July 2021 .
This page was set to be reviewed before 6 July 2021. This might mean the content is out of date.