Accessing Application Log Data in OpenSearch
Overview
This document is intended to assist engineers in accessing application and system logs stored in a centralized OpenSearch cluster.
Accessing OpenSearch Dashboards
The Cloud Platform collects, indexes, and presents your application and system log data, enabling you to query using OpenSearch Dashboards’ standard query language (based on Lucene query syntax).
To access OpenSearch Dashboards, follow the link below and authenticate with your GitHub credentials:
https://app-logs.cloud-platform.service.justice.gov.uk/_dashboards/app/home#/
Security Feature: Document-Level Security
We have implemented Document-Level Security in OpenSearch. This is a security improvement which will mean that teams are constrained to viewing only logs which they have permission to view.
Permissions are based on GitHub team membership and correlated namespace access.
If you have access to a namespace you have access to that namespace’s logs and all cloud platform namespaces in OpenSearch but no other user namespace logs.
If you cannot view certain logs, please ensure that you are in the correct GitHub team or contact the corresponding GitHub team administrators to add you.
Sensitive data
Even with Document-Level Security in place, it’s important to prevent sensitive information from being logged. Information in log files may still be accessible to individuals who have access to the relevant namespaces. Therefore, ensure that your applications are not logging any sensitive data.
Examples of sensitive data which should not be logged include:
- Personally Identifiable Information (PII): Names, email addresses, postal addresses, phone numbers, etc.
- User-Generated Data: Users may input PII and other sensitive information into text fields, regardless of advisories.
- Financial Data: Credit card numbers, bank account details, or any other financial information.
- Authentication Secrets: Passwords, API keys, tokens, or any other secrets.
Using OpenSearch Dashboards
As a quick example, we will filter down to the logs of a particular environment.
1) On the OpenSearch Dashboards homepage, select the Disvocer tab.
2) Ensure the index pattern selected is live_kubernetes_cluster-*.
3) Click on + Add filter.
4) Filter kubernetes.namespace_name, with operator is, and set the value equal to your namespace.
The log entries will contain any data that your pods wrote to STDOUT/STDERR.
Accessing Ingress Logs
A special case is the Nginx Ingress, which serves as a reverse proxy for all the other applications in the cluster. Its logs originate in the ingress-controllers namespace and are indexed separately.
To search:
1) On the same OpenSearch Discover tab, change the index pattern to live_kubernetes_ingress-* in the drop-down on the left
2) Click on + Add filter.
3) Filter log_processed.kubernetes_namespace, with operator is, and set the value equal to your namespace.
Accessing namespace events
Messages generated by events which are not part of the applications themselves (e.g. deployment misconfiguration, pods failing on startup due to lack of resources, actions forbidden by security restrictions) are trapped by a shared Eventrouter which annotates and stores them in the logging namespace.
To view such events:
1) On the same OpenSearch Discover tab, use the index pattern live_eventrouter*
2) Click on + Add filter.
3) Filter metadata.namespace, with operator is, and set the value equal to your namespace.
For more in-depth guides on using OpenSearch Dashboards, refer to the links below: