Accessing Application Log Data in OpenSearch
Overview
This document is intended to assist engineers in accessing application and system logs stored in a centralized OpenSearch cluster.
Accessing OpenSearch Dashboards
The Cloud Platform collects, indexes, and presents your application and system log data, enabling you to query using OpenSearch Dashboards’ standard query language (based on Lucene query syntax).
To access OpenSearch Dashboards, follow the link below and authenticate with your GitHub credentials:
https://app-logs.cloud-platform.service.justice.gov.uk/_dashboards/app/home#/
Important Notes on Log Availability in OpenSearch
Our OpenSearch Index State Management (ISM) policy is configured as follows:
Hot to Warm: after 1 day
Warm to Cold: after 30 days
Cold to Delete: after 366 days
What this means for users
Logs stored in hot and warm indices are fully visible in the OpenSearch console. As a result, logs from the past 30 days are available by default.
However, cold indices are not searchable in the console - this is expected behavior. Logs are stored in cold storage for long-term retention and cost efficiency.
To access logs older than 30 days, we need to manually migrate the relevant cold indices back to the warm tier, which makes them searchable again.
You may currently see logs going back to January 2025 and this is due to a known indexing issue. We are working to resolve this index issue.
If you would like to retrieve logs for a specific date, please contact us on the #ask-cloud-platform Slack channel.
Notes on Cold → Warm Migration
This process:
- Adds load to the OpenSearch cluster
- Increases hot instance resource usage
- May impact live OpenSearch cluster performance for other users
To reduce these risks, we recommend requesting migration of specific dates only, rather than large time ranges.
Currently logs are available as far back as November 2024.
Migrating a cold index affects all namespaces within that index — not just one.
Security Feature: Document-Level Security
We have implemented Document-Level Security in OpenSearch. This is a security improvement which will mean that teams are constrained to viewing only logs which they have permission to view.
Permissions are based on GitHub team membership and correlated namespace access.
If you have access to a namespace you have access to that namespace’s logs and all cloud platform namespaces in OpenSearch but no other user namespace logs.
If you cannot view certain logs, please ensure that you are in the correct GitHub team or contact the corresponding GitHub team administrators to add you.
Sensitive data
Even with Document-Level Security in place, it’s important to prevent sensitive information from being logged. Information in log files may still be accessible to individuals who have access to the relevant namespaces. Therefore, ensure that your applications are not logging any sensitive data.
Examples of sensitive data which should not be logged include:
- Personally Identifiable Information (PII): Names, email addresses, postal addresses, phone numbers, etc.
- User-Generated Data: Users may input PII and other sensitive information into text fields, regardless of advisories.
- Financial Data: Credit card numbers, bank account details, or any other financial information.
- Authentication Secrets: Passwords, API keys, tokens, or any other secrets.
Using OpenSearch Dashboards
As a quick example, we will filter down to the logs of a particular environment.
1) On the OpenSearch Dashboards homepage, select the Discover tab.
2) Ensure the index pattern selected is live_kubernetes_cluster-*.
3) Click on + Add filter.
4) Filter kubernetes.namespace_name, with operator is, and set the value equal to your namespace.
The log entries will contain any data that your pods wrote to STDOUT/STDERR.
Accessing Ingress Logs
A special case is the Nginx Ingress, which serves as a reverse proxy for all the other applications in the cluster. Its logs originate in the ingress-controllers namespace and are indexed separately.
To search:
1) On the same OpenSearch Discover tab, change the index pattern to live_kubernetes_ingress-* in the drop-down on the left
2) Click on + Add filter.
3) Filter log_processed.kubernetes_namespace, with operator is, and set the value equal to your namespace.
Accessing namespace events
Messages generated by events which are not part of the applications themselves (e.g. deployment misconfiguration, pods failing on startup due to lack of resources, actions forbidden by security restrictions) are trapped by a shared Eventrouter which annotates and stores them in the logging namespace.
To view such events:
1) On the same OpenSearch Discover tab, use the index pattern live_eventrouter*
2) Click on + Add filter.
3) Filter metadata.namespace, with operator is, and set the value equal to your namespace.
For more in-depth guides on using OpenSearch Dashboards, refer to the links below: