Observability in Game Warden

Guide to using Grafana, Loki, Mimir, and Alloy to track application logs, metrics, and alerts in Game Warden.

Game Warden's observability stack provides real-time visibility into application behavior, performance, and logs. It leverages the following tools:

Grafana is a data visualization platform that allows you to query, analyze, and visualize your application logs and metrics. With Grafana, you can build interactive dashboards and panels for operational insights into your deployed services.

Grafana also supports log-based alerting, enabling you to set up alerts based on specific log patterns or errors detected in your application.

Note
This guide is intended for application engineers who are already familiar with technical concepts and need to explore customer-specific data within Grafana. It does not cover basic Grafana functionality.

Access Grafana dashboard

Note
Access to Grafana are not granted automatically and must be explicitly assigned. If you were invited by your team, you will receive baseline developer and logging access automatically. If you signed up without a team invitation, or find you are missing Grafana access, contact a 2F administrator to have the appropriate permissions granted to your account.

Second Front team manages the Grafana endpoints you use. Most production endpoints follow the same convention as your AFWERX endpoint:

grafana-<company-id>.il2.afwerx.dso.mil

Note that DEV, STG, and PRD environments each have their own unique URLs. If you need to identify your specific environment endpoints, contact your Mission Success Manager.

You can access Grafana using your Platform One (P1) SSO credentials and Government Access Card.

  1. Go to your Game Warden Grafana endpoint and click Sign in with Game Warden SSO.
  2. On the Log in or sign up page, click Continue With P1 SSO.
  3. Enter your P1 credentials and Multi-Factor Authentication (MFA) code.

Common tasks in Grafana

Review the tabs below for common tasks you'll perform in Grafana.

The Grafana home dashboard includes a dedicated Alerts panel for you to track which alerts are actively firing within your application namespace.

Landing Page

Available features:

The stack includes over 200 preloaded Prometheus alerts and 200+ recording rules for tracking both application and infrastructure-level behavior. These alerts are also listed on the Alerting page in Grafana.

Tip
For detailed alert descriptions, see the Prometheus runbooks.

Alert behavior:

By default, Prometheus configures alerts to send notifications to Slack. Certain alerts—such as those related to Kubernetes control plane components—are automatically silenced in cloud environments, as they are only relevant for bare-metal deployments. You'll receive alerts only for your application namespaces. Meanwhile, the Second Front team tracks and manages infrastructure and platform-level alerts.

Manage custom alerts:

  • You cannot edit default Prometheus/Mimir alerts directly (for example, silencing or adjusting intervals).
  • You can create custom Grafana alerts using Grafana's native alert engine:
    • These alerts support querying both logs and metrics.
    • Second Front performs a one-time configuration to route custom alerts to Slack.

Troubleshooting: Example scenarios

Below are two common scenarios that highlight the importance of checking beyond container logs and incorporating Istio traffic and authentication insights in your troubleshooting process.

Scenario 1: Missing traffic or 'No Healthy Upstream'

When traffic is completely absent from your service, it's often due to either an authentication issue or an Istio-related networking issue.

In Kubernetes clusters, while you can use container logs to look at service-specific events, much of the networking flow occurs through Istio sidecars. These sidecars act as independent containers within your pods and are not always reflected in the main container logs.

To investigate, go to the Explorer page in Grafana and query logs based on the destination of the virtual service in question.

For example, if the service name is api, query logs for Istio interactions and virtual service forwarding. Below is an example LogQL query:

{namespace="YOUR_NAMESPACE", container="istio-proxy"} |= "api"
  • {namespace="YOUR_NAMESPACE", container="istio-proxy"} => This targets logs from the Istio sidecar (istio-proxy) within your namespace.
  • |= "api" => Filters the logs to include entries that mention "api", which is typically how the virtual service or destination service would appear in the logs.

Look for log entries such as no_healthy_upstream, which suggests that traffic is not reaching a healthy endpoint or is being blocked by Istio.

Troubleshooting scenarios

These logs can arise due to:

  • Destination endpoint issues (for example, the target service is unavailable or misconfigured).
  • Istio routing rules that do not match the intended destination.
Scenario 2: Authentication failure

Game Warden deploys a cluster-wide authentication service called authservice, which manages traffic authorization based on rules defined for each application.

When traffic is blocked due to authentication issues, you'll see log entries indicating failed authentication attempts, often using codes such as UAEX ext_authz_denied.

Troubleshooting scenarios

To troubleshoot:

  • Use namespace-scoped queries in Grafana to capture logs from all relevant services, not just individual containers.
  • Look for authentication errors in logs and identify which service or token is failing the checks.
  • Determine if the authorization policies need to be adjusted or if authservice rules need updates to accommodate new traffic patterns.
Best practices for troubleshooting
  • While container-scoped logs are helpful for isolating issues in a specific service, using namespace-scoped logs provides better visibility into how services interact—especially for traffic routing and authentication flows.
  • In most cases, start with namespace-scoped queries to understand the full service communication context, and narrow down to container logs only when investigating a specific process or isolated error.

Best practices

Use structured logs:

  • Format log messages as JSON or key-value pairs to make them easier to parse and analyze.
  • Structured logging improves searchability, enables more precise queries, and ensures compatibility with observability tools like Loki and Grafana.
  • Use structured logging libraries available in your programming language or set consistent key-value formatting.

Include log levels:

  • Incorporate standard log levels such as DEBUG, INFO, WARNING, and ERROR.
  • Most logging libraries include log levels automatically. Make sure these levels are not stripped out or suppressed in production logs.
  • Log levels let you filter logs by severity so you can identify critical issues.

Balance logs and metrics:

  • If you're frequently logging an INFO-level message just to track the frequency of an event, consider creating a metric instead.
  • Metrics (like counters and gauges) are more efficient for tracking event counts and trends over time, while logs should focus on diagnostics and warnings.
  • Reserve logs for events that need investigation (for example, unexpected errors or critical warnings).

Support and resources

To request technical support or customizations:

  • Submit a pairing session request via the Support Ticketing system.
  • After onboarding, customers receive up to four hours of pairing support for observability-related issues.
August 5, 2026
Was this page helpful?
← PreviousPlatform Processes & Security ControlsNext →Managed Services