Grafana Developer Troubleshooting Guide

This guide helps developers troubleshoot applications in our Kubernetes environment using the Grafana observability stack.

Target audience: Developers working with Keycloak, ESB, RDS, Customer Profile Service, and other applications.

Use cases:

Quick Access

I need to…	Go to
Start troubleshooting	HN Overview Dashboard
Investigate pod restarts/OOMs	HN Pod Health Dashboard
Check cluster capacity	HN Node Health Dashboard
Debug ESB/AWS integrations	HN ESB Dashboard
Check ingress errors	HN Ingress Performance
Check authentication issues	User Logins Dashboard
Explore logs	Logs Drilldown
Explore metrics	Metrics Drilldown
Explore traces	Traces Drilldown

We use the Grafana LGTM stack (Loki, Grafana, Tempo, Mimir/Prometheus):

Component	What it stores	Retention	Use it for
Prometheus	Metrics (numbers over time)	7 days	CPU, memory, request rates, error rates
Loki	Logs (application output)	7 days	Error messages, debug info, audit trails
Tempo	Traces (request flows)	7 days	End-to-end request debugging
Grafana	Visualization	-	Dashboards, exploration, alerting

Work through the sections in order, or jump to a specific playbook when troubleshooting.

Section	What you’ll learn	Time
01 - Dashboards Overview	Tour of pre-built dashboards	~15 min
02 - Metrics & Prometheus	Finding and querying metrics	~20 min
03 - Logs & Loki	Log structure, LogQL queries	~20 min
04 - Traces & Tempo	Distributed tracing, TraceQL	~15 min
05 - Correlation	Connecting logs, metrics, and traces	~10 min
06 - Alerting	Understanding alerts and notifications	~10 min

Scenario	When to use
Pod Keeps Restarting / OOM	Pod restarts, OOMKilled events
Stuck Deployment	Deployment not progressing
Application Errors	Users report errors, 5xx responses
Performance Degradation	Slow responses, high latency

Topic	Description
Query Cheat Sheet	Common PromQL and LogQL queries
Dashboard Reference	All dashboards and their purpose
Architecture	How the observability stack works

Environment	URL
Non-production	grafana.nonprod.hollandsnieuwe.nl

Ready? Start with the Dashboards Overview to learn about the pre-built dashboards.