Metrics For Your Web Application's Dashboards

Whenever I create a dashboard for an application, it’s generally the same handful of metrics I look to. They’re the ones I always use to orient myself quickly when Pagerduty fires. They give me the grand overview, and then I’ll know what logging queries to start writing, code to look at, box to SSH into, or mitigation to activate. The same metrics are able to tell me during the day whether the system is ok, and I use them to do napkin math on e.g. capacity planning and imminent bottlenecks:

• Web Backend (e.g. Django, Node, Rails, Go, ..)
  • Response Time p50, p90, p99, sum, avg †
  • Throughput by HTTP status †
  • Worker Utilization ¹
  • Request Queuing Time ²
  • Service calls †
    • Database(s), caches, internal services, third-party APIs, ..
    • Enqueued jobs are important!
    • Circuit Breaker tripping † /min
    • Errors, throughput, latency p50, p90, p99
  • Throttling †
  • Cache hits and misses % †
  • CPU and Memory Utilization
  • Exception counts † /min
• Job Backend (e.g. Sidekiq, Celery, Bull, ..)
  • Job Execution Time p50, p90, p99, sum, avg †
  • Throughput by Job Status {error, success, retry} †
  • Worker Utilization ³
  • Time in Queue † ⁴
  • Queue Sizes † ⁵
    • Don’t forget scheduled jobs and retries!
  • Service calls p50, p90, p99, count, by type †
  • Throttling †
  • CPU and Memory Utilization
  • Exception counts † /min

† Metrics where you need the ability to slice by endpoint or job, tenant_id, app_id, worker_id, zone, hostname, and queue (for jobs). This is paramount to be able to figure out if it’s a single endpoint, tenant, or app that’s causing problems.

You can likely cobble a workable chunk of this together from your existing service provider and APM. The value is for you to know what metrics to pay attention to, and which key ones you’re missing. The holy grail is one dashboard for web, and one for job. The more incidents you have, the more problematic it becomes that you need to visit a dozen URLs to get the metrics you need.

If you have little of this and need somewhere to start, start with logs. They’re the lowest common denominator, and if you’re productive in a good logging system that will you very far. You can build all these dashboards with logs alone. Jumping into the detailed logs is usually the next step you take during an incident if it’s not immediately clear what to do from the metrics.

Use the canonical log line pattern (see figure below), resist emitting random logs throughout the request as this makes analysis difficult. A canonical log line is a log emitted at the end of the request with everything that happened during the request. This makes querying the logs bliss.

Surprisingly, there aren’t good libraries available for the canonical log line pattern, so I recommend rolling your own. Create a middleware in your job and web stack to emit the log at the end of the request. If you need to accumulate metrics throughout the request for the canonical log line, create a thread-local dictionary for them that you flush in the middleware.

For response time from services, you will need to emit inline logs or metrics. Consider using an OpenTelemetry library so you only need to instrument once and can later add sinks for canonical logs (the sum), metrics, profiling, and traces.

Notably absent here is monitoring a database, which would take its own post.

Hope this helps you step up your monitoring game. If there’s a metric you feel strongly that’s missing, please let me know!

You might also like...

• Neural Network From Scratch
• Using checksums to verify syncing 100M database records
• MySQL transactions per second vs fsyncs per second
• Shitlist Driven Development
• Increase HTTP Performance by Fitting In the Initial TCP Slow Start Window