Metrics and Observability¶

ShinyHub exposes Prometheus metrics for the server process itself (the control plane), emits a structured access log for every request, and - when tracing is enabled - records control-plane spans correlated with that access log. This is separate from the per-app CPU/RAM sampling shown in the dashboard and from the per-app proxy trace buffer documented in tracing.md.

The /metrics endpoint¶

Metrics are opt-in and served on their own listener, separate from the main application port so server internals are never exposed on a routable interface by accident:

metrics:
  enabled: true
  addr: "127.0.0.1:9090"   # default when enabled and unset

Environment overrides (last-wins over YAML):

The endpoint defaults to loopback. Operators scraping from another host set addr to a private interface behind their own network controls (the conventional Prometheus pattern). When enabled: false no handler and no listener are created.

Scrape it like any Prometheus target:

scrape_configs:
  - job_name: shinyhub
    static_configs:
      - targets: ["shinyhub-host:9090"]

Exposed series¶

Process and build¶

Control-plane HTTP¶

Labeled by the matched chi route pattern (not the raw path), so high-cardinality path parameters and unmatched 404 scans cannot explode the series count.

Data-plane admission¶

The reason label is a closed vocabulary. The same value is returned on the X-Shinyhub-Reject response header, so a rejected request can be traced from the client back to this counter. Reasons differ in what they mean you should do, which is why they are not collapsed into one:

render-deferred counts page loads deferred, not sessions refused, and it is inflated by design: one waiting browser re-polls roughly every 1.75 s until capacity frees, so a single user can contribute dozens of increments. Use it to see that users are waiting; use render-paced to count sessions actually turned away. Alerting on render-deferred as if it were a refusal rate will page you for one patient user.

Both session gauges are exported per control-plane instance, like every metric here. On a single-node deployment they are exact. In a clustered deployment, scrape every instance and aggregate in PromQL (sum by (slug) (...)) rather than reading one instance in isolation - the example alert below already does this.

Fleet and lifecycle¶

Application-log durability and delivery¶

These process-local signals cover the shared-log path used by clustered Postgres deployments. They deliberately carry no app, replica, or run labels: those identities are unbounded and belong in the log viewer, not in Prometheus series. Scrape every control-plane instance and aggregate with sum(...) in HA.

The flush and buffer metrics remain present at zero on single-node SQLite deployments, where the viewer reads its local files directly. Retention cleanup counters can still rise there. Like all Prometheus counters, cleanup and loss totals reset when a ShinyHub process restarts.

After a follower read fails, its polling interval backs off exponentially with jitter to a five-second ceiling. A successful read restores the normal 200 ms cadence, while a locally committed log chunk wakes the follower immediately. This bounds database pressure during an outage without delaying healthy local delivery.

Example alerts¶

groups:
  - name: shinyhub
    rules:
      - alert: ShinyHubDeployFailures
        expr: increase(shinyhub_deploys_total{result="failure"}[15m]) > 0
        annotations:
          summary: "A ShinyHub deploy failed in the last 15m"

      - alert: ShinyHubReplicaFlapping
        expr: increase(shinyhub_replica_restarts_total[10m]) > 5
        annotations:
          summary: "A ShinyHub replica is crash-restarting repeatedly"

      - alert: ShinyHubSessionsNearCap
        # sum by (slug) aggregates across control-plane instances; on a single
        # node it is simply the one series.
        expr: sum by (slug) (shinyhub_app_sessions) / sum by (slug) (shinyhub_app_sessions_limit) > 0.9
        for: 5m
        annotations:
          summary: "{{ $labels.slug }} is above 90% of its admission ceiling"

      - alert: ShinyHubAppLogPersistenceErrors
        expr: sum(increase(shinyhub_app_log_flush_attempts_total{result="error"}[10m])) > 0
        annotations:
          summary: "Shared app-log persistence has failed"

      - alert: ShinyHubAppLogBacklogStuck
        expr: sum(shinyhub_app_log_pending_bytes) > 0
        for: 5m
        annotations:
          summary: "Shared app-log bytes have remained queued for 5m"

      - alert: ShinyHubAppLogDataDropped
        expr: sum(increase(shinyhub_app_log_buffer_dropped_bytes_total[5m])) > 0
        annotations:
          summary: "Shared app-log history may be incomplete"

      - alert: ShinyHubAppLogFollowErrors
        expr: sum(increase(shinyhub_app_log_follow_errors_total[10m])) > 0
        annotations:
          summary: "Shared app-log live delivery has encountered database errors"

Access log¶

Every request emits one structured api_access record through log/slog (replacing chi's unstructured stock logger), so the control plane has a single structured log stream a log aggregator can ingest. Fields:

• request_id - per-request correlation ID (see below)
• method, path, route (matched pattern), status
• bytes, duration_ms
• client_ip - the trusted-proxy-aware client IP (honest even when ShinyHub sits behind an edge proxy; see server.trusted_proxies)
• trace_id - present when tracing is enabled and a span is active

Request-ID correlation¶

Each request is assigned a correlation ID echoed on the response as the X-Request-Id header and threaded through the request context so downstream handlers tag their own logs with the same ID. A well-formed inbound X-Request-Id (e.g. minted by a trusted edge proxy) is honored so a request stays correlated across tiers; a malformed or oversized value is rejected and replaced, closing a log- and header-injection vector.

Log <-> trace correlation¶

When server tracing is enabled (see tracing.md), the access log and the trace are linked in both directions:

• the api_access record carries the active span's trace_id, so you can pivot from a log line to the trace, and
• the server span carries the request_id attribute, so you can pivot from a trace back to the log line.

Server (control-plane) tracing¶

Enabling tracing also instruments the control-plane API: ShinyHub emits one server span per request and spans for background lifecycle operations (lifecycle.wake, lifecycle.restart, lifecycle.hibernate, tagged with shinyhub.app.slug), exported to the same OTLP endpoint the managed apps use. Spans use OpenTelemetry HTTP semantic-convention attribute names (http.request.method, http.route, http.response.status_code) and carry a resource identifying the instance (service.name, service.version, service.instance.id). An inbound traceparent is adopted as the parent, so a client/edge trace links through ShinyHub to the app it proxies.

This reuses the existing tracing config block; there is no separate server-tracing switch. See tracing.md for the configuration fields and the per-app proxy trace buffer.