Zero-downtime upgrades (systemd)¶
ShinyHub hands new connections to a successor during a compatible upgrade.
Existing app sessions keep using the retiring proxy until they close or its
drain deadline expires; sessions still open at that deadline are disconnected.
On an acknowledged shinyhub reload request (or SIGHUP), the running control plane re-execs the
new binary, hands off its listening sockets to the successor, drains in-flight
WebSocket sessions, releases the ownership lease, and exits. App processes keep
running throughout (server.shutdown_apps: adopt, the default).
Deployment acceptance tests¶
make test-systemd-upgrade runs on a disposable Linux host with systemd,
passwordless sudo, flock, nginx, uv, Go, Node and the render driver's Playwright
dependencies installed. It uses synthetic accounts and fresh SQLite state;
it does not connect to an existing ShinyHub deployment. CI runs it, and the
release workflow requires it before publishing. The Make targets serialize these
fixtures with a host lock: independently launched native servers can otherwise
compete for worker ports while starting. Run these targets on a disposable host
without other ShinyHub instances.
The test selects the latest non-yanked stable PyPI version below the candidate
version, installs its wheel with uv tool, starts the reference
systemd unit, and deploys a real Shiny app behind a loopback nginx TLS proxy.
It builds the candidate wheel from the current source, reinstalls it through
Ansible, and invokes systemctl reload. Fresh authenticated HTTP probes must
succeed throughout installation and handoff. The original browser must remain
reactive on the same WebSocket beyond the drain deadline; a new browser session
must also work. Served version, active server PID, supervisor PID and service
restart count are checked. When a role override is used, its source files are fingerprinted in the report.
Sampling detects observed failures; it cannot prove
the absence of every possible transient failure under arbitrary load.
After the continuity checks finish, the test kills only the active successor
while the original session and retired relay remain alive. It requires exactly
one automatic systemd restart, removal of the old service's processes, and a
fresh reactive Shiny session on the candidate version. This separate crash
phase expects the old WebSocket to disconnect; continuity is checked during the
earlier planned handoff. Both reference and role units must use serve --supervise
with a stable direct-child MainPID, without systemd PIDFile= tracking.
Another crash check kills the supervisor's direct child and requires a
retryable failure, one automatic restart, and a new reactive session.
The role fixture first migrates a plain-serve/PIDFile= service through the
role's actual unit-install task and restart handlers, before opening sessions.
Set SHINYHUB_E2E_LEGACY_UNIT_TEMPLATE to a previous unit template to rehearse
that exact starting definition; the report records its SHA-256. Without it,
the fixture synthesizes the legacy supervision shape from the current unit.
After crash recovery, a same-version reload verifies that closing the preserved
browser lets the supervisor's direct child retire without a restart or loss of
the successor PID file. A separate future-schema database verifies the supervised
binary exits with status 7 on permanent startup refusal.
To test a deployment repository's actual package tasks, handlers and hardened
unit, set SHINYHUB_E2E_ROLE_DIR to its shinyhub Ansible role directory. The
role must provide tasks/main.yml, tasks/package.yml, and templates/shinyhub.service.j2, with
an Install ShinyHub systemd unit task and restart handlers, service name and
runtime directory overrides. The fixture supplies its own
configuration, credentials and paths. The reference fixture starts with its
selected unit installed. The role fixture first performs the legacy unit
migration and its planned restart; both then verify package-only upgrades.
Production SSO, the production proxy configuration, and other config,
environment, or unit changes that require a restart remain outside this test.
Run make test-systemd-upgrade-negative for the negative control. It deliberately
uses stop/start instead of reload and passes only when the first failure is the
original WebSocket continuity assertion, after the candidate has started serving,
and the HTTPS probes also observed failures during stop/start.
Setup failures never count as successful controls. CI and the release gate run
this control too. Cleanup only stops the fixture's uniquely named units.
make test-browser-session-relays-e2e runs the same-version native reload checks
on macOS or Linux without systemd. It verifies the original WebSocket objects
remain open without reconnecting beyond the fixture's drain deadline, a distinct
successor session can recalculate, and closing all browser sessions reduces the
CLI's retired-generation and session counts to zero without changing the active
PID. Authenticated HTTP probes run throughout the handoffs and retirement.
make test-entra-oidc-browser separately checks that an emulator-authenticated
OIDC session survives an acknowledged reload and a full page navigation without
another provider sign-in. These isolated tests do not validate production DNS,
proxy routing, or Microsoft tenant policies.
One-time setup¶
- Install the binary at
/usr/local/bin/shinyhub(a real built binary;go runcannot perform graceful reloads). - Set a PID file in
shinyhub.yaml: - Install
deploy/systemd/shinyhub.service, thensystemctl daemon-reloadandsystemctl enable --now shinyhub.
Upgrading¶
Special procedure: 0.12.x to 0.13.0¶
The schedule-provenance upgrade is an intentional exception to the normal
zero-downtime procedure. Use a maintenance window and stop the 0.12.x control
plane; do not use systemctl reload for this version boundary. Version
0.13.0 installs a database admission fence that prevents a still-running 0.12.x
server from launching another unattributed schedule process.
- While 0.12.x is still running, inventory every enabled schedule with
on_success = "roll". Version 0.12.x allowed these schedules on local Docker; 0.13.0 requires every data-producing schedule to use effectiveworker_isolation = "multiplex"and a local native runtime. Move each app to that topology, or disable/remove the producer policy before upgrading. - Replace every manifest
run_on_registerdeclaration with an explicitdeploy_trigger: normallybundle_changefor a cache that must match its bundle, orfirst_deployfor a genuine one-time bootstrap. - Let all in-flight schedule runs finish, verify their terminal state with
shinyhub schedule runs, then stop the old control plane withsystemctl stop shinyhub. Confirm no scheduled-job process or descendant is left on the host; reboot if that is the only reliable way to establish the process fence. - Install 0.13.0 and start the service. Startup validates every enabled stored
producer before serving. An error names the app, schedule, isolation, tier,
or runtime that still needs correction. Stop the service again before
correcting a startup error so
Restart=on-failurecannot retry around your maintenance work. - If startup reports
legacy schedule writer fence, keep all old servers and the new service stopped, verify the listed process trees are gone, and run the new binary's offline resolver:
systemctl stop shinyhub
shinyhub resolve-legacy-schedule-writers
shinyhub resolve-legacy-schedule-writers --acknowledge-processes-stopped
systemctl start shinyhub
The first invocation is read-only and lists the affected app, schedule, and
run IDs. The acknowledged invocation records conservative data uncertainty;
it does not declare the cache safe.
6. Reapply the updated manifests after startup. Migrated schedules default to
deploy_trigger = "never" because 0.12.x did not persist
run_on_register. If the resolver marked a producer as requiring repair,
rerun that exact schedule under the new fence and wait for success:
Check shinyhub apps show APP and shinyhub schedule ls APP; do not return
the app to service while compatibility_quarantined or
producer_repair_required remains true.
For later compatible releases, use the normal handoff:
# 1. Replace the binary in place with the new version.
install -m 0755 ./shinyhub /usr/local/bin/shinyhub
# 2. Trigger the zero-downtime handoff.
systemctl reload shinyhub
The reference systemd unit uses shinyhub reload --wait 10m. The command
returns success only after the successor acknowledges readiness and its PID
matches server.pid_file. Standalone successors must regain mutation ownership;
session-preserving standalone reloads also verify that previously serving apps
have reachable, adopted replicas meeting ready_before.
The predecessor counts ready replicas using the same checks as the successor;
only normal replica slots count toward the predecessor's obligation, since
scheduled-roll surge replicas are temporary. Apps with no ready normal replicas
create no serving obligation. A host already serving
below its configured warm floor does not need to reach that floor to reload;
the predecessor logs a warning with the ready count and configured floor.
This includes apps that begin serving while the command waits for admitted work.
An unavailable app is named in the readiness failure rather than reporting success.
Clustered successors may be healthy standbys after their first pool sync. A refusal
or failed readiness check returns nonzero, so systemctl reload reports failure.
A refusal before handoff leaves the current process serving; after an accepted
handoff, inspect the successor and the named app before retrying.
The unit allows fourteen minutes for the default ten-minute transition wait,
startup acknowledgement, and a separate readiness budget. --wait only bounds
the transition wait before handoff. After handoff acknowledgement, readiness has
its own server.upgrade_timeout + 1m limit (2 minutes with the default 60s
upgrade timeout), regardless of --wait. On timeout, the CLI names the blockers
and reports them to the successor for an error log in the server journal. This
report is best effort if the successor's control socket is unavailable.
If you increase --wait or server.upgrade_timeout, set TimeoutStartSec to at least
wait + 2 * upgrade_timeout + 75s.
For a local invocation without systemd:
During a waiting session-preserving reload, new schedule runs and scheduled activation claims are paused so newly due producers and queued hourly rolls do not extend the wait. Already-admitted schedule runs finish their execution and durable result handling before handoff; blockers name the app, schedule, and run. This prevents retirement from interrupting an in-place producer and quarantining its serving app. An already-admitted roll finishes normally, and retries for work that already owns runtime state remain admitted, including synchronous producers needed by an admitted deployment. A canceled or failed reload resumes schedule admissions and the activation queue; the successor resumes durable queued work after a successful handoff. Schedules missed during the pause follow their configured missed-run policy. Draining replicas remain a reload blocker; preserving their sessions across reload is not yet supported. Connection-identity blockers name the app, connection ID, deployment, and replica so they can be matched to WebSocket logs.
These admission protections run in the retiring server. When upgrading from
0.20.6, arrange the first reload after active producers and rolls have finished;
replacing the binary cannot change the already-running process's retirement logic.
The readiness replica cap is also captured by the retiring server; a first reload
from 0.20.7 or 0.20.8 still uses that server's uncapped warm-floor requirement.
For that first reload, temporarily lower the live app's min_warm_replicas to
its current ready count, then restore the target once the fixed server is running.
If it has already handed off with a nonzero exit, verify the successor PID and
readiness before retrying.
make test-browser-reload-producers-e2e exercises this boundary with real native
processes, a cron producer overlapping a scheduled roll, and a Shiny browser
session. It verifies exact replica adoption, uninterrupted page probes, and
session continuity beyond the reload drain deadline. A second case deliberately
fails a producer and verifies that an accepted handoff with no ready app replica
returns nonzero with the app named. New manual schedule requests during the
reload pause return 503 with Retry-After: 1 and can be retried afterward.
Without --wait, temporary blockers fail immediately. With it, the command
waits up to that duration for app operations, candidates, draining generations,
serving activations, parent retirement, or relay capacity. Elastic pools and successor errors fail
immediately. Canceling the caller cancels the transition wait. Once a handoff
starts, the server uses server.upgrade_timeout to finish or recover it; if the
acknowledgement connection is lost, verify the published live PID before retrying.
The private .reload directory beside the PID file contains process-specific
control sockets and must remain accessible while the server is running. Keep
its path short enough for the platform's Unix socket limit. Socket setup failures
are logged and leave the server serving; acknowledged reload then fails clearly
until the configuration is corrected. Crash-left sockets for the current PID
are reclaimed without replacing live sockets or other files.
When upgrading a server that predates acknowledged reloads, replace the binary
and use the existing signal-based unit once (or send SIGHUP to its MainPID).
Confirm the successor is ready, then install the updated unit and run
systemctl daemon-reload. The new command refuses to silently fall back to a
signal when the old server has no reload control socket. Existing custom units
that still use /bin/kill -HUP continue to report signal delivery, not the
handoff result; update ExecReload to use the command above.
A continuous client sees no connection-refused gap; in-flight sessions drain on
the old process for up to server.reload_drain_timeout (defaults to
server.drain_timeout, normally 60s) before any
straggler is force-closed.
For a longer upgrade drain without extending app-version retirement:
This setting is read by the retiring process at its own startup. Editing the file immediately before reload changes the successor's future drain budget, not the already-running parent's budget.
This postpones the upgrade disconnect; it does not transfer existing sessions
to the successor. The retiring proxy continues checking session authorization
throughout its drain. A SIGINT/SIGTERM shortens a reload drain to at most the
ordinary drain_timeout; it does not reset an earlier reload deadline.
For stable single-host native multiplex apps, session relays can remove the reload deadline entirely:
This is opt-in. Configure it before starting the process whose sessions you want to retain; the already-running parent's settings govern that handoff. The old process keeps its existing WebSocket tunnels, closes its database and lifecycle control plane, and retires its session lease when its final session closes. If it launched native backends, it keeps only their stdout/stderr readers and process reapers alive until those backends exit, preserving logs without database access. The retiring process logs how many launched replicas it is waiting for. Do not kill it while those replicas are running: their output pipes still belong to it, and killing it breaks their logs or can terminate them with SIGPIPE. Adopted backends do not create another output reader on each reload. A ready successor reads sealed leases for the exact backend processes, counts them for admission and hibernation, and performs authorization and usage persistence for the relays. Later reloads can proceed while earlier relays remain alive. Login expiration, access revocation and ordinary service stop still close sessions as usual.
A preserving handoff cannot begin while an app operation, candidate or previous
generation is still draining, or when any pool uses elastic isolation. The
current process keeps serving; the upgrade error names the blocking apps and
reasons (elastic pool, draining generation, candidate generation, or serving
activation). An acknowledged reload can wait for temporary blockers with
--wait; a signal-based reload is refused immediately. At the relay limit,
another preserving reload is refused
instead of disconnecting older sessions, reserving capacity even if a new session
arrives during handoff. Close old tabs to
reclaim capacity. The limit bounds retained session relays, not their age or output-only
processes still needed by native backends. Retained tunnels continue running the old proxy code, so use an ordinary
service restart when a security fix must replace every running proxy immediately.
Compatible app deployments and rolling restarts may run after the control plane
has retired. Their own drain_timeout still applies to retained backend sessions;
preserving a hub reload does not preserve a session across replacement of its
app process. Stop-first changes remain fenced while relayed sessions exist.
A waiting reload begins its handoff after the app-generation drain finishes. This
mode preserves WebSocket transport; XHR fallback transport and recovery after
an owner crash are outside its continuity guarantee.
Relay IPC uses the private app-data registry when its path fits the Unix socket
limit, then a short private RUNTIME_DIRECTORY if available. For unusually long
paths it falls back to private /tmp/shinyhub-relay-* directories; exclude those
live directories from age-based temporary-file cleanup. Removing a live socket
interrupts authorization checks and closes retained sessions after the verification
grace period.
Do not replace the binary with a legacy build while relays are alive. The
side-effect-free relay-protocol preflight refuses a handoff to a binary lacking
relay support. Auth-secret rotation invalidates relay RPC authentication and
ends retained sessions after the fail-closed authorization grace (at least 90s).
Runtime leases live in the private .reload-relays directory beneath app-data;
a private, uid-scoped local Unix socket carries owner RPCs. Relay protocols use
frozen versioned identity and usage fields, so future upgrades must retain that
compatibility while older relays remain alive.
Run the real Shiny acceptance test with
SHINYHUB_E2E_SESSION_RELAYS=1 node scripts/browser-lifecycle-e2e.mjs. It holds
reactive sessions across repeated reloads, probes page loads, checks the relay
limit, deploys a replacement, and revokes access to a relayed session.
Tunables (shinyhub.yaml)¶
| Key | Default | Meaning |
|---|---|---|
server.env_file |
(empty) | Private reloadable environment settings; see systemd. |
server.max_retired_lifetime |
24h |
Maximum preserved-session lifetime after control-plane retirement. Remaining sessions receive WebSocket 1012 when a complete outbound frame permits it; partial frames or slow transports close directly. |
server.pid_file |
(empty) | PID file naming the active server; required for acknowledged reloads and supervision. |
server.upgrade_timeout |
60s |
How long the old process waits for the successor to become ready before aborting the upgrade and continuing to serve. |
server.drain_timeout |
60s |
App-generation retirement, replica removal, and ordinary service shutdown session drain. |
server.reload_sessions |
drain |
Opt-in preserve for stable single-host native multiplex WebSockets. Environment: SHINYHUB_RELOAD_SESSIONS. |
server.reload_relay_limit |
4 |
Maximum retained relay processes, from 1 to 32. Environment: SHINYHUB_RELOAD_RELAY_LIMIT. |
server.reload_drain_timeout |
server.drain_timeout |
Positive duration for established WebSockets on a retiring proxy after a successful reload. Environment override: SHINYHUB_RELOAD_DRAIN_TIMEOUT. |
Notes & limits¶
- Repeated upgrades in drain mode. The handoff refuses another upgrade
while the successor's parent is still alive. A long reload drain therefore
delays the next upgrade; use
reload --waitto wait for the retiring proxy to exit, or retry after it exits. It also retains that process's memory and database connection. - Backend protection during handoff. The successor cannot count sessions still flowing through its parent. While that parent is alive, idle hibernation and lifecycle mutations such as deploy, stop, resize and non-rolling restart are fenced until the parent exits and recovery cleanup finishes. Rolling restarts are accepted into the durable queue and wait. This deliberately trades lifecycle availability for session continuity in drain mode. Session-preserving relays supply cross-process accounting after their control plane retires.
- Multiplex scope. Multiplex workers can be re-adopted. Elastic workers
(
groupedandper_session) have process-local client bindings and are rebuilt during recovery; a longer proxy drain does not preserve their backend sessions. - Service stop budget. Set systemd
TimeoutStopSecabove your ordinarydrain_timeoutplus HTTP and ownership shutdown (allow at least another minute). For a ten-minute drain, a twelve-minute stop budget is appropriate. The supervisor's termination policy can close sessions sooner than ShinyHub. - App changes remain separate. A deploy or schedule roll may retire the backend an old proxy connection uses before the reload drain ends. The longer reload budget does not extend that backend's generation deadline.
- Failure is safe. If the successor fails to start within
upgrade_timeout, the old process keeps serving - the upgrade simply does not happen. - Startup preflight. The handoff re-execs the process's own
argv[0], so at boot the server checks thatargv[0]resolves to an executable (the samePATHlookup the handoff uses) and logs a warning when it does not. With the warning present every reload fails safe with the current process still serving; fix how the service launches the binary (an absoluteExecStartpath, or a name resolvable on the unit'sPATH) before relying on reloads. - Database migrations must be backward-compatible and non-blocking. During the handoff window both versions briefly run against the same SQLite database. The successor applies any new migrations at startup while the previous version is still serving on the old schema. An upgrade that adds migrations must therefore use additive / expand-contract changes (the previous version keeps working against the new schema) - never a destructive rename or column drop in the same release - and must avoid long-running/locking migrations (SQLite runs them in a transaction that can block the old process); split large backfills or table rewrites out of the upgrade and run them separately afterward. Same-version restarts apply no migrations and are always safe.
- A rollback point is taken automatically. Before applying pending
migrations the successor copies the SQLite database aside as
<dsn>.pre-migration-v<version>-<timestamp>.sqlite, and logs the path. It is written only when migrations are pending, and older snapshots are pruned down to a configurable retention count afterward. See Schema migrations on startup for the opt-out and the Postgres equivalent. - Rolling back to an older binary needs the snapshot. An older build refuses
to serve a database a newer one migrated, exiting
7(schema_incompatible); the unit setsRestartPreventExitStatus=7so systemd stops rather than restart-loops. Put back the pre-migration snapshot along with the old binary, following Getting back to the older build - a snapshot is moved into place, not fed to
shinyhub restore, and its-walsidecar has to go or the rollback silently undoes itself. - systemd supervision. The reference unit runs
serve --supervise. Its stable direct child remainsMainPID; each ready server reports that supervisor PID throughNotifyAccess=all. The supervisor watches the active server'sserver.pid_filewith Linux pidfds. An unexpected active-server exit fails the supervisor and triggersRestart=on-failure, even while retired parents or replicas remain alive. Do not configure a systemdPIDFile=that would retargetMainPIDaway from the supervisor. Installing supervision on an existing unit requires one announced restart; bareservestill supports the older notification behavior, with its non-child tracking limitation. - Scope. This is the systemd/VM path and covers all app runtimes (native, Docker, Fargate, remote-worker). Multi-pod Kubernetes rolling upgrades are part of the separate high-availability project.
- Platform. Linux and macOS only (tableflip does not support Windows).
Page polling during a handoff¶
/app/APP/.shinyhub/version.json returns 503 while the active version is not
locally routable, including the successor's route recovery window.
/app/APP/.shinyhub/ready returns 503 with ready: false until local readiness
is satisfied. The readiness response includes Retry-After: 1. Both responses
are temporary polling outcomes; the page overlay retries without showing an
error. A persistent 503 requires investigating readiness or recovery.
HTTP-only probes never produce the “no WebSocket has connected” warning. The warning requires an upstream WebSocket upgrade to fail and no successful upgrade to clear the diagnostic during the grace window.
Retired generation visibility and lifetime¶
shinyhub status --config /etc/shinyhub/shinyhub.yaml inspects the host-local
PID file and locked relay registry. JSON output includes pid and retired
(count, sessions, unknown, and each generation's PID, retirement time and
session count). Older relay snapshots have no PID/time metadata. Unknown live
snapshots remain counted and explicitly flagged, rather than being treated as idle.
Prometheus exposes shinyhub_retired_generations, shinyhub_retired_sessions,
and shinyhub_retired_state_unknown from the same registry.
server.max_retired_lifetime (or SHINYHUB_MAX_RETIRED_LIFETIME) defaults to
24 hours and must be positive. It bounds preserved WebSockets independently of
drain_timeout; service stop still uses the ordinary shutdown grace. A retiring
server logs retired process exiting after its sessions and owned output finish.
New shared-user native launches use independent, capped output pumps, allowing
the retiring server to exit without breaking replica stdout. Isolated native
replicas use the journal. Legacy replicas whose stdout still terminates in an
older controller need that output owner until the replicas exit; a handoff cannot
retroactively replace their pipe endpoints. Their output-only owners stay in the
registry with zero sessions. Replace those replicas in an announced app rollout
before expecting the lifetime limit to eliminate every legacy process.