Operations Runbook

Day-2 operations: costs, quotas, backups, users, troubleshooting.

Cost accounting & the price book

The gateway tracks what your fleet costs and what its traffic is
worth — the worth numbers are yours, never derived:

1. Config → Costs & hosts: declare each physical box (IPs, overhead
   watts, hardware cost, purchase date, amortization years) and your
   electricity rate.
2. Set your price book on Usage & costs → Your price book → Edit prices (price_book): prompt / cached / generated
   $/1M. These are the numbers the "value" line uses. There is no
   recommended pricing; pricing is policy, and policy is yours.
3. Usage & costs (/admin/costs) shows the value-vs-cost daily chart: green (value at
   your prices) above red (actual cost = GPU energy + overhead + capex
   accrual) means the system pays for itself. History persists in SQLite
   (cost_history table) and survives restarts.

Engine energy comes from the NInfer fork's NVML accounting — see the
NInfer runbook. Hosts without fork telemetry show
overhead + capex only.

Services and GPUs. A service is one engine (host + port); a GPU is
a physical card. Several services can share a card, and one service can
span several. Every engine on a card reports that card's whole draw, so
list each box's cards under gpus (one per line, name = ports):

RTX PRO 6000 = 8006
RTX 4090 = 8001, 8009

A card with no ports runs every service on the box; a port on two cards
spans both. The gateway then counts each card (or group of cards joined
by a spanning service) once, and splits its energy between the services
on it by tokens served. Without a gpus list, each service is assumed to
have its own card, and the Usage & costs page warns when a box has several
services reporting energy.

Alerts

The gateway checks every 30 seconds and notifies you when something breaks. Each problem is sent once when it starts and once when it clears.

Set it up in Config → Alerts: add a webhook, Save, then Send a test alert.

• Phone push: install ntfy, subscribe to a long, unguessable topic, and add https://ntfy.sh/<topic> with format ntfy. Self-hosted ntfy works the same way.
• Slack / Discord: an incoming-webhook URL with format Slack or Discord.
• Anything else: format JSON posts {kind, severity, title, message, resolved, at}.

Each webhook chooses what it gets: everything, warnings and up (default), or critical only. You can switch individual alert kinds off.

Active alerts and the last notifications are on Overview (/admin/overview). When a service is gone for good, Stop watching it there; otherwise it keeps counting as down for a week. Every alert is also written to the gateway log (journalctl -u llm-gateway | grep ALERT).

Per-user quotas

Admins → Users → set a daily token limit per user. Exceeding it returns
429 with a retry hint until midnight UTC — the same UTC-day boundary
usage and cost accounting bucket by (the admin UI shows the rollover in
your local time). Admins and LAN-trusted unkeyed traffic are exempt by
design.

Service accounts (FR-14)

Machine identities for automated tools (CI, bots, monitoring, sidecars):
named accounts that open only the LLM plane (/v1 chat, models,
memory, RAG, agent) through a Bearer key — no login, no platform APIs, no
UI. Usage and the daily token limit attribute to the account name, so the
system-level view of "what did this tool cost" is just the per-user
analytics with the account name in the row.

• Manage: Admins → Users → Service accounts card (same page as the
  human user table) — create (mints the first key, shown once), add /
  rotate / revoke keys, set a daily token limit, set a purpose note,
  disable (kills every key of the account at once, reversible), delete
  (purges the account's keys + settings; usage history stays attributed
  to the name). API: GET/POST /admin/service-accounts
  (admin session or admin key; every mutation writes a
  svc.account.* audit row, FR-4).

• Quotas: a limit set on a service account enforces exactly like a
  user's (429 + Retry-After until UTC midnight). Service names are NOT
  in the exempt set — only admin, operator, and LAN-trusted local
  are.

• Name rules: platform username shape; local/operator reserved;
  collides with an existing human (PB) username or an existing service
  account (case-insensitive) — those collisions are refused at
  create-time in BOTH directions (a human create_user or OIDC
  provisioning under a service name is 409), so the usage attribution
  can never split.

• Wiring a tool:

  # mint (admin session cookie, or an admin key as Bearer)
  curl -s -b "$ADMIN_COOKIE" -H 'Content-Type: application/json' \
    -d '{"action":"create","name":"ci-lint","note":"repo CI linter"}' \
    http://127.0.0.1:8033/admin/service-accounts
  # → {"name":"ci-lint","key":"sk-…","key_id":"svc", …} — store the key
  # in the tool's environment / secret store; it is shown exactly once.
  
  # the tool then just uses it like any other gateway key:
  curl -s http://127.0.0.1:8033/v1/chat/completions \
    -H "Authorization: Bearer sk-…" \
    -H 'Content-Type: application/json' \
    -d '{"model":"<pool-or-alias>","messages":[{"role":"user","content":"hi"}]}'
  

• Usage: today's tokens + per-key tallies appear on the card; the
  per-user analytics (/admin/performance) lists the account like any
  user; a disabled account's keys are rejected instantly (the gate is in
  key resolution, both the cached and re-verified paths).

Users, keys, sessions

• Invites: Admins → Users → invite (credential hand-off; the user
  must change the password on first login).
• Keys: capped by gateway.max_active_keys (default 20; 0 =
  unlimited, set it on the Config page); rotation keeps the old key alive
  for a grace window; revocation kills sessions instantly (epoch bump).
• Native-client tokens (FR-13): the gateway issues short-lived access
  • rotating refresh tokens to first-party apps (POST /oauth/token,
    form-encoded grant_type=password / refresh_token). A device lists its
    sessions (GET /oauth/sessions), kills one (DELETE /oauth/sessions/<id>),
    or wipes them all (POST /oauth/revoke-all — the "something is
    compromised" lever; it also bumps the session epoch, so in-flight access
    tokens die before their 1-hour TTL). Both mutations are audit-logged.
    Tokens are stored hashed (the API-key PBKDF2 scheme), never plaintext.
• PB dashboard: http://127.0.0.1:8090/_/ — superuser credentials in
  /var/lib/llm-gateway/pb/.superuser-env (or re-seed with
  llm-gateway superuser upsert <email> <pass>).
• Deleting an account: Admins → Users → delete, or deleting the account
  in the PocketBase dashboard — both run the same cascade. The account's API
  keys, OAuth sessions, device tokens, hook tokens, space key wraps, runners,
  prefs, and connected GitHub/MCP credentials are purged; its mailbox is
  deleted; bookkeeping businesses it owns are removed. Personal content
  (chats, documents, files, notes, memories) and space/team memberships are
  kept — a membership removal needs the space's key rotation, not a bare row
  delete.

SQL workbench

Admins → SQL (/admin/sql/page) is a full workbench over the
gateway's own SQLite database (the embedded PocketBase data.db —
accounts, usage, chat and document metadata). It is admin-only; every run
is written to the journal (journalctl -u llm-gateway | grep 'admin sql')
with the acting user and the statement head.

• Query — raw SQL, one statement at a time for reads, multi-statement
  for writes (one paste = one transaction: it commits atomically or rolls
  back as a whole). Write statements are flagged in the UI, need a
  confirm, and can be previewed (dry-run): the run executes in the
  transaction and rolls back, reporting the affected rows without
  persisting anything. Results are capped at 2 000 rows; reads time out
  after 30 s. Two conveniences: describe a query in plain English and
  Generate SQL (the live schema + your prompt go to your default chat
  model; it drops one cleaned statement into the editor for review —
  nothing runs automatically), and clicking a table in the tree fills the
  editor with SELECT * … LIMIT 100 (its row-count badge opens the Data
  tab instead).
• Data — page through a table with an optional WHERE / ORDER BY.
• Inspect — columns, indexes and foreign keys of the selected table.
• History — the last 50 runs in this browser (local only; click one
  to re-load it into the editor).

Result grids export to CSV. The API (/admin/sql, /admin/sql/generate,
/admin/sql/browse, /admin/sql/table) also accepts an admin Bearer key,
so scripted maintenance (VACUUM, one-off UPDATEs) can run outside the
browser — VACUUM cannot be dry-run (it cannot run inside a transaction).

While a write run is open, the gateway's own PocketBase writes queue
behind it (SQLite is single-writer; the run's BEGIN IMMEDIATE holds the
write lock until its COMMIT/ROLLBACK, bounded by a 5-minute
per-run deadline). Keep write runs short.

Backups

What matters, in order:

sudo systemctl stop llm-gateway
tar czf llm-gateway-backup.tgz /var/lib/llm-gateway/pb /opt/llm-gateway/{users.json,llm_gateway.conf}
# the secrets key goes somewhere else (another disk, a password manager)
cp -p /opt/llm-gateway/users.json.key /secure/place/
sudo systemctl start llm-gateway

Encrypted chat & document data (vault)

Since the vault change, each user's chat history and document-search data
are encrypted per-user (SPEC §12). The DEK never leaves gateway memory;
only a wrapped DEK + public salt sit in the vault table inside data.db.

• Backups of pb/ (and users.json) are therefore inherently safe:
  a copied data.db + users.json + PB password hash cannot be opened
  without each user's current password. No extra off-site key to store.
• Restoring a backup preserves the protection; the user's next sign-in
  unlocks it as usual.
• Password change re-keys the data key in the same request (the
  gateway derives both wrap keys server-side — the passwords are in that
  request anyway). If you ever see vault rekey … ENCRYPTED DATA IS NOW LOCKED in the journal, the user's encrypted data is unreachable: treat
  that user's chat/document history as lost (their API access and keys
  still work).
• Admin password reset can only re-key while the user's session is
  live (the DEK is cached). Reset a locked-out user's password while they
  have no session and their encrypted data becomes permanently
  unreachable — the admin UI response carries an explicit WARNING when
  that happens.

Secrets in users.json

The session signing secret, each user's UI key and connector (MCP) auth headers are stored encrypted in users.json (AES-256-GCM). The key is in users.json.key next to it (mode 600; override the path with LLM_GATEWAY_SECRETS_KEY_FILE), created on first start. Older files with plain values are encrypted automatically on the next start.

Gateway-level integration secrets live here too, set from Admin → Config — the gateway never writes them to the config file (a value an operator put there by hand still works as a fallback): the GitHub OAuth client secret, the QuickBooks Online app client secret, the OIDC client secret, and the offsite S3 secret access key. Each is write-only in the UI — the page reports whether one is configured and which layer supplies it (stored → env var → config file), never the value. The OIDC secret applies immediately; the S3 key applies after the next restart (the offsite client is built once at boot).

• Back the key up, but not in the same place as users.json: together they reveal the secrets.
• Lose the key and the gateway refuses to start, saying so. Restore the key, or move users.json's encrypted values aside: users then sign in again, UI keys are re-issued and connectors need their tokens re-entered.
• A gateway older than this change can't read the encrypted values: rolling back signs everyone out and breaks connectors until they're re-entered.

Data flow (why restarts are safe)

Usage is recorded in memory → synced to SQLite every 60s and at
shutdown → usage.json/cost_history.json are mirrors. Every boot
re-merges the mirrors into SQLite (idempotent max-merge). Losing at most
one minute of tallies requires killing the process twice inside the same
minute.

Public exposure (Cloudflare tunnel)

The tunnel targets port 8033. 127.0.0.1 is never LAN-trusted (that's
the tunnel path), so tunnel traffic needs keys. Security posture: HMAC
session cookies with per-user epochs, per-IP rate limits, login backoff,
2 MiB body cap, no version banner.

Troubleshooting

Upgrading

Releases are tagged vX.Y.Z; install.sh re-run handles it (see
Installation Runbook). Skim the release notes for
config-schema notes — unknown fields survive (config overlay keeps old
values), but behavior changes are called out per release.

Security triage (dependabot)

GitHub dependabot alerts are triaged in the repo, not just on the
dashboard (the alerts' comment endpoint is unavailable on this repo, so
this section is the record). 2026-10-23 state:

• uuid (webui-host, dev-only transitive via storybook tooling) —
  pinned to 13.0.2 via an npm overrides entry in
  webui-host/package.json (devDep pin so the lockfile carries it
  deterministically for npm ci). The advisory (silent partial writes in
  v3/v5/v6 buffer paths, GHSA-w5hq-g745-h8pq) is closed by the
  bump; the package is dev-scope and never in the shipped bundle.
• sprintf-js (webui-host, dev-only transitive via loglevel) —
  dismissed inaccurate 2026-10-23: the advisory
  (GHSA-hp3w-g68c-fv31, unbounded-precision DoS) was withdrawn
  2026-10-22 as a caller-handling issue, not a library defect.
• disintegration/imaging (Go, transitive via PocketBase's image
  path) — dismissed tolerable_risk 2026-10-23: the library is
  unmaintained and v1.6.2 is its last release, so there is no patched
  version to bump to. Severity is low (crafted-TIFF panic) and the
  embedded PocketBase is loopback-only (127.0.0.1:8090), so the panic
  is not remotely reachable. Re-open if imaging ships a fix or PocketBase
  vendors it out.