> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hr-easy.nlead.ch/llms.txt
> Use this file to discover all available pages before exploring further.

# Runbooks

> The operations playbooks — what exists, and when to reach for each.

The playbooks live in `docs/operations/` in the repository. They are intentionally
short — five to ten minutes of reading, not exhaustive treatment — and each is
self-contained.

## When to reach for which

| Situation                                                                 | Playbook                      |
| ------------------------------------------------------------------------- | ----------------------------- |
| A bad deploy is in production right now                                   | `rollback.md`                 |
| The site is down, users cannot log in, or a service is degraded           | `incident-runbook.md`         |
| You need to restore the database from backup                              | `backup-restore.md`           |
| Performance is degrading and you suspect resource limits                  | `capacity-planning.md`        |
| You are rotating the magic-link signing secret                            | `magic-link-rotation.md`      |
| You are rotating the Microsoft Graph client secret                        | `graph-secret-rotation.md`    |
| M365 login fails with `AADSTS7000222`, or you are rotating the SSO secret | `sso-secret-rotation.md`      |
| The applicant e-mail scan is not attaching or moving mail                 | `applicant-email-scan.md`     |
| You are spinning up the platform for a new customer                       | `customer-onboarding.md`      |
| You want Application Insights alerts wired in                             | `app-insights-alerts.md`      |
| You want secrets in Key Vault rather than App Service configuration       | `key-vault-migration.md`      |
| Terraform apply fails on `roleAssignments/write` or `/delete` with a 403  | `terraform-ci-rbac.md`        |
| Terraform state needs to move or be recovered                             | `terraform-remote-state.md`   |
| The production domain is live and you want it HSTS-preloaded              | `hsts-preload.md`             |
| A user reports a Chrome "Dangerous site" warning                          | `safe-browsing-warning.md`    |
| DOCX to PDF conversion has stopped working                                | `libreoffice-sidecar.md`      |
| The BFS Gemeindeliste refresh has stopped                                 | `bfs-gemeindeliste-cron.md`   |
| You are rolling out multiple Rechtsträger on a live installation          | `multi-entity-rollout-sot.md` |

## The three you will reach for under pressure

<AccordionGroup>
  <Accordion title="Rollback" icon="rotate-left">
    A bad release in production. Redeploy the previous image; treat database migrations as
    forward-only — rolling one back is a restore, not a redeploy, which is why the rollback and
    restore playbooks are separate documents.
  </Accordion>

  <Accordion title="Incident runbook" icon="triangle-exclamation">
    Site down, sign-in broken, or a degraded dependency. Covers triage order, what to check first,
    and how to communicate.

    <Warning>
      The runbook does **not** currently cover the nDSG breach-assessment and EDÖB notification
      decision. For an incident with a suspected data exposure, escalate to the controller's
      data-protection contact alongside the technical response — the playbook will not prompt you.
    </Warning>
  </Accordion>

  <Accordion title="Backup and restore" icon="database">
    Point-in-time restore of PostgreSQL, and what to do about blob storage and Key Vault alongside
    it. A database restored without its matching documents is half a restore.
  </Accordion>
</AccordionGroup>

## Secret rotation

Three secrets have their own playbooks because rotating them naively logs people out,
breaks the mailbox, or gets silently reverted by the next deploy:

* **Magic-link secret** — rotating it invalidates every outstanding link, including
  onboarding links valid for seven days. The playbook covers the overlap window.
* **Graph client secret** — expiry shows up as the mailbox reporting auth-expired in
  the messages surface. The mailbox diagnostic identifies it; the playbook rotates it.
* **M365 SSO client secret** — expiry shows up as `AADSTS7000222` on the login page
  and blocks every M365 sign-in. The deploy workflows overwrite the App Service value
  from a GitHub environment secret, so the rotation must update both — the playbook
  covers that trap.

Database credentials rotate on a schedule through a dedicated workflow.

## Customer onboarding

Standing up a new installation: Azure resources, Entra registration, the shared mailbox,
branding, seed data, feature flags, the first administrator, and the customer-side IT
contact. The checklist ends with a verification section — as they all do.

## Customer overlays

The playbooks are customer-agnostic and use placeholders (`<app>`, `<rg>`, `<vault>`).
What specifically exists in a given customer's environment — the resource inventory,
contacts, decision log — lives under `docs/customers/<customer>/`.

Keep the two apart. A playbook with one customer's resource names in it is a playbook
that cannot be used for the next customer.

## Ownership

Each installation has a primary on-call — the platform maintainer — and a customer-side
IT contact for M365 and Azure billing, recorded during onboarding.

<Note>
  Update the relevant playbook in the same PR as any production change that affects operations. A
  stale playbook is worse than a missing one, because it is trusted at 03:00.
</Note>