# Game clock reconciliation recovery

This runbook is intentionally fail-closed. Do not force a profile to `RUNNING`
or delete an outbox row merely because its process is alive.

## Observe

Read only the target game schema. Record `world_state.clock_phase`,
`clock_revision`, `deadline_generation`, the latest `clock_suspension`, all its
participant checksums, and the matching `clock_projection_outbox`. Compare that
target revision with `sammo:{profile}:clock:active-revision`. Never print DB or
Redis credentials.

## Status meaning

- `SUSPENDED`: the cut is durable; no alignment DB transaction has committed.
- `RECONCILING` with `PENDING`/`FAILED` outbox: DB schedules moved, Redis is not
  authoritative yet, and gameplay must remain stopped.
- `RECONCILING` with `APPLIED` outbox: verify Redis active revision and all
  participant checksums before finalizing.
- `RUNNING`: DB revision, deadline generation, and Redis active revision must
  agree. A mismatch is an incident and workers must not dequeue.

## Retry

Retry the same suspension ID and target revision through the clock-operation
service. The service must re-read participant checksums and either return the
already-applied result or resume the pending outbox. Never create a replacement
revision to hide a failed target revision.

For `UNIFICATION_WAIT`, never rerun invader creation as a separate repair.
The input event, aligned schedules, optional rate, deterministic invader IDs,
reserved turns, and outbox committed together. A committed command with a
`RECONCILING` world therefore needs only the same outbox retry. If the command
transaction rolled back, the original prompt and source revision remain and
the same response can be retried without changing IDs or RNG results.

When an outbox row is `FAILED`, the profile must remain `RECONCILING`. A retry
is safe in both crash locations:

- before Redis commit, the Lua operation reapplies from the source revision;
- after Redis commit but before DB finalization, the Lua operation returns the
  already-active target result and DB finalization resumes without shifting any
  tournament deadline twice.

`lastError` naming a legacy tournament deadline means the tournament lacks its
tick dual-write. Do not force the active revision; migrate or prove the
tournament inactive, then retry the same outbox.

## Rollback

There is no blind inverse update. Before enabling exact reconciliation in an
environment, keep the normal database backup required for schema migrations.
If participant verification shows an unexpected mutation, stop the profile,
retain the ledger/outbox evidence, and restore the whole game schema from that
backup. Redis projections are then rebuilt from the restored DB revision.

The conditional clock suite exercises `SUSPENDED`, `RECONCILING/PENDING`,
`RECONCILING/FAILED` before and after Redis commit, recovered `APPLIED`, and
final `RUNNING`. The admin status endpoint exposes the phase, revision,
participant checksums, and outbox error needed to choose the matching step.