feat: 시계 reconciliation 워커와 명령 경계 완성

This commit is contained in:
2026-09-03 09:40:48 +00:00
parent ae7d55ef47
commit a3e2bf90ae
46 changed files with 3011 additions and 219 deletions
+24 -8
View File
@@ -6,6 +6,10 @@
"coveredFields": [
"input_event.accepted_game_tick",
"input_event.accepted_clock_revision",
"input_event.accepted_deadline_generation",
"input_event.processing_game_tick",
"input_event.processing_clock_revision",
"input_event.processing_deadline_generation",
"world_state.clock_tick",
"world_state.last_turn_tick",
"world_state.clock_revision",
@@ -125,8 +129,16 @@
{
"key": "accepted-command-coordinate",
"policy": "KEEP",
"authorityFields": ["input_event.accepted_game_tick", "input_event.accepted_clock_revision"],
"projectionFields": [],
"authorityFields": [
"input_event.accepted_game_tick",
"input_event.accepted_clock_revision",
"input_event.accepted_deadline_generation"
],
"projectionFields": [
"input_event.processing_game_tick",
"input_event.processing_clock_revision",
"input_event.processing_deadline_generation"
],
"owner": "game-api/input-event"
},
{
@@ -139,11 +151,15 @@
},
{
"key": "movable-json-rule-anchors",
"policy": "FORBID",
"authorityFields": ["world_state.meta.turntime", "world_state.meta.starttime", "world_state.meta.tnmt_time"],
"policy": "SHIFT",
"authorityFields": [
"world_state.meta.turntime",
"world_state.meta.starttime",
"world_state.meta.tnmt_time"
],
"projectionFields": [],
"owner": "game-engine/world-meta",
"migration": "Register typed columns or explicit participant adapters before exact reconciliation can complete."
"owner": "game-engine/clock-operation",
"migration": "Explicit adapter shifts registered ISO projections while typed tick columns remain authoritative."
},
{
"key": "unification-wait",
@@ -179,12 +195,12 @@
{
"keyPattern": "sammo:{profile}:auction:timer",
"policy": "REBUILD",
"status": "implemented-without-clock-revision-fence"
"status": "implemented-with-clock-revision-phase-generation-fence"
},
{
"keyPattern": "sammo:{profile}:tournament:state",
"policy": "REBUILD",
"status": "legacy-date-dual-write-required"
"status": "implemented-with-tick-dual-write-and-clock-fence"
}
],
"wallOnly": [
+19 -4
View File
@@ -65,6 +65,19 @@ pop is one Redis operation: verify revision/phase, read `-inf..nowTick`, and
remove the claimed members. A failed Redis rebuild therefore leaves the game in
`RECONCILING`; process liveness alone is not readiness.
`applyNextClockProjection()` claims rows with `FOR UPDATE SKIP LOCKED` and uses
PostgreSQL UTC wall time for claim/retry timestamps. One Redis Lua operation
compares the active source revision, rebuilds the auction timer, conditionally
replaces the exact tournament source snapshot, and writes target revision plus
deadline generation. The DB finalizer then re-acquires the clock-operation lock
and changes `RECONCILING -> RUNNING` only when target revision and generation
still match. If the process dies after the Lua commit, retry observes the
already-active target revision and performs only the DB finalizer.
An active legacy tournament containing only `nextAt`/`bettingCloseAt` is a
fail-closed migration boundary. Reconciliation remains incomplete until its
authoritative `nextTick`/`bettingCloseTick` dual-write exists.
## Lock order
All mutation paths use this order:
@@ -101,7 +114,9 @@ future anchored realtime profiles as `PREOPEN`, and other profiles as
`RUNNING`. Existing DateTime columns remain projections while tick columns are
authoritative.
Exact reconciliation stays disabled while any registry participant is
`FORBID`. In particular, Redis-only tournament dates and unification wait must
be moved to durable tick/revision contracts before the operation can reach
`RUNNING`. Removing these guards to make a partial operation pass is prohibited.
Exact reconciliation stays disabled while an active registry participant is
`FORBID`. Tournament writes now carry tick/revision/generation coordinates and
are revision-fenced in Redis. The remaining unification wait participant must
be moved from its `lastTurnTime` workaround to a durable suspension before that
workflow can reach `RUNNING`. Removing this guard to make a partial operation
pass is prohibited.
@@ -11,7 +11,7 @@ mean deployment or production validation.
## Milestone 1 - authority and inventory
- [x] Branded `GameTick`, `ObservedGameInstant`, `ScheduleInstant`,
`WallInstant`, and `ClockRevision` boundaries.
`WallInstant`, and `ClockRevision` boundaries.
- [x] Explicit clock phase and monotonic RUNNING projection.
- [x] Exact alignment arithmetic preserving millisecond/sub-turn remainder.
- [x] Opening tick zero and PREOPEN executable floor in the shared seeder.
@@ -19,49 +19,50 @@ mean deployment or production validation.
- [x] Suspension, participant-checksum, and Redis projection outbox tables.
- [x] Machine-readable DB/Redis/JSON participant inventory and architecture gate.
- [x] Turn flush lock prefix and phase/revision/generation fence.
- [ ] Empty and upgraded database migration execution evidence.
- [x] Empty and upgraded database migration execution evidence.
## Milestone 2 - exact DB reconciliation
- [ ] Suspension start command with DB wall time and idempotent source revision.
- [ ] Exact resume plan transaction with deterministic participant lock order.
- [ ] SHIFT adapters for cursor, generals, active auctions, message expiry, vote
end, select pool, and NPC selection windows.
- [ ] KEEP checksum adapters for occurrences and history.
- [ ] Explicit `LEGACY_COMPLETE_TURNS` and bounded `CATCH_UP` policies.
- [ ] Property tests for remaining distance, ordering, and history invariants.
- [ ] 24-hour and 65m17.250s PostgreSQL integration evidence.
- [x] Suspension start command with DB wall time and idempotent source revision.
- [x] Exact resume plan transaction with deterministic participant lock order.
- [x] SHIFT adapters for cursor, generals, active auctions, message expiry, vote
end, select pool, and NPC selection windows.
- [x] KEEP checksum adapters for occurrences and accepted command coordinates.
- [x] Explicit `LEGACY_COMPLETE_TURNS` and bounded `CATCH_UP` policies.
- [x] Property tests for remaining distance, ordering, and history invariants.
- [x] 24-hour and 65m17.250s PostgreSQL integration evidence.
## Milestone 3 - revisioned Redis and workers
- [ ] Projection outbox claimer/retry/recovery state machine.
- [ ] Redis active revision and atomic due-pop script.
- [ ] Auction OPEN/FINALIZING revision and generation fence.
- [ ] Tournament durable tick dual-write and projection rebuild.
- [ ] DB-commit/Redis-failure restart tests and readiness integration.
- [x] Projection outbox claimer/retry/recovery state machine.
- [x] Redis active revision and atomic due-pop script.
- [x] Auction OPEN/FINALIZING revision and generation fence.
- [x] Tournament durable tick dual-write and projection rebuild.
- [x] DB-commit/Redis-failure crash-restart test.
- [x] Readiness integration in Gateway/API process health.
## Milestone 4 - command and lifecycle workflows
- [ ] All durable input events record accepted tick and accepted revision.
- [ ] Processing converts accepted coordinates across revisions or fails closed.
- [x] All durable input events record accepted tick and accepted revision.
- [x] Processing converts accepted coordinates across revisions or fails closed.
- [ ] Gateway pause/resume/open orchestration writes the DB clock phase.
- [ ] Unification wait becomes a durable `UNIFICATION_WAIT` suspension.
- [ ] Alignment, optional rate change, invader IDs/RNG, creation, first schedule,
outbox, verification, and RUNNING transition form one retry-safe workflow.
outbox, verification, and RUNNING transition form one retry-safe workflow.
- [ ] Multi-host drift and general-access/clock-operation deadlock tests.
## Milestone 5 - test-branch release gate
- [ ] Full typecheck, architecture, lint, unit, build, and non-conditional
integration suites.
integration suites.
- [ ] Dedicated PostgreSQL/Redis conditional integration suite with skip count
recorded.
recorded.
- [ ] Recovery runbook exercised from each incomplete status.
- [ ] Admin status/readiness exposes revision, phase, participant checksums, and
incomplete outbox state.
- [x] Admin status/readiness exposes revision, phase, participant checksums, and
incomplete outbox state.
- [ ] User-test deployment evidence is recorded separately from Git push.
- [ ] All `FORBID` inventory entries are removed by typed migrations or proven
inactive preconditions.
inactive preconditions.
## Evidence log
@@ -78,4 +79,25 @@ mean deployment or production validation.
- `pnpm check:architecture`: package boundaries passed; 21 authoritative clock
fields and 18 participants were registered.
- Migration SQL was generated, formatted, validated, and registered as the
release manifest head. Empty/upgraded PostgreSQL execution is still pending.
release manifest head. All 43 migrations applied to the empty dedicated
PostgreSQL instance. A second schema upgraded from the previous head with an
existing manual `world_state` row and verified `MANUAL:1:1` plus all three
reconciliation tables.
- Dedicated PostgreSQL/Redis fixtures proved exact 24-hour and 65m17.250s gaps,
schedule distance/order preservation, occurrence checksums, live-lease
fencing, a single durable outbox, Redis target revision, and recovery after a
crash between Redis commit and DB finalization.
### 2026-09-03 - revisioned workers and input coordinates
- `CI=1 TURBO_CONCURRENCY=1 pnpm typecheck`: 21/21 tasks passed after the
worker/input contract changes.
- `CI=1 TURBO_CONCURRENCY=1 pnpm test`: 12/12 package tasks passed.
- Dedicated PostgreSQL runs passed `databaseCommandQueue.integration.test.ts`
6/6 and `runtimeClockShiftPersistence.integration.test.ts` 3/3 when executed
sequentially; the latter now clears each single-world fixture before the next
case. Dedicated Redis passed tournament source revision 4/4, including an
atomic stale-clock rejection. API input-event integration passed 13/13.
- The 24-hour/65m17.250s reconciliation suite passed 2/2 after the other DB
suites. Conditional files share a deliberately dedicated schema and are run
sequentially to prevent their fixture cleanup from racing another file.
+12
View File
@@ -28,6 +28,18 @@ service. The service must re-read participant checksums and either return the
already-applied result or resume the pending outbox. Never create a replacement
revision to hide a failed target revision.
When an outbox row is `FAILED`, the profile must remain `RECONCILING`. A retry
is safe in both crash locations:
- before Redis commit, the Lua operation reapplies from the source revision;
- after Redis commit but before DB finalization, the Lua operation returns the
already-active target result and DB finalization resumes without shifting any
tournament deadline twice.
`lastError` naming a legacy tournament deadline means the tournament lacks its
tick dual-write. Do not force the active revision; migrate or prove the
tournament inactive, then retry the same outbox.
## Rollback
There is no blind inverse update. Before enabling exact reconciliation in an