feat: 시계 reconciliation 워커와 명령 경계 완성
This commit is contained in:
@@ -6,6 +6,10 @@
|
||||
"coveredFields": [
|
||||
"input_event.accepted_game_tick",
|
||||
"input_event.accepted_clock_revision",
|
||||
"input_event.accepted_deadline_generation",
|
||||
"input_event.processing_game_tick",
|
||||
"input_event.processing_clock_revision",
|
||||
"input_event.processing_deadline_generation",
|
||||
"world_state.clock_tick",
|
||||
"world_state.last_turn_tick",
|
||||
"world_state.clock_revision",
|
||||
@@ -125,8 +129,16 @@
|
||||
{
|
||||
"key": "accepted-command-coordinate",
|
||||
"policy": "KEEP",
|
||||
"authorityFields": ["input_event.accepted_game_tick", "input_event.accepted_clock_revision"],
|
||||
"projectionFields": [],
|
||||
"authorityFields": [
|
||||
"input_event.accepted_game_tick",
|
||||
"input_event.accepted_clock_revision",
|
||||
"input_event.accepted_deadline_generation"
|
||||
],
|
||||
"projectionFields": [
|
||||
"input_event.processing_game_tick",
|
||||
"input_event.processing_clock_revision",
|
||||
"input_event.processing_deadline_generation"
|
||||
],
|
||||
"owner": "game-api/input-event"
|
||||
},
|
||||
{
|
||||
@@ -139,11 +151,15 @@
|
||||
},
|
||||
{
|
||||
"key": "movable-json-rule-anchors",
|
||||
"policy": "FORBID",
|
||||
"authorityFields": ["world_state.meta.turntime", "world_state.meta.starttime", "world_state.meta.tnmt_time"],
|
||||
"policy": "SHIFT",
|
||||
"authorityFields": [
|
||||
"world_state.meta.turntime",
|
||||
"world_state.meta.starttime",
|
||||
"world_state.meta.tnmt_time"
|
||||
],
|
||||
"projectionFields": [],
|
||||
"owner": "game-engine/world-meta",
|
||||
"migration": "Register typed columns or explicit participant adapters before exact reconciliation can complete."
|
||||
"owner": "game-engine/clock-operation",
|
||||
"migration": "Explicit adapter shifts registered ISO projections while typed tick columns remain authoritative."
|
||||
},
|
||||
{
|
||||
"key": "unification-wait",
|
||||
@@ -179,12 +195,12 @@
|
||||
{
|
||||
"keyPattern": "sammo:{profile}:auction:timer",
|
||||
"policy": "REBUILD",
|
||||
"status": "implemented-without-clock-revision-fence"
|
||||
"status": "implemented-with-clock-revision-phase-generation-fence"
|
||||
},
|
||||
{
|
||||
"keyPattern": "sammo:{profile}:tournament:state",
|
||||
"policy": "REBUILD",
|
||||
"status": "legacy-date-dual-write-required"
|
||||
"status": "implemented-with-tick-dual-write-and-clock-fence"
|
||||
}
|
||||
],
|
||||
"wallOnly": [
|
||||
|
||||
@@ -65,6 +65,19 @@ pop is one Redis operation: verify revision/phase, read `-inf..nowTick`, and
|
||||
remove the claimed members. A failed Redis rebuild therefore leaves the game in
|
||||
`RECONCILING`; process liveness alone is not readiness.
|
||||
|
||||
`applyNextClockProjection()` claims rows with `FOR UPDATE SKIP LOCKED` and uses
|
||||
PostgreSQL UTC wall time for claim/retry timestamps. One Redis Lua operation
|
||||
compares the active source revision, rebuilds the auction timer, conditionally
|
||||
replaces the exact tournament source snapshot, and writes target revision plus
|
||||
deadline generation. The DB finalizer then re-acquires the clock-operation lock
|
||||
and changes `RECONCILING -> RUNNING` only when target revision and generation
|
||||
still match. If the process dies after the Lua commit, retry observes the
|
||||
already-active target revision and performs only the DB finalizer.
|
||||
|
||||
An active legacy tournament containing only `nextAt`/`bettingCloseAt` is a
|
||||
fail-closed migration boundary. Reconciliation remains incomplete until its
|
||||
authoritative `nextTick`/`bettingCloseTick` dual-write exists.
|
||||
|
||||
## Lock order
|
||||
|
||||
All mutation paths use this order:
|
||||
@@ -101,7 +114,9 @@ future anchored realtime profiles as `PREOPEN`, and other profiles as
|
||||
`RUNNING`. Existing DateTime columns remain projections while tick columns are
|
||||
authoritative.
|
||||
|
||||
Exact reconciliation stays disabled while any registry participant is
|
||||
`FORBID`. In particular, Redis-only tournament dates and unification wait must
|
||||
be moved to durable tick/revision contracts before the operation can reach
|
||||
`RUNNING`. Removing these guards to make a partial operation pass is prohibited.
|
||||
Exact reconciliation stays disabled while an active registry participant is
|
||||
`FORBID`. Tournament writes now carry tick/revision/generation coordinates and
|
||||
are revision-fenced in Redis. The remaining unification wait participant must
|
||||
be moved from its `lastTurnTime` workaround to a durable suspension before that
|
||||
workflow can reach `RUNNING`. Removing this guard to make a partial operation
|
||||
pass is prohibited.
|
||||
|
||||
@@ -11,7 +11,7 @@ mean deployment or production validation.
|
||||
## Milestone 1 - authority and inventory
|
||||
|
||||
- [x] Branded `GameTick`, `ObservedGameInstant`, `ScheduleInstant`,
|
||||
`WallInstant`, and `ClockRevision` boundaries.
|
||||
`WallInstant`, and `ClockRevision` boundaries.
|
||||
- [x] Explicit clock phase and monotonic RUNNING projection.
|
||||
- [x] Exact alignment arithmetic preserving millisecond/sub-turn remainder.
|
||||
- [x] Opening tick zero and PREOPEN executable floor in the shared seeder.
|
||||
@@ -19,49 +19,50 @@ mean deployment or production validation.
|
||||
- [x] Suspension, participant-checksum, and Redis projection outbox tables.
|
||||
- [x] Machine-readable DB/Redis/JSON participant inventory and architecture gate.
|
||||
- [x] Turn flush lock prefix and phase/revision/generation fence.
|
||||
- [ ] Empty and upgraded database migration execution evidence.
|
||||
- [x] Empty and upgraded database migration execution evidence.
|
||||
|
||||
## Milestone 2 - exact DB reconciliation
|
||||
|
||||
- [ ] Suspension start command with DB wall time and idempotent source revision.
|
||||
- [ ] Exact resume plan transaction with deterministic participant lock order.
|
||||
- [ ] SHIFT adapters for cursor, generals, active auctions, message expiry, vote
|
||||
end, select pool, and NPC selection windows.
|
||||
- [ ] KEEP checksum adapters for occurrences and history.
|
||||
- [ ] Explicit `LEGACY_COMPLETE_TURNS` and bounded `CATCH_UP` policies.
|
||||
- [ ] Property tests for remaining distance, ordering, and history invariants.
|
||||
- [ ] 24-hour and 65m17.250s PostgreSQL integration evidence.
|
||||
- [x] Suspension start command with DB wall time and idempotent source revision.
|
||||
- [x] Exact resume plan transaction with deterministic participant lock order.
|
||||
- [x] SHIFT adapters for cursor, generals, active auctions, message expiry, vote
|
||||
end, select pool, and NPC selection windows.
|
||||
- [x] KEEP checksum adapters for occurrences and accepted command coordinates.
|
||||
- [x] Explicit `LEGACY_COMPLETE_TURNS` and bounded `CATCH_UP` policies.
|
||||
- [x] Property tests for remaining distance, ordering, and history invariants.
|
||||
- [x] 24-hour and 65m17.250s PostgreSQL integration evidence.
|
||||
|
||||
## Milestone 3 - revisioned Redis and workers
|
||||
|
||||
- [ ] Projection outbox claimer/retry/recovery state machine.
|
||||
- [ ] Redis active revision and atomic due-pop script.
|
||||
- [ ] Auction OPEN/FINALIZING revision and generation fence.
|
||||
- [ ] Tournament durable tick dual-write and projection rebuild.
|
||||
- [ ] DB-commit/Redis-failure restart tests and readiness integration.
|
||||
- [x] Projection outbox claimer/retry/recovery state machine.
|
||||
- [x] Redis active revision and atomic due-pop script.
|
||||
- [x] Auction OPEN/FINALIZING revision and generation fence.
|
||||
- [x] Tournament durable tick dual-write and projection rebuild.
|
||||
- [x] DB-commit/Redis-failure crash-restart test.
|
||||
- [x] Readiness integration in Gateway/API process health.
|
||||
|
||||
## Milestone 4 - command and lifecycle workflows
|
||||
|
||||
- [ ] All durable input events record accepted tick and accepted revision.
|
||||
- [ ] Processing converts accepted coordinates across revisions or fails closed.
|
||||
- [x] All durable input events record accepted tick and accepted revision.
|
||||
- [x] Processing converts accepted coordinates across revisions or fails closed.
|
||||
- [ ] Gateway pause/resume/open orchestration writes the DB clock phase.
|
||||
- [ ] Unification wait becomes a durable `UNIFICATION_WAIT` suspension.
|
||||
- [ ] Alignment, optional rate change, invader IDs/RNG, creation, first schedule,
|
||||
outbox, verification, and RUNNING transition form one retry-safe workflow.
|
||||
outbox, verification, and RUNNING transition form one retry-safe workflow.
|
||||
- [ ] Multi-host drift and general-access/clock-operation deadlock tests.
|
||||
|
||||
## Milestone 5 - test-branch release gate
|
||||
|
||||
- [ ] Full typecheck, architecture, lint, unit, build, and non-conditional
|
||||
integration suites.
|
||||
integration suites.
|
||||
- [ ] Dedicated PostgreSQL/Redis conditional integration suite with skip count
|
||||
recorded.
|
||||
recorded.
|
||||
- [ ] Recovery runbook exercised from each incomplete status.
|
||||
- [ ] Admin status/readiness exposes revision, phase, participant checksums, and
|
||||
incomplete outbox state.
|
||||
- [x] Admin status/readiness exposes revision, phase, participant checksums, and
|
||||
incomplete outbox state.
|
||||
- [ ] User-test deployment evidence is recorded separately from Git push.
|
||||
- [ ] All `FORBID` inventory entries are removed by typed migrations or proven
|
||||
inactive preconditions.
|
||||
inactive preconditions.
|
||||
|
||||
## Evidence log
|
||||
|
||||
@@ -78,4 +79,25 @@ mean deployment or production validation.
|
||||
- `pnpm check:architecture`: package boundaries passed; 21 authoritative clock
|
||||
fields and 18 participants were registered.
|
||||
- Migration SQL was generated, formatted, validated, and registered as the
|
||||
release manifest head. Empty/upgraded PostgreSQL execution is still pending.
|
||||
release manifest head. All 43 migrations applied to the empty dedicated
|
||||
PostgreSQL instance. A second schema upgraded from the previous head with an
|
||||
existing manual `world_state` row and verified `MANUAL:1:1` plus all three
|
||||
reconciliation tables.
|
||||
- Dedicated PostgreSQL/Redis fixtures proved exact 24-hour and 65m17.250s gaps,
|
||||
schedule distance/order preservation, occurrence checksums, live-lease
|
||||
fencing, a single durable outbox, Redis target revision, and recovery after a
|
||||
crash between Redis commit and DB finalization.
|
||||
|
||||
### 2026-09-03 - revisioned workers and input coordinates
|
||||
|
||||
- `CI=1 TURBO_CONCURRENCY=1 pnpm typecheck`: 21/21 tasks passed after the
|
||||
worker/input contract changes.
|
||||
- `CI=1 TURBO_CONCURRENCY=1 pnpm test`: 12/12 package tasks passed.
|
||||
- Dedicated PostgreSQL runs passed `databaseCommandQueue.integration.test.ts`
|
||||
6/6 and `runtimeClockShiftPersistence.integration.test.ts` 3/3 when executed
|
||||
sequentially; the latter now clears each single-world fixture before the next
|
||||
case. Dedicated Redis passed tournament source revision 4/4, including an
|
||||
atomic stale-clock rejection. API input-event integration passed 13/13.
|
||||
- The 24-hour/65m17.250s reconciliation suite passed 2/2 after the other DB
|
||||
suites. Conditional files share a deliberately dedicated schema and are run
|
||||
sequentially to prevent their fixture cleanup from racing another file.
|
||||
|
||||
@@ -28,6 +28,18 @@ service. The service must re-read participant checksums and either return the
|
||||
already-applied result or resume the pending outbox. Never create a replacement
|
||||
revision to hide a failed target revision.
|
||||
|
||||
When an outbox row is `FAILED`, the profile must remain `RECONCILING`. A retry
|
||||
is safe in both crash locations:
|
||||
|
||||
- before Redis commit, the Lua operation reapplies from the source revision;
|
||||
- after Redis commit but before DB finalization, the Lua operation returns the
|
||||
already-active target result and DB finalization resumes without shifting any
|
||||
tournament deadline twice.
|
||||
|
||||
`lastError` naming a legacy tournament deadline means the tournament lacks its
|
||||
tick dual-write. Do not force the active revision; migrate or prove the
|
||||
tournament inactive, then retry the same outbox.
|
||||
|
||||
## Rollback
|
||||
|
||||
There is no blind inverse update. Before enabling exact reconciliation in an
|
||||
|
||||
Reference in New Issue
Block a user