From efb7be3cb9aae05e9ea6c461c3517c5acc8f167a Mon Sep 17 00:00:00 2001 From: hided62 Date: Mon, 17 Aug 2026 15:43:14 +0000 Subject: [PATCH] =?UTF-8?q?feat:=20=EB=8B=A4=EC=A4=91=20=ED=94=84=EB=A1=9C?= =?UTF-8?q?=ED=95=84=20DB=20=EB=B6=80=ED=95=98=20=EA=B3=84=EC=B8=A1?= =?UTF-8?q?=EC=9D=84=20=EC=B6=94=EA=B0=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit production 턴 런타임의 lease, flush, outbox와 Redis 발행을 한 달 단위로 측정한다. nya와 pya 1분 프로필의 격리 fixture 및 현재 process·잠금 경계 문서를 함께 추가한다. --- docs/architecture/runtime.md | 30 ++ tools/load-tests/README.md | 39 ++- tools/load-tests/compose.capacity.yml | 4 +- .../config/nya-10-users-800-npcs-1m.json | 89 +++++ .../config/pya-10-users-800-npcs-1m.json | 89 +++++ tools/load-tests/package.json | 1 + tools/load-tests/src/cli.ts | 24 +- tools/load-tests/src/turnFlush.ts | 328 ++++++++++++++++++ tools/load-tests/test/config.test.ts | 22 ++ 9 files changed, 619 insertions(+), 7 deletions(-) create mode 100644 tools/load-tests/config/nya-10-users-800-npcs-1m.json create mode 100644 tools/load-tests/config/pya-10-users-800-npcs-1m.json create mode 100644 tools/load-tests/src/turnFlush.ts diff --git a/docs/architecture/runtime.md b/docs/architecture/runtime.md index 32e5350b..807c32d0 100644 --- a/docs/architecture/runtime.md +++ b/docs/architecture/runtime.md @@ -17,6 +17,20 @@ Gateway API와 game API는 기본적으로 `0.0.0.0`에 bind합니다. 실제 port와 prefix는 환경 변수와 배포 profile이 결정합니다. +현재 PM2 조립에서 game profile 하나는 frontend, API, turn daemon, auction, +battle-sim, tournament worker의 여섯 process를 만듭니다. 각 정의에는 +`instances`나 cluster `exec_mode`가 없으므로 모두 단일 fork입니다. frontend도 +Caddy 정적 파일이 아니라 profile별 Vite preview Node process이고, API도 하나의 +Fastify process입니다. worker 역할 분리는 API event loop의 작업을 줄이지만 +frontend/API replica나 장애 대체 backend를 제공하지는 않습니다. + +Profile은 PostgreSQL schema와 Redis namespace를 분리하지만 같은 database, +PostgreSQL instance, runtime cgroup을 공유합니다. 현재 `PrismaPg` adapter에는 +role별 pool 상한을 명시하지 않아 각 DB 사용 process가 `pg` 기본 pool 상한을 +독립적으로 가질 수 있습니다. 따라서 profile 수를 늘릴 때는 process RSS뿐 아니라 +API, daemon, 세 worker와 Gateway 계열의 합산 connection budget을 PostgreSQL +`max_connections` 안에서 먼저 정해야 합니다. + ## Gateway 실행 `resolveGatewayApiConfigFromEnv()`가 PostgreSQL schema, Redis prefix, session @@ -168,6 +182,22 @@ Checkpoint의 단일 소유자는 `InMemoryTurnWorld`이며 state store는 이 예약 턴은 revision/CAS와 lease를 사용합니다. API의 편집과 daemon의 실행이 경합해도 오래된 revision이 새 queue를 덮어쓰지 않게 합니다. +정상 gameplay 경로는 table 전체를 배타 잠그지 않습니다. 서로 다른 profile +schema의 row lock은 직접 충돌하지 않지만 다음 직렬화 지점은 남습니다. + +- daemon flush마다 profile별 `turn_daemon_lease`와 단일 `world_state` 행을 갱신합니다. +- `read_model_revision`의 전역 entity와 input-event revision/CAS는 같은 profile에서 hot row가 될 수 있습니다. +- outbox dispatcher는 `FOR UPDATE SKIP LOCKED`로 claim 경쟁을 분산합니다. +- 경매, 베팅, 메시지, 장수 선택·생성은 대상 row lock 또는 advisory lock을 사용합니다. +- PostgreSQL advisory lock은 schema가 아니라 database 범위입니다. key에 profile/schema를 포함하지 않은 일부 + 기능별 lock은 서로 다른 profile 사이에서도 같은 key일 때 잠깐 직렬화될 수 있습니다. + +월 경계 flush는 dirty world, 장수·국가·도시, 로그와 outbox를 한 transaction에 +저장하므로 일반 장수 1턴보다 lock 보유 시간이 깁니다. profile별 월 경계 시각이 +겹치면 row 자체는 달라도 PostgreSQL CPU/I/O, connection과 runtime memory에서 +경합합니다. Migration과 `RESET`의 강한 lock은 일반 운영 중 실행하지 않고 +orchestrator의 process 정지·배포 경계에서 다룹니다. + ## 월간 경계 Calendar handler는 turn time이 월 경계를 지날 때 scenario event table의 diff --git a/tools/load-tests/README.md b/tools/load-tests/README.md index 4297f8c6..7e47387b 100644 --- a/tools/load-tests/README.md +++ b/tools/load-tests/README.md @@ -36,6 +36,8 @@ versioned data-directory 계약에 맞춰 volume은 `/var/lib/postgresql`에 붙 `15442/16379`이며 `CAPACITY_POSTGRES_PORT`/`CAPACITY_REDIS_PORT`로 충돌 없이 바꿀 수 있다. `prepare`는 PostgreSQL password, API token/image secret과 정확한 URL을 무작위 생성해 Git ignored `secrets/`의 새 파일 세 개에 `0600`으로 저장한다. 기존 파일을 덮어쓰거나 비밀값을 stdout에 쓰지 않는다. +기존 fixture volume을 보존하면서 별도 실행이 필요하면 +`CAPACITY_COMPOSE_PROJECT_NAME`과 `CAPACITY_POSTGRES_VOLUME_NAME`을 함께 고유하게 지정한다. ```sh pnpm --filter @sammo-ts/load-tests prepare:capacity \ @@ -152,7 +154,34 @@ pnpm --filter @sammo-ts/game-engine profile:npc-capacity-1200 이 프로필은 자연 통일 소요시간 시험이 아니라 고정 1개월 engine 처리량 시험이다. 기존 `profile:npc-unification-timing`의 무보정 자연 진행 의미는 바꾸지 않는다. -### 4. 명시적 cleanup +### 4. E2 PostgreSQL flush와 profile 간 경합 + +`measure-turn-flush`는 검증된 fixture를 production loader로 읽고 daemon lease/fencing, 장수 턴, +dirty-state transaction, journal/outbox와 commit 이후 Redis 발행을 거쳐 정확히 한 월 경계를 실행한다. +정상 realtime daemon처럼 장수는 한 transaction에 하나씩 commit한다. 권위 순서는 `turn_tick`이며, +JavaScript `Date`의 밀리초보다 세밀한 tick을 포함하도록 cutoff를 보정한다. + +```sh +pnpm --filter @sammo-ts/load-tests measure-turn-flush \ + --config tools/load-tests/config/300-users-900-npcs-5m.json \ + --confirm load_capacity_300_900_5m \ + --output tools/load-tests/results/turn-flush.json +``` + +결과에는 장수 transaction과 월 transaction/Redis 발행 latency, 처리량, 시작·종료 장수 수, +process CPU/RSS/event-loop lag, database-wide `pg_stat_database` delta와 connection/active/lock-wait 최대값이 +들어간다. PostgreSQL delta에는 별도 observer sampler의 read transaction도 포함되므로 transaction 수를 +daemon commit 수와 동일하다고 해석하지 않는다. + +`nya`와 `pya` 동시 1분 경합용 config는 각각 Redis DB 14/13과 별도 `load_` schema를 사용한다. +같은 PostgreSQL/CPU에서 두 fixture를 seed한 뒤 두 `measure-turn-flush` process를 동시에 시작하여 +schema 간 row-lock 격리와 공유 CPU/I/O/connection 경합을 확인한다. 이 실행은 실제 운영 profile이나 +공개 URL을 대상으로 하지 않는다. + +- `tools/load-tests/config/nya-10-users-800-npcs-1m.json` +- `tools/load-tests/config/pya-10-users-800-npcs-1m.json` + +### 5. 명시적 cleanup token 파일은 별도로 안전하게 삭제하고, fixture schema/Redis token은 schema명을 그대로 확인 인자로 주어 정리한다. named volume은 보존한다. 데이터 폐기가 필요하지 않으면 이 명령을 실행하지 않는다. @@ -166,9 +195,11 @@ docker compose -f tools/load-tests/compose.capacity.yml down ## 아직 남은 측정 경계 -- `seed`/`verify-fixture`는 실제 PostgreSQL schema와 Redis access-token 상태를 만든다. 그러나 E2의 daemon - fast-forward, 한 달치 PostgreSQL flush/outbox publish, schedule lag와 DB statement count를 하나로 - 계측하는 실행기는 아직 없다. 따라서 E1이나 API/SSE driver 결과를 E2 합격으로 대체하지 않는다. +- `measure-turn-flush`는 한 달을 wall-clock보다 빠르게 replay하는 처리량 시험이다. 실제 schedule lag, + 장시간 pool wait와 autovacuum/checkpoint 영향을 보려면 profile별 속도로 pacing한 soak가 별도로 필요하다. +- 월 경계 latency는 실행당 표본이 하나다. scenario 진행 시점과 월별 event 차이를 포괄하지 않는다. +- 두 1분 profile 동시 실행은 최악 turn-rate 조합의 국소 증거이며, 여섯 profile의 전체 PM2 RSS, + Gateway·worker connection과 운영 container cgroup을 재현하지 않는다. - own/global phase의 mutation stimulus는 driver가 만들지 않는다. 격리 runtime의 실제 engine/API mutation과 함께 실행하지 않았다면 A2/A3/M1 전체 합격으로 보고하지 않는다. - 이 repository에서 실행한 로컬 E1 수치는 source-tree 회귀 근거다. dev-sam2026 동급 4 CPU/8 GiB container diff --git a/tools/load-tests/compose.capacity.yml b/tools/load-tests/compose.capacity.yml index feb2448c..ce1688fc 100644 --- a/tools/load-tests/compose.capacity.yml +++ b/tools/load-tests/compose.capacity.yml @@ -1,4 +1,4 @@ -name: sammo-capacity-fixture +name: ${CAPACITY_COMPOSE_PROJECT_NAME:-sammo-capacity-fixture} services: postgres: @@ -39,4 +39,4 @@ secrets: volumes: capacity_postgres_data: - name: sammo_capacity_fixture_postgres_data + name: ${CAPACITY_POSTGRES_VOLUME_NAME:-sammo_capacity_fixture_postgres_data} diff --git a/tools/load-tests/config/nya-10-users-800-npcs-1m.json b/tools/load-tests/config/nya-10-users-800-npcs-1m.json new file mode 100644 index 00000000..04207179 --- /dev/null +++ b/tools/load-tests/config/nya-10-users-800-npcs-1m.json @@ -0,0 +1,89 @@ +{ + "$schema": "./load-test.schema.json", + "version": 1, + "name": "nya-10-users-800-npcs-1m", + "target": { + "baseUrl": "http://127.0.0.1:15001", + "trpcPath": "/api/trpc", + "ssePath": "/events", + "publicProfile": false, + "allowedHosts": ["127.0.0.1", "localhost"] + }, + "isolation": { + "postgresSchema": "load_capacity_nya_10_800_1m", + "redisPrefix": "load-tests:capacity-nya-10-800-1m:", + "redisDatabase": 14, + "profileName": "load-tests:capacity-nya-10-800-1m" + }, + "capacity": { + "authenticatedViewers": 10, + "npcGenerals": 800, + "humanGenerals": 10, + "turnIntervalMs": 60000 + }, + "runtimeMetadata": { + "fixtureSha256": "sha256:0000000000000000000000000000000000000000000000000000000000000000", + "imageDigest": "replace-before-measurement", + "postgresVersion": "replace-before-measurement", + "redisVersion": "replace-before-measurement" + }, + "phases": [ + { + "name": "idle-calibration", + "kind": "idle", + "durationMs": 1000, + "sseConnections": 10, + "requestIntervalMs": null, + "operations": [] + }, + { + "name": "own-calibration", + "kind": "own", + "durationMs": 1000, + "sseConnections": 10, + "requestIntervalMs": 1000, + "operations": [ + { + "name": "own-context", + "procedure": "dashboard.getContextBundleDelta", + "type": "query", + "weight": 1, + "input": { + "include": { "context": true, "commandTable": true, "boardAccess": true }, + "forceSnapshot": false + } + } + ] + }, + { + "name": "global-calibration", + "kind": "global", + "durationMs": 1000, + "sseConnections": 10, + "requestIntervalMs": 1000, + "operations": [ + { "name": "global-front", "procedure": "general.getFrontStatus", "type": "query", "weight": 1 } + ] + }, + { + "name": "mixed-calibration", + "kind": "mixed", + "durationMs": 1000, + "sseConnections": 10, + "requestIntervalMs": 1000, + "operations": [ + { + "name": "own-context", + "procedure": "dashboard.getContextBundleDelta", + "type": "query", + "weight": 1, + "input": { + "include": { "context": true, "commandTable": true, "boardAccess": true }, + "forceSnapshot": false + } + }, + { "name": "global-front", "procedure": "general.getFrontStatus", "type": "query", "weight": 1 } + ] + } + ] +} diff --git a/tools/load-tests/config/pya-10-users-800-npcs-1m.json b/tools/load-tests/config/pya-10-users-800-npcs-1m.json new file mode 100644 index 00000000..89fd7960 --- /dev/null +++ b/tools/load-tests/config/pya-10-users-800-npcs-1m.json @@ -0,0 +1,89 @@ +{ + "$schema": "./load-test.schema.json", + "version": 1, + "name": "pya-10-users-800-npcs-1m", + "target": { + "baseUrl": "http://127.0.0.1:15001", + "trpcPath": "/api/trpc", + "ssePath": "/events", + "publicProfile": false, + "allowedHosts": ["127.0.0.1", "localhost"] + }, + "isolation": { + "postgresSchema": "load_capacity_pya_10_800_1m", + "redisPrefix": "load-tests:capacity-pya-10-800-1m:", + "redisDatabase": 13, + "profileName": "load-tests:capacity-pya-10-800-1m" + }, + "capacity": { + "authenticatedViewers": 10, + "npcGenerals": 800, + "humanGenerals": 10, + "turnIntervalMs": 60000 + }, + "runtimeMetadata": { + "fixtureSha256": "sha256:0000000000000000000000000000000000000000000000000000000000000000", + "imageDigest": "replace-before-measurement", + "postgresVersion": "replace-before-measurement", + "redisVersion": "replace-before-measurement" + }, + "phases": [ + { + "name": "idle-calibration", + "kind": "idle", + "durationMs": 1000, + "sseConnections": 10, + "requestIntervalMs": null, + "operations": [] + }, + { + "name": "own-calibration", + "kind": "own", + "durationMs": 1000, + "sseConnections": 10, + "requestIntervalMs": 1000, + "operations": [ + { + "name": "own-context", + "procedure": "dashboard.getContextBundleDelta", + "type": "query", + "weight": 1, + "input": { + "include": { "context": true, "commandTable": true, "boardAccess": true }, + "forceSnapshot": false + } + } + ] + }, + { + "name": "global-calibration", + "kind": "global", + "durationMs": 1000, + "sseConnections": 10, + "requestIntervalMs": 1000, + "operations": [ + { "name": "global-front", "procedure": "general.getFrontStatus", "type": "query", "weight": 1 } + ] + }, + { + "name": "mixed-calibration", + "kind": "mixed", + "durationMs": 1000, + "sseConnections": 10, + "requestIntervalMs": 1000, + "operations": [ + { + "name": "own-context", + "procedure": "dashboard.getContextBundleDelta", + "type": "query", + "weight": 1, + "input": { + "include": { "context": true, "commandTable": true, "boardAccess": true }, + "forceSnapshot": false + } + }, + { "name": "global-front", "procedure": "general.getFrontStatus", "type": "query", "weight": 1 } + ] + } + ] +} diff --git a/tools/load-tests/package.json b/tools/load-tests/package.json index 4c56e577..8c9c5d7f 100644 --- a/tools/load-tests/package.json +++ b/tools/load-tests/package.json @@ -11,6 +11,7 @@ "seed": "pnpm -w exec tsx tools/load-tests/src/cli.ts seed", "verify-fixture": "pnpm -w exec tsx tools/load-tests/src/cli.ts verify-fixture", "activate-coverage": "pnpm -w exec tsx tools/load-tests/src/cli.ts activate-coverage", + "measure-turn-flush": "pnpm -w exec tsx tools/load-tests/src/cli.ts measure-turn-flush", "materialize-calibration": "pnpm -w exec tsx tools/load-tests/src/cli.ts materialize-calibration", "cleanup": "pnpm -w exec tsx tools/load-tests/src/cli.ts cleanup", "test": "pnpm -w exec tsx --test tools/load-tests/test/*.test.ts", diff --git a/tools/load-tests/src/cli.ts b/tools/load-tests/src/cli.ts index b22df765..c6c9f5c5 100644 --- a/tools/load-tests/src/cli.ts +++ b/tools/load-tests/src/cli.ts @@ -11,6 +11,7 @@ import { verifyCapacityFixture, } from './fixture.js'; import { describeDryRun, runLoadTest } from './runner.js'; +import { measureTurnFlush } from './turnFlush.js'; type Command = | 'run' @@ -20,12 +21,13 @@ type Command = | 'seed' | 'verify-fixture' | 'activate-coverage' + | 'measure-turn-flush' | 'materialize-calibration' | 'cleanup'; const usage = (): never => { process.stderr.write( - 'usage: cli.ts --config [--tokens <0600-gitignored-file>] [--output ] [--confirm ]\n' + 'usage: cli.ts --config [--tokens <0600-gitignored-file>] [--output ] [--confirm ]\n' ); process.exit(64); }; @@ -43,6 +45,7 @@ const parseArguments = ( 'seed', 'verify-fixture', 'activate-coverage', + 'measure-turn-flush', 'materialize-calibration', 'cleanup', ].includes(command ?? '') @@ -69,6 +72,11 @@ const parseArguments = ( (!values.get('--confirm') || values.has('--tokens') || values.has('--output')) ) usage(); + if ( + command === 'measure-turn-flush' && + (!values.get('--confirm') || !values.get('--output') || values.has('--tokens')) + ) + usage(); if ( command === 'materialize-calibration' && (!values.get('--output') || values.has('--tokens') || values.has('--confirm')) @@ -116,6 +124,20 @@ const main = async (): Promise => { process.stdout.write(`${JSON.stringify(await activateCapacityCoverage(config, args.confirm!))}\n`); return; } + if (args.command === 'measure-turn-flush') { + const output = path.resolve(args.output!); + await mkdir(path.dirname(output), { recursive: true }); + const result = await measureTurnFlush({ config, confirmation: args.confirm! }); + await writeFile(output, `${JSON.stringify(result, null, 2)}\n`, { + encoding: 'utf8', + flag: 'wx', + mode: 0o600, + }); + process.stdout.write( + `${JSON.stringify({ completed: true, processedGenerals: result.throughput.processedGenerals, outputWritten: true })}\n` + ); + return; + } if (args.command === 'materialize-calibration') { process.stdout.write( `${JSON.stringify( diff --git a/tools/load-tests/src/turnFlush.ts b/tools/load-tests/src/turnFlush.ts new file mode 100644 index 00000000..74765594 --- /dev/null +++ b/tools/load-tests/src/turnFlush.ts @@ -0,0 +1,328 @@ +import { monitorEventLoopDelay } from 'node:perf_hooks'; + +import { + createTurnDaemonRuntime, + getNextTickTime, + type TurnCheckpoint, + type TurnRunResult, +} from '@sammo-ts/game-engine'; +import { createGamePostgresConnector, GamePrisma } from '@sammo-ts/infra'; + +import type { LoadConfig } from './config.js'; +import { verifyCapacityFixture } from './fixture.js'; +import { summarizeDistribution } from './metrics.js'; + +type DatabaseStatsRow = { + xactCommit: bigint; + xactRollback: bigint; + blocksRead: bigint; + blocksHit: bigint; + tuplesReturned: bigint; + tuplesFetched: bigint; + tuplesInserted: bigint; + tuplesUpdated: bigint; + tuplesDeleted: bigint; + tempFiles: bigint; + tempBytes: bigint; + deadlocks: bigint; +}; + +type ActivityRow = { + connections: bigint; + active: bigint; + waitingLocks: bigint; +}; + +const readDatabaseStats = async ( + database: ReturnType['prisma'] +): Promise => { + const rows = await database.$queryRaw(GamePrisma.sql` + SELECT + xact_commit AS "xactCommit", + xact_rollback AS "xactRollback", + blks_read AS "blocksRead", + blks_hit AS "blocksHit", + tup_returned AS "tuplesReturned", + tup_fetched AS "tuplesFetched", + tup_inserted AS "tuplesInserted", + tup_updated AS "tuplesUpdated", + tup_deleted AS "tuplesDeleted", + temp_files AS "tempFiles", + temp_bytes AS "tempBytes", + deadlocks + FROM pg_stat_database + WHERE datname = current_database() + `); + const row = rows[0]; + if (!row) throw new Error('pg_stat_database did not return the current database'); + return row; +}; + +const subtractDatabaseStats = (before: DatabaseStatsRow, after: DatabaseStatsRow): Record => + Object.fromEntries( + Object.keys(before).map((key) => { + const name = key as keyof DatabaseStatsRow; + return [key, (after[name] - before[name]).toString()]; + }) + ); + +const readActivity = async ( + database: ReturnType['prisma'] +): Promise => { + const rows = await database.$queryRaw(GamePrisma.sql` + SELECT + (SELECT count(*) FROM pg_stat_activity WHERE datname = current_database()) AS connections, + ( + SELECT count(*) + FROM pg_stat_activity + WHERE datname = current_database() AND state = 'active' AND pid <> pg_backend_pid() + ) AS active, + ( + SELECT count(*) + FROM pg_locks + WHERE NOT granted AND database = (SELECT oid FROM pg_database WHERE datname = current_database()) + ) AS "waitingLocks" + `); + const row = rows[0]; + if (!row) throw new Error('PostgreSQL activity sampler returned no row'); + return row; +}; + +const round = (value: number): number => Math.round(value * 1000) / 1000; + +const includeSubMillisecondGameTick = (turnTime: Date): Date => + // Game ticks are finer than JavaScript Date's millisecond precision. The + // loader projects authoritative turn_tick to a floored Date, so replaying + // that exact Date can map to a tick just before the general is due. + new Date(turnTime.getTime() + 1); + +/** + * Runs exactly one logical month through the production loader, lease/fencing, + * dirty-state flush, journal/outbox and Redis publication boundaries. General + * turns are committed one at a time in chronological order, matching a healthy + * realtime daemon rather than a catch-up chunk. + */ +export const measureTurnFlush = async (options: { + config: LoadConfig; + confirmation: string; + env?: NodeJS.ProcessEnv; +}) => { + const env = options.env ?? process.env; + if (options.confirmation !== options.config.isolation.postgresSchema) { + throw new Error('turn-flush confirmation must exactly equal isolation.postgresSchema'); + } + const databaseUrl = env.LOAD_TEST_DATABASE_URL; + const redisUrl = env.LOAD_TEST_REDIS_URL; + if (!databaseUrl || !redisUrl) { + throw new Error('LOAD_TEST_DATABASE_URL and LOAD_TEST_REDIS_URL are required'); + } + const fixture = await verifyCapacityFixture(options.config, env); + if (!fixture.valid) throw new Error('fixture verification failed; refusing turn-flush measurement'); + + const observer = createGamePostgresConnector({ url: databaseUrl }); + await observer.connect(); + let beforeStats: DatabaseStatsRow; + try { + beforeStats = await readDatabaseStats(observer.prisma); + } catch (error) { + await observer.disconnect().catch(() => undefined); + throw error; + } + const activity = { + samples: 0, + failures: 0, + maxConnections: 0, + maxActive: 0, + maxWaitingLocks: 0, + }; + let sampling = true; + const sampleActivity = async (): Promise => { + while (sampling) { + try { + const sample = await readActivity(observer.prisma); + activity.samples += 1; + activity.maxConnections = Math.max(activity.maxConnections, Number(sample.connections)); + activity.maxActive = Math.max(activity.maxActive, Number(sample.active)); + activity.maxWaitingLocks = Math.max(activity.maxWaitingLocks, Number(sample.waitingLocks)); + } catch { + activity.failures += 1; + } + await new Promise((resolve) => setTimeout(resolve, 10)); + } + }; + const activityPromise = sampleActivity(); + + const histogram = monitorEventLoopDelay({ resolution: 20 }); + histogram.enable(); + const cpuStart = process.cpuUsage(); + const wallStartNs = process.hrtime.bigint(); + let maxRssBytes = process.memoryUsage().rss; + const generalTransactionMs: number[] = []; + const monthlyTransactionMs: number[] = []; + const publicationMs: number[] = []; + let processedGenerals = 0; + let processedMonths = 0; + let runtime: Awaited> | null = null; + let startYearMonth: string | null = null; + let endYearMonth: string | null = null; + let initialGeneralCount: number | null = null; + let finalGeneralCount: number | null = null; + let runError: unknown; + + try { + runtime = await createTurnDaemonRuntime({ + // Omitting profileName deliberately disables the Gateway admin-action + // consumer. The load schema contains game tables only; the scoped + // profile value still isolates the lease and Redis channel. + profile: options.config.isolation.profileName, + databaseUrl, + redisUrl, + gameClockMode: 'manual', + enableDatabaseFlush: true, + enableLeaseHeartbeat: true, + databaseTransactionTimeoutMs: 30_000, + }); + const initialState = runtime.world.getState(); + initialGeneralCount = runtime.world.listGenerals().length; + startYearMonth = `${initialState.currentYear}-${String(initialState.currentMonth).padStart(2, '0')}`; + const tickMinutes = Math.max(1, Math.round(initialState.tickSeconds / 60)); + const boundary = getNextTickTime(initialState.lastTurnTime, tickMinutes); + let checkpoint: TurnCheckpoint | undefined = await runtime.stateStore.loadCheckpoint(); + + const execute = async (target: Date, maxGenerals: number): Promise => { + const started = performance.now(); + const result = await runtime!.stateManager.transaction(async () => { + await runtime!.stateStore.advanceGameClockTo(target, new Date()); + const next = await runtime!.processor.run( + target, + { budgetMs: 30_000, maxGenerals, catchUpCap: 1 }, + checkpoint + ); + await runtime!.stateStore.saveLastTurnTime(new Date(next.lastTurnTime)); + await runtime!.stateStore.saveCheckpoint(next.checkpoint); + await runtime!.hooks?.flushChanges?.(next); + return next; + }); + const transactionMs = performance.now() - started; + checkpoint = result.checkpoint; + maxRssBytes = Math.max(maxRssBytes, process.memoryUsage().rss); + const publishStarted = performance.now(); + await runtime!.hooks?.publishEvents?.(result); + publicationMs.push(performance.now() - publishStarted); + if (result.processedTurns > 0) monthlyTransactionMs.push(transactionMs); + else generalTransactionMs.push(transactionMs); + processedGenerals += result.processedGenerals; + processedMonths += result.processedTurns; + return result; + }; + + while (true) { + const nextGeneral = await runtime.stateStore.loadNextGeneralTurnTime(); + if (!nextGeneral || nextGeneral.getTime() >= boundary.getTime()) break; + const result = await execute(includeSubMillisecondGameTick(nextGeneral), 1); + if (result.processedGenerals !== 1 || result.processedTurns !== 0) { + throw new Error( + `chronological turn-flush run expected one general and zero months; got ${result.processedGenerals} generals and ${result.processedTurns} months` + ); + } + } + + const monthly = await execute(boundary, 200); + if (monthly.processedTurns !== 1) { + throw new Error('turn-flush measurement did not cross exactly one monthly boundary'); + } + const finalState = runtime.world.getState(); + finalGeneralCount = runtime.world.listGenerals().length; + endYearMonth = `${finalState.currentYear}-${String(finalState.currentMonth).padStart(2, '0')}`; + } catch (error) { + runError = error; + } finally { + try { + await runtime?.close(); + } catch (error) { + runError ??= error; + } + sampling = false; + try { + await activityPromise; + } catch (error) { + runError ??= error; + } + } + if (runError !== undefined) { + histogram.disable(); + await observer.disconnect().catch(() => undefined); + throw runError; + } + if ( + startYearMonth === null || + endYearMonth === null || + initialGeneralCount === null || + finalGeneralCount === null + ) { + histogram.disable(); + await observer.disconnect().catch(() => undefined); + throw new Error('turn-flush measurement completed without a full result'); + } + + let afterStats: DatabaseStatsRow; + try { + afterStats = await readDatabaseStats(observer.prisma); + } catch (error) { + histogram.disable(); + await observer.disconnect().catch(() => undefined); + throw error; + } + await observer.disconnect(); + histogram.disable(); + const elapsedMs = Number(process.hrtime.bigint() - wallStartNs) / 1_000_000; + const cpu = process.cpuUsage(cpuStart); + const cpuMs = (cpu.user + cpu.system) / 1_000; + const fromNs = (value: number): number => (Number.isFinite(value) ? round(value / 1_000_000) : 0); + + return { + version: 1, + fixture: { + name: options.config.name, + fixtureSha256: fixture.fixtureSha256, + capacity: options.config.capacity, + }, + mode: 'chronological-one-general-per-transaction-plus-month-boundary', + startYearMonth, + endYearMonth, + elapsedMs: round(elapsedMs), + throughput: { + generalTurnsPerSecond: round(processedGenerals / Math.max(elapsedMs / 1_000, 0.001)), + processedGenerals, + processedMonths, + }, + population: { + initialGenerals: initialGeneralCount, + finalGenerals: finalGeneralCount, + generalDelta: finalGeneralCount - initialGeneralCount, + }, + latencyMs: { + generalTransaction: summarizeDistribution(generalTransactionMs), + monthlyTransaction: summarizeDistribution(monthlyTransactionMs), + redisPublication: summarizeDistribution(publicationMs), + }, + postgres: { + statsScope: 'database-wide-including-observer-sampler', + statsDelta: subtractDatabaseStats(beforeStats, afterStats), + activity, + }, + process: { + cpuPercentOfOneCore: round((cpuMs / Math.max(elapsedMs, 1)) * 100), + maxRssBytes, + eventLoopLagMs: { + min: fromNs(histogram.min), + max: fromNs(histogram.max), + mean: fromNs(histogram.mean), + p50: fromNs(histogram.percentile(50)), + p95: fromNs(histogram.percentile(95)), + p99: fromNs(histogram.percentile(99)), + }, + }, + measuredAt: new Date().toISOString(), + }; +}; diff --git a/tools/load-tests/test/config.test.ts b/tools/load-tests/test/config.test.ts index f82d2c32..97fd67ab 100644 --- a/tools/load-tests/test/config.test.ts +++ b/tools/load-tests/test/config.test.ts @@ -6,6 +6,10 @@ import test from 'node:test'; import { assertRuntimeMetadataFinalized, canonicalJson, expandWeightedOperations, loadTokens, validateLoadConfig } from '../src/config.js'; const samplePath = new URL('../config/300-users-900-npcs-5m.json', import.meta.url); +const oneMinuteProfilePaths = [ + new URL('../config/nya-10-users-800-npcs-1m.json', import.meta.url), + new URL('../config/pya-10-users-800-npcs-1m.json', import.meta.url), +]; void test('the 300 viewer, 900 NPC, five-minute sample validates', async () => { const config = validateLoadConfig(JSON.parse(await readFile(samplePath, 'utf8'))); @@ -18,6 +22,24 @@ void test('the 300 viewer, 900 NPC, five-minute sample validates', async () => { assert.deepEqual(new Set(config.phases.map((phase) => phase.kind)), new Set(['idle', 'own', 'global', 'mixed'])); }); +void test('nya and pya one-minute profiles use distinct database and Redis isolation', async () => { + const configs = await Promise.all( + oneMinuteProfilePaths.map(async (configPath) => + validateLoadConfig(JSON.parse(await readFile(configPath, 'utf8'))) + ) + ); + assert.deepEqual( + configs.map((config) => config.capacity), + [ + { authenticatedViewers: 10, npcGenerals: 800, humanGenerals: 10, turnIntervalMs: 60_000 }, + { authenticatedViewers: 10, npcGenerals: 800, humanGenerals: 10, turnIntervalMs: 60_000 }, + ] + ); + assert.equal(new Set(configs.map((config) => config.isolation.postgresSchema)).size, 2); + assert.equal(new Set(configs.map((config) => config.isolation.redisDatabase)).size, 2); + assert.equal(new Set(configs.map((config) => config.isolation.profileName)).size, 2); +}); + void test('validation rejects public, non-allowlisted, and mutating targets', async () => { const raw = JSON.parse(await readFile(samplePath, 'utf8')) as Record; raw.target.publicProfile = true;