-
Type:
Task
-
Resolution: Duplicate
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Not Applicable
-
Storage Engines
-
206.359
-
None
-
None
Issue Summary
On the disaggMmkAzureByok cluster (Azure BYOK, disaggregated storage, 9.1.0-rc1021), node shard-00-00 (zone use1-az6) took 86 seconds to install the startup checkpoint. The peer node shard-00-01 had no checkpoint to pick up and completed startup in 329ms.
The stall is entirely in KMIP decrypt against the local proxy at 127.0.0.1:35696, not in WiredTiger checkpoint pickup mechanics.
Timeline (shard-00-00, all UTC)
| Time | Event |
|---|---|
| 00:43:48.321 | Starting to install the startup checkpoint |
| 00:43:48.415 | WT: "Loading persisted crypt key: lsn=7685184299280105489" |
| 00:43:48.433 | Calling KMIP decrypt |
| 00:43:58.911 | Socket recv() timeout (10s) |
| 00:44:00.664 | KMIP operation failed (attempt 1/5, SocketException RECV_TIMEOUT) |
| 00:44:19.313 | Socket recv() timeout (19s) |
| 00:44:20.018 | KMIP operation failed (attempt 2/5) |
| 00:44:35.368 | Socket recv() timeout (15s) |
| 00:44:36.294 | KMIP operation failed (attempt 3/5) |
| 00:44:52.560 | Socket recv() timeout (16s) |
| 00:44:53.112 | KMIP operation failed (attempt 4/5) |
| 00:45:12.272 | KMIP decrypt succeeded |
| 00:45:14.089 | Finished startup recovery (51ms) |
| 00:45:14.384 | Finished installing the startup checkpoint (86063ms total) |
Key Facts
- Database size: 1.1 MB (database_size=1173584), 14 collections
- Actual checkpoint pickup work (apply_checkpoint_meta + catalog init + recovery): ~51ms
- KMIP decrypt retries: 4 failures across ~84 seconds from key_provider->load_key() at conn_layered_page_log.c:288
- Peer shard-00-01 skipped KMIP entirely ("no last completed checkpoint") — startup took 329ms
- Local KMIP proxy configured via kmip.port: 35696, kmip.serverName: "127.0.0.1"
Questions
- Why did the local KMIP proxy at 127.0.0.1:35696 repeatedly timeout during this node's startup but not the peer's?
- Is this a startup race between mongod and the KMIP proxy process?
- Should the KMIP retry logic have a shorter initial timeout or faster backoff to reduce stall duration?
- Can we add disagg_pick_up_checkpoint_time_startup to Grafana dashboards to catch this in the future?
Logs
Mongod logs for both shard-00-00 and shard-00-01 attached. FTDC tarballs were empty (no metrics captured).