-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: 8.2.11
-
Component/s: None
-
None
-
Storage Engines - Transactions
-
ALL
-
SE Transactions - 2026-12-04
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Environment
- MongoDB 8.2.11 Community, Windows Server 2022, PSA replica set (primary, secondary, arbiter)
- Primary: 96 GB RAM, 12 cores, default WiredTiger cache (51 GB), journal and data on the same volume
- Workload: about 1,700 inserts, 2,000 updates and 3,900 finds per second, arriving in 16 s bursts every minute. Documents get frequent small updates ($v:2 diffs) and are upserted by _id. A TTL index removes about 16M documents per day.
Problem
After 4 to 6 weeks of uptime, the primary drops into a slow state and stays there until it is restarted or stepped down. It started recurring about every 3 months; since the TTL indexes were added, about every month. Observed on Oct 28-30 in FTDC and logs; restarted Oct 30, uptime then 42 days. First column with numbers is the slow state, next column is after restart of primary(so when everything is perfectly fine)
| Checkpoint duration (most recent time (msecs)) | 160,000-187,000 | 6,500-13,000 |
| Time spent on per-tree checkpoint work | 107-136 s | 2.7-6 s |
| Pages reconciled by checkpoint per second | 26 | 100 |
| maximum milliseconds spent in a reconciliation call | 2,431 | 78 |
| checkpoint blocked page eviction per second | 17.3 | 0.16 |
| Transaction IDs pinned by checkpoint (max) | ~289,000 | ~773 |
| config.transactions in cache (123 docs, 20 KB on disk) | up to 88 MB | 17 MB |
| time spent on the most recent linear walk of extents during first-fit (max) | 86 ms | 12 ms |
| Average read latency (opLatencies.reads) | 17 ms | 0.3 ms |
| Read working time (opWorkingTime.reads) | ~48 s/s | ~0.1 s/s |
What is not the cause
- The cache: about 40.8 of 51 GB used, application threads did no eviction, and no operation timed out waiting for cache space.
- Query plans: keys and documents examined per document returned stayed at 2.0.
- Ticket queues and disk: queue times under 0.05 ms, and checkpoint writes under 1 MB/s.
- CPU: 30-40% busy.
- Replication: lag under 15 s, and the written and applied optimes on the primary were equal.
What we observe
In the slow state, finds that read about 100 documents entirely from cache take about 113 ms, with no lock, ticket or disk waits. WiredTiger cursor and page work per operation is unchanged compared with the healthy state. Checkpoints run almost continuously and log "Checkpoint has been running for N seconds" dozens of times a day; the healthy state logs none.
Our reading: long checkpoints block eviction and pin old update chains, which makes reads and the next checkpoint slower, and the loop only breaks on restart. Possibly related: WT-18826 and WT-18765.
Available
FTDC from the primary (Oct 28 to Nov 1, covering the slow state, the restart and the recovery), the primary's mongod logs for Oct 28-31, and the secondary's logs for Oct 24-31.
Questions
- Is this a known issue on 8.2.x?
- Is there a fix, or a configuration that avoids it, short of restarting the primary every few weeks?
- is related to
-
WT-18765 History store dirty content accumulates unevictably during the checkpoint history store pass
-
- Open
-