-
Type:
Bug
-
Resolution: Declined
-
Priority:
Major - P3
-
None
-
Affects Version/s: 3.2.10
-
Component/s: None
-
None
-
Fully Compatible
-
ALL
-
None
-
3
-
None
-
None
-
None
-
None
-
None
-
None
I have a cluster with 9 shards ，each shard write QPS is 5k/s，the WT cacheSize is configured with 16GB。Mongod emory usage will keep growing over time，eventually leads to mongod OOM.
By serverStatus，found that wiredTiger stats reports only 13GB of cache used, but according to ps mongod is using more than 20GB or more
When a mongod restarted(killed by OOM)，get the mongod process status.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 70068 user_00 20 0 28.4g 19.3g 11724 S 17.3 30.9 3216:28 mongod
I can collect more information if needed.
wiredTiger cache：
{
"bytes belonging to page images in the cache" : 10253478724,
"bytes currently in the cache" : 13785399117,
"bytes not belonging to page images in the cache" : 3531920392,
"bytes read into cache" : NumberLong("8793441827502"),
"bytes written from cache" : NumberLong("6370331017071"),
"checkpoint blocked page eviction" : 1198232,
"eviction calls to get a page" : 851138375,
"eviction calls to get a page found queue empty" : 46568598,
"eviction calls to get a page found queue empty after locking" : 19944658,
"eviction currently operating in aggressive mode" : 0,
"eviction empty score" : 0,
"eviction server candidate queue empty when topping up" : 23092098,
"eviction server candidate queue not empty when topping up" : 22233159,
"eviction server evicting pages" : 130724295,
"eviction server slept, because we did not make progress with eviction" : 19705012,
"eviction server unable to reach eviction goal" : 3,
"eviction state" : 16,
"eviction walks abandoned" : 6521373,
"eviction worker thread evicting pages" : 604798543,
"failed eviction of pages that exceeded the in-memory maximum" : 543025,
"files with active eviction walks" : 0,
"files with new eviction walks started" : 73260089,
"hazard pointer blocked page eviction" : 28016,
"hazard pointer check calls" : 760851318,
"hazard pointer check entries walked" : 16319500537,
"hazard pointer maximum array length" : 17,
"in-memory page passed criteria to be split" : 319959,
"in-memory page splits" : 160950,
"internal pages evicted" : 23906542,
"internal pages split during eviction" : 5284,
"leaf pages split during eviction" : 220766,
"lookaside table insert calls" : 0,
"lookaside table remove calls" : 0,
"maximum bytes configured" : 17179869184,
"maximum page size at eviction" : 332557,
"modified pages evicted" : 178301222,
"modified pages evicted by application threads" : 19687859,
"overflow pages read into cache" : 0,
"overflow values cached in memory" : 0,
"page split during eviction deepened the tree" : 0,
"page written requiring lookaside records" : 0,
"pages currently held in the cache" : 507652,
"pages evicted because they exceeded the in-memory maximum" : 153690,
"pages evicted because they had chains of deleted items" : 1222974,
"pages evicted by application threads" : 23174903,
"pages queued for eviction" : 2428617936,
"pages queued for urgent eviction" : 731847,
"pages queued for urgent eviction during walk" : 762491,
"pages read into cache" : 490708559,
"pages read into cache requiring lookaside entries" : 0,
"pages requested from the cache" : 4883482550,
"pages seen by eviction walk" : 6736070390,
"pages selected for eviction unable to be evicted" : 85413953,
"pages walked for eviction" : 294676888895,
"pages written from cache" : 307273400,
"pages written requiring in-memory restoration" : 335501,
"percentage overhead" : 8,
"tracked bytes belonging to internal pages in the cache" : 653852348,
"tracked bytes belonging to leaf pages in the cache" : 13131546769,
"tracked dirty bytes in the cache" : 635490210,
"tracked dirty pages in the cache" : 13415,
"unmodified pages evicted" : 496733004
}
I restarted mongod with --setParameter heapProfilingEnabled=true, after a couple days got a heap-profile result，see details heap-profile-mongod.out