-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Labels:None
A customer has reported a failure where __wt_txn_recover allocated an unreasonable amount of memory, 17GB, on a host with a total of 15GB of memory. This allocation fails and as a result WiredTiger cannot recover.
In the reported scenario MongoDB would then attempt to restart and immediately hits the same issue with the same amount of memory being allocated. The fix in this situation was to resync the problematic node.
Logs:
{"t":{"$date":"2021-07-22T20:03:08.243+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":12,"message":"[1626984188:241896][325102:0x7f4cb0f4cbc0], txn-recover: __realloc_func, 124: memory allocation of 17124907440 bytes failed: Cannot allocate memory"}} {"t":{"$date":"2021-07-22T20:03:09.066+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":12,"message":"[1626984189:66653][325102:0x7f4cb0f4cbc0], txn-recover: __wt_txn_recover, 1045: Recovery failed: Cannot allocate memory"}}
{"t":{"$date":"2021-07-22T20:03:09.070+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":0,"message":"[1626984189:70767][325102:0x7f4cb0f4cbc0], connection: __wt_cache_destroy, 364: cache server: exiting with 34 pages in memory and 0 pages evicted"}}{"t":{"$date":"2021-07-22T20:03:09.070+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":0,"message":"[1626984189:70802][325102:0x7f4cb0f4cbc0], connection: __wt_cache_destroy, 367: cache server: exiting with 483555 image bytes in memory"}}