[SERVER-19562] Restore error: insertion error with out-of-memory condition Created: 23/Jul/15  Updated: 14/Oct/15  Resolved: 30/Jul/15

Status: Closed
Project: Core Server
Component/s: Admin, WiredTiger
Affects Version/s: 3.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Andrey Kostin Assignee: Ramon Fernandez Marina
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Gentoo, Docker 1.7.0


Attachments: HTML File Dockerfile     File mongod.conf    
Issue Links:
Duplicate
duplicates SERVER-17386 Cursor cache causes excessive memory ... Closed
Related
related to TOOLS-939 Error restoring database "insertion e... Closed
Operating System: Linux
Participants:

 Description   

Failed to mongorestore database. MongoDB is running in docker container with memory constraint 4220MB. WiredTiger storage cacheSizeGB is set to 4. Tried to restore backup with 120GB in GridFS and failed:

2015-07-23T23:48:35.968+0300    [##############..........]  anydb.fs.chunks  69.2 GB/117.9 GB  (58.7%)
2015-07-23T23:48:36.395+0300    Failed: restore error: anydb.fs.chunks: error restoring from fs.chunks.bson: insertion error: EOF

2015-07-23T20:47:30.965+0000 I WRITE    [conn4] insert anydb.fs.chunks ninserted:211 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { w: 1 } }, Collection: { acquireCount: { w: 1 } } } 245ms
2015-07-23T20:47:31.381+0000 I WRITE    [conn4] insert anydb.fs.chunks ninserted:207 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { w: 1 } }, Collection: { acquireCount: { w: 1 } } } 268ms
2015-07-23T20:48:02.582+0000 E STORAGE  [conn4] WiredTiger (12) [1437684466:622716][1:0x7f9014b34700], session.commit_transaction: journal/WiredTigerLog.0000000707 write error: failed to write 217600 bytes at offset 64507904: Cannot allocate memory
2015-07-23T20:48:02.658+0000 I -        [conn4] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 305
2015-07-23T20:48:35.250+0000 E STORAGE  WiredTiger (12) [1437684512:848464][1:0x7f9017d3c700], file:anydb/collection/9--8126075289585441344.wt, session.checkpoint: anydb/collection/9--8126075289585441344.wt write error: failed to write 196608 bytes at offset 73760747520: Cannot allocate memory
2015-07-23T20:48:35.250+0000 E STORAGE  WiredTiger (12) [1437684515:250525][1:0x7f9017d3c700], checkpoint-server: checkpoint server error: Cannot allocate memory
2015-07-23T20:48:35.250+0000 E STORAGE  WiredTiger (-31804) [1437684515:250560][1:0x7f9017d3c700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-07-23T20:48:35.250+0000 I -        Fatal Assertion 28558
2015-07-23T20:48:35.682+0000 I CONTROL  
 0xf605f9 0xf09361 0xeece61 0xd893ea 0x1395d89 0x1395f45 0x13963e4 0x132b363 0x7f901bd3d182 0x7f901ae3defd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B605F9"},{"b":"400000","o":"B09361"},{"b":"400000","o":"AECE61"},{"b":"400000","o":"9893EA"},{"b":"400000","o":"F95D89"},{"b":"400000","o":"F95F45"},{"b":"400000","o":"F963E4"},{"b":"400000","o":"F2B363"},{"b":"7F901BD35000","o":"8182"},{"b":"7F901AD43000","o":"FAEFD"}],"processInfo":{ "mongodbVersion" : "3.0.4", "gitVersion" : "0481c958daeb2969800511e7475dc66986fa9ed5", "uname" : { "sysname" : "Linux", "release" : "3.17.7-gentoo", "version" : "#2 SMP Sun Dec 28 23:21:45 EET 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFA0ED4000", "elfType" : 3 }, { "b" : "7F901BD35000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F901BB2D000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F901B929000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F901B625000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F901B31F000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F901B109000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F901AD43000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F901BF53000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf605f9]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf09361]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xeece61]
 mongod(+0x9893EA) [0xd893ea]
 mongod(__wt_eventv+0x489) [0x1395d89]
 mongod(__wt_err+0x95) [0x1395f45]
 mongod(__wt_panic+0x24) [0x13963e4]
 mongod(+0xF2B363) [0x132b363]
 libpthread.so.0(+0x8182) [0x7f901bd3d182]
 libc.so.6(clone+0x6D) [0x7f901ae3defd]
-----  END BACKTRACE  -----
2015-07-23T20:48:35.682+0000 I -        
 
***aborting after fassert() failure



 Comments   
Comment by Ramon Fernandez Marina [ 30/Jul/15 ]

lisio, we released 3.0.5 earlier this week, which replaces all the 3.0.5-rcX release candidates. I'm going to mark this ticket as a duplicate of SERVER-17386 and close it. If during your testing with 3.0.5 the problem reappears please let us know so we can reopen this ticket and investigate further.

Regards,
Ramón.

Comment by Andrey Kostin [ 24/Jul/15 ]

During the second run with the same settings it's just worked without problems, so now I'm now quite sure that testing with 3.0.5-rc2 will be informative.

Comment by Andrey Kostin [ 23/Jul/15 ]

Is it a problem with RAM? A little bit confused with the following line (total server RAM is less than 73GB):

write error: failed to write 196608 bytes at offset 73760747520: Cannot allocate memory

Right now I'm trying to do the same for the second time in order to reproduce the issue using the same resources. After that I'll try 3.0.5-rc2 with cacheSizeGB:1 as you suggested.

Comment by Ramon Fernandez Marina [ 23/Jul/15 ]

Hi lisio, this could be the same situation described in SERVER-17386, or a simple case of not having enough resources, or a combination of both. If you only have 4220MB available, setting the WT cache size to 4GB may easily trigger the out of memory condition that you're seeing.

You may want to download 3.0.5-rc2, released earlier today, which contains a fix for SERVER-17386. Even then, leaving so little memory for the system and mongod may not prevent the out of memory issue that you're running into. What I'd recommend is that you try 3.0.5-rc2 with a 1GB cache for WT – could you please try this and report back to make sure there are no other issues lurking?

Thanks,
Ramón.

Generated at Thu Feb 08 03:51:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.