[SERVER-16062] Segfault in shell on Ordered Bulk Insert/Find-Update with Large amount of docs Created: 10/Nov/14  Updated: 14/Nov/14  Resolved: 14/Nov/14

Status: Closed
Project: Core Server
Component/s: Shell, Write Ops
Affects Version/s: 2.6.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Mark Garrett Assignee: Unassigned
Resolution: Done Votes: 0
Labels: segfault
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OSX 10.10; iMac (27-inch, Late 2009); 2.8 GHz Intel Core i7; 8 GB 1067 MHz DDR3


Attachments: PNG File memoryScreencap.png    
Operating System: ALL
Steps To Reproduce:

Using an empty collection with no indexes, try:

> var ordered = db.mars.initializeOrderedBulkOp();
> for (u=0; u < 300; u++) {
...     var id = new ObjectId();
...     for (m=1; m <=12; m++) {
...         ordered.insert({_id:'2014-' + m + '-FACILITY2-' + id.str, admins:[]});
...         for (med=1; med <= 30; med++) {
...             var medID = new ObjectId();
...             for (ad = 1; ad <= 40; ad++) {
...                 ordered.find({_id:"2014-" + m + "-FACILITY2-" + id.str}).update({$push:{"admins":{m:medID, sd:new ISODate(), c:"A", u:ObjectId(), ad: new ISODate(), v:{pulse: 66, bp: "120/80"}}}});
...             }
...         }
...     }
... }
ordered.execute();

This will attempt to create 4.3 million inserts/updates. This will crash before execute() step before machine runs out of memory.

Returns:

#
# Fatal error in CALL_AND_RETRY_2
# Allocation failed - process out of memory
#
 
2014-11-10T15:58:58.256-0600 mongo got signal 6 (Abort trap: 6), stack trace: 
2014-11-10T15:58:58.258-0600 0x106a8074a 0x1068b18d0 0x7fff8dc9ef1a 0x0 0x7fff8b021b73 0x106cf9519 0x106b68d79 0x106b4b020 0x106b23046 0x106ba9edd 0x106bbfd89 0x106bbf9e6 0x106b705a0 0x106b6ef1e 0x106cc1c89 0x106d3ba62 0x353b7c906362 0x353b7c90d1f1 0x353b7c972999 
 0   mongo                               0x0000000106a8074a _ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE + 58
 1   mongo                               0x00000001068b18d0 _Z12quitAbruptlyi + 272
 2   libsystem_platform.dylib            0x00007fff8dc9ef1a _sigtramp + 26
 3   ???                                 0x0000000000000000 0x0 + 0
 4   libsystem_c.dylib                   0x00007fff8b021b73 abort + 129
 5   mongo                               0x0000000106cf9519 _ZN2v88internal2OS5AbortEv + 9
 6   mongo                               0x0000000106b68d79 _Z9API_FatalPKcS0_z + 185
 7   mongo                               0x0000000106b4b020 _ZN2v8L24DefaultFatalErrorHandlerEPKcS1_ + 112
 8   mongo                               0x0000000106b23046 _ZN2v88internal2V823FatalProcessOutOfMemoryEPKcb + 566
 9   mongo                               0x0000000106ba9edd _ZN2v88internal7Factory19NewTypeFeedbackInfoEv + 365
 10  mongo                               0x0000000106bbfd89 _ZN2v88internal17FullCodeGenerator24PopulateTypeFeedbackInfoENS0_6HandleINS0_4CodeEEE + 25
 11  mongo                               0x0000000106bbf9e6 _ZN2v88internal17FullCodeGenerator8MakeCodeEPNS0_15CompilationInfoE + 486
 12  mongo                               0x0000000106b705a0 _ZN2v88internalL12GenerateCodeEPNS0_15CompilationInfoE + 240
 13  mongo                               0x0000000106b6ef1e _ZN2v88internal8Compiler11CompileLazyEPNS0_15CompilationInfoE + 398
 14  mongo                               0x0000000106cc1c89 _ZN2v88internal10JSFunction11CompileLazyENS0_6HandleIS1_EENS0_18ClearExceptionFlagE + 377
 15  mongo                               0x0000000106d3ba62 _ZN2v88internal19Runtime_LazyCompileENS0_9ArgumentsEPNS0_7IsolateE + 50
 16  ???                                 0x0000353b7c906362 0x0 + 58529609180002
 17  ???                                 0x0000353b7c90d1f1 0x0 + 58529609208305
 18  ???                                 0x0000353b7c972999 0x0 + 58529609623961

Participants:

 Description   

Using a script to generate large Bulk.insert()/Bulk.updateOne() operations can cause out of memory segfault.



 Comments   
Comment by Ramon Fernandez Marina [ 14/Nov/14 ]

This is a known issue with V8 memory limitations. To work around this limitation you may want to consider building smaller sets of bulk operations.

Comment by Mark Garrett [ 11/Nov/14 ]

Update: I tried again this morning, and got a signal 11 Segfault for the same javascript. I've attached a screenshot of the memory situation on the server right before mongo shell crashed out. Yesterday the faults seemed to sometimes be a signal 6, sometimes a signal 11.

2014-11-11T09:56:07.270-0600 mongo got signal 11 (Segmentation fault: 11), stack trace: 
2014-11-11T09:56:07.271-0600 0x109aa774a 0x1098d88d0 0x7fff8dc9ef1a 0x109bfaaba 0x109c0f4bb 0x109bf6e57 0x109bf62e9 0x109bf4cbf 0x109bf46bc 0x109bd8456 0x109d6263c 0x3a1438006362 0x3a143800cc46 0x3a143807e620 0x3a143800cfa7 
 0   mongo                               0x0000000109aa774a _ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE + 58
 1   mongo                               0x00000001098d88d0 _Z12quitAbruptlyi + 272
 2   libsystem_platform.dylib            0x00007fff8dc9ef1a _sigtramp + 26
 3   mongo                               0x0000000109bfaaba _ZN2v88internal4Heap18AllocateFixedArrayEi + 26
 4   mongo                               0x0000000109c0f4bb _ZN2v88internal19FlexibleBodyVisitorINS0_17NewSpaceScavengerENS0_8JSObject14BodyDescriptorEiE16VisitSpecializedILi24EEEiPNS0_3MapEPNS0_10HeapObjectE + 91
 5   mongo                               0x0000000109bf6e57 _ZN2v88internal4Heap10DoScavengeEPNS0_13ObjectVisitorEPh + 119
 6   mongo                               0x0000000109bf62e9 _ZN2v88internal4Heap8ScavengeEv + 1705
 7   mongo                               0x0000000109bf4cbf _ZN2v88internal4Heap24PerformGarbageCollectionENS0_16GarbageCollectorEPNS0_8GCTracerE + 447
 8   mongo                               0x0000000109bf46bc _ZN2v88internal4Heap14CollectGarbageENS0_15AllocationSpaceENS0_16GarbageCollectorEPKcS5_ + 412
 9   mongo                               0x0000000109bd8456 _ZN2v88internal7Factory11NewJSObjectENS0_6HandleINS0_10JSFunctionEEENS0_13PretenureFlagE + 118
 10  mongo                               0x0000000109d6263c _ZN2v88internal17Runtime_NewObjectENS0_9ArgumentsEPNS0_7IsolateE + 492
 11  ???                                 0x00003a1438006362 0x0 + 63858513306466
 12  ???                                 0x00003a143800cc46 0x0 + 63858513333318
 13  ???                                 0x00003a143807e620 0x0 + 63858513798688
 14  ???                                 0x00003a143800cfa7 0x0 + 63858513334183

Generated at Thu Feb 08 03:39:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.