[SERVER-9117] Mongo segfault on mapreduce - repeatable every time Created: 25/Mar/13  Updated: 10/Dec/14  Resolved: 29/Mar/13

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: 2.4.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ian Patton Assignee: Ben Becker
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

osx


Issue Links:
Duplicate
duplicates SERVER-9051 v8 heap allocation failure can lead t... Closed
Operating System: ALL
Steps To Reproduce:

1 map=function(){
2 var dealer_identifier = this.source+"_"+this.source_dealer_id
3 emit(dealer_identifier, {cars: [

{source: this.source, source_id: this.source_id}

] });
4 }
5
6 reduce=function(key,values){
7 reducedValue =

{cars: [] }

;
8 values.forEach(function(value){
9 for(x in value.cars)

{ 10 reducedValue.cars.push(value.cars[x]); 11 }

12 });
13 return reducedValue;
14 }
15
16 db.getCollection("atc_and_cdc_reduced").mapReduce(map, reduce,

{out: "import_data_dealer_cars"}

);

Participants:

 Description   

I have a mapreduce job that causes a segfault every time it runs.

The map seems to work fine, but the reduce causes the following error, which then corrupts the database files

map=function(){
  var dealer_identifier = this.source+"_"+this.source_dealer_id
  emit(dealer_identifier, {cars: [ {source: this.source, source_id: this.source_id} ] });
}
 
reduce=function(key,values){
  reducedValue = {cars: [] };
  values.forEach(function(value){
    for(x in value.cars){
      reducedValue.cars.push(value.cars[x]);
    }
  });
  return reducedValue;
}
 
db.getCollection("atc_and_cdc_reduced").mapReduce(map, reduce, {out: "import_data_dealer_cars"});



 Comments   
Comment by Ben Becker [ 29/Mar/13 ]

Hi Ian,

I believe the cause of this issue was v8 running out of memory, which can result in a segfault (see SERVER-9051). The reduce function was likely hitting the memory limit due to SERVER-9186. I haven't been able to reproduce with the attached script; likely due to requirements of the dataset.

Would you be able to test this on version 2.4.2 when it is released? If so, please reopen this ticket and let us know.

Thanks,
Ben

Comment by Ben Becker [ 25/Mar/13 ]

Hi Ian,

Which field did you have to clone, exactly? The source_id from the reduce function's 'values' argument?

When you weren't cloning the object, how did the corruption manifest itself?

Thanks,
Ben

Comment by Ian Patton [ 25/Mar/13 ]

cloning the hash does indeed fix the problem. Maybe you guys should add a safeguard against crashing the mongod, but otherwise this was a typical javascript issue.

Ian

Comment by Ben Becker [ 25/Mar/13 ]

Hi Ian,

Thanks for the report; I'm looking into this now.

What are the symptoms of database file corruption that you found?
Can you also provide a small sample of the dataset which caused this?

Thanks,
Ben

Comment by Ian Patton [ 25/Mar/13 ]

I think I just needed to clone the hash instead of pushing a reference to it. I will see if that resolves the issue and follow up.

Comment by Ian Patton [ 25/Mar/13 ]

From the log:

Mon Mar 25 10:44:16.001 [conn13] 		M/R: (1/3) Emit Progress: 4140100/4221729	98%
Mon Mar 25 10:44:19.011 [conn13] 		M/R: (1/3) Emit Progress: 4188300/4221729	99%
Mon Mar 25 10:44:28.456 Invalid access at address: 0 from thread: conn13
 
Mon Mar 25 10:44:28.456 Got signal: 11 (Segmentation fault: 11).
 
Mon Mar 25 10:44:28.497 Backtrace:
0x1006d95bb 0x10000197b 0x100002031 0x7fff94a8794a 0x1008a7933 0x100894ee9 0x1007f596f 0x100676485 0x100676f2c 0x1006765cb 0x100677a60 0x100665a84 0x1001b75ab 0x1001be22c 0x1001af2d3 0x1001b5a42 0x1001c4c68 0x1001fc8b0 0x1001fdbca 0x1001fef70 
 0   mongod                              0x00000001006d95bb _ZN5mongo15printStackTraceERSo + 43
 1   mongod                              0x000000010000197b _ZN5mongo10abruptQuitEi + 987
 2   mongod                              0x0000000100002031 _ZN5mongo24abruptQuitWithAddrSignalEiP9__siginfoPv + 673
 3   libsystem_c.dylib                   0x00007fff94a8794a _sigtramp + 26
 4   mongod                              0x00000001008a7933 _ZN2v88internal4Heap21ProcessWeakReferencesEPNS0_18WeakObjectRetainerE + 1779
 5   mongod                              0x0000000100894ee9 _ZN2v88internal11GetPropertyENS0_6HandleINS0_6ObjectEEES3_ + 73
 6   mongod                              0x00000001007f596f _ZN2v86Object3GetENS_6HandleINS_5ValueEEE + 223
 7   mongod                              0x0000000100676485 _ZN5mongo7V8Scope9v8ToMongoEN2v86HandleINS1_6ObjectEEEi + 677
 8   mongod                              0x0000000100676f2c _ZN5mongo7V8Scope16v8ToMongoElementERNS_14BSONObjBuilderERKSsN2v86HandleINS5_5ValueEEEiPNS_7BSONObjE + 620
 9   mongod                              0x00000001006765cb _ZN5mongo7V8Scope9v8ToMongoEN2v86HandleINS1_6ObjectEEEi + 1003
 10  mongod                              0x0000000100677a60 _ZN5mongo7V8Scope9getObjectEPKc + 208
 11  mongod                              0x0000000100665a84 _ZN5mongo5Scope6appendERNS_14BSONObjBuilderEPKcS4_ + 948
 12  mongod                              0x00000001001b75ab _ZN5mongo2mr9JSReducer7_reduceERKSt6vectorINS_7BSONObjESaIS3_EERS3_Ri + 3051
 13  mongod                              0x00000001001be22c _ZN5mongo2mr9JSReducer11finalReduceERKSt6vectorINS_7BSONObjESaIS3_EEPNS0_9FinalizerE + 140
 14  mongod                              0x00000001001af2d3 _ZN5mongo2mr5State11finalReduceERSt6vectorINS_7BSONObjESaIS3_EE + 99
 15  mongod                              0x00000001001b5a42 _ZN5mongo2mr5State11finalReduceEPNS_5CurOpERNS_19ProgressMeterHolderE + 4290
 16  mongod                              0x00000001001c4c68 _ZN5mongo2mr16MapReduceCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 3800
 17  mongod                              0x00000001001fc8b0 _ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 48
 18  mongod                              0x00000001001fdbca _ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 2090
 19  mongod                              0x00000001001fef70 _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 672

Generated at Thu Feb 08 03:19:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.