[SERVER-6757] map reduce double free when dropping source collection during yield Created: 13/Aug/12  Updated: 11/Jul/16  Resolved: 21/Aug/12

Status: Closed
Project: Core Server
Component/s: MapReduce, Stability
Affects Version/s: None
Fix Version/s: 2.2.0-rc2, 2.3.0

Type: Bug Priority: Major - P3
Reporter: Aaron Staple Assignee: Aaron Staple
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-6818 audit map / reduce yield recovery cases Closed
Operating System: ALL
Participants:

 Description   

The auto_ptr<ClientCursor> holdCursor is not released properly on a yield recovery failure (for example when the source collection is dropped). Using a ClientCursor::Holder is one way to handle this case safely. There may be other similarly unsafe auto_ptr<ClientCursor> variables in mr.cpp.

Test

c = db.c;
c.drop();
 
for( i = 0; i < 1000; ++i ) {
    c.save( {} );
}
db.getLastError();
 
s = startParallelShell( "sleep( 1000 ); db.c.drop();" );
 
c.mapReduce( function() { sleep( 3 ); }, function() {}, { out:{ inline:1 } } );
 
s();

Output

Mon Aug 13 10:53:21 [conn5] CMD: drop test.c
Mon Aug 13 10:53:21 [conn4]   warning assertion failure false src/mongo/db/clientcursor.cpp 341
0x106c9b395 0x106e6590b 0x106d788ea 0x1071585a1 0x107195355 0x106ab64a9 0x10717819e 0x107179f20 0x10717b380 0x107120cc5 0x10712179b 0x106fe66f2 0x106feb5f0 0x1069006e7 0x106d44563 0x106d45bc0 0x106d45c22 0x106d45c4d 0x106d39e69 0x108b708bf 
Mon Aug 13 10:53:21 [conn5] end connection 127.0.0.1:56302 (1 connection now open)
 0   mongod                              0x0000000106c9b395 _ZN5mongo15printStackTraceERSo + 37
 1   mongod                              0x0000000106e6590b _ZN5mongo10logContextEPKc + 123
 2   mongod                              0x0000000106d788ea _ZN5mongo9wassertedEPKcS1_j + 538
 3   mongod                              0x00000001071585a1 _ZN5mongo12ClientCursorD1Ev + 57
 4   mongod                              0x0000000107195355 _ZNSt8auto_ptrIN5mongo12ClientCursorEED1Ev + 45
 5   mongod                              0x0000000106ab64a9 _ZN5mongo2mr16MapReduceCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 12421
 6   mongod                              0x000000010717819e _ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb + 142
 7   mongod                              0x0000000107179f20 _ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 4608
 8   mongod                              0x000000010717b380 _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 1472
 9   mongod                              0x0000000107120cc5 _ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 117
 10  mongod                              0x000000010712179b _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 1595
 11  mongod                              0x0000000106fe66f2 _ZN5mongoL13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 418
 12  mongod                              0x0000000106feb5f0 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1136
 13  mongod                              0x00000001069006e7 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 261
 14  mongod                              0x0000000106d44563 _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 2275
 15  mongod                              0x0000000106d45bc0 _ZN5boost3_bi5list1INS0_5valueIPN5mongo13MessagingPortEEEEclIPFvS5_ENS0_5list0EEEvNS0_4typeIvEERT_RT0_i + 78
 16  mongod                              0x0000000106d45c22 _ZN5boost3_bi6bind_tIvPFvPN5mongo13MessagingPortEENS0_5list1INS0_5valueIS4_EEEEEclEv + 92
 17  mongod                              0x0000000106d45c4d _ZN5boost6detail11thread_dataINS_3_bi6bind_tIvPFvPN5mongo13MessagingPortEENS2_5list1INS2_5valueIS6_EEEEEEE3runEv + 37
 18  mongod                              0x0000000106d39e69 thread_proxy + 169
 19  libsystem_c.dylib                   0x0000000108b708bf _pthread_start + 335
mongod(71587,0x10aaa0000) malloc: *** error for object 0x7fd7c9d09640: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Mon Aug 13 10:53:21 Got signal: 6 (Abort trap: 6).
 
Mon Aug 13 10:53:21 Backtrace:
0x106c9b395 0x1068ebb3e 0x108bc4cfa 0 0x108b63a7a 0x108bc284c 0x108911702 0x108911740 0x107158b18 0x107195355 0x106ab64a9 0x10717819e 0x107179f20 0x10717b380 0x107120cc5 0x10712179b 0x106fe66f2 0x106feb5f0 0x1069006e7 0x106d44563 
 0   mongod                              0x0000000106c9b395 _ZN5mongo15printStackTraceERSo + 37
 1   mongod                              0x00000001068ebb3e _ZN5mongo10abruptQuitEi + 446
 2   libsystem_c.dylib                   0x0000000108bc4cfa _sigtramp + 26
 3   ???                                 0x0000000000000000 0x0 + 0
 4   libsystem_c.dylib                   0x0000000108b63a7a abort + 143
 5   libsystem_c.dylib                   0x0000000108bc284c free + 389
 6   libstdc++.6.dylib                   0x0000000108911702 _ZNSs4_Rep10_M_disposeERKSaIcE + 60
 7   libstdc++.6.dylib                   0x0000000108911740 _ZNSsD2Ev + 44
 8   mongod                              0x0000000107158b18 _ZN5mongo12ClientCursorD1Ev + 1456
 9   mongod                              0x0000000107195355 _ZNSt8auto_ptrIN5mongo12ClientCursorEED1Ev + 45
 10  mongod                              0x0000000106ab64a9 _ZN5mongo2mr16MapReduceCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 12421
 11  mongod                              0x000000010717819e _ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb + 142
 12  mongod                              0x0000000107179f20 _ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 4608
 13  mongod                              0x000000010717b380 _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 1472
 14  mongod                              0x0000000107120cc5 _ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 117
 15  mongod                              0x000000010712179b _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 1595
 16  mongod                              0x0000000106fe66f2 _ZN5mongoL13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 418
 17  mongod                              0x0000000106feb5f0 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1136
 18  mongod                              0x00000001069006e7 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 261
 19  mongod                              0x0000000106d44563 _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 2275



 Comments   
Comment by auto [ 22/Aug/12 ]

Author:

{u'date': u'2012-08-21T16:54:24-07:00', u'email': u'aaron@10gen.com', u'name': u'Aaron'}

Message: SERVER-6757 Map reduce collection drop test.
Branch: master
https://github.com/mongodb/mongo/commit/1bac89b6f3b79507cac2e3272f03789467286e29

Comment by auto [ 21/Aug/12 ]

Author:

{u'date': u'2012-08-21T15:50:29-07:00', u'email': u'aaron@10gen.com', u'name': u'Aaron'}

Message: SERVER-6757 Store holdCursor in a ClientCursor::Holder to prevent a double free on failed yield recovery.
Branch: v2.2
https://github.com/mongodb/mongo/commit/45d66f6b12d8e6faee340c915340256ae1f0a221

Comment by auto [ 21/Aug/12 ]

Author:

{u'date': u'2012-08-21T15:50:29-07:00', u'email': u'aaron@10gen.com', u'name': u'Aaron'}

Message: SERVER-6757 Store holdCursor in a ClientCursor::Holder to prevent a double free on failed yield recovery.
Branch: master
https://github.com/mongodb/mongo/commit/4cf1628f59deafd1c608aaf0a9930ca68430c8dd

Comment by auto [ 21/Aug/12 ]

Author:

{u'date': u'2012-08-21T13:57:28-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}

Message: SERVER-6757 - need to clear holdCursor when collection dropped during yield
Branch: v2.2
https://github.com/mongodb/mongo/commit/3f518bf76c1db505449bf8b14b8afd084736006e

Comment by auto [ 21/Aug/12 ]

Author:

{u'date': u'2012-08-21T13:57:28-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}

Message: SERVER-6757 - need to clear holdCursor when collection dropped during yield
Branch: master
https://github.com/mongodb/mongo/commit/084125a26dfba0313a016e3c88b10f0b33c35737

Comment by Aaron Staple [ 13/Aug/12 ]

This appears to be new in 2.1. There was no 'holdCursor' in 2.0.

Comment by Aaron Staple [ 13/Aug/12 ]

Tentatively scheduled for 2.2.0-rc1.

Generated at Thu Feb 08 03:12:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.