[SERVER-13672] Segmentation Fault Constantly on OS X Created: 21/Apr/14  Updated: 10/Dec/14  Resolved: 23/Apr/14

Status: Closed
Project: Core Server
Component/s: Write Ops
Affects Version/s: 2.6.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: James Simpson Assignee: J Rassi
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-13516 Array updates on documents with more ... Closed
Participants:

 Description   

We currently run 2.4.10 in production and I develop with it locally on OS X with no issues. I upgraded to 2.6.0 and mongod crashes over and over. I tried deleting everything, reinstalling mongo and rebuilding my local database, but the issue persists. Downgrading back to 2.4.10 fixes the issue. Here's the most recent example:

2014-04-21T16:45:23.860-0500 [initandlisten] MongoDB starting : pid=24871 port=27017 dbpath=/Users/James/mongo 64-bit host=Jamess-MacBook-Pro.local
2014-04-21T16:45:23.861-0500 [initandlisten] db version v2.6.0
2014-04-21T16:45:23.861-0500 [initandlisten] git version: nogitversion
2014-04-21T16:45:23.861-0500 [initandlisten] build info: Darwin minimavericks.local 13.1.0 Darwin Kernel Version 13.1.0: Thu Jan 16 19:40:37 PST 2014; root:xnu-2422.90.20~2/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49
2014-04-21T16:45:23.861-0500 [initandlisten] allocator: tcmalloc
2014-04-21T16:45:23.861-0500 [initandlisten] options: { processManagement: { fork: true }, storage: { dbPath: "/Users/James/mongo" }, systemLog: { destination: "file", path: "/Users/James/mongo/mongodb.log" } }
2014-04-21T16:45:23.863-0500 [initandlisten] journal dir=/Users/James/mongo/journal
2014-04-21T16:45:23.864-0500 [initandlisten] recover begin
2014-04-21T16:45:23.864-0500 [initandlisten] recover lsn: 1985857
2014-04-21T16:45:23.864-0500 [initandlisten] recover /Users/James/mongo/journal/j._2
2014-04-21T16:45:23.878-0500 [initandlisten] recover skipping application of section seq:22951 < lsn:1985857
2014-04-21T16:45:23.879-0500 [initandlisten] recover skipping application of section seq:285957 < lsn:1985857
2014-04-21T16:45:23.879-0500 [initandlisten] recover skipping application of section seq:743487 < lsn:1985857
2014-04-21T16:45:23.879-0500 [initandlisten] recover skipping application of section seq:800987 < lsn:1985857
2014-04-21T16:45:23.879-0500 [initandlisten] recover skipping application of section seq:858577 < lsn:1985857
2014-04-21T16:45:23.880-0500 [initandlisten] recover skipping application of section seq:973107 < lsn:1985857
2014-04-21T16:45:23.880-0500 [initandlisten] recover skipping application of section seq:1030417 < lsn:1985857
2014-04-21T16:45:23.880-0500 [initandlisten] recover skipping application of section seq:1087937 < lsn:1985857
2014-04-21T16:45:23.880-0500 [initandlisten] recover skipping application of section seq:1145477 < lsn:1985857
2014-04-21T16:45:23.881-0500 [initandlisten] recover skipping application of section more...
2014-04-21T16:45:23.914-0500 [initandlisten] recover cleaning up
2014-04-21T16:45:23.914-0500 [initandlisten] removeJournalFiles
2014-04-21T16:45:23.928-0500 [initandlisten] recover done
2014-04-21T16:45:24.254-0500 [initandlisten] waiting for connections on port 27017
2014-04-21T16:45:27.749-0500 [initandlisten] connection accepted from 127.0.0.1:59753 #1 (1 connection now open)
2014-04-21T16:45:27.762-0500 [conn1] end connection 127.0.0.1:59753 (0 connections now open)
2014-04-21T16:45:27.764-0500 [initandlisten] connection accepted from 127.0.0.1:59754 #2 (1 connection now open)
2014-04-21T16:45:27.765-0500 [initandlisten] connection accepted from 127.0.0.1:59755 #3 (2 connections now open)
2014-04-21T16:45:27.765-0500 [initandlisten] connection accepted from 127.0.0.1:59756 #4 (3 connections now open)
2014-04-21T16:45:27.766-0500 [initandlisten] connection accepted from 127.0.0.1:59757 #5 (4 connections now open)
2014-04-21T16:45:27.766-0500 [initandlisten] connection accepted from 127.0.0.1:59758 #6 (5 connections now open)
2014-04-21T16:45:29.260-0500 [conn4] SEVERE: Invalid access at address: 0x211918acc2
2014-04-21T16:45:29.266-0500 [conn4] SEVERE: Got signal: 11 (Segmentation fault: 11).
Backtrace:0x10dd153ea 0x10dd14e86 0x10dd14f94 0x7fff8e78d5aa 0x10d62232a 0x10d61e78f 0x10d61e540 0x10d61e01f 0x10d93fe36 0x10d94aa8f 0x10d7b6d0c 0x10d7b5d3a 0x10d7b46ff 0x10d7bc09c 0x10d7ca869 0x10d7cb64e 0x10d7cc921 0x10da072ac 0x10d8b918f 0x10d60541e 
 0   mongod                              0x000000010dd153ea _ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE + 58
 1   mongod                              0x000000010dd14e86 _ZN5mongo12_GLOBAL__N_110abruptQuitEi + 198
 2   mongod                              0x000000010dd14f94 _ZN5mongo12_GLOBAL__N_124abruptQuitWithAddrSignalEiP9__siginfoPv + 212
 3   libsystem_platform.dylib            0x00007fff8e78d5aa _sigtramp + 26
 4   mongod                              0x000000010d62232a _ZN5mongo20compareElementValuesERKNS_11BSONElementES2_ + 3066
 5   mongod                              0x000000010d61e78f _ZNK5mongo11mutablebson8Document4Impl13writeChildrenINS_16BSONArrayBuilderEEEvjPT_ + 111
 6   mongod                              0x000000010d61e540 _ZNK5mongo11mutablebson8Document4Impl12writeElementINS_14BSONObjBuilderEEEvjPT_PKNS_10StringDataE + 592
 7   mongod                              0x000000010d61e01f _ZNK5mongo11mutablebson8Document4Impl13writeChildrenINS_14BSONObjBuilderEEEvjPT_ + 111
 8   mongod                              0x000000010d93fe36 _ZN5mongo6updateERKNS_13UpdateRequestEPNS_7OpDebugEPNS_12UpdateDriverEPNS_14CanonicalQueryE + 3046
 9   mongod                              0x000000010d94aa8f _ZN5mongo14UpdateExecutor7executeEv + 111
 10  mongod                              0x000000010d7b6d0c _ZN5mongo18WriteBatchExecutor10execUpdateERKNS_12BatchItemRefEPNS_7BSONObjEPPNS_16WriteErrorDetailE + 956
 11  mongod                              0x000000010d7b5d3a _ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPNSt3__16vectorIPNS_19BatchedUpsertDetailENS4_9allocatorIS7_EEEEPNS5_IPNS_16WriteErrorDetailENS8_ISD_EEEE + 154
 12  mongod                              0x000000010d7b46ff _ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE + 2031
 13  mongod                              0x000000010d7bc09c _ZN5mongo8WriteCmd3runERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEERNS_7BSONObjEiRS7_RNS_14BSONObjBuilderEb + 620
 14  mongod                              0x000000010d7ca869 _ZN5mongo12_execCommandEPNS_7CommandERKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderEb + 41
 15  mongod                              0x000000010d7cb64e _ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 2286
 16  mongod                              0x000000010d7cc921 _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 1313
 17  mongod                              0x000000010da072ac _ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 876
 18  mongod                              0x000000010d8b918f _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 2351
 19  mongod                              0x000000010d60541e _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 190



 Comments   
Comment by J Rassi [ 23/Apr/14 ]

Thanks!

Comment by James Simpson [ 23/Apr/14 ]

I can confirm that 2.6.1-rc0 fixes the issue.

Comment by James Simpson [ 22/Apr/14 ]

Perfect, I'll test that out tomorrow then and verify that the issue is resolved. Thanks!

Comment by J Rassi [ 22/Apr/14 ]

I note that the update predicate causing the crash is { $set: { ent.$.b: 1 } }. Based on that and the stack trace, I'm going to diagnose the issue as SERVER-13516 and resolve this ticket as a dup.

James, note that we are planning to release MongoDB 2.6.1-rc0 (release candidate for MongoDB 2.6.1) tomorrow, which includes a fix for this issue. It will be available at <http://www.mongodb.org/downloads>. Would you be willing to test the release candidate on your dev machine tomorrow, verify that it fixes your problem, and report back on this ticket with the result?

Comment by James Simpson [ 22/Apr/14 ]

I hadn't seen that issue, but after looking at that I have a feeling you are right. Regardless, here's the output with log level 3:

2014-04-22T09:51:26.411-0500 [conn41] processing update :  query: { _id: ObjectId('52c52c0b4f5b210000000027'), ent: { $elemMatch: { dt: new Date(1388788374000) } }, uid: ObjectId('51f4c454cc02eb0000000006') } updated: { $set: { ent.$.b: 1 } } god: 0 upsert: 0 multi: 0 callLogOp: 1 fromMigration: 0 fromReplications: 0
2014-04-22T09:51:26.411-0500 [conn41] Using cached query plan: query: { _id: ObjectId('52c52c0b4f5b210000000027'), ent: { $elemMatch: { dt: new Date(1388788374000) } }, uid: ObjectId('51f4c454cc02eb0000000006') } sort: {} projection: {}, planSummary: IXSCAN { _id: 1 }
2014-04-22T09:51:26.412-0500 [conn41] update validate options --  updatedFields: Fields:[ ent.141.b,] immutableAndSingleValueFields.size:0 fromRepl: 0 validate:1
2014-04-22T09:51:26.412-0500 [conn41] SEVERE: Invalid access at address: 0x21112d3fc2
2014-04-22T09:51:26.417-0500 [conn41] SEVERE: Got signal: 11 (Segmentation fault: 11).
Backtrace:0x105b6a3ea 0x105b69e86 0x105b69f94 0x7fff8e78d5aa 0xc0 0x10547378f 0x105473540 0x10547301f 0x105794e36 0x10579fa8f 0x10560bd0c 0x10560ad3a 0x1056096ff 0x10561109c 0x10561f869 0x10562064e 0x105621921 0x10585c2ac 0x10570e18f 0x10545a41e 
 0   mongod                              0x0000000105b6a3ea _ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE + 58
 1   mongod                              0x0000000105b69e86 _ZN5mongo12_GLOBAL__N_110abruptQuitEi + 198
 2   mongod                              0x0000000105b69f94 _ZN5mongo12_GLOBAL__N_124abruptQuitWithAddrSignalEiP9__siginfoPv + 212
 3   libsystem_platform.dylib            0x00007fff8e78d5aa _sigtramp + 26
 4   ???                                 0x00000000000000c0 0x0 + 192
 5   mongod                              0x000000010547378f _ZNK5mongo11mutablebson8Document4Impl13writeChildrenINS_16BSONArrayBuilderEEEvjPT_ + 111
 6   mongod                              0x0000000105473540 _ZNK5mongo11mutablebson8Document4Impl12writeElementINS_14BSONObjBuilderEEEvjPT_PKNS_10StringDataE + 592
 7   mongod                              0x000000010547301f _ZNK5mongo11mutablebson8Document4Impl13writeChildrenINS_14BSONObjBuilderEEEvjPT_ + 111
 8   mongod                              0x0000000105794e36 _ZN5mongo6updateERKNS_13UpdateRequestEPNS_7OpDebugEPNS_12UpdateDriverEPNS_14CanonicalQueryE + 3046
 9   mongod                              0x000000010579fa8f _ZN5mongo14UpdateExecutor7executeEv + 111
 10  mongod                              0x000000010560bd0c _ZN5mongo18WriteBatchExecutor10execUpdateERKNS_12BatchItemRefEPNS_7BSONObjEPPNS_16WriteErrorDetailE + 956
 11  mongod                              0x000000010560ad3a _ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPNSt3__16vectorIPNS_19BatchedUpsertDetailENS4_9allocatorIS7_EEEEPNS5_IPNS_16WriteErrorDetailENS8_ISD_EEEE + 154
 12  mongod                              0x00000001056096ff _ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE + 2031
 13  mongod                              0x000000010561109c _ZN5mongo8WriteCmd3runERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEERNS_7BSONObjEiRS7_RNS_14BSONObjBuilderEb + 620
 14  mongod                              0x000000010561f869 _ZN5mongo12_execCommandEPNS_7CommandERKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderEb + 41
 15  mongod                              0x000000010562064e _ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 2286
 16  mongod                              0x0000000105621921 _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 1313
 17  mongod                              0x000000010585c2ac _ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 876
 18  mongod                              0x000000010570e18f _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 2351
 19  mongod                              0x000000010545a41e _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 190

Comment by J Rassi [ 21/Apr/14 ]

Hi James,

Could you restart mongod with log level 3 enabled (add "-vvv" to the mongod command-line options) and paste a new log excerpt showing the crash?

I suspect this is a dup of SERVER-13516.

~ Jason Rassi

Generated at Thu Feb 08 03:32:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.