-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
Affects Version/s: 3.2.3
-
Component/s: WiredTiger
-
Labels:
-
Fully Compatible
-
ALL
-
-
Integration 12 (04/04/16)
I'm executing a $sample aggregation through the node.js driver with a large sample size (e.g. 10000 on a collection of size 1M), which reproducibly causes a seg fault and crashes the server. In several attempts with smaller sample sizes (100, 1000) the crash did not occur.
I was only able to reproduce on WiredTiger, not MMAP.
Every time it crashes, I can see "duplicate document" messages in the log file in the getMore command, followed by an "Invalid access at address: 0x90."
Relevant log lines and stack trace:
2016-02-26T18:00:56.537+1100 I COMMAND [conn8] command mongodb.fanclub2 command: aggregate { aggregate: "fanclub2", pipeline: [ { $sample: { size: 5000 } } ], allowDiskUse: true, cursor: { batchSize: 1000 } } cursorid:46731770331 keyUpdates:0 writeConflicts:0 numYields:25 reslen:401111 locks:{ Global: { acquireCount: { r: 56 } }, Database: { acquireCount: { r: 28 } }, Collection: { acquireCount: { r: 28 } } } protocol:op_query 234ms
2016-02-26T18:00:56.569+1100 D COMMAND [conn8] run command mongodb.$cmd { getMore: 46731770331, collection: "fanclub2", batchSize: 1000 }
2016-02-26T18:00:56.570+1100 D QUERY [conn8] $sample encountered duplicate document: {_id: 543fb212fe4dce3730b58fe1, membership_status: "PENDING", name: "David W Wood", phone_no: "+1441773104", last_login: 2012-07-29T22:42:23, address: {city: "Fresno, California", street: "12 Fir St", postal_code: "58844", country: "USA"}, last_position: {type: "Point", coordinates: [-178.813, 51.7581]}, email: "cutesy49@comcast.net"}
2016-02-26T18:00:56.572+1100 D QUERY [conn8] $sample encountered duplicate document: {_id: 53182d2cfe4dce37c22212e7, favorite_feature: "Replication", membership_status: "PENDING", name: "Ella P Allen", gender: "male", phone_no: "+452001029501", last_login: 2011-05-30T17:12:21, address: {city: "Orlando, Florida", street: "96 East Interlaken St", postal_code: "58824", country: "USA"}, last_position: {type: "Point", coordinates: [-175.925, -9.12909]}, email: "dimples84@icloud.com"}
2016-02-26T18:00:56.573+1100 F - [conn8] Invalid access at address: 0x90
2016-02-26T18:00:56.581+1100 F - [conn8] Got signal: 11 (Segmentation fault: 11).
0x10fec2119 0x10fec1c28 0x7fff92f83eaa 0x1 0x10f82f7c9 0x10fa48622 0x10f8374c7 0x10fa48622 0x10f7780e8 0x10f7f1b62 0x10f7f0b97 0x10f749d7a 0x10f8e6a22 0x10f8e336a 0x10f5ea64f 0x10fe8489e 0x10fe84041 0x7fff89192c13 0x7fff89192b90 0x7fff89190375
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"10F5E0000","o":"8E2119","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"},{"b":"10F5E0000","o":"8E1C28","s":"_ZN5mongo12_GLOBAL__N_124abruptQuitWithAddrSignalEiP9__siginfoPv"},{"b":"7FFF92F7F000","o":"4EAA","s":"_sigtramp"},{"b":"0","o":"1"},{"b":"10F5E0000","o":"24F7C9","s":"_ZN5mongo18MultiIteratorStage28doDetachFromOperationContextEv"},{"b":"10F5E0000","o":"468622","s":"_ZN5mongo12PlanExecutor26detachFromOperationContextEv"},{"b":"10F5E0000","o":"2574C7","s":"_ZN5mongo18PipelineProxyStage28doDetachFromOperationContextEv"},{"b":"10F5E0000","o":"468622","s":"_ZN5mongo12PlanExecutor26detachFromOperationContextEv"},{"b":"10F5E0000","o":"1980E8","s":"_ZN5mongo10GetMoreCmd3runEPNS_16OperationContextERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEERNS_7BSONObjEiRS9_RNS_14BSONObjBuilderE"},{"b":"10F5E0000","o":"211B62","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},{"b":"10F5E0000","o":"210B97","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"10F5E0000","o":"169D7A","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"10F5E0000","o":"306A22","s":"_ZN5mongoL15receivedCommandEPNS_16OperationContextERKNS_15NamespaceStringERNS_6ClientERNS_10DbResponseERNS_7MessageE"},{"b":"10F5E0000","o":"30336A","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"10F5E0000","o":"A64F","s":"_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE"},{"b":"10F5E0000","o":"8A489E","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},{"b":"10F5E0000","o":"8A4041","s":"_ZNSt3__114__thread_proxyINS_5tupleIJNS_6__bindIPFPvS3_EJPN5mongo12_GLOBAL__N_124MessagingPortWithHandlerEEEEEEEEES3_S3_"},{"b":"7FFF8918F000","o":"3C13","s":"_pthread_body"},{"b":"7FFF8918F000","o":"3B90","s":"_pthread_body"},{"b":"7FFF8918F000","o":"1375","s":"thread_start"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Darwin", "release" : "15.3.0", "version" : "Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/Users/tr/.mongodb/versions/mongodb-current/bin/mongod", "machType" : 2, "b" : "10F5E0000", "vmaddr" : "100000000", "buildId" : "8A20E1DCBF543EE3AE6F68D820D4C1B6" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF8CDF8000", "vmaddr" : "7FFF89215000", "buildId" : "5A4257EF31453BB387A40D2404A9462D" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF84CF5000", "vmaddr" : "7FFF81112000", "buildId" : "8FC3D139805534989AC56467CB7F4D14" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF89234000", "vmaddr" : "7FFF85651000", "buildId" : "6B245C0AF3EA383BA5425B0D0456A41B" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF8E4FA000", "vmaddr" : "7FFF8A917000", "buildId" : "766BC3F541F33315BABC72718A98EA92" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF8B0EF000", "vmaddr" : "7FFF8750C000", "buildId" : "D3C4AB4023B43BC68C385B8758D14E80" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF92F93000", "vmaddr" : "7FFF8F3B0000", "buildId" : "F51332690B22388CA57C079667B6291E" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF97082000", "vmaddr" : "7FFF9349F000", "buildId" : "C6BD205F4ECE37EEBCABA76F39CDCFFA" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF8E58A000", "vmaddr" : "7FFF8A9A7000", "buildId" : "324C91892AF33356847F6F4CE1C6E901" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF8FBBA000", "vmaddr" : "7FFF8BFD7000", "buildId" : "AA629043C6F632FE8007E3478E99ACA7" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF8FBB9000", "vmaddr" : "7FFF8BFD6000", "buildId" : "09397E0160663179A50C2CE666FDA929" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF8A810000", "vmaddr" : "7FFF86C2D000", "buildId" : "EDF719D6D2BB38DD8C944272BEFDA2CD" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF8E606000", "vmaddr" : "7FFF8AA23000", "buildId" : "CB745E1F48853F96B38B2093DF488FD5" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF909AF000", "vmaddr" : "7FFF8CDCC000", "buildId" : "163CF63A74553D1FAE578C4475A9204C" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF918BF000", "vmaddr" : "7FFF8DCDC000", "buildId" : "B8D1A5FCCFD53AAB8A1014DDC129710A" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF8E5BB000", "vmaddr" : "7FFF8A9D8000", "buildId" : "9B500E4EE462321E828E5524DC984C1B" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF93725000", "vmaddr" : "7FFF8FB42000", "buildId" : "49D423297DE9341392C3A473A7E9CF35" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF85141000", "vmaddr" : "7FFF8155E000", "buildId" : "EAB38A6C86713B13B50090EC1B912063" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF84B6B000", "vmaddr" : "7FFF80F88000", "buildId" : "5FD7907036CC3D02BEA7BB5D2AE97D5D" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFF8E682000", "vmaddr" : "7FFF8AA9F000", "buildId" : "1B3F5AFCFFCD3ECB8B9A5538366FB20D" }, { "path" : "/usr/lib/system/libsystem_coretls.dylib", "machType" : 6, "b" : "7FFF907B0000", "vmaddr" : "7FFF8CBCD000", "buildId" : "75C97D880A633093AE83DE33EB7405CE" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF92ED0000", "vmaddr" : "7FFF8F2ED000", "buildId" : "945B5FB1DA913D45A961A8FAD53C1E7E" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF8EED0000", "vmaddr" : "7FFF8B2ED000", "buildId" : "6513635B4ADE3B45BF63ED7AC565B0C9" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF8EB62000", "vmaddr" : "7FFF8AF7F000", "buildId" : "9CEB6C3B1CAF3C32A9FD93BC72CBCEA1" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF8844F000", "vmaddr" : "7FFF8486C000", "buildId" : "26655445CA97321EB221801CB378D1AA" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF8A990000", "vmaddr" : "7FFF86DAD000", "buildId" : "9EECAB18F02534C48E327EFFA6720EFC" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF8A8FA000", "vmaddr" : "7FFF86D17000", "buildId" : "865FE79AA22D3733A14FFC7B37F3AECD" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFF87FDA000", "vmaddr" : "7FFF843F7000", "buildId" : "DC8A102ABF0231A4891465C34DF6B592" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF90B15000", "vmaddr" : "7FFF8CF32000", "buildId" : "243FADE1255A3B788033F336CD64B817" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF92F7F000", "vmaddr" : "7FFF8F39C000", "buildId" : "D3A27E107F083603ACC87A92B2C04BAB" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8918F000", "vmaddr" : "7FFF855AC000", "buildId" : "327CECD0B88131538FCC4FD4818B7F16" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF91B5A000", "vmaddr" : "7FFF8DF77000", "buildId" : "3E0036AFFC6433528DA46B550C2C2562" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFF88D19000", "vmaddr" : "7FFF85136000", "buildId" : "FD6ECF2C148932CA981B9045B5EB1FAA" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFF8A9B6000", "vmaddr" : "7FFF86DD3000", "buildId" : "B485369FE3A1319E998C89AAF606079E" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF8CDA4000", "vmaddr" : "7FFF891C1000", "buildId" : "1D0F8265F0263CBD93D3F8DF14FFCE68" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF9086B000", "vmaddr" : "7FFF8CC88000", "buildId" : "124E0F0523503774A32C7F5BF38EDE73" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF8CFDF000", "vmaddr" : "7FFF893FC000", "buildId" : "61AB46109304354C9E9BD57198AE9866" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF884AB000", "vmaddr" : "7FFF848C8000", "buildId" : "9F45830DF1D53CDF94611A5477ED7D1E" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF90A2E000", "vmaddr" : "7FFF8CE4B000", "buildId" : "999E610F41FC32A3ADCA5EC049B65DFB" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF88C0D000", "vmaddr" : "7FFF8502A000", "buildId" : "DCCC81773D0935BC97842A04FEC4C71B" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF98B94000", "vmaddr" : "7FFF94FB1000", "buildId" : "4243B6B421E9355B9C5A95A216233B96" } ] }}
mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x10fec2119]
mongod(_ZN5mongo12_GLOBAL__N_124abruptQuitWithAddrSignalEiP9__siginfoPv+0xF8) [0x10fec1c28]
libsystem_platform.dylib(_sigtramp+0x1A) [0x7fff92f83eaa]
??? [0x1]
mongod(_ZN5mongo18MultiIteratorStage28doDetachFromOperationContextEv+0x29) [0x10f82f7c9]
mongod(_ZN5mongo12PlanExecutor26detachFromOperationContextEv+0x22) [0x10fa48622]
mongod(_ZN5mongo18PipelineProxyStage28doDetachFromOperationContextEv+0x47) [0x10f8374c7]
mongod(_ZN5mongo12PlanExecutor26detachFromOperationContextEv+0x22) [0x10fa48622]
mongod(_ZN5mongo10GetMoreCmd3runEPNS_16OperationContextERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEERNS_7BSONObjEiRS9_RNS_14BSONObjBuilderE+0x1288) [0x10f7780e8]
mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x8D2) [0x10f7f1b62]
mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x987) [0x10f7f0b97]
mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x2CA) [0x10f749d7a]
mongod(_ZN5mongoL15receivedCommandEPNS_16OperationContextERKNS_15NamespaceStringERNS_6ClientERNS_10DbResponseERNS_7MessageE+0x112) [0x10f8e6a22]
mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x7EA) [0x10f8e336a]
mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xFF) [0x10f5ea64f]
mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x27E) [0x10fe8489e]
mongod(_ZNSt3__114__thread_proxyINS_5tupleIJNS_6__bindIPFPvS3_EJPN5mongo12_GLOBAL__N_124MessagingPortWithHandlerEEEEEEEEES3_S3_+0x61) [0x10fe84041]
libsystem_pthread.dylib(_pthread_body+0x83) [0x7fff89192c13]
libsystem_pthread.dylib(_pthread_body+0x0) [0x7fff89192b90]
libsystem_pthread.dylib(thread_start+0xD) [0x7fff89190375]
----- END BACKTRACE -----
Full log file (verbosity 2) attached.