[SERVER-21706] Certain parameters to mapReduce trigger segmentation fault in a sharded cluster Created: 30/Nov/15  Updated: 16/Nov/16  Resolved: 15/Dec/15

Status: Closed
Project: Core Server
Component/s: MapReduce, Sharding
Affects Version/s: 3.2.0-rc4
Fix Version/s: 3.2.1, 3.3.0

Type: Bug Priority: Major - P3
Reporter: Robert Guo (Inactive) Assignee: Misha Tyulenev
Resolution: Done Votes: 0
Labels: code-and-test
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-24597 mapreduce.sharded finish attempts to ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Sprint: Sharding E (01/08/16)
Participants:
Linked BF Score: 0

 Description   

segmentation fault on the config server primary

c20002| 2015-11-30T18:28:13.367-0500 I SHARDING [conn26] first cluster operation detected, adding sharding hook to enable versioning and authentication to remote servers
c20002| 2015-11-30T18:28:13.379-0500 F -        [conn26] Invalid access at address: 0x20
c20002| 2015-11-30T18:28:13.383-0500 F -        [conn26] Got signal: 11 (Segmentation fault: 11).
c20002| 
c20002|  0x1080854c9 0x10808500f 0x7fff8434d52a 0x10843024f 0x107f5e55c 0x10792ef5d 0x10798de82 0x10798d022 0x1078e3a16 0x107a8222d 0x1077885ea 0x108031253 0x108030a61 0x7fff8a7e19b1 0x7fff8a7e192e 0x7fff8a7df385
c20002|  mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x1080854c9]
c20002|  mongod(_ZN5mongo12_GLOBAL__N_124abruptQuitWithAddrSignalEiP9__siginfoPv+0x12F) [0x10808500f]
c20002|  libsystem_platform.dylib(_sigtramp+0x1A) [0x7fff8434d52a]
c20002|  mongod(_ZN2js3jit22CanEnterBaselineMethodEP9JSContextRNS_8RunStateE+0xAF) [0x10843024f]
c20002|  mongod(_ZN5mongo13ShardRegistry8getShardEPNS_16OperationContextERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEE+0x2C) [0x107f5e55c]
c20002|  mongod(_ZN5mongo2mr22MapReduceFinishCommand3runEPNS_16OperationContextERKNSt3__112basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEERNS_7BSONObjEiRSA_RNS_14BSONObjBuilderE+0x6AD) [0x10792ef5d]
c20002|  mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x9D2) [0x10798de82]
c20002|  mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0xC62) [0x10798d022]
c20002|  mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x2A6) [0x1078e3a16]
c20002|  mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x17CD) [0x107a8222d]
c20002|  mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xEA) [0x1077885ea]
c20002|  mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x253) [0x108031253]
c20002|  mongod(_ZNSt3__114__thread_proxyINS_5tupleIJNS_6__bindIPFPvS3_EJPN5mongo12_GLOBAL__N_124MessagingPortWithHandlerEEEEEEEEES3_S3_+0x61) [0x108030a61]
c20002|  libsystem_pthread.dylib(_pthread_body+0x83) [0x7fff8a7e19b1]
c20002|  libsystem_pthread.dylib(_pthread_body+0x0) [0x7fff8a7e192e]
c20002|  libsystem_pthread.dylib(thread_start+0xD) [0x7fff8a7df385]



 Comments   
Comment by Githook User [ 15/Dec/15 ]

Author:

{u'username': u'mikety', u'name': u'Misha Tyulenev', u'email': u'misha@mongodb.com'}

Message: SERVER-21706 error when mapReduce outputs to config or admin db

(cherry picked from commit 3e1a3872c996fcc72c8cf20ea38e9fe86a11caa4)
Branch: v3.2
https://github.com/mongodb/mongo/commit/126cecc6366342ffe9d4561ba6ff7532f596c019

Comment by Githook User [ 15/Dec/15 ]

Author:

{u'username': u'mikety', u'name': u'Misha Tyulenev', u'email': u'misha@mongodb.com'}

Message: SERVER-21706 error when mapReduce outputs to config or admin db
Branch: master
https://github.com/mongodb/mongo/commit/3e1a3872c996fcc72c8cf20ea38e9fe86a11caa4

Comment by Andy Schwerin [ 01/Dec/15 ]

Looking at the code, this is only a problem for databases stored on the config servers. This is because the config servers do not have a sharding state object (and should not have one in the current design).

Comment by Robert Guo (Inactive) [ 01/Dec/15 ]

It's different than the other ticket. Here it seems like there's a nullptr for a ShardRegistry. The other one is for a ChunkManager.

Kamran and I tried it on a number of valid and non-existant databases and only "config" and "admin" are affected.

Comment by Andy Schwerin [ 01/Dec/15 ]

Does this have to be on the admin/config database?

Comment by Andy Schwerin [ 01/Dec/15 ]

Is this different from SERVER-21596?

Comment by Kamran K. [ 30/Nov/15 ]

c20002| ==31287== ERROR: AddressSanitizer: SEGV on unknown address 0x000000000030 (pc 0x7f3550a7b414 sp 0x7f3511c47c90 bp 0x7f3511c47cb0 T91)
c20002| AddressSanitizer can not provide additional info.
c20002|     #0 0x7f3550a7b413 in __GI___pthread_mutex_lock /build/buildd/eglibc-2.19/nptl/../nptl/pthread_mutex_lock.c:63
c20002|     #1 0x1e534a9 in __gthread_mutex_lock(pthread_mutex_t*) /usr/include/x86_64-linux-gnu/c++/4.8/bits/gthr-default.h:748
c20002|     #2 0x1e55b73 in std::mutex::lock() /usr/include/c++/4.8/mutex:134
c20002|     #3 0x1e560a0 in std::lock_guard<std::mutex>::lock_guard(std::mutex&) /usr/include/c++/4.8/mutex:414
c20002|     #4 0x2c87735 in mongo::ShardRegistry::_findUsingLookUp(std::string const&) /home/s/code/mongo/mongo/src/mongo/s/client/shard_registry.cpp:403
c20002|     #5 0x2c85878 in mongo::ShardRegistry::getShard(mongo::OperationContext*, std::string const&) /home/s/code/mongo/mongo/src/mongo/s/client/shard_registry.cpp:221
c20002|     #6 0x20ae8cf in mongo::mr::MapReduceFinishCommand::run(mongo::OperationContext*, std::string const&, mongo::BSONObj&, int, std::string&, mongo::BSONObjBuilder&) /home/s/code/mongo/mongo/src/mongo/db/commands/mr.cpp:1612
c20002|     #7 0x2153310 in mongo::Command::run(mongo::OperationContext*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) /home/s/code/mongo/mongo/src/mongo/db/dbcommands.cpp:1432
c20002|     #8 0x2151d24 in mongo::Command::execCommand(mongo::OperationContext*, mongo::Command*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) /home/s/code/mongo/mongo/src/mongo/db/dbcommands.cpp:1317
c20002|     #9 0x2042a09 in mongo::runCommands(mongo::OperationContext*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) /home/s/code/mongo/mongo/src/mongo/db/commands.cpp:498
c20002|     #10 0x2358f65 in mongo::(anonymous namespace)::receivedRpc(mongo::OperationContext*, mongo::Client&, mongo::DbResponse&, mongo::Message&) /home/s/code/mongo/mongo/src/mongo/db/instance.cpp:293
c20002|     #11 0x235a7d2 in mongo::assembleResponse(mongo::OperationContext*, mongo::Message&, mongo::DbResponse&, mongo::HostAndPort const&) /home/s/code/mongo/mongo/src/mongo/db/instance.cpp:522
c20002|     #12 0x1dd25da in mongo::MyMessageHandler::process(mongo::Message&, mongo::AbstractMessagingPort*) /home/s/code/mongo/mongo/src/mongo/db/db.cpp:171
c20002|     #13 0x2e20fd2 in mongo::PortMessageServer::handleIncomingMsg(void*) /home/s/code/mongo/mongo/src/mongo/util/net/message_server_port.cpp:229
c20002|     #14 0x7f35528b2b97 (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x18b97)
c20002|     #15 0x7f3550a79181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312
c20002|     #16 0x7f35507a647c in clone /build/buildd/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111
c20002| SUMMARY: AddressSanitizer: SEGV /build/buildd/eglibc-2.19/nptl/../nptl/pthread_mutex_lock.c:66 __GI___pthread_mutex_lock
c20002| Thread T91 created by T0 here:
c20002|     #0 0x7f35528a4b5b in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.0+0xab5b)
c20002|     #1 0x2e20629 in mongo::PortMessageServer::accepted(std::shared_ptr<mongo::Socket>, long long) /home/s/code/mongo/mongo/src/mongo/util/net/message_server_port.cpp:148
c20002|     #2 0x2e151bf in mongo::Listener::initAndListen() /home/s/code/mongo/mongo/src/mongo/util/net/listen.cpp:352
c20002|     #3 0x2e20949 in mongo::PortMessageServer::run() /home/s/code/mongo/mongo/src/mongo/util/net/message_server_port.cpp:176
c20002|     #4 0x1dc95e4 in mongo::_initAndListen(int) /home/s/code/mongo/mongo/src/mongo/db/db.cpp:652
c20002|     #5 0x1dc9a87 in mongo::initAndListen(int) /home/s/code/mongo/mongo/src/mongo/db/db.cpp:657
c20002|     #6 0x1dcb44a in mongoDbMain(int, char**, char**) /home/s/code/mongo/mongo/src/mongo/db/db.cpp:891
c20002|     #7 0x1dc9e31 in main /home/s/code/mongo/mongo/src/mongo/db/db.cpp:702
c20002|     #8 0x7f35506cdec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287
c20002| ==31287== ABORTING

Generated at Thu Feb 08 03:58:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.