[SERVER-24596] Deadlock waiting for majority snapshot during map reduce output stage Created: 15/Jun/16  Updated: 19/Nov/16  Resolved: 14/Nov/16

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: None
Fix Version/s: 3.4.0-rc4

Type: Bug Priority: Major - P3
Reporter: Spencer Brody (Inactive) Assignee: Marko Vojvodic
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-27097 Coverity analysis defect 100050: Dere... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Query 17 (07/15/16), Query 2016-11-21
Participants:
Linked BF Score: 0

 Description   

When running the mapreduce.shardedfinish command against a mongod with majority reads enabled, it's possible to deadlock. This is because we take a global lock here, then use DBDirectClient to run a count(), which in turn will use AutoGetCollectionForRead to take database locks which in turn calls ReplicationCoordinatorImpl::waitUntilSnapshotCommitted. AugoGetCollectionForRead attempts to yield its locks when waiting for the snapshot, but it won't be able to yield the global lock held by the map reduce code above where it called into DBDirectClient.



 Comments   
Comment by Githook User [ 14/Nov/16 ]

Author:

{u'username': u'm-vojvodic', u'name': u'Marko Vojvodic', u'email': u'marko.vojvodic@mongodb.com'}

Message: SERVER-24596 Replace DBDirectClient::count call with Collection::numRecords to avoid deadlock in State::postProcessCollection
Branch: master
https://github.com/mongodb/mongo/commit/493b1f4bb89d721e630bcd99b1aac2808c8a5161

Comment by Eric Milkie [ 15/Jun/16 ]

I think the easiest way to fix this is to fix the map/reduce code that is using DBDirectClient to run count. It appears to be written that way now simply because it is the original code that Antoine and Tony wrote back in 2010 and hasn't been updated to use newer server APIs.

Comment by Spencer Brody (Inactive) [ 15/Jun/16 ]

Stack trace from BF-2498 that demonstrates this issue when attempting to run the shardedfinish stage against a config server included below. It was a bug that the mapReduce output was running against the config server, but I suspect the same issue could exist against a shard running with majority reads enabled.

 [2016/05/26 14:43:07.504] Thread 932 (Thread 0x2b1450b7a940 (LWP 19928)):
 [2016/05/26 14:43:07.504] #0  0x00002b1422581019 in pthread_cond_wait@@GLIBC_2.3.2 ()
 [2016/05/26 14:43:07.504]    from /lib64/libpthread.so.0
 [2016/05/26 14:43:07.504] #1  0x0000000001ef870c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
 [2016/05/26 14:43:07.504] #2  0x000000000108d14b in mongo::repl::ReplicationCoordinatorImpl::waitUntilSnapshotCommitted(mongo::OperationContext*, mongo::SnapshotName const&) ()
 [2016/05/26 14:43:07.504] #3  0x0000000000ca99a6 in mongo::AutoGetCollectionForRead::_ensureMajorityCommittedSnapshotIsValid(mongo::NamespaceString const&) ()
 [2016/05/26 14:43:07.504] #4  0x0000000000ca9b8e in mongo::AutoGetCollectionForRead::AutoGetCollectionForRead(mongo::OperationContext*, mongo::NamespaceString const&) ()
 [2016/05/26 14:43:07.504] #5  0x0000000000be7fe6 in mongo::(anonymous namespace)::CmdCount::run(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, mongo::BSONObjBuilder&) ()
 [2016/05/26 14:43:07.504] #6  0x0000000000cb5370 in mongo::DBDirectClient::count(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, int, int, int) ()
 [2016/05/26 14:43:07.504] #7  0x0000000000c2e183 in mongo::mr::_safeCount(mongo::OperationContext*, mongo::DBDirectClient&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, int, int, int) ()
 [2016/05/26 14:43:07.504] #8  0x0000000000c31ff4 in mongo::mr::State::postProcessCollectionNonAtomic(mongo::OperationContext*, mongo::CurOp*, mongo::ProgressMeterHolder&) ()
 [2016/05/26 14:43:07.504] #9  0x0000000000c32d7e in mongo::mr::State::postProcessCollection(mongo::OperationContext*, mongo::CurOp*, mongo::ProgressMeterHolder&) ()
 [2016/05/26 14:43:07.504] #10 0x0000000000c3e724 in mongo::mr::MapReduceFinishCommand::run(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, mongo::BSONObjBuilder&) ()
 [2016/05/26 14:43:07.504] #11 0x0000000000cab70c in mongo::Command::run(mongo::OperationContext*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) ()
 [2016/05/26 14:43:07.504] #12 0x0000000000cac8ea in mongo::Command::execCommand(mongo::OperationContext*, mongo::Command*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) ()
 [2016/05/26 14:43:07.505] #13 0x0000000001129ba6 in mongo::runCommands(mongo::OperationContext*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) ()
 [2016/05/26 14:43:07.505] #14 0x0000000000dd3b65 in mongo::assembleResponse(mongo::OperationContext*, mongo::Message&, mongo::DbResponse&, mongo::HostAndPort const&) ()
 [2016/05/26 14:43:07.505] #15 0x0000000000a4346a in mongo::MyMessageHandler::process(mongo::Message&, mongo::AbstractMessagingPort*) ()
 [2016/05/26 14:43:07.505] #16 0x00000000014beb03 in mongo::PortMessageServer::handleIncomingMsg(void*)
 [2016/05/26 14:43:07.505]     ()
 [2016/05/26 14:43:07.505] #17 0x00002b142257c83d in start_thread () from /lib64/libpthread.so.0
 [2016/05/26 14:43:07.505] #18 0x00002b1422866fdd in clone () from /lib64/libc.so.6

Generated at Thu Feb 08 04:06:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.