[SERVER-17117] boost::lock_error in CachedPlanStage::updateCache when destructing CountStage Created: 29/Jan/15  Updated: 18/Sep/15  Resolved: 05/Feb/15

Status: Closed
Project: Core Server
Component/s: Concurrency, Querying
Affects Version/s: 3.0.0-rc7
Fix Version/s: 3.0.0-rc8, 3.1.0

Type: Bug Priority: Critical - P2
Reporter: Kamran K. Assignee: David Storch
Resolution: Done Votes: 0
Labels: 28qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-17236 '!_killed' invariant failure in Cache... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Participants:

 Description   

While running a longevity test against WiredTiger, I triggered a boost::lock_error that stemmed from CachedPlanStage::updateCache.

std::exception::what(): boost::lock_error
Actual exception type: boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >
 
mongod(mongo::printStackTrace(std::ostream&) 0x27) [/home/ubuntu/mongo/src/mongo/util/stacktrace_posix.cpp:104]
mongod( 0x151A087) [/home/ubuntu/mongo/src/mongo/util/signal_handlers_synchronous.cpp:180]
libstdc.so.6( 0x5E836) [0x7ffff750a836]
libstdc.so.6( 0x5D8F9) [0x7ffff75098f9]
libstdc.so.6(__gxx_personality_v0 0x52A) [0x7ffff750a4aa]
libgcc_s.so.1( 0xFFF3) [0x7ffff6f9fff3]
libgcc_s.so.1(_Unwind_Resume 0x57) [0x7ffff6fa0517]
mongod(mongo::CachedPlanStage::updateCache() 0x372) [/home/ubuntu/mongo/src/mongo/db/exec/cached_plan.cpp:198]
mongod(mongo::CachedPlanStage::~CachedPlanStage() 0x39) [/home/ubuntu/mongo/src/mongo/db/exec/cached_plan.cpp:67]
mongod(mongo::CachedPlanStage::~CachedPlanStage() 0x18) [/home/ubuntu/mongo/src/mongo/db/exec/cached_plan.cpp:73]
mongod(void boost::checked_delete<mongo::PlanStage>(mongo::PlanStage*) 0x2A) [/home/ubuntu/mongo/src/third_party/boost/boost/checked_delete.hpp:40]
mongod(boost::scoped_ptr<mongo::PlanStage>::~scoped_ptr() 0x1B) [/home/ubuntu/mongo/src/third_party/boost/boost/smart_ptr/scoped_ptr.hpp:81]
mongod(mongo::CountStage::~CountStage() 0x50) [/home/ubuntu/mongo/src/mongo/db/exec/count.cpp:58]
mongod(mongo::CountStage::~CountStage() 0x18) [/home/ubuntu/mongo/src/mongo/db/exec/count.cpp:58]
mongod(void boost::checked_delete<mongo::PlanStage>(mongo::PlanStage*) 0x2A) [/home/ubuntu/mongo/src/third_party/boost/boost/checked_delete.hpp:40]
mongod(boost::scoped_ptr<mongo::PlanStage>::~scoped_ptr() 0x1B) [/home/ubuntu/mongo/src/third_party/boost/boost/smart_ptr/scoped_ptr.hpp:81]
mongod(mongo::PlanExecutor::~PlanExecutor() 0x4C) [/home/ubuntu/mongo/src/mongo/db/query/plan_executor.cpp:197]
mongod(void boost::checked_delete<mongo::PlanExecutor>(mongo::PlanExecutor*) 0x1E) [/home/ubuntu/mongo/src/third_party/boost/boost/checked_delete.hpp:39 (discriminator 1)]
mongod(boost::scoped_ptr<mongo::PlanExecutor>::~scoped_ptr() 0x1B) [/home/ubuntu/mongo/src/third_party/boost/boost/smart_ptr/scoped_ptr.hpp:81]
mongod(mongo::CmdCount::run(mongo::OperationContext*, std::string const&, mongo::BSONObj&, int, std::string&, mongo::BSONObjBuilder&, bool) 0x368) [/home/ubuntu/mongo/src/mongo/db/commands/count.cpp:137]
mongod(mongo::_execCommand(mongo::OperationContext*, mongo::Command*, std::string const&, mongo::BSONObj&, int, std::string&, mongo::BSONObjBuilder&, bool) 0x96) [/home/ubuntu/mongo/src/mongo/db/dbcommands.cpp:1273]
mongod(mongo::Command::execCommand(mongo::OperationContext*, mongo::Command*, int, char const*, mongo::BSONObj&, mongo::BSONObjBuilder&, bool) 0xB9E) [/home/ubuntu/mongo/src/mongo/db/dbcommands.cpp:1489]
mongod(mongo::_runCommands(mongo::OperationContext*, char const*, mongo::BSONObj&, mongo::_BufBuilder<mongo::TrivialAllocator>&, mongo::BSONObjBuilder&, bool, int) 0x491) [/home/ubuntu/mongo/src/mongo/db/dbcommands.cpp:1561]
mongod( 0x1135D2A) [/home/ubuntu/mongo/src/mongo/db/query/find.cpp:137]
mongod(mongo::runQuery(mongo::OperationContext*, mongo::Message&, mongo::QueryMessage&, mongo::NamespaceString const&, mongo::CurOp&, mongo::Message&, bool) 0x372) [0x1537d52]
mongod( 0x103DEFA) [/home/ubuntu/mongo/src/mongo/db/instance.cpp:220 (discriminator 1)]
mongod(mongo::assembleResponse(mongo::OperationContext*, mongo::Message&, mongo::DbResponse&, mongo::HostAndPort const&, bool) 0x4F9) [/home/ubuntu/mongo/src/mongo/db/instance.cpp:403]
mongod(mongo::MyMessageHandler::process(mongo::Message&, mongo::AbstractMessagingPort*, mongo::LastError*) 0xF6) [/home/ubuntu/mongo/src/mongo/db/db.cpp:206 (discriminator 1)]
mongod(mongo::PortMessageServer::handleIncomingMsg(void*) 0x40D) [/home/ubuntu/mongo/src/mongo/util/net/message_server_port.cpp:231]
libpthread.so.0( 0x8182) [0x7ffff7bc4182]
libc.so.6(clone 0x6D) [0x7ffff6cc500d]


Version: 091f56d3b73453



 Comments   
Comment by Githook User [ 05/Feb/15 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-17117 propagate kill notifications to CachedPlanStage

(cherry picked from commit 5936e08901898c8c8cc3e31de96cfe4264c171c6)
Branch: v3.0
https://github.com/mongodb/mongo/commit/3e9b48ff434a6ccda0890e752f2034845c042f21

Comment by Githook User [ 05/Feb/15 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-17117 propagate kill notifications to CachedPlanStage
Branch: master
https://github.com/mongodb/mongo/commit/5936e08901898c8c8cc3e31de96cfe4264c171c6

Comment by David Storch [ 04/Feb/15 ]

The kill flag on the count query is being successfully set by the database drop operation. However, the CachedPlanStage is attempting to access the plan cache from its destructor. The boost::lock_error is thrown because we are accessing a plan cache instance that has already been destroyed and we attempt to lock a mutex after its destruction.

We need to prevent the CachedPlanStage from accessing a collection that no longer exists.

Generated at Thu Feb 08 03:43:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.