[SERVER-13921] Database crashed when running C++ driver tests Created: 13/May/14  Updated: 12/Jan/15  Resolved: 12/Jan/15

Status: Closed
Project: Core Server
Component/s: JavaScript
Affects Version/s: 2.7.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tyler Brock Assignee: Andy Schwerin
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux 64-bit


Attachments: File mongorun.log    
Operating System: ALL
Participants:

 Description   

2014-05-13T10:11:36.215-0400 [conn2623] dropDatabase gridfs-test starting
2014-05-13T10:11:36.217-0400 [conn2620] SEVERE: Got signal: 11 (Segmentation fault).
0xee8816 0xee7c42 0xee80a6 0x7ff4de9914b0 0xff40e6 0xe5db47 0xe52a1d 0xe52a69 0x796889 0xe4e3cb 0xe4ec09 0x909d08 0x913942 0x968877 0x9696ac 0x96a471 0xba3b03 0xa5ae07 0x79c178 0xea0fd1 
 mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xee8816]
 mongod(+0xae7c42) [0xee7c42]
 mongod(+0xae80a6) [0xee80a6]
 /usr/lib/libpthread.so.0(+0xf4b0) [0x7ff4de9914b0]
 mongod(_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x16) [0xff40e6]
 mongod(_ZN5mongo10ObjTrackerINS_10BSONHolderEED1Ev+0x67) [0xe5db47]
 mongod(_ZN5mongo7V8ScopeD1Ev+0x12d) [0xe52a1d]
 mongod(_ZN5mongo7V8ScopeD0Ev+0x9) [0xe52a69]
 mongod(_ZN5boost6detail12shared_countD1Ev+0x39) [0x796889]
 mongod(_ZN5mongo11PooledScopeD1Ev+0xab) [0xe4e3cb]
 mongod(_ZN5mongo11PooledScopeD0Ev+0x9) [0xe4ec09]
 mongod(_ZN5mongo2mr5StateD1Ev+0x108) [0x909d08]
 mongod(_ZN5mongo2mr16MapReduceCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xe62) [0x913942]
 mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x27) [0x968877]
 mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xbcc) [0x9696ac]
 mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6f1) [0x96a471]
 mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x803) [0xba3b03]
 mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x7d7) [0xa5ae07]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xb8) [0x79c178]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x471) [0xea0fd1]



 Comments   
Comment by Ramon Fernandez Marina [ 16/Jul/14 ]

rassi@10gen.com, tyler@10gen.com, are you still interested in chasing this ticket down? I'm trying to figure out what fixVersion should be, if it's not "debugging with submitter" because we don't have a reproducer?

Cc thomasr

Comment by Andy Schwerin [ 12/Jun/14 ]

Looks like maybe the thread-local isolate or maybe its heap pointer were set to NULL. As this is during cleanup, it suggests maybe there's a cleanup race – maybe two threads ended up with pointers to the same ObjTracer<BSONHolder>.

Needs a repro.

Comment by Tyler Brock [ 21/May/14 ]

Tried running the gridfs_test and clientTest in loops for a while and couldn't get it to crash.

Comment by Tyler Brock [ 19/May/14 ]

Attaching log output for successful run.

Comment by Tyler Brock [ 19/May/14 ]

I don't have the logs for that run but basically there are a series of tests that each use the same namespace and drop the collection before each run. I haven't been able to reproduce locally after running the tests a bit more. If it pops up again we might be able to look into it further.

Comment by J Rassi [ 14/May/14 ]

I don't think it's related to an out-of-memory condition. V8::AdjustAmountOfExternalAllocatedMemory() is called unconditionally from the BSONHolder constructor, and it'd segfault if it has a dangling reference to an already-freed isolate (and indeed it looks like dropDatabase was being run concurrently).

Please upload the whole log for this test run, if you still have it. Could you describe and/or link to the tests you were running?

Comment by Tyler Brock [ 13/May/14 ]

Build flags: --ssl
Git version: ac19e03a1c66037e17e6f057c0a0bb004b2a3977
Reproduction has failed so far, it happened as I was testing the C++ driver. Could be that I just ran out of memory, I have no swap on this box.

Comment by J Rassi [ 13/May/14 ]

A few questions:

  • What build flags did you use to compile the server?
  • What's the git hash of your build?
  • What happens when you try to reproduce this?
Comment by Tyler Brock [ 13/May/14 ]

The demangled form:

0xee8816 0xee7c42 0xee80a6 0x7ff4de9914b0 0xff40e6 0xe5db47 0xe52a1d 0xe52a69 0x796889 0xe4e3cb 0xe4ec09 0x909d08 0x913942 0x968877 0x9696ac 0x96a471 0xba3b03 0xa5ae07 0x79c178 0xea0fd1 
 mongod(mongo::printStackTrace(std::basic_ostream<char, std::char_traits<char> >&)+0x26) [0xee8816]
 mongod(+0xae7c42) [0xee7c42]
 mongod(+0xae80a6) [0xee80a6]
 /usr/lib/libpthread.so.0(+0xf4b0) [0x7ff4de9914b0]
 mongod(v8::V8::AdjustAmountOfExternalAllocatedMemory(long)+0x16) [0xff40e6]
 mongod(mongo::ObjTracker<mongo::BSONHolder>::~ObjTracker()+0x67) [0xe5db47]
 mongod(mongo::V8Scope::~V8Scope()+0x12d) [0xe52a1d]
 mongod(mongo::V8Scope::~V8Scope()+0x9) [0xe52a69]
 mongod(boost::detail::shared_count::~shared_count()+0x39) [0x796889]
 mongod(mongo::PooledScope::~PooledScope()+0xab) [0xe4e3cb]
 mongod(mongo::PooledScope::~PooledScope()+0x9) [0xe4ec09]
 mongod(mongo::mr::State::~State()+0x108) [0x909d08]
 mongod(mongo::mr::MapReduceCommand::run(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj&, int, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, mongo::BSONObjBuilder&, bool)+0xe62) [0x913942]
 mongod(mongo::_execCommand(mongo::Command*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj&, int, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, mongo::BSONObjBuilder&, bool)+0x27) [0x968877]
 mongod(mongo::Command::execCommand(mongo::Command*, mongo::Client&, int, char const*, mongo::BSONObj&, mongo::BSONObjBuilder&, bool)+0xbcc) [0x9696ac]
 mongod(mongo::_runCommands(char const*, mongo::BSONObj&, mongo::_BufBuilder<mongo::TrivialAllocator>&, mongo::BSONObjBuilder&, bool, int)+0x6f1) [0x96a471]
 mongod(mongo::newRunQuery(mongo::Message&, mongo::QueryMessage&, mongo::CurOp&, mongo::Message&)+0x803) [0xba3b03]
 mongod(mongo::assembleResponse(mongo::Message&, mongo::DbResponse&, mongo::HostAndPort const&)+0x7d7) [0xa5ae07]
 mongod(mongo::MyMessageHandler::process(mongo::Message&, mongo::AbstractMessagingPort*, mongo::LastError*)+0xb8) [0x79c178]
 mongod(mongo::PortMessageServer::handleIncomingMsg(void*)+0x471) [0xea0fd1]

Generated at Thu Feb 08 03:33:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.