[CXX-787] multithreaded application hangs during bulk load Created: 23/Dec/15  Updated: 01/Jan/16  Resolved: 01/Jan/16

Status: Closed
Project: C++ Driver
Component/s: API
Affects Version/s: legacy-1.0.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Judy Han [X] Assignee: Andrew Morrow (Inactive)
Resolution: Done Votes: 0
Labels: legacy-cxx
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

We have a multi-threaded app doing bulk loading (each thread loading different set of data), with 10 thread, all threads are hanging with the same call stack.

#0 0x0000003ba820eb5c in recv () from /lib64/libpthread.so.0
#1 0x00007f291f2d36cf in mongo::Socket::_recv(char*, int) () from /opt/mtx/lib/libmongoclient.so
#2 0x00007f291f2d36f9 in mongo::Socket::unsafe_recv(char*, int) () from /opt/mtx/lib/libmongoclient.so
#3 0x00007f291f2d373f in mongo::Socket::recv(char*, int) () from /opt/mtx/lib/libmongoclient.so
#4 0x00007f291f2cb514 in mongo::MessagingPort::recv(mongo::Message&) () from /opt/mtx/lib/libmongoclient.so
#5 0x00007f291f2cc4c2 in mongo::MessagingPort::recv(mongo::Message const&, mongo::Message&) () from /opt/mtx/lib/libmongoclient.so
#6 0x00007f291f25c629 in mongo::DBClientConnection::call(mongo::Message&, mongo::Message&, bool, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*) () from /opt/mtx/lib/libmongoclient.so
#7 0x00007f291f2790b1 in mongo::DBClientCursor::init() () from /opt/mtx/lib/libmongoclient.so
#8 0x00007f291f24906d in mongo::DBClientBase::query(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::Query, int, int, mongo::BSONObj const*, int, int) () from /opt/mtx/lib/libmongoclient.so
#9 0x00007f291f25c4d3 in mongo::DBClientConnection::query(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::Query, int, int, mongo::BSONObj const*, int, int) () from /opt/mtx/lib/libmongoclient.so
#10 0x00007f291f25b2d2 in mongo::DBClientInterface::findN(std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> >&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::Query, int, int, mongo::BSONObj const*, int, int) () from /opt/mtx/lib/libmongoclient.so
#11 0x00007f291f25b640 in mongo::DBClientInterface::findOne(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::Query const&, mongo::BSONObj const*, int) () from /opt/mtx/lib/libmongoclient.so
#12 0x00007f291f2551e0 in mongo::DBClientWithCommands::runCommand(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, mongo::BSONObj&, int) () from /opt/mtx/lib/libmongoclient.so
#13 0x00007f291f255353 in mongo::DBClientConnection::runCommand(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, mongo::BSONObj&, int) () from /opt/mtx/lib/libmongoclient.so
#14 0x00007f291f245488 in mongo::CommandWriter::_send(mongo::BSONObjBuilder*, mongo::WriteConcern const*, mongo::StringData const&) () from /opt/mtx/lib/libmongoclient.so
#15 0x00007f291f245f1c in mongo::CommandWriter::write(mongo::StringData const&, std::vector<mongo::WriteOperation*, std::allocator<mongo::WriteOperation*> > const&, bool, mongo::WriteConcern const*, mongo::WriteResult*) () from /opt/mtx/lib/libmongoclient.so
#16 0x00007f291f24a93d in mongo::DBClientBase::_write(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<mongo::WriteOperation*, std::allocator<mongo::WriteOperation*> > const&, bool, mongo::WriteConcern const*, mongo::WriteResult*) () from /opt/mtx/lib/libmongoclient.so
#17 0x00007f291f565b0f in EventLoader::EventLoaderWorkerTask::commitBulk(mongo::BulkOperationBuilder&, std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> >&, char const*) () from /opt/mtx/lib/libevent_loader.so



 Comments   
Comment by Andrew Morrow (Inactive) [ 01/Jan/16 ]

Thanks for getting back to me with those answers, and I'm happy to hear that you were able to develop an understanding of the behavior you are observing. I'm closing this ticket, but feel free to ping it if you have further questions about the C++ driver.

Comment by Judy Han [X] [ 30/Dec/15 ]

Hi Andrew,
Thanks for the response. Thanks for your questions.
We just figured out what the problem is - we are missing one index and the database is quite big and the upsert is taking a long long time...
Please feel free to close the ticket.
Thanks you very much!
Judy

Here are the answers to your questions in case you still want to know:
1. yes, each thread use a separate DBClientBase object.
2. yes, I am able to connect via mongo from the same machine where I ran the app and able to issue queries and inserts, but updates are hanging.
3. mongodb_writeconcern is 1, mongodb_wtimeout is 1 ms, the replication
4. no replication, no sharding, here is the configuration file:

Comment by Andrew Morrow (Inactive) [ 30/Dec/15 ]

Hi Judy -

That call stack looks pretty normal. It looks more like the client thread is just waiting for a response from the server. I have a few questions:

  • Does each thread use a separate DBClientBase object? They aren't safe to share across threads.
  • Are the servers with which you are communicating acting normally? In other words, during the bulk load, can you issue queries and updates against them?
  • What WriteConcern are you using? Could this simply be awaiting replication?
  • What is the topology of the cluster (replication? sharding?)
Generated at Wed Feb 07 22:00:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.