Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16252

DBConnectionPool::flush should fully flush connection pool

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0-rc2
    • Component/s: Networking
    • Labels:
      None
    • Operating System:
      ALL

      Description

      Currently the connection pool flush operation uses isMaster to check if connections are still "alive" - this may cause additional overhead and complexity for a command used when the network state is sad. It can also trigger a deadlock when the ReplicaSetMonitor needs to refresh:

      Thread 2 (Thread 0x7f5c64838700 (LWP 5100)):
      #0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
      #1  0x00007f5c6ae34065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2  0x00007f5c6ae33eba in __pthread_mutex_lock (mutex=0x7f5c6b1724a0) at pthread_mutex_lock.c:61
      #3  0x0000000000dabe18 in boost::timed_mutex::lock (this=0x7f5c6b1724a0) at src/third_party/boost/boost/thread/pthread/mutex.hpp:154
      #4  0x0000000000dae256 in boost::unique_lock<boost::timed_mutex>::lock (this=0x7f5c648369a0) at src/third_party/boost/boost/thread/locks.hpp:412
      #5  0x0000000000dad031 in boost::unique_lock<boost::timed_mutex>::unique_lock (this=0x7f5c648369a0, m_=...)
          at src/third_party/boost/boost/thread/locks.hpp:290
      #6  0x0000000000dac173 in mongo::mutex::scoped_lock::scoped_lock (this=0x7f5c648369a0, m=...) at src/mongo/util/concurrency/mutex.h:115
      #7  0x0000000000da9912 in mongo::DBConnectionPool::_get (this=0x20bb1e0 <mongo::pool>, ident=..., socketTimeout=5)
          at src/mongo/client/connpool.cpp:205
      #8  0x0000000000da9bc4 in mongo::DBConnectionPool::get (this=0x20bb1e0 <mongo::pool>, url=..., socketTimeout=5)
          at src/mongo/client/connpool.cpp:234
      #9  0x0000000000e03dea in mongo::ScopedDbConnection::ScopedDbConnection (this=0x7f5c64836b70, host=..., socketTimeout=5)
          at src/mongo/client/connpool.h:308
      #10 0x0000000000dfebba in mongo::ReplicaSetMonitor::Refresher::_refreshUntilMatches (this=0x7f5c64836c40, criteria=0x7f5c64836cd0)
      ---Type <return> to continue, or q <return> to quit---
          at src/mongo/client/replica_set_monitor.cpp:760
      #11 0x0000000000e03567 in mongo::ReplicaSetMonitor::Refresher::refreshUntilMatches (this=0x7f5c64836c40, criteria=...)
          at src/mongo/client/replica_set_monitor.h:239
      #12 0x0000000000dfb70f in mongo::ReplicaSetMonitor::getHostOrRefresh (this=0x7f5c68c47120, criteria=...)
          at src/mongo/client/replica_set_monitor.cpp:288
      #13 0x0000000000dfb8ae in mongo::ReplicaSetMonitor::getMasterOrUassert (this=0x7f5c68c47120) at src/mongo/client/replica_set_monitor.cpp:302
      #14 0x0000000000dcea2f in mongo::DBClientReplicaSet::checkMaster (this=0x7f5c69542c20) at src/mongo/client/dbclient_rs.cpp:291
      #15 0x0000000000dd1b25 in mongo::DBClientReplicaSet::findOne (this=0x7f5c69542c20, ns=..., query=..., fieldsToReturn=0x0, queryOptions=0)
          at src/mongo/client/dbclient_rs.cpp:630
      #16 0x0000000000db5da5 in mongo::DBClientWithCommands::runCommand (this=0x7f5c69542c20, dbname=..., cmd=..., info=..., options=0)
          at src/mongo/client/dbclient.cpp:448
      #17 0x0000000000db86b4 in mongo::DBClientWithCommands::isMaster (this=0x7f5c69542c20, isMaster=@0x7f5c6483724e: true, info=0x7f5c648371e0)
          at src/mongo/client/dbclient.cpp:776
      #18 0x0000000000da9276 in mongo::PoolForHost::flush (this=0x7f5c6b1902c0) at src/mongo/client/connpool.cpp:127
      #19 0x0000000000daa4ba in mongo::DBConnectionPool::flush (this=0x20bb1e0 <mongo::pool>) at src/mongo/client/connpool.cpp:302
      #20 0x0000000000e90dea in mongo::PoolFlushCmd::run (this=0x20bdce0 <mongo::poolFlushCmd>, txn=0x0, result=...) at src/mongo/db/commands.cpp:356
      #21 0x0000000001185197 in mongo::Command::execCommandClientBasic (txn=0x0, c=0x20bdce0 <mongo::poolFlushCmd>, client=..., queryOptions=0, 
          ns=0x7f5c695aa434 "admin.$cmd", cmdObj=..., result=..., fromRepl=false) at src/mongo/s/s_only.cpp:148
      #22 0x000000000110b36c in mongo::Command::runAgainstRegistered (ns=0x7f5c695aa434 "admin.$cmd", jsobj=..., anObjBuilder=..., queryOptions=0)
          at src/mongo/s/commands_public.cpp:2696
      #23 0x000000000119c49b in mongo::Strategy::clientCommandOp (this=0x7f5c6b2386e0, r=...) at src/mongo/s/strategy.cpp:296
      #24 0x0000000001183f39 in mongo::Request::process (this=0x7f5c64837b50, attempt=0) at src/mongo/s/request.cpp:122
      #25 0x0000000000d633af in mongo::ShardedMessageHandler::process (this=0x7fff0c363a60, m=..., p=0x7f5c69546fa0, le=0x7f5c68b2c1a0)
          at src/mongo/s/server.cpp:148
      #26 0x000000000124f601 in mongo::PortMessageServer::handleIncomingMsg (arg=0x7f5c68b01220) at src/mongo/util/net/message_server_port.cpp:234
      #27 0x00007f5c6ae31e9a in start_thread (arg=0x7f5c64838700) at pthread_create.c:308
      #28 0x00007f5c69f3dccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
      #29 0x0000000000000000 in ?? ()
       
      Thread 1 (Thread 0x7f5c6b24a740 (LWP 4703)):
      #0  0x00007f5c69f37033 in select () at ../sysdeps/unix/syscall-template.S:82
      #1  0x00000000012479f3 in mongo::Listener::initAndListen (this=0x7f5c695983e8) at src/mongo/util/net/listen.cpp:256
      #2  0x000000000124f0ea in mongo::PortMessageServer::run (this=0x7f5c695983e0) at src/mongo/util/net/message_server_port.cpp:158
      #3  0x0000000000d5e6bd in mongo::start (opts=...) at src/mongo/s/server.cpp:200
      #4  0x0000000000d5edea in runMongosServer (doUpgrade=false) at src/mongo/s/server.cpp:297
      #5  0x0000000000d5f21a in _main () at src/mongo/s/server.cpp:364
      #6  0x0000000000d5f62b in mongoSMain (argc=10, argv=0x7fff0c364148, envp=0x7fff0c3641a0) at src/mongo/s/server.cpp:434
      #7  0x0000000000d5f928 in main (argc=10, argv=0x7fff0c364148, envp=0x7fff0c3641a0) at src/mongo/s/server.cpp:468

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              renctan Randolph Tan
              Reporter:
              greg_10gen Greg Studer
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: