Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22862

Deadlock between ReplicaSetMonitor updating the connection string for a shard and reloading the ShardRegistry

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.4, 3.3.3
    • Component/s: Sharding
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:
    • Sprint:
      Sharding 11 (03/11/16)
    • Linked BF Score:
      0

      Description

       [2016/02/24 15:16:18.727] Thread 10 (Thread 0x2b4968191940 (LWP 15297)):
       [2016/02/24 15:16:18.727] #0  0x00002b495f481654 in __lll_lock_wait () from /lib64/libpthread.so.0
       [2016/02/24 15:16:18.727] #1  0x00002b495f47cf4a in _L_lock_1034 () from /lib64/libpthread.so.0
       [2016/02/24 15:16:18.727] #2  0x00002b495f47ce0c in pthread_mutex_lock () from /lib64/libpthread.so.0
       [2016/02/24 15:16:18.727] #3  0x0000000000701dd8 in mongo::ReplicaSetMonitor::getServerAddress() const
       [2016/02/24 15:16:18.727]     ()
       [2016/02/24 15:16:18.727] #4  0x00000000006ffc66 in mongo::RemoteCommandTargeterRS::connectionString()
       [2016/02/24 15:16:18.727]     ()
       [2016/02/24 15:16:18.727] #5  0x0000000000af3e57 in mongo::ShardRegistry::_updateLookupMapsForShard_inlock(std::shared_ptr<mongo::Shard>, boost::optional<mongo::ConnectionString const&>) ()
       [2016/02/24 15:16:18.727] #6  0x0000000000af4770 in mongo::ShardRegistry::_addShard_inlock(mongo::ShardType const&, bool) ()
       [2016/02/24 15:16:18.727] #7  0x0000000000af55ad in mongo::ShardRegistry::reload(mongo::OperationContext*) ()
       [2016/02/24 15:16:18.728] #8  0x0000000000a27dd6 in mongo::Balancer::run() ()
       [2016/02/24 15:16:18.728] #9  0x0000000000bc4d80 in mongo::BackgroundJob::jobBody() ()
       [2016/02/24 15:16:18.728] #10 0x0000000000e15190 in execute_native_thread_routine ()
       [2016/02/24 15:16:18.728] #11 0x00002b495f47a83d in start_thread () from /lib64/libpthread.so.0
       [2016/02/24 15:16:18.728] #12 0x00002b495f764fdd in clone () from /lib64/libc.so.6
       [2016/02/24 15:16:18.728] #13 0x0000000000000000 in ?? ()
      

       [2016/02/24 15:16:18.731] Thread 5 (Thread 0x2b496a995940 (LWP 15448)):
       [2016/02/24 15:16:18.731] #0  0x00002b495f481654 in __lll_lock_wait () from /lib64/libpthread.so.0
       [2016/02/24 15:16:18.731] #1  0x00002b495f47cf4a in _L_lock_1034 () from /lib64/libpthread.so.0
       [2016/02/24 15:16:18.732] #2  0x00002b495f47ce0c in pthread_mutex_lock () from /lib64/libpthread.so.0
       [2016/02/24 15:16:18.732] #3  0x0000000000af3152 in mongo::ShardRegistry::lookupRSName(std::string const&) const ()
       [2016/02/24 15:16:18.732] #4  0x0000000000b5061f in mongo::ConfigServer::replicaSetChangeShardRegistryUpdateHook(std::string const&, std::string const&) ()
       [2016/02/24 15:16:18.732] #5  0x0000000000707884 in mongo::ReplicaSetMonitor::Refresher::receivedIsMasterFromMaster(mongo::ReplicaSetMonitor::IsMasterReply const&) ()
       [2016/02/24 15:16:18.732] #6  0x0000000000708313 in mongo::ReplicaSetMonitor::Refresher::receivedIsMaster(mongo::HostAndPort const&, long, mongo::BSONObj const&) ()
       [2016/02/24 15:16:18.732] #7  0x000000000070890d in mongo::ReplicaSetMonitor::Refresher::_refreshUntilMatches(mongo::ReadPreferenceSetting const*) ()
       [2016/02/24 15:16:18.732] #8  0x00000000007096c1 in mongo::(anonymous namespace)::ReplicaSetMonitorWatcher::run() ()
       [2016/02/24 15:16:18.732] #9  0x0000000000bc4d80 in mongo::BackgroundJob::jobBody() ()
       [2016/02/24 15:16:18.732] #10 0x0000000000e15190 in execute_native_thread_routine ()
       [2016/02/24 15:16:18.732] #11 0x00002b495f47a83d in start_thread () from /lib64/libpthread.so.0
       [2016/02/24 15:16:18.732] #12 0x00002b495f764fdd in clone () from /lib64/libc.so.6
       [2016/02/24 15:16:18.732] #13 0x0000000000000000 in ?? ()
      

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: