Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32446

Mongod crashes when config servers upgraded to RS

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.2.18
    • Fix Version/s: 3.2.19
    • Component/s: Sharding
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Sprint:
      Sharding 2018-01-29, Sharding 2018-02-12
    • Case:

      Description

      Mongod 3.2.18 crashes with segfault during upgrade of config servers (primary running with sccc AND RS). The crash occured while primary config node was stepped down prior to sccc mode being removed.
      Here's the parsed stack trace using http://downloads.mongodb.org/linux/mongodb-linux-x86_64-rhel62-debugsymbols-3.2.18.tgz:

       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/util/stacktrace_posix.cpp:171:0: mongo::printStackTrace(std::ostream&)
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/util/signal_handlers_synchronous.cpp:180:0: mongo::(anonymous namespace)::printSignalAndBacktrace(int)
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/util/signal_handlers_synchronous.cpp:276:0: mongo::(anonymous namespace)::abruptQuitWithAddrSignal(int, siginfo*, void*)
       ??:0:0: ??
       ??:0:0: ??
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/bits/char_traits.h:271:0: std::char_traits<char>::copy(char*, char const*, unsigned long)
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/bits/basic_string.h:359:0: std::string::_M_copy(char*, char const*, unsigned long)
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/bits/basic_string.h:404:0: std::string::_S_copy_chars(char*, char const*, char const*)
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/bits/basic_string.tcc:140:0: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag)
       /data/mci/f5d89cb468d530e4577e158d8bd78d5e/toolchain-builder/build-gcc-v1.sh-vDX/x86_64-mongodb-linux/libstdc++-v3/include/bits/basic_string.h:1725:0: _S_construct_aux<char const*>
       /data/mci/f5d89cb468d530e4577e158d8bd78d5e/toolchain-builder/build-gcc-v1.sh-vDX/x86_64-mongodb-linux/libstdc++-v3/include/bits/basic_string.h:1746:0: _S_construct<char const*>
       /data/mci/f5d89cb468d530e4577e158d8bd78d5e/toolchain-builder/build-gcc-v1.sh-vDX/x86_64-mongodb-linux/libstdc++-v3/include/bits/basic_string.tcc:207:0: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, unsigned long, std::allocator<char> const&)
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/base/string_data.h:177:0: mongo::StringData::toString() const
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/client/connection_string.cpp:46:0: mongo::ConnectionString::ConnectionString(mongo::StringData, std::vector<mongo::HostAndPort, std::allocator<mongo::HostAndPort> >)
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/client/connection_string.cpp:83:0: mongo::ConnectionString::forReplicaSet(mongo::StringData, std::vector<mongo::HostAndPort, std::allocator<mongo::HostAndPort> >)
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/s/client/sharding_network_connection_hook.cpp:83:0: mongo::ShardingNetworkConnectionHook::validateHostImpl(mongo::HostAndPort const&, mongo::executor::RemoteCommandResponse const&, bool)
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/s/sharding_initialization.cpp:173:0: operator()
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/functional:2057:0: std::_Function_handler<mongo::Status (mongo::HostAndPort const&, mongo::executor::RemoteCommandResponse const&), mongo::initializeGlobalShardingState(mongo::OperationContext*, mongo::ConnectionString const&, bool)::'lambda'(mongo::HostAndPort const&, mongo::executor::RemoteCommandResponse const&)>::_M_invoke(std::_Any_data const&, mongo::HostAndPort const&, mongo::executor::RemoteCommandResponse const&)
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/functional:2464:0: std::function<mongo::Status (mongo::HostAndPort const&, mongo::executor::RemoteCommandResponse const&)>::operator()(mongo::HostAndPort const&, mongo::executor::RemoteCommandResponse const&) const
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/client/syncclusterconnection.cpp:207:0: operator()
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/functional:2057:0: std::_Function_handler<mongo::Status (mongo::executor::RemoteCommandResponse const&), mongo::SyncClusterConnection::_connect(std::string const&)::'lambda'(mongo::executor::RemoteCommandResponse const&)>::_M_invoke(std::_Any_data const&, mongo::executor::RemoteCommandResponse const&)
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/functional:2464:0: std::function<mongo::Status (mongo::executor::RemoteCommandResponse const&)>::operator()(mongo::executor::RemoteCommandResponse const&) const
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/client/dbclient.cpp:950:0: mongo::DBClientConnection::connect(mongo::HostAndPort const&)
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/client/syncclusterconnection.cpp:215:0: mongo::SyncClusterConnection::_connect(std::string const&)
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/client/syncclusterconnection.cpp:71:0: mongo::SyncClusterConnection::SyncClusterConnection(std::list<mongo::HostAndPort, std::allocator<mongo::HostAndPort> > const&, double)
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/client/connection_string_connect.cpp:77:0: mongo::ConnectionString::connect(std::string&, double) const
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/client/connpool.cpp:292:0: mongo::DBConnectionPool::get(std::string const&, double)
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/client/connpool.cpp:494:0: mongo::ScopedDbConnection::ScopedDbConnection(std::string const&, double)
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/s/catalog/legacy/legacy_dist_lock_pinger.cpp:71:0: mongo::LegacyDistLockPinger::_distLockPingThread(mongo::ConnectionString, std::string const&, std::chrono::duration<long, std::ratio<1l, 1000l> >)
       /data/mci/c1844f0cf65f07c4d744b1b1343bf2dd/src/src/mongo/s/catalog/legacy/legacy_dist_lock_pinger.cpp:213:0: mongo::LegacyDistLockPinger::distLockPingThread(mongo::ConnectionString, long long, std::string const&, std::chrono::duration<long, std::ratio<1l, 1000l> >)
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/functional:601:0: operator()<mongo::ConnectionString&, long long int&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::chrono::duration<long int, std::ratio<1l, 1000l> >&, void>
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/functional:1296:0: __call<void, 0ul, 1ul, 2ul, 3ul, 4ul>
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/functional:1355:0: operator()<, void>
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/functional:1732:0: _M_invoke<>
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/functional:1720:0: std::_Bind_simple<std::_Bind<std::_Mem_fn<void (mongo::LegacyDistLockPinger::*)(mongo::ConnectionString, long long, std::string const&, std::chrono::duration<long, std::ratio<1l, 1000l> >)> (mongo::LegacyDistLockPinger*, mongo::ConnectionString, long long, std::string, std::chrono::duration<long, std::ratio<1l, 1000l> >)> ()>::operator()()
       /opt/mongodbtoolchain/v1/include/c++/4.8.2/thread:115:0: std::thread::_Impl<std::_Bind_simple<std::_Bind<std::_Mem_fn<void (mongo::LegacyDistLockPinger::*)(mongo::ConnectionString, long long, std::string const&, std::chrono::duration<long, std::ratio<1l, 1000l> >)> (mongo::LegacyDistLockPinger*, mongo::ConnectionString, long long, std::string, std::chrono::duration<long, std::ratio<1l, 1000l> >)> ()> >::_M_run()
       /data/mci/f5d89cb468d530e4577e158d8bd78d5e/toolchain-builder/build-gcc-v1.sh-vDX/x86_64-mongodb-linux/libstdc++-v3/src/c++11/../../../../../gcc-4.8.2/libstdc++-v3/src/c++11/thread.cc:84:0: execute_native_thread_routine
       ??:0:0: ??
       ??:0:0: ??
      

      The same root issue as in SERVER-25029 and a similar fix needs to be made here:
      https://github.com/mongodb/mongo/blob/r3.2.18/src/mongo/s/client/sharding_network_connection_hook.cpp?utf8=%E2%9C%93#L81-L85

      The lines https://github.com/mongodb/mongo/blob/r3.2.18/src/mongo/s/client/sharding_connection_hook.cpp#L185-L187 are missing in the code pointed out.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: