Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-28319

Invariant failure with 1000 of fake shards in mongos

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.2
    • Component/s: Querying, Sharding
    • Labels:
      None
    • Query
    • ALL

      CSRS with 1 config server and 3 real shards (standalone), 1 mongos

      Added 1000 fake shards in config metadata and assigned chunks to some of them for the sharded collection "test.test". Restarted the cluster.

      When doing db.test.find({}), mongos crashes with

      2017-03-15T10:27:04.575-0400 I -        [conn1] Invariant failure remote.status.isOK() src/mongo/s/query/async_results_merger.cpp 345"

      Doesn't reproduce without fake shards or with a single fake shard. The latter leads me to believe that it's not a problem with the way I introduce the fake shards.

      Logs and config metadata attached.

      The call stack of the crash is:

      2017-03-15T10:04:47.448-0400 I -        [conn1] Invariant failure remote.status.isOK() src/mongo/s/query/async_results_merger.cpp 345
      ...
       0x1017b4a0a 0x1017b4230 0x7fff814d1f1a 0x102df9e00 0x7fff85e4c9b3 0x10174869e 0x1016e9670 0x1016f7b5b 0x1016edc52 0x1016f45ce 0x10168af8a 0x101705b1b 0x1016d89ae 0x1016d7907 0x101191b18 0x10174268d 0x101742d41 0x7fff8162405a 0x7fff81623fd7 0x7fff816213ed
      ----- BEGIN BACKTRACE -----
       mongos(mongo::printStackTrace(std::__1::basic_ostream<char, std::__1::char_traits<char> >&) 0x3A) [0x1017b4a0a]
       mongos(mongo::(anonymous namespace)::abruptQuit(int) 0xC0) [0x1017b4230]
       libsystem_platform.dylib(_sigtramp 0x1A) [0x7fff814d1f1a]
       ??? [0x102df9e00]
       libsystem_c.dylib(abort 0x81) [0x7fff85e4c9b3]
       mongos(mongo::invariantFailed(char const*, char const*, unsigned int) 0x2FE) [0x10174869e]
       mongos(mongo::AsyncResultsMerger::nextEvent() 0x2F0) [0x1016e9670]
       mongos(mongo::RouterStageMerge::next() 0x5B) [0x1016f7b5b]
       mongos(mongo::ClusterClientCursorImpl::next() 0xA2) [0x1016edc52]
       mongos(mongo::ClusterFind::runQuery(mongo::OperationContext*, mongo::CanonicalQuery const&, mongo::ReadPreferenceSetting const&, std::__1::vector<mongo::BSONObj, std::__1::allocator<mongo::BSONObj> >*, mongo::BSONObj*) 0x1A9E) [0x1016f45ce]
       mongos(mongo::(anonymous namespace)::ClusterFindCmd::run(mongo::OperationContext*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, mongo::BSONObj&, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, mongo::BSONObjBuilder&) 0x4DA) [0x10168af8a]
       mongos(mongo::Command::execCommandClient(mongo::OperationContext*, mongo::Command*, int, char const*, mongo::BSONObj&, mongo::BSONObjBuilder&) 0x6FB) [0x101705b1b]
       mongos(mongo::(anonymous namespace)::runAgainstRegistered(mongo::OperationContext*, char const*, mongo::BSONObj&, mongo::BSONObjBuilder&, int) 0x12E) [0x1016d89ae]
       mongos(mongo::Strategy::clientCommandOp(mongo::OperationContext*, mongo::NamespaceString const&, mongo::DbMessage*) 0xC47) [0x1016d7907]
       mongos(mongo::ServiceEntryPointMongos::_sessionLoop(std::__1::shared_ptr<mongo::transport::Session> const&) 0x9F8) [0x101191b18]
       mongos(mongo::(anonymous namespace)::runFunc(void*) 0x2AD) [0x10174268d]
       mongos(void* std::__1::__thread_proxy<std::__1::tuple<std::__1::__bind<void* (&)(void*), mongo::(anonymous namespace)::Context*> > >(void*) 0x61) [0x101742d41]
       libsystem_pthread.dylib(_pthread_body 0x83) [0x7fff8162405a]
       libsystem_pthread.dylib(_pthread_body 0x0) [0x7fff81623fd7]
       libsystem_pthread.dylib(thread_start 0xD) [0x7fff816213ed]
      -----  END BACKTRACE  -----
      

        1. config_dump.tgz
          36 kB
        2. mongos.log
          319 kB

            Assignee:
            backlog-server-query Backlog - Query Team (Inactive)
            Reporter:
            alex.komyagin@mongodb.com Alexander Komyagin (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: