Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-31746

Invariant failure on sharded $changeStream UUID mismatch

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.6.0-rc1
    • Labels:
      None
    • ALL

      On mongod, when we find that the UUID of a sharded collection does not match the UUID expected by the $changeStream pipeline, we uassert and throw a plain AssertionException with ErrorCodes::StaleConfig. However, this trips the invariant at service_entry_point_mongod.cpp#760, which requires that all ErrorCodes::StaleConfig exceptions be of type StaleConfigException.

      Stacktrace:

          frame #0: 0x00007fff66d20fce libsystem_kernel.dylib`__pthread_kill + 10
          frame #1: 0x00007fff66e5e150 libsystem_pthread.dylib`pthread_kill + 333
          frame #2: 0x00007fff66c7d32a libsystem_c.dylib`abort + 127
          frame #3: 0x000000010dd1afce mongod`mongo::invariantFailed(expr="sce", file="src/mongo/db/service_entry_point_mongod.cpp", line=760) at assert_util.cpp:130
          frame #4: 0x000000010af3e5a6 mongod`mongo::(anonymous namespace)::execCommandDatabase(opCtx=0x00007f9c7fd60460, command=0x000000010eb3bec0, request=0x000070000d9e9810, replyBuilder=0x00007f9c7fd5af40) at service_entry_point_mongod.cpp:760
          frame #5: 0x000000010af3acc3 mongod`mongo::(anonymous namespace)::runCommands(this=0x000070000d9e9cf0)::$_4::operator()() const at service_entry_point_mongod.cpp:871
          frame #6: 0x000000010af362f1 mongod`mongo::(anonymous namespace)::runCommands(opCtx=0x00007f9c7fd60460, message=0x00007f9c7ff9d008) at service_entry_point_mongod.cpp:824
          frame #7: 0x000000010af34d6e mongod`mongo::ServiceEntryPointMongod::handleRequest(this=0x00007f9c7fe00440, opCtx=0x00007f9c7fd60460, m=0x00007f9c7ff9d008) at service_entry_point_mongod.cpp:1135
          frame #8: 0x000000010af55857 mongod`mongo::ServiceStateMachine::_processMessage(this=0x00007f9c7ff9cf60, guard=0x000070000d9eb098) at service_state_machine.cpp:307
          frame #9: 0x000000010af54bb0 mongod`mongo::ServiceStateMachine::_runNextInGuard(this=0x00007f9c7ff9cf60, guard=0x000070000d9eb098) at service_state_machine.cpp:401
          frame #10: 0x000000010af55456 mongod`mongo::ServiceStateMachine::runNext(this=0x00007f9c7ff9cf60) at service_state_machine.cpp:365
          frame #11: 0x000000010af63198 mongod`mongo::ServiceStateMachine::scheduleNext(this=0x000070000d9eb408)::$_4::operator()() const at service_state_machine.cpp:429
          frame #12: 0x000000010af63175 mongod`void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(this=0x000070000d9eb408)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()::operator()() const at service_state_machine.h:164
          frame #13: 0x000000010af6314d mongod`void std::__1::__invoke_void_return_wrapper<void>::__call<void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()&>(void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()&&&) [inlined] decltype(__f=0x000070000d9eb408)::$_4>(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()&>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&) at type_traits:4291
          frame #14: 0x000000010af6313c mongod`void std::__1::__invoke_void_return_wrapper<void>::__call<void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(__args=0x000070000d9eb408)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()&>(void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()&&&) at __functional_base:359
          frame #15: 0x000000010af62e89 mongod`std::__1::__function::__func<void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'(), std::__1::allocator<void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()>, void ()>::operator(this=0x000070000d9eb400)() at functional:1552
          frame #16: 0x000000010ae0b7ab mongod`std::__1::function<void ()>::operator(this=0x000070000d9eb400)() const at functional:1911
          frame #17: 0x000000010cf56fd7 mongod`mongo::transport::ServiceExecutorSynchronous::schedule(this=0x00007f9c7fe00d90, task=0x000070000d9eb400, flags=kMayRecurse)>, mongo::transport::ServiceExecutor::ScheduleFlags) at service_executor_synchronous.cpp:118
          frame #18: 0x000000010af56577 mongod`void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(this=0x00007f9c7ff9cf60, func=0x000070000d9eb460, flags=kMayRecurse)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags) at service_state_machine.h:163
          frame #19: 0x000000010af54437 mongod`mongo::ServiceStateMachine::scheduleNext(this=0x00007f9c7ff9cf60, flags=kMayRecurse) at service_state_machine.cpp:429
          frame #20: 0x000000010af539ca mongod`mongo::ServiceStateMachine::_sourceCallback(this=0x00007f9c7ff9cf60, status=Status @ 0x000070000d9eb9b8) at service_state_machine.cpp:228
          frame #21: 0x000000010af54828 mongod`mongo::ServiceStateMachine::_runNextInGuard(this=0x00007f9c7ff9cf60, guard=0x000070000d9ebb18) at service_state_machine.cpp:388
          frame #22: 0x000000010af55456 mongod`mongo::ServiceStateMachine::runNext(this=0x00007f9c7ff9cf60) at service_state_machine.cpp:365
          frame #23: 0x000000010af63198 mongod`mongo::ServiceStateMachine::scheduleNext(this=0x00007f9c81843e38)::$_4::operator()() const at service_state_machine.cpp:429
          frame #24: 0x000000010af63175 mongod`void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(this=0x00007f9c81843e38)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()::operator()() const at service_state_machine.h:164
          frame #25: 0x000000010af6314d mongod`void std::__1::__invoke_void_return_wrapper<void>::__call<void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()&>(void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()&&&) [inlined] decltype(__f=0x00007f9c81843e38)::$_4>(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()&>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&) at type_traits:4291
          frame #26: 0x000000010af6313c mongod`void std::__1::__invoke_void_return_wrapper<void>::__call<void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(__args=0x00007f9c81843e38)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()&>(void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()&&&) at __functional_base:359
          frame #27: 0x000000010af62e89 mongod`std::__1::__function::__func<void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'(), std::__1::allocator<void mongo::ServiceStateMachine::_scheduleFunc<mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::ServiceStateMachine::scheduleNext(mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&, mongo::transport::ServiceExecutor::ScheduleFlags)::'lambda'()>, void ()>::operator(this=0x00007f9c81843e30)() at functional:1552
          frame #28: 0x000000010ae0b7ab mongod`std::__1::function<void ()>::operator(this=0x00007f9c81843e30)() const at functional:1911
          frame #29: 0x000000010cf5966a mongod`mongo::transport::ServiceExecutorSynchronous::schedule(this=0x00007f9c7ff9d1a0)>, mongo::transport::ServiceExecutor::ScheduleFlags)::$_2::operator()() const at service_executor_synchronous.cpp:135
          frame #30: 0x000000010cf5952d mongod`void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::transport::ServiceExecutorSynchronous::schedule(std::__1::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::$_2&>(mongo::transport::ServiceExecutorSynchronous::schedule(std::__1::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::$_2&&&) [inlined] decltype(__f=0x00007f9c7ff9d1a0)>, mongo::transport::ServiceExecutor::ScheduleFlags)::$_2&>(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<mongo::transport::ServiceExecutorSynchronous::schedule(std::__1::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::$_2&>(mongo::transport::ServiceExecutorSynchronous::schedule(std::__1::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::$_2&&&) at type_traits:4291
          frame #31: 0x000000010cf5951c mongod`void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::transport::ServiceExecutorSynchronous::schedule(__args=0x00007f9c7ff9d1a0)>, mongo::transport::ServiceExecutor::ScheduleFlags)::$_2&>(mongo::transport::ServiceExecutorSynchronous::schedule(std::__1::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::$_2&&&) at __functional_base:359
          frame #32: 0x000000010cf592e9 mongod`std::__1::__function::__func<mongo::transport::ServiceExecutorSynchronous::schedule(std::__1::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::$_2, std::__1::allocator<mongo::transport::ServiceExecutorSynchronous::schedule(std::__1::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::$_2>, void ()>::operator(this=0x00007f9c7ff9d190)() at functional:1552
          frame #33: 0x000000010ae0b7ab mongod`std::__1::function<void ()>::operator(this=0x00007f9c7ff9d190)() const at functional:1911
          frame #34: 0x000000010dadec5b mongod`mongo::(anonymous namespace)::runFunc(ctx=0x00007f9c7ff8e920) at service_entry_point_utils.cpp:55
          frame #35: 0x00007fff66e5b6c1 libsystem_pthread.dylib`_pthread_body + 340
          frame #36: 0x00007fff66e5b56d libsystem_pthread.dylib`_pthread_start + 377
          frame #37: 0x00007fff66e5ac5d libsystem_pthread.dylib`thread_start + 13
      

            Assignee:
            Unassigned Unassigned
            Reporter:
            bernard.gorman@mongodb.com Bernard Gorman
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: