Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-41163

During initial sync, failing to apply an update thats in a prepared txn hits an invariant

    • Fully Compatible
    • ALL
    • v4.2
    • Hide

      I've attached the test. Here are the logs.
      Here's the stacktrace:

      [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.171-0400 d34588| 2019-05-15T16:43:12.171-0400 F  -        [repl-writer-worker-2] Invariant failure isCrudOpType() src/mongo/db/repl/oplog_entry.cpp 292
      [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.171-0400 d34588| 2019-05-15T16:43:12.171-0400 F  -        [repl-writer-worker-2]
      [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.171-0400 d34588|
      [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.172-0400 d34588| ***aborting after invariant() failure
      [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.172-0400 d34588|
      [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.172-0400 d34588|
      [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.172-0400 d34588| 2019-05-15T16:43:12.172-0400 F  -        [repl-writer-worker-2] Got signal: 6 (Aborted).
      [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.173-0400 d34588|  0x7faebee287a1 0x7faebee2802e 0x7faebee280cf 0x7faebcc37890 0x7faebc872e97 0x7faebc874801 0x7faebeda8bc6 0x7faebfd07e60 0x7faec1b5e492 0x7faec1b6013a 0x7faec1b6255d 0x7faec1b54de8 0x7faec1b5bffa 0x7faec10cfb16 0x7faec10d0d10 0x7faec10d2755 0x7faebd4b957f 0x7faebcc2c6db 0x7faebc95588f
      [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.173-0400 d34588| ----- BEGIN BACKTRACE -----
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/stacktrace_posix.cpp:174:39: mongo::printStackTrace(std::ostream&)
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/signal_handlers_synchronous.cpp:182:20: mongo::(anonymous namespace)::printSignalAndBacktrace(int)
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/signal_handlers_synchronous.cpp:239:28: mongo::(anonymous namespace)::abruptQuit(int)
       ??:0:0: ??
       /build/glibc-OTsEL5/glibc-2.27/signal/../sysdeps/unix/sysv/linux/nptl-signals.h:80:0: __libc_signal_restore_set
       /build/glibc-OTsEL5/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:48:0: gsignal
       /build/glibc-OTsEL5/glibc-2.27/stdlib/abort.c:79:0: abort
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/assert_util.cpp:104:15: mongo::invariantFailed(char const*, char const*, unsigned int)
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/invariant.h:69:33: void mongo::invariantWithLocation<bool>(bool const&, char const*, char const*, unsigned int)
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/db/repl/oplog_entry.cpp:292:5: mongo::repl::OplogEntry::getIdElement() const [clone .cold.94]
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/db/repl/sync_tail.cpp:895:53: mongo::repl::SyncTail::getMissingDoc(mongo::OperationContext*, mongo::repl::OplogEntry const&)
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/db/repl/sync_tail.cpp:951:57: mongo::repl::SyncTail::fetchAndInsertMissingDocument(mongo::OperationContext*, mongo::repl::OplogEntry const&)
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/db/repl/sync_tail.cpp:1071:58: mongo::repl::multiSyncApply(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*)
       /opt/mongodbtoolchain/revisions/e84eb3fd219668197589e62dba14b9914712642d/stow/gcc-v3.ipd/include/c++/8.2.0/bits/std_function.h:283:40: std::_Function_handler<mongo::Status (mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*), mongo::Status (*)(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*)>::_M_invoke(std::_Any_data const&, mongo::OperationContext*&&, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*&&, mongo::repl::SyncTail*&&, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*&&)
       /opt/mongodbtoolchain/revisions/e84eb3fd219668197589e62dba14b9914712642d/stow/gcc-v3.ipd/include/c++/8.2.0/bits/std_function.h:687:71: std::function<mongo::Status (mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*)>::operator()(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*) const
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/db/repl/sync_tail.cpp:1278:92: operator()
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/interruptible.h:179:47: runWithoutInterruptionExceptAtGlobalShutdown<mongo::repl::SyncTail::_applyOps(std::vector<std::vector<const mongo::repl::OplogEntry*> >&, std::vector<mongo::Status>*, std::vector<std::vector<mongo::MultikeyPathInfo> >*)::<lambda(auto:20)> [with auto:20 = mongo::Status]::<lambda()> >
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/db/repl/sync_tail.cpp:1277:20: operator()<mongo::Status>
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/functional.h:144:10: callRegularVoid<mongo::repl::SyncTail::_applyOps(std::vector<std::vector<const mongo::repl::OplogEntry*> >&, std::vector<mongo::Status>*, std::vector<std::vector<mongo::MultikeyPathInfo> >*)::<lambda(auto:20)> >
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/functional.h:158:39: _ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_4repl8SyncTail9_applyOpsERSt6vectorIS7_IPKNS5_10OplogEntryESaISA_EESaISC_EEPS7_IS1_SaIS1_EEPS7_IS7_INS_16MultikeyPathInfoESaISJ_EESaISL_EEEUlT_E_EEDaOSP_EN12SpecificImpl4callEOS1_
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/functional.h:108:54: mongo::unique_function<void (mongo::Status)>::operator()(mongo::Status) const
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/concurrency/thread_pool.cpp:341:9: mongo::ThreadPool::_doOneTask(std::unique_lock<std::mutex>*)
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/concurrency/thread_pool.cpp:294:19: mongo::ThreadPool::_consumeTasks()
       /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/concurrency/thread_pool.cpp:247:24: mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
       ??:0:0: ??
       ??:0:0: start_thread
       /build/glibc-OTsEL5/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95:0: clone
      [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.176-0400 d34588| -----  END BACKTRACE  -----
      
      Show
      I've attached the test. Here are the logs . Here's the stacktrace: [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.171-0400 d34588| 2019-05-15T16:43:12.171-0400 F - [repl-writer-worker-2] Invariant failure isCrudOpType() src/mongo/db/repl/oplog_entry.cpp 292 [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.171-0400 d34588| 2019-05-15T16:43:12.171-0400 F - [repl-writer-worker-2] [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.171-0400 d34588| [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.172-0400 d34588| ***aborting after invariant() failure [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.172-0400 d34588| [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.172-0400 d34588| [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.172-0400 d34588| 2019-05-15T16:43:12.172-0400 F - [repl-writer-worker-2] Got signal: 6 (Aborted). [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.173-0400 d34588| 0x7faebee287a1 0x7faebee2802e 0x7faebee280cf 0x7faebcc37890 0x7faebc872e97 0x7faebc874801 0x7faebeda8bc6 0x7faebfd07e60 0x7faec1b5e492 0x7faec1b6013a 0x7faec1b6255d 0x7faec1b54de8 0x7faec1b5bffa 0x7faec10cfb16 0x7faec10d0d10 0x7faec10d2755 0x7faebd4b957f 0x7faebcc2c6db 0x7faebc95588f [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.173-0400 d34588| ----- BEGIN BACKTRACE ----- /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/stacktrace_posix.cpp:174:39: mongo::printStackTrace(std::ostream&) /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/signal_handlers_synchronous.cpp:182:20: mongo::(anonymous namespace)::printSignalAndBacktrace(int) /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/signal_handlers_synchronous.cpp:239:28: mongo::(anonymous namespace)::abruptQuit(int) ??:0:0: ?? /build/glibc-OTsEL5/glibc-2.27/signal/../sysdeps/unix/sysv/linux/nptl-signals.h:80:0: __libc_signal_restore_set /build/glibc-OTsEL5/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:48:0: gsignal /build/glibc-OTsEL5/glibc-2.27/stdlib/abort.c:79:0: abort /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/assert_util.cpp:104:15: mongo::invariantFailed(char const*, char const*, unsigned int) /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/invariant.h:69:33: void mongo::invariantWithLocation<bool>(bool const&, char const*, char const*, unsigned int) /mnt/home/vlad/mongodb/github/mongo/src/mongo/db/repl/oplog_entry.cpp:292:5: mongo::repl::OplogEntry::getIdElement() const [clone .cold.94] /mnt/home/vlad/mongodb/github/mongo/src/mongo/db/repl/sync_tail.cpp:895:53: mongo::repl::SyncTail::getMissingDoc(mongo::OperationContext*, mongo::repl::OplogEntry const&) /mnt/home/vlad/mongodb/github/mongo/src/mongo/db/repl/sync_tail.cpp:951:57: mongo::repl::SyncTail::fetchAndInsertMissingDocument(mongo::OperationContext*, mongo::repl::OplogEntry const&) /mnt/home/vlad/mongodb/github/mongo/src/mongo/db/repl/sync_tail.cpp:1071:58: mongo::repl::multiSyncApply(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*) /opt/mongodbtoolchain/revisions/e84eb3fd219668197589e62dba14b9914712642d/stow/gcc-v3.ipd/include/c++/8.2.0/bits/std_function.h:283:40: std::_Function_handler<mongo::Status (mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*), mongo::Status (*)(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*)>::_M_invoke(std::_Any_data const&, mongo::OperationContext*&&, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*&&, mongo::repl::SyncTail*&&, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*&&) /opt/mongodbtoolchain/revisions/e84eb3fd219668197589e62dba14b9914712642d/stow/gcc-v3.ipd/include/c++/8.2.0/bits/std_function.h:687:71: std::function<mongo::Status (mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*)>::operator()(mongo::OperationContext*, std::vector<mongo::repl::OplogEntry const*, std::allocator<mongo::repl::OplogEntry const*> >*, mongo::repl::SyncTail*, std::vector<mongo::MultikeyPathInfo, std::allocator<mongo::MultikeyPathInfo> >*) const /mnt/home/vlad/mongodb/github/mongo/src/mongo/db/repl/sync_tail.cpp:1278:92: operator() /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/interruptible.h:179:47: runWithoutInterruptionExceptAtGlobalShutdown<mongo::repl::SyncTail::_applyOps(std::vector<std::vector<const mongo::repl::OplogEntry*> >&, std::vector<mongo::Status>*, std::vector<std::vector<mongo::MultikeyPathInfo> >*)::<lambda(auto:20)> [with auto:20 = mongo::Status]::<lambda()> > /mnt/home/vlad/mongodb/github/mongo/src/mongo/db/repl/sync_tail.cpp:1277:20: operator()<mongo::Status> /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/functional.h:144:10: callRegularVoid<mongo::repl::SyncTail::_applyOps(std::vector<std::vector<const mongo::repl::OplogEntry*> >&, std::vector<mongo::Status>*, std::vector<std::vector<mongo::MultikeyPathInfo> >*)::<lambda(auto:20)> > /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/functional.h:158:39: _ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_4repl8SyncTail9_applyOpsERSt6vectorIS7_IPKNS5_10OplogEntryESaISA_EESaISC_EEPS7_IS1_SaIS1_EEPS7_IS7_INS_16MultikeyPathInfoESaISJ_EESaISL_EEEUlT_E_EEDaOSP_EN12SpecificImpl4callEOS1_ /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/functional.h:108:54: mongo::unique_function<void (mongo::Status)>::operator()(mongo::Status) const /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/concurrency/thread_pool.cpp:341:9: mongo::ThreadPool::_doOneTask(std::unique_lock<std::mutex>*) /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/concurrency/thread_pool.cpp:294:19: mongo::ThreadPool::_consumeTasks() /mnt/home/vlad/mongodb/github/mongo/src/mongo/util/concurrency/thread_pool.cpp:247:24: mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ??:0:0: ?? ??:0:0: start_thread /build/glibc-OTsEL5/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95:0: clone [js_test:initsync_fuzzer-cacf2-1557881965671-05] 2019-05-15T16:43:12.176-0400 d34588| ----- END BACKTRACE -----
    • Repl 2019-06-03, Repl 2019-06-17, Repl 2019-07-01
    • 8

      When an update fails to get applied due to missing documents, we fetch the missing document using the oplog entry to figure out what document to retrieve.

      However, the oplog entry for the update in this case is the commitTransaction command. Currently, we assert that the oplog entry can only be an insert, delete or update. We thus hit this invariant when we try and retrieve the missing doc.

            Assignee:
            pavithra.vetriselvan@mongodb.com Pavithra Vetriselvan
            Reporter:
            vlad.rachev@mongodb.com Vlad Rachev (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: