-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: Replication
-
None
-
Fully Compatible
-
ALL
-
-
Repl 2016-10-10
The issue here was that the CollectionBulkLoaderImpl was created in the runner task, and init was called in the task which led to a failure where the loader wasn't returned. The destructor was called, which waited for runner to join, which couldn't happen since this was within the active task in the runner.
Original description
The CollectionBulkLoaderImpl is constructed with an active TaskRunner because it is given the same instance as StorageInterfaceImpl::createCollectionForBulkLoading() created. If CollectionBulkLoaderImpl::init() returns an error status, then CollectionBulkLoaderImpl::commit() is never called, which means the CollectionBulkLoaderImpl never calls TaskRunner::runSynchronousTask() to set TaskRunner::_active back to false.
Thread 3 (Thread 0x7f3dacc7f700 (LWP 28390)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3dc7f8815c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/mci/toolchain-builder/build-gcc-v2.sh-cAV/x86_64-mongodb-linux/libstdc++-v3/include/x86_64-mongodb-linux/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../gcc-5.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007f3dc6d17c6b in wait<(lambda at src/mongo/db/repl/task_runner.cpp:130:25)> (this=0x7f3dccca3938, __lock=..., __p=...) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/condition_variable:98 #4 mongo::repl::TaskRunner::join (this=0x7f3dccca3900) at src/mongo/db/repl/task_runner.cpp:130 #5 0x00007f3dc6d59498 in mongo::repl::CollectionBulkLoaderImpl::~CollectionBulkLoaderImpl (this=0x7f3dccca3a80) at src/mongo/db/repl/collection_bulk_loader_impl.cpp:81 #6 0x00007f3dc6d59cbe in mongo::repl::CollectionBulkLoaderImpl::~CollectionBulkLoaderImpl (this=0x7f3dccca3a80) at src/mongo/db/repl/collection_bulk_loader_impl.cpp:80 #7 0x00007f3dc6d57af5 in operator() (__ptr=0x7f3dccca393c, this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/bits/unique_ptr.h:76 #8 ~unique_ptr (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/bits/unique_ptr.h:236 #9 operator() (this=<optimized out>, txn=0x7f3dccb3f2c0) at src/mongo/db/repl/storage_interface_impl.cpp:297 #10 std::_Function_handler<mongo::Status (mongo::OperationContext*), mongo::repl::StorageInterfaceImpl::createCollectionForBulkLoading(mongo::NamespaceString const&, mongo::CollectionOptions const&, mongo::BSONObj, std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> > const&)::$_0>::_M_invoke(std::_Any_data const&, mongo::OperationContext*&&) (__functor=..., __args=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/functional:1856 #11 0x00007f3dc6d192aa in operator() (this=0x80, __args=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/functional:2271 #12 operator() (this=0x7f3dc95d6bf0, txn=<optimized out>, taskStatus=...) at src/mongo/db/repl/task_runner.cpp:230 #13 std::_Function_handler<mongo::repl::TaskRunner::NextAction (mongo::OperationContext*, mongo::Status const&), mongo::repl::TaskRunner::runSynchronousTask(std::function<mongo::Status (mongo::OperationContext*)>, mongo::repl::TaskRunner::NextAction)::$_4>::_M_invoke(std::_Any_data const&, mongo::OperationContext*&&, mongo::Status const&) (__functor=..., __args=..., __args=...) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/functional:1856 #14 0x00007f3dc6d189ed in operator() (this=0x7f3dccca393c, __args=..., __args=...) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/functional:2271 #15 mongo::repl::(anonymous namespace)::runSingleTask(std::function<mongo::repl::TaskRunner::NextAction (mongo::OperationContext*, mongo::Status const&)> const&, mongo::OperationContext*, mongo::Status const&) (task=..., txn=<optimized out>, status=...) at src/mongo/db/repl/task_runner.cpp:66 #16 0x00007f3dc6d182c0 in mongo::repl::TaskRunner::_runTasks (this=0x7f3dccca3900) at src/mongo/db/repl/task_runner.cpp:151 #17 0x00007f3dc611d8ad in operator() (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/functional:2271 #18 mongo::ThreadPool::_doOneTask (this=0x7f3dcd065dc0, lk=0x7f3dacc7e6d0) at src/mongo/util/concurrency/thread_pool.cpp:326 #19 0x00007f3dc611ebcd in mongo::ThreadPool::_consumeTasks (this=0x7f3dcd065dc0) at src/mongo/util/concurrency/thread_pool.cpp:278 #20 0x00007f3dc611e5d5 in mongo::ThreadPool::_workerThreadBody (pool=0x7f3dcd065dc0, threadName=...) at src/mongo/util/concurrency/thread_pool.cpp:228 #21 0x00007f3dc69162d0 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../gcc-5.3.0/libstdc++-v3/src/c++11/thread.cc:84 #22 0x00007f3dc302e184 in start_thread (arg=0x7f3dacc7f700) at pthread_create.c:312 #23 0x00007f3dc2d5b37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 ... Thread 9 (Thread 0x7f3daec83700 (LWP 28354)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3dc7f8815c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/mci/toolchain-builder/build-gcc-v2.sh-cAV/x86_64-mongodb-linux/libstdc++-v3/include/x86_64-mongodb-linux/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../gcc-5.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007f3dc6d18f7b in wait<(lambda at src/mongo/db/repl/task_runner.cpp:250:31)> (this=0x100000000, __lock=..., __p=...) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/condition_variable:98 #4 mongo::repl::TaskRunner::runSynchronousTask(std::function<mongo::Status (mongo::OperationContext*)>, mongo::repl::TaskRunner::NextAction) (this=<optimized out>, func=..., nextAction=<optimized out>) at src/mongo/db/repl/task_runner.cpp:250 #5 0x00007f3dc6d54454 in mongo::repl::StorageInterfaceImpl::createCollectionForBulkLoading (this=<optimized out>, nss=..., options=..., idIndexSpec=..., secondaryIndexSpecs=...) at src/mongo/db/repl/storage_interface_impl.cpp:255 #6 0x00007f3dc7073bba in mongo::repl::CollectionCloner::_beginCollectionCallback (this=0x7f3dccb92690, cbd=...) at src/mongo/db/repl/collection_cloner.cpp:341 #7 0x00007f3dc7074bec in operator() (this=<optimized out>, __args=...) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/functional:2271 #8 operator() (this=<optimized out>, txn=<optimized out>, status=...) at src/mongo/db/repl/collection_cloner.cpp:118 #9 std::_Function_handler<mongo::repl::TaskRunner::NextAction (mongo::OperationContext*, mongo::Status const&), mongo::repl::CollectionCloner::CollectionCloner(mongo::executor::TaskExecutor*, mongo::OldThreadPool*, mongo::HostAndPort const&, mongo::NamespaceString const&, mongo::CollectionOptions const&, std::function<void (mongo::Status const&)> const&, mongo::repl::StorageInterface*)::$_0::operator()(std::function<void (mongo::executor::TaskExecutor::CallbackArgs const&)> const&) const::{lambda(mongo::OperationContext*, mongo::Status const&)#1}>::_M_invoke(std::_Any_data const&, mongo::OperationContext*&&, mongo::Status const&) (__functor=..., __args=..., __args=...) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/functional:1856 #10 0x00007f3dc6d189ed in operator() (this=0x7f3daec82094, __args=..., __args=...) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/functional:2271 #11 mongo::repl::(anonymous namespace)::runSingleTask(std::function<mongo::repl::TaskRunner::NextAction (mongo::OperationContext*, mongo::Status const&)> const&, mongo::OperationContext*, mongo::Status const&) (task=..., txn=<optimized out>, status=...) at src/mongo/db/repl/task_runner.cpp:66 #12 0x00007f3dc6d182c0 in mongo::repl::TaskRunner::_runTasks (this=0x7f3dccb92c18) at src/mongo/db/repl/task_runner.cpp:151 #13 0x00007f3dc611d8ad in operator() (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/functional:2271 #14 mongo::ThreadPool::_doOneTask (this=0x7f3dcd064000, lk=0x7f3daec826d0) at src/mongo/util/concurrency/thread_pool.cpp:326 #15 0x00007f3dc611ebcd in mongo::ThreadPool::_consumeTasks (this=0x7f3dcd064000) at src/mongo/util/concurrency/thread_pool.cpp:278 #16 0x00007f3dc611e5d5 in mongo::ThreadPool::_workerThreadBody (pool=0x7f3dcd064000, threadName=...) at src/mongo/util/concurrency/thread_pool.cpp:228 #17 0x00007f3dc69162d0 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../gcc-5.3.0/libstdc++-v3/src/c++11/thread.cc:84 #18 0x00007f3dc302e184 in start_thread (arg=0x7f3daec83700) at pthread_create.c:312 #19 0x00007f3dc2d5b37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
This issue does not affect the version of initial sync in MongoDB 3.2.
python buildscripts/resmoke.py --executor=replica_sets jstests/replsets/buildindexes.js --mongodSetParameters='{use3dot2InitialSync: true, initialSyncOplogBuffer: "inMemoryBlockingQueue"}'
- depends on
-
SERVER-25131 CollectionBulkLoaderImpl should release locks on task errors
- Closed
- is duplicated by
-
SERVER-26341 TaskRunner destroyed prior to task return
- Closed
- is related to
-
SERVER-25725 Running {setFeatureCompatibilityVersion: "3.4"} on a 3.4 primary should kill 3.2 secondaries
- Closed
- related to
-
SERVER-26335 Initial sync with buildIndexes=false fails
- Closed