Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-47412

Make the database cloner ensure collections are actually unsharded before cloning

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Sharding
    • Sprint:
      Sharding 2020-04-20

      Description

      The operation observers are being used, among other reasons, to maintain internal statistics on the collection and database usage, which can be used, for example, to detect hotspots.

      Changing getCollectionDescription_DEPRECATED for the intended getCollectionDescription that now throws on SSV on the sharding operation observers might cause a backtrace similar to the one below:

       src/mongo/util/stacktrace_posix.cpp:253:21: LibunwindStepIteration
       src/mongo/util/stacktrace_posix.cpp:424:36: mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) (.constprop.587)
       src/mongo/util/stacktrace_posix.cpp:477:44: mongo::printStackTrace()
       src/mongo/util/assert_util.cpp:81:24: mongo::DBException::traceIfNeeded(mongo::DBException const&)
       src/mongo/util/assert_util.h:134:22: mongo::DBException::DBException(mongo::Status const&)
       src/mongo/util/assert_util.h:151:66: _ZN5mongo18AssertionExceptionC4ERKNS_6StatusE
       src/mongo/util/assert_util.h:182:71: mongo::error_details::ExceptionForImpl<(mongo::ErrorCodes::Error)13388, mongo::ExceptionForCat<(mongo::ErrorCategory)3>, mongo::ExceptionForCat<(mongo::ErrorCategory)4> >::ExceptionForImpl(mongo::Status const&)
       build/optdebug/mongo/base/error_codes.cpp:2393:63: mongo::error_details::throwExceptionForStatus(mongo::Status const&)
       src/mongo/util/assert_util.cpp:257:43: mongo::uassertedWithLocation(mongo::Status const&, char const*, unsigned int)
       src/mongo/db/s/collection_sharding_runtime.cpp:151:5: operator()
       src/mongo/db/s/collection_sharding_runtime.cpp:151:5: mongo::CollectionShardingRuntime::getCollectionDescription() (.cold.960)
       src/mongo/db/s/shard_server_op_observer.cpp:241:57: mongo::ShardServerOpObserver::onInserts(mongo::OperationContext*, mongo::NamespaceString const&, boost::optional<mongo::UUID>, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, bool)
       src/mongo/s/server.cpp:370:52: mongo::(anonymous namespace)::initializeSharding(mongo::OperationContext*)
       src/mongo/db/catalog/collection_impl.cpp:491:58: mongo::CollectionImpl::insertDocuments(mongo::OperationContext*, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, mongo::OpDebug*, bool)
       src/mongo/db/catalog/collection_impl.cpp:531:81: mongo::CollectionImpl::insertDocument(mongo::OperationContext*, mongo::InsertStatement const&, mongo::OpDebug*, bool)
       src/mongo/db/cloner.cpp:240:94: mongo::Cloner::Fun::operator()(mongo::DBClientCursorBatchIterator&)::'lambda8'()::operator()() const
       src/mongo/db/concurrency/write_conflict_exception.h:94:21: auto mongo::writeConflictRetry<mongo::Cloner::Fun::operator()(mongo::DBClientCursorBatchIterator&)::'lambda8'()>(mongo::OperationContext*, mongo::StringData, mongo::StringData, mongo::Cloner::Fun::operator()(mongo::DBClientCursorBatchIterator&)::'lambda8'()&&)
       src/mongo/db/cloner.cpp:232:31: mongo::Cloner::Fun::operator()(mongo::DBClientCursorBatchIterator&)
       /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void (mongo::DBClientCursorBatchIterator&)>::operator()(mongo::DBClientCursorBatchIterator&) const
       src/mongo/client/dbclient_base.cpp:824:10: mongo::DBClientBase::query(std::function<void (mongo::DBClientCursorBatchIterator&)>, mongo::NamespaceStringOrUUID const&, mongo::Query, mongo::BSONObj const*, int, int, boost::optional<mongo::BSONObj>)
       src/mongo/db/cloner.cpp:311:21: mongo::Cloner::copy(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::NamespaceString const&, mongo::BSONObj const&, mongo::BSONObj const&, mongo::NamespaceString const&, mongo::CloneOptions const&, mongo::Query)
       src/mongo/db/cloner.cpp:839:17: mongo::Cloner::copyDb(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::CloneOptions const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> >)
       src/mongo/db/s/clone_catalog_data_command.cpp:136:9: mongo::(anonymous namespace)::CloneCatalogDataCommand::run(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, mongo::BSONObjBuilder&)
       src/mongo/db/commands.h:799:19: mongo::BasicCommand::runWithReplyBuilder(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, mongo::rpc::ReplyBuilderInterface*)
       src/mongo/db/commands.cpp:756:48: mongo::BasicCommandWithReplyBuilderInterface::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)
       src/mongo/db/commands.cpp:184:20: mongo::CommandHelpers::runCommandInvocation(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*)
       src/mongo/db/service_entry_point_common.cpp:829:53: mongo::(anonymous namespace)::runCommandImpl(mongo::OperationContext*, mongo::CommandInvocation*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::LogicalTime, mongo::ServiceEntryPointCommon::Hooks const&, mongo::BSONObjBuilder*, mongo::OperationSessionInfoFromClient const&)
       src/mongo/db/service_entry_point_common.cpp:1186:32: mongo::(anonymous namespace)::execCommandDatabase(mongo::OperationContext*, mongo::Command*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::ServiceEntryPointCommon::Hooks const&)
       src/mongo/db/service_entry_point_common.cpp:1362:32: operator()
       src/mongo/db/service_entry_point_common.cpp:1382:6: mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)
       src/mongo/db/service_entry_point_common.cpp:1686:38: mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)
       src/mongo/db/service_entry_point_mongod.cpp:277:68: mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)
       src/mongo/transport/service_state_machine.cpp:470:72: mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard)
       src/mongo/transport/service_state_machine.cpp:548:32: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
       src/mongo/transport/service_state_machine.cpp:592:29: operator()
       /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::'lambda'()>::_M_invoke(std::_Any_data const&)
       /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
       src/mongo/transport/service_executor_synchronous.cpp:107:17: mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)
       src/mongo/transport/service_state_machine.cpp:595:80: mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)
       src/mongo/transport/service_state_machine.cpp:373:38: mongo::ServiceStateMachine::_sourceCallback(mongo::Status)
       src/mongo/transport/service_state_machine.cpp:329:24: operator()
       src/mongo/util/future_impl.h:237:16: auto mongo::future_details::call<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::'lambda0'(mongo::StatusWith<mongo::Message>)&, mongo::StatusWith<mongo::Message> >(mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::'lambda0'(mongo::StatusWith<mongo::Message>)&, mongo::StatusWith<mongo::Message>&&)
       src/mongo/util/future_impl.h:851:32: operator()
       src/mongo/util/future_impl.h:1163:27: generalImpl<mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda(mongo::Message&&)>, mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda(mongo::Status&&)>, mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda()> >
       src/mongo/util/future_impl.h:865:14: getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)> >
       src/mongo/util/future.h:348:9: getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)> >
       src/mongo/transport/service_state_machine.cpp:324:29: mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)
       src/mongo/transport/service_state_machine.cpp:545:31: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
       src/mongo/transport/service_state_machine.cpp:592:29: operator()
       /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::'lambda'()>::_M_invoke(std::_Any_data const&)
       /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
       src/mongo/transport/service_executor_synchronous.cpp:124:36: operator()
       /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)::'lambda0'()>::_M_invoke(std::_Any_data const&)
       /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
       src/mongo/transport/service_entry_point_utils.cpp:100:14: operator()
       /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::launchServiceWorkerThread(std::function<void ()>)::'lambda1'()>::_M_invoke(std::_Any_data const&)
       /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
       src/mongo/transport/service_entry_point_utils.cpp:58:15: mongo::(anonymous namespace)::runFunc(void*)
       ??:0:0: start_thread
       /build/glibc-OTsEL5/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95:0: clone
      

      This is caused to the fact that on the observer it is not expected a SSV when inserting data to a collection. This form part of the new assurances of the CollectionShardingState API and is being enforced as part of the project PM-1645.

      The cloner should refresh the filtering metadata and make sure all the collections are unsharded before doing the copy operation.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              marcos.grillo Marcos José Grillo Ramirez
              Reporter:
              marcos.grillo Marcos José Grillo Ramirez
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: