[SERVER-20445] Add support for majority read concern level to Aggregation $out Created: 16/Sep/15  Updated: 03/Apr/19  Resolved: 13/Nov/18

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, MapReduce
Affects Version/s: None
Fix Version/s: 4.1.6

Type: Improvement Priority: Major - P3
Reporter: Mathias Stearn Assignee: Martin Neupauer
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-20445.patch     Text File decipheredBt.txt     Text File decipheredBt.txt     Text File readMajorityOut.patch     Text File readMajorityOut.patch    
Issue Links:
Depends
is depended on by DRIVERS-610 Add support for majority read concern... Closed
is depended on by PHPLIB-290 Remove aggregate() and restriction on... Closed
Documented
is documented by DOCS-12598 Docs for SERVER-20445: Add support fo... Closed
Related
related to DRIVERS-564 Add support for majority read concern... Closed
related to SERVER-36925 All write operations with a query com... Closed
related to SERVER-38117 Add support for majority read concern... Closed
related to SERVER-20214 Stop supporting majority read concern... Closed
related to SERVER-36255 Prohibit $out with readConcern level ... Closed
related to DOCS-10647 Document readConcern option for inlin... Closed
Backwards Compatibility: Fully Compatible
Sprint: Query 2018-07-16, Query 2018-07-30, Query 2018-10-22, Query 2018-11-05, Query 2018-11-19
Participants:

 Description   

For SERVER-20214 we are removing support for majority read concern level in these operations because we cannot write in that mode. This ticket is to reenable support.

The current plan is to emulate majority read concern level by waiting for w:majority in the portions of those operations that write (aggregations including $out and all map reduces) and keep the read concern level for the portions that don't (aggregations not including $out including all work from the shard phase). It is unclear right now which category inline map reduce falls under. Part of this work is to ensure that this approach is valid.



 Comments   
Comment by Githook User [ 14/Nov/18 ]

Author:

{'name': 'Martin Neupauer', 'email': 'martin.neupauer@mongodb.com', 'username': 'MartinNeupauer'}

Message: SERVER-20445 quick follow up - enable more tests of $out with readConcern majority.
Branch: master
https://github.com/mongodb/mongo/commit/2f0271a86b127a95cd04d878673d9c097de7a975

Comment by Githook User [ 13/Nov/18 ]

Author:

{'name': 'Martin Neupauer', 'email': 'martin.neupauer@mongodb.com', 'username': 'MartinNeupauer'}

Message: SERVER-20445 Add support for majority read concern level to MapReduce and Aggregation $out.
Branch: master
https://github.com/mongodb/mongo/commit/dc9e1ee045af74c74360ffce2bec88868b08e1dc

Comment by Ian Whalen (Inactive) [ 17/Sep/18 ]

CC david.storch on sprint/assignee change.

Comment by Kyle Suarez [ 11/Sep/18 ]

I ran out of work otherwise and will work on this as much as I can and post any findings here. If I can't finish it and it isn't immediately handed off, I'll throw this back into Query 2018-10-08.

Comment by Kyle Suarez [ 06/Sep/18 ]

I won't have time to get to this in my last week here so we're throwing it back into the pile and pushing it to a future sprint. I've uploaded my work-in-progress here as SERVER-20445.patch, based on commit 0bb846536e3dda6905bd211db4e4a137705eeb51.

Simply removing the prohibition on readConcern level "majority" for modes "insertDocuments" and "replaceDocuments" worked, but I faced a problem when it came to actually performing the inserts/updates. The reading portion of the aggregation will read from the majority snapshot, but now the writes will as well. Take a look at the new test I've written:

jstests/replsets/aggregate_out_read_concern_majority.cpp

66
    // An aggregation using readConcern level "majority" can read committed data.
67
    assertOutSeesState("old");
68
    rst.awaitReplication();
69
 
70
    // Stop replication on all secondaries to prevent the majority commit point from advancing.
71
    stopReplicationOnSecondaries(rst);
72
 
73
    // Perform a write that will not be majority committed (yet). A readConcern level "majority"
74
    // $out will not see uncommitted data.
75
    assert.commandWorked(
76
        sourceColl.update({state: "old"}, {$set: {state: "new"}}, kWriteConcernW1));
77
    assertOutSeesState("old");

The assertion on line 77 won't hold true because updates won't see the uncommitted data at all. What we really want is for the write portion of $out to not use the majority snapshot (that is, it should effectively be operating at readConcern level "local").

Comment by Kyle Suarez [ 23/Aug/18 ]

Adding this back to the $out project. This work is now in-scope again, but only for modes "insertDocuments" and "replaceDocuments". We won't support readConcern level "majority" for "replaceCollection" and will document that appropriately.

Comment by Charlie Swanson [ 03/Aug/18 ]

For help with triaging, this is the stack trace that Minji and I were looking at when we first came across this:

[js_test:read_majority_out] 2018-08-03T10:43:32.897-0400 d30021| 2018-08-03T10:43:32.889-0400 F -        [conn1] Invariant failure _collection->getCatalogEntry()->getTotalIndexCount(opCtx) == count src/mongo/db/catalog/index_catalog_impl.cpp 1134
[js_test:read_majority_out] 2018-08-03T10:43:32.897-0400 d30021| 2018-08-03T10:43:32.889-0400 F -        [conn1]
[js_test:read_majority_out] 2018-08-03T10:43:32.897-0400 d30021|
[js_test:read_majority_out] 2018-08-03T10:43:32.897-0400 d30021| ***aborting after invariant() failure
[js_test:read_majority_out] 2018-08-03T10:43:32.897-0400 d30021|
[js_test:read_majority_out] 2018-08-03T10:43:32.897-0400 d30021|
...
 /home/charlie/github/mongo/src/mongo/util/stacktrace_posix.cpp:172:0: mongo::printStackTrace(std::ostream&)
 /home/charlie/github/mongo/src/mongo/util/signal_handlers_synchronous.cpp:180:0: mongo::(anonymous namespace)::printSignalAndBacktrace(int)
 /home/charlie/github/mongo/src/mongo/util/signal_handlers_synchronous.cpp:237:0: mongo::(anonymous namespace)::abruptQuit(int)
 ??:0:0: ??
 /build/glibc-Cl5G7W/glibc-2.23/signal/../sysdeps/unix/sysv/linux/raise.c:54:0: gsignal
 /build/glibc-Cl5G7W/glibc-2.23/stdlib/abort.c:89:0: abort
 /home/charlie/github/mongo/src/mongo/util/assert_util.cpp:104:0: mongo::invariantFailed(char const*, char const*, unsigned int)
 /home/charlie/github/mongo/src/mongo/util/invariant.h:67:0: void mongo::invariantWithLocation<bool>(bool const&, char const*, char const*, unsigned int)
 /home/charlie/github/mongo/src/mongo/db/catalog/index_catalog_impl.cpp:1134:0: mongo::IndexCatalogImpl::numIndexesTotal(mongo::OperationContext*) const
 /home/charlie/github/mongo/src/mongo/db/catalog/index_catalog.h:297:0: mongo::IndexCatalog::numIndexesTotal(mongo::OperationContext*) const
 /home/charlie/github/mongo/src/mongo/db/catalog/drop_collection.cpp:89:0: mongo::dropCollection(mongo::OperationContext*, mongo::NamespaceString const&, mongo::BSONObjBuilder&, mongo::repl::OpTime const&, mongo::DropCollectionSystemCollectionMode)::{lambda()#1}::operator()() const
 /home/charlie/github/mongo/src/mongo/db/concurrency/write_conflict_exception.h:91:0: auto mongo::writeConflictRetry<mongo::dropCollection(mongo::OperationContext*, mongo::NamespaceString const&, mongo::BSONObjBuilder&, mongo::repl::OpTime const&, mongo::DropCollectionSystemCollectionMode)::{lambda()#1}>(mongo::OperationContext*, mongo::StringData, mongo::dropCollection(mongo::OperationContext*, mongo::NamespaceString const&, mongo::BSONObjBuilder&, mongo::repl::OpTime const&, mongo::DropCollectionSystemCollectionMode)::{lambda()#1}, mongo::dropCollection(mongo::OperationContext*, mongo::NamespaceString const&, mongo::BSONObjBuilder&, mongo::repl::OpTime const&, mongo::DropCollectionSystemCollectionMode)::{lambda()#1}&&)
 /home/charlie/github/mongo/src/mongo/db/catalog/drop_collection.cpp:113:0: mongo::dropCollection(mongo::OperationContext*, mongo::NamespaceString const&, mongo::BSONObjBuilder&, mongo::repl::OpTime const&, mongo::DropCollectionSystemCollectionMode)
 /home/charlie/github/mongo/src/mongo/db/commands/dbcommands.cpp:314:0: mongo::(anonymous namespace)::CmdDrop::errmsgRun(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, mongo::BSONObjBuilder&)
 /home/charlie/github/mongo/src/mongo/db/commands.cpp:530:0: mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, mongo::BSONObjBuilder&)
 /home/charlie/github/mongo/src/mongo/db/commands.cpp:436:0: mongo::BasicCommand::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_common.cpp:468:0: mongo::(anonymous namespace)::invokeInTransaction(mongo::OperationContext*, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*)
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_common.cpp:541:0: mongo::(anonymous namespace)::runCommandImpl(mongo::OperationContext*, mongo::CommandInvocation*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::LogicalTime, mongo::ServiceEntryPointCommon::Hooks const&, mongo::BSONObjBuilder*, boost::optional<mongo::OperationSessionInfoFromClient> const&)
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_common.cpp:883:0: mongo::(anonymous namespace)::execCommandDatabase(mongo::OperationContext*, mongo::Command*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::ServiceEntryPointCommon::Hooks const&)
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_common.cpp:1031:0: mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)::{lambda()#1}::operator()() const
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_common.cpp:1046:0: mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_common.cpp:1320:0: mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_mongod.cpp:123:0: mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)
 /home/charlie/github/mongo/src/mongo/db/dbdirectclient.cpp:145:0: mongo::(anonymous namespace)::loopbackBuildResponse(mongo::OperationContext*, mongo::LastError*, mongo::Message&)
 /home/charlie/github/mongo/src/mongo/db/dbdirectclient.cpp:150:0: mongo::DBDirectClient::call(mongo::Message&, mongo::Message&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)
 /home/charlie/github/mongo/src/mongo/client/dbclient_base.cpp:231:0: mongo::DBClientBase::runCommandWithTarget(mongo::OpMsgRequest)
 /home/charlie/github/mongo/src/mongo/client/dbclient_base.cpp:269:0: mongo::DBClientBase::runCommandWithTarget(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj, mongo::BSONObj&, int)
 /home/charlie/github/mongo/src/mongo/client/dbclient_base.cpp:289:0: mongo::DBClientBase::runCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj, mongo::BSONObj&, int)
 /home/charlie/github/mongo/src/mongo/client/dbclient_base.h:423:0: mongo::DBClientBase::dropCollection(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::WriteConcernOptions const&, mongo::BSONObj*)
 /home/charlie/github/mongo/src/mongo/db/pipeline/document_source_out_drop_target.h:45:0: mongo::DocumentSourceOutDropTarget::~DocumentSourceOutDropTarget()
 /home/charlie/github/mongo/src/mongo/db/pipeline/document_source_out_drop_target.h:52:0: mongo::DocumentSourceOutDropTarget::~DocumentSourceOutDropTarget()
 /home/charlie/github/mongo/src/mongo/util/intrusive_counter.h:58:0: mongo::intrusive_ptr_release(mongo::RefCountable const*)
 /home/charlie/github/mongo/src/third_party/boost-1.60.0/boost/smart_ptr/intrusive_ptr.hpp:97:0: boost::intrusive_ptr<mongo::DocumentSource>::~intrusive_ptr()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/bits/stl_list.h:106:0: std::_List_node<boost::intrusive_ptr<mongo::DocumentSource> >::~_List_node()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/ext/new_allocator.h:124:0: void __gnu_cxx::new_allocator<std::_List_node<boost::intrusive_ptr<mongo::DocumentSource> > >::destroy<std::_List_node<boost::intrusive_ptr<mongo::DocumentSource> > >(std::_List_node<boost::intrusive_ptr<mongo::DocumentSource> >*)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/bits/list.tcc:75:0: std::__cxx11::_List_base<boost::intrusive_ptr<mongo::DocumentSource>, std::allocator<boost::intrusive_ptr<mongo::DocumentSource> > >::_M_clear()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/bits/stl_list.h:446:0: std::__cxx11::_List_base<boost::intrusive_ptr<mongo::DocumentSource>, std::allocator<boost::intrusive_ptr<mongo::DocumentSource> > >::~_List_base()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/bits/stl_list.h:507:0: std::__cxx11::list<boost::intrusive_ptr<mongo::DocumentSource>, std::allocator<boost::intrusive_ptr<mongo::DocumentSource> > >::~list()
 /home/charlie/github/mongo/src/mongo/db/pipeline/pipeline.cpp:88:0: mongo::Pipeline::~Pipeline()
 /home/charlie/github/mongo/src/mongo/db/pipeline/pipeline.h:424:0: mongo::PipelineDeleter::operator()(mongo::Pipeline*)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/bits/unique_ptr.h:236:0: std::unique_ptr<mongo::Pipeline, mongo::PipelineDeleter>::~unique_ptr()
 /home/charlie/github/mongo/src/mongo/db/exec/pipeline_proxy.h:47:0: mongo::PipelineProxyStage::~PipelineProxyStage()
 /home/charlie/github/mongo/src/mongo/db/exec/pipeline_proxy.h:47:0: mongo::PipelineProxyStage::~PipelineProxyStage()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/bits/unique_ptr.h:76:0: std::default_delete<mongo::PlanStage>::operator()(mongo::PlanStage*) const
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/bits/unique_ptr.h:236:0: std::unique_ptr<mongo::PlanStage, std::default_delete<mongo::PlanStage> >::~unique_ptr()
 /home/charlie/github/mongo/src/mongo/db/query/plan_executor.cpp:280:0: mongo::PlanExecutor::~PlanExecutor()
 /home/charlie/github/mongo/src/mongo/db/query/plan_executor.cpp:737:0: mongo::PlanExecutor::Deleter::operator()(mongo::PlanExecutor*)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/bits/unique_ptr.h:236:0: std::unique_ptr<mongo::PlanExecutor, mongo::PlanExecutor::Deleter>::~unique_ptr()
 /home/charlie/github/mongo/src/mongo/db/clientcursor.cpp:108:0: mongo::ClientCursor::~ClientCursor()
 /home/charlie/github/mongo/src/mongo/db/clientcursor.cpp:239:0: mongo::ClientCursorPin::deleteUnderlying()
 /home/charlie/github/mongo/src/mongo/db/commands/run_aggregate.cpp:592:0: mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, mongo::rpc::ReplyBuilderInterface*)::{lambda(std::vector<mongo::ClientCursorPin, std::allocator<mongo::ClientCursorPin> >*)#2}::operator()(std::vector<mongo::ClientCursorPin, std::allocator<mongo::ClientCursorPin> >*) const
 /home/charlie/github/mongo/src/mongo/util/scopeguard.h:167:0: mongo::ScopeGuardImpl1<mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, mongo::rpc::ReplyBuilderInterface*)::{lambda(std::vector<mongo::ClientCursorPin, std::allocator<mongo::ClientCursorPin> >*)#2}, std::vector<mongo::ClientCursorPin, std::allocator<mongo::ClientCursorPin> >*>::Execute()
 /home/charlie/github/mongo/src/mongo/util/scopeguard.h:100:0: void mongo::ScopeGuardImplBase::SafeExecute<mongo::ScopeGuardImpl1<mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, mongo::rpc::ReplyBuilderInterface*)::{lambda(std::vector<mongo::ClientCursorPin, std::allocator<mongo::ClientCursorPin> >*)#2}, std::vector<mongo::ClientCursorPin, std::allocator<mongo::ClientCursorPin> >*> >(mongo::ScopeGuardImpl1<mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, mongo::rpc::ReplyBuilderInterface*)::{lambda(std::vector<mongo::ClientCursorPin, std::allocator<mongo::ClientCursorPin> >*)#2}, std::vector<mongo::ClientCursorPin, std::allocator<mongo::ClientCursorPin> >*>&)
 /home/charlie/github/mongo/src/mongo/util/scopeguard.h:163:0: mongo::ScopeGuardImpl1<mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, mongo::rpc::ReplyBuilderInterface*)::{lambda(std::vector<mongo::ClientCursorPin, std::allocator<mongo::ClientCursorPin> >*)#2}, std::vector<mongo::ClientCursorPin, std::allocator<mongo::ClientCursorPin> >*>::~ScopeGuardImpl1()
 /home/charlie/github/mongo/src/mongo/db/commands/run_aggregate.cpp:595:0: mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, mongo::rpc::ReplyBuilderInterface*)
 /home/charlie/github/mongo/src/mongo/db/commands/pipeline_command.cpp:85:0: mongo::(anonymous namespace)::PipelineCommand::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_common.cpp:468:0: mongo::(anonymous namespace)::invokeInTransaction(mongo::OperationContext*, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*)
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_common.cpp:541:0: mongo::(anonymous namespace)::runCommandImpl(mongo::OperationContext*, mongo::CommandInvocation*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::LogicalTime, mongo::ServiceEntryPointCommon::Hooks const&, mongo::BSONObjBuilder*, boost::optional<mongo::OperationSessionInfoFromClient> const&)
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_common.cpp:883:0: mongo::(anonymous namespace)::execCommandDatabase(mongo::OperationContext*, mongo::Command*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::ServiceEntryPointCommon::Hooks const&)
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_common.cpp:1031:0: mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)::{lambda()#1}::operator()() const
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_common.cpp:1046:0: mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_common.cpp:1320:0: mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)
 /home/charlie/github/mongo/src/mongo/db/service_entry_point_mongod.cpp:123:0: mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)
 /home/charlie/github/mongo/src/mongo/transport/service_state_machine.cpp:378:0: mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard)
 /home/charlie/github/mongo/src/mongo/transport/service_state_machine.cpp:439:0: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
 /home/charlie/github/mongo/src/mongo/transport/service_state_machine.cpp:479:0: mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::{lambda()#1}::operator()() const
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1871:0: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<void ()>::operator()() const
 /home/charlie/github/mongo/src/mongo/transport/service_executor_synchronous.cpp:115:0: mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)
 /home/charlie/github/mongo/src/mongo/transport/service_state_machine.cpp:483:0: mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)
 /home/charlie/github/mongo/src/mongo/transport/service_state_machine.cpp:305:0: mongo::ServiceStateMachine::_sourceCallback(mongo::Status)
 /home/charlie/github/mongo/src/mongo/transport/service_state_machine.cpp:259:0: mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::{lambda(mongo::StatusWith<mongo::Message>)#2}::operator()(mongo::StatusWith<mongo::Message>) const
 /home/charlie/github/mongo/src/mongo/util/future.h:112:0: auto mongo::future_details::call<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::{lambda(mongo::StatusWith<mongo::Message>)#2}&, mongo::Message>(mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::{lambda(mongo::StatusWith<mongo::Message>)#2}&, mongo::Message&&)
 /home/charlie/github/mongo/src/mongo/util/future.h:787:0: void mongo::future_details::Future<mongo::Message>::getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::{lambda(mongo::StatusWith<mongo::Message>)#2}>(mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::{lambda(mongo::StatusWith<mongo::Message>)#2}&&) &&::{lambda(mongo::Message&&)#1}::operator()(mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::{lambda(mongo::StatusWith<mongo::Message>)#2}&&) const
 /home/charlie/github/mongo/src/mongo/util/future.h:1091:0: auto mongo::future_details::Future<mongo::Message>::generalImpl<void mongo::future_details::Future<mongo::Message>::getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::{lambda(mongo::StatusWith<mongo::Message>)#2}>(mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::{lambda(mongo::StatusWith<mongo::Message>)#2}&&) &&::{lambda(mongo::Message&&)#1}, void mongo::future_details::Future<mongo::Message>::getAsync<{lambda(mongo::StatusWith<mongo::Message>)#2}>({lambda(mongo::StatusWith<mongo::Message>)#2}) &&::{lambda(mongo::Status&&)#2}, void mongo::future_details::Future<mongo::Message>::getAsync<{lambda(mongo::StatusWith<mongo::Message>)#2}>({lambda(mongo::StatusWith<mongo::Message>)#2}) &&::{lambda()#3}>(void mongo::future_details::Future<mongo::Message>::getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::{lambda(mongo::StatusWith<mongo::Message>)#2}>(mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::{lambda(mongo::StatusWith<mongo::Message>)#2}&&) &&::{lambda(mongo::Message&&)#1}, void mongo::future_details::Future<mongo::Message>::getAsync<{lambda(mongo::StatusWith<mongo::Message>)#2}>({lambda(mongo::StatusWith<mongo::Message>)#2}) &&::{lambda(mongo::Status&&)#2}&&, void mongo::future_details::Future<mongo::Message>::getAsync<{lambda(mongo::StatusWith<mongo::Message>)#2}>({lambda(mongo::StatusWith<mongo::Message>)#2}) &&::{lambda()#3}&&)
 /home/charlie/github/mongo/src/mongo/util/future.h:801:0: void mongo::future_details::Future<mongo::Message>::getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::{lambda(mongo::StatusWith<mongo::Message>)#2}>(mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::{lambda(mongo::StatusWith<mongo::Message>)#2}&&) &&
 /home/charlie/github/mongo/src/mongo/transport/service_state_machine.cpp:260:0: mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)
 /home/charlie/github/mongo/src/mongo/transport/service_state_machine.cpp:436:0: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
 /home/charlie/github/mongo/src/mongo/transport/service_state_machine.cpp:479:0: mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::{lambda()#1}::operator()() const
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1871:0: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<void ()>::operator()() const
 /home/charlie/github/mongo/src/mongo/transport/service_executor_synchronous.cpp:132:0: mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)::{lambda()#1}::operator()() const
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1871:0: std::_Function_handler<void (), mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<void ()>::operator()() const
 /home/charlie/github/mongo/src/mongo/transport/service_entry_point_utils.cpp:55:0: mongo::(anonymous namespace)::runFunc(void*)
 ??:0:0: start_thread
 /build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109:0: clone 

In this trace, it looks like the $out has failed and is trying to drop the temp collection. While doing so, it looks like the index catalog thinks it's corrupt. My best guess is that somehow a read of the catalog tables cannot find the _id index created by the $out stage because it has not yet been majority committed, but I can't quite tell what's going on here. At this point in the test, we have manually set a stable timestamp as the majority commit point, then run the $out without moving the commit point forward, so all changes made by $out will not be part of the stable checkpoint used by majority read concern. I'm not familiar enough with the storage code to know whether/how that will affect catalog operations like this.

Comment by Minji Kim [ 02/Aug/18 ]

Here is the patch and the git hash of the commit before it: 

readMajorityOut.patch

77c14d01a12dc01ad8e4a464d1c0222dd501fc57

Comment by Charlie Swanson [ 02/Aug/18 ]

Given the complications with this ticket, I'm bumping this out of the current sprint and back into Needs Triage.

Comment by Minji Kim [ 02/Aug/18 ]

Yes, my bad! I've attached the correct backtrace here. 

decipheredBt.txt 

There are actually multiple backtraces that pop up when you run the read_majority_out.js file. However, there are too many to post, and it's unclear how useful all of them are. The one above is just the very first backtrace to pop up. 

Comment by Charlie Swanson [ 02/Aug/18 ]

minji.kim the backtrace attached there doesn't look correct... There are a bunch of frames like this:

?:0:0: _fini
 ??:0:0: _fini
 ??:0:0: _fini
 ??:0:0: _fini
 ??:0:0: _fini
 ??:0:0: _fini
 ??:0:0: _fini
 ??:0:0: _fini

Also, I recall we found earlier that the stack trace indicated we were somewhere in renaming the temporary $out collection into place.

Comment by Minji Kim [ 02/Aug/18 ]

When I created and ran tests for read majority in $out, there were issues with creating and renaming of the new collection at the end of the $out process. 

I have attached the relevant patch here:

readMajorityOut.patch

Here is the deciphered backtrace: 

decipheredBt.txt

 ^^ 

Comment by James Wahlin [ 13/Mar/18 ]

From discussion with milkie we can most likely allow $out with readConcernLevel majority at this point, without further changes other than removing the restriction in document_source_out.cpp and added testing.

Generated at Thu Feb 08 03:54:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.