[SERVER-36920] Remove skipValidationNamespaces for the oplog after WiredTiger segmentation fault has been resolved. Created: 29/Aug/18  Updated: 08/Jan/24  Resolved: 27/Sep/18

Status: Closed
Project: Core Server
Component/s: Internal Code
Affects Version/s: None
Fix Version/s: 4.1.4

Type: Bug Priority: Major - P3
Reporter: Blake Oler Assignee: Blake Oler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on WT-4333 WiredTiger cursor cache doesn't handl... Closed
is depended on by SERVER-37227 Reintroduce enableMajorityReadConcern... Closed
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Storage Engines 2018-09-24, Sharding 2018-10-08
Participants:
Linked BF Score: 58

 Description   

The segmentation faults occur inside the validateCollections hook after [geo_s2ordering.js, views_rename.js, cursor5.js]. The tests were running in a custom jsCore passthrough suite with the logical session refresh set to 100ms.

geo_s2ordering.js suite

views_rename.js, cursor5.js suite

Current theory is that a session was reaped while a refresh was being attempted. Further investigation would confirm or deny,

geo_s2ordering.js stacktrace

 /data/mci/0dec23a244e697b846e3603b40845b44/src/src/mongo/util/stacktrace_posix.cpp:172:0: mongo::printStackTrace(std::ostream&)
 /data/mci/0dec23a244e697b846e3603b40845b44/src/src/mongo/util/signal_handlers_synchronous.cpp:180:0: mongo::(anonymous namespace)::printSignalAndBacktrace(int)
 /data/mci/0dec23a244e697b846e3603b40845b44/src/src/mongo/util/signal_handlers_synchronous.cpp:276:0: mongo::(anonymous namespace)::abruptQuitWithAddrSignal(int, siginfo*, void*)
 /data/mci/6fd4c41497f6db6e1f7ce1891b5368c9/src/src/third_party/mozjs-45/extract/js/src/asmjs/AsmJSSignalHandlers.cpp:1159:0: AsmJSFaultHandler(int, siginfo*, void*)
 ??:0:0: ??
 /data/mci/6fd4c41497f6db6e1f7ce1891b5368c9/src/src/third_party/wiredtiger/src/btree/row_srch.c:292:0: __wt_row_search
 /data/mci/6fd4c41497f6db6e1f7ce1891b5368c9/src/src/third_party/wiredtiger/src/btree/bt_cursor.c:385:0: __cursor_row_search
 /data/mci/6fd4c41497f6db6e1f7ce1891b5368c9/src/src/third_party/wiredtiger/src/btree/bt_cursor.c:813:0: __wt_btcur_insert
 /data/mci/d0714778f2150c8968af23a43dc7dbb3/src/src/third_party/wiredtiger/src/cursor/cur_file.c:265:0: __curfile_insert
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp:1334:0: mongo::WiredTigerRecordStore::_insertRecords(mongo::OperationContext*, mongo::Record*, mongo::Timestamp const*, unsigned long)
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp:1412:0: mongo::WiredTigerRecordStore::insertRecordsWithDocWriter(mongo::OperationContext*, mongo::DocWriter const* const*, mongo::Timestamp const*, unsigned long, mongo::RecordId*)
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/catalog/collection_impl.cpp:310:0: mongo::CollectionImpl::insertDocumentsForOplog(mongo::OperationContext*, mongo::DocWriter const* const*, mongo::Timestamp*, unsigned long)
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/catalog/collection.h:503:0: mongo::Collection::insertDocumentsForOplog(mongo::OperationContext*, mongo::DocWriter const* const*, mongo::Timestamp*, unsigned long)
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/repl/oplog.cpp:401:0: mongo::repl::_logOpsInner(mongo::OperationContext*, mongo::NamespaceString const&, mongo::DocWriter const* const*, mongo::Timestamp*, unsigned long, mongo::Collection*, mongo::repl::OpTime)
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/repl/oplog.cpp:589:0: mongo::repl::logInsertOps(mongo::OperationContext*, mongo::NamespaceString const&, boost::optional<mongo::UUID>, mongo::Session*, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, bool, mongo::Date_t)
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/op_observer_impl.cpp:403:0: mongo::OpObserverImpl::onInserts(mongo::OperationContext*, mongo::NamespaceString const&, boost::optional<mongo::UUID>, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, bool)
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/op_observer_registry.h:77:0: mongo::OpObserverRegistry::onInserts(mongo::OperationContext*, mongo::NamespaceString const&, boost::optional<mongo::UUID>, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, bool)
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/catalog/collection_impl.cpp:364:0: mongo::CollectionImpl::insertDocuments(mongo::OperationContext*, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, mongo::OpDebug*, bool)
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/catalog/collection_impl.cpp:378:0: mongo::CollectionImpl::insertDocument(mongo::OperationContext*, mongo::InsertStatement const&, mongo::OpDebug*, bool)
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/catalog/collection.h:492:0: mongo::Collection::insertDocument(mongo::OperationContext*, mongo::InsertStatement const&, mongo::OpDebug*, bool)
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/exec/update.cpp:467:0: operator()
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/concurrency/write_conflict_exception.h:91:0: writeConflictRetry<mongo::UpdateStage::doInsert()::<lambda()> >
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/exec/update.cpp:475:0: mongo::UpdateStage::doInsert()
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/exec/update.cpp:510:0: mongo::UpdateStage::doWork(unsigned long*)
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/exec/plan_stage.cpp:46:0: mongo::PlanStage::work(unsigned long*)
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/query/plan_executor.cpp:563:0: mongo::PlanExecutor::getNextImpl(mongo::Snapshotted<mongo::BSONObj>*, mongo::RecordId*)
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/query/plan_executor.cpp:408:0: mongo::PlanExecutor::getNext(mongo::BSONObj*, mongo::RecordId*)
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/query/plan_executor.cpp:692:0: mongo::PlanExecutor::executePlan()
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/ops/write_ops_exec.cpp:658:0: performSingleUpdateOp
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/ops/write_ops_exec.cpp:736:0: mongo::performUpdates(mongo::OperationContext*, mongo::write_ops::Update const&)
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/commands/write_commands/write_commands.cpp:342:0: mongo::(anonymous namespace)::CmdUpdate::Invocation::runImpl(mongo::OperationContext*, mongo::BSONObjBuilder&) const
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/db/commands/write_commands/write_commands.cpp:232:0: mongo::(anonymous namespace)::WriteCommand::InvocationBase::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/service_entry_point_common.cpp:450:0: mongo::(anonymous namespace)::invokeInTransaction(mongo::OperationContext*, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*) [clone .constprop.332]
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/service_entry_point_common.cpp:527:0: runCommandImpl
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/service_entry_point_common.cpp:868:0: mongo::(anonymous namespace)::execCommandDatabase(mongo::OperationContext*, mongo::Command*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::ServiceEntryPointCommon::Hooks const&) [clone .constprop.323]
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/service_entry_point_common.cpp:1018:0: mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)::{lambda()#1}::operator()() const
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/service_entry_point_common.cpp:1033:0: receivedCommands
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/service_entry_point_common.cpp:1308:0: mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/service_entry_point_mongod.cpp:123:0: mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/dbdirectclient.cpp:145:0: mongo::(anonymous namespace)::loopbackBuildResponse(mongo::OperationContext*, mongo::LastError*, mongo::Message&) [clone .constprop.78]
 /data/mci/29a22ee711ac9ae9227034f98233a7d3/src/src/mongo/db/dbdirectclient.cpp:150:0: mongo::DBDirectClient::call(mongo::Message&, mongo::Message&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/client/dbclient_base.cpp:231:0: mongo::DBClientBase::runCommandWithTarget(mongo::OpMsgRequest)
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/client/dbclient_base.cpp:269:0: mongo::DBClientBase::runCommandWithTarget(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj, mongo::BSONObj&, int)
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/client/dbclient_base.cpp:289:0: mongo::DBClientBase::runCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj, mongo::BSONObj&, int)
 /data/mci/30d296d182b2fe9b0d1ab2c9dff868c5/src/src/mongo/db/sessions_collection.cpp:166:0: operator()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1857:0: std::_Function_handler<mongo::Status (mongo::BSONObj), mongo::SessionsCollection::makeSendFnForBatchWrite(mongo::NamespaceString const&, mongo::DBClientBase*)::{lambda(mongo::BSONObj)#1}>::_M_invoke(std::_Any_data const&, mongo::BSONObj&&)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<mongo::Status (mongo::BSONObj)>::operator()(mongo::BSONObj) const
 /data/mci/30d296d182b2fe9b0d1ab2c9dff868c5/src/src/mongo/db/sessions_collection.cpp:152:0: mongo::Status mongo::(anonymous namespace)::runBulkCmd<mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONObjBuilder*)#1}&, mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionRecord const&)#2}&, std::function<mongo::Status (mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > >(mongo::StringData, mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONObjBuilder*)#1}&, mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionRecord const&)#2}&, std::function<mongo::Status (mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&)::{lambda(mongo::BSONArrayBuilder)#2}::operator()(mongo::BSONArrayBuilder) const [clone .isra.170]
 /data/mci/30d296d182b2fe9b0d1ab2c9dff868c5/src/src/mongo/db/sessions_collection.cpp:106:0: operator()
 /data/mci/30d296d182b2fe9b0d1ab2c9dff868c5/src/src/mongo/db/sessions_collection.cpp:124:0: mongo::Status mongo::(anonymous namespace)::runBulkGeneric<mongo::Status mongo::(anonymous namespace)::runBulkCmd<mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONObjBuilder*)#1}&, mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionRecord const&)#2}&, std::function<mongo::Status (mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > >(mongo::StringData, mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONObjBuilder*)#1}&, mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionRecord const&)#2}&, std::function<mongo::Status (mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&)::{lambda()#1}, mongo::LogicalSessionRecord const&, mongo::Status mongo::(anonymous namespace)::runBulkCmd<mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONObjBuilder*)#1}, {lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionRecord const&)#2}, mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionRecord const&)#2}, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > >(mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionRecord const&)#2}&, mongo::StringData, mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONObjBuilder*)#1}&, mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionRecord const&)#2}&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> >)::{lambda(mongo::BSONArrayBuilder)#2}, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > >(std::function<mongo::Status (mongo::BSONObj)>&, mongo::Status mongo::(anonymous namespace)::runBulkCmd<mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONObjBuilder*)#1}&, mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionRecord const&)#2}&, std::function<mongo::Status (mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > >(mongo::StringData, mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONObjBuilder*)#1}&, mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionRecord const&)#2}&, std::function<mongo::Status (mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&)::{lambda()#1}, mongo::LogicalSessionRecord const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> >) [clone .isra.200]
 /data/mci/30d296d182b2fe9b0d1ab2c9dff868c5/src/src/mongo/db/sessions_collection.cpp:155:0: runBulkCmd<mongo::SessionsCollection::doRefresh(const mongo::NamespaceString&, const LogicalSessionRecordSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONObjBuilder*)>&, mongo::SessionsCollection::doRefresh(const mongo::NamespaceString&, const LogicalSessionRecordSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONArrayBuilder*, const mongo::LogicalSessionRecord&)>&, std::function<mongo::Status(mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash> >
 /data/mci/30d296d182b2fe9b0d1ab2c9dff868c5/src/src/mongo/db/sessions_collection.cpp:225:0: mongo::SessionsCollection::doRefresh(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&, std::function<mongo::Status (mongo::BSONObj)>)
 /data/mci/30d296d182b2fe9b0d1ab2c9dff868c5/src/src/mongo/db/sessions_collection_rs.cpp:183:0: operator()
 /data/mci/30d296d182b2fe9b0d1ab2c9dff868c5/src/src/mongo/db/sessions_collection_rs.cpp:134:0: operator()
 /data/mci/30d296d182b2fe9b0d1ab2c9dff868c5/src/src/mongo/db/sessions_collection_rs.cpp:108:0: boost::optional<decltype (((declval<mongo::NamespaceString const&>)())())> mongo::(anonymous namespace)::runIfStandaloneOrPrimary<decltype (((declval<mongo::SessionsCollectionRS::refreshSessions(mongo::OperationContext*, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&)::{lambda(mongo::DBClientBase*)#2}>)())(static_cast<mongo::DBClientBase>((decltype(nullptr))0))) mongo::(anonymous namespace)::dispatch<mongo::SessionsCollectionRS::refreshSessions(mongo::OperationContext*, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&)::{lambda()#1}, mongo::SessionsCollectionRS::refreshSessions(mongo::OperationContext*, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&)::{lambda(mongo::DBClientBase*)#2}>(mongo::NamespaceString const&, mongo::LockMode, mongo::OperationContext*, mongo::SessionsCollectionRS::refreshSessions(mongo::OperationContext*, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&)::{lambda()#1}, {lambda(mongo::DBClientBase*)#2})::{lambda()#1}>(mongo::NamespaceString, mongo::NamespaceString const, mongo::OperationContext*, mongo::NamespaceString const&) [clone .constprop.168]
 /data/mci/30d296d182b2fe9b0d1ab2c9dff868c5/src/src/mongo/db/sessions_collection_rs.cpp:134:0: dispatch<mongo::SessionsCollectionRS::refreshSessions(mongo::OperationContext*, const LogicalSessionRecordSet&)::<lambda()>, mongo::SessionsCollectionRS::refreshSessions(mongo::OperationContext*, const LogicalSessionRecordSet&)::<lambda(mongo::DBClientBase*)> >
 /data/mci/30d296d182b2fe9b0d1ab2c9dff868c5/src/src/mongo/db/sessions_collection_rs.cpp:190:0: mongo::SessionsCollectionRS::refreshSessions(mongo::OperationContext*, std::unordered_set<mongo::LogicalSessionRecord, mongo::LogicalSessionRecordHash, std::equal_to<mongo::LogicalSessionRecord>, std::allocator<mongo::LogicalSessionRecord> > const&)
 /data/mci/ffa6ce63c3a77a0f5ac1fb16a501e578/src/src/mongo/db/logical_session_cache_impl.cpp:343:0: mongo::LogicalSessionCacheImpl::_refresh(mongo::Client*)
 /data/mci/ffa6ce63c3a77a0f5ac1fb16a501e578/src/src/mongo/db/logical_session_cache_impl.cpp:176:0: mongo::LogicalSessionCacheImpl::_periodicRefresh(mongo::Client*)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<void (mongo::Client*)>::operator()(mongo::Client*) const
 /data/mci/d9390258e742ca52130dae278d1efb43/src/src/mongo/util/periodic_runner_impl.cpp:118:0: operator()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1531:0: _M_invoke<>
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1520:0: operator()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/thread:115:0: std::thread::_Impl<std::_Bind_simple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#1} ()> >::_M_run()
 /data/mci/5094ed0504fef7be989d31e353bed7b1/toolchain-builder/build-gcc-v2.sh-7sl/x86_64-mongodb-linux/libstdc++-v3/src/c++11/../../../../../gcc-5.4.0/libstdc++-v3/src/c++11/thread.cc:84:0: execute_native_thread_routine

Note: As part of the fix for this ticket, also remove the skipValidationNamespaces: local.oplog.rs designation for ValidateCollections created in SERVER-37187.



 Comments   
Comment by Githook User [ 27/Sep/18 ]

Author:

{'name': 'Blake Oler', 'email': 'blake.oler@mongodb.com', 'username': 'BlakeIsBlake'}

Message: SERVER-36920 Remove skipValidationNamespaces for the oplog after WiredTiger segmentation fault has been resolved.
Branch: master
https://github.com/mongodb/mongo/commit/44c90e7ddbab97cd04000c58b5b8d29c989fc68e

Comment by Eric Milkie [ 26/Sep/18 ]

Sorry – I missed the Note at the end of the description. Can you change the title of this ticket to reflect the work? It currently describes the problem, rather than the work needed to fix.

Comment by Blake Oler [ 26/Sep/18 ]

Completion of this ticket relies upon the removal of the skipValidationNamespaces for ValidateCollections created in SERVER-37187.

Comment by Eric Milkie [ 26/Sep/18 ]

This is now fixed with this commit on master:

commit fecda01a45da991501f9eb0863f98b242421a00a
Author: Luke Chen <luke.chen@mongodb.com>
Date: Wed Sep 26 12:51:03 2018 +1000

Import wiredtiger: f2959cdb9f863f26dd0c3e3067a1af6f0d6061b4 from branch mongodb-4.2

ref: 7db12ec1b5..f2959cdb9f
for: 4.1.4

WT-4333 WiredTiger cursor cache doesn't handle all possible locked handle states

Comment by Eric Milkie [ 25/Sep/18 ]

I put this in Waiting because it's waiting for the linked WT ticket to appear in the mongodb master branch.

Comment by Eric Milkie [ 18/Sep/18 ]

Both of the links in the original description point to crashes while inserting documents into the oplog while another thread is actively attempting to do a verify on the oplog collection. This is a known problem that we are actively investigating.
In my reproducer for the crash, I modified my local code to not cache logical sessions at all, since having outstanding sessions will lock the oplog table and prevent the verify from occurring. It makes sense that increasing the logical session refresh rate to something very frequent, as was done in this ticket, had a similar effect.

Comment by Vamsi Boyapati [ 31/Aug/18 ]

Apart from the function names, no useful information is available to diagnose

 /data/mci/6fd4c41497f6db6e1f7ce1891b5368c9/src/src/third_party/wiredtiger/src/btree/row_srch.c:292:0: __wt_row_search
 /data/mci/6fd4c41497f6db6e1f7ce1891b5368c9/src/src/third_party/wiredtiger/src/btree/bt_cursor.c:385:0: __cursor_row_search
 /data/mci/6fd4c41497f6db6e1f7ce1891b5368c9/src/src/third_party/wiredtiger/src/btree/bt_cursor.c:813:0: __wt_btcur_insert
 /data/mci/d0714778f2150c8968af23a43dc7dbb3/src/src/third_party/wiredtiger/src/cursor/cur_file.c:265:0: __curfile_insert

for now, returning back to platforms team.

Comment by Alexander Gorrod [ 31/Aug/18 ]

vamsi.krishna please take a look at this, and help if possible.

Comment by Mira Carey [ 29/Aug/18 ]

I think it comes down to whether or not we actually initialize a js_runtime (if we don't, no signal handler is installed).

Comment by Andrew Morrow (Inactive) [ 29/Aug/18 ]

Oh, I agree, it isn't the source of the problem. I also agree the signal ultimately ends up routed correctly. But what is strange to me is why we don't always see it, if it is always there.

Comment by Mira Carey [ 29/Aug/18 ]

From a quick look, I think spidermonkey installs its signal handler regardless of whether or not it actually uses signals for interruption (because to jit or not to jit is a per runtime option, and setting up the signal handlers is a process wide operation).

So it's in there. And maybe it shouldn't be. But it probably isn't the actual source of the problem (it forwards signals it doesn't know about, which seems to be working in this case)

Comment by Blake Oler [ 29/Aug/18 ]

After discussing with renctan, we decided that it seems unlikely that an upsert turned into an insert would be an issue with Sharding-related code. Sending to the Storage backlog for a peek!

E: I'll let it wait until mira.carey@mongodb.com takes a look

Generated at Thu Feb 08 04:44:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.