[SERVER-49940] Determine why create index concurrency limit in 4.2.3 might not be working for my benchmarks Created: 28/Jul/20  Updated: 14/Sep/20  Resolved: 14/Sep/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.2.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Mark Callaghan (Inactive) Assignee: Mark Callaghan (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Create indexes concurrently for large collections

Participants:

 Description   

I frequently run two benchmark workloads – Linkbench and the insert benchmark. Secondary indexes are created for both after there is some data in the indexed table. For Linkbench there is one secondary index while in the insert benchmark I use 3 secondary indexes per collection and usually use 8 collections.

For the insert benchmark creating the secondary indexes is done concurrently with one client per collection (so 8 in parallel). There is only one secondary index for Linkbench. 

Creating the secondary index is faster in 4.4 than 4.2 for Linkbench. But for the insert benchmark it is almost 2X faster in 4.2 than 4.4.

My scripts use "indexed rows/second" rather than time to report performance (so larger == faster) and that is ips in the data that my test scripts report. There is a lot of data here but I will explain what I see using the results in this section

  • for IO-bound Linkbench indexed rows/s is 247146 and 271190 for 4.2.8 without and with Snappy compression for the database compared to 113216 and 145950 for 4.4.0rc14 without and with Snappy compression.
  • 4.4 uses less CPU than 4.2 – see the cpupq column which is CPU per indexed row
  • 4.4 databases (see dbgb1) are larger than 4.2 which might explain why 4.4 reads more from storage / indexed row than 4.2 (see rkbpq). But this doesn't explain a 2X perf difference
  • rkbps (storage read KB/s from iostat) is almost 2X larger in 4.2 than 4.4 and that might explain the 2X perf difference.
  • however from a CPU-bound test where the collections are cached and there are no storage reads, 4.2 is still a lot faster (maybe 1.5X) than 4.4. Results are here

Some performance results from Linkbench are here and the indexed rows/s rate (ips) is better for 4.4 than 4.2

Next steps for me:

  • Figure out why 4.2 can read from storage almost 2X faster despite using the same setup via DSI (c3.8xlarge with 20k EBS IOPs)
  • Re-learn what the limits are for memory used by create index sorts in 4.2 and 4.4
  • upload ftdc


 Comments   
Comment by Mark Callaghan (Inactive) [ 11/Sep/20 ]

I don't think there is more to do here. AFAIK there are separate tickets open to update the docs

Comment by Louis Williams [ 29/Jul/20 ]

My memory has not served me well. In 4.2 we don't run createIndex commands on the thread pool on primaries (The default value here is false). The thread pool is only used on secondaries. We lowered the limit because createIndexes commands on primaries can build several indexes at once, but replicate multiple oplog entries, overwhelming secondaries.

So the behavior you observed makes sense. If you were running in a replica set configuration, you would likely see significant replication lag due to the limited thread pool resources on the secondaries blocking replication.

Comment by Mark Callaghan (Inactive) [ 29/Jul/20 ]

louis.williams - let know if someone on your side wants to continue on this. From what I see, my workloads in 4.2 are not limited to 3 concurrent index builds.

From looking at vmstat output during the create index with 8 concurrent index builds, 4.2.2 and 4.2.3 look the same. They also take a similar amount of time.

From 3 sets of stack traces obtained via PMP for 4.2.3 it looks like 8 create index commands are running concurrently:

      3 mongo::BSONObj::woCompare(mongo::BSONObj,std::pair<mongo::BSONObj,,void,void,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 tc_free,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 mongo::SortedFileWriter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted<mongo::BSONObj>*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 mongo::BSONObj::woCompare(mongo::BSONObj,std::_Deque_iterator<std::pair<mongo::BSONObj,,void,void,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 __strchr_sse2,mongo::dotted_path_support::extractElementAtPathOrArrayAlongPath(mongo::BSONObj,mongo::BtreeKeyGenerator::_extractNextElement(mongo::BSONObj,mongo::BtreeKeyGenerator::_getKeysWithArray(std::vector<char,mongo::BtreeKeyGenerator::getKeys(mongo::BSONObj,mongo::AbstractIndexAccessMethod::getKeys(mongo::BSONObj,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone

      3 pread64,__posix_file_read,__wt_block_read_off,__wt_bm_read,__wt_bt_read,__wt_page_in_func,__tree_walk_internal,__wt_btcur_next,__curfile_next,int,mongo::WiredTigerRecordStoreCursorBase::next(),mongo::CollectionScan::doWork(unsigned,mongo::PlanStage::work(unsigned,mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted<mongo::BSONObj>*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 tcmalloc::DLL_Prepend(tcmalloc::Span*,,tcmalloc::CentralFreeList::ReleaseToSpans(void*),tcmalloc::CentralFreeList::InsertRange(void*,,tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,,tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 mongo::WiredTigerRecoveryUnit::beginUnitOfWork(mongo::OperationContext*),mongo::WriteUnitOfWork::WriteUnitOfWork(mongo::OperationContext*),mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 mongo::BtreeKeyGenerator::_extractNextElement(mongo::BSONObj,mongo::BtreeKeyGenerator::_getKeysWithArray(std::vector<char,mongo::BtreeKeyGenerator::getKeys(mongo::BSONObj,mongo::AbstractIndexAccessMethod::getKeys(mongo::BSONObj,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 mongo::BSONObj::woCompare(mongo::BSONObj,std::pair<mongo::BSONObj,,void,void,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 __cursor_row_slot_return,__wt_btcur_next,__curfile_next,int,mongo::WiredTigerRecordStoreCursorBase::next(),mongo::CollectionScan::doWork(unsigned,mongo::PlanStage::work(unsigned,mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted<mongo::BSONObj>*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone

      2 mongo::BSONObj::woCompare(mongo::BSONObj,std::pair<mongo::BSONObj,,void,void,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::AbstractIndexAccessMethod::commitBulk(mongo::OperationContext*,,mongo::MultiIndexBlock::dumpInsertsFromBulk(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      2 mongo::BSONObj::woCompare(mongo::BSONObj,std::_Deque_iterator<std::pair<mongo::BSONObj,,void,void,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::AbstractIndexAccessMethod::commitBulk(mongo::OperationContext*,,mongo::MultiIndexBlock::dumpInsertsFromBulk(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 mongo::SortedFileWriter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::sorter::NoLimitSorter<mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 mongo::LockManager::unlock(mongo::LockRequest*),mongo::LockerImpl::_unlockImpl(mongo::FastMapNoAlloc<mongo::ResourceId,,mongo::LockerImpl::unlock(mongo::ResourceId),mongo::LockerImpl::saveLockStateAndUnlock(mongo::Locker::LockSnapshot*),mongo::PlanYieldPolicy::_yieldAllLocks(mongo::OperationContext*,,mongo::PlanYieldPolicy::yieldOrInterrupt(std::function<void,mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted<mongo::BSONObj>*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 mongo::BSONObj::getField(mongo::StringData),mongo::BtreeKeyGenerator::_extractNextElement(mongo::BSONObj,mongo::BtreeKeyGenerator::_getKeysWithArray(std::vector<char,mongo::BtreeKeyGenerator::getKeys(mongo::BSONObj,mongo::AbstractIndexAccessMethod::getKeys(mongo::BSONObj,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone
      1 __wt_page_inmem,__wt_page_in_func,__wt_row_search,__wt_btcur_search_near,__curfile_search_near,int,mongo::WiredTigerRecordStoreCursorBase::restore(),mongo::CollectionScan::doRestoreStateRequiresCollection(),mongo::RequiresCollectionStageBase<mongo::Collection,mongo::PlanExecutorImpl::restoreStateWithoutRetrying(),mongo::PlanYieldPolicy::yieldOrInterrupt(std::function<void,mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted<mongo::BSONObj>*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler<void,std::_Function_handler<void,mongo::(anonymous,start_thread,clone

Comment by Mark Callaghan (Inactive) [ 28/Jul/20 ]

It will take a few days to get all of the results. But from the first version I tested (rc14) create index is now faster in 4.4 than 4.2 for the insert benchmark once I increased the concurrency limit. So I will keep this open now to figure out why I don't see the impact of the concurrency limit starting in 4.2.3

Comment by Louis Williams [ 28/Jul/20 ]

Since this has come up before, index builds in 4.4 should be faster in almost every way due to KeyString sorting improvements we made. That said, 4.4 introduced some slowness due to the two-phase commit protocol, but unless there’s a large amount of replication lag, it shouldn’t be significant at scale. The observation that index builds are 2x slower seems related only to concurrent behavior when building more indexes than the default limit of 3, but I am interested to see what we find here.

Comment by Louis Williams [ 28/Jul/20 ]

The maxNumActiveUserIndexBuilds server parameter is hardcoded in 4.2 and not tunable. I filed SERVER-49948 to fix that.

Comment by Louis Williams [ 28/Jul/20 ]

Yes, actually. Wiredtiger has "read once" cursors that we use for index builds. Enabled by the useReadOnceCursorsForIndexBuilds by default. I did a small writeup in SERVER-37590.

Comment by Mark Callaghan (Inactive) [ 28/Jul/20 ]

louis.williams - are there plans to get and use a "don't cache" flag in WiredTiger so that scans done for create index don't wipe the cache? We added such an option to MyRocks (MySQL+RocksDB) and used it for full scans done during daily logical backup.

Comment by Mark Callaghan (Inactive) [ 28/Jul/20 ]

I have more to investigate. I have index build results for all 4.2 releases and I don't see a change at 4.2.3.

Comment by Louis Williams [ 28/Jul/20 ]

There is actually more history than I remember:

  • A limit of 10 was originally introduced by SERVER-38323 (for 4.1.7). Undocumented as far as I can tell.
  • The limit was lowered to 3 by SERVER-44984 (for 4.2.3 and 4.4.0-rc0) and the default memory usage was lowered to 200MB. Only the memory usage change was documented by DOCS-13432.
  • SERVER-47155 changed the implementation to be less buggy and introduced the new parameter that only exists in 4.4.

In your tests, both 4.2.8 and 4.4.0-rc* should be limited to 3 concurrent index builds, so maybe there is more to investigate there.

Comment by Louis Williams [ 28/Jul/20 ]

I'm sorry you ran into this issue. I updated the description in SERVER-47155 to provide more context for the change, which was to bound resource utilization of index builds by a safe default.

Comment by Mark Callaghan (Inactive) [ 28/Jul/20 ]

I was not aware of that change and will repeat my tests. Can server-47155 get updated to explain why this was added?

Also, I don't see docs for this:
https://docs.mongodb.com/master/reference/operator/aggregation/lookup/?searchProperty=current&query=maxNumActiveUserIndexBuilds

Comment by Louis Williams [ 28/Jul/20 ]

In 4.4, we introduced a limit for the maximum number of concurrent index builds to 3 by default (see SERVER-47155). Are you raising the maxNumActiveUserIndexBuilds parameter in your test that is running 8 concurrent index builds?

Comment by Mark Callaghan (Inactive) [ 28/Jul/20 ]

For insert benchmark with 1 collection and serial create index, creating secondary indexes is slightly faster in 4.4 than in 4.2. I will run a few more tests to determine how that changes as concurrency is increased.

Generated at Thu Feb 08 05:21:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.