-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Storage Engines, Storage Engines - Foundations
-
None
-
None
I reproduced an issue that demonstrates WT bulk cursors allocating unbounded memory untracked by the WT cache in the form of WT_IKEY structs. The out-of-cache memory is eventually converted into cache-tracked memory when the bulk cursor is closed.
During the index build process, after collection scanning and sorting is complete (marker A), the memory usage starts to rise dramatically, up to 2GB outside of the configured WT cache size.
When run with the mongodb heap profiler, the following stack is implicated:
{
"t":
{
"$date": "2025-09-26T20:04:39.071+00:00"
},
"s": "I",
"c": "-",
"id": 8592501,
"ctx": "ftdc",
"msg": "heapProfile stack",
"attr":
{
"stackNum": 445,
"stackObj":
{
"0": "tcmalloc::tcmalloc_internal::SampleifyAllocation()",
"1": "tcmalloc::tcmalloc_internal::alloc_small_sampled_hooks_or_perthread<>()",
"2": "__libc_calloc",
"3": "__wt_calloc",
"4": "__wt_row_ikey_alloc",
"5": "__rec_split_write",
"6": "__wti_rec_split",
"7": "__wt_bulk_insert_row",
"8": "__curbulk_insert_row",
"9": "mongo::wiredTigerCursorInsert()",
"10": "mongo::(anonymous namespace)::StandardBulkBuilder::addKey()",
"11": "mongo::SortedDataIndexAccessMethod::BaseBulkBuilder::commit()",
"12": "mongo::MultiIndexBlock::dumpInsertsFromBulk()",
"13": "mongo::MultiIndexBlock::insertAllDocumentsInCollection()",
"14": "mongo::IndexBuildsManager::startBuildingIndex()",
"15": "mongo::IndexBuildsCoordinator::_scanCollectionAndInsertSortedKeysIntoIndex()",
"16": "mongo::IndexBuildsCoordinator::_buildHybridIndex()",
"17": "mongo::IndexBuildsCoordinator::_runIndexBuildInner()",
"18": "mongo::IndexBuildsCoordinator::_runIndexBuild()",
"19": "mongo::IndexBuildsCoordinatorMongod::_startIndexBuild()::{lambda()#1}::operator()<>()",
"20": "mongo::unique_function<>::makeImpl<>()::SpecificImpl::call()",
"21": "mongo::ThreadPool::Impl::_doOneTask()",
"22": "mongo::ThreadPool::Impl::_consumeTasks()",
"23": "mongo::ThreadPool::Impl::_workerThreadBody()",
"24": "std::thread::_State_impl<>::_M_run()",
"25": "execute_native_thread_routine",
"26": "0xffffb7add5c8"
}
}
}
At marker B, when the bulk cursor is closed, that memory appears to somehow be transferred, but in a way that is now tracked in the cache:
{
"t":
{
"$date": "2025-09-26T20:06:44.127+00:00"
},
"s": "I",
"c": "-",
"id": 8592501,
"ctx": "ftdc",
"msg": "heapProfile stack",
"attr":
{
"stackNum": 490,
"stackObj":
{
"0": "tcmalloc::tcmalloc_internal::SampleifyAllocation()",
"1": "tcmalloc::tcmalloc_internal::alloc_small_sampled_hooks_or_perthread<>()",
"2": "__libc_calloc",
"3": "__wt_calloc",
"4": "__wti_row_ikey",
"5": "__wt_multi_to_ref",
"6": "__wt_split_multi",
"7": "__wt_evict",
"8": "__wt_evict_file",
"9": "__checkpoint_tree.isra.0",
"10": "__wt_checkpoint_close",
"11": "__wt_conn_dhandle_close",
"12": "__wt_session_release_dhandle_v2",
"13": "__curfile_close",
"14": "mongo::(anonymous namespace)::StandardBulkBuilder::~StandardBulkBuilder()",
"15": "mongo::SortedDataIndexAccessMethod::BaseBulkBuilder::commit()",
"16": "mongo::MultiIndexBlock::dumpInsertsFromBulk()",
"17": "mongo::MultiIndexBlock::insertAllDocumentsInCollection()",
"18": "mongo::IndexBuildsManager::startBuildingIndex()",
"19": "mongo::IndexBuildsCoordinator::_scanCollectionAndInsertSortedKeysIntoIndex()",
"20": "mongo::IndexBuildsCoordinator::_buildHybridIndex()",
"21": "mongo::IndexBuildsCoordinator::_runIndexBuildInner()",
"22": "mongo::IndexBuildsCoordinator::_runIndexBuild()",
"23": "mongo::IndexBuildsCoordinatorMongod::_startIndexBuild()::{lambda()#1}::operator()<>()",
"24": "mongo::unique_function<>::makeImpl<>()::SpecificImpl::call()",
"25": "mongo::ThreadPool::Impl::_doOneTask()",
"26": "mongo::ThreadPool::Impl::_consumeTasks()",
"27": "mongo::ThreadPool::Impl::_workerThreadBody()",
"28": "std::thread::_State_impl<>::_M_run()",
"29": "execute_native_thread_routine",
"30": "0xffffb7add5c8"
}
}
}
- is related to
-
SERVER-111614 Increase size of index builds in perf testing
-
- Needs Scheduling
-
- related to
-
-
- Needs Scheduling
-