-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Cursors
-
Storage Engines - Foundations
-
SE Foundations - 2025-11-21
-
5
-
Technical Services
I reproduced an issue that demonstrates WT bulk cursors allocating unbounded memory untracked by the WT cache in the form of WT_IKEY structs. The out-of-cache memory is eventually converted into cache-tracked memory when the bulk cursor is closed.
During the index build process, after collection scanning and sorting is complete (marker A), the memory usage starts to rise dramatically, up to 2GB outside of the configured WT cache size.
When run with the mongodb heap profiler, the following stack is implicated:
{
"t":
{
"$date": "2025-09-26T20:04:39.071+00:00"
},
"s": "I",
"c": "-",
"id": 8592501,
"ctx": "ftdc",
"msg": "heapProfile stack",
"attr":
{
"stackNum": 445,
"stackObj":
{
"0": "tcmalloc::tcmalloc_internal::SampleifyAllocation()",
"1": "tcmalloc::tcmalloc_internal::alloc_small_sampled_hooks_or_perthread<>()",
"2": "__libc_calloc",
"3": "__wt_calloc",
"4": "__wt_row_ikey_alloc",
"5": "__rec_split_write",
"6": "__wti_rec_split",
"7": "__wt_bulk_insert_row",
"8": "__curbulk_insert_row",
"9": "mongo::wiredTigerCursorInsert()",
"10": "mongo::(anonymous namespace)::StandardBulkBuilder::addKey()",
"11": "mongo::SortedDataIndexAccessMethod::BaseBulkBuilder::commit()",
"12": "mongo::MultiIndexBlock::dumpInsertsFromBulk()",
"13": "mongo::MultiIndexBlock::insertAllDocumentsInCollection()",
"14": "mongo::IndexBuildsManager::startBuildingIndex()",
"15": "mongo::IndexBuildsCoordinator::_scanCollectionAndInsertSortedKeysIntoIndex()",
"16": "mongo::IndexBuildsCoordinator::_buildHybridIndex()",
"17": "mongo::IndexBuildsCoordinator::_runIndexBuildInner()",
"18": "mongo::IndexBuildsCoordinator::_runIndexBuild()",
"19": "mongo::IndexBuildsCoordinatorMongod::_startIndexBuild()::{lambda()#1}::operator()<>()",
"20": "mongo::unique_function<>::makeImpl<>()::SpecificImpl::call()",
"21": "mongo::ThreadPool::Impl::_doOneTask()",
"22": "mongo::ThreadPool::Impl::_consumeTasks()",
"23": "mongo::ThreadPool::Impl::_workerThreadBody()",
"24": "std::thread::_State_impl<>::_M_run()",
"25": "execute_native_thread_routine",
"26": "0xffffb7add5c8"
}
}
}
At marker B, when the bulk cursor is closed, that memory appears to somehow be transferred, but in a way that is now tracked in the cache:
{
"t":
{
"$date": "2025-09-26T20:06:44.127+00:00"
},
"s": "I",
"c": "-",
"id": 8592501,
"ctx": "ftdc",
"msg": "heapProfile stack",
"attr":
{
"stackNum": 490,
"stackObj":
{
"0": "tcmalloc::tcmalloc_internal::SampleifyAllocation()",
"1": "tcmalloc::tcmalloc_internal::alloc_small_sampled_hooks_or_perthread<>()",
"2": "__libc_calloc",
"3": "__wt_calloc",
"4": "__wti_row_ikey",
"5": "__wt_multi_to_ref",
"6": "__wt_split_multi",
"7": "__wt_evict",
"8": "__wt_evict_file",
"9": "__checkpoint_tree.isra.0",
"10": "__wt_checkpoint_close",
"11": "__wt_conn_dhandle_close",
"12": "__wt_session_release_dhandle_v2",
"13": "__curfile_close",
"14": "mongo::(anonymous namespace)::StandardBulkBuilder::~StandardBulkBuilder()",
"15": "mongo::SortedDataIndexAccessMethod::BaseBulkBuilder::commit()",
"16": "mongo::MultiIndexBlock::dumpInsertsFromBulk()",
"17": "mongo::MultiIndexBlock::insertAllDocumentsInCollection()",
"18": "mongo::IndexBuildsManager::startBuildingIndex()",
"19": "mongo::IndexBuildsCoordinator::_scanCollectionAndInsertSortedKeysIntoIndex()",
"20": "mongo::IndexBuildsCoordinator::_buildHybridIndex()",
"21": "mongo::IndexBuildsCoordinator::_runIndexBuildInner()",
"22": "mongo::IndexBuildsCoordinator::_runIndexBuild()",
"23": "mongo::IndexBuildsCoordinatorMongod::_startIndexBuild()::{lambda()#1}::operator()<>()",
"24": "mongo::unique_function<>::makeImpl<>()::SpecificImpl::call()",
"25": "mongo::ThreadPool::Impl::_doOneTask()",
"26": "mongo::ThreadPool::Impl::_consumeTasks()",
"27": "mongo::ThreadPool::Impl::_workerThreadBody()",
"28": "std::thread::_State_impl<>::_M_run()",
"29": "execute_native_thread_routine",
"30": "0xffffb7add5c8"
}
}
}
- is depended on by
-
SERVER-115329 Limit memory utilization of WT cursors in index build
-
- Blocked
-
- is related to
-
-
- Backlog
-
-
-
- Closed
-
-
SERVER-111614 Increase size of index builds in perf testing
-
- Closed
-
- related to
-
-
- Closed
-
-
-
- Closed
-