[SERVER-83145] Shared buffer fragment incorrectly tracks memory usage in freeUnused() Created: 10/Nov/23  Updated: 29/Jan/24  Resolved: 25/Jan/24

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 6.0.4, 6.2.0-rc0
Fix Version/s: 7.0.6, 6.0.14, 8.0.0-rc0, 7.3.0-rc2

Type: Bug Priority: Major - P3
Reporter: Shin Yee Tan Assignee: Louis Williams
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Problem/Incident
is caused by SERVER-68125 Index build on multi-key fields can c... Closed
Related
related to SERVER-82037 Memory used by sorter spills can grow... Backlog
Assigned Teams:
Storage Execution
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v7.3, v7.0, v6.0
Steps To Reproduce:

const maxMemUsageMB = 50;
const replSet = new ReplSetTest({
    nodes: 1,
    nodeOptions:

{         setParameter: \{maxIndexBuildMemoryUsageMegabytes: maxMemUsageMB, heapProfilingEnabled: true}

    },
});
replSet.startSet();
replSet.initiate();

const primary = replSet.getPrimary();
const testDB = primary.getDB('test');
const coll = testDB.index_build_wildcard;

 
const docs = 1000;
const bigArr = new Array(100000).fill('x'.repeat(10));
let d = 0;
while (d < docs) {
constbulk=coll.initializeUnorderedBulkOp();
for (leti=0; i<10; i++)

{ bulk.insert(\{a: (d+i) +'x'.repeat(1000), arr: bigArr}

);
}
d+=10;
print(d);
assert.commandWorked(bulk.execute());
}

coll.createIndex({"$**": 1});

Sprint: Execution Team 2023-12-11, Execution Team 2024-01-08, Execution Team 2024-01-22, Execution Team 2024-02-05
Participants:
Case:

 Description   

We decrement from memUsage when we free unused buffers from _activeBuffers here. The calculation is incorrect and undercounts actual freed memory causing the memUsage stat to balloon. This can be a problem when the memusage stat grows past the in-memory limit for the sorter resulting in spilling to disk on every key added to the sorter, eventually leading to an OOM issue.



 Comments   
Comment by Githook User [ 29/Jan/24 ]

Author:

{'name': 'Louis Williams', 'email': 'louiswilliams@users.noreply.github.com', 'username': 'louiswilliams'}

Message: SERVER-83145 Correctly track memory usage in SharedBufferFragment (#18234)

GitOrigin-RevId: cf21f604fdeef621f7782092ad637a89e57e1549
Branch: v6.0
https://github.com/mongodb/mongo/commit/1b600236e98f6d5e84df36611df2b2b6d79aa896

Comment by Githook User [ 26/Jan/24 ]

Author:

{'name': 'Louis Williams', 'email': 'louiswilliams@users.noreply.github.com', 'username': 'louiswilliams'}

Message: SERVER-83145 Correctly track memory usage in SharedBufferFragment (#18234)

GitOrigin-RevId: 6c4aec482bb15f917219a01aec332de557f77ac8
Branch: v7.3
https://github.com/mongodb/mongo/commit/e76c1f5de4438982de8028668e3dd7a1b8797c39

Comment by Githook User [ 26/Jan/24 ]

Author:

{'name': 'Louis Williams', 'email': 'louiswilliams@users.noreply.github.com', 'username': 'louiswilliams'}

Message: SERVER-83145 Correctly track memory usage in SharedBufferFragment (#18234)

GitOrigin-RevId: e6685de50d2a7556e2f2f9a8c52d30145d1256c9
Branch: v7.0
https://github.com/mongodb/mongo/commit/f880039d428b51a1260b65a1524cb5f30f2a19eb

Comment by Githook User [ 25/Jan/24 ]

Author:

{'name': 'Louis Williams', 'email': 'louiswilliams@users.noreply.github.com', 'username': 'louiswilliams'}

Message: SERVER-83145 Correctly track memory usage in SharedBufferFragment (#18234)

GitOrigin-RevId: 4ea6987aa035bbb75cf585b7390f70e5921b9ac5
Branch: master
https://github.com/mongodb/mongo/commit/51d0db82a4b3f89fc031898d944d6c8f1088f1b2

Generated at Thu Feb 08 06:51:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.