[SERVER-37926] Excess allocated memory associated with cursors due to WT "modify" operations Created: 05/Nov/18  Updated: 28/Apr/19  Resolved: 12/Nov/18

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 4.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Bruce Lucas (Inactive) Assignee: Vamsi Boyapati
Resolution: Duplicate Votes: 0
Labels: dmd-perf
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot After WT-4418.png     PNG File Screen Shot Before WT-4418.png     PNG File comparison.png    
Issue Links:
Duplicate
duplicates WT-4418 Don't keep key/value memory buffers a... Closed
Related
is related to SERVER-37795 WiredTiger again uses way more memory... Closed
Operating System: ALL
Sprint: Storage Engines 2018-11-19
Participants:
Story Points: 1

 Description   

In SERVER-37795 we observed excess allocated memory associated with cursors due to WT "modify" operations in 3.6.8. This should not occur in 4.0 because improved cursor caching should limit the number of cursors, but we still see excess allocated memory:

The excess memory in 4.0.3 is more than double that of 3.6.8. Note that the rate of updates is somewhat higher in 4.0.3 than 3.6.8 (probably SERVER-36221), but that doesn't appear to be enough larger to account for the >2x larger excess memory allocated in 4.0.3.

Possibly related: "cached cursor count" seems to grow indefinitely, although we only have that information for the oplog collection and not globally.

Notes:

  • Verified that the memory is associated with cursors by disabling cursor caching in 3.6.8 and the excess memory disappeared.
  • Uesed the heap profiler to confirmed that the excess allocated memory is coming from the same allocation sites in 4.0.3 as 3.6.8.

Repro:

load("/home/bdlucas/mongodb/git/mongo/jstests/libs/parallelTester.js")
 
size = 1000000
bunch = 100
 
// start a bunch of threads inserting to a bunch of collections
// in 3.6.8 this creates a big bunch of cursors (bunch^2, maybe)
// but not in 4.0.3
threads = []
for (var t=0; t<bunch; t++) {
    thread = new ScopedThread(function(t) {
        c = db['c' + t]
        while (true)
            c.insert({})
    }, t)
    threads.push(thread)
    thread.start()
}
 
// while that's going on do some small updates to large documents
// this uses WT "modify" triggering the issue in SERVER-37795 in 3.6.8
// that shouldn't happen in 4.0.3 but it does
for (i = 0; i < 100; i++) {
    c = db['c' + i]
    x = 'x'.repeat(size)
    c.insert({_id: 0, x: x})
}
while (true) {
    for (i = 0; i < 100; i++) {
        c = db['c' + i]
        c.update({_id: 0}, {$inc: {i: 1}})
        c.find({_id: 0}).itcount()
    }
}



 Comments   
Comment by Vamsi Boyapati [ 12/Nov/18 ]

I have checked this symptom of increasing memory with & with-out WT-4418 fix. WT-4418, fixes memory increase specified in this ticket. Closing this as duplicate of WT-4418

Attached the screenshots of FTDC with & with out WT-4418.

Generated at Thu Feb 08 04:47:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.