[SERVER-13181] cap compact index build heap memory usage Created: 13/Mar/14  Updated: 08/Jan/24  Resolved: 26/Oct/16

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: None
Fix Version/s: 3.0.0

Type: Bug Priority: Major - P3
Reporter: Davide Italiano Assignee: Backlog - Storage Execution Team
Resolution: Done Votes: 0
Labels: 26qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File compact_populate.cpp     PNG File memory_usage_compact.png    
Issue Links:
Depends
is depended on by SERVER-12332 choose std::sort or stable_sort for e... Closed
Related
Assigned Teams:
Storage Execution
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Platforms 15 (06/03/16)
Participants:

 Description   

Running a longevity compact test I realized mongod memory usage increases all over the time until it eats a quantity of memory >= total available RAM and get kicked out of the scheduling by the OS.
The following test run continuously in a loop eventually reproduce the behavior described above.

 
var func = function(i) {
    var new_db = "foo" + i.toString();
    db = db.getSiblingDB(new_db);
    var ns = "goo" + i.toString();
    return db.runCommand({compact: ns});
    }
 
 
for (var k = 1; k <= 12; ++k) {
    var new_db = "foo" + k.toString();
    db = db.getSiblingDB(new_db);
    var ns = "goo" + k.toString();
    db[ns].ensureIndex({name: 1});
 
}
 
for (var j = 1; j <= 12; ++j) {
    var start = new Date().getTime();
    threads = [];
 
    for (var k = 1; k <= j; ++k) {
        t = new ScopedThread(func, k);
        threads.push(t);
        t.start();
    }
 
    for (var k in threads) {
        var t = threads[k];
        threads[k].join();
        //printjson(t.returnData());
    }
 
    var elapsed = new Date().getTime() - start;
    print(elapsed);
}

top/htop clearly show on a machine equipped with 48GB of RAM 98% of it eaten by mongod. Running the same test against 2.4.9 shows memory usage doesn't go over 7-8% in the same workload.

The dataset is made by 12 different collections sitting in 12 different databases – every collection is made by 1million documents of the form

{_id: ObjectId(), name: random_string}

The code used to generate the random strings can be found here

#!/bin/bash
 
STRLENGTH=32
NUMFILES=16
RANGE=1000000
 
for i in `seq 1 $NUMFILES`
do
    rm -rf output${i}.txt
done
 
for i in `seq 1 $NUMFILES`
do
    for j in `seq 1 $RANGE`
    do
        base64 /dev/urandom | head -c ${STRLENGTH} >> output${i}.txt
        echo "" >> output${i}.txt
    done
done

and the code to populate the database can be found attached (uses the C++ driver)



 Comments   
Comment by Eric Milkie [ 26/Oct/16 ]

I believe the memory usage pattern Davide saw on his test is due to SERVER-25318. Since there are no further memory leaks, I am going to close this ticket.

Comment by Davide Italiano [ 19/Mar/14 ]

I'm calling compact on 64 collections in parallel and ram usage approaches 80% (on the aforementioned 32GB machine ram) but it looks like doesn't go further. Still a lot wrt < 10% usage in 2.4.9

Comment by Davide Italiano [ 18/Mar/14 ]

I see a label: memory usage (top). top(1) by default report memory usage as percentage of overall available RAM (which in this case is 32GB, for what it's worth).

Comment by Mark Benvenuto [ 18/Mar/14 ]

What is the chart in since i did not see a Y-axis label. Is it in GB?

Comment by Githook User [ 17/Mar/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-13181: Fix build break

(cherry picked from commit 57eea817bbf74f34113b61e3fbae36d08132b770)
Branch: v2.6
https://github.com/mongodb/mongo/commit/7226cdfa5199d4d77171cb274a5ff142a93dfc2c

Comment by Githook User [ 17/Mar/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-13181: Fix memory leaks in indexing

(cherry picked from commit 4f05d9ec581d0ab770200e3629ba25eee3715585)
Branch: v2.6
https://github.com/mongodb/mongo/commit/5109563d5e35a25e246248c761adfaa0e929f2bc

Comment by Githook User [ 17/Mar/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-13181: Fix BtreeBulk leaks

  • Fix leak in buildAnIndex - initiateBulk allocates a BtreeBulk which is never deleted
  • Fix leak in MultiIndexBlock - we are leaking BtreeBulk since we use a raw pointer

(cherry picked from commit 25aac5429dca3dfae108c6137a12d1ed48483b82)
Branch: v2.6
https://github.com/mongodb/mongo/commit/0838167a2769daf211677afcb3bd577f8b99ffd7

Comment by Githook User [ 17/Mar/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-13181: Query Memory Leaks

(cherry picked from commit 7510a554b8f38db470216c8213563b7c25a70896)
Branch: v2.6
https://github.com/mongodb/mongo/commit/e08a0f09d064c9a5bb6212e89b04cef7b94ac073

Comment by Githook User [ 17/Mar/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-13181: Fix build break
Branch: master
https://github.com/mongodb/mongo/commit/57eea817bbf74f34113b61e3fbae36d08132b770

Comment by Githook User [ 17/Mar/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: Revert "SERVER-13181: Fix Memory leaks in WriteOpResult - switch struct to class, and tighten ownership of pointers with explicit set methods"

This reverts commit eaa83e8afb6bd800cbab0de01b051f2e1460a432.
Branch: master
https://github.com/mongodb/mongo/commit/8150cc2965d024e5df989f1fd6742e7b67cf03ec

Comment by Githook User [ 17/Mar/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-13181: Fix memory leaks in indexing
Branch: master
https://github.com/mongodb/mongo/commit/4f05d9ec581d0ab770200e3629ba25eee3715585

Comment by Githook User [ 17/Mar/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-13181: Fix Memory leaks in WriteOpResult - switch struct to class, and tighten ownership of pointers with explicit set methods
Branch: master
https://github.com/mongodb/mongo/commit/eaa83e8afb6bd800cbab0de01b051f2e1460a432

Comment by Githook User [ 17/Mar/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-13181: Fix BtreeBulk leaks

Comment by Githook User [ 17/Mar/14 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-13181: Query Memory Leaks
Branch: master
https://github.com/mongodb/mongo/commit/7510a554b8f38db470216c8213563b7c25a70896

Generated at Thu Feb 08 03:30:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.