[SERVER-13181] cap compact index build heap memory usage Created: 13/Mar/14 Updated: 08/Jan/24 Resolved: 26/Oct/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance |
| Affects Version/s: | None |
| Fix Version/s: | 3.0.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Davide Italiano | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Done | Votes: | 0 |
| Labels: | 26qa | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Assigned Teams: |
Storage Execution
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Operating System: | ALL | ||||||||||||
| Sprint: | Platforms 15 (06/03/16) | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
Running a longevity compact test I realized mongod memory usage increases all over the time until it eats a quantity of memory >= total available RAM and get kicked out of the scheduling by the OS.
top/htop clearly show on a machine equipped with 48GB of RAM 98% of it eaten by mongod. Running the same test against 2.4.9 shows memory usage doesn't go over 7-8% in the same workload. The dataset is made by 12 different collections sitting in 12 different databases – every collection is made by 1million documents of the form
The code used to generate the random strings can be found here
and the code to populate the database can be found attached (uses the C++ driver) |
| Comments |
| Comment by Eric Milkie [ 26/Oct/16 ] |
|
I believe the memory usage pattern Davide saw on his test is due to |
| Comment by Davide Italiano [ 19/Mar/14 ] |
|
I'm calling compact on 64 collections in parallel and ram usage approaches 80% (on the aforementioned 32GB machine ram) but it looks like doesn't go further. Still a lot wrt < 10% usage in 2.4.9 |
| Comment by Davide Italiano [ 18/Mar/14 ] |
|
I see a label: memory usage (top). top(1) by default report memory usage as percentage of overall available RAM (which in this case is 32GB, for what it's worth). |
| Comment by Mark Benvenuto [ 18/Mar/14 ] |
|
What is the chart in since i did not see a Y-axis label. Is it in GB? |
| Comment by Githook User [ 17/Mar/14 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: (cherry picked from commit 57eea817bbf74f34113b61e3fbae36d08132b770) |
| Comment by Githook User [ 17/Mar/14 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: (cherry picked from commit 4f05d9ec581d0ab770200e3629ba25eee3715585) |
| Comment by Githook User [ 17/Mar/14 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message:
(cherry picked from commit 25aac5429dca3dfae108c6137a12d1ed48483b82) |
| Comment by Githook User [ 17/Mar/14 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: (cherry picked from commit 7510a554b8f38db470216c8213563b7c25a70896) |
| Comment by Githook User [ 17/Mar/14 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: |
| Comment by Githook User [ 17/Mar/14 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: Revert " This reverts commit eaa83e8afb6bd800cbab0de01b051f2e1460a432. |
| Comment by Githook User [ 17/Mar/14 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: |
| Comment by Githook User [ 17/Mar/14 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: |
| Comment by Githook User [ 17/Mar/14 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message:
|
| Comment by Githook User [ 17/Mar/14 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: |