[SERVER-45137] Increasing memory allocation in Top::record with high rate of collection creates and drops Created: 13/Dec/19 Updated: 29/Oct/23 Resolved: 31/Jan/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Catalog |
| Affects Version/s: | 4.2.1 |
| Fix Version/s: | 4.2.4, 4.3.4 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Bruce Lucas (Inactive) | Assignee: | Gregory Wlodarek |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | KP42 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||
| Backport Requested: |
v4.2
|
||||||||||||||||||||||||
| Sprint: | Execution Team 2020-01-27, Execution Team 2020-02-10 | ||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||
| Description |
|
Heap profiler shows memory usage increasing over time on the secondary. The profile is characteristic of a buffer that grows by doubling each time.
Allocated here:
Reproduced by a high rate of collection creates and drops using mapReduce directed to the primary:
In this repro the mapReduce commands are done on the primary, but the memory increase only seems to happen on the secondary. I don't know whether it is specific to collections created and dropped by mapReduce, and I don't know if it's a 4.2 regression (but I note there appear to have been some considerable code changes in this area). |
| Comments |
| Comment by Githook User [ 11/Feb/20 ] |
|
Author: {'username': 'GWlodarek', 'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com'}Message: create mode 100644 jstests/noPassthroughWithMongod/top_rename.js (cherry picked from commit c2d35dd6214978959a9cfc5dcb813d62ae8981ef) create mode 100644 jstests/noPassthroughWithMongod/top_rename.js |
| Comment by Githook User [ 31/Jan/20 ] |
|
Author: {'username': 'GWlodarek', 'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com'}Message: create mode 100644 jstests/noPassthroughWithMongod/top_rename.js |
| Comment by Gregory Wlodarek [ 27/Jan/20 ] |
|
After taking another look at this, I've figured out what was causing the _usage map in Top to keep growing in size. Recently, we've refactored the rename collection helpers and replaced OldClientContext with AutoStatsTracker. In the rename code, we create a top-level AutoStatsTracker and after the renaming logic was done, we'd remove the source collection namespace from the _usage map in Top. However, the destructor of our top-level AutoStatsTracker would run afterwards, which calls Top::record() and records the source collection namespace into the _usage map again. |
| Comment by Gregory Wlodarek [ 21/Jan/20 ] |
|
After taking an initial look at this, I believe we never erase from the _usage map in Top, which causes the map's memory footprint to keep growing until the server OOMs. We perform inserts into the map on a hashed namespace string here: https://github.com/mongodb/mongo/blob/b7846ff4dceec36e344b0f87c48783dffa2c6a32/src/mongo/db/stats/top.cpp#L85-L95 But when we perform erase operations on the map, we use an unhashed namespace string here: https://github.com/mongodb/mongo/blob/b7846ff4dceec36e344b0f87c48783dffa2c6a32/src/mongo/db/stats/top.cpp#L145
Edit: this assumption turned out to be false because the StringMap will automatically hash strings if they weren't already hashed. |
| Comment by Eric Milkie [ 04/Jan/20 ] |
|
We scheduled some investigation work for this starting Jan 13 (next week). |
| Comment by Kuan Huang [ 03/Jan/20 ] |
|
Hi, I'd like to follow up on this ticket. In order to keep our operation going, we are doing lots of extra work to bypass this issue. Any eta on when this issue can be resolved? Thank you and we'd really appreciate your response. |