[SERVER-71683] unbounded memory growth during tenant migrations Created: 29/Nov/22 Updated: 29/Oct/23 Resolved: 07/Dec/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 6.1.1, 6.2.0-rc3, 6.3.0-rc0 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Eric Milkie | Assignee: | Suganthi Mani |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Operating System: | ALL | ||||||||||||
| Backport Requested: |
v6.2
|
||||||||||||
| Steps To Reproduce: | git am the attached patch. |
||||||||||||
| Sprint: | Server Serverless 2022-12-12 | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
It appears that there is no backpressure between reading from a donor and writing on a recipient; there is an in-memory buffer that lives on the recipient that tenant migration writer threads pull from to perform writes. This buffer can grow without bound if writing on the recipient is significantly slower compared to reading on the donor. |
| Comments |
| Comment by Githook User [ 07/Dec/22 ] |
|
Author: {'name': 'Suganthi Mani', 'email': 'suganthi.mani@mongodb.com', 'username': 'smani87'}Message: (cherry picked from commit 5fee6fff13b8a0b9f96f6bbe228afcd9514ac952) |
| Comment by Githook User [ 07/Dec/22 ] |
|
Author: {'name': 'Suganthi Mani', 'email': 'suganthi.mani@mongodb.com', 'username': 'smani87'}Message: (cherry picked from commit 5fee6fff13b8a0b9f96f6bbe228afcd9514ac952) |
| Comment by Suganthi Mani [ 07/Dec/22 ] |
|
Just for those who are watching this ticket, we considered following 3 options ( We chose option#1 for following reasons 1)Simplicity 2) This is a problem only for MTM protocol and this protocol will be retired soon and will be replaced with split and merge.) Option#1: Tenant collection cloner reads the next batch from socket buffer to in-memory buffer only after inserting all the documents in the in-memory buffer to the collection, by running insert docs in-line with handleNextBatch()
Option#2: Put explicit max size limit to the in-memory buffer - TenantCollectionCloner::_documentsToInsert
Option#3: Like
|
| Comment by Githook User [ 07/Dec/22 ] |
|
Author: {'name': 'Suganthi Mani', 'email': 'suganthi.mani@mongodb.com', 'username': 'smani87'}Message: |
| Comment by Suganthi Mani [ 01/Dec/22 ] |
|
Reposting milkie@mongodb.com's slack response on why this ticket is marked as P2-Critical
|