[SERVER-68139] Resharding command fails if the projection sort is bigger than 100MB Created: 19/Jul/22 Updated: 29/Oct/23 Resolved: 20/Sep/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 5.0.14, 6.0.3, 6.1.0-rc3, 6.2.0-rc0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Rachita Dhawan | Assignee: | Nandini Bhartiya |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | sharding-nyc-subteam1 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Backport Requested: |
v6.1, v6.0, v5.0
|
||||||||||||||||
| Sprint: | Sharding 2022-09-05, Sharding 2022-09-19, Sharding 2022-10-03 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||
| Story Points: | 3 | ||||||||||||||||
| Description |
|
Problem: The resharding command fails if the newShardKeyProjection size * samplePerChunk * intialNumOfChunks is bigger than 100MB. newShardKeyProjection: size of newShardKey document Default value of initialNumOfChunks is the existing number of chunks before the resharding. Default value of samplePerChunk: 10 Description: Resharding creates a pipeline query command and the query is then sorted on the new Shard Key as part of the projection. If the projection of over 100MB is passed onto sort, the following error is thrown
The error is due to
Steps to reproduce:
|
| Comments |
| Comment by Githook User [ 21/Sep/22 ] |
|
Author: {'name': 'nandinibhartiyaMDB', 'email': 'nandini.bhartiya@mongodb.com', 'username': 'nandinibhartiyaMDB'}Message: (cherry picked from commit 901ce6eed061d21df66d82ded539d294da6d3de4) |
| Comment by Githook User [ 21/Sep/22 ] |
|
Author: {'name': 'nandinibhartiyaMDB', 'email': 'nandini.bhartiya@mongodb.com', 'username': 'nandinibhartiyaMDB'}Message: (cherry picked from commit 901ce6eed061d21df66d82ded539d294da6d3de4) |
| Comment by Githook User [ 21/Sep/22 ] |
|
Author: {'name': 'nandinibhartiyaMDB', 'email': 'nandini.bhartiya@mongodb.com', 'username': 'nandinibhartiyaMDB'}Message: (cherry picked from commit 901ce6eed061d21df66d82ded539d294da6d3de4) |
| Comment by Githook User [ 19/Sep/22 ] |
|
Author: {'name': 'nandinibhartiyaMDB', 'email': 'nandini.bhartiya@mongodb.com', 'username': 'nandinibhartiyaMDB'}Message: |
| Comment by Max Hirschhorn [ 04/Aug/22 ] |
|
We should add C++ test cases to initial_split_policy_test.cpp which exercise $sort needing to spill to disk. The C++ test cases should reproduce the "Sort exceeded memory limit" error without needing to have >100MB worth of data. It looks like the DocumentSourceSort stage supports passing in maxMemoryUsageBytes. Either this mechanism can be used directly or perhaps it is already behind a general failpoint. Feel free to reach out to the Query team over Slack for additional guidance on the testing. |