[SERVER-61909] Hang inserting or deleting document with large number of index entries Created: 03/Dec/21 Updated: 29/Oct/23 Resolved: 22/Nov/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 6.2.0-rc3, 6.3.0-rc0, 6.0.5, 5.0.16 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Bruce Lucas (Inactive) | Assignee: | Yujin Kang Park |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Minor Change | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Backport Requested: |
v6.2, v6.0, v5.0
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | Execution Team 2022-10-17, Execution Team 2022-10-31, Execution Team 2022-11-14, Execution Team 2022-11-28 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Linked BF Score: | 164 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
Inserting a document that creates a large number of index entries can create a large amount of dirty data in a single transaction, causing it to be canceled and retried indefinitely, resulting in a hang. For example on a node with a 256 MB cache, create a text index then insert a document with a large string to be indexed, or equivalently a lot of terms to be indexed:
This will hang after a few documents, with high cache pressure, and the following emited repeatedly in the log:
This will effectively make the server inoperational due to cache pressure. If it occurs on the secondaries they will stall because it will prevent completion of the current batch. This is a regression as these inserts complete successfully (even if somewhat slowly) in 4.2. I think this is related to
|
| Comments |
| Comment by Githook User [ 20/Mar/23 ] |
|
Author: {'name': 'Yu Jin Kang Park', 'email': 'yujin.kang@mongodb.com', 'username': 'ykangpark'}Message: (cherry picked from commit 9ddbd3512044ca6508cb72d9a321391d45c18140) |
| Comment by Githook User [ 06/Feb/23 ] |
|
Author: {'name': 'Yu Jin Kang Park', 'email': 'yujin.kang@mongodb.com', 'username': 'ykangpark'}Message: (cherry picked from commit 9ddbd3512044ca6508cb72d9a321391d45c18140) |
| Comment by Githook User [ 12/Dec/22 ] |
|
Author: {'name': 'Yu Jin Kang Park', 'email': 'yujin.kang@mongodb.com', 'username': 'ykangpark'}Message: |
| Comment by Githook User [ 02/Dec/22 ] |
|
Author: {'name': 'Yu Jin Kang Park', 'email': 'yujin.kang@mongodb.com', 'username': 'ykangpark'}Message: (cherry picked from commit 9ddbd3512044ca6508cb72d9a321391d45c18140) |
| Comment by Yujin Kang Park [ 22/Nov/22 ] |
|
Some more details can be found here: |
| Comment by Githook User [ 21/Nov/22 ] |
|
Author: {'name': 'Yu Jin Kang Park', 'email': 'yujin.kang@mongodb.com', 'username': 'ykangpark'}Message: |
| Comment by Yujin Kang Park [ 28/Oct/22 ] |
|
Adding dependency on |
| Comment by Josef Ahmad [ 20/Sep/22 ] |
|
I'm going to re-evaluate this ticket once |
| Comment by Josef Ahmad [ 17/Feb/22 ] |
|
I'm making this ticket dependent on |
| Comment by Bruce Lucas (Inactive) [ 09/Dec/21 ] |
|
This can also occur deleting such a document. |
| Comment by Eric Milkie [ 03/Dec/21 ] |
|
Ultimately, the project to permit uncommitted writes to be evicted will solve this. I don't have good ideas on how to avoid this problem in the interim, other than turning off or tuning the oldest-transaction-rolled-back behavior. The issue here is that because that rollback behavior is asynchronous and non-deterministic, it allows situations where the transaction manages to commit on a primary node but the equivalent replicated transactions on secondaries always get rolled back. |