[SERVER-56274] TTL deletes are much slower on descending indexes than ascending indexes Created: 22/Apr/21 Updated: 29/Oct/23 Resolved: 03/Oct/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.4.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Louis Williams | Assignee: | Backlog - Storage Engines Team |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Assigned Teams: |
Storage Engines
|
||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||
| Sprint: | Execution Team 2021-06-28, Execution Team 2021-07-12, Execution Team 2021-08-23, Execution Team 2021-09-06 | ||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||
| Description |
|
For unknown reasons, TTL deletes that use a descending index (e.g. {lastUse: -1}) take at least an order of magnitude more time than deletes on an equivalent, ascending index (e.g.{lastUse: 1}). Investigate and attempt to fix this problem. |
| Comments |
| Comment by Louis Williams [ 03/Oct/22 ] | ||||||||||||||||||||
|
This performance problem has gone away since the work combined from | ||||||||||||||||||||
| Comment by Louis Williams [ 01/Dec/21 ] | ||||||||||||||||||||
|
Hi jenny.ren@mail.utoronto.ca, your analysis seems correct to me. A descending index on time requires the TTL monitor to traverse in reverse order, which means that restoring the cursor after each delete traverses in forward order, having to skip over deleted records. Thanks for the investigation! | ||||||||||||||||||||
| Comment by Jen X [ 01/Dec/21 ] | ||||||||||||||||||||
|
Hi, I'd like to share some findings on the root cause. The following snippet of code is called when repositioning the cursor (in "__wt_btcur_search_near"):
As you can see, the search is hard-coded to search one direction (next) first, and when nothing is found, it searches the reverse direction (prev). For the ascending index, the cursor moves in the same direction as "__wt_btcur_next", so it finds the next valid record right away and there is no need to call "__wt_btcur_prev". For descending index, the direction the cursor moves is reversed, but the code still calls "__wt_btcur_next" first only to find deleted records. Then it calls "__wt_btcur_prev" to successfully reposition the cursor, but extra work is wasted. If you change the code to execute "__wt_btcur_prev" first, the ascending index delete becomes the slow one and descending index delete becomes the fast one.
| ||||||||||||||||||||
| Comment by Henrik Edin [ 15/Sep/21 ] | ||||||||||||||||||||
|
Assigning to Storage Engines per Louis's comment. | ||||||||||||||||||||
| Comment by Louis Williams [ 03/Jun/21 ] | ||||||||||||||||||||
|
The common theme with this and I'm not sure why this only happens on descending indexes. Descending TTL indexes delete keys from the end of an index and traverse in reverse order. Maybe there is some deficiency that traversing tombstones at the end of a table and not the front? But that doesn't explain the behavior in Since Storage Engines is going to scope a project per this comment, I'm going to assign this to their backlog. | ||||||||||||||||||||
| Comment by Louis Williams [ 01/Jun/21 ] | ||||||||||||||||||||
|
I was able to reproduce this regression in the attached test (ttl_delete_repro.js). (Note: I started mongod using --replSet rs --setParameter ttlMonitorSleepSecs=1) The output is as follows:
This appears to have the same symptoms and unexplained behavior as Ascending: Descending: | ||||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 22/Apr/21 ] | ||||||||||||||||||||
|
In my testing I saw 1) a regression in both ascending and descending delete rates from 4.4 to 4.2, but 2) a much larger regression in descending than ascending.
|