[SERVER-19122] Out-of-order _id index inserts on secondary contribute to replica lag under WiredTiger Created: 24/Jun/15 Updated: 06/Dec/22 Resolved: 02/Apr/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Bruce Lucas (Inactive) | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Assigned Teams: |
Storage Execution
|
||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Sprint: | QuInt A (10/12/15), QuInt B (11/02/15) | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
In some use cases inserts into the _id index are done in monotonically increasing _id order on the primary, and that is a case that WT optimizes for. However because of the way oplog entries are divided among the worker threads in SyncTail::fillWriterVectors on the secondary, the entries in the _id index may not be inserted in order on the secondary, and this is a less optimal path under WT. This can occur for example if multiple collections are being inserted into, one thread per collection, using a monotonic _id, such as ObjectId; in this case inserts into the _id index are in order in each WT index table on the primary, but not on the secondary. (This issue does not occur with multiple threads inserting into a single collection because the _id index inserts are not in order on either the primary or the secondary, so parity is maintained). We can control whether inserts are ordered or disordered on the primary by either
Inserts are always disordered on the secondary, so by observing secondary performance relative to primary while varying ordering on the primary as above we can measure the relative impact of the disordered inserts on the secondary. Secondary processing rate relative to primary was measured by comparing number of entries inserted into oplog on secondary with number on primary at a particular point in time during the run:
Note that performance of primary relative to secondary is reduced by either using random _ids, or by inserting into a single collection with multiple threads, both of which disorder the inserts into the _id index on the primary, to match the disordered inserts on the secondary. Finally, we can achieve parity between the primary and the secondary by both
It's unclear whether this issue can be addressed in the mongod layer, or whether an improvement in WT for the out-of-order insertion case is needed. |
| Comments |
| Comment by Daniel Gottlieb (Inactive) [ 20/Oct/17 ] | ||||||||||||||||||||||||||||||||||||||
|
milkie marking as 3.7 Desired - conversation at nyc storage meeting was that we will likely be doing some work on this soon. give a shout if that's incorrect. | ||||||||||||||||||||||||||||||||||||||
| Comment by Martin Bligh [ 28/Aug/15 ] | ||||||||||||||||||||||||||||||||||||||
|
Observation from today: Part of the "out of orderness" is oplog optimes on primary do not strictly match ordering of the _id Another interesting experiment ... run primary flat out for 60s. Get about 200-210K/s on primary, secondary is only getting about 140K/s. | ||||||||||||||||||||||||||||||||||||||
| Comment by Martin Bligh [ 28/Aug/15 ] | ||||||||||||||||||||||||||||||||||||||
|
Tried turning off the oplog commits on the secondary (just force commenting out in code). Even without oplog, secondary only goes half the speed of the primary. Specifically __wt_search_insert seems to be all added overhead - Michael, is this just straight overhead of not doing an append-only insert? PRIMARY
SECONDARY, no oplog
| ||||||||||||||||||||||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 24/Jun/15 ] | ||||||||||||||||||||||||||||||||||||||
|
Created |