[SERVER-31099] Automate testing when oldest_timestamp stalls Created: 15/Sep/17 Updated: 30/Oct/23 Resolved: 01/Nov/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | None |
| Fix Version/s: | 3.6.0-rc3 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Eric Milkie | Assignee: | Sulabh Mahajan |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||
| Sprint: | Storage 2017-10-02, Storage 2017-10-23, Storage 2017-11-13 | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
Devise tests for behavior of the system when timestamped writes continue to happen while the oldest_timestamp ceases to be updated. This situation can happen when a majority of secondaries stop replicating in a replica set. |
| Comments |
| Comment by Githook User [ 01/Nov/17 ] |
|
Author: {'email': 'sulabh.mahajan@mongodb.com', 'name': 'Sulabh Mahajan', 'username': 'sulabhM'}Message: |
| Comment by Sulabh Mahajan [ 27/Oct/17 ] |
|
I retested today with latest WT develop and mongo master. I don't see stalls now:
|
| Comment by Alexander Gorrod [ 27/Oct/17 ] |
|
sulabh.mahajan there has been some additional work done in |
| Comment by Sulabh Mahajan [ 27/Sep/17 ] |
|
milkie unfortunately that's true. I have discussed these results with michael.cahill, so he is aware of these stalls. These stalls correspond to the checkpoints reading back data from the LAS file and then writing out the checkpoint. The work with |
| Comment by Eric Milkie [ 27/Sep/17 ] |
|
If I am reading this graph correctly, it says that for about 100 seconds there were 0 writes per second? (From ~380 to ~500.) |
| Comment by Sulabh Mahajan [ 27/Sep/17 ] |
|
Attached is perf degradation graph as LAS file gets used because of pinned timestamp, for MongoDB-3.6 with changes being made by Michael for
|
| Comment by Eric Milkie [ 21/Sep/17 ] |
|
Thanks for that testing, Sulabh. |
| Comment by Sulabh Mahajan [ 21/Sep/17 ] |
|
I did some testing for this ticket today. With the setup and workload from 1. On mongodb master I got similar stall as in |
| Comment by Eric Milkie [ 20/Sep/17 ] |
|
Coincidentally, redbeard0531 has already done a bit of testing here, although unintentionally. While testing the performance of the server with timestamps, he encountered a bug in one-node replica sets for the inMemory storage engine that caused oldest_timestamp to never be updated. I'll be filing a ticket about this soon. Mathias can also assist Sulabh with this oldest_timestamp testing in general. |
| Comment by Alexander Gorrod [ 20/Sep/17 ] |
|
sulabh.mahajan Please take a look at this ticket, and think about crafting a use case. I expect the workload to be similar to the work done in |