[SERVER-20874] replSet write stall under stress with wiredTiger Created: 09/Oct/15 Updated: 12/Nov/15 Resolved: 11/Nov/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, WiredTiger |
| Affects Version/s: | 3.1.9 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Blocker - P1 |
| Reporter: | Rui Zhang (Inactive) | Assignee: | Michael Cahill (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | sys-perf-reg | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||
| Issue Links: |
|
||||
| Operating System: | ALL | ||||
| Steps To Reproduce: | Setup replSet with secondaries |
||||
| Sprint: | Performance A (10/08/15), Performance B (11/02/15) | ||||
| Participants: | |||||
| Description |
| Comments |
| Comment by Rui Zhang (Inactive) [ 11/Nov/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The recent sys-perf test passed, and the latest longevity test also successfully done with loading phase. I also did manual test, the latest master is fine. Close this as gone away longevity ongoing run: https://evergreen.mongodb.com/task/mongo_longevity_linux_wt_shard_shard_cluster_test_fd539883bfc534a7f16aa352d503c903e9d367df_15_11_11_17_47_43 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Alexander Gorrod [ 26/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I've reproduced this performance issue using the tip of master (3d74ad6da4dfeb293cd77ae71583740f008e7007). Since there wasn't information about the replica set configuration in the ticket, I created one:
With matching configurations for two secondaries.
I start all of the nodes on a single machine, and connect them into a replica set. Then I start running the YCSB load against my new replica set as per the instructions above. The results are poor:
I suspect this is exacerbated by me choosing a relatively small cache size (4GB), which means that the WiredTiger needs to do more work to maintain old snapshots. Looking at the state of the system, I see that old named snapshots are being removed, but they are a long way behind the current transaction in the system. For example, in a debugger I see:
That means that the current transaction ID is ~13.5 million. The oldest named live snapshot is ~0.5 million. Meaning that WiredTiger needs to keep the updates from 13 million transactions available to satisfy the oldest named snapshot. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Martin Bligh [ 23/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Dan and I were just discussing if this might be related to | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Fangcheng Sun [ 16/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
We have similar issue on Mongod 3.0.4 and 3.0.7. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Rui Zhang (Inactive) [ 13/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
add call-tree from gdbmon (call-tree-replSet-stall.html), this is during the phase with very low insert rate at ~10-20 insert per second.
looked at the call-graph, it seems wait is mostly in eviction related function, maybe WT related issue? I am re-run V0 protocol just to be sure it passes. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 12/Oct/15 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
rui.zhang, can you clearly state the reproduction steps needed?
|