[SERVER-24773] Secondary block on initialSync when the next oplog needed was removed on primary Created: 24/Jun/16 Updated: 15/Nov/21 Resolved: 05/Jul/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | 3.0.10, 3.2.3 |
| Fix Version/s: | 3.2.8, 3.3.10 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Zhang Youdong | Assignee: | Judah Schvimer |
| Resolution: | Done | Votes: | 0 |
| Labels: | code-only | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Backwards Compatibility: | Fully Compatible |
| Operating System: | ALL |
| Backport Completed: | |
| Steps To Reproduce: | 1. prepare replicaset with big dataset or small oplog. |
| Sprint: | Repl 16 (06/24/16), Repl 17 (07/15/16) |
| Participants: |
| Description |
|
During initialSync, secondary need to apply oplog serveral times which will call _applyOplogUntil, if the next oplog was removed on primary, the secondary will block on this call, resync command will have no effect in this sutiation, and the process cannot be killed normally.
|
| Comments |
| Comment by Ramon Fernandez Marina [ 08/Jul/16 ] | |||||||||||
|
All, MongoDB release candidate 3.2.8-rc1 is now available for download, and includes a fix for this issue. The MongoDB 3.2.8 release is scheduled for next week. Regards, | |||||||||||
| Comment by Zhang Youdong [ 06/Jul/16 ] | |||||||||||
|
Judah Schvimer I got your point,great thanks. | |||||||||||
| Comment by Judah Schvimer [ 05/Jul/16 ] | |||||||||||
|
The resync command will be fixed in 3.4 with other initial sync work we are doing. For now you should be able to resync simply by shutting down and restarting your server. We will try to fix this in 3.2 in a future commit. | |||||||||||
| Comment by Zhang Youdong [ 05/Jul/16 ] | |||||||||||
|
Judah Schvimer I think _initialSyncRequestedFlag also should be checked to make the 『resync』 command work in this situation. | |||||||||||
| Comment by Githook User [ 05/Jul/16 ] | |||||||||||
|
Author: {u'username': u'judahschvimer', u'name': u'Judah Schvimer', u'email': u'judah@mongodb.com'}Message: (cherry picked from commit 08ee223880898fc2153cf4eedc124fc2e4dfc133) | |||||||||||
| Comment by Githook User [ 05/Jul/16 ] | |||||||||||
|
Author: {u'username': u'judahschvimer', u'name': u'Judah Schvimer', u'email': u'judah@mongodb.com'}Message: | |||||||||||
| Comment by Zhang Youdong [ 02/Jul/16 ] | |||||||||||
|
Judah Schvimer , that's great, resync and shutdown is really important for users in this case, they cannot do anything else. | |||||||||||
| Comment by Judah Schvimer [ 01/Jul/16 ] | |||||||||||
|
Hi, Thank you very much for the detailed clarification and for finding this bug. What you're saying is correct and I've been able to reproduce it on 3.2. I'm also glad to hear that you enjoyed MongoDB World! We will fix _applyOplogUntil to be shutdown aware and terminate on shutdown. Thanks, | |||||||||||
| Comment by Zhang Youdong [ 01/Jul/16 ] | |||||||||||
|
hi,Judah Schvimer, sorry for reply so late, I am atending the MongoDB World 2016 in New York these days, it's really a great conference. This problem is hard to reproduce by constructing data and oplog config, but it think it's abviously by reading the source code, and I will keep working on how to reproduce it . Here is the first three event sequences
the oplog from t1 to t2 is fetched by produceThread, if the some oplog in this range is removed between t1 and t2, the _initialSyncApplyOplog will never return. Next I will explain why resync and shutdown will not work in this case. In SyncTail::oplogApplication(), if the resync command is received, it will go to the very begining to do initial sync.
but in InitialSync::oplogApplication When it's waiting the next oplog, a resync will not restart the initial sync. | |||||||||||
| Comment by Judah Schvimer [ 27/Jun/16 ] | |||||||||||
|
Hi Zhang, I have tried to reproduce your failure, but have been unable to get to a state where I cannot kill the process normally. I started up a node with a 1MB oplog. While inside of initial sync, I did 40 inserts of 100KB documents so that the first ones were forced off the end of the oplog. The new node then said it was too stale and went into RECOVERY mode, at which point the _applyOplogUntil thread continued to run as you specified. I connected to the new node from a second shell and ran
Thanks, |