[SERVER-12058] Primary should abort if encountered problems writing to the oplog Created: 11/Dec/13 Updated: 21/Sep/17 Resolved: 13/Jun/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, Stability |
| Affects Version/s: | 2.4.8 |
| Fix Version/s: | 2.6.6, 2.7.3 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Alexander Komyagin | Assignee: | Andy Schwerin |
| Resolution: | Done | Votes: | 1 |
| Labels: | elections | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||
| Backport Completed: | |||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||
| Description |
|
In a replica set if primary has problems writing to the oplog, for example because of corruption there, it still continues applying writes, but they never go to oplog. Thus they are not replicating. Additionally, once the problem is identified, there is no easy way to recover the applied data if it's not in the oplog. Primary should abort if it encountered data access related assertions in
|
| Comments |
| Comment by Githook User [ 24/Nov/14 ] |
|
Author: {u'name': u'Dan Pasette', u'email': u'dan@10mongodb.com'}Message: Expect the server to ABRUPT_EXIT when we catch clock skew exceptions in logOp |
| Comment by Githook User [ 24/Nov/14 ] |
|
Author: {u'username': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@mongodb.com'}Message: Letting exceptions escape from logOp() can lead to inconsistencies in the oplog. (cherry picked from commit 8a47cfef3d41decde32b1a425229f1dc1c810f65) Conflicts: |
| Comment by Andy Schwerin [ 18/Aug/14 ] |
|
That is correct. |
| Comment by Kevin Pulo [ 18/Aug/14 ] |
|
Thanks Andy. To be 100% clear, this means that once the mongod has been restarted after this abort, the problematic operation will not have been applied to the data files, right? |
| Comment by Andy Schwerin [ 18/Aug/14 ] |
|
Terminating the server is the only way to roll back the write operation at this point in 2.6. Leaving the server up causes it to contain a write that will not replicate, and cannot be rolled back via replication rollback (which requires the operation to be present on the oplog). |
| Comment by Kevin Pulo [ 18/Aug/14 ] |
|
For anyone watching on, note that the fix here is to abort (ie. exit) the mongod when a problem is detected recording an operation in the oplog (as opposed to the stepDown that is mentioned in the original ticket description). Presumably this is because stepDown would immediately promote to primary one of the secondaries which must be missing the problematic operation. This would leave the replica set running in an inconsistent state, which is basically as bad as leaving the original primary running (and potentially worse in some cases). |
| Comment by Githook User [ 13/Jun/14 ] |
|
Author: {u'username': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@mongodb.com'}Message: Letting exceptions escape from logOp() can lead to inconsistencies in the oplog. |