[SERVER-12058] Primary should abort if encountered problems writing to the oplog Created: 11/Dec/13  Updated: 21/Sep/17  Resolved: 13/Jun/14

Status: Closed
Project: Core Server
Component/s: Replication, Stability
Affects Version/s: 2.4.8
Fix Version/s: 2.6.6, 2.7.3

Type: Improvement Priority: Major - P3
Reporter: Alexander Komyagin Assignee: Andy Schwerin
Resolution: Done Votes: 1
Labels: elections
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
is related to SERVER-6981 maxPasses assertion (on allocation fa... Closed
is related to SERVER-17561 Fragmentation/non-linearity in mmapv1... Closed
is related to SERVER-1558 Documents should write checksum on wr... Closed
Tested
Backport Completed:
Participants:
Case:

 Description   

In a replica set if primary has problems writing to the oplog, for example because of corruption there, it still continues applying writes, but they never go to oplog. Thus they are not replicating. Additionally, once the problem is identified, there is no easy way to recover the applied data if it's not in the oplog.

Primary should abort if it encountered data access related assertions in
the oplog collection, including:

  • invalid bsonobjsize
  • delete bucket corrupted


 Comments   
Comment by Githook User [ 24/Nov/14 ]

Author:

{u'name': u'Dan Pasette', u'email': u'dan@10mongodb.com'}

Message: SERVER-12058 fix repl/master1.js

Expect the server to ABRUPT_EXIT when we catch clock skew exceptions in logOp
Branch: v2.6
https://github.com/mongodb/mongo/commit/f81009f1d2604d153cdeda9fb106dc86247e12f4

Comment by Githook User [ 24/Nov/14 ]

Author:

{u'username': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@mongodb.com'}

Message: SERVER-12058 Abort mongod before letting exceptions esacpe from logOp().

Letting exceptions escape from logOp() can lead to inconsistencies in the oplog.

(cherry picked from commit 8a47cfef3d41decde32b1a425229f1dc1c810f65)

Conflicts:
src/mongo/db/repl/oplog.cpp
Branch: v2.6
https://github.com/mongodb/mongo/commit/d5c4eef6c9a6c82f7c84845f63a8d19fa12382c0

Comment by Andy Schwerin [ 18/Aug/14 ]

That is correct.

Comment by Kevin Pulo [ 18/Aug/14 ]

Thanks Andy. To be 100% clear, this means that once the mongod has been restarted after this abort, the problematic operation will not have been applied to the data files, right?

Comment by Andy Schwerin [ 18/Aug/14 ]

Terminating the server is the only way to roll back the write operation at this point in 2.6. Leaving the server up causes it to contain a write that will not replicate, and cannot be rolled back via replication rollback (which requires the operation to be present on the oplog).

Comment by Kevin Pulo [ 18/Aug/14 ]

For anyone watching on, note that the fix here is to abort (ie. exit) the mongod when a problem is detected recording an operation in the oplog (as opposed to the stepDown that is mentioned in the original ticket description). Presumably this is because stepDown would immediately promote to primary one of the secondaries which must be missing the problematic operation. This would leave the replica set running in an inconsistent state, which is basically as bad as leaving the original primary running (and potentially worse in some cases).

Comment by Githook User [ 13/Jun/14 ]

Author:

{u'username': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@mongodb.com'}

Message: SERVER-12058 Abort mongod before letting exceptions esacpe from logOp().

Letting exceptions escape from logOp() can lead to inconsistencies in the oplog.
Branch: master
https://github.com/mongodb/mongo/commit/8a47cfef3d41decde32b1a425229f1dc1c810f65

Generated at Thu Feb 08 03:27:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.