[SERVER-26685] Primary journal and oplog visibility order must be consistent Created: 18/Oct/16  Updated: 04/Apr/19  Resolved: 07/Nov/16

Status: Closed
Project: Core Server
Component/s: Replication, Storage
Affects Version/s: None
Fix Version/s: 3.2.11, 3.4.0-rc3

Type: Bug Priority: Major - P3
Reporter: Mathias Stearn Assignee: Mathias Stearn
Resolution: Done Votes: 0
Labels: code-and-test
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
is related to SERVER-26981 Update mongo-perf Closed
is related to SERVER-27823 Update microbenchmarks to run against... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Sprint: Repl 2016-11-21
Participants:
Linked BF Score: 0

 Description   

Journal commit order isn't required to match oplog optime order on document-locking storage engines. We have a mechanism to hide oplog ops that are ahead of uncommitted ops, but it currently does not consider durability. This can lead to a scenario where a secondary has ops 1,2,3 while a former primary restarts after a crash with just 1 and 3 but no 2. Since they both have the same highest point (3) they will assume they are consistent.

The fix for this involves two semantic changes related to reading from the oplog:
1) If there are any hidden writes at the time an optime is assigned, we must wait for that write to be durable before making it visible.
2) Reverse oplog cursors should ignore visibility rules and always return the newest committed op.



 Comments   
Comment by Githook User [ 14/Nov/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-26685 Tie oplog visibility to durability

(cherry picked from commit 8a2f2fc1883f5deb1b23915cd7a47686a623ba87)
Branch: v3.2
https://github.com/mongodb/mongo/commit/da854a0793e210a4236dedf9d8895dee1f098125

Comment by Githook User [ 14/Nov/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-26685 move ephemeral bypass for waitUntilDurable into SessionCache
Branch: v3.2
https://github.com/mongodb/mongo/commit/0b3f66a03f21ce0aa0d86f3035ce4b67bcbbcba4

Comment by Githook User [ 14/Nov/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-26685 register Database::AddCollectionChange in the correct order

Doing it before creating the RecordStore lead to rollback of Collection
creation happening after RecordStore creation. This meant the Collection
destructor would call setCappedCallback() on an already destroyed RecordStore.

(cherry picked from commit f985c0ce3fa7efb0e857747f0a72bdef3326ac55)
Branch: v3.2
https://github.com/mongodb/mongo/commit/6271e9904a3a107294434eb61bf62b95c543d7e7

Comment by Githook User [ 09/Nov/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-26685 register Database::AddCollectionChange in the correct order

Doing it before creating the RecordStore lead to rollback of Collection
creation happening after RecordStore creation. This meant the Collection
destructor would call setCappedCallback() on an already destroyed RecordStore.
Branch: master
https://github.com/mongodb/mongo/commit/f985c0ce3fa7efb0e857747f0a72bdef3326ac55

Comment by Githook User [ 07/Nov/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-26685 move ephemeral bypass for waitUntilDurable into SessionCache
Branch: master
https://github.com/mongodb/mongo/commit/4720f9651f5e77af6adc9421a2bbf8c610280de7

Comment by Mathias Stearn [ 04/Nov/16 ]

Reopening due to test failures with inMemory storage engine

Comment by Githook User [ 03/Nov/16 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-26685 Tie oplog visibility to durability
Branch: master
https://github.com/mongodb/mongo/commit/8a2f2fc1883f5deb1b23915cd7a47686a623ba87

Generated at Thu Feb 08 04:12:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.