[SERVER-45912] rollback_fuzzer_unclean_shutdowns can fail after restarting a node during rollback Created: 31/Jan/20  Updated: 29/Oct/23  Resolved: 14/Feb/20

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.3.4

Type: Bug Priority: Major - P3
Reporter: A. Jesse Jiryu Davis Assignee: A. Jesse Jiryu Davis
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Problem/Incident
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:
Linked BF Score: 29

 Description   

The following race can occasionally occur:

  1. rollback_fuzzer_unclean_shutdowns starts a 3-node set
  2. it causes Node 1 to enter rollback
  3. Node 1 increments its Rollback ID
  4. the test restarts Node 1 before it can truncate its oplog
  5. Node 1 restarts, it's still on the wrong branch of history, so it enters rollback again
  6. Node 1 increments its RBID again
  7. In rollback_test.js, transitionToSteadyStateOperations asserts Node 1's RBID is exactly the test's previous known RBID + 1, but fails because it was incremented twice

The same issue affects code in rollback_test_deluxe, although I don't think that copy of the bug can be triggered in fuzz testing.



 Comments   
Comment by Githook User [ 14/Feb/20 ]

Author:

{'username': 'vessy-mongodb', 'name': 'Vesselina Ratcheva', 'email': 'vesselina.ratcheva@10gen.com'}

Message: SERVER-45912 Fix typo in rollback_test.js
Branch: master
https://github.com/mongodb/mongo/commit/149aae77fd00cbb0d5760881e76eae631e1f0e11

Comment by Githook User [ 13/Feb/20 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'username': 'ajdavis', 'email': 'jesse@mongodb.com'}

Message: SERVER-45912 Handle shutdown during rollback in fuzz tests

In rollback_fuzzer_unclean_shutdowns we can restart a node while it's in
ROLLBACK, after it has incremented its Rollback ID but before it has truncated
its oplog. In that case it will restart, pass through states SECONDARY and
ROLLBACK, and increment its Rollback ID again after restarting. Update
rollback_test.js to handle cases where the node enters and exits ROLLBACK and
increments its RBID several times.
Branch: master
https://github.com/mongodb/mongo/commit/9aa9fa4a2844f0fe7890e01d621960a0c64607f6

Generated at Thu Feb 08 05:10:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.