-
Type:
Task
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Cluster Scalability
-
Fully Compatible
-
ClusterScalability 8Jun-22June, ClusterScalability 22Jun-6Jul, ClusterScalability 6Jul-20Jul
-
None
-
None
-
None
-
None
-
None
-
None
-
None
The new commands coordinator uses to communicate with the participants are blocking and stepdowns sever the communication and resets the process. We have stepdown suites today, but the issue is that stepdown is very disruptive to many services and it takes time to elect and fully transition to primary. This is why suites today only attempt to do stepdowns every 8 secs. This leaves very few opportunities to exercise the stepdown logic, especially during the "interesting" moments in resharding.
The idea is to add a facility to allow performing a light weight "stepdown" to the resharding state machine and rerun the state machine stepup procedure. This will allow us to exercise the resharding stepdown/recovery logic more often than a real stepdown. This is not a real stepdown so it won't catch the class of bugs related to a slightly lagged secondary stepping up, but it can still catch classes of bugs related to the recovery logic.
We have to make sure that things that resharding related logic that lives outside the state machines (like writes in the donor shards setting the destinedRecipient fields) will still behave correctly while the "lightweight stepdown" is ongoing (or design the test so that we will overlook that limitation). We might also need to temporarily return some kind of retryable error while the state machine is "restarting".