[SERVER-67960] Promotion of new 4.2 config server primary stuck creating index on chunks.ns_1_min_1 Created: 11/Jul/22 Updated: 20/Sep/22 Resolved: 20/Sep/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Iván Groenewold | Assignee: | Chris Kelly |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Participants: |
| Description |
|
Summary The config server replica set has 3 nodes running 4.0.28, and I have added 3 new nodes running 4.2.20 for a total of six servers. I have set priority to 1 in just 2 nodes (one 4.0 and one 4.2) to be able to control the promotion process (the rest of the nodes all have prio:0). When trying to promote the 4.2 server via rs.stepDown(), the process doesn't complete and clients start complaining that they are unable to reach primary. Looking at db.currentOp() on the 4.2 host that is trying to become primary I see the following operation which seems to be blocking the promotion process:
The config.chunks collection only has 9 chunks so the index creation should be very fast. |
| Comments |
| Comment by Chris Kelly [ 20/Sep/22 ] | |||||||||
|
Thanks for your report - just to add: MongoDB uses a write concern of {{"majority" }}when writing to config servers already in 4.2. In 4.4+, setDefaultRWConcern is introduced instead. Starting in 5.0, we completely ignore getLastErrorDefaults (
Regards, Christopher | |||||||||
| Comment by Iván Groenewold [ 15/Jul/22 ] | |||||||||
|
For anyone that runs into a similar issue the problem in this case was the config server replica set was provisioned with a non-standard default write concern:
instead of the default values:
changing to w:1 fixed the problem. For some reason this setting didn't cause issues in elections in versions earlier than 4.2 |