[SERVER-35651] Arbiters do not track FCV changes Created: 18/Jun/18 Updated: 29/Oct/23 Resolved: 11/Feb/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, Upgrade/Downgrade |
| Affects Version/s: | None |
| Fix Version/s: | 4.1.8 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Eric Milkie | Assignee: | Daniel Gottlieb (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Minor Change | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Sprint: | Repl 2018-07-02, Repl 2018-07-16, Storage NYC 2019-02-11 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||
| Description |
|
Because arbiters do not track FCV (since they do not follow the oplog and thus do not currently know when it is upgraded), the WiredTiger data format for arbiters is stuck on 2.9 (the version associated with MongoDB 3.2). We must keep dragging along the parsing code and associated test code for that version for each new release of MongoDB, indefinitely. Some ways to fix this would be:
|
| Comments |
| Comment by Daniel Gottlieb (Inactive) [ 11/Feb/19 ] |
|
louisa.berger this patch will be in the next development release. However, WT hasn't bumped the data file version. That means a 4.0 binary can* come up on data files left behind by 4.1.8. I apologize for the difficulties. |
| Comment by Githook User [ 11/Feb/19 ] |
|
Author: {'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}Message: To downgrade binaries for an arbiter, the user must delete the dbpath. |
| Comment by Tess Avitabile (Inactive) [ 23/Jan/19 ] |
|
milkie, would it make sense for the storage team to do this work? Replication would require guidance on this. |
| Comment by Eric Milkie [ 23/Jan/19 ] |
|
The server work for this is to change the startup logic for arbiters to immediately upgrade the datafile format to 4.2 instead of leaving it alone. The first version we would be able to remove the parsing code would be 4.4, since 4.2 mongod binaries need to be able to parse arbiter datafiles upgraded from 4.0. |
| Comment by Alyson Cabral (Inactive) [ 23/Jan/19 ] |
|
I don't think there's server work here. milkie has the code been removed? We should create a docs ticket and an automation ticket to change the downgrade instructions. |
| Comment by Gregory McKeon (Inactive) [ 10/Sep/18 ] |
|
alyson.cabral is there anything else for you to do here, or can we hand this back to the repl team for scheduling? |
| Comment by Eric Milkie [ 25/Jul/18 ] |
|
Dropping the local database of arbiters on downgrade should be the only downstream work necessary for this (other than adding new documentation as appropriate.) |
| Comment by Louisa Berger [ 25/Jul/18 ] |
|
sure, let's chat about what this actually entails – i'm unclear what changes automation would have to make |
| Comment by Gregory McKeon (Inactive) [ 25/Jul/18 ] |
|
alyson.cabral louisa.berger is going to start following this ticket - can you sync with her? |
| Comment by Alyson Cabral (Inactive) [ 20/Jun/18 ] |
|
I would like us to make sure the automation agent is prepared to make these changes to support downgrade automatically. We should lineup with their timeline. |
| Comment by Spencer Brody (Inactive) [ 20/Jun/18 ] |
|
What do you think we need to make a final decision? You, me, Tess and alyson.cabral are all on board. |
| Comment by Eric Milkie [ 20/Jun/18 ] |
|
I don’t think we have a full decision yet on what we are doing. |
| Comment by Tess Avitabile (Inactive) [ 20/Jun/18 ] |
|
Got it. I think it's acceptable that you will lose data stored in the local db on arbiters upon downgrade. milkie, do you want to close this ticket and notify docs and cloud when you change the upgrade logic for arbiters? |
| Comment by Eric Milkie [ 20/Jun/18 ] |
|
Right, the only issue with deleting the data files would be if the user had stored anything in the local db. |
| Comment by Spencer Brody (Inactive) [ 20/Jun/18 ] |
|
I think we probably don't need to give any instructions on re-establishing the config. Assuming they don't also delete the data files from their data bearing nodes, those nodes will have the config and will start sending heartbeats to the arbiter on startup, which will in turn inform the arbiter of the config. |
| Comment by Eric Milkie [ 20/Jun/18 ] |
|
If we were to choose the first option, the first version we would be able to remove the parsing code would be 4.4, since 4.2 mongod binaries need to be able to parse arbiter datafiles upgraded from 4.0. What we would do now in the master branch is to change the upgrade logic for arbiters to immediately upgrade the datafile format to 4.2 instead of leaving it alone. |
| Comment by Tess Avitabile (Inactive) [ 19/Jun/18 ] |
|
milkie, are you planning on removing this code for 4.2 (so we only need to change the downgrade instructions for 4.2)? spencer, are there additional downgrade instructions, such as how to give the downgraded arbiter the config? |
| Comment by Spencer Brody (Inactive) [ 18/Jun/18 ] |
|
I for one am pretty comfortable with the first option, and that seems like the least amount of work. |