[SERVER-35651] Arbiters do not track FCV changes Created: 18/Jun/18  Updated: 29/Oct/23  Resolved: 11/Feb/19

Status: Closed
Project: Core Server
Component/s: Replication, Upgrade/Downgrade
Affects Version/s: None
Fix Version/s: 4.1.8

Type: Bug Priority: Major - P3
Reporter: Eric Milkie Assignee: Daniel Gottlieb (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Documented
is documented by DOCS-12471 Docs for SERVER-35651: Arbiters do no... Closed
Related
is related to DOCS-13029 Note that arbiters will always return... Closed
Backwards Compatibility: Minor Change
Operating System: ALL
Sprint: Repl 2018-07-02, Repl 2018-07-16, Storage NYC 2019-02-11
Participants:
Case:

 Description   

Because arbiters do not track FCV (since they do not follow the oplog and thus do not currently know when it is upgraded), the WiredTiger data format for arbiters is stuck on 2.9 (the version associated with MongoDB 3.2). We must keep dragging along the parsing code and associated test code for that version for each new release of MongoDB, indefinitely.

Some ways to fix this would be:

  • upgrade arbiters to the 4.2 data format immediately, the first time a 4.2 binary is run, and change the downgrade instructions to delete all arbiter datafiles on downgrade.
  • change the way FCV changes are propagated so arbiters hear about them.
  • deprecate and remove support for non-replicating arbiters.


 Comments   
Comment by Daniel Gottlieb (Inactive) [ 11/Feb/19 ]

louisa.berger this patch will be in the next development release. However, WT hasn't bumped the data file version. That means a 4.0 binary can* come up on data files left behind by 4.1.8. I apologize for the difficulties.

Comment by Githook User [ 11/Feb/19 ]

Author:

{'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}

Message: SERVER-35651: Don't downgrade data files when a 4.2 binary running as an arbiter is shut down.

To downgrade binaries for an arbiter, the user must delete the dbpath.
Branch: master
https://github.com/mongodb/mongo/commit/646ff4b0ab9669f2260c6a017bbf3d45d1e48e6a

Comment by Tess Avitabile (Inactive) [ 23/Jan/19 ]

milkie, would it make sense for the storage team to do this work? Replication would require guidance on this.

Comment by Eric Milkie [ 23/Jan/19 ]

The server work for this is to change the startup logic for arbiters to immediately upgrade the datafile format to 4.2 instead of leaving it alone.  The first version we would be able to remove the parsing code would be 4.4, since 4.2 mongod binaries need to be able to parse arbiter datafiles upgraded from 4.0.

Comment by Alyson Cabral (Inactive) [ 23/Jan/19 ]

I don't think there's server work here. milkie has the code been removed? We should create a docs ticket and an automation ticket to change the downgrade instructions.

Comment by Gregory McKeon (Inactive) [ 10/Sep/18 ]

alyson.cabral is there anything else for you to do here, or can we hand this back to the repl team for scheduling?

Comment by Eric Milkie [ 25/Jul/18 ]

Dropping the local database of arbiters on downgrade should be the only downstream work necessary for this (other than adding new documentation as appropriate.)

Comment by Louisa Berger [ 25/Jul/18 ]

sure, let's chat about what this actually entails – i'm unclear what changes automation would have to make

Comment by Gregory McKeon (Inactive) [ 25/Jul/18 ]

alyson.cabral louisa.berger is going to start following this ticket - can you sync with her?

Comment by Alyson Cabral (Inactive) [ 20/Jun/18 ]

I would like us to make sure the automation agent is prepared to make these changes to support downgrade automatically. We should lineup with their timeline.

Comment by Spencer Brody (Inactive) [ 20/Jun/18 ]

What do you think we need to make a final decision? You, me, Tess and alyson.cabral are all on board.

Comment by Eric Milkie [ 20/Jun/18 ]

I don’t think we have a full decision yet on what we are doing.

Comment by Tess Avitabile (Inactive) [ 20/Jun/18 ]

Got it. I think it's acceptable that you will lose data stored in the local db on arbiters upon downgrade.

milkie, do you want to close this ticket and notify docs and cloud when you change the upgrade logic for arbiters?

Comment by Eric Milkie [ 20/Jun/18 ]

Right, the only issue with deleting the data files would be if the user had stored anything in the local db.
Also automation will need to amend their downgrade scripts.

Comment by Spencer Brody (Inactive) [ 20/Jun/18 ]

I think we probably don't need to give any instructions on re-establishing the config. Assuming they don't also delete the data files from their data bearing nodes, those nodes will have the config and will start sending heartbeats to the arbiter on startup, which will in turn inform the arbiter of the config.

Comment by Eric Milkie [ 20/Jun/18 ]

If we were to choose the first option, the first version we would be able to remove the parsing code would be 4.4, since 4.2 mongod binaries need to be able to parse arbiter datafiles upgraded from 4.0. What we would do now in the master branch is to change the upgrade logic for arbiters to immediately upgrade the datafile format to 4.2 instead of leaving it alone.

Comment by Tess Avitabile (Inactive) [ 19/Jun/18 ]

milkie, are you planning on removing this code for 4.2 (so we only need to change the downgrade instructions for 4.2)?

spencer, are there additional downgrade instructions, such as how to give the downgraded arbiter the config?

Comment by Spencer Brody (Inactive) [ 18/Jun/18 ]

I for one am pretty comfortable with the first option, and that seems like the least amount of work.

alyson.cabral

Generated at Thu Feb 08 04:40:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.