Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-57228

Config Server crashes when updating FCV using an inconsistent FCV document

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 5.0.0-rc1, 5.1.0-rc0
    • Affects Version/s: None
    • Component/s: Sharding
    • None
    • Fully Compatible
    • ALL
    • v5.0
    • Hide

      The steps to reproduce the problem with a sharded cluster from a Mongo Shell connected to the Mongo Router are:

      // This is an environment using FCV 5.0
mongos> db.system.version.find()
{ "_id" : "featureCompatibilityVersion", "version" : "5.0" }

// Explicitly change the persisted document required by the FCV logic providing inconsistent information
mongos> db.system.version.update({"_id" : "featureCompatibilityVersion"}, {$set: {"version" : "4.4", "targetVersion" : "5.0"}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

// This causes the Config Server's crash
mongos> db.adminCommand({"setFeatureCompatibilityVersion" : "5.0"})
      Show
      The steps to reproduce the problem with a sharded cluster from a Mongo Shell connected to the Mongo Router are: // This is an environment using FCV 5.0 mongos> db.system.version.find() { "_id" : "featureCompatibilityVersion" , "version" : "5.0" } // Explicitly change the persisted document required by the FCV logic providing inconsistent information mongos> db.system.version.update({ "_id" : "featureCompatibilityVersion" }, {$set: { "version" : "4.4" , "targetVersion" : "5.0" }}) WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 }) // This causes the Config Server's crash mongos> db.adminCommand({ "setFeatureCompatibilityVersion" : "5.0" })
    • Sharding EMEA 2021-05-31
    • 160

      When receiving an FCV update request, the Config Server relies on the current document admin.system.version {"_id": "featureCompatibilityVersion"} to determine whether to recover from a previous interrupted run.

      The Config Server could crash during a FCV upgrade and after an explicit (and illegal) amend of such document. The invalid persisted information leads the Config Server to execute a wrong logic that requires the availability of missing information in that document (that is changeTimestamp), and then hit and invariant.

      In this scenario, the FCV document is syntactically but not semantically correct.

            Assignee:
            antonio.fuschetto@mongodb.com Antonio Fuschetto
            Reporter:
            antonio.fuschetto@mongodb.com Antonio Fuschetto
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: