Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-8601

Weird logging behavior when re-attempting 2.2 -> 2.4 upgrade on mongos after first attempt fails

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Minor - P4 Minor - P4
    • 2.4.0-rc2
    • Affects Version/s: 2.4.0-rc0
    • Component/s: None
    • Labels:
      None
    • ALL

      • set up a 2.0 sharded cluster
      • create lots (100+) sharded collections/dbs
      • upgrade cluster to 2.2
      • run a 2.4 mongos with --upgrade flag to start upgrade process
      • while copying chunks is in progress, kill -9 the mongos
      • try to run the upgrade process again

      mongos appears to have hanged on startup, doesn't print anything in the log for a while but eventually it all shows up on one line without linebreaks:

      Sat Feb 16 23:25:21.626 [mongosMain] ERROR: waited 11s for distributed lock configUpgrade for upgrading config database to new format v4waited 22s for distributed lock configUpgrade for upgrading config database to new format v4waited 33s for distributed lock configUpgrade for upgrading config database to new format v4waited 44s for distributed lock configUpgrade for upgrading config database to new format v4waited 55s for distributed lock configUpgrade for upgrading config database to new format v4waited 66s for distributed lock configUpgrade for upgrading config database to new format v4waited 77s for distributed lock configUpgrade for upgrading config database to new format v4waited 88s for distributed lock configUpgrade for upgrading config database to new format v4waited 99s for distributed lock configUpgrade for upgrading config database to new format v4waited 110s for distributed lock configUpgrade for upgrading config database to new format v4waited 121s for distributed lock configUpgrade for upgrading config database to new format v4waited 132s for distributed lock configUpgrade for upgrading config database to new format v4waited 143s for distributed lock configUpgrade for upgrading config database to new format v4waited 154s for distributed lock configUpgrade for upgrading config database to new format v4waited 165s for distributed lock configUpgrade for upgrading config database to new format v4waited 176s for distributed lock configUpgrade for upgrading config database to new format v4waited 187s for distributed lock configUpgrade for upgrading config database to new format v4waited 198s for distributed lock configUpgrade for upgrading config database to new format v4waited 209s for distributed lock configUpgrade for upgrading config database to new format v4waited 220s for distributed lock configUpgrade for upgrading config database to new format v4waited 231s for distributed lock configUpgrade for upgrading config database to new format v4waited 242s for distributed lock configUpgrade for upgrading config database to new format v4waited 253s for distributed lock configUpgrade for upgrading config database to new format v4waited 264s for distributed lock configUpgrade for upgrading config database to new format v4waited 275s for distributed lock configUpgrade for upgrading config database to new format v4waited 286s for distributed lock configUpgrade for upgrading config database to new format v4waited 297s for distributed lock configUpgrade for upgrading config database to new format v4waited 308s for distributed lock configUpgrade for upgrading config database to new format v4waited 319s for distributed lock configUpgrade for upgrading config database to new format v4waited 330s for distributed lock configUpgrade for upgrading config database to new format v4waited 341s for distributed lock configUpgrade for upgrading config database to new format v4waited 352s for distributed lock configUpgrade for upgrading config database to new format v4waited 363s for distributed lock configUpgrade for upgrading config database to new format v4waited 374s for distributed lock configUpgrade for upgrading config database to new format v4waited 385s for distributed lock configUpgrade for upgrading config database to new format v4waited 396s for distributed lock configUpgrade for upgrading config database to new format v4waited 407s for distributed lock configUpgrade for upgrading config database to new format v4waited 418s for distributed lock configUpgrade for upgrading config database to new format v4waited 429s for distributed lock configUpgrade for upgrading config database to new format v4waited 440s for distributed lock configUpgrade for upgrading config database to new format v4waited 451s for distributed lock configUpgrade for upgrading config database to new format v4waited 462s for distributed lock configUpgrade for upgrading config database to new format v4waited 473s for distributed lock configUpgrade for upgrading config database to new format v4waited 484s for distributed lock configUpgrade for upgrading config database to new format v4waited 495s for distributed lock configUpgrade for upgrading config database to new format v4waited 507s for distributed lock configUpgrade for upgrading config database to new format v4waited 518s for distributed lock configUpgrade for upgrading config database to new format v4waited 529s for distributed lock configUpgrade for upgrading config database to new format v4waited 540s for distributed lock configUpgrade for upgrading config database to new format v4waited 551s for distributed lock configUpgrade for upgrading config database to new format v4waited 562s for distributed lock configUpgrade for upgrading config database to new format v4waited 573s for distributed lock configUpgrade for upgrading config database to new format v4waited 584s for distributed lock configUpgrade for upgrading config database to new format v4waited 595s for distributed lock configUpgrade for upgrading config database to new format v4waited 606s for distributed lock configUpgrade for upgrading config database to new format v4waited 617s for distributed lock configUpgrade for upgrading config database to new format v4waited 628s for distributed lock configUpgrade for upgrading config database to new format v4waited 639s for distributed lock configUpgrade for upgrading config database to new format v4waited 650s for distributed lock configUpgrade for upgrading config database to new format v4waited 661s for distributed lock configUpgrade for upgrading config database to new format v4waited 672s for distributed lock configUpgrade for upgrading config database to new format v4waited 683s for distributed lock configUpgrade for upgrading config database to new format v4waited 694s for distributed lock configUpgrade for upgrading config database to new format v4waited 705s for distributed lock configUpgrade for upgrading config database to new format v4waited 716s for distributed lock configUpgrade for upgrading config database to new format v4waited 727s for distributed lock configUpgrade for upgrading config database to new format v4waited 738s for distributed lock configUpgrade for upgrading config database to new format v4waited 749s for distributed lock configUpgrade for upgrading config database to new format v4waited 760s for distributed lock configUpgrade for upgrading config database to new format v4waited 771s for distributed lock configUpgrade for upgrading config database to new format v4waited 782s for distributed lock configUpgrade for upgrading config database to new format v4waited 793s for distributed lock configUpgrade for upgrading config database to new format v4waited 804s for distributed lock configUpgrade for upgrading config database to new format v4waited 815s for distributed lock configUpgrade for upgrading config database to new format v4waited 826s for distributed lock configUpgrade for upgrading config database to new format v4waited 837s for distributed lock configUpgrade for upgrading config database to new format v4waited 848s for distributed lock configUpgrade for upgrading config database to new format v4waited 859s for distributed lock configUpgrade for upgrading config database to new format v4waited 870s for distributed lock configUpgrade for upgrading config database to new format v4waited 881s for distributed lock configUpgrade for upgrading config database to new format v4waited 892s for distributed lock configUpgrade for upgrading config database to new format v4error upgrading config database to v4 :: caused by :: could not acquire upgrade lock for config upgrade to v4 :: caused by :: could not acquire distributed lock configUpgrade for upgrading config database to new format v4 after 900s, other lock may be held: { _id: "configUpgrade", process: "mikes-MacBook-Pro.local:20998:1361074195:16807", state: 2, ts: ObjectId('51205813cc55cc686161adea'), when: new Date(1361074195417), who: "mikes-MacBook-Pro.local:20998:1361074195:16807:mongosMain:282475249", why: "upgrading config database to new format v4" }
      

            Assignee:
            greg_10gen Greg Studer
            Reporter:
            mikeo@mongodb.com Michael O'Brien
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: