Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20856

During CSRS upgrade, config server gets stuck in STARTUP2 when restarted with --replSet and --configsvrMode=sccc set

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 3.2.0-rc2
    • Affects Version/s: 3.1.9, 3.2.0-rc0
    • Component/s: Replication
    • Labels:
      None
    • Fully Compatible
    • ALL
    • Hide
      1. Start up a 3.1.9 config server
      2. Do rs.initiate({_id: "csrs", version: 1, configsvr: true, members: [ {_id: 0, host: "HOSTNAME:PORT"}]})
      3. Stop config server
      4. Start config server with --replSet=csrs --configsvrMode=sccc also set
      5. Watch config server get stuck in STARTUP2
      Show
      Start up a 3.1.9 config server Do rs.initiate({_id: "csrs", version: 1, configsvr: true, members: [ {_id: 0, host: "HOSTNAME:PORT"}]}) Stop config server Start config server with --replSet=csrs --configsvrMode=sccc also set Watch config server get stuck in STARTUP2
    • Sharding B (10/30/15)

      During the CSRS upgrade, the first restart of the first config server (with --replSet and --configsvrMode=sccc set) results in the first config server getting stuck in STARTUP2.

      Log file of first config server:

      /tmp/mms-automation/test//versions/mongodb-osx-x86_64-3.1.9/bin/mongod --dbpath=db1 --configsvr --replSet=csrs --configsvrMode=sccc
      2015-10-09T17:18:02.287-0400 I -        [initandlisten] Detected data files in db1 created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2015-10-09T17:18:02.287-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=9G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2015-10-09T17:18:03.316-0400 I STORAGE  [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.$main
      2015-10-09T17:18:03.316-0400 I STORAGE  [initandlisten] Scanning the oplog to determine where to place markers for when to truncate
      2015-10-09T17:18:03.317-0400 I STORAGE  [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
      2015-10-09T17:18:03.317-0400 I STORAGE  [initandlisten] Scanning the oplog to determine where to place markers for when to truncate
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] MongoDB starting : pid=28419 port=27019 dbpath=db1 64-bit host=neurofunk.local
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] 
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] ** NOTE: This is a development version (3.1.9) of MongoDB.
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] **       Not recommended for production.
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] 
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] 
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] db version v3.1.9
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] git version: 9c317d92145a77ab582acb215b3ed41af2f1b33c
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] allocator: system
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] modules: none
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] build environment:
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten]     distarch: x86_64
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten]     target_arch: x86_64
      2015-10-09T17:18:03.327-0400 I CONTROL  [initandlisten] options: { replication: { replSet: "csrs" }, sharding: { clusterRole: "configsvr", configsvrMode: "sccc" }, storage: { dbPath: "db1" } }
      2015-10-09T17:18:03.337-0400 I REPL     [initandlisten] Did not find local voted for document at startup;  NoMatchingDocument Did not find replica set lastVote document in local.replset.election
      2015-10-09T17:18:03.338-0400 I FTDC     [initandlisten] Starting full-time diagnostic data capture with directory 'db1/diagnostic.data'
      2015-10-09T17:18:03.338-0400 W REPL     [ReplicationExecutor] Failed to load timestamp of most recently applied operation; NoMatchingDocument Did not find any entries in local.oplog.rs
      2015-10-09T17:18:03.338-0400 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "csrs", version: 1, configsvr: true, protocolVersion: 1, members: [ { _id: 0, host: "neurofunk.local:27019", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, electionTimeoutOffsetLimitMillis: 2000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
      2015-10-09T17:18:03.338-0400 I REPL     [ReplicationExecutor] This node is neurofunk.local:27019 in the config
      2015-10-09T17:18:03.338-0400 I REPL     [ReplicationExecutor] transition to STARTUP2
      2015-10-09T17:18:03.339-0400 I REPL     [ReplicationExecutor] Starting replication applier threads
      2015-10-09T17:18:03.340-0400 I NETWORK  [initandlisten] waiting for conne2015-10-09T17:18:03.340-0400 I REPL     [rsSync] no valid sync sources found in current replset to do an initial sync
      2015-10-09T17:18:04.345-0400 I REPL     [rsSync] initial sync pending
      2015-10-09T17:18:04.345-0400 I REPL     [rsSync] no valid sync sources found in current replset to do an initial sync
      2015-10-09T17:18:05.350-0400 I REPL     [rsSync] initial sync pending
      2015-10-09T17:18:05.350-0400 I REPL     [rsSync] no valid sync sources found in current replset to do an initial sync
      2015-10-09T17:18:06.353-0400 I REPL     [rsSync] initial sync pending
      2015-10-09T17:18:06.353-0400 I REPL     [rsSync] no valid sync sources found in current replset to do an initial sync
      2015-10-09T17:18:07.358-0400 I REPL     [rsSync] initial sync pending
      2015-10-09T17:18:07.359-0400 I REPL     [rsSync] no valid sync sources found in current replset to do an initial sync
      

            Assignee:
            schwerin@mongodb.com Andy Schwerin
            Reporter:
            tim.olsen@mongodb.com Timothy Olsen (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: