Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22016

Fatal assertion 28723 trying to rollback applyOps on a CSRS config server

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Fixed
    • None
    • 3.2.3, 3.3.0
    • Sharding
    • Fully Compatible
    • ALL

    Description

      We deploy replica set between multi location(data center), and We use from mongodb 3.2.
      Here is some problems on some machines, The configsvrs after some machines after some minutes down.
      We create replica set for configsvr from 12 members, that The replica set have 7 vote for primary and the rest is vote=0 and priority=0.
      How to fix my issue?

      configsvr logs:

      2015-12-26T23:46:39.737+0330 I REPL     [ReplicationExecutor] Error in heartbeat request to shard2:47041; HostUnreachable Connection refused
      2015-12-26T23:46:40.425+0330 F REPL     [rsBackgroundSync] can't rollback this command yet: { applyOps: [ { op: "u", b: true, ns: "config.chunks", o: { _id: "INN.destinations-destination_code_MinKey", lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('567eb3043d88f022d5c0cf86'), ns: "INN.destinations", min: { destination_code: MinKey }, max: { destination_code: 0.0 }, shard: "shard1_1" }, o2: { _id: "INN.destinations-destination_code_MinKey" } }, { op: "u", b: true, ns: "config.chunks", o: { _id: "INN.destinations-destination_code_0.0", lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('567eb3043d88f022d5c0cf86'), ns: "INN.destinations", min: { destination_code: 0.0 }, max: { destination_code: MaxKey }, shard: "shard1_1" }, o2: { _id: "INN.destinations-destination_code_0.0" } } ], maxTimeMS: 30000 }
      2015-12-26T23:46:40.425+0330 I REPL     [rsBackgroundSync] cmdname=applyOps
      2015-12-26T23:46:40.425+0330 E REPL     [rsBackgroundSync] replica set fatal exception
      2015-12-26T23:46:40.425+0330 I REPL     [rsBackgroundSync] rollback finished
      2015-12-26T23:46:40.425+0330 I -        [rsBackgroundSync] Fatal assertion 28723 UnrecoverableRollbackError need to rollback, but unable to determine common point between local and remote oplog: replica set fatal exception @ 18752
      2015-12-26T23:46:40.425+0330 I -        [rsBackgroundSync]
       
      ***aborting after fassert() failure
      

      Attachments

        Activity

          People

            schwerin@mongodb.com Andy Schwerin
            ali.hallaji1@gmail.com Ali Hallaji
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: