Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-36569

Config Server unable to restart staging Opslog missing entries error.

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Incomplete
    • Icon: Major - P3 Major - P3
    • None
    • None
    • Replication, Sharding
    • None
    • ALL

    Description

      I have a 13 server MongoDB(4.0.0) cluster consisting of 3 Config Servers(configured as a replication set). the Primary of the CS is crashes stating the error below. Please find the log attachments for more details.

      Rollback common point is { ts: Timestamp(1533794677, 1), t: 5 }
      2018-08-10T06:06:09.177+0000 I ROLLBACK [rsBackgroundSync] finding record store counts
      2018-08-10T06:06:09.177+0000 I REPL [rsBackgroundSync] Incremented the rollback ID to 230
      2018-08-10T06:06:09.177+0000 I ROLLBACK [rsBackgroundSync] Preparing to write deleted documents to a rollback file for collection config.lockpings with uuid ec8d2b3e-3d7e-43b2-acae-b5e68f407945 to /data_storages/data/rollback/config.lockpings/removed.2018-08-10T06-06-09.0.bson
      2018-08-10T06:06:09.178+0000 I STORAGE [rsBackgroundSync] closeCatalog: closing all databases
      2018-08-10T06:06:09.178+0000 I STORAGE [rsBackgroundSync] closeCatalog: closing storage engine catalog
      2018-08-10T06:06:09.178+0000 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down
      2018-08-10T06:06:09.178+0000 F ROLLBACK [rsBackgroundSync] RecoverToStableTimestamp failed. :: caused by :: UnrecoverableRollbackError: No stable timestamp available to recover to. Initial data timestamp: Timestamp(1533794690, 1), Stable timestamp: Timestamp(0, 0)
      2018-08-10T06:06:09.178+0000 I ROLLBACK [rsBackgroundSync] Rollback summary:
      2018-08-10T06:06:09.178+0000 I ROLLBACK [rsBackgroundSync] start time: 2018-08-10T06:06:09.160+0000
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] end time: 2018-08-10T06:06:09.170+0000
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] sync source: ip-172-30-0-149:57042
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] rollback data file directory: /data_storages/data/rollback/config.lockpings
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] rollback id: 230
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] last optime on branch of history rolled back: { ts: Timestamp(1533794690, 1), t: 5 }
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] common point optime: { ts: Timestamp(1533794677, 1), t: 5 }
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] last wall clock time on the branch of history rolled back: 2018-08-09T06:04:50.652+0000
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] common point wall clock time: 2018-08-09T06:04:37.479+0000
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] difference in wall clock times: 13 second(s)
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] shard identity document rolled back: false
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] config server config version document rolled back: false
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] affected sessions: none
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] affected namespaces:
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] config.lockpings
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] counts of interesting commands rolled back:
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] update: 1
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] total number of entries rolled back (including no-ops): 2
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] transition to SECONDARY
      2018-08-10T06:06:09.179+0000 I REPL [rsBackgroundSync] transition to SECONDARY from ROLLBACK
      2018-08-10T06:06:09.179+0000 F REPL [rsBackgroundSync] Rollback failed with unrecoverable error: UnrecoverableRollbackError: Recover to stable timestamp failed.
      2018-08-10T06:06:09.179+0000 F - [rsBackgroundSync] Fatal assertion 50666 UnrecoverableRollbackError: Recover to stable timestamp failed. at src/mongo/db/repl/bgsync.cpp

      Attachments

        1. 1.png
          1.png
          330 kB
        2. 2.png
          2.png
          336 kB
        3. 3.png
          3.png
          319 kB
        4. 4.png
          4.png
          328 kB
        5. 5.png
          5.png
          328 kB
        6. config_primary.log
          23 kB
        7. config_secondary.log
          26 kB

        Activity

          People

            nick.brewer Nick Brewer
            prasadsurase Prasad Surase
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: