Config Server unable to restart staging Opslog missing entries error.

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Incomplete
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Replication, Sharding
    • None
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      I have a 13 server MongoDB(4.0.0) cluster consisting of 3 Config Servers(configured as a replication set). the Primary of the CS is crashes stating the error below. Please find the log attachments for more details.

      Rollback common point is { ts: Timestamp(1533794677, 1), t: 5 }
      2018-08-10T06:06:09.177+0000 I ROLLBACK [rsBackgroundSync] finding record store counts
      2018-08-10T06:06:09.177+0000 I REPL [rsBackgroundSync] Incremented the rollback ID to 230
      2018-08-10T06:06:09.177+0000 I ROLLBACK [rsBackgroundSync] Preparing to write deleted documents to a rollback file for collection config.lockpings with uuid ec8d2b3e-3d7e-43b2-acae-b5e68f407945 to /data_storages/data/rollback/config.lockpings/removed.2018-08-10T06-06-09.0.bson
      2018-08-10T06:06:09.178+0000 I STORAGE [rsBackgroundSync] closeCatalog: closing all databases
      2018-08-10T06:06:09.178+0000 I STORAGE [rsBackgroundSync] closeCatalog: closing storage engine catalog
      2018-08-10T06:06:09.178+0000 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down
      2018-08-10T06:06:09.178+0000 F ROLLBACK [rsBackgroundSync] RecoverToStableTimestamp failed. :: caused by :: UnrecoverableRollbackError: No stable timestamp available to recover to. Initial data timestamp: Timestamp(1533794690, 1), Stable timestamp: Timestamp(0, 0)
      2018-08-10T06:06:09.178+0000 I ROLLBACK [rsBackgroundSync] Rollback summary:
      2018-08-10T06:06:09.178+0000 I ROLLBACK [rsBackgroundSync] start time: 2018-08-10T06:06:09.160+0000
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] end time: 2018-08-10T06:06:09.170+0000
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] sync source: ip-172-30-0-149:57042
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] rollback data file directory: /data_storages/data/rollback/config.lockpings
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] rollback id: 230
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] last optime on branch of history rolled back: { ts: Timestamp(1533794690, 1), t: 5 }
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] common point optime: { ts: Timestamp(1533794677, 1), t: 5 }
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] last wall clock time on the branch of history rolled back: 2018-08-09T06:04:50.652+0000
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] common point wall clock time: 2018-08-09T06:04:37.479+0000
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] difference in wall clock times: 13 second(s)
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] shard identity document rolled back: false
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] config server config version document rolled back: false
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] affected sessions: none
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] affected namespaces:
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] config.lockpings
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] counts of interesting commands rolled back:
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] update: 1
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] total number of entries rolled back (including no-ops): 2
      2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] transition to SECONDARY
      2018-08-10T06:06:09.179+0000 I REPL [rsBackgroundSync] transition to SECONDARY from ROLLBACK
      2018-08-10T06:06:09.179+0000 F REPL [rsBackgroundSync] Rollback failed with unrecoverable error: UnrecoverableRollbackError: Recover to stable timestamp failed.
      2018-08-10T06:06:09.179+0000 F - [rsBackgroundSync] Fatal assertion 50666 UnrecoverableRollbackError: Recover to stable timestamp failed. at src/mongo/db/repl/bgsync.cpp

        1. config_secondary.log
          26 kB
          Prasad Surase
        2. config_primary.log
          23 kB
          Prasad Surase
        3. 5.png
          328 kB
          Prasad Surase
        4. 4.png
          328 kB
          Prasad Surase
        5. 3.png
          319 kB
          Prasad Surase
        6. 2.png
          336 kB
          Prasad Surase
        7. 1.png
          330 kB
          Prasad Surase

            Assignee:
            Nick Brewer (Inactive)
            Reporter:
            Prasad Surase
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: