Details
-
Bug
-
Resolution: Incomplete
-
Major - P3
-
None
-
None
-
None
-
ALL
Description
I have a 13 server MongoDB(4.0.0) cluster consisting of 3 Config Servers(configured as a replication set). the Primary of the CS is crashes stating the error below. Please find the log attachments for more details.
Rollback common point is { ts: Timestamp(1533794677, 1), t: 5 }
2018-08-10T06:06:09.177+0000 I ROLLBACK [rsBackgroundSync] finding record store counts
2018-08-10T06:06:09.177+0000 I REPL [rsBackgroundSync] Incremented the rollback ID to 230
2018-08-10T06:06:09.177+0000 I ROLLBACK [rsBackgroundSync] Preparing to write deleted documents to a rollback file for collection config.lockpings with uuid ec8d2b3e-3d7e-43b2-acae-b5e68f407945 to /data_storages/data/rollback/config.lockpings/removed.2018-08-10T06-06-09.0.bson
2018-08-10T06:06:09.178+0000 I STORAGE [rsBackgroundSync] closeCatalog: closing all databases
2018-08-10T06:06:09.178+0000 I STORAGE [rsBackgroundSync] closeCatalog: closing storage engine catalog
2018-08-10T06:06:09.178+0000 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down
2018-08-10T06:06:09.178+0000 F ROLLBACK [rsBackgroundSync] RecoverToStableTimestamp failed. :: caused by :: UnrecoverableRollbackError: No stable timestamp available to recover to. Initial data timestamp: Timestamp(1533794690, 1), Stable timestamp: Timestamp(0, 0)
2018-08-10T06:06:09.178+0000 I ROLLBACK [rsBackgroundSync] Rollback summary:
2018-08-10T06:06:09.178+0000 I ROLLBACK [rsBackgroundSync] start time: 2018-08-10T06:06:09.160+0000
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] end time: 2018-08-10T06:06:09.170+0000
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] sync source: ip-172-30-0-149:57042
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] rollback data file directory: /data_storages/data/rollback/config.lockpings
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] rollback id: 230
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] last optime on branch of history rolled back: { ts: Timestamp(1533794690, 1), t: 5 }
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] common point optime: { ts: Timestamp(1533794677, 1), t: 5 }
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] last wall clock time on the branch of history rolled back: 2018-08-09T06:04:50.652+0000
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] common point wall clock time: 2018-08-09T06:04:37.479+0000
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] difference in wall clock times: 13 second(s)
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] shard identity document rolled back: false
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] config server config version document rolled back: false
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] affected sessions: none
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] affected namespaces:
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] config.lockpings
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] counts of interesting commands rolled back:
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] update: 1
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] total number of entries rolled back (including no-ops): 2
2018-08-10T06:06:09.179+0000 I ROLLBACK [rsBackgroundSync] transition to SECONDARY
2018-08-10T06:06:09.179+0000 I REPL [rsBackgroundSync] transition to SECONDARY from ROLLBACK
2018-08-10T06:06:09.179+0000 F REPL [rsBackgroundSync] Rollback failed with unrecoverable error: UnrecoverableRollbackError: Recover to stable timestamp failed.
2018-08-10T06:06:09.179+0000 F - [rsBackgroundSync] Fatal assertion 50666 UnrecoverableRollbackError: Recover to stable timestamp failed. at src/mongo/db/repl/bgsync.cpp