-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 4.0.12
-
Component/s: None
-
Labels:None
-
Environment:Centos 7
-
ALL
Mongo 4.0.12 ( ~2.3 TB)
After an unclean shutdown of a replica set member, the replica member tried to do a replication recovery by replaying stored operation from the oplog.
2020-01-23T21:44:10.390-0600 I REPL [initandlisten] Replaying stored operations from { : Timestamp(1578569519, 42) } (exclusive) to { : Timestamp(1579831824, 148) } (inclusive).
I couldn't wait for it to replay 15 days of stored operations because that was the only member in the replica set(the secondary was down for some other problem). I decided to start it as a standalone and drop the local database, but got
:> db.dropDatabase()
{ "ok" : 0, "errmsg" : "not authorized on local to execute command { dropDatabase: 1.0, lsid: { id: UUID(\"60d44cb0-c23e-4c0e-a9b1-c6939e326c40\") }, $readPreference: { mode: \"secondaryPreferred\" }, $db: \"local\" }", "code" : 13, "codeName" : "Unauthorized"}
I was only able to delete the collection oplog.rs, but when the node starts as a replica set, the member is in `STARTUP2` state and tries to perform an initial sync.
2020-01-23T1... E REPL [replication-0] Initial sync attempt failed -- attempts left: 0 cause: InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.... 2020-01-23T... F REPL [replication-0] The maximum number of retries have been exhausted for initial sync. 2020-01-23T... I STORAGE [replication-0] Finishing collection drop for local.temp_oplog_buffer (d663e044-dbec-4167-b95a-c9a703595672). 2020-01-23T22:... E REPL [replication-0] Initial sync failed, shutting down now. Restart the server to attempt a new initial sync. 2020-01-23T22:... F - [replication-0] Fatal assertion 40088 InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync. at src/mongo/db/repl/replication_coordinator_impl.cpp 727 2020-01-23T22:... F - [replication-0] ***aborting after fassert() failure
starting the node with a different rs name doesn't help either
2020-01-23T19:... I REPL [conn117] replSet set names do not match, ours: rs1; remote node's: rs5 ... 2020-01-23T1... E REPL [replication-0] Initial sync attempt failed -- attempts left: 0 cause: InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync. 2020-01-23T22::... F REPL [replication-0] The maximum number of retries have been exhausted for initial sync. 2020-01-23T22:... E REPL [replication-0] Initial sync failed, shutting down now. Restart the server to attempt a new initial sync. 2020-01-23T22:... F - [replication-0] Fatal assertion 40088 InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync. at src/mongo/db/repl/replication_coordinator_impl.cpp 727 2020-01-23T22:... F - [replication-0] ***aborting after fassert() failure
I want to be able to restart the node as a replica set, but it looks like this can't be done without dropping the local database.