-
Type: Bug
-
Resolution: Won't Fix
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Replication, Storage
-
Labels:None
-
Storage Execution
-
ALL
-
25
We saw this failure on evergreen: https://evergreen.mongodb.com/task/mongodb_mongo_master_ubuntu1404_rocksdb_sharding_auth_ea31111dc95eb309269545348c34791b472f6c25_17_08_15_22_36_34/0
A secondary crashed during intial sync:
[js_test:sharding_rs1] 2017-08-16T10:25:17.000+0000 d20762| 2017-08-16T10:25:16.999+0000 I REPL [repl writer worker 5] Finished cloning data: OK. Beginning oplog replay. [js_test:sharding_rs1] 2017-08-16T10:25:17.011+0000 d20762| 2017-08-16T10:25:17.010+0000 E REPL [replication-1] Possible rollback on sync source ip-10-186-5-61:20760. Currently at { : Timestamp 1502879111000|2 }. Started at { : Timestamp 1502879111000|3 } [js_test:sharding_rs1] 2017-08-16T10:25:17.011+0000 d20762| 2017-08-16T10:25:17.011+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending connection to host ip-10-186-5-61:20760 due to bad connection status; 2 connections to that host remain open [js_test:sharding_rs1] 2017-08-16T10:25:17.012+0000 d20762| 2017-08-16T10:25:17.011+0000 I REPL [replication-0] Finished fetching oplog during initial sync: CallbackCanceled: error in fetcher batch callback: oplog fetcher is shutting down. Last fetched optime and hash: { ts: Tim [js_test:sharding_rs1] 2017-08-16T10:25:17.012+0000 d20762| 2017-08-16T10:25:17.011+0000 I REPL [replication-0] Initial sync attempt finishing up. [js_test:sharding_rs1] 2017-08-16T10:25:17.012+0000 d20762| 2017-08-16T10:25:17.011+0000 I REPL [replication-0] Initial Sync Attempt Statistics: { failedInitialSyncAttempts: 0, maxFailedInitialSyncAttempts: 1, initialSyncStart: new Date(1502879111978), initialSyncAttempts: [], f [js_test:sharding_rs1] 2017-08-16T10:25:17.013+0000 d20762| 2017-08-16T10:25:17.011+0000 E REPL [replication-0] Initial sync attempt failed -- attempts left: 0 cause: OplogOutOfOrder: Possible rollback on sync source ip-10-186-5-61:20760. Currently at { : Timestamp 1502879111000 [js_test:sharding_rs1] 2017-08-16T10:25:17.013+0000 d20762| 2017-08-16T10:25:17.011+0000 F REPL [replication-0] The maximum number of retries have been exhausted for initial sync. [js_test:sharding_rs1] 2017-08-16T10:25:17.015+0000 d20762| 2017-08-16T10:25:17.013+0000 E REPL [replication-0] Initial sync failed, shutting down now. Restart the server to attempt a new initial sync. [js_test:sharding_rs1] 2017-08-16T10:25:17.015+0000 d20762| 2017-08-16T10:25:17.013+0000 F - [replication-0] Fatal assertion 40088 OplogOutOfOrder: Possible rollback on sync source ip-10-186-5-61:20760. Currently at { : Timestamp 1502879111000|2 }. Started at { : Timestamp 15 [js_test:sharding_rs1] 2017-08-16T10:25:17.015+0000 d20762| 2017-08-16T10:25:17.013+0000 F - [replication-0] [js_test:sharding_rs1] 2017-08-16T10:25:17.016+0000 d20762| [js_test:sharding_rs1] 2017-08-16T10:25:17.016+0000 d20762| ***aborting after fassert() failure [js_test:sharding_rs1] 2017-08-16T10:25:17.016+0000 d20762| [js_test:sharding_rs1] 2017-08-16T10:25:17.016+0000 d20762|