Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-30901

RocksDB Initial sync crash due to stopTimestamp < startTimestamp

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Major - P3 Major - P3
    • None
    • None
    • Replication, Storage
    • None
    • Storage Execution
    • ALL
    • 25

    Description

      We saw this failure on evergreen: https://evergreen.mongodb.com/task/mongodb_mongo_master_ubuntu1404_rocksdb_sharding_auth_ea31111dc95eb309269545348c34791b472f6c25_17_08_15_22_36_34/0

      A secondary crashed during intial sync:

      [js_test:sharding_rs1] 2017-08-16T10:25:17.000+0000 d20762| 2017-08-16T10:25:16.999+0000 I REPL     [repl writer worker 5] Finished cloning data: OK. Beginning oplog replay.
      [js_test:sharding_rs1] 2017-08-16T10:25:17.011+0000 d20762| 2017-08-16T10:25:17.010+0000 E REPL     [replication-1] Possible rollback on sync source ip-10-186-5-61:20760. Currently at { : Timestamp 1502879111000|2 }. Started at { : Timestamp 1502879111000|3 }
      [js_test:sharding_rs1] 2017-08-16T10:25:17.011+0000 d20762| 2017-08-16T10:25:17.011+0000 I ASIO     [NetworkInterfaceASIO-RS-0] Ending connection to host ip-10-186-5-61:20760 due to bad connection status; 2 connections to that host remain open
      [js_test:sharding_rs1] 2017-08-16T10:25:17.012+0000 d20762| 2017-08-16T10:25:17.011+0000 I REPL     [replication-0] Finished fetching oplog during initial sync: CallbackCanceled: error in fetcher batch callback: oplog fetcher is shutting down. Last fetched optime and hash: { ts: Tim
      [js_test:sharding_rs1] 2017-08-16T10:25:17.012+0000 d20762| 2017-08-16T10:25:17.011+0000 I REPL     [replication-0] Initial sync attempt finishing up.
      [js_test:sharding_rs1] 2017-08-16T10:25:17.012+0000 d20762| 2017-08-16T10:25:17.011+0000 I REPL     [replication-0] Initial Sync Attempt Statistics: { failedInitialSyncAttempts: 0, maxFailedInitialSyncAttempts: 1, initialSyncStart: new Date(1502879111978), initialSyncAttempts: [], f
      [js_test:sharding_rs1] 2017-08-16T10:25:17.013+0000 d20762| 2017-08-16T10:25:17.011+0000 E REPL     [replication-0] Initial sync attempt failed -- attempts left: 0 cause: OplogOutOfOrder: Possible rollback on sync source ip-10-186-5-61:20760. Currently at { : Timestamp 1502879111000
      [js_test:sharding_rs1] 2017-08-16T10:25:17.013+0000 d20762| 2017-08-16T10:25:17.011+0000 F REPL     [replication-0] The maximum number of retries have been exhausted for initial sync.
      [js_test:sharding_rs1] 2017-08-16T10:25:17.015+0000 d20762| 2017-08-16T10:25:17.013+0000 E REPL     [replication-0] Initial sync failed, shutting down now. Restart the server to attempt a new initial sync.
      [js_test:sharding_rs1] 2017-08-16T10:25:17.015+0000 d20762| 2017-08-16T10:25:17.013+0000 F -        [replication-0] Fatal assertion 40088 OplogOutOfOrder: Possible rollback on sync source ip-10-186-5-61:20760. Currently at { : Timestamp 1502879111000|2 }. Started at { : Timestamp 15
      [js_test:sharding_rs1] 2017-08-16T10:25:17.015+0000 d20762| 2017-08-16T10:25:17.013+0000 F -        [replication-0]
      [js_test:sharding_rs1] 2017-08-16T10:25:17.016+0000 d20762|
      [js_test:sharding_rs1] 2017-08-16T10:25:17.016+0000 d20762| ***aborting after fassert() failure
      [js_test:sharding_rs1] 2017-08-16T10:25:17.016+0000 d20762|
      [js_test:sharding_rs1] 2017-08-16T10:25:17.016+0000 d20762|
      

      Attachments

        Activity

          People

            backlog-server-execution Backlog - Storage Execution Team
            spencer@mongodb.com Spencer Brody (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: