Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-38863

secondary init sync error(Restarting oplog query due to error: NetworkInterfaceExceededTimeLimit: error in fetcher batch callback :: caused by :: timed out.)

    • Type: Icon: Question Question
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None

      when add a senondary ,the init sync process can not fished ,secondary mongod log print the followng message :

      2019-01-07T05:45:36.576+0800 I - [repl writer worker 9] libtorrent.fs.chunks collection clone progress: 686378/927304 74% (documents copied)
      2019-01-07T05:47:20.420+0800 I - [repl writer worker 9] libtorrent.fs.chunks collection clone progress: 690780/927304 74% (documents copied)
      2019-01-07T05:49:00.834+0800 I - [repl writer worker 9] libtorrent.fs.chunks collection clone progress: 695150/927304 74% (documents copied)
      2019-01-07T05:49:44.268+0800 I ASIO [ShardRegistry] Dropping all pooled connections to moveShard2_configServer2:27019 due to NetworkInterfaceExceededTimeLimit: Connection pool has been idle for longer than the host timeout
      2019-01-07T05:49:44.278+0800 I ASIO [ShardRegistry] Connecting to moveShard2_configServer2:27019
      2019-01-07T05:50:09.168+0800 I REPL [replication-19] Restarting oplog query due to error: NetworkInterfaceExceededTimeLimit: error in fetcher batch callback :: caused by :: timed out. Last fetched optime (with hash): { ts: Timestamp(1546840107, 74), t: 7 }[8020115046077724190]. Restarts remaining: 10
      2019-01-07T05:50:09.168+0800 I REPL [replication-19] Scheduled new oplog query Fetcher source: moveShard5:27017 database: local query: { find: "oplog.rs", filter: { ts:

      { $gte: Timestamp(1546840107, 74) }

      }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000, batchSize: 13981010, term: 7, readConcern: { afterClusterTime: Timestamp(1546840107, 74) } } query metadata: { $replData: 1, $oplogQueryData: 1, $readPreference:

      { mode: "secondaryPreferred" }

      } active: 1 findNetworkTimeout: 7000ms getMoreNetworkTimeout: 10000ms shutting down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler request: RemoteCommand 95373 – target:moveShard5:27017 db:local cmd:{ find: "oplog.rs", filter: { ts:

      { $gte: Timestamp(1546840107, 74) }

      }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000, batchSize: 13981010, term: 7, readConcern: { afterClusterTime: Timestamp(1546840107, 74) } } active: 1 callbackHandle.valid: 1 callbackHandle.cancelled: 0 attempt: 1 retryPolicy: RetryPolicyImpl maxAttempts: 1 maxTimeMillis: -1ms
      2019-01-07T05:50:09.168+0800 I ASIO [RS] Ending connection to host moveShard5:27017 due to bad connection status; 1 connections to that host remain open
      2019-01-07T05:50:09.168+0800 I ASIO [RS] Connecting to moveShard5:27017
      2019-01-07T05:50:16.168+0800 I REPL [replication-20] Restarting oplog query due to error: NetworkInterfaceExceededTimeLimit: error in fetcher batch callback :: caused by :: timed out. Last fetched optime (with hash): { ts: Timestamp(1546840107, 74), t: 7 }[8020115046077724190]. Restarts remaining: 9
      2019-01-07T05:50:16.168+0800 I ASIO [RS] Ending connection to host moveShard5:27017 due to bad connection status; 1 connections to that host remain open
      2019-01-07T05:50:16.168+0800 I REPL [replication-20] Scheduled new oplog query Fetcher source: moveShard5:27017 database: local query: { find: "oplog.rs", filter: { ts:

      { $gte: Timestamp(1546840107, 74) }

      }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000, batchSize: 13981010, term: 7, readConcern: { afterClusterTime: Timestamp(1546840107, 74) } } query metadata: { $replData: 1, $oplogQueryData: 1, $readPreference:

      { mode: "secondaryPreferred" }

      } active: 1 findNetworkTimeout: 7000ms getMoreNetworkTimeout: 10000ms shutting down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler request: RemoteCommand 95402 – target:moveShard5:27017 db:local cmd:{ find: "oplog.rs", filter: { ts:

      { $gte: Timestamp(1546840107, 74) }

      }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000, batchSize: 13981010, term: 7, readConcern: { afterClusterTime: Timestamp(1546840107, 74) } } active: 1 callbackHandle.valid: 1 callbackHandle.cancelled: 0 attempt: 1 retryPolicy: RetryPolicyImpl maxAttempts: 1 maxTimeMillis: -1ms
      2019-01-07T05:50:16.168+0800 I ASIO [RS] Connecting to moveShard5:27017
      2019-01-07T05:50:23.168+0800 I REPL [replication-19] Restarting oplog query due to error: NetworkInterfaceExceededTimeLimit: error in fetcher batch callback :: caused by :: timed out. Last fetched optime (with hash): { ts: Timestamp(1546840107, 74), t: 7 }[8020115046077724190]. Restarts remaining: 8
      2019-01-07T05:50:23.168+0800 I ASIO [RS] Ending connection to host moveShard5:27017 due to bad connection status; 1 connections to that host remain open
      2019-01-07T05:50:23.168+0800 I REPL [replication-19] Scheduled new oplog query Fetcher source: moveShard5:27017 database: local query: { find: "oplog.rs", filter: { ts:

      { $gte: Timestamp(1546840107, 74) }

      }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000, batchSize: 13981010, term: 7, readConcern: { afterClusterTime: Timestamp(1546840107, 74) } } query metadata: { $replData: 1, $oplogQueryData: 1, $readPreference:

      { mode: "secondaryPreferred" }

      } active: 1 findNetworkTimeout: 7000ms getMoreNetworkTimeout: 10000ms shutting down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler request: RemoteCommand 95432 – target:moveShard5:27017 db:local cmd:{ find: "oplog.rs", filter: { ts:

      { $gte: Timestamp(1546840107, 74) }

      }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000, batchSize: 13981010, term: 7, readConcern: { afterClusterTime: Timestamp(1546840107, 74) } } active: 1 callbackHandle.valid: 1 callbackHandle.cancelled: 0 attempt: 1 retryPolicy: RetryPolicyImpl maxAttempts: 1 maxTimeMillis: -1ms

      details in the attachments 

        1. metrics.2019-01-06T15-23-14Z-00000
          3.97 MB
        2. moveShard5.log
          3.88 MB

            Assignee:
            dmitry.agranat@mongodb.com Dmitry Agranat
            Reporter:
            jidepeng 纪德鹏
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: