• Type: Icon: Question Question
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.0.2
    • Component/s: MMAPv1
    • Labels:
      None

      I'm seeing a lot of these errors in the log for one of the shard:

      2015-05-21T10:07:25.440-0700 W NETWORK [LockPinger] Failed to connect to 172.16.128.61:27017 after 5000 milliseconds, giving up.
      2015-05-21T10:07:25.440-0700 I NETWORK [LockPinger] SyncClusterConnection connect fail to: ir-mngc-l002w.sfoprod.local:27017 errmsg: couldn't connect to server ir-mngc-l002w.sfoprod.local:27017 (172.16.128.61), connection attempt failed
      2015-05-21T10:07:25.440-0700 I NETWORK [LockPinger] SyncClusterConnection connecting to [ir-mngc-l002c.sfoprod.local:27017]
      2015-05-21T10:07:25.443-0700 I NETWORK [LockPinger] SyncClusterConnection connecting to [ir-mngc-l002e.sfoprod.local:27017]
      2015-05-21T10:07:25.480-0700 I NETWORK [LockPinger] trying reconnect to ir-mngc-l002w.sfoprod.local:27017 (172.16.128.61) failed
      2015-05-21T10:07:27.139-0700 I REPL [ReplicationExecutor] syncing from: ir-mngs3-l002e.sfoprod.local:27017
      2015-05-21T10:07:27.208-0700 I REPL [SyncSourceFeedback] replset setting syncSourceFeedback to ir-mngs3-l002e.sfoprod.local:27017
      2015-05-21T10:07:27.309-0700 I REPL [rsBackgroundSync] replSet our last op time fetched: May 21 10:06:50:9f
      2015-05-21T10:07:27.309-0700 I REPL [rsBackgroundSync] replset source's GTE: May 21 10:07:25:1
      2015-05-21T10:07:27.309-0700 I REPL [rsBackgroundSync] beginning rollback
      2015-05-21T10:07:27.309-0700 I REPL [rsBackgroundSync] rollback 0
      2015-05-21T10:07:27.310-0700 I REPL [ReplicationExecutor] transition to ROLLBACK
      2015-05-21T10:07:27.310-0700 I REPL [rsBackgroundSync] rollback 1
      2015-05-21T10:07:27.310-0700 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint
      2015-05-21T10:07:27.379-0700 I REPL [rsBackgroundSync] replSet info rollback our last optime: May 21 10:06:50:9f
      2015-05-21T10:07:27.379-0700 I REPL [rsBackgroundSync] replSet info rollback their last optime: May 21 10:07:25:14
      2015-05-21T10:07:27.379-0700 I REPL [rsBackgroundSync] replSet info rollback diff in end of log times: -35 seconds
      2015-05-21T10:07:27.434-0700 I REPL [rsBackgroundSync] replSet rollback found matching events at May 21 10:06:29:255
      2015-05-21T10:07:27.434-0700 I REPL [rsBackgroundSync] replSet rollback findcommonpoint scanned : 8873
      2015-05-21T10:07:27.434-0700 I REPL [rsBackgroundSync] replSet rollback 3 fixup
      2015-05-21T10:07:28.628-0700 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 172.16.128.36:27017, reason: errno:115 Operation now in progress
      2015-05-21T10:07:28.629-0700 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set s2
      2015-05-21T10:07:29.049-0700 W NETWORK [ReplExecNetThread-130] Failed to connect to 172.16.128.38:27017, reason: errno:115 Operation now in progress
      2015-05-21T10:07:29.049-0700 I REPL [ReplicationExecutor] Error in heartbeat request to ir-mngs3-l002w.sfoprod.local:27017; Location18915 Failed attempt to connect to ir-mngs3-l002w.sfoprod.local:27017; couldn't connect to server ir-mngs3-l002w.sfoprod.local:27017 (172.16.128.38), connection attempt failed
      2015-05-21T10:07:29.672-0700 I NETWORK [initandlisten] connection accepted from 172.17.128.94:44923 #9766 (38 connections now open)
      2015-05-21T10:07:30.484-0700 W NETWORK [LockPinger] Failed to connect to 172.16.128.61:27017 after 5000 milliseconds, giving up.
      2015-05-21T10:07:30.484-0700 I NETWORK [LockPinger] reconnect ir-mngc-l002w.sfoprod.local:27017 (172.16.128.61) failed failed couldn't connect to server ir-mngc-l002w.sfoprod.local:27017 (172.16.128.61), connection attempt failed
      2015-05-21T10:07:30.594-0700 I NETWORK [LockPinger] scoped connection to ir-mngc-l002w.sfoprod.local:27017,ir-mngc-l002c.sfoprod.local:27017,ir-mngc-l002e.sfoprod.local:27017 not being returned to the pool

      We were doing our load test and were also experiencing some network issues.

            Assignee:
            Unassigned Unassigned
            Reporter:
            van.pham@wbgames.com Van Pham
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: