2016-02-03T17:53:02.750+0000 D QUERY [main] Skipping RLP Initialization, BT Root not set. 2016-02-03T17:53:02.757+0000 I CONTROL [initandlisten] MongoDB starting : pid=13731 port=27017 dbpath=/data 64-bit host=ip-10-0-51-223 2016-02-03T17:53:02.757+0000 I CONTROL [initandlisten] db version v3.3.1-17-g3e41f72 2016-02-03T17:53:02.757+0000 I CONTROL [initandlisten] git version: 3e41f72b36656a58f28fea3314e50aa7dcac47c9 2016-02-03T17:53:02.757+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1k-fips 8 Jan 2015 2016-02-03T17:53:02.757+0000 I CONTROL [initandlisten] allocator: tcmalloc 2016-02-03T17:53:02.757+0000 I CONTROL [initandlisten] modules: enterprise 2016-02-03T17:53:02.757+0000 I CONTROL [initandlisten] build environment: 2016-02-03T17:53:02.757+0000 I CONTROL [initandlisten] distmod: rhel62 2016-02-03T17:53:02.757+0000 I CONTROL [initandlisten] distarch: x86_64 2016-02-03T17:53:02.757+0000 I CONTROL [initandlisten] target_arch: x86_64 2016-02-03T17:53:02.757+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, operationProfiling: { slowOpThresholdMs: 5000 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { replSetName: "rsmydomain2" }, storage: { dbPath: "/data", engine: "mmapv1", indexBuildRetry: false }, systemLog: { destination: "file", logAppend: true, logRotate: "rename", path: "/data/mongod.log", verbosity: 2 } } 2016-02-03T17:53:02.757+0000 D NETWORK [initandlisten] fd limit hard:65535 soft:65535 max conn: 52428 2016-02-03T17:53:02.788+0000 I JOURNAL [initandlisten] journal dir=/data/journal 2016-02-03T17:53:02.788+0000 D COMMAND [DataFileSync] BackgroundJob starting: DataFileSync 2016-02-03T17:53:02.788+0000 I JOURNAL [initandlisten] recover : no journal files present, no recovery needed 2016-02-03T17:53:02.788+0000 D STORAGE [initandlisten] flushing directory /data/journal 2016-02-03T17:53:03.202+0000 D STORAGE [initandlisten] flushing directory /data/journal 2016-02-03T17:53:03.239+0000 I JOURNAL [initandlisten] preallocateIsFaster=true 7.5 2016-02-03T17:53:03.239+0000 D STORAGE [initandlisten] flushing directory /data/journal 2016-02-03T17:53:03.305+0000 D STORAGE [initandlisten] flushing directory /data/journal 2016-02-03T17:53:03.343+0000 D STORAGE [initandlisten] flushing directory /data/journal 2016-02-03T17:53:03.345+0000 I JOURNAL [durability] Durability thread started 2016-02-03T17:53:03.346+0000 I JOURNAL [journal writer] Journal writer thread started 2016-02-03T17:53:03.347+0000 D STORAGE [initandlisten] flushing directory /data 2016-02-03T17:53:03.347+0000 I CONTROL [initandlisten] 2016-02-03T17:53:03.347+0000 I CONTROL [initandlisten] ** NOTE: This is a development version (3.3.1-17-g3e41f72) of MongoDB. 2016-02-03T17:53:03.347+0000 I CONTROL [initandlisten] ** Not recommended for production. 2016-02-03T17:53:03.347+0000 I CONTROL [initandlisten] 2016-02-03T17:53:03.347+0000 D COMMAND [SNMPAgent] BackgroundJob starting: SNMPAgent 2016-02-03T17:53:03.347+0000 D NETWORK [SNMPAgent] SNMPAgent not enabled 2016-02-03T17:53:03.347+0000 D STORAGE [initandlisten] enter repairDatabases (to check pdfile version #) 2016-02-03T17:53:03.348+0000 D STORAGE [initandlisten] Recovering database: mydomain 2016-02-03T17:53:03.364+0000 D STORAGE [initandlisten] mydomain.AccountUsage: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.FunnelReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.UserFunnelDurationReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.RetentionFrequencyReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.SessionTouchEvents: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.DeviceCodeInternalReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.HeatmapReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.DebugSymbolsCache: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.OSVersionReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.RetentionReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.ActionReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.RetentionZeroReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.CountryReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.CustomEventPropertyReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.GestureReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.DurationReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.AppScreenReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.ClientHistoryReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.AppReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.PendingSessions: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.system.indexes: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.Clients: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.ConnectivityReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.ResolutionReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.365+0000 D STORAGE [initandlisten] mydomain.UserFunnelStates: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.366+0000 D STORAGE [initandlisten] mydomain.CustomEventReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.366+0000 D STORAGE [initandlisten] mydomain.Sessions: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.366+0000 D STORAGE [initandlisten] mydomain.system.namespaces: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.366+0000 D STORAGE [initandlisten] mydomain.AppVersionReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.366+0000 D STORAGE [initandlisten] mydomain.UserFunnelReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.366+0000 D STORAGE [initandlisten] mydomain.NavigationReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.366+0000 D STORAGE [initandlisten] mydomain.HeatmapRectReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.366+0000 D STORAGE [initandlisten] mydomain.DeviceReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.366+0000 D STORAGE [initandlisten] mydomain.AppUsage: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.366+0000 D STORAGE [initandlisten] mydomain.NavigationPathReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.366+0000 D STORAGE [initandlisten] mydomain.RetentionTimeReports: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.366+0000 D STORAGE [initandlisten] Recovering database: local 2016-02-03T17:53:03.369+0000 D STORAGE [initandlisten] local.startup_log: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.369+0000 D STORAGE [initandlisten] local.system.replset: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.369+0000 D STORAGE [initandlisten] local.system.indexes: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.369+0000 D STORAGE [initandlisten] local.slaves: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.369+0000 D STORAGE [initandlisten] local.me: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.369+0000 D STORAGE [initandlisten] local.temp: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.369+0000 D STORAGE [initandlisten] local.oplog.rs: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.369+0000 D STORAGE [initandlisten] local.system.namespaces: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.369+0000 D STORAGE [initandlisten] local.replset.minvalid: clearing plan cache - collection info cache reset 2016-02-03T17:53:03.369+0000 D STORAGE [initandlisten] done repairDatabases 2016-02-03T17:53:03.369+0000 D QUERY [initandlisten] Running query: query: {} sort: {} projection: {} 2016-02-03T17:53:03.369+0000 D QUERY [initandlisten] Collection admin.system.roles does not exist. Using EOF plan: query: {} sort: {} projection: {} 2016-02-03T17:53:03.369+0000 I COMMAND [initandlisten] query admin.system.roles planSummary: EOF ntoreturn:0 ntoskip:0 keysExamined:0 docsExamined:0 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:0 reslen:20 locks:{ Global: { acquireCount: { r: 4, W: 2 } }, MMAPV1Journal: { acquireCount: { r: 1, w: 3 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { R: 1 } } } 0ms 2016-02-03T17:53:03.370+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.370+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.370+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.370+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.370+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.370+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.370+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.370+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.370+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.370+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.370+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.370+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes 2016-02-03T17:53:03.371+0000 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes 2016-02-03T17:53:03.371+0000 D INDEX [initandlisten] checking complete 2016-02-03T17:53:03.371+0000 I REPL [initandlisten] Did not find local voted for document at startup; NoMatchingDocument: Did not find replica set lastVote document in local.replset.election 2016-02-03T17:53:03.371+0000 D ASIO [NetworkInterfaceASIO-Replication-0] The NetworkInterfaceASIO worker thread is spinning up 2016-02-03T17:53:03.372+0000 D EXECUTOR [replExecDBWorker-0] starting thread in pool replExecDBWorker-Pool 2016-02-03T17:53:03.372+0000 D EXECUTOR [replExecDBWorker-2] starting thread in pool replExecDBWorker-Pool 2016-02-03T17:53:03.372+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/diagnostic.data' 2016-02-03T17:53:03.372+0000 D EXECUTOR [replExecDBWorker-1] starting thread in pool replExecDBWorker-Pool 2016-02-03T17:53:03.372+0000 D NETWORK [ReplicationExecutor] getBoundAddrs(): [ 127.0.0.1] [ 10.0.51.223] 2016-02-03T17:53:03.372+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker 2016-02-03T17:53:03.372+0000 D COMMAND [PeriodicTaskRunner] BackgroundJob starting: PeriodicTaskRunner 2016-02-03T17:53:03.372+0000 D COMMAND [TTLMonitor] BackgroundJob starting: TTLMonitor 2016-02-03T17:53:03.372+0000 I NETWORK [initandlisten] waiting for connections on port 27017 2016-02-03T17:53:03.372+0000 D COMMAND [ClientCursorMonitor] BackgroundJob starting: ClientCursorMonitor 2016-02-03T17:53:03.373+0000 W NETWORK [HostnameCanonicalizationWorker] Failed to obtain address information for hostname ip-10-0-51-223: Name or service not known 2016-02-03T17:53:03.381+0000 D NETWORK [ReplicationExecutor] getAddrsForHost("in.db2m1.roi.test.mydomain.com:27017"): [ 10.0.208.168] 2016-02-03T17:53:03.381+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:53:03.382+0000 D NETWORK [ReplicationExecutor] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:53:03.382+0000 D NETWORK [ReplicationExecutor] getBoundAddrs(): [ 127.0.0.1] [ 10.0.51.223] 2016-02-03T17:53:03.391+0000 D NETWORK [ReplicationExecutor] getAddrsForHost("in.db2arb.roi.test.mydomain.com:27017"): [ 10.0.86.213] 2016-02-03T17:53:03.392+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:53:03.392+0000 D NETWORK [ReplicationExecutor] connected to server in.db2arb.roi.test.mydomain.com:27017 (10.0.86.213) 2016-02-03T17:53:03.393+0000 W REPL [ReplicationExecutor] Locally stored replica set configuration does not have a valid entry for the current node; waiting for reconfig or remote heartbeat; Got "NodeNotFound: No host described in new configuration 197501 for replica set rsmydomain2 maps to this node" while validating { _id: "rsmydomain2", version: 197501, protocolVersion: 1, members: [ { _id: 0, host: "in.db2m1.roi.test.mydomain.com:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "in.db2arb.roi.test.mydomain.com:27017", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 2.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } 2016-02-03T17:53:03.393+0000 D REPL [ReplicationExecutor] Updated term in topology coordinator to 0 due to new config 2016-02-03T17:53:03.393+0000 I REPL [ReplicationExecutor] New replica set config in use: { _id: "rsmydomain2", version: 197501, protocolVersion: 1, members: [ { _id: 0, host: "in.db2m1.roi.test.mydomain.com:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "in.db2arb.roi.test.mydomain.com:27017", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 2.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } 2016-02-03T17:53:03.393+0000 I REPL [ReplicationExecutor] This node is not a member of the config 2016-02-03T17:53:03.393+0000 I REPL [ReplicationExecutor] transition to REMOVED 2016-02-03T17:53:03.393+0000 D REPL [ReplicationExecutor] Current term is now -1 2016-02-03T17:53:03.393+0000 I REPL [ReplicationExecutor] Starting replication applier threads 2016-02-03T17:53:03.393+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] The NetworkInterfaceASIO worker thread is spinning up 2016-02-03T17:53:03.393+0000 D EXECUTOR [rsBackgroundSync-0] starting thread in pool rsBackgroundSync 2016-02-03T17:53:03.393+0000 D REPL [rsBackgroundSync] bgsync fetch queue set to: (term: -1, timestamp: Jan 21 09:03:36:430) -6286199844409637543 2016-02-03T17:53:03.393+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:04.393+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:05.394+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:06.394+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:07.394+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:08.394+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:09.395+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:10.395+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:11.395+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:12.396+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:13.396+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:14.396+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:15.396+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:16.397+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:17.397+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:18.397+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:19.397+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:20.398+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:21.398+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:22.398+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:23.398+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:24.399+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:25.399+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:26.399+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:27.399+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:28.400+0000 D REPL [ReplicationExecutor] Cannot sync from any members because we are not in the replica set config 2016-02-03T17:53:28.769+0000 I NETWORK [initandlisten] connection accepted from 10.0.208.168:58094 #1 (1 connection now open) 2016-02-03T17:53:28.770+0000 D COMMAND [conn1] run command admin.$cmd { _isSelf: 1 } 2016-02-03T17:53:28.770+0000 I COMMAND [conn1] command admin.$cmd command: _isSelf { _isSelf: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:53 locks:{} protocol:op_query 0ms 2016-02-03T17:53:28.770+0000 D NETWORK [conn1] SocketException: remote: 10.0.208.168:58094 error: 9001 socket exception [CLOSED] server [10.0.208.168:58094] 2016-02-03T17:53:28.770+0000 I NETWORK [conn1] end connection 10.0.208.168:58094 (0 connections now open) 2016-02-03T17:53:28.771+0000 I NETWORK [initandlisten] connection accepted from 10.0.208.168:58095 #2 (1 connection now open) 2016-02-03T17:53:28.771+0000 D COMMAND [conn2] run command admin.$cmd { isMaster: 1 } 2016-02-03T17:53:28.771+0000 I COMMAND [conn2] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:402 locks:{} protocol:op_query 0ms 2016-02-03T17:53:28.771+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:28.771+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:28.772+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:28.772Z 2016-02-03T17:53:28.772+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 1 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:38.772+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197501, from: "", fromId: -1, term: 0 } 2016-02-03T17:53:28.772+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Connecting to in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:28.772+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:288 locks:{} protocol:op_command 0ms 2016-02-03T17:53:28.772+0000 I NETWORK [initandlisten] connection accepted from 10.0.208.168:58096 #3 (2 connections now open) 2016-02-03T17:53:28.772+0000 D COMMAND [conn3] run command admin.$cmd { isMaster: 1 } 2016-02-03T17:53:28.772+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:28.772+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:28.772+0000 I COMMAND [conn3] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:402 locks:{} protocol:op_query 0ms 2016-02-03T17:53:28.772+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:288 locks:{} protocol:op_command 0ms 2016-02-03T17:53:28.773+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 2 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:28.773+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:28.773+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:28.773+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:288 locks:{} protocol:op_command 0ms 2016-02-03T17:53:28.773+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:28.773+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 1 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:28.773+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 1 out: Operation aborted. 2016-02-03T17:53:28.774+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:33.773Z 2016-02-03T17:53:28.774+0000 D REPL [ReplicationExecutor] Received new config via heartbeat with version 197502 2016-02-03T17:53:28.774+0000 D NETWORK [replExecDBWorker-0] getBoundAddrs(): [ 127.0.0.1] [ 10.0.51.223] 2016-02-03T17:53:28.774+0000 D NETWORK [replExecDBWorker-0] getAddrsForHost("in.db2m1.roi.test.mydomain.com:27017"): [ 10.0.208.168] 2016-02-03T17:53:28.774+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:53:28.775+0000 D NETWORK [replExecDBWorker-0] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:53:28.775+0000 D NETWORK [replExecDBWorker-0] getBoundAddrs(): [ 127.0.0.1] [ 10.0.51.223] 2016-02-03T17:53:28.775+0000 D NETWORK [replExecDBWorker-0] getAddrsForHost("in.db2arb.roi.test.mydomain.com:27017"): [ 10.0.86.213] 2016-02-03T17:53:28.775+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:53:28.776+0000 D NETWORK [replExecDBWorker-0] connected to server in.db2arb.roi.test.mydomain.com:27017 (10.0.86.213) 2016-02-03T17:53:28.776+0000 D NETWORK [replExecDBWorker-0] getBoundAddrs(): [ 127.0.0.1] [ 10.0.51.223] 2016-02-03T17:53:28.785+0000 D NETWORK [replExecDBWorker-0] getAddrsForHost("in.db2m2.roi.test.mydomain.com:27017"): [ 10.0.51.223] 2016-02-03T17:53:28.785+0000 D QUERY [replExecDBWorker-0] Only one plan is available; it will be run but will not be cached. query: {} sort: {} projection: {}, planSummary: COLLSCAN 2016-02-03T17:53:28.786+0000 I REPL [ReplicationExecutor] New replica set config in use: { _id: "rsmydomain2", version: 197502, protocolVersion: 1, members: [ { _id: 0, host: "in.db2m1.roi.test.mydomain.com:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "in.db2arb.roi.test.mydomain.com:27017", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 2.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 3, host: "in.db2m2.roi.test.mydomain.com:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } 2016-02-03T17:53:28.786+0000 I REPL [ReplicationExecutor] This node is in.db2m2.roi.test.mydomain.com:27017 in the config 2016-02-03T17:53:28.786+0000 I REPL [ReplicationExecutor] transition to STARTUP2 2016-02-03T17:53:28.786+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:28.786Z 2016-02-03T17:53:28.786+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:28.786Z 2016-02-03T17:53:28.786+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 4 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:38.786+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:28.786+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 4 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:28.786+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 5 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:38.786+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:28.786+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Connecting to in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:28.786+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 4 out: Operation aborted. 2016-02-03T17:53:28.786+0000 I REPL [ReplicationExecutor] Member in.db2m1.roi.test.mydomain.com:27017 is now in state PRIMARY 2016-02-03T17:53:28.787+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:33.786Z 2016-02-03T17:53:28.787+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 6 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:28.787+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:28.787+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 5 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:28.788+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 5 out: Operation aborted. 2016-02-03T17:53:28.788+0000 I REPL [ReplicationExecutor] Member in.db2arb.roi.test.mydomain.com:27017 is now in state ARBITER 2016-02-03T17:53:28.788+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:33.788Z 2016-02-03T17:53:28.797+0000 I NETWORK [initandlisten] connection accepted from 10.0.86.213:35281 #4 (3 connections now open) 2016-02-03T17:53:28.797+0000 D COMMAND [conn4] run command admin.$cmd { isMaster: 1 } 2016-02-03T17:53:28.797+0000 I COMMAND [conn4] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:485 locks:{} protocol:op_query 0ms 2016-02-03T17:53:28.797+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197501, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:53:28.797+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:53:28.798+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197501, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:1072 locks:{} protocol:op_command 0ms 2016-02-03T17:53:28.817+0000 I NETWORK [initandlisten] connection accepted from 10.0.86.213:35282 #5 (4 connections now open) 2016-02-03T17:53:28.817+0000 D COMMAND [conn5] run command admin.$cmd { _isSelf: 1 } 2016-02-03T17:53:28.817+0000 I COMMAND [conn5] command admin.$cmd command: _isSelf { _isSelf: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:53 locks:{} protocol:op_query 0ms 2016-02-03T17:53:28.817+0000 D NETWORK [conn5] SocketException: remote: 10.0.86.213:35282 error: 9001 socket exception [CLOSED] server [10.0.86.213:35282] 2016-02-03T17:53:28.817+0000 I NETWORK [conn5] end connection 10.0.86.213:35282 (3 connections now open) 2016-02-03T17:53:28.818+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:53:28.818+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:53:28.818+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:320 locks:{} protocol:op_command 0ms 2016-02-03T17:53:30.773+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:30.773+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:30.773+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:320 locks:{} protocol:op_command 0ms 2016-02-03T17:53:32.773+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:32.773+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:32.773+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:320 locks:{} protocol:op_command 0ms 2016-02-03T17:53:33.178+0000 I NETWORK [initandlisten] connection accepted from 10.0.97.218:60826 #6 (4 connections now open) 2016-02-03T17:53:33.178+0000 D COMMAND [conn6] run command admin.$cmd { isMaster: 1 } 2016-02-03T17:53:33.178+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:485 locks:{} protocol:op_query 0ms 2016-02-03T17:53:33.178+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:33.178+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:33.179+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:33.179+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:33.394+0000 I REPL [ReplicationExecutor] transition to RECOVERING 2016-02-03T17:53:33.394+0000 D EXECUTOR [repl writer worker 0] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.394+0000 D EXECUTOR [repl writer worker 1] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.394+0000 D EXECUTOR [repl writer worker 2] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.394+0000 D EXECUTOR [repl writer worker 3] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.394+0000 D EXECUTOR [repl writer worker 4] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.394+0000 D EXECUTOR [repl writer worker 6] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.394+0000 D EXECUTOR [repl writer worker 8] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.394+0000 D EXECUTOR [repl writer worker 5] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.394+0000 D EXECUTOR [repl writer worker 7] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl writer worker 12] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl writer worker 11] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl writer worker 10] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl writer worker 9] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl writer worker 14] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl writer worker 15] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl prefetch worker 0] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl prefetch worker 2] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl prefetch worker 3] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl prefetch worker 4] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl prefetch worker 1] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl writer worker 13] starting thread in pool repl writer worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl prefetch worker 5] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl prefetch worker 10] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl prefetch worker 8] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl prefetch worker 9] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl prefetch worker 7] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.395+0000 D EXECUTOR [repl prefetch worker 11] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.396+0000 D EXECUTOR [repl prefetch worker 6] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.396+0000 D EXECUTOR [repl prefetch worker 12] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.396+0000 D EXECUTOR [repl prefetch worker 13] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.396+0000 D EXECUTOR [repl prefetch worker 14] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.396+0000 D EXECUTOR [repl prefetch worker 15] starting thread in pool repl prefetch worker Pool 2016-02-03T17:53:33.396+0000 I REPL [ReplicationExecutor] transition to SECONDARY 2016-02-03T17:53:33.786+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 9 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:43.786+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:33.786+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 9 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:33.787+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 9 out: Operation aborted. 2016-02-03T17:53:33.787+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:38.787Z 2016-02-03T17:53:33.788+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 11 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:43.788+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:33.788+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 11 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:33.789+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 11 out: Operation aborted. 2016-02-03T17:53:33.789+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:38.789Z 2016-02-03T17:53:33.819+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:53:33.819+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:53:33.819+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:320 locks:{} protocol:op_command 0ms 2016-02-03T17:53:34.401+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:34.402+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:53:34.402+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:53:34.403+0000 D REPL [SyncSourceFeedback] resetting connection in sync source feedback 2016-02-03T17:53:34.403+0000 I REPL [SyncSourceFeedback] setting syncSourceFeedback to in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:34.403+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:53:34.403+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 13 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:53:44.403+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:53:34.403+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Connecting to in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:34.403+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:53:34.404+0000 D NETWORK [SyncSourceFeedback] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:53:34.404+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 14 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:34.404+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:53:34.404+0000 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:34.404+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 13 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:34.460+0000 I NETWORK [initandlisten] connection accepted from 10.0.208.168:58099 #7 (5 connections now open) 2016-02-03T17:53:34.461+0000 D COMMAND [conn7] run command admin.$cmd { isMaster: 1 } 2016-02-03T17:53:34.461+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:485 locks:{} protocol:op_query 0ms 2016-02-03T17:53:34.461+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:34.461+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:34.461+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:34.461+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:34.775+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:34.775+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:34.775+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:36.776+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:36.776+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:36.776+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:38.630+0000 I NETWORK [initandlisten] connection accepted from 10.0.43.87:34041 #8 (6 connections now open) 2016-02-03T17:53:38.630+0000 D COMMAND [conn8] run command admin.$cmd { isMaster: 1 } 2016-02-03T17:53:38.630+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:485 locks:{} protocol:op_query 0ms 2016-02-03T17:53:38.630+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:38.630+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:38.631+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:38.631+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:38.690+0000 I NETWORK [initandlisten] connection accepted from 10.0.138.113:40819 #9 (7 connections now open) 2016-02-03T17:53:38.690+0000 D COMMAND [conn9] run command admin.$cmd { isMaster: 1 } 2016-02-03T17:53:38.691+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:485 locks:{} protocol:op_query 0ms 2016-02-03T17:53:38.691+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:38.691+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:38.691+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:38.691+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:38.776+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:38.776+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:38.777+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:38.787+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 15 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:48.787+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:38.787+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 15 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:38.787+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 15 out: Operation aborted. 2016-02-03T17:53:38.787+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:40.787Z 2016-02-03T17:53:38.790+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 17 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:48.790+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:38.790+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 17 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:38.790+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 17 out: Operation aborted. 2016-02-03T17:53:38.790+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:40.790Z 2016-02-03T17:53:38.820+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:53:38.820+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:53:38.820+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:39.405+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:53:39.469+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 13 out: Operation aborted. 2016-02-03T17:53:39.469+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:53:39.469+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:39.469+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:39.469+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:53:39.470+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:53:39.471+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:53:39.471+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:53:39.471+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 20 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:53:49.471+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:53:39.471+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 20 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:40.778+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:40.778+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:40.778+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:40.787+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 21 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:50.787+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:40.787+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 21 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:40.788+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 21 out: Operation aborted. 2016-02-03T17:53:40.788+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:42.788Z 2016-02-03T17:53:40.790+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 23 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:50.790+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:40.790+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 23 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:40.791+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 23 out: Operation aborted. 2016-02-03T17:53:40.791+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:42.791Z 2016-02-03T17:53:42.778+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:42.778+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:42.779+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:42.789+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 25 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:52.789+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:42.789+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 25 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:42.789+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 25 out: Operation aborted. 2016-02-03T17:53:42.789+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:44.789Z 2016-02-03T17:53:42.792+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 27 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:52.792+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:42.792+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 27 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:42.792+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 27 out: Operation aborted. 2016-02-03T17:53:42.792+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:44.792Z 2016-02-03T17:53:43.181+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:43.181+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:43.821+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:53:43.821+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:53:43.821+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:44.464+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:44.464+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:44.472+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:53:44.537+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:53:44.537+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 20 out: Operation aborted. 2016-02-03T17:53:44.537+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:44.537+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:44.537+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:53:44.538+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:53:44.539+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:53:44.539+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:53:44.539+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 30 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:53:54.539+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:53:44.539+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 30 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:44.779+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:44.780+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:44.780+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:44.789+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 31 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:54.789+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:44.789+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 31 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:44.789+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 31 out: Operation aborted. 2016-02-03T17:53:44.790+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:46.789Z 2016-02-03T17:53:44.792+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 33 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:54.792+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:44.792+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 33 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:44.792+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 33 out: Operation aborted. 2016-02-03T17:53:44.792+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:46.792Z 2016-02-03T17:53:46.781+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:46.781+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:46.781+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:46.789+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 35 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:56.789+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:46.789+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 35 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:46.790+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 35 out: Operation aborted. 2016-02-03T17:53:46.790+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:48.790Z 2016-02-03T17:53:46.792+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 37 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:56.792+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:46.792+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 37 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:46.793+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 37 out: Operation aborted. 2016-02-03T17:53:46.793+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:48.793Z 2016-02-03T17:53:48.633+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:48.634+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:48.694+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:48.694+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:48.781+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:48.781+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:48.781+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:48.790+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 39 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:58.790+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:48.790+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 39 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:48.791+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 39 out: Operation aborted. 2016-02-03T17:53:48.791+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:50.791Z 2016-02-03T17:53:48.793+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 41 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:53:58.793+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:48.793+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 41 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:48.794+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 41 out: Operation aborted. 2016-02-03T17:53:48.794+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:50.794Z 2016-02-03T17:53:48.822+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:53:48.822+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:53:48.822+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:49.539+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:53:49.604+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 30 out: Operation aborted. 2016-02-03T17:53:49.604+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:53:49.604+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:49.605+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:49.605+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:53:49.605+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:53:49.606+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:53:49.606+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:53:49.606+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 44 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:53:59.606+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:53:49.606+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 44 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:50.783+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:50.783+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:50.783+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:50.791+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 45 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:00.791+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:50.791+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 45 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:50.792+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 45 out: Operation aborted. 2016-02-03T17:53:50.792+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:52.792Z 2016-02-03T17:53:50.794+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 47 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:00.794+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:50.794+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 47 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:50.795+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 47 out: Operation aborted. 2016-02-03T17:53:50.795+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:52.795Z 2016-02-03T17:53:52.784+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:52.784+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:52.784+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:52.792+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 49 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:02.792+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:52.792+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 49 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:52.793+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 49 out: Operation aborted. 2016-02-03T17:53:52.793+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:54.793Z 2016-02-03T17:53:52.795+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 51 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:02.795+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:52.795+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 51 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:52.796+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 51 out: Operation aborted. 2016-02-03T17:53:52.796+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:54.796Z 2016-02-03T17:53:53.184+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:53.184+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:53.823+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:53:53.823+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:53:53.824+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:54.467+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:54.467+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:54.607+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:53:54.673+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 44 out: Operation aborted. 2016-02-03T17:53:54.673+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:53:54.673+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:54.673+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:54.673+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:53:54.674+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:53:54.675+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:53:54.675+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:53:54.675+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 54 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:54:04.675+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:53:54.675+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 54 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:54.785+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:54.786+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:54.786+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:54.793+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 55 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:04.793+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:54.793+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 55 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:54.793+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 55 out: Operation aborted. 2016-02-03T17:53:54.793+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:56.793Z 2016-02-03T17:53:54.797+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 57 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:04.797+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:54.797+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 57 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:54.797+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 57 out: Operation aborted. 2016-02-03T17:53:54.797+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:56.797Z 2016-02-03T17:53:56.786+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:56.786+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:56.786+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:56.793+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 59 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:06.793+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:56.793+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 59 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:56.794+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 59 out: Operation aborted. 2016-02-03T17:53:56.794+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:53:58.794Z 2016-02-03T17:53:56.797+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 61 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:06.797+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:56.797+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 61 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:56.798+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 61 out: Operation aborted. 2016-02-03T17:53:56.798+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:53:58.798Z 2016-02-03T17:53:58.636+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:58.636+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:58.697+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:53:58.697+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:53:58.787+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:53:58.787+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:53:58.787+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:58.794+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 63 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:08.794+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:58.794+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 63 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:58.795+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 63 out: Operation aborted. 2016-02-03T17:53:58.795+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:00.795Z 2016-02-03T17:53:58.798+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 65 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:08.798+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:53:58.798+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 65 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:53:58.799+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 65 out: Operation aborted. 2016-02-03T17:53:58.799+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:00.799Z 2016-02-03T17:53:58.824+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:53:58.824+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:53:58.824+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:53:59.675+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:53:59.738+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 54 out: Operation aborted. 2016-02-03T17:53:59.738+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:53:59.738+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:59.738+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:53:59.739+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:53:59.739+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:53:59.740+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:53:59.740+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:53:59.740+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 68 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:54:09.740+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:53:59.740+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 68 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:00.788+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:00.788+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:00.788+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:00.795+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 69 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:10.795+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:00.795+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 69 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:00.796+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 69 out: Operation aborted. 2016-02-03T17:54:00.796+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:02.796Z 2016-02-03T17:54:00.799+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 71 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:10.799+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:00.799+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 71 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:00.800+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 71 out: Operation aborted. 2016-02-03T17:54:00.800+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:02.800Z 2016-02-03T17:54:02.789+0000 I STORAGE [DataFileSync] flushing mmaps took 1ms for 782 files 2016-02-03T17:54:02.789+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:02.789+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:02.789+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:02.796+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 73 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:12.796+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:02.796+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 73 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:02.797+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 73 out: Operation aborted. 2016-02-03T17:54:02.797+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:04.797Z 2016-02-03T17:54:02.800+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 75 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:12.800+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:02.800+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 75 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:02.801+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 75 out: Operation aborted. 2016-02-03T17:54:02.801+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:04.801Z 2016-02-03T17:54:03.186+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:03.186+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:03.372+0000 D - [PeriodicTaskRunner] cleaning up unused lock buckets of the global lock manager 2016-02-03T17:54:03.373+0000 D INDEX [TTLMonitor] TTL -- ns: mydomain.PendingSessions key: { st: 1.0 } 2016-02-03T17:54:03.826+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:54:03.826+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:54:03.827+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:04.469+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:04.469+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:04.740+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:04.789+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:04.789+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:04.789+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:04.798+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 77 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:14.798+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:04.798+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 77 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:04.798+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 77 out: Operation aborted. 2016-02-03T17:54:04.798+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:06.798Z 2016-02-03T17:54:04.801+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 79 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:14.801+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:04.802+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 79 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:04.802+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 79 out: Operation aborted. 2016-02-03T17:54:04.802+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:06.802Z 2016-02-03T17:54:04.804+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 68 out: Operation aborted. 2016-02-03T17:54:04.804+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:54:04.804+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:04.804+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:04.812+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:54:04.812+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:54:04.813+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:04.813+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:54:04.813+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 82 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:54:14.813+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:54:04.813+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 82 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:06.791+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:06.791+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:06.791+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:06.798+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 83 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:16.798+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:06.798+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 83 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:06.799+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 83 out: Operation aborted. 2016-02-03T17:54:06.799+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:08.799Z 2016-02-03T17:54:06.802+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 85 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:16.802+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:06.802+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 85 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:06.803+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 85 out: Operation aborted. 2016-02-03T17:54:06.803+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:08.803Z 2016-02-03T17:54:08.639+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:08.639+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:08.699+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:08.699+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:08.791+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:08.791+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:08.792+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:08.799+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 87 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:18.799+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:08.799+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 87 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:08.799+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 87 out: Operation aborted. 2016-02-03T17:54:08.799+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:10.799Z 2016-02-03T17:54:08.804+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 89 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:18.804+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:08.804+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 89 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:08.804+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 89 out: Operation aborted. 2016-02-03T17:54:08.804+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:10.804Z 2016-02-03T17:54:08.827+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:54:08.828+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:54:08.828+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:09.814+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:09.878+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 82 out: Operation aborted. 2016-02-03T17:54:09.878+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:54:09.878+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:09.878+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:09.879+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:54:09.879+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:54:09.880+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:09.880+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:54:09.880+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 92 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:54:19.880+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:54:09.880+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 92 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:10.793+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:10.793+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:10.793+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:10.799+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 93 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:20.799+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:10.799+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 93 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:10.800+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 93 out: Operation aborted. 2016-02-03T17:54:10.800+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:12.800Z 2016-02-03T17:54:10.804+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 95 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:20.804+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:10.804+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 95 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:10.805+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 95 out: Operation aborted. 2016-02-03T17:54:10.805+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:12.805Z 2016-02-03T17:54:12.794+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:12.794+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:12.794+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:12.800+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 97 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:22.800+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:12.800+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 97 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:12.801+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 97 out: Operation aborted. 2016-02-03T17:54:12.801+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:14.801Z 2016-02-03T17:54:12.805+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 99 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:22.805+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:12.805+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 99 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:12.806+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 99 out: Operation aborted. 2016-02-03T17:54:12.806+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:14.806Z 2016-02-03T17:54:13.189+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:13.189+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:13.828+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:54:13.828+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:54:13.828+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:14.472+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:14.472+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:14.795+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:14.795+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:14.795+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:14.801+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 101 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:24.801+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:14.801+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 101 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:14.802+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 101 out: Operation aborted. 2016-02-03T17:54:14.802+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:16.802Z 2016-02-03T17:54:14.806+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 103 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:24.806+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:14.806+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 103 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:14.807+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 103 out: Operation aborted. 2016-02-03T17:54:14.807+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:16.807Z 2016-02-03T17:54:14.881+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:14.945+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 92 out: Operation aborted. 2016-02-03T17:54:14.945+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:54:14.945+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:14.946+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:14.946+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:54:14.946+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:54:14.947+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:54:14.947+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:14.947+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 106 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:54:24.947+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:54:14.947+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 106 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:16.796+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:16.796+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:16.796+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:16.802+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 107 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:26.802+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:16.802+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 107 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:16.803+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 107 out: Operation aborted. 2016-02-03T17:54:16.803+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:18.803Z 2016-02-03T17:54:16.807+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 109 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:26.807+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:16.807+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 109 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:16.808+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 109 out: Operation aborted. 2016-02-03T17:54:16.808+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:18.808Z 2016-02-03T17:54:18.641+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:18.642+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:18.702+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:18.702+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:18.796+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:18.796+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:18.796+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:18.804+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 111 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:28.803+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:18.804+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 111 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:18.804+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 111 out: Operation aborted. 2016-02-03T17:54:18.804+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:20.804Z 2016-02-03T17:54:18.808+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 113 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:28.808+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:18.808+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 113 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:18.809+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 113 out: Operation aborted. 2016-02-03T17:54:18.809+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:20.809Z 2016-02-03T17:54:18.830+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:54:18.830+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:54:18.830+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:19.948+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:20.015+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:54:20.015+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 106 out: Operation aborted. 2016-02-03T17:54:20.015+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:20.015+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:20.015+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:54:20.016+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:54:20.017+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:20.017+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:54:20.017+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 116 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:54:30.017+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:54:20.017+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 116 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:20.798+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:20.798+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:20.798+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:20.804+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 117 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:30.804+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:20.804+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 117 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:20.805+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 117 out: Operation aborted. 2016-02-03T17:54:20.805+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:22.805Z 2016-02-03T17:54:20.809+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 119 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:30.809+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:20.809+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 119 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:20.809+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 119 out: Operation aborted. 2016-02-03T17:54:20.809+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:22.809Z 2016-02-03T17:54:22.799+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:22.799+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:22.799+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:22.805+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 121 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:32.805+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:22.805+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 121 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:22.806+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 121 out: Operation aborted. 2016-02-03T17:54:22.806+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:24.806Z 2016-02-03T17:54:22.809+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 123 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:32.809+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:22.809+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 123 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:22.810+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 123 out: Operation aborted. 2016-02-03T17:54:22.810+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:24.810Z 2016-02-03T17:54:23.191+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:23.192+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:23.830+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:54:23.830+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:54:23.830+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:24.474+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:24.475+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:24.800+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:24.800+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:24.800+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:24.806+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 125 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:34.806+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:24.806+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 125 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:24.807+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 125 out: Operation aborted. 2016-02-03T17:54:24.807+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:26.807Z 2016-02-03T17:54:24.810+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 127 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:34.810+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:24.810+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 127 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:24.810+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 127 out: Operation aborted. 2016-02-03T17:54:24.811+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:26.810Z 2016-02-03T17:54:25.017+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:25.083+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:54:25.083+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 116 out: Operation aborted. 2016-02-03T17:54:25.083+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:25.083+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:25.084+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:54:25.084+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:54:25.085+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:25.085+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:54:25.085+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 130 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:54:35.085+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:54:25.085+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 130 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:26.801+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:26.801+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:54:26.801+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:26.807+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 131 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:36.807+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:26.807+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 131 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:26.808+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 131 out: Operation aborted. 2016-02-03T17:54:26.808+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:28.808Z 2016-02-03T17:54:26.810+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 133 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:36.810+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:26.810+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 133 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:26.811+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 133 out: Operation aborted. 2016-02-03T17:54:26.811+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:28.811Z 2016-02-03T17:54:28.644+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:28.644+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:28.705+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:28.705+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:28.802+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:28.802+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:28.802+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:28.808+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 135 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:38.808+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:28.808+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 135 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:28.809+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 135 out: Operation aborted. 2016-02-03T17:54:28.809+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:30.809Z 2016-02-03T17:54:28.811+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 137 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:38.811+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:28.811+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 137 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:28.812+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 137 out: Operation aborted. 2016-02-03T17:54:28.812+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:30.812Z 2016-02-03T17:54:28.832+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:54:28.832+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:54:28.832+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:30.085+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:30.153+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 130 out: Operation aborted. 2016-02-03T17:54:30.153+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:54:30.153+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:30.153+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:30.153+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:54:30.154+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:54:30.155+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:30.155+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:54:30.155+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 140 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:54:40.155+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:54:30.155+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 140 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:30.803+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:30.803+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:30.803+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:30.809+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 141 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:40.809+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:30.809+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 141 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:30.810+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 141 out: Operation aborted. 2016-02-03T17:54:30.810+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:32.810Z 2016-02-03T17:54:30.812+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 143 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:40.812+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:30.812+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 143 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:30.813+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 143 out: Operation aborted. 2016-02-03T17:54:30.813+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:32.813Z 2016-02-03T17:54:32.804+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:32.804+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:32.804+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:32.810+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 145 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:42.810+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:32.810+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 145 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:32.811+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 145 out: Operation aborted. 2016-02-03T17:54:32.811+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:34.811Z 2016-02-03T17:54:32.813+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 147 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:42.813+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:32.813+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 147 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:32.814+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 147 out: Operation aborted. 2016-02-03T17:54:32.814+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:34.814Z 2016-02-03T17:54:33.194+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:33.194+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:33.832+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:54:33.832+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:54:33.832+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:34.477+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:34.477+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:34.804+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:34.805+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:34.805+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:34.811+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 149 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:44.811+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:34.811+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 149 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:34.811+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 149 out: Operation aborted. 2016-02-03T17:54:34.812+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:36.811Z 2016-02-03T17:54:34.814+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 151 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:44.814+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:34.814+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 151 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:34.814+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 151 out: Operation aborted. 2016-02-03T17:54:34.814+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:36.814Z 2016-02-03T17:54:35.155+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:35.222+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 140 out: Operation aborted. 2016-02-03T17:54:35.222+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:54:35.222+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:35.222+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:35.223+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:54:35.223+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:54:35.224+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:54:35.224+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:35.224+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 154 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:54:45.224+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:54:35.224+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 154 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:36.806+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:36.806+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:36.806+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:36.811+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 155 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:46.811+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:36.811+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 155 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:36.812+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 155 out: Operation aborted. 2016-02-03T17:54:36.812+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:38.812Z 2016-02-03T17:54:36.814+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 157 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:46.814+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:36.814+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 157 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:36.815+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 157 out: Operation aborted. 2016-02-03T17:54:36.815+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:38.815Z 2016-02-03T17:54:38.647+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:38.647+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:38.708+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:38.708+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:38.807+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:38.807+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:38.807+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:38.812+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 159 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:48.812+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:38.812+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 159 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:38.813+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 159 out: Operation aborted. 2016-02-03T17:54:38.813+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:40.813Z 2016-02-03T17:54:38.815+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 161 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:48.815+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:38.815+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 161 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:38.815+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 161 out: Operation aborted. 2016-02-03T17:54:38.815+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:40.815Z 2016-02-03T17:54:38.834+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:54:38.834+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:54:38.834+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:40.224+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:40.292+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:54:40.292+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 154 out: Operation aborted. 2016-02-03T17:54:40.292+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:40.292+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:40.293+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:54:40.293+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:54:40.294+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:40.294+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:54:40.294+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 164 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:54:50.294+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:54:40.294+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 164 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:40.808+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:40.808+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:40.808+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:40.813+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 165 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:50.813+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:40.813+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 165 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:40.814+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 165 out: Operation aborted. 2016-02-03T17:54:40.814+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:42.814Z 2016-02-03T17:54:40.815+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 167 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:50.815+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:40.815+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 167 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:40.816+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 167 out: Operation aborted. 2016-02-03T17:54:40.816+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:42.816Z 2016-02-03T17:54:42.809+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:42.809+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:42.809+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:42.814+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 169 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:52.814+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:42.814+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 169 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:42.815+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 169 out: Operation aborted. 2016-02-03T17:54:42.815+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:44.815Z 2016-02-03T17:54:42.816+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 171 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:52.816+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:42.816+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 171 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:42.816+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 171 out: Operation aborted. 2016-02-03T17:54:42.817+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:44.817Z 2016-02-03T17:54:43.197+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:43.197+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:43.834+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:54:43.834+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:54:43.834+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:44.480+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:44.480+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:44.809+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:44.809+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:44.810+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:44.815+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 173 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:54.815+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:44.815+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 173 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:44.815+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 173 out: Operation aborted. 2016-02-03T17:54:44.815+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:46.815Z 2016-02-03T17:54:44.818+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 175 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:54.818+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:44.818+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 175 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:44.818+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 175 out: Operation aborted. 2016-02-03T17:54:44.818+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:46.818Z 2016-02-03T17:54:45.294+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:45.359+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:54:45.359+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 164 out: Operation aborted. 2016-02-03T17:54:45.359+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:45.359+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:45.359+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:54:45.360+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:54:45.361+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:45.361+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:54:45.361+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 178 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:54:55.361+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:54:45.361+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 178 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:46.811+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:46.811+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:46.811+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:46.815+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 179 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:56.815+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:46.815+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 179 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:46.816+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 179 out: Operation aborted. 2016-02-03T17:54:46.816+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:48.816Z 2016-02-03T17:54:46.818+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 181 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:56.818+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:46.818+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 181 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:46.818+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 181 out: Operation aborted. 2016-02-03T17:54:46.818+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:48.818Z 2016-02-03T17:54:48.650+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:48.650+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:48.710+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:48.710+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:48.811+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:48.811+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:48.812+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:48.817+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 183 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:58.817+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:48.817+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 183 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:48.817+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 183 out: Operation aborted. 2016-02-03T17:54:48.817+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:50.817Z 2016-02-03T17:54:48.818+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 185 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:54:58.818+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:48.819+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 185 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:48.819+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 185 out: Operation aborted. 2016-02-03T17:54:48.819+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:50.819Z 2016-02-03T17:54:48.835+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:54:48.835+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:54:48.835+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:50.361+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:50.426+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 178 out: Operation aborted. 2016-02-03T17:54:50.426+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:54:50.426+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:50.426+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:50.426+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:54:50.426+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:54:50.427+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:50.427+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:54:50.427+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 188 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:55:00.427+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:54:50.427+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 188 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:50.812+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:50.812+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:50.812+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:50.818+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 189 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:00.818+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:50.818+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 189 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:50.818+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 189 out: Operation aborted. 2016-02-03T17:54:50.818+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:52.818Z 2016-02-03T17:54:50.819+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 191 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:00.819+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:50.819+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 191 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:50.820+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 191 out: Operation aborted. 2016-02-03T17:54:50.820+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:52.820Z 2016-02-03T17:54:52.814+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:52.814+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:52.814+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:52.818+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 193 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:02.818+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:52.818+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 193 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:52.819+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 193 out: Operation aborted. 2016-02-03T17:54:52.819+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:54.819Z 2016-02-03T17:54:52.820+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 195 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:02.820+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:52.820+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 195 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:52.820+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 195 out: Operation aborted. 2016-02-03T17:54:52.820+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:54.820Z 2016-02-03T17:54:53.199+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:53.199+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:53.836+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:54:53.836+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:54:53.836+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:54.483+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:54.483+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:54.814+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:54.814+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:54.814+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:54.819+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 197 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:04.819+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:54.820+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 197 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:54.820+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 198 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:04.820+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:54.820+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 198 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:54.820+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 197 out: Operation aborted. 2016-02-03T17:54:54.820+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:56.820Z 2016-02-03T17:54:54.820+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 198 out: Operation aborted. 2016-02-03T17:54:54.820+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:56.820Z 2016-02-03T17:54:55.428+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:55.493+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 188 out: Operation aborted. 2016-02-03T17:54:55.493+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:54:55.494+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:55.494+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:55.494+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:54:55.494+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:54:55.495+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:54:55.495+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:54:55.495+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 202 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:55:05.495+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:54:55.495+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 202 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:56.816+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:56.816+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:56.816+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:56.820+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 203 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:06.820+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:56.820+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 204 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:06.820+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:56.820+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 203 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:56.820+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 204 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:56.820+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 203 out: Operation aborted. 2016-02-03T17:54:56.820+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:54:58.820Z 2016-02-03T17:54:56.821+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 204 out: Operation aborted. 2016-02-03T17:54:56.821+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:54:58.821Z 2016-02-03T17:54:58.652+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:58.653+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:58.713+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:54:58.713+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:54:58.817+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:54:58.817+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:54:58.817+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:54:58.820+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 207 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:08.820+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:58.820+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 207 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:54:58.821+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 207 out: Operation aborted. 2016-02-03T17:54:58.821+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:00.821Z 2016-02-03T17:54:58.821+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 209 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:08.821+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:54:58.821+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 209 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:54:58.821+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 209 out: Operation aborted. 2016-02-03T17:54:58.821+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:00.821Z 2016-02-03T17:54:58.838+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:54:58.838+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:54:58.838+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:00.496+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:00.558+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 202 out: Operation aborted. 2016-02-03T17:55:00.558+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:55:00.558+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:00.559+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:00.559+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:55:00.559+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:55:00.560+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:00.560+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:55:00.560+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 212 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:55:10.560+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:55:00.560+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 212 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:00.817+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:00.817+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:00.817+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:00.822+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 213 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:10.822+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:00.822+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 214 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:10.822+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:00.822+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 213 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:00.822+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 214 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:00.822+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 213 out: Operation aborted. 2016-02-03T17:55:00.822+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:02.822Z 2016-02-03T17:55:00.822+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 214 out: Operation aborted. 2016-02-03T17:55:00.822+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:02.822Z 2016-02-03T17:55:02.789+0000 I STORAGE [DataFileSync] flushing mmaps took 1ms for 782 files 2016-02-03T17:55:02.819+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:02.819+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:02.819+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:02.822+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 217 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:12.822+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:02.822+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 217 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:02.822+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 218 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:12.822+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:02.822+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 218 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:02.823+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 217 out: Operation aborted. 2016-02-03T17:55:02.823+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:04.823Z 2016-02-03T17:55:02.823+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 218 out: Operation aborted. 2016-02-03T17:55:02.823+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:04.823Z 2016-02-03T17:55:03.202+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:03.202+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:03.372+0000 D - [PeriodicTaskRunner] cleaning up unused lock buckets of the global lock manager 2016-02-03T17:55:03.374+0000 D INDEX [TTLMonitor] TTL -- ns: mydomain.PendingSessions key: { st: 1.0 } 2016-02-03T17:55:03.838+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:55:03.838+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:55:03.838+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:04.486+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:04.486+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:04.819+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:04.819+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:04.819+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:04.823+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 221 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:14.823+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:04.823+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 222 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:14.823+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:04.823+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 221 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:04.824+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 222 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:04.824+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 221 out: Operation aborted. 2016-02-03T17:55:04.824+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:06.824Z 2016-02-03T17:55:04.824+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 222 out: Operation aborted. 2016-02-03T17:55:04.824+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:06.824Z 2016-02-03T17:55:05.561+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:05.652+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 212 out: Operation aborted. 2016-02-03T17:55:05.652+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:55:05.652+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:05.652+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:05.661+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:55:05.661+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:55:05.662+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:05.662+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:55:05.662+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 226 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:55:15.662+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:55:05.662+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 226 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:06.821+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:06.821+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:06.821+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:06.824+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 227 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:16.824+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:06.824+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 228 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:16.824+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:06.824+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 227 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:06.824+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 228 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:06.825+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 227 out: Operation aborted. 2016-02-03T17:55:06.825+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:08.825Z 2016-02-03T17:55:06.825+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 228 out: Operation aborted. 2016-02-03T17:55:06.825+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:08.825Z 2016-02-03T17:55:08.655+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:08.655+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:08.715+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:08.715+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:08.821+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:08.821+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:08.821+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:08.826+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 231 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:18.826+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:08.826+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 232 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:18.826+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:08.826+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 231 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:08.826+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 232 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:08.826+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 231 out: Operation aborted. 2016-02-03T17:55:08.826+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:10.826Z 2016-02-03T17:55:08.826+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 232 out: Operation aborted. 2016-02-03T17:55:08.826+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:10.826Z 2016-02-03T17:55:08.840+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:55:08.840+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:55:08.840+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:10.663+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:10.729+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 226 out: Operation aborted. 2016-02-03T17:55:10.729+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:55:10.729+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:10.729+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:10.730+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:55:10.730+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:55:10.731+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:10.731+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:55:10.731+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 236 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:55:20.731+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:55:10.731+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 236 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:10.823+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:10.823+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:10.823+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:10.826+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 237 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:20.826+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:10.826+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 237 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:10.826+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 238 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:20.826+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:10.826+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 238 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:10.827+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:12.827Z 2016-02-03T17:55:10.827+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 237 out: Operation aborted. 2016-02-03T17:55:10.827+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 238 out: Operation aborted. 2016-02-03T17:55:10.827+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:12.827Z 2016-02-03T17:55:12.824+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:12.824+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:12.824+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:12.827+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 241 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:22.827+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:12.827+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 242 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:22.827+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:12.827+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 241 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:12.827+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 242 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:12.828+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 241 out: Operation aborted. 2016-02-03T17:55:12.828+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:14.828Z 2016-02-03T17:55:12.828+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 242 out: Operation aborted. 2016-02-03T17:55:12.828+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:14.828Z 2016-02-03T17:55:13.204+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:13.205+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:13.841+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:55:13.841+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:55:13.841+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:14.489+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:14.489+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:14.824+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:14.824+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:14.824+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:14.828+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 245 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:24.828+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:14.828+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 245 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:14.828+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 246 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:24.828+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:14.828+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 246 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:14.829+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 245 out: Operation aborted. 2016-02-03T17:55:14.829+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:16.829Z 2016-02-03T17:55:14.829+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 246 out: Operation aborted. 2016-02-03T17:55:14.829+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:16.829Z 2016-02-03T17:55:15.731+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:15.804+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:55:15.804+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 236 out: Operation aborted. 2016-02-03T17:55:15.804+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:15.804+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:15.804+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:55:15.805+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:55:15.805+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:15.805+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:55:15.805+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 250 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:55:25.805+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:55:15.806+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 250 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:16.826+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:16.826+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:16.826+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:16.829+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 251 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:26.829+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:16.829+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 251 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:16.829+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 252 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:26.829+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:16.829+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 252 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:16.830+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 251 out: Operation aborted. 2016-02-03T17:55:16.830+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:18.830Z 2016-02-03T17:55:16.830+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:18.830Z 2016-02-03T17:55:16.830+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 252 out: Operation aborted. 2016-02-03T17:55:18.658+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:18.658+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:18.718+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:18.718+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:18.827+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:18.827+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:18.827+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:18.830+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 255 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:28.830+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:18.830+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 256 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:28.830+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:18.830+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 255 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:18.830+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 256 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:18.831+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 255 out: Operation aborted. 2016-02-03T17:55:18.831+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:20.831Z 2016-02-03T17:55:18.831+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:20.831Z 2016-02-03T17:55:18.831+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 256 out: Operation aborted. 2016-02-03T17:55:18.842+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:55:18.842+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:55:18.842+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:20.806+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:20.828+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:20.828+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:20.828+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:20.831+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 259 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:30.831+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:20.831+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 260 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:30.831+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:20.831+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 259 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:20.831+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 260 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:20.832+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 259 out: Operation aborted. 2016-02-03T17:55:20.832+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:22.832Z 2016-02-03T17:55:20.832+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:22.832Z 2016-02-03T17:55:20.832+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 260 out: Operation aborted. 2016-02-03T17:55:20.876+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 250 out: Operation aborted. 2016-02-03T17:55:20.876+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:55:20.876+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:20.876+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:20.877+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:55:20.877+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:55:20.878+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:20.878+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:55:20.878+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 264 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:55:30.878+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:55:20.878+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 264 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:22.829+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:22.829+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:22.829+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:22.832+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 265 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:32.832+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:22.832+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 265 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:22.832+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 266 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:32.832+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:22.832+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 266 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:22.833+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 265 out: Operation aborted. 2016-02-03T17:55:22.833+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:24.833Z 2016-02-03T17:55:22.833+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 266 out: Operation aborted. 2016-02-03T17:55:22.833+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:24.833Z 2016-02-03T17:55:23.207+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:23.207+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:23.842+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:55:23.842+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:55:23.842+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:24.492+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:24.492+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:24.830+0000 D COMMAND [conn3] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:24.830+0000 D COMMAND [conn3] command: replSetHeartbeat 2016-02-03T17:55:24.830+0000 I COMMAND [conn3] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:24.833+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 269 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:34.833+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:24.833+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 270 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:34.833+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:24.833+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 269 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:24.833+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 270 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:24.834+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 269 out: Operation aborted. 2016-02-03T17:55:24.834+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:26.834Z 2016-02-03T17:55:24.834+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:26.834Z 2016-02-03T17:55:24.834+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 270 out: Operation aborted. 2016-02-03T17:55:25.878+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:25.945+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:55:25.945+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 264 out: Operation aborted. 2016-02-03T17:55:25.945+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:25.945+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:25.945+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:55:25.946+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:55:25.947+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:25.947+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:55:25.947+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 274 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:55:35.947+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:55:25.947+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 274 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:26.831+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:26.831+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:26.831+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:26.834+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 275 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:36.834+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:26.834+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 276 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:36.834+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:26.834+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 275 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:26.834+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 276 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:26.835+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 275 out: Operation aborted. 2016-02-03T17:55:26.835+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:28.835Z 2016-02-03T17:55:26.835+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:28.835Z 2016-02-03T17:55:26.835+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 276 out: Operation aborted. 2016-02-03T17:55:28.661+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:28.661+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:28.721+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:28.721+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:28.832+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:28.832+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:28.832+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:28.835+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 279 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:38.835+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:28.835+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 280 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:38.835+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:28.835+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 279 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:28.835+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 280 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:28.836+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 279 out: Operation aborted. 2016-02-03T17:55:28.836+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:30.836Z 2016-02-03T17:55:28.836+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 280 out: Operation aborted. 2016-02-03T17:55:28.836+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:30.836Z 2016-02-03T17:55:28.843+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:55:28.843+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:55:28.843+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:30.833+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:30.833+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:30.833+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:30.836+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 283 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:40.836+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:30.836+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 284 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:40.836+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:30.836+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 283 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:30.836+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 284 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:30.837+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 283 out: Operation aborted. 2016-02-03T17:55:30.837+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:32.837Z 2016-02-03T17:55:30.837+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 284 out: Operation aborted. 2016-02-03T17:55:30.837+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:32.837Z 2016-02-03T17:55:30.947+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:31.015+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:55:31.015+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 274 out: Operation aborted. 2016-02-03T17:55:31.015+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:31.015+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:31.016+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:55:31.016+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:55:31.017+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:31.017+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:55:31.017+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 288 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:55:41.017+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:55:31.017+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 288 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:32.834+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:32.834+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:32.834+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:32.837+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 289 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:42.837+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:32.837+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 289 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:32.837+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 290 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:42.837+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:32.837+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 290 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:32.838+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 289 out: Operation aborted. 2016-02-03T17:55:32.838+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:34.838Z 2016-02-03T17:55:32.838+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:34.838Z 2016-02-03T17:55:32.838+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 290 out: Operation aborted. 2016-02-03T17:55:33.210+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:33.210+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:33.844+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:55:33.844+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:55:33.844+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:34.494+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:34.495+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:34.835+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:34.835+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:34.835+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:34.838+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 293 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:44.838+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:34.838+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 294 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:44.838+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:34.838+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 293 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:34.838+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 294 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:34.838+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 293 out: Operation aborted. 2016-02-03T17:55:34.838+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:36.838Z 2016-02-03T17:55:34.839+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 294 out: Operation aborted. 2016-02-03T17:55:34.839+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:36.839Z 2016-02-03T17:55:36.018+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:36.102+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:55:36.102+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 288 out: Operation aborted. 2016-02-03T17:55:36.102+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:36.102+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:36.102+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:55:36.102+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:55:36.103+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:36.103+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:55:36.103+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 298 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:55:46.103+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:55:36.103+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 298 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:36.835+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:36.835+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:36.835+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:36.839+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 299 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:46.839+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:36.839+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 300 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:46.839+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:36.839+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 299 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:36.839+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 300 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:36.839+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 299 out: Operation aborted. 2016-02-03T17:55:36.839+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:38.839Z 2016-02-03T17:55:36.839+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:38.839Z 2016-02-03T17:55:36.839+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 300 out: Operation aborted. 2016-02-03T17:55:38.663+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:38.663+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:38.723+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:38.723+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:38.836+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:38.836+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:38.836+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:38.840+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 303 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:48.840+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:38.840+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 304 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:48.840+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:38.840+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 303 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:38.840+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 304 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:38.840+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 303 out: Operation aborted. 2016-02-03T17:55:38.840+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:40.840Z 2016-02-03T17:55:38.840+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 304 out: Operation aborted. 2016-02-03T17:55:38.840+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:40.840Z 2016-02-03T17:55:38.846+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:55:38.846+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:55:38.846+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:40.838+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:40.838+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:40.838+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:40.840+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 307 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:50.840+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:40.840+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 308 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:50.840+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:40.840+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 307 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:40.840+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 308 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:40.841+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 307 out: Operation aborted. 2016-02-03T17:55:40.841+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:42.841Z 2016-02-03T17:55:40.841+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 308 out: Operation aborted. 2016-02-03T17:55:40.841+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:42.841Z 2016-02-03T17:55:41.104+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:41.185+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 298 out: Operation aborted. 2016-02-03T17:55:41.185+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:55:41.185+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:41.185+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:41.185+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:55:41.186+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:55:41.187+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:41.187+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:55:41.187+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 312 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:55:51.187+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:55:41.187+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 312 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:42.839+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:42.839+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:42.839+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:42.841+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 313 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:52.841+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:42.841+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 313 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:42.841+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 314 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:52.841+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:42.841+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 314 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:42.842+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 313 out: Operation aborted. 2016-02-03T17:55:42.842+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:44.842Z 2016-02-03T17:55:42.842+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 314 out: Operation aborted. 2016-02-03T17:55:42.842+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:44.842Z 2016-02-03T17:55:43.212+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:43.212+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:43.847+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:55:43.847+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:55:43.847+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:44.497+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:44.497+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:44.839+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:44.839+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:44.840+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:44.842+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 317 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:54.842+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:44.842+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 318 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:54.842+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:44.842+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 317 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:44.842+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 318 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:44.842+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:46.842Z 2016-02-03T17:55:44.842+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 317 out: Operation aborted. 2016-02-03T17:55:44.842+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 318 out: Operation aborted. 2016-02-03T17:55:44.842+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:46.842Z 2016-02-03T17:55:46.187+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:46.268+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:55:46.268+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 312 out: Operation aborted. 2016-02-03T17:55:46.268+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:46.268+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:46.269+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:55:46.269+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:55:46.270+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:46.270+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:55:46.270+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 322 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:55:56.270+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:55:46.270+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 322 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:46.841+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:46.841+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:46.841+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:46.842+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 323 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:56.842+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:46.842+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 323 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:46.842+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 324 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:56.842+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:46.842+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 324 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:46.843+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 323 out: Operation aborted. 2016-02-03T17:55:46.843+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:48.843Z 2016-02-03T17:55:46.843+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:48.843Z 2016-02-03T17:55:46.843+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 324 out: Operation aborted. 2016-02-03T17:55:48.666+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:48.666+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:48.726+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:48.726+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:48.843+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:48.843+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:48.843+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 327 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:58.843+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:48.843+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 327 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:48.843+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 328 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:55:58.843+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:48.843+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 328 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:48.843+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:48.844+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:50.844Z 2016-02-03T17:55:48.844+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 328 out: Operation aborted. 2016-02-03T17:55:48.844+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 327 out: Operation aborted. 2016-02-03T17:55:48.844+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:50.844Z 2016-02-03T17:55:48.848+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:55:48.848+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:55:48.848+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:50.844+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:50.844+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:50.844+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 331 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:00.844+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:50.844+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 331 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:50.844+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 332 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:00.844+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:50.844+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 332 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:50.844+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:50.845+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 331 out: Operation aborted. 2016-02-03T17:55:50.845+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:52.845Z 2016-02-03T17:55:50.845+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 332 out: Operation aborted. 2016-02-03T17:55:50.845+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:52.845Z 2016-02-03T17:55:51.270+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:51.352+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 322 out: Operation aborted. 2016-02-03T17:55:51.352+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:55:51.352+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:51.353+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:51.353+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:55:51.353+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:55:51.354+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:51.354+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:55:51.354+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 336 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:56:01.354+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:55:51.354+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 336 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:52.845+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:52.845+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:52.845+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 337 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:02.845+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:52.845+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 338 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:02.845+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:52.845+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 337 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:52.845+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 338 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:52.845+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:52.846+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 337 out: Operation aborted. 2016-02-03T17:55:52.846+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:54.846Z 2016-02-03T17:55:52.846+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:54.846Z 2016-02-03T17:55:52.846+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 338 out: Operation aborted. 2016-02-03T17:55:53.215+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:53.215+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:53.849+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:55:53.849+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:55:53.849+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:54.500+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:54.500+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:54.845+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:54.845+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:54.845+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:54.847+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 341 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:04.847+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:54.847+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 342 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:04.847+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:54.847+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 341 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:54.847+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 342 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:54.847+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 341 out: Operation aborted. 2016-02-03T17:55:54.847+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:56.847Z 2016-02-03T17:55:54.847+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:56.847Z 2016-02-03T17:55:54.847+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 342 out: Operation aborted. 2016-02-03T17:55:56.355+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:56.451+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:55:56.451+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 336 out: Operation aborted. 2016-02-03T17:55:56.451+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:56.451+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:56.452+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:55:56.452+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:55:56.453+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:55:56.453+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:55:56.453+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 346 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:56:06.453+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:55:56.453+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 346 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:56.847+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:56.847+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:56.847+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 347 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:06.847+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:56.847+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 347 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:56.847+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 348 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:06.847+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:56.847+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 348 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:56.847+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:56.848+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 348 out: Operation aborted. 2016-02-03T17:55:56.848+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:55:58.848Z 2016-02-03T17:55:56.848+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:55:58.848Z 2016-02-03T17:55:56.848+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 347 out: Operation aborted. 2016-02-03T17:55:58.668+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:58.669+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:58.728+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:55:58.728+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:55:58.848+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:55:58.848+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:55:58.849+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 351 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:08.849+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:58.849+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 351 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:55:58.849+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 352 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:08.849+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:55:58.849+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 352 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:55:58.849+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:55:58.849+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 351 out: Operation aborted. 2016-02-03T17:55:58.849+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:56:00.849Z 2016-02-03T17:55:58.849+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 352 out: Operation aborted. 2016-02-03T17:55:58.849+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:56:00.849Z 2016-02-03T17:55:58.850+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:55:58.850+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:55:58.850+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:00.849+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:56:00.849+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:56:00.849+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 355 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:10.849+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:00.849+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 355 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:00.849+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 356 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:10.849+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:00.849+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 356 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:56:00.849+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:00.850+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:56:02.850Z 2016-02-03T17:56:00.850+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 356 out: Operation aborted. 2016-02-03T17:56:00.850+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 355 out: Operation aborted. 2016-02-03T17:56:00.850+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:56:02.850Z 2016-02-03T17:56:01.453+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:56:01.521+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 346 out: Operation aborted. 2016-02-03T17:56:01.521+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:56:01.521+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:01.522+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:01.522+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:56:01.522+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:56:01.523+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:56:01.523+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:56:01.523+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 360 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:56:11.523+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:56:01.523+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 360 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:02.789+0000 I STORAGE [DataFileSync] flushing mmaps took 1ms for 782 files 2016-02-03T17:56:02.850+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:56:02.850+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:56:02.850+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 361 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:12.850+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:02.850+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 362 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:12.850+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:02.850+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 361 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:56:02.850+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 362 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:02.850+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:02.851+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 361 out: Operation aborted. 2016-02-03T17:56:02.851+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:56:04.851Z 2016-02-03T17:56:02.851+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 362 out: Operation aborted. 2016-02-03T17:56:02.851+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:56:04.851Z 2016-02-03T17:56:03.217+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:56:03.217+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:56:03.373+0000 D - [PeriodicTaskRunner] cleaning up unused lock buckets of the global lock manager 2016-02-03T17:56:03.375+0000 D INDEX [TTLMonitor] TTL -- ns: mydomain.PendingSessions key: { st: 1.0 } 2016-02-03T17:56:03.850+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:56:03.850+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:56:03.850+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:04.503+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:56:04.503+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:56:04.850+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:56:04.850+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:56:04.850+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:04.852+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 365 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:14.852+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:04.852+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 366 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:14.852+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:04.852+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 365 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:56:04.852+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 366 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:04.852+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 366 out: Operation aborted. 2016-02-03T17:56:04.852+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:56:06.852Z 2016-02-03T17:56:04.852+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:56:06.852Z 2016-02-03T17:56:04.852+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 365 out: Operation aborted. 2016-02-03T17:56:06.524+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:56:06.591+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:56:06.591+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 360 out: Operation aborted. 2016-02-03T17:56:06.591+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:06.591+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:06.600+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:56:06.600+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:56:06.601+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:56:06.601+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:56:06.601+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 370 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:56:16.601+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:56:06.601+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 370 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:06.851+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:56:06.851+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:56:06.851+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:06.852+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 371 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:16.852+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:06.852+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 371 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:06.852+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 372 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:16.852+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:06.853+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 372 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:56:06.853+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:56:08.853Z 2016-02-03T17:56:06.853+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 371 out: Operation aborted. 2016-02-03T17:56:06.853+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 372 out: Operation aborted. 2016-02-03T17:56:06.853+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:56:08.853Z 2016-02-03T17:56:08.671+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:56:08.671+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:56:08.731+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:56:08.731+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:56:08.851+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:56:08.851+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:56:08.851+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:08.852+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:56:08.852+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:56:08.852+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:08.853+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 375 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:18.853+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:08.853+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 375 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:08.853+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 376 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:18.853+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:08.853+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 376 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:56:08.853+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 375 out: Operation aborted. 2016-02-03T17:56:08.853+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:56:10.853Z 2016-02-03T17:56:08.854+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:56:10.854Z 2016-02-03T17:56:08.854+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 376 out: Operation aborted. 2016-02-03T17:56:10.852+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:56:10.852+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:56:10.852+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:10.853+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 379 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:20.853+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:10.854+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 379 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:10.854+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 379 out: Operation aborted. 2016-02-03T17:56:10.854+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:56:12.854Z 2016-02-03T17:56:10.854+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 381 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:20.854+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:10.854+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 381 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:56:10.855+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 381 out: Operation aborted. 2016-02-03T17:56:10.855+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:56:12.855Z 2016-02-03T17:56:11.602+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:56:11.691+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 370 out: Operation aborted. 2016-02-03T17:56:11.691+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:56:11.691+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:11.692+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:11.692+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:56:11.692+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:56:11.693+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:56:11.693+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:56:11.693+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 384 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:56:21.693+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:56:11.693+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 384 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:12.854+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:56:12.854+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:56:12.854+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 385 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:22.854+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:12.854+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 385 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:12.854+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:12.855+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 385 out: Operation aborted. 2016-02-03T17:56:12.855+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:56:14.855Z 2016-02-03T17:56:12.855+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 387 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:22.855+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:12.855+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 387 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:56:12.855+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 387 out: Operation aborted. 2016-02-03T17:56:12.855+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:56:14.855Z 2016-02-03T17:56:13.220+0000 D COMMAND [conn6] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:56:13.220+0000 I COMMAND [conn6] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:56:13.853+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:56:13.853+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:56:13.853+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:14.506+0000 D COMMAND [conn7] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:56:14.506+0000 I COMMAND [conn7] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:56:14.854+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:56:14.854+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:56:14.854+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:14.856+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 389 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:24.856+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:14.856+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 389 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:14.856+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 390 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:24.856+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:14.856+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 390 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:56:14.856+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:56:16.856Z 2016-02-03T17:56:14.856+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 389 out: Operation aborted. 2016-02-03T17:56:14.856+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 390 out: Operation aborted. 2016-02-03T17:56:14.856+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:56:16.856Z 2016-02-03T17:56:16.694+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:56:16.764+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Failed to time operation 384 out: Operation aborted. 2016-02-03T17:56:16.764+0000 D REPL [rsBackgroundSync-0] Error returned while fetching the oplog: ExceededTimeLimit: operation exceeded time limit 2016-02-03T17:56:16.764+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:16.764+0000 I REPL [ReplicationExecutor] syncing from: in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:16.765+0000 D COMMAND [ConnectBG] BackgroundJob starting: ConnectBG 2016-02-03T17:56:16.765+0000 D NETWORK [rsBackgroundSync] connected to server in.db2m1.roi.test.mydomain.com:27017 (10.0.208.168) 2016-02-03T17:56:16.766+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454522008000|208, t: 13 }, memberId: 2, cfgver: 197502 }, { _id: ObjectId('56b23d66643c646738e2dfd3'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 197502 } ] } 2016-02-03T17:56:16.766+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m1.roi.test.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-03T17:56:16.766+0000 D ASIO [rsBackgroundSync] startCommand: RemoteCommand 394 -- target:in.db2m1.roi.test.mydomain.com:27017 db:local expDate:2016-02-03T17:56:26.766+0000 cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp 1453367016000|1072 } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 5000, term: 13 } 2016-02-03T17:56:16.766+0000 D ASIO [NetworkInterfaceASIO-BGSync-0] Starting asynchronous command 394 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:16.855+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:56:16.855+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:56:16.856+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 395 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:26.856+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:16.856+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:16.856+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 395 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:16.856+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 396 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:26.856+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:16.856+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 396 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:56:16.856+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 396 out: Operation aborted. 2016-02-03T17:56:16.856+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:56:18.856Z 2016-02-03T17:56:16.856+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 395 out: Operation aborted. 2016-02-03T17:56:16.856+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:56:18.856Z 2016-02-03T17:56:18.674+0000 D COMMAND [conn8] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:56:18.674+0000 I COMMAND [conn8] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:56:18.734+0000 D COMMAND [conn9] run command admin.$cmd { ismaster: 1 } 2016-02-03T17:56:18.734+0000 I COMMAND [conn9] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:470 locks:{} protocol:op_command 0ms 2016-02-03T17:56:18.854+0000 D COMMAND [conn4] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } 2016-02-03T17:56:18.854+0000 D COMMAND [conn4] command: replSetHeartbeat 2016-02-03T17:56:18.854+0000 I COMMAND [conn4] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2arb.roi.test.mydomain.com:27017", fromId: 2, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:18.857+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 399 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:28.857+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:18.857+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 399 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:56:18.857+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 400 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:28.857+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:18.857+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 400 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:18.857+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:56:18.857+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:56:18.857+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:18.857+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 400 out: Operation aborted. 2016-02-03T17:56:18.857+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:56:20.857Z 2016-02-03T17:56:18.857+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:56:20.857Z 2016-02-03T17:56:18.857+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 399 out: Operation aborted. 2016-02-03T17:56:20.857+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 403 -- target:in.db2m1.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:30.857+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:20.858+0000 D ASIO [ReplicationExecutor] startCommand: RemoteCommand 404 -- target:in.db2arb.roi.test.mydomain.com:27017 db:admin expDate:2016-02-03T17:56:30.858+0000 cmd:{ replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m2.roi.test.mydomain.com:27017", fromId: 3, term: 13 } 2016-02-03T17:56:20.858+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 403 on host in.db2m1.roi.test.mydomain.com:27017 2016-02-03T17:56:20.858+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Starting asynchronous command 404 on host in.db2arb.roi.test.mydomain.com:27017 2016-02-03T17:56:20.858+0000 D COMMAND [conn2] run command admin.$cmd { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } 2016-02-03T17:56:20.858+0000 D COMMAND [conn2] command: replSetHeartbeat 2016-02-03T17:56:20.858+0000 I COMMAND [conn2] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "rsmydomain2", configVersion: 197502, from: "in.db2m1.roi.test.mydomain.com:27017", fromId: 0, term: 13 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:370 locks:{} protocol:op_command 0ms 2016-02-03T17:56:20.858+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 403 out: Operation aborted. 2016-02-03T17:56:20.858+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.roi.test.mydomain.com:27017 at 2016-02-03T17:56:22.858Z 2016-02-03T17:56:20.858+0000 D ASIO [NetworkInterfaceASIO-Replication-0] Failed to time operation 404 out: Operation aborted. 2016-02-03T17:56:20.858+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.roi.test.mydomain.com:27017 at 2016-02-03T17:56:22.858Z