2016-06-01T23:26:01.270-0400 I CONTROL [initandlisten] MongoDB starting : pid=18046 port=29113 dbpath=/data_mount_netapp/mongodata/shard027/ 64-bit host=bxb-dev-oas013 2016-06-01T23:26:01.270-0400 I CONTROL [initandlisten] db version v3.2.5 2016-06-01T23:26:01.270-0400 I CONTROL [initandlisten] git version: 34e65e5383f7ea1726332cb175b73077ec4a1b02 2016-06-01T23:26:01.270-0400 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2016-06-01T23:26:01.270-0400 I CONTROL [initandlisten] allocator: tcmalloc 2016-06-01T23:26:01.270-0400 I CONTROL [initandlisten] modules: none 2016-06-01T23:26:01.270-0400 I CONTROL [initandlisten] build environment: 2016-06-01T23:26:01.270-0400 I CONTROL [initandlisten] distmod: rhel62 2016-06-01T23:26:01.270-0400 I CONTROL [initandlisten] distarch: x86_64 2016-06-01T23:26:01.270-0400 I CONTROL [initandlisten] target_arch: x86_64 2016-06-01T23:26:01.270-0400 I CONTROL [initandlisten] options: { net: { port: 29113 }, processManagement: { fork: true, pidFilePath: "/data_mount_netapp/mongodata/shard027/mongodb.pid" }, replication: { replSet: "shard027" }, storage: { dbPath: "/data_mount_netapp/mongodata/shard027/", directoryPerDB: true, wiredTiger: { engineConfig: { cacheSizeGB: 20 } } }, systemLog: { destination: "file", path: "/data_mount_netapp/mongodata/shard027/mongod.log", quiet: true } } 2016-06-01T23:26:01.285-0400 I - [initandlisten] Detected data files in /data_mount_netapp/mongodata/shard027/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2016-06-01T23:26:01.291-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=20G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2016-06-01T23:26:03.012-0400 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2016-06-01T23:26:03.012-0400 I STORAGE [initandlisten] The size storer reports that the oplog contains 7492144 records totaling to 35378498836 bytes 2016-06-01T23:26:03.073-0400 I STORAGE [initandlisten] Sampling from the oplog between May 23 11:44:05:2 and Jun 1 21:11:34:2 to determine where to place markers for truncation 2016-06-01T23:26:03.073-0400 I STORAGE [initandlisten] Taking 658 samples and assuming that each section of oplog contains approximately 113694 records totaling to 536872095 bytes 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 23 15:00:34:77 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 23 16:40:23:8b 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 23 16:52:51:1b1 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 23 16:53:13:26d 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 07:10:54:5c 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 08:25:45:64 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 09:23:56:3 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 10:34:54:7c 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 11:27:24:198 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 13:23:11:5b 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 13:51:37:1dc 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 13:56:15:346 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 14:09:12:7d 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 14:42:18:77 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 15:33:03:38a 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 15:35:40:3c5 2016-06-01T23:26:26.297-0400 I STORAGE [initandlisten] Placing a marker at optime May 24 15:37:43:1d13 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 25 04:32:34:99 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 25 10:57:47:7b 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 25 11:22:54:23 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 25 14:57:15:4bc 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 25 15:04:20:c6 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 25 15:05:48:87 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 25 16:16:29:2ac 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 26 05:35:18:120 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 26 12:19:23:2c 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 26 14:00:02:5c 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 26 14:36:26:a0 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 26 15:39:58:1c1b 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 26 15:42:46:92 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 26 16:09:00:51 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 26 23:03:47:2 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 27 01:41:20:14 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 27 09:42:19:16d 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 27 13:14:52:5 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 27 13:23:26:c5 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 27 13:39:10:19e 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 27 16:05:26:28 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 27 16:05:54:19a 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 29 22:29:38:d 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 30 09:31:55:21 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 01:17:54:60 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 07:34:26:16 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 11:21:03:c8 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 14:20:52:a2 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 14:25:45:6a 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 14:35:23:8b 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 14:52:50:5f 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 14:57:42:49 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 15:05:07:b6 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 15:07:57:67 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 15:11:02:3c 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 15:20:40:d2 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 15:28:14:56 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 16:09:38:f4 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime May 31 17:35:59:82 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime Jun 1 05:04:48:bd 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime Jun 1 10:26:21:95 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime Jun 1 10:58:05:7f 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime Jun 1 11:26:26:ab 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime Jun 1 12:17:50:c5 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime Jun 1 12:36:06:87 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime Jun 1 13:05:57:21 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime Jun 1 14:16:26:2 2016-06-01T23:26:26.298-0400 I STORAGE [initandlisten] Placing a marker at optime Jun 1 16:22:17:ce 2016-06-01T23:26:27.596-0400 I CONTROL [initandlisten] 2016-06-01T23:26:27.596-0400 I CONTROL [initandlisten] ** WARNING: Insecure configuration, access control is not enabled and no --bind_ip has been specified. 2016-06-01T23:26:27.596-0400 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted, 2016-06-01T23:26:27.596-0400 I CONTROL [initandlisten] ** and the server listens on all available network interfaces. 2016-06-01T23:26:27.596-0400 I CONTROL [initandlisten] 2016-06-01T23:26:27.597-0400 I CONTROL [initandlisten] 2016-06-01T23:26:27.597-0400 I CONTROL [initandlisten] ** WARNING: /proc/sys/vm/overcommit_memory is 2 2016-06-01T23:26:27.597-0400 I CONTROL [initandlisten] ** Journaling works best with it set to 0 or 1 2016-06-01T23:26:27.597-0400 I CONTROL [initandlisten] 2016-06-01T23:26:27.597-0400 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. 2016-06-01T23:26:27.597-0400 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2016-06-01T23:26:27.597-0400 I CONTROL [initandlisten] 2016-06-01T23:26:27.597-0400 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. 2016-06-01T23:26:27.597-0400 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2016-06-01T23:26:27.597-0400 I CONTROL [initandlisten] 2016-06-01T23:26:27.805-0400 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data_mount_netapp/mongodata/shard027/diagnostic.data' 2016-06-01T23:26:27.805-0400 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker 2016-06-01T23:26:27.807-0400 I NETWORK [initandlisten] waiting for connections on port 29113 2016-06-01T23:26:27.809-0400 I REPL [ReplicationExecutor] New replica set config in use: { _id: "shard027", version: 418432, protocolVersion: 1, members: [ { _id: 0, host: "bxb-dev-oas003.public.oas.intdata.com:29111", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 5.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "bxb-dev-oas017.public.oas.intdata.com:29112", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 3.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "bxb-dev-oas013.public.oas.intdata.com:29113", arbiterOnly: false, buildIndexes: true, hidden: true, priority: 0.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5718e53bb94737969da5facf') } } 2016-06-01T23:26:27.809-0400 I REPL [ReplicationExecutor] This node is bxb-dev-oas013.public.oas.intdata.com:29113 in the config 2016-06-01T23:26:27.809-0400 I REPL [ReplicationExecutor] transition to STARTUP2 2016-06-01T23:26:27.809-0400 I REPL [ReplicationExecutor] Starting replication applier threads 2016-06-01T23:26:27.810-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-01T23:26:27.812-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-01T23:26:27.812-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-01T23:26:27.812-0400 I REPL [ReplicationExecutor] Member bxb-dev-oas003.public.oas.intdata.com:29111 is now in state PRIMARY 2016-06-01T23:26:27.812-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-01T23:26:27.812-0400 I REPL [ReplicationExecutor] Member bxb-dev-oas017.public.oas.intdata.com:29112 is now in state SECONDARY 2016-06-01T23:26:33.812-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-01T23:26:33.813-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-01T23:26:33.814-0400 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-01T23:34:47.928-0400 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-01T23:50:07.972-0400 I INDEX [repl writer worker 9] build index on: trancheinfodb_20160602.ti_overnight properties: { v: 1, key: { id: 1 }, name: "id_1", ns: "trancheinfodb_20160602.ti_overnight" } 2016-06-01T23:50:07.973-0400 I INDEX [repl writer worker 9] building index using bulk method 2016-06-01T23:50:08.006-0400 I INDEX [repl writer worker 9] build index done. scanned 7135 total records. 0 secs 2016-06-01T23:50:08.033-0400 I INDEX [repl writer worker 10] build index on: trancheinfodb_20160602.ti_overnight properties: { v: 1, key: { bmstatus: 1 }, name: "bmstatus_1", ns: "trancheinfodb_20160602.ti_overnight" } 2016-06-01T23:50:08.033-0400 I INDEX [repl writer worker 10] building index using bulk method 2016-06-01T23:50:08.053-0400 I INDEX [repl writer worker 10] build index done. scanned 7135 total records. 0 secs 2016-06-01T23:50:08.121-0400 I INDEX [repl writer worker 11] build index on: trancheinfodb_20160602.ti_overnight properties: { v: 1, key: { dealname: 1 }, name: "dealname_1", ns: "trancheinfodb_20160602.ti_overnight" } 2016-06-01T23:50:08.121-0400 I INDEX [repl writer worker 11] building index using bulk method 2016-06-01T23:50:08.151-0400 I INDEX [repl writer worker 11] build index done. scanned 7135 total records. 0 secs 2016-06-01T23:50:08.171-0400 I INDEX [repl writer worker 8] build index on: trancheinfodb_20160602.ti_overnight properties: { v: 1, key: { id.cusip: 1 }, name: "id.cusip_1", ns: "trancheinfodb_20160602.ti_overnight" } 2016-06-01T23:50:08.171-0400 I INDEX [repl writer worker 8] building index using bulk method 2016-06-01T23:50:08.206-0400 I INDEX [repl writer worker 8] build index done. scanned 7135 total records. 0 secs 2016-06-01T23:50:13.764-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-01T23:50:13.766-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-01T23:50:13.767-0400 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-01T23:50:19.921-0400 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T05:50:30.053-0400 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable 2016-06-02T05:50:30.053-0400 I NETWORK [initandlisten] failed to create thread after accepting new connection, closing connection 2016-06-02T07:03:43.638-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 69, timestamp: Jun 2 07:02:34:1aec) which is more than 30s behind member bxb-dev-oas017.public.oas.intdata.com:29112 whose most recent OpTime is (term: 69, timestamp: Jun 2 07:03:43:18f) 2016-06-02T07:03:43.639-0400 I REPL [ReplicationExecutor] could not find member to sync from 2016-06-02T07:03:44.639-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T07:03:44.647-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T07:04:04.411-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865443000|28, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:59 reslen:2038056 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 438963 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 439ms 2016-06-02T07:04:05.178-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865444000|3, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:0 reslen:292 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 116712 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 116ms 2016-06-02T07:04:05.328-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865444000|20, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:38 reslen:1313878 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 149091 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 149ms 2016-06-02T07:04:07.207-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865444000|20, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:115 reslen:3973111 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 1862240 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 1863ms 2016-06-02T07:04:07.450-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865444000|58, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:96 reslen:3316599 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 195549 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 196ms 2016-06-02T07:04:07.738-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865444000|173, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:97 reslen:3351285 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 253241 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 254ms 2016-06-02T07:04:10.640-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865449000|110, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:17 reslen:587721 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 157916 } }, Database: { acquireCount: { r: 1 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 17671 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 175ms 2016-06-02T07:04:10.893-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865450000|1, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:101 reslen:3491013 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 245265 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 246ms 2016-06-02T07:04:13.760-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865451000|124, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:27 reslen:933278 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 155343 } }, Database: { acquireCount: { r: 1 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 137 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 155ms 2016-06-02T07:04:14.439-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865451000|124, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:31 reslen:1071417 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 668121 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 668ms 2016-06-02T07:04:14.613-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865452000|31, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:14 reslen:483901 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 123310 } }, Database: { acquireCount: { r: 1 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 150 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 123ms 2016-06-02T07:04:16.695-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865455000|31, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:26 reslen:898369 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 222430 } }, Database: { acquireCount: { r: 1 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 159 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 222ms 2016-06-02T07:04:16.808-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865455000|31, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:34 reslen:1174573 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 102885 } }, Database: { acquireCount: { r: 1 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 132 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 103ms 2016-06-02T07:04:17.875-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865455000|57, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:89 reslen:3072094 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 1054791 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 1055ms 2016-06-02T07:04:18.529-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865455000|180, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:122 reslen:4216339 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 582792 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 584ms 2016-06-02T07:04:20.022-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865456000|122, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:122 reslen:4216042 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 1322261 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 1374ms 2016-06-02T07:04:21.539-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865456000|209, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:122 reslen:4216020 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 1472924 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 1474ms 2016-06-02T07:04:23.100-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865458000|52, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:16 reslen:553251 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 1288492 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 1288ms 2016-06-02T07:04:24.169-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865461000|125, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:122 reslen:4215319 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 947068 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 948ms 2016-06-02T07:04:25.477-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865462000|112, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:122 reslen:4214894 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 1169757 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 1170ms 2016-06-02T07:04:26.789-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865463000|15, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:69 reslen:2383578 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 1261781 } }, Database: { acquireCount: { r: 1 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 6516 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 1269ms 2016-06-02T07:04:30.518-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865464000|21, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:122 reslen:4215459 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 3652805 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 3654ms 2016-06-02T07:04:57.900-0400 I COMMAND [conn4349] command local.oplog.rs command: getMore { getMore: 15779247254, collection: "oplog.rs", maxTimeMS: 5000, term: 69, lastKnownCommittedOpTime: { ts: Timestamp 1464865464000|21, t: 69 } } cursorid:15779247254 keyUpdates:0 writeConflicts:0 exception: operation exceeded time limit code:50 numYields:0 reslen:74 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 27313666 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_command 27314ms 2016-06-02T10:25:17.070-0400 I INDEX [repl writer worker 7] build index on: trancheinfodb_20160602.ti_work_1 properties: { v: 1, key: { id: 1 }, name: "id_1", ns: "trancheinfodb_20160602.ti_work_1" } 2016-06-02T10:25:17.070-0400 I INDEX [repl writer worker 7] building index using bulk method 2016-06-02T10:25:17.101-0400 I INDEX [repl writer worker 7] build index done. scanned 7135 total records. 0 secs 2016-06-02T10:25:17.191-0400 I INDEX [repl writer worker 10] build index on: trancheinfodb_20160602.ti_work_1 properties: { v: 1, key: { bmstatus: 1 }, name: "bmstatus_1", ns: "trancheinfodb_20160602.ti_work_1" } 2016-06-02T10:25:17.191-0400 I INDEX [repl writer worker 10] building index using bulk method 2016-06-02T10:25:17.226-0400 I INDEX [repl writer worker 10] build index done. scanned 7135 total records. 0 secs 2016-06-02T10:25:17.248-0400 I INDEX [repl writer worker 6] build index on: trancheinfodb_20160602.ti_work_1 properties: { v: 1, key: { dealname: 1 }, name: "dealname_1", ns: "trancheinfodb_20160602.ti_work_1" } 2016-06-02T10:25:17.248-0400 I INDEX [repl writer worker 6] building index using bulk method 2016-06-02T10:25:17.298-0400 I INDEX [repl writer worker 6] build index done. scanned 7135 total records. 0 secs 2016-06-02T10:25:17.326-0400 I INDEX [repl writer worker 13] build index on: trancheinfodb_20160602.ti_work_1 properties: { v: 1, key: { id.cusip: 1 }, name: "id.cusip_1", ns: "trancheinfodb_20160602.ti_work_1" } 2016-06-02T10:25:17.326-0400 I INDEX [repl writer worker 13] building index using bulk method 2016-06-02T10:25:17.368-0400 I INDEX [repl writer worker 13] build index done. scanned 7135 total records. 0 secs 2016-06-02T10:25:18.595-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 69, timestamp: Jun 2 10:00:53:74) which is more than 30s behind member bxb-dev-oas017.public.oas.intdata.com:29112 whose most recent OpTime is (term: 69, timestamp: Jun 2 10:25:18:c4) 2016-06-02T10:25:18.595-0400 I REPL [ReplicationExecutor] could not find member to sync from 2016-06-02T10:25:19.595-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T10:25:19.597-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T10:27:11.349-0400 I INDEX [repl writer worker 10] build index on: trancheinfodb_20160602.ti_1 properties: { v: 1, key: { id: 1 }, name: "id_1", ns: "trancheinfodb_20160602.ti_1" } 2016-06-02T10:27:11.349-0400 I INDEX [repl writer worker 10] building index using bulk method 2016-06-02T10:27:11.727-0400 I INDEX [repl writer worker 10] build index done. scanned 7135 total records. 0 secs 2016-06-02T10:27:12.268-0400 I INDEX [repl writer worker 5] build index on: trancheinfodb_20160602.ti_1 properties: { v: 1, key: { bmstatus: 1 }, name: "bmstatus_1", ns: "trancheinfodb_20160602.ti_1" } 2016-06-02T10:27:12.268-0400 I INDEX [repl writer worker 5] building index using bulk method 2016-06-02T10:27:12.517-0400 I INDEX [repl writer worker 5] build index done. scanned 7135 total records. 0 secs 2016-06-02T10:27:13.473-0400 I INDEX [repl writer worker 0] build index on: trancheinfodb_20160602.ti_1 properties: { v: 1, key: { dealname: 1 }, name: "dealname_1", ns: "trancheinfodb_20160602.ti_1" } 2016-06-02T10:27:13.473-0400 I INDEX [repl writer worker 0] building index using bulk method 2016-06-02T10:27:13.915-0400 I INDEX [repl writer worker 0] build index done. scanned 7135 total records. 0 secs 2016-06-02T10:27:14.751-0400 I INDEX [repl writer worker 1] build index on: trancheinfodb_20160602.ti_1 properties: { v: 1, key: { id.cusip: 1 }, name: "id.cusip_1", ns: "trancheinfodb_20160602.ti_1" } 2016-06-02T10:27:14.751-0400 I INDEX [repl writer worker 1] building index using bulk method 2016-06-02T10:27:15.209-0400 I INDEX [repl writer worker 1] build index done. scanned 7135 total records. 0 secs 2016-06-02T11:15:32.519-0400 I INDEX [repl writer worker 7] build index on: trancheinfodb_20160602.ti_work_2 properties: { v: 1, key: { id: 1 }, name: "id_1", ns: "trancheinfodb_20160602.ti_work_2" } 2016-06-02T11:15:32.519-0400 I INDEX [repl writer worker 7] building index using bulk method 2016-06-02T11:15:32.574-0400 I INDEX [repl writer worker 7] build index done. scanned 7135 total records. 0 secs 2016-06-02T11:15:32.645-0400 I INDEX [repl writer worker 14] build index on: trancheinfodb_20160602.ti_work_2 properties: { v: 1, key: { bmstatus: 1 }, name: "bmstatus_1", ns: "trancheinfodb_20160602.ti_work_2" } 2016-06-02T11:15:32.645-0400 I INDEX [repl writer worker 14] building index using bulk method 2016-06-02T11:15:32.696-0400 I INDEX [repl writer worker 14] build index done. scanned 7135 total records. 0 secs 2016-06-02T11:15:32.716-0400 I INDEX [repl writer worker 13] build index on: trancheinfodb_20160602.ti_work_2 properties: { v: 1, key: { dealname: 1 }, name: "dealname_1", ns: "trancheinfodb_20160602.ti_work_2" } 2016-06-02T11:15:32.716-0400 I INDEX [repl writer worker 13] building index using bulk method 2016-06-02T11:15:32.742-0400 I INDEX [repl writer worker 13] build index done. scanned 7135 total records. 0 secs 2016-06-02T11:15:32.762-0400 I INDEX [repl writer worker 9] build index on: trancheinfodb_20160602.ti_work_2 properties: { v: 1, key: { id.cusip: 1 }, name: "id.cusip_1", ns: "trancheinfodb_20160602.ti_work_2" } 2016-06-02T11:15:32.762-0400 I INDEX [repl writer worker 9] building index using bulk method 2016-06-02T11:15:32.792-0400 I INDEX [repl writer worker 9] build index done. scanned 7135 total records. 0 secs 2016-06-02T11:15:45.952-0400 I INDEX [repl writer worker 11] build index on: trancheinfodb_20160602.ti_2 properties: { v: 1, key: { id: 1 }, name: "id_1", ns: "trancheinfodb_20160602.ti_2" } 2016-06-02T11:15:45.952-0400 I INDEX [repl writer worker 11] building index using bulk method 2016-06-02T11:15:45.981-0400 I INDEX [repl writer worker 11] build index done. scanned 7135 total records. 0 secs 2016-06-02T11:15:45.991-0400 I INDEX [repl writer worker 7] build index on: trancheinfodb_20160602.ti_2 properties: { v: 1, key: { bmstatus: 1 }, name: "bmstatus_1", ns: "trancheinfodb_20160602.ti_2" } 2016-06-02T11:15:45.991-0400 I INDEX [repl writer worker 7] building index using bulk method 2016-06-02T11:15:46.009-0400 I INDEX [repl writer worker 7] build index done. scanned 7135 total records. 0 secs 2016-06-02T11:15:46.022-0400 I INDEX [repl writer worker 5] build index on: trancheinfodb_20160602.ti_2 properties: { v: 1, key: { dealname: 1 }, name: "dealname_1", ns: "trancheinfodb_20160602.ti_2" } 2016-06-02T11:15:46.022-0400 I INDEX [repl writer worker 5] building index using bulk method 2016-06-02T11:15:46.043-0400 I INDEX [repl writer worker 5] build index done. scanned 7135 total records. 0 secs 2016-06-02T11:15:46.053-0400 I INDEX [repl writer worker 4] build index on: trancheinfodb_20160602.ti_2 properties: { v: 1, key: { id.cusip: 1 }, name: "id.cusip_1", ns: "trancheinfodb_20160602.ti_2" } 2016-06-02T11:15:46.053-0400 I INDEX [repl writer worker 4] building index using bulk method 2016-06-02T11:15:46.074-0400 I INDEX [repl writer worker 4] build index done. scanned 7135 total records. 0 secs 2016-06-02T11:51:18.529-0400 I INDEX [repl writer worker 9] build index on: trancheinfodb_20160603.dealuniverse properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.dealuniverse" } 2016-06-02T11:51:18.529-0400 I INDEX [repl writer worker 9] building index using bulk method 2016-06-02T11:51:18.532-0400 I INDEX [repl writer worker 9] build index done. scanned 0 total records. 0 secs 2016-06-02T11:51:18.623-0400 I INDEX [repl writer worker 13] build index on: trancheinfodb_20160603.ti_work properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_work" } 2016-06-02T11:51:18.623-0400 I INDEX [repl writer worker 13] building index using bulk method 2016-06-02T11:51:18.625-0400 I INDEX [repl writer worker 13] build index done. scanned 0 total records. 0 secs 2016-06-02T11:51:18.651-0400 I INDEX [repl writer worker 8] build index on: trancheinfodb_20160603.ti_work4pm properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_work4pm" } 2016-06-02T11:51:18.651-0400 I INDEX [repl writer worker 8] building index using bulk method 2016-06-02T11:51:18.654-0400 I INDEX [repl writer worker 8] build index done. scanned 0 total records. 0 secs 2016-06-02T11:51:18.701-0400 I INDEX [repl writer worker 2] build index on: trancheinfodb_20160603.cmoprice properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.cmoprice" } 2016-06-02T11:51:18.701-0400 I INDEX [repl writer worker 2] building index using bulk method 2016-06-02T11:51:18.704-0400 I INDEX [repl writer worker 2] build index done. scanned 0 total records. 0 secs 2016-06-02T11:51:18.753-0400 I INDEX [repl writer worker 3] build index on: trancheinfodb_20160603.ti_overflow properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_overflow" } 2016-06-02T11:51:18.753-0400 I INDEX [repl writer worker 3] building index using bulk method 2016-06-02T11:51:18.755-0400 I INDEX [repl writer worker 3] build index done. scanned 0 total records. 0 secs 2016-06-02T11:51:18.804-0400 I INDEX [repl writer worker 15] build index on: trancheinfodb_20160603.stripprice properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.stripprice" } 2016-06-02T11:51:18.804-0400 I INDEX [repl writer worker 15] building index using bulk method 2016-06-02T11:51:18.807-0400 I INDEX [repl writer worker 15] build index done. scanned 0 total records. 0 secs 2016-06-02T11:51:18.853-0400 I INDEX [repl writer worker 6] build index on: trancheinfodb_20160603.ti_overnight properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_overnight" } 2016-06-02T11:51:18.853-0400 I INDEX [repl writer worker 6] building index using bulk method 2016-06-02T11:51:18.856-0400 I INDEX [repl writer worker 6] build index done. scanned 0 total records. 0 secs 2016-06-02T11:51:20.971-0400 I INDEX [repl writer worker 7] build index on: trancheinfodb_20160603.tr_shared properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.tr_shared" } 2016-06-02T11:51:20.971-0400 I INDEX [repl writer worker 7] building index using bulk method 2016-06-02T11:51:20.974-0400 I INDEX [repl writer worker 7] build index done. scanned 0 total records. 0 secs 2016-06-02T11:51:21.026-0400 I INDEX [repl writer worker 12] build index on: trancheinfodb_20160603.ti_1 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_1" } 2016-06-02T11:51:21.026-0400 I INDEX [repl writer worker 12] building index using bulk method 2016-06-02T11:51:21.029-0400 I INDEX [repl writer worker 12] build index done. scanned 0 total records. 0 secs 2016-06-02T11:51:21.061-0400 I INDEX [repl writer worker 1] build index on: trancheinfodb_20160603.ti_work_1 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_work_1" } 2016-06-02T11:51:21.061-0400 I INDEX [repl writer worker 1] building index using bulk method 2016-06-02T11:51:21.063-0400 I INDEX [repl writer worker 1] build index done. scanned 0 total records. 0 secs 2016-06-02T11:51:21.110-0400 I INDEX [repl writer worker 5] build index on: trancheinfodb_20160603.pr_1 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.pr_1" } 2016-06-02T11:51:21.110-0400 I INDEX [repl writer worker 5] building index using bulk method 2016-06-02T11:51:21.112-0400 I INDEX [repl writer worker 5] build index done. scanned 0 total records. 0 secs 2016-06-02T11:53:26.910-0400 I INDEX [repl writer worker 10] build index on: trancheinfodb_20160603.ti_2 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_2" } 2016-06-02T11:53:26.910-0400 I INDEX [repl writer worker 10] building index using bulk method 2016-06-02T11:53:26.912-0400 I INDEX [repl writer worker 10] build index done. scanned 0 total records. 0 secs 2016-06-02T11:53:26.959-0400 I INDEX [repl writer worker 11] build index on: trancheinfodb_20160603.ti_work_2 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_work_2" } 2016-06-02T11:53:26.959-0400 I INDEX [repl writer worker 11] building index using bulk method 2016-06-02T11:53:26.962-0400 I INDEX [repl writer worker 11] build index done. scanned 0 total records. 0 secs 2016-06-02T11:53:27.018-0400 I INDEX [repl writer worker 14] build index on: trancheinfodb_20160603.pr_2 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.pr_2" } 2016-06-02T11:53:27.018-0400 I INDEX [repl writer worker 14] building index using bulk method 2016-06-02T11:53:27.020-0400 I INDEX [repl writer worker 14] build index done. scanned 0 total records. 0 secs 2016-06-02T11:53:27.099-0400 I INDEX [repl writer worker 0] build index on: trancheinfodb_20160603.ti_3 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_3" } 2016-06-02T11:53:27.099-0400 I INDEX [repl writer worker 0] building index using bulk method 2016-06-02T11:53:27.101-0400 I INDEX [repl writer worker 0] build index done. scanned 0 total records. 0 secs 2016-06-02T11:53:27.131-0400 I INDEX [repl writer worker 4] build index on: trancheinfodb_20160603.ti_work_3 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_work_3" } 2016-06-02T11:53:27.131-0400 I INDEX [repl writer worker 4] building index using bulk method 2016-06-02T11:53:27.134-0400 I INDEX [repl writer worker 4] build index done. scanned 0 total records. 0 secs 2016-06-02T11:53:28.788-0400 I INDEX [repl writer worker 9] build index on: trancheinfodb_20160603.pr_3 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.pr_3" } 2016-06-02T11:53:28.788-0400 I INDEX [repl writer worker 9] building index using bulk method 2016-06-02T11:53:28.795-0400 I INDEX [repl writer worker 9] build index done. scanned 0 total records. 0 secs 2016-06-02T11:53:28.832-0400 I INDEX [repl writer worker 13] build index on: trancheinfodb_20160603.ti_4 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_4" } 2016-06-02T11:53:28.832-0400 I INDEX [repl writer worker 13] building index using bulk method 2016-06-02T11:53:28.835-0400 I INDEX [repl writer worker 13] build index done. scanned 0 total records. 0 secs 2016-06-02T11:53:28.889-0400 I INDEX [repl writer worker 8] build index on: trancheinfodb_20160603.pr_4 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.pr_4" } 2016-06-02T11:53:28.889-0400 I INDEX [repl writer worker 8] building index using bulk method 2016-06-02T11:53:28.891-0400 I INDEX [repl writer worker 8] build index done. scanned 0 total records. 0 secs 2016-06-02T11:53:28.932-0400 I INDEX [repl writer worker 2] build index on: trancheinfodb_20160603.ti_5 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_5" } 2016-06-02T11:53:28.932-0400 I INDEX [repl writer worker 2] building index using bulk method 2016-06-02T11:53:28.935-0400 I INDEX [repl writer worker 2] build index done. scanned 0 total records. 0 secs 2016-06-02T11:53:28.981-0400 I INDEX [repl writer worker 3] build index on: trancheinfodb_20160603.ti_work_5 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_work_5" } 2016-06-02T11:53:28.981-0400 I INDEX [repl writer worker 3] building index using bulk method 2016-06-02T11:53:28.984-0400 I INDEX [repl writer worker 3] build index done. scanned 0 total records. 0 secs 2016-06-02T11:53:29.032-0400 I INDEX [repl writer worker 15] build index on: trancheinfodb_20160603.ti_work_4 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_work_4" } 2016-06-02T11:53:29.032-0400 I INDEX [repl writer worker 15] building index using bulk method 2016-06-02T11:53:29.034-0400 I INDEX [repl writer worker 15] build index done. scanned 0 total records. 0 secs 2016-06-02T11:55:50.617-0400 I INDEX [repl writer worker 6] build index on: trancheinfodb_20160603.pr_5 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.pr_5" } 2016-06-02T11:55:50.617-0400 I INDEX [repl writer worker 6] building index using bulk method 2016-06-02T11:55:50.619-0400 I INDEX [repl writer worker 6] build index done. scanned 0 total records. 0 secs 2016-06-02T11:55:50.668-0400 I INDEX [repl writer worker 7] build index on: trancheinfodb_20160603.ti_6 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_6" } 2016-06-02T11:55:50.668-0400 I INDEX [repl writer worker 7] building index using bulk method 2016-06-02T11:55:50.670-0400 I INDEX [repl writer worker 7] build index done. scanned 0 total records. 0 secs 2016-06-02T11:55:50.717-0400 I INDEX [repl writer worker 12] build index on: trancheinfodb_20160603.ti_work_6 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_work_6" } 2016-06-02T11:55:50.717-0400 I INDEX [repl writer worker 12] building index using bulk method 2016-06-02T11:55:50.719-0400 I INDEX [repl writer worker 12] build index done. scanned 0 total records. 0 secs 2016-06-02T11:55:50.781-0400 I INDEX [repl writer worker 1] build index on: trancheinfodb_20160603.pr_6 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.pr_6" } 2016-06-02T11:55:50.781-0400 I INDEX [repl writer worker 1] building index using bulk method 2016-06-02T11:55:50.784-0400 I INDEX [repl writer worker 1] build index done. scanned 0 total records. 0 secs 2016-06-02T11:55:50.819-0400 I INDEX [repl writer worker 5] build index on: trancheinfodb_20160603.ti_7 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_7" } 2016-06-02T11:55:50.819-0400 I INDEX [repl writer worker 5] building index using bulk method 2016-06-02T11:55:50.821-0400 I INDEX [repl writer worker 5] build index done. scanned 0 total records. 0 secs 2016-06-02T11:55:50.867-0400 I INDEX [repl writer worker 10] build index on: trancheinfodb_20160603.ti_work_7 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_work_7" } 2016-06-02T11:55:50.867-0400 I INDEX [repl writer worker 10] building index using bulk method 2016-06-02T11:55:50.870-0400 I INDEX [repl writer worker 10] build index done. scanned 0 total records. 0 secs 2016-06-02T11:55:53.245-0400 I INDEX [repl writer worker 11] build index on: trancheinfodb_20160603.pr_7 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.pr_7" } 2016-06-02T11:55:53.245-0400 I INDEX [repl writer worker 11] building index using bulk method 2016-06-02T11:55:53.248-0400 I INDEX [repl writer worker 11] build index done. scanned 0 total records. 0 secs 2016-06-02T11:55:53.294-0400 I INDEX [repl writer worker 14] build index on: trancheinfodb_20160603.ti_8 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_8" } 2016-06-02T11:55:53.294-0400 I INDEX [repl writer worker 14] building index using bulk method 2016-06-02T11:55:53.297-0400 I INDEX [repl writer worker 14] build index done. scanned 0 total records. 0 secs 2016-06-02T11:55:53.342-0400 I INDEX [repl writer worker 0] build index on: trancheinfodb_20160603.ti_work_8 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_work_8" } 2016-06-02T11:55:53.342-0400 I INDEX [repl writer worker 0] building index using bulk method 2016-06-02T11:55:53.345-0400 I INDEX [repl writer worker 0] build index done. scanned 0 total records. 0 secs 2016-06-02T11:55:53.392-0400 I INDEX [repl writer worker 4] build index on: trancheinfodb_20160603.pr_8 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.pr_8" } 2016-06-02T11:55:53.392-0400 I INDEX [repl writer worker 4] building index using bulk method 2016-06-02T11:55:53.394-0400 I INDEX [repl writer worker 4] build index done. scanned 0 total records. 0 secs 2016-06-02T11:55:53.440-0400 I INDEX [repl writer worker 9] build index on: trancheinfodb_20160603.ti_9 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_9" } 2016-06-02T11:55:53.440-0400 I INDEX [repl writer worker 9] building index using bulk method 2016-06-02T11:55:53.443-0400 I INDEX [repl writer worker 9] build index done. scanned 0 total records. 0 secs 2016-06-02T11:57:30.545-0400 I INDEX [repl writer worker 13] build index on: trancheinfodb_20160603.ti_work_9 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_work_9" } 2016-06-02T11:57:30.545-0400 I INDEX [repl writer worker 13] building index using bulk method 2016-06-02T11:57:30.547-0400 I INDEX [repl writer worker 13] build index done. scanned 0 total records. 0 secs 2016-06-02T11:57:30.586-0400 I INDEX [repl writer worker 8] build index on: trancheinfodb_20160603.pr_9 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.pr_9" } 2016-06-02T11:57:30.586-0400 I INDEX [repl writer worker 8] building index using bulk method 2016-06-02T11:57:30.588-0400 I INDEX [repl writer worker 8] build index done. scanned 0 total records. 0 secs 2016-06-02T11:57:30.635-0400 I INDEX [repl writer worker 2] build index on: trancheinfodb_20160603.ti_10 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_10" } 2016-06-02T11:57:30.635-0400 I INDEX [repl writer worker 2] building index using bulk method 2016-06-02T11:57:30.637-0400 I INDEX [repl writer worker 2] build index done. scanned 0 total records. 0 secs 2016-06-02T11:57:36.181-0400 I INDEX [repl writer worker 3] build index on: trancheinfodb_20160603.ti_work_10 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.ti_work_10" } 2016-06-02T11:57:36.181-0400 I INDEX [repl writer worker 3] building index using bulk method 2016-06-02T11:57:36.184-0400 I INDEX [repl writer worker 3] build index done. scanned 0 total records. 0 secs 2016-06-02T11:57:36.261-0400 I INDEX [repl writer worker 15] build index on: trancheinfodb_20160603.pr_10 properties: { v: 1, key: { bucket_id: 1.0 }, name: "bucket_id_1", ns: "trancheinfodb_20160603.pr_10" } 2016-06-02T11:57:36.261-0400 I INDEX [repl writer worker 15] building index using bulk method 2016-06-02T11:57:36.264-0400 I INDEX [repl writer worker 15] build index done. scanned 0 total records. 0 secs 2016-06-02T12:00:43.442-0400 I INDEX [repl writer worker 9] build index on: trancheinfodb_20160603.dealuniverse properties: { v: 1, key: { dealName: 1 }, name: "dealName_", ns: "trancheinfodb_20160603.dealuniverse" } 2016-06-02T12:00:43.442-0400 I INDEX [repl writer worker 9] building index using bulk method 2016-06-02T12:00:43.449-0400 I INDEX [repl writer worker 9] build index done. scanned 0 total records. 0 secs 2016-06-02T12:00:43.468-0400 I INDEX [repl writer worker 10] build index on: trancheinfodb_20160603.dealuniverse properties: { v: 1, key: { calcStatus.status: 1 }, name: "calcStatus.status_", ns: "trancheinfodb_20160603.dealuniverse" } 2016-06-02T12:00:43.469-0400 I INDEX [repl writer worker 10] building index using bulk method 2016-06-02T12:00:43.476-0400 I INDEX [repl writer worker 10] build index done. scanned 0 total records. 0 secs 2016-06-02T12:00:43.496-0400 I INDEX [repl writer worker 6] build index on: trancheinfodb_20160603.dealuniverse properties: { v: 1, key: { dealName: 1, calcStatus: 1 }, name: "dealName_CalcStatus_", ns: "trancheinfodb_20160603.dealuniverse" } 2016-06-02T12:00:43.496-0400 I INDEX [repl writer worker 6] building index using bulk method 2016-06-02T12:00:43.502-0400 I INDEX [repl writer worker 6] build index done. scanned 0 total records. 0 secs 2016-06-02T12:00:43.522-0400 I INDEX [repl writer worker 13] build index on: trancheinfodb_20160603.dealuniverse properties: { v: 1, key: { calcStatus.status: 1, bucket_id: 1 }, name: "calcStatus.status_bucketId_", ns: "trancheinfodb_20160603.dealuniverse" } 2016-06-02T12:00:43.522-0400 I INDEX [repl writer worker 13] building index using bulk method 2016-06-02T12:00:43.530-0400 I INDEX [repl writer worker 13] build index done. scanned 0 total records. 0 secs 2016-06-02T12:00:43.562-0400 I INDEX [repl writer worker 11] build index on: trancheinfodb_20160603.dealuniverse properties: { v: 1, key: { calcStatus.status: 1, calcStatus.bucketId: 1 }, name: "calcStatus.status_calcStatus.bucketId_", ns: "trancheinfodb_20160603.dealuniverse" } 2016-06-02T12:00:43.562-0400 I INDEX [repl writer worker 11] building index using bulk method 2016-06-02T12:00:43.572-0400 I INDEX [repl writer worker 11] build index done. scanned 0 total records. 0 secs 2016-06-02T12:00:43.599-0400 I INDEX [repl writer worker 7] build index on: trancheinfodb_20160603.dealuniverse properties: { v: 1, key: { calcStatus.status: 1, calcStatus.bucketId: 1, bucket_id: 1 }, name: "calcStatus.status_calcStatus.bucketId_bucketId_", ns: "trancheinfodb_20160603.dealuniverse" } 2016-06-02T12:00:43.599-0400 I INDEX [repl writer worker 7] building index using bulk method 2016-06-02T12:00:43.607-0400 I INDEX [repl writer worker 7] build index done. scanned 0 total records. 0 secs 2016-06-02T12:00:43.620-0400 I INDEX [repl writer worker 8] build index on: trancheinfodb_20160603.dealuniverse properties: { v: 1, key: { calcStatus.bucketId: 1, bucket_id: 1 }, name: "calcStatus.bucketId_bucketId_", ns: "trancheinfodb_20160603.dealuniverse" } 2016-06-02T12:00:43.620-0400 I INDEX [repl writer worker 8] building index using bulk method 2016-06-02T12:00:43.623-0400 I INDEX [repl writer worker 8] build index done. scanned 0 total records. 0 secs 2016-06-02T12:02:44.496-0400 I INDEX [repl writer worker 5] build index on: trancheinfodb_20160602.ti_work_4 properties: { v: 1, key: { id: 1 }, name: "id_1", ns: "trancheinfodb_20160602.ti_work_4" } 2016-06-02T12:02:44.496-0400 I INDEX [repl writer worker 5] building index using bulk method 2016-06-02T12:02:44.549-0400 I INDEX [repl writer worker 5] build index done. scanned 7135 total records. 0 secs 2016-06-02T12:02:44.613-0400 I INDEX [repl writer worker 8] build index on: trancheinfodb_20160602.ti_work_4 properties: { v: 1, key: { bmstatus: 1 }, name: "bmstatus_1", ns: "trancheinfodb_20160602.ti_work_4" } 2016-06-02T12:02:44.613-0400 I INDEX [repl writer worker 8] building index using bulk method 2016-06-02T12:02:44.657-0400 I INDEX [repl writer worker 8] build index done. scanned 7135 total records. 0 secs 2016-06-02T12:02:44.688-0400 I INDEX [repl writer worker 1] build index on: trancheinfodb_20160602.ti_work_4 properties: { v: 1, key: { dealname: 1 }, name: "dealname_1", ns: "trancheinfodb_20160602.ti_work_4" } 2016-06-02T12:02:44.688-0400 I INDEX [repl writer worker 1] building index using bulk method 2016-06-02T12:02:44.717-0400 I INDEX [repl writer worker 1] build index done. scanned 7135 total records. 0 secs 2016-06-02T12:02:44.748-0400 I INDEX [repl writer worker 7] build index on: trancheinfodb_20160602.ti_work_4 properties: { v: 1, key: { id.cusip: 1 }, name: "id.cusip_1", ns: "trancheinfodb_20160602.ti_work_4" } 2016-06-02T12:02:44.748-0400 I INDEX [repl writer worker 7] building index using bulk method 2016-06-02T12:02:44.783-0400 I INDEX [repl writer worker 7] build index done. scanned 7135 total records. 0 secs 2016-06-02T12:02:45.007-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 69, timestamp: Jun 2 12:01:13:a08) which is more than 30s behind member bxb-dev-oas017.public.oas.intdata.com:29112 whose most recent OpTime is (term: 69, timestamp: Jun 2 12:02:45:1) 2016-06-02T12:02:45.007-0400 I REPL [ReplicationExecutor] could not find member to sync from 2016-06-02T12:02:51.008-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T12:02:51.016-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T12:02:51.017-0400 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T12:02:53.696-0400 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T12:04:31.093-0400 I INDEX [repl writer worker 7] build index on: trancheinfodb_20160602.ti_4 properties: { v: 1, key: { id: 1 }, name: "id_1", ns: "trancheinfodb_20160602.ti_4" } 2016-06-02T12:04:31.093-0400 I INDEX [repl writer worker 7] building index using bulk method 2016-06-02T12:04:32.351-0400 I INDEX [repl writer worker 7] build index done. scanned 7135 total records. 1 secs 2016-06-02T12:04:33.107-0400 I INDEX [repl writer worker 3] build index on: trancheinfodb_20160602.ti_4 properties: { v: 1, key: { bmstatus: 1 }, name: "bmstatus_1", ns: "trancheinfodb_20160602.ti_4" } 2016-06-02T12:04:33.107-0400 I INDEX [repl writer worker 3] building index using bulk method 2016-06-02T12:04:33.676-0400 I INDEX [repl writer worker 3] build index done. scanned 7135 total records. 0 secs 2016-06-02T12:04:34.250-0400 I INDEX [repl writer worker 10] build index on: trancheinfodb_20160602.ti_4 properties: { v: 1, key: { dealname: 1 }, name: "dealname_1", ns: "trancheinfodb_20160602.ti_4" } 2016-06-02T12:04:34.251-0400 I INDEX [repl writer worker 10] building index using bulk method 2016-06-02T12:04:34.618-0400 I INDEX [repl writer worker 10] build index done. scanned 7135 total records. 0 secs 2016-06-02T12:04:35.010-0400 I INDEX [repl writer worker 4] build index on: trancheinfodb_20160602.ti_4 properties: { v: 1, key: { id.cusip: 1 }, name: "id.cusip_1", ns: "trancheinfodb_20160602.ti_4" } 2016-06-02T12:04:35.010-0400 I INDEX [repl writer worker 4] building index using bulk method 2016-06-02T12:04:35.080-0400 I INDEX [repl writer worker 4] build index done. scanned 7135 total records. 0 secs 2016-06-02T12:56:52.539-0400 I INDEX [repl writer worker 6] build index on: trancheinfodb_20160602.ti_overflow properties: { v: 1, key: { id: 1 }, name: "id_1", ns: "trancheinfodb_20160602.ti_overflow" } 2016-06-02T12:56:52.539-0400 I INDEX [repl writer worker 6] building index using bulk method 2016-06-02T12:56:52.721-0400 I INDEX [repl writer worker 6] build index done. scanned 7135 total records. 0 secs 2016-06-02T12:56:52.728-0400 I INDEX [repl writer worker 13] build index on: trancheinfodb_20160602.ti_overflow properties: { v: 1, key: { dealname: 1 }, name: "dealname_1", ns: "trancheinfodb_20160602.ti_overflow" } 2016-06-02T12:56:52.728-0400 I INDEX [repl writer worker 13] building index using bulk method 2016-06-02T12:56:52.820-0400 I INDEX [repl writer worker 13] build index done. scanned 7135 total records. 0 secs 2016-06-02T12:56:52.836-0400 I INDEX [repl writer worker 7] build index on: trancheinfodb_20160602.ti_overflow properties: { v: 1, key: { id.cusip: 1 }, name: "id.cusip_1", ns: "trancheinfodb_20160602.ti_overflow" } 2016-06-02T12:56:52.836-0400 I INDEX [repl writer worker 7] building index using bulk method 2016-06-02T12:56:52.934-0400 I INDEX [repl writer worker 7] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:16:25.565-0400 I INDEX [repl writer worker 11] build index on: trancheinfodb_20160602.ti_work_3 properties: { v: 1, key: { id: 1 }, name: "id_1", ns: "trancheinfodb_20160602.ti_work_3" } 2016-06-02T13:16:25.565-0400 I INDEX [repl writer worker 11] building index using bulk method 2016-06-02T13:16:25.697-0400 I INDEX [repl writer worker 11] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:16:25.705-0400 I INDEX [repl writer worker 3] build index on: trancheinfodb_20160602.ti_work_3 properties: { v: 1, key: { bmstatus: 1 }, name: "bmstatus_1", ns: "trancheinfodb_20160602.ti_work_3" } 2016-06-02T13:16:25.705-0400 I INDEX [repl writer worker 3] building index using bulk method 2016-06-02T13:16:25.783-0400 I INDEX [repl writer worker 3] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:16:25.801-0400 I INDEX [repl writer worker 6] build index on: trancheinfodb_20160602.ti_work_3 properties: { v: 1, key: { dealname: 1 }, name: "dealname_1", ns: "trancheinfodb_20160602.ti_work_3" } 2016-06-02T13:16:25.801-0400 I INDEX [repl writer worker 6] building index using bulk method 2016-06-02T13:16:25.856-0400 I INDEX [repl writer worker 6] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:16:25.943-0400 I INDEX [repl writer worker 7] build index on: trancheinfodb_20160602.ti_work_3 properties: { v: 1, key: { id.cusip: 1 }, name: "id.cusip_1", ns: "trancheinfodb_20160602.ti_work_3" } 2016-06-02T13:16:25.943-0400 I INDEX [repl writer worker 7] building index using bulk method 2016-06-02T13:16:26.028-0400 I INDEX [repl writer worker 7] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:17:32.264-0400 I INDEX [repl writer worker 7] build index on: trancheinfodb_20160602.ti_3 properties: { v: 1, key: { id: 1 }, name: "id_1", ns: "trancheinfodb_20160602.ti_3" } 2016-06-02T13:17:32.264-0400 I INDEX [repl writer worker 7] building index using bulk method 2016-06-02T13:17:32.407-0400 I INDEX [repl writer worker 7] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:17:32.415-0400 I INDEX [repl writer worker 14] build index on: trancheinfodb_20160602.ti_3 properties: { v: 1, key: { bmstatus: 1 }, name: "bmstatus_1", ns: "trancheinfodb_20160602.ti_3" } 2016-06-02T13:17:32.415-0400 I INDEX [repl writer worker 14] building index using bulk method 2016-06-02T13:17:32.513-0400 I INDEX [repl writer worker 14] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:17:32.521-0400 I INDEX [repl writer worker 13] build index on: trancheinfodb_20160602.ti_3 properties: { v: 1, key: { dealname: 1 }, name: "dealname_1", ns: "trancheinfodb_20160602.ti_3" } 2016-06-02T13:17:32.521-0400 I INDEX [repl writer worker 13] building index using bulk method 2016-06-02T13:17:32.632-0400 I INDEX [repl writer worker 13] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:17:32.640-0400 I INDEX [repl writer worker 11] build index on: trancheinfodb_20160602.ti_3 properties: { v: 1, key: { id.cusip: 1 }, name: "id.cusip_1", ns: "trancheinfodb_20160602.ti_3" } 2016-06-02T13:17:32.640-0400 I INDEX [repl writer worker 11] building index using bulk method 2016-06-02T13:17:32.756-0400 I INDEX [repl writer worker 11] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:18:16.906-0400 I INDEX [repl writer worker 8] build index on: trancheinfodb_20160602.ti_work_5 properties: { v: 1, key: { id: 1 }, name: "id_1", ns: "trancheinfodb_20160602.ti_work_5" } 2016-06-02T13:18:16.906-0400 I INDEX [repl writer worker 8] building index using bulk method 2016-06-02T13:18:17.057-0400 I INDEX [repl writer worker 8] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:18:17.075-0400 I INDEX [repl writer worker 1] build index on: trancheinfodb_20160602.ti_work_5 properties: { v: 1, key: { bmstatus: 1 }, name: "bmstatus_1", ns: "trancheinfodb_20160602.ti_work_5" } 2016-06-02T13:18:17.075-0400 I INDEX [repl writer worker 1] building index using bulk method 2016-06-02T13:18:17.197-0400 I INDEX [repl writer worker 1] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:18:17.222-0400 I INDEX [repl writer worker 4] build index on: trancheinfodb_20160602.ti_work_5 properties: { v: 1, key: { dealname: 1 }, name: "dealname_1", ns: "trancheinfodb_20160602.ti_work_5" } 2016-06-02T13:18:17.222-0400 I INDEX [repl writer worker 4] building index using bulk method 2016-06-02T13:18:17.303-0400 I INDEX [repl writer worker 4] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:18:17.347-0400 I INDEX [repl writer worker 3] build index on: trancheinfodb_20160602.ti_work_5 properties: { v: 1, key: { id.cusip: 1 }, name: "id.cusip_1", ns: "trancheinfodb_20160602.ti_work_5" } 2016-06-02T13:18:17.347-0400 I INDEX [repl writer worker 3] building index using bulk method 2016-06-02T13:18:17.454-0400 I INDEX [repl writer worker 3] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:21:22.839-0400 I INDEX [repl writer worker 6] build index on: trancheinfodb_20160602.ti_5 properties: { v: 1, key: { id: 1 }, name: "id_1", ns: "trancheinfodb_20160602.ti_5" } 2016-06-02T13:21:22.839-0400 I INDEX [repl writer worker 6] building index using bulk method 2016-06-02T13:21:25.321-0400 I INDEX [repl writer worker 6] build index done. scanned 7135 total records. 2 secs 2016-06-02T13:21:25.383-0400 I INDEX [repl writer worker 14] build index on: trancheinfodb_20160602.ti_5 properties: { v: 1, key: { bmstatus: 1 }, name: "bmstatus_1", ns: "trancheinfodb_20160602.ti_5" } 2016-06-02T13:21:25.383-0400 I INDEX [repl writer worker 14] building index using bulk method 2016-06-02T13:21:25.456-0400 I INDEX [repl writer worker 14] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:21:25.509-0400 I INDEX [repl writer worker 5] build index on: trancheinfodb_20160602.ti_5 properties: { v: 1, key: { dealname: 1 }, name: "dealname_1", ns: "trancheinfodb_20160602.ti_5" } 2016-06-02T13:21:25.509-0400 I INDEX [repl writer worker 5] building index using bulk method 2016-06-02T13:21:25.562-0400 I INDEX [repl writer worker 5] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:21:25.571-0400 I INDEX [repl writer worker 11] build index on: trancheinfodb_20160602.ti_5 properties: { v: 1, key: { id.cusip: 1 }, name: "id.cusip_1", ns: "trancheinfodb_20160602.ti_5" } 2016-06-02T13:21:25.571-0400 I INDEX [repl writer worker 11] building index using bulk method 2016-06-02T13:21:25.617-0400 I INDEX [repl writer worker 11] build index done. scanned 7135 total records. 0 secs 2016-06-02T13:37:12.891-0400 I REPL [ReplicationExecutor] could not find member to sync from 2016-06-02T13:37:15.431-0400 I REPL [ReplicationExecutor] Error in heartbeat request to bxb-dev-oas003.public.oas.intdata.com:29111; ExceededTimeLimit: Couldn't get a connection within the time limit 2016-06-02T13:37:17.891-0400 I REPL [ReplicationExecutor] Member bxb-dev-oas017.public.oas.intdata.com:29112 is now in state PRIMARY 2016-06-02T13:37:17.891-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T13:37:17.932-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T13:37:30.431-0400 I REPL [ReplicationExecutor] Error in heartbeat request to bxb-dev-oas003.public.oas.intdata.com:29111; ExceededTimeLimit: Couldn't get a connection within the time limit 2016-06-02T13:37:30.933-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T13:37:32.431-0400 I REPL [ReplicationExecutor] Member bxb-dev-oas003.public.oas.intdata.com:29111 is now in state SECONDARY 2016-06-02T13:38:08.434-0400 I ASIO [ReplicationExecutor] dropping unhealthy pooled connection to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T13:38:08.434-0400 I ASIO [ReplicationExecutor] after drop, pool was empty, going to spawn some connections 2016-06-02T13:38:08.454-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T13:38:12.814-0400 I REPL [ReplicationExecutor] could not find member to sync from 2016-06-02T13:38:12.814-0400 I ASIO [ReplicationExecutor] dropping unhealthy pooled connection to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T13:38:12.814-0400 I ASIO [ReplicationExecutor] after drop, pool was empty, going to spawn some connections 2016-06-02T13:38:12.855-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T13:38:12.855-0400 I REPL [ReplicationExecutor] Member bxb-dev-oas017.public.oas.intdata.com:29112 is now in state SECONDARY 2016-06-02T13:38:17.816-0400 I REPL [ReplicationExecutor] Member bxb-dev-oas003.public.oas.intdata.com:29111 is now in state PRIMARY 2016-06-02T13:38:18.815-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T13:38:18.849-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T13:38:18.849-0400 I ASIO [rsBackgroundSync] dropping unhealthy pooled connection to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T13:38:18.849-0400 I ASIO [rsBackgroundSync] after drop, pool was empty, going to spawn some connections 2016-06-02T13:38:18.869-0400 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T13:52:17.636-0400 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:12:47.629-0400 I REPL [ReplicationExecutor] could not find member to sync from 2016-06-02T14:12:57.193-0400 I REPL [ReplicationExecutor] Error in heartbeat request to bxb-dev-oas003.public.oas.intdata.com:29111; ExceededTimeLimit: Couldn't get a connection within the time limit 2016-06-02T14:12:57.630-0400 I REPL [ReplicationExecutor] Member bxb-dev-oas017.public.oas.intdata.com:29112 is now in state PRIMARY 2016-06-02T14:12:57.630-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:12:57.656-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:13:12.193-0400 I REPL [ReplicationExecutor] Error in heartbeat request to bxb-dev-oas003.public.oas.intdata.com:29111; ExceededTimeLimit: Couldn't get a connection within the time limit 2016-06-02T14:13:24.193-0400 I REPL [ReplicationExecutor] Error in heartbeat request to bxb-dev-oas003.public.oas.intdata.com:29111; ExceededTimeLimit: Couldn't get a connection within the time limit 2016-06-02T14:13:36.194-0400 I REPL [ReplicationExecutor] Error in heartbeat request to bxb-dev-oas003.public.oas.intdata.com:29111; ExceededTimeLimit: Couldn't get a connection within the time limit 2016-06-02T14:13:48.194-0400 I REPL [ReplicationExecutor] Error in heartbeat request to bxb-dev-oas003.public.oas.intdata.com:29111; ExceededTimeLimit: Couldn't get a connection within the time limit 2016-06-02T14:14:00.194-0400 I REPL [ReplicationExecutor] Error in heartbeat request to bxb-dev-oas003.public.oas.intdata.com:29111; ExceededTimeLimit: Couldn't get a connection within the time limit 2016-06-02T14:14:12.194-0400 I REPL [ReplicationExecutor] Error in heartbeat request to bxb-dev-oas003.public.oas.intdata.com:29111; ExceededTimeLimit: Couldn't get a connection within the time limit 2016-06-02T14:14:14.792-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:14.795-0400 I REPL [ReplicationExecutor] Member bxb-dev-oas003.public.oas.intdata.com:29111 is now in state PRIMARY 2016-06-02T14:14:16.795-0400 I ASIO [ReplicationExecutor] dropping unhealthy pooled connection to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:16.795-0400 I ASIO [ReplicationExecutor] after drop, pool was empty, going to spawn some connections 2016-06-02T14:14:16.813-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:16.818-0400 I REPL [ReplicationExecutor] Member bxb-dev-oas003.public.oas.intdata.com:29111 is now in state SECONDARY 2016-06-02T14:14:18.824-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:18.824-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:18.875-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:18.894-0400 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:18.910-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:18.923-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:18.924-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:18.924-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:18.947-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:19.023-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:19.024-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:19.024-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:19.024-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:19.024-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:19.052-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:19.052-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:19.052-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:19.052-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:19.052-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:19.077-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:19.077-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:19.085-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:19.085-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:19.085-0400 I NETWORK [SyncSourceFeedback] Socket say send() errno:9 Bad file descriptor 10.170.49.89:29111 2016-06-02T14:14:19.095-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:19.095-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:19.101-0400 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update: socket exception [SEND_ERROR] for 10.170.49.89:29111 2016-06-02T14:14:19.101-0400 I REPL [SyncSourceFeedback] updateUpstream failed: Location9001: socket exception [SEND_ERROR] for 10.170.49.89:29111, will retry 2016-06-02T14:14:19.119-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:19.130-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:24.123-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.151-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.159-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:24.159-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:24.159-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:24.159-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:24.159-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:24.161-0400 E NETWORK [SyncSourceFeedback] disableNagle failed: errno:9 Bad file descriptor 2016-06-02T14:14:24.161-0400 E NETWORK [SyncSourceFeedback] SO_KEEPALIVE failed: errno:9 Bad file descriptor 2016-06-02T14:14:24.161-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPIDLE: errno:9 Bad file descriptor 2016-06-02T14:14:24.161-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPINTVL: errno:9 Bad file descriptor 2016-06-02T14:14:24.161-0400 W NETWORK [SyncSourceFeedback] Could not resolve local address for socket with fd -1: Unknown error 2016-06-02T14:14:24.161-0400 I NETWORK [SyncSourceFeedback] Socket say send() errno:9 Bad file descriptor 10.170.49.89:29111 2016-06-02T14:14:24.161-0400 I REPL [SyncSourceFeedback] socket exception [SEND_ERROR] for 10.170.49.89:29111 2016-06-02T14:14:24.200-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:24.203-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:24.204-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:24.204-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:24.204-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:24.214-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:24.214-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:24.214-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:24.214-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:24.214-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:24.228-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:24.228-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:24.228-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:24.228-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.228-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:24.238-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:24.238-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:24.267-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:24.270-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:24.270-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.271-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:24.304-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:24.304-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:24.304-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:24.304-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:24.304-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:24.304-0400 I REPL [SyncSourceFeedback] network error while attempting to run command 'isMaster' on host 'bxb-dev-oas017.public.oas.intdata.com:29112' 2016-06-02T14:14:24.304-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.329-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:24.332-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:24.332-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:24.332-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:24.333-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:24.347-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:24.348-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:24.348-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:24.348-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:24.348-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:24.358-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:24.358-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:24.359-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:24.359-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:24.362-0400 I REPL [WTJournalFlusher] My durable progress ((term: 72, timestamp: Jun 2 14:13:43:9)) is ahead of my applied progress ((term: 71, timestamp: Jun 2 14:12:37:1). This is likely due to a rollback. slaveInfo: { id: 2, rid: ObjectId('57431fe2f1f17b0bf2d8a64d'), host: "bxb-dev-oas013.public.oas.intdata.com:29113", lastDurableOpTime: { ts: Timestamp 1464891157000|1, t: 71 }, lastAppliedOpTime: { ts: Timestamp 1464891157000|1, t: 71 }, self: true, lastUpdated: new Date(1464891264359) } 2016-06-02T14:14:24.369-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:24.369-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:24.395-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:24.397-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:24.398-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.407-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:24.443-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.451-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:24.451-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:24.451-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:24.451-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:24.451-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:24.462-0400 E NETWORK [SyncSourceFeedback] disableNagle failed: errno:9 Bad file descriptor 2016-06-02T14:14:24.462-0400 E NETWORK [SyncSourceFeedback] SO_KEEPALIVE failed: errno:9 Bad file descriptor 2016-06-02T14:14:24.462-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPIDLE: errno:9 Bad file descriptor 2016-06-02T14:14:24.462-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPINTVL: errno:9 Bad file descriptor 2016-06-02T14:14:24.462-0400 W NETWORK [SyncSourceFeedback] Could not resolve local address for socket with fd -1: Unknown error 2016-06-02T14:14:24.462-0400 I NETWORK [SyncSourceFeedback] Socket say send() errno:9 Bad file descriptor 10.170.49.89:29111 2016-06-02T14:14:24.462-0400 I REPL [SyncSourceFeedback] socket exception [SEND_ERROR] for 10.170.49.89:29111 2016-06-02T14:14:24.467-0400 W NETWORK [rsBackgroundSync] Failed to connect to 10.170.49.89:29111, reason: errno:106 Transport endpoint is already connected 2016-06-02T14:14:24.467-0400 E REPL [rsBackgroundSync] sync producer problem: 6 couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:24.567-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.612-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.619-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:24.619-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:24.619-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:24.620-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:24.651-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:24.654-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:24.654-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:24.654-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:24.654-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:24.679-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:24.679-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:24.679-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:24.679-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:24.679-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:24.690-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:24.690-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:24.691-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:24.691-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:24.701-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:24.701-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:24.725-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:24.733-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:24.733-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.734-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:24.757-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.764-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:24.765-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:24.765-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:24.765-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:24.765-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:24.774-0400 W NETWORK [SyncSourceFeedback] Failed to connect to 10.170.49.89:29111, reason: errno:9 Bad file descriptor 2016-06-02T14:14:24.774-0400 I REPL [SyncSourceFeedback] couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:24.791-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:24.796-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:24.796-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:24.796-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:24.796-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:24.815-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:24.815-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:24.815-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:24.815-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:24.815-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:24.826-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:24.826-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:24.826-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:24.826-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.826-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:24.837-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:24.837-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:24.876-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:24.878-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:24.878-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.879-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:24.919-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.926-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:24.927-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:24.927-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:24.927-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:24.927-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:24.940-0400 W NETWORK [SyncSourceFeedback] Failed to connect to 10.170.49.89:29111, reason: errno:9 Bad file descriptor 2016-06-02T14:14:24.940-0400 I REPL [SyncSourceFeedback] couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:24.965-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:24.969-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:24.970-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:24.970-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:24.970-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:24.979-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:24.979-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:24.979-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:24.979-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:24.979-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:24.990-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:24.990-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:24.990-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:24.990-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:24.990-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:25.001-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:25.003-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:25.043-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:25.043-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.083-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.091-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:25.096-0400 I REPL [rsBackgroundSync] Waiting for all operations from (term: 71, timestamp: Jun 2 14:12:37:1) until (term: 72, timestamp: Jun 2 14:13:43:9) to be applied before starting rollback. 2016-06-02T14:14:25.141-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:25.147-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:25.147-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:25.147-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:25.147-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:25.182-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:25.187-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:25.187-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:25.187-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:25.187-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:25.200-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:25.201-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:25.201-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:25.201-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:25.201-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:25.211-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:25.211-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:25.211-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:25.211-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:25.222-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:25.222-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:25.244-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:25.246-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:25.246-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.247-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:25.284-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.292-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:25.292-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:25.292-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:25.292-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:25.292-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:25.306-0400 E NETWORK [SyncSourceFeedback] disableNagle failed: errno:9 Bad file descriptor 2016-06-02T14:14:25.306-0400 E NETWORK [SyncSourceFeedback] SO_KEEPALIVE failed: errno:9 Bad file descriptor 2016-06-02T14:14:25.306-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPIDLE: errno:9 Bad file descriptor 2016-06-02T14:14:25.306-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPINTVL: errno:9 Bad file descriptor 2016-06-02T14:14:25.306-0400 W NETWORK [SyncSourceFeedback] Could not resolve local address for socket with fd -1: Unknown error 2016-06-02T14:14:25.306-0400 I NETWORK [SyncSourceFeedback] Socket say send() errno:9 Bad file descriptor 10.170.49.89:29111 2016-06-02T14:14:25.306-0400 I REPL [SyncSourceFeedback] socket exception [SEND_ERROR] for 10.170.49.89:29111 2016-06-02T14:14:25.307-0400 W NETWORK [rsBackgroundSync] Failed to connect to 10.170.49.89:29111, reason: errno:106 Transport endpoint is already connected 2016-06-02T14:14:25.308-0400 E REPL [rsBackgroundSync] sync producer problem: 6 couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:25.408-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.454-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.462-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:25.462-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:25.462-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:25.462-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:25.499-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:25.501-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:25.501-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:25.501-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:25.501-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:25.519-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:25.519-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:25.519-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:25.519-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:25.519-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:25.539-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:25.539-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:25.539-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:25.539-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:25.549-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:25.550-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:25.592-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:25.594-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:25.595-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.595-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:25.638-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.645-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:25.645-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:25.645-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:25.645-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:25.646-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:25.650-0400 W NETWORK [SyncSourceFeedback] Failed to connect to 10.170.49.89:29111, reason: errno:9 Bad file descriptor 2016-06-02T14:14:25.650-0400 I REPL [SyncSourceFeedback] couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:25.677-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:25.679-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:25.679-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:25.679-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:25.679-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:25.703-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:25.704-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:25.704-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:25.704-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:25.704-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:25.715-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:25.715-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:25.715-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:25.715-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:25.715-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.725-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:25.725-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:25.744-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:25.747-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:25.747-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.747-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:25.777-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.785-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:25.785-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:25.785-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:25.785-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:25.785-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:25.797-0400 E NETWORK [SyncSourceFeedback] disableNagle failed: errno:9 Bad file descriptor 2016-06-02T14:14:25.797-0400 E NETWORK [SyncSourceFeedback] SO_KEEPALIVE failed: errno:9 Bad file descriptor 2016-06-02T14:14:25.797-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPIDLE: errno:9 Bad file descriptor 2016-06-02T14:14:25.797-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPINTVL: errno:9 Bad file descriptor 2016-06-02T14:14:25.797-0400 W NETWORK [SyncSourceFeedback] Could not resolve local address for socket with fd -1: Unknown error 2016-06-02T14:14:25.797-0400 I NETWORK [SyncSourceFeedback] Socket say send() errno:9 Bad file descriptor 10.170.49.89:29111 2016-06-02T14:14:25.797-0400 I REPL [SyncSourceFeedback] socket exception [SEND_ERROR] for 10.170.49.89:29111 2016-06-02T14:14:25.801-0400 W NETWORK [rsBackgroundSync] Failed to connect to 10.170.49.89:29111, reason: errno:106 Transport endpoint is already connected 2016-06-02T14:14:25.801-0400 E REPL [rsBackgroundSync] sync producer problem: 6 couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:25.901-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.985-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:25.994-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:25.994-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:25.994-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:25.994-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:26.036-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:26.040-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:26.040-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:26.040-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:26.040-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:26.061-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:26.061-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:26.061-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:26.061-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:26.061-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:26.072-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:26.072-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:26.072-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:26.072-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:26.074-0400 I REPL [WTJournalFlusher] My durable progress ((term: 72, timestamp: Jun 2 14:13:43:9)) is ahead of my applied progress ((term: 71, timestamp: Jun 2 14:12:37:1). This is likely due to a rollback. slaveInfo: { id: 2, rid: ObjectId('57431fe2f1f17b0bf2d8a64d'), host: "bxb-dev-oas013.public.oas.intdata.com:29113", lastDurableOpTime: { ts: Timestamp 1464891157000|1, t: 71 }, lastAppliedOpTime: { ts: Timestamp 1464891157000|1, t: 71 }, self: true, lastUpdated: new Date(1464891266072) } 2016-06-02T14:14:26.082-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:26.083-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:26.105-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:26.108-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:26.108-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:26.108-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:26.154-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:26.162-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:26.162-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:26.162-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:26.162-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:26.164-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:26.165-0400 E NETWORK [SyncSourceFeedback] disableNagle failed: errno:9 Bad file descriptor 2016-06-02T14:14:26.165-0400 E NETWORK [SyncSourceFeedback] SO_KEEPALIVE failed: errno:9 Bad file descriptor 2016-06-02T14:14:26.165-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPIDLE: errno:9 Bad file descriptor 2016-06-02T14:14:26.165-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPINTVL: errno:9 Bad file descriptor 2016-06-02T14:14:26.165-0400 W NETWORK [SyncSourceFeedback] Could not resolve local address for socket with fd -1: Unknown error 2016-06-02T14:14:26.165-0400 I NETWORK [SyncSourceFeedback] Socket say send() errno:9 Bad file descriptor 10.170.49.89:29111 2016-06-02T14:14:26.165-0400 I REPL [SyncSourceFeedback] socket exception [SEND_ERROR] for 10.170.49.89:29111 2016-06-02T14:14:26.192-0400 W NETWORK [rsBackgroundSync] Failed to connect to 10.170.49.89:29111, reason: errno:106 Transport endpoint is already connected 2016-06-02T14:14:26.192-0400 E REPL [rsBackgroundSync] sync producer problem: 6 couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:26.292-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:26.334-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:26.343-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:26.344-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:26.344-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:26.344-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:26.402-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:26.406-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:26.407-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:26.407-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:26.407-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:26.425-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:26.426-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:26.426-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:26.426-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:26.426-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:26.438-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:26.438-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:26.438-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:26.438-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:26.439-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:26.439-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:26.472-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:26.475-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:26.475-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:26.476-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:26.521-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:26.534-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:26.534-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:26.534-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:26.534-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:26.535-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:26.536-0400 W NETWORK [SyncSourceFeedback] Failed to connect to 10.170.49.89:29111, reason: errno:9 Bad file descriptor 2016-06-02T14:14:26.536-0400 I REPL [SyncSourceFeedback] couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:26.572-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:26.576-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:26.576-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:26.576-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:26.577-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:26.589-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:26.589-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:26.589-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:26.589-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:26.589-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:26.601-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:26.601-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:26.601-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:26.601-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:26.601-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:26.611-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:26.611-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:26.657-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:26.660-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:26.660-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:26.660-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:26.711-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:26.719-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:26.719-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:26.719-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:26.719-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:26.719-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:26.730-0400 E NETWORK [SyncSourceFeedback] disableNagle failed: errno:9 Bad file descriptor 2016-06-02T14:14:26.730-0400 E NETWORK [SyncSourceFeedback] SO_KEEPALIVE failed: errno:9 Bad file descriptor 2016-06-02T14:14:26.730-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPIDLE: errno:9 Bad file descriptor 2016-06-02T14:14:26.730-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPINTVL: errno:9 Bad file descriptor 2016-06-02T14:14:26.730-0400 W NETWORK [SyncSourceFeedback] Could not resolve local address for socket with fd -1: Unknown error 2016-06-02T14:14:26.730-0400 I NETWORK [SyncSourceFeedback] Socket say send() errno:9 Bad file descriptor 10.170.49.89:29111 2016-06-02T14:14:26.730-0400 I REPL [SyncSourceFeedback] socket exception [SEND_ERROR] for 10.170.49.89:29111 2016-06-02T14:14:26.772-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:26.776-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:26.776-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:26.776-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:26.777-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:26.795-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:26.795-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:26.795-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:26.795-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:26.795-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:26.806-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:26.806-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:26.806-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:26.806-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:26.806-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:26.816-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:26.816-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:26.850-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:26.853-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:26.853-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:26.853-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:26.883-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:26.890-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:26.890-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:26.890-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:26.890-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:26.891-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:26.893-0400 E NETWORK [SyncSourceFeedback] disableNagle failed: errno:9 Bad file descriptor 2016-06-02T14:14:26.893-0400 E NETWORK [SyncSourceFeedback] SO_KEEPALIVE failed: errno:9 Bad file descriptor 2016-06-02T14:14:26.893-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPIDLE: errno:9 Bad file descriptor 2016-06-02T14:14:26.893-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPINTVL: errno:9 Bad file descriptor 2016-06-02T14:14:26.893-0400 W NETWORK [SyncSourceFeedback] Could not resolve local address for socket with fd -1: Unknown error 2016-06-02T14:14:26.893-0400 I NETWORK [SyncSourceFeedback] Socket say send() errno:9 Bad file descriptor 10.170.49.89:29111 2016-06-02T14:14:26.893-0400 I REPL [SyncSourceFeedback] socket exception [SEND_ERROR] for 10.170.49.89:29111 2016-06-02T14:14:26.899-0400 W NETWORK [rsBackgroundSync] Failed to connect to 10.170.49.89:29111, reason: errno:106 Transport endpoint is already connected 2016-06-02T14:14:26.899-0400 E REPL [rsBackgroundSync] sync producer problem: 6 couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:26.999-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.037-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.045-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:27.045-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:27.045-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:27.045-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:27.098-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:27.099-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:27.099-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:27.099-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:27.100-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:27.113-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:27.113-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:27.113-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:27.113-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:27.113-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:27.129-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:27.129-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:27.129-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:27.129-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:27.139-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:27.139-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:27.172-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:27.174-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:27.174-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:27.174-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.218-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.225-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:27.225-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:27.225-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:27.225-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:27.226-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:27.240-0400 E NETWORK [SyncSourceFeedback] disableNagle failed: errno:9 Bad file descriptor 2016-06-02T14:14:27.240-0400 E NETWORK [SyncSourceFeedback] SO_KEEPALIVE failed: errno:9 Bad file descriptor 2016-06-02T14:14:27.240-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPIDLE: errno:9 Bad file descriptor 2016-06-02T14:14:27.240-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPINTVL: errno:9 Bad file descriptor 2016-06-02T14:14:27.240-0400 W NETWORK [SyncSourceFeedback] Could not resolve local address for socket with fd -1: Unknown error 2016-06-02T14:14:27.240-0400 I NETWORK [SyncSourceFeedback] Socket say send() errno:9 Bad file descriptor 10.170.49.89:29111 2016-06-02T14:14:27.240-0400 I REPL [SyncSourceFeedback] socket exception [SEND_ERROR] for 10.170.49.89:29111 2016-06-02T14:14:27.242-0400 W NETWORK [rsBackgroundSync] Failed to connect to 10.170.49.89:29111, reason: errno:106 Transport endpoint is already connected 2016-06-02T14:14:27.242-0400 E REPL [rsBackgroundSync] sync producer problem: 6 couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:27.343-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.376-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.387-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:27.387-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:27.387-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:27.387-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:27.420-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:27.423-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:27.423-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:27.423-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:27.424-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:27.436-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:27.436-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:27.436-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:27.436-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:27.436-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:27.458-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:27.458-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:27.458-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:27.458-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:27.468-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:27.468-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:27.509-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:27.512-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:27.512-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:27.512-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.560-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.568-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:27.568-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:27.568-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:27.568-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:27.570-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:27.578-0400 W NETWORK [SyncSourceFeedback] Failed to connect to 10.170.49.89:29111, reason: errno:9 Bad file descriptor 2016-06-02T14:14:27.578-0400 I REPL [SyncSourceFeedback] couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:27.613-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:27.616-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:27.616-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:27.616-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:27.616-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:27.631-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:27.631-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:27.631-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:27.631-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:27.631-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:27.642-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:27.642-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:27.643-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:27.643-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.643-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:27.653-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:27.653-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:27.680-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:27.683-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:27.683-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.683-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:27.723-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.743-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:27.743-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:27.743-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:27.743-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:27.745-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:27.754-0400 I REPL [SyncSourceFeedback] network error while attempting to run command 'isMaster' on host 'bxb-dev-oas003.public.oas.intdata.com:29111' 2016-06-02T14:14:27.762-0400 I NETWORK [conn8557] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50386] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8564] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50393] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8558] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50387] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8563] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50392] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8550] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50379] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8545] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50374] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8543] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50372] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8544] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50373] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8549] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50378] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8562] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50391] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8559] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50388] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8556] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50385] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8553] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50382] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8565] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50394] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8569] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50398] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8548] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50377] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8546] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50375] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8568] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50397] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8567] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50396] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8542] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50371] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8560] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50389] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8547] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50376] 2016-06-02T14:14:27.762-0400 I NETWORK [conn8551] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50380] 2016-06-02T14:14:27.766-0400 I NETWORK [conn8570] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.170.49.99:50399] 2016-06-02T14:14:27.788-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:27.789-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:27.789-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:27.789-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:27.789-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:27.798-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:27.798-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:27.798-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:27.798-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:27.798-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:27.809-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:27.809-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:27.809-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:27.809-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:27.809-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.819-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:27.819-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:27.855-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:27.857-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:27.857-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.863-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:27.896-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:27.903-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:27.903-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:27.904-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:27.904-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:27.904-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:27.912-0400 E NETWORK [SyncSourceFeedback] disableNagle failed: errno:9 Bad file descriptor 2016-06-02T14:14:27.912-0400 E NETWORK [SyncSourceFeedback] SO_KEEPALIVE failed: errno:9 Bad file descriptor 2016-06-02T14:14:27.912-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPIDLE: errno:9 Bad file descriptor 2016-06-02T14:14:27.912-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPINTVL: errno:9 Bad file descriptor 2016-06-02T14:14:27.912-0400 W NETWORK [SyncSourceFeedback] Could not resolve local address for socket with fd -1: Unknown error 2016-06-02T14:14:27.912-0400 I NETWORK [SyncSourceFeedback] Socket say send() errno:9 Bad file descriptor 10.170.49.89:29111 2016-06-02T14:14:27.912-0400 I REPL [SyncSourceFeedback] socket exception [SEND_ERROR] for 10.170.49.89:29111 2016-06-02T14:14:27.929-0400 W NETWORK [rsBackgroundSync] Failed to connect to 10.170.49.89:29111, reason: errno:106 Transport endpoint is already connected 2016-06-02T14:14:27.929-0400 E REPL [rsBackgroundSync] sync producer problem: 6 couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:28.029-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.057-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.064-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:28.064-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:28.064-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:28.064-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:28.089-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:28.090-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:28.090-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:28.090-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:28.090-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:28.107-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:28.107-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:28.107-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:28.107-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:28.107-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:28.118-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:28.118-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:28.118-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:28.118-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:28.128-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:28.128-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:28.155-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:28.157-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:28.157-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.158-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:28.235-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.239-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:28.239-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:28.239-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:28.239-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:28.240-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:28.246-0400 W NETWORK [SyncSourceFeedback] Failed to connect to 10.170.49.89:29111, reason: errno:9 Bad file descriptor 2016-06-02T14:14:28.247-0400 I REPL [SyncSourceFeedback] couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:28.271-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:28.275-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:28.275-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:28.275-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:28.276-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:28.288-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:28.288-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:28.288-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:28.288-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:28.288-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:28.299-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:28.299-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:28.299-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:28.299-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.299-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:28.309-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:28.310-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:28.331-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:28.331-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:28.331-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.367-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.374-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:28.374-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:28.374-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:28.374-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:28.375-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:28.381-0400 E NETWORK [SyncSourceFeedback] disableNagle failed: errno:9 Bad file descriptor 2016-06-02T14:14:28.381-0400 E NETWORK [SyncSourceFeedback] SO_KEEPALIVE failed: errno:9 Bad file descriptor 2016-06-02T14:14:28.381-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPIDLE: errno:9 Bad file descriptor 2016-06-02T14:14:28.381-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPINTVL: errno:9 Bad file descriptor 2016-06-02T14:14:28.381-0400 W NETWORK [SyncSourceFeedback] Could not resolve local address for socket with fd -1: Unknown error 2016-06-02T14:14:28.381-0400 I NETWORK [SyncSourceFeedback] Socket say send() errno:9 Bad file descriptor 10.170.49.89:29111 2016-06-02T14:14:28.381-0400 I REPL [SyncSourceFeedback] socket exception [SEND_ERROR] for 10.170.49.89:29111 2016-06-02T14:14:28.384-0400 W NETWORK [rsBackgroundSync] Failed to connect to 10.170.49.89:29111, reason: errno:106 Transport endpoint is already connected 2016-06-02T14:14:28.385-0400 E REPL [rsBackgroundSync] sync producer problem: 6 couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:28.485-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.511-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.519-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:28.519-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:28.519-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:28.519-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:28.554-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:28.555-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:28.555-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:28.555-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:28.555-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:28.575-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:28.575-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:28.575-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:28.575-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:28.576-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:28.586-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:28.586-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:28.587-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:28.587-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:28.597-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:28.597-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:28.619-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:28.622-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:28.622-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.630-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:28.659-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:28.659-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:28.659-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:28.659-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:28.659-0400 I REPL [SyncSourceFeedback] network error while attempting to run command 'isMaster' on host 'bxb-dev-oas017.public.oas.intdata.com:29112' 2016-06-02T14:14:28.659-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.659-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:28.693-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:28.695-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:28.695-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:28.695-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:28.695-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:28.717-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:28.717-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:28.717-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:28.717-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:28.717-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:28.728-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:28.728-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:28.728-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:28.728-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:28.738-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:28.738-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:28.820-0400 I REPL [ReplicationExecutor] Member bxb-dev-oas003.public.oas.intdata.com:29111 is now in state PRIMARY 2016-06-02T14:14:28.858-0400 I ASIO [rsBackgroundSync] dropping unhealthy pooled connection to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:28.858-0400 I ASIO [rsBackgroundSync] dropping unhealthy pooled connection to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:28.858-0400 I ASIO [rsBackgroundSync] after drop, pool was empty, going to spawn some connections 2016-06-02T14:14:28.858-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:28.879-0400 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:28.879-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:28.924-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:28.924-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.930-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:28.958-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:28.965-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:28.965-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:28.965-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:28.965-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:28.965-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:28.986-0400 W NETWORK [SyncSourceFeedback] Failed to connect to 10.170.49.89:29111, reason: errno:106 Transport endpoint is already connected 2016-06-02T14:14:28.986-0400 I REPL [SyncSourceFeedback] couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:28.996-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:29.000-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:29.000-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:16:b 2016-06-02T14:14:29.000-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -33 seconds 2016-06-02T14:14:29.001-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:29.010-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:29.010-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:29.010-0400 I REPL [rsBackgroundSync] minvalid=(term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:29.010-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:29.010-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:29.023-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:29.023-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:29.023-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:29.023-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:29.023-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:29.033-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:29.033-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:29.057-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:29.057-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:29.058-0400 I REPL [ReplicationExecutor] transition to SECONDARY 2016-06-02T14:14:29.110-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:29.117-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:29.117-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:29.117-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:29.117-0400 I REPL [ReplicationExecutor] transition to ROLLBACK 2016-06-02T14:14:29.117-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:29.129-0400 E NETWORK [SyncSourceFeedback] disableNagle failed: errno:9 Bad file descriptor 2016-06-02T14:14:29.129-0400 E NETWORK [SyncSourceFeedback] SO_KEEPALIVE failed: errno:9 Bad file descriptor 2016-06-02T14:14:29.129-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPIDLE: errno:9 Bad file descriptor 2016-06-02T14:14:29.129-0400 E NETWORK [SyncSourceFeedback] can't get TCP_KEEPINTVL: errno:9 Bad file descriptor 2016-06-02T14:14:29.129-0400 W NETWORK [SyncSourceFeedback] Could not resolve local address for socket with fd -1: Unknown error 2016-06-02T14:14:29.129-0400 I NETWORK [SyncSourceFeedback] Socket say send() errno:9 Bad file descriptor 10.170.49.89:29111 2016-06-02T14:14:29.129-0400 I REPL [SyncSourceFeedback] socket exception [SEND_ERROR] for 10.170.49.89:29111 2016-06-02T14:14:29.137-0400 W NETWORK [rsBackgroundSync] Failed to connect to 10.170.49.89:29111, reason: errno:106 Transport endpoint is already connected 2016-06-02T14:14:29.137-0400 E REPL [rsBackgroundSync] sync producer problem: 6 couldn't connect to server bxb-dev-oas003.public.oas.intdata.com:29111, connection attempt failed 2016-06-02T14:14:29.237-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:29.270-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:29.277-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:29.277-0400 I REPL [rsBackgroundSync] beginning rollback 2016-06-02T14:14:29.277-0400 I REPL [rsBackgroundSync] rollback 0 2016-06-02T14:14:29.277-0400 I REPL [rsBackgroundSync] rollback 1 2016-06-02T14:14:29.320-0400 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint 2016-06-02T14:14:29.324-0400 I REPL [rsBackgroundSync] rollback our last optime: Jun 2 14:13:43:9 2016-06-02T14:14:29.324-0400 I REPL [rsBackgroundSync] rollback their last optime: Jun 2 14:14:29:1 2016-06-02T14:14:29.324-0400 I REPL [rsBackgroundSync] rollback diff in end of log times: -46 seconds 2016-06-02T14:14:29.325-0400 I REPL [rsBackgroundSync] rollback 3 fixup 2016-06-02T14:14:29.333-0400 I REPL [rsBackgroundSync] rollback 3.5 2016-06-02T14:14:29.333-0400 I REPL [rsBackgroundSync] rollback 4 n:1 2016-06-02T14:14:29.333-0400 I REPL [rsBackgroundSync] minvalid=(term: 73, timestamp: Jun 2 14:14:29:1) 2016-06-02T14:14:29.334-0400 I REPL [rsBackgroundSync] rollback 4.6 2016-06-02T14:14:29.334-0400 I REPL [rsBackgroundSync] rollback 4.7 2016-06-02T14:14:29.345-0400 I REPL [rsBackgroundSync] rollback 5 d:0 u:9 2016-06-02T14:14:29.345-0400 I REPL [rsBackgroundSync] rollback 6 2016-06-02T14:14:29.345-0400 I REPL [rsBackgroundSync] rollback done 2016-06-02T14:14:29.345-0400 I REPL [rsBackgroundSync] rollback finished 2016-06-02T14:14:29.355-0400 I REPL [ReplicationExecutor] transition to RECOVERING 2016-06-02T14:14:29.355-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:29.392-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas017.public.oas.intdata.com:29112 2016-06-02T14:14:29.395-0400 I REPL [ReplicationExecutor] re-evaluating sync source because our current sync source's most recent OpTime is (term: 72, timestamp: Jun 2 14:13:43:9) which is more than 30s behind member bxb-dev-oas003.public.oas.intdata.com:29111 whose most recent OpTime is (term: 71, timestamp: Jun 2 14:14:16:b) 2016-06-02T14:14:29.395-0400 I REPL [ReplicationExecutor] syncing from: bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:29.444-0400 I REPL [SyncSourceFeedback] setting syncSourceFeedback to bxb-dev-oas003.public.oas.intdata.com:29111 2016-06-02T14:14:29.452-0400 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 72, timestamp: Jun 2 14:13:43:9). source's GTE: (term: 71, timestamp: Jun 2 14:14:14:1) hashes: (-5715498605584477543/1286036717366498379) 2016-06-02T14:14:29.460-0400 I - [rsBackgroundSync] Fatal assertion 18750 UnrecoverableRollbackError: need to rollback, but in inconsistent state. minvalid: (term: 73, timestamp: Jun 2 14:14:29:1) > our last optime: (term: 72, timestamp: Jun 2 14:13:43:9) 2016-06-02T14:14:29.460-0400 I - [rsBackgroundSync] ***aborting after fassert() failure