2016-02-05T03:20:37.013+0000 I CONTROL [initandlisten] MongoDB starting : pid=32323 port=27017 dbpath=/mnt/mongodb 64-bit host=ip-10-232-151-100 2016-02-05T03:20:37.013+0000 I CONTROL [initandlisten] db version v3.2.1 2016-02-05T03:20:37.013+0000 I CONTROL [initandlisten] git version: a14d55980c2cdc565d4704a7e3ad37e4e535c1b2 2016-02-05T03:20:37.013+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010 2016-02-05T03:20:37.013+0000 I CONTROL [initandlisten] allocator: tcmalloc 2016-02-05T03:20:37.013+0000 I CONTROL [initandlisten] modules: none 2016-02-05T03:20:37.013+0000 I CONTROL [initandlisten] build environment: 2016-02-05T03:20:37.013+0000 I CONTROL [initandlisten] distmod: amazon 2016-02-05T03:20:37.013+0000 I CONTROL [initandlisten] distarch: x86_64 2016-02-05T03:20:37.013+0000 I CONTROL [initandlisten] target_arch: x86_64 2016-02-05T03:20:37.013+0000 I CONTROL [initandlisten] options: { processManagement: { fork: true }, replication: { oplogSizeMB: 90000, replSet: "rsMesh2" }, storage: { dbPath: "/mnt/mongodb", engine: "wiredTiger", wiredTiger: { engineConfig: { cacheSizeGB: 24 } } }, systemLog: { destination: "file", path: "/mnt/mongodb/mongo.log" } } 2016-02-05T03:20:37.034+0000 W - [initandlisten] Detected unclean shutdown - /mnt/mongodb/mongod.lock is not empty. 2016-02-05T03:20:37.034+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2016-02-05T03:20:37.034+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=24G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2016-02-05T03:20:37.296+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2016-02-05T03:20:37.296+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 239664823 records totaling to 94639593280 bytes 2016-02-05T03:20:37.297+0000 I STORAGE [initandlisten] Sampling from the oplog between Feb 3 05:35:07:5da and Feb 5 01:56:40:9 to determine where to place markers for truncation 2016-02-05T03:20:37.297+0000 I STORAGE [initandlisten] Taking 1002 samples and assuming that each section of oplog contains approximately 2389868 records totaling to 943718534 bytes 2016-02-05T03:20:38.559+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 06:22:44:e6d 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 07:15:30:28 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 07:49:34:ef 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 08:26:14:943 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 08:39:58:3c0 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 09:12:34:10c 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 09:57:11:20a 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 10:33:33:1af 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 11:14:02:859 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 11:50:54:ab 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 12:18:26:130a 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 12:44:01:215 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 13:10:40:39c 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 13:35:31:91d 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 14:01:31:161 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 14:26:01:354 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 14:48:51:338 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 15:13:10:3df 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 15:37:02:222 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 15:59:04:168 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 16:20:21:636 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 16:43:24:11c 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 17:45:19:400 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 18:03:44:84e 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 18:20:05:1359 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 18:33:52:15e 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 18:51:35:2097 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 19:05:37:96c 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 19:20:12:383 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 19:33:50:9f7 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 19:45:58:c31 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 20:04:10:749 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 20:26:54:21 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 20:50:10:2b 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 21:10:11:18a 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 21:34:19:246 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 21:58:06:34c 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 22:19:54:305 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 22:45:22:458 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 23:11:23:61 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 3 23:37:24:e5 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 00:07:03:1e7 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 00:28:32:6b4 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 00:51:13:49a 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 01:17:24:16f 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 01:44:05:4c2 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 02:08:36:52b 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 02:38:30:d8 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 03:09:31:393 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 03:39:01:28a 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 04:11:39:186 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 04:49:27:299 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 05:25:04:6cd 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 06:10:13:134 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 06:40:54:16a 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 07:15:43:cb9 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 07:47:54:1ac 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 08:16:36:109 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 08:53:52:1e9 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 09:24:27:19 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 09:57:59:52d 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 10:25:25:ff 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 10:57:17:16e 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 11:23:38:24 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 11:50:14:19f 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 12:14:25:23b 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 12:30:31:44c 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 12:49:27:b73 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 13:10:37:156 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 13:31:10:8db 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 13:53:37:275 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 14:18:53:358 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 14:45:17:2b3 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 15:07:49:be3 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 15:33:23:b0 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 15:59:03:3f7 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 16:18:23:68d 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 16:42:17:408 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 17:04:41:185 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 17:24:52:38 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 17:50:35:2a1 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 18:19:05:3aa 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 18:39:32:240 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 19:07:00:98d 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 19:17:20:6c5 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 19:33:43:340 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 20:20:50:fa 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 20:49:30:190 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 21:10:07:5d6 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 21:33:40:4b8 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 21:59:24:c8 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 22:21:08:620 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 22:59:34:37e 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 23:19:36:1d1 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 4 23:55:47:4ef 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 5 00:16:28:3f 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 5 00:53:51:cb6 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 5 01:06:09:80f 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 5 01:22:57:193 2016-02-05T03:20:38.560+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 5 01:48:29:2de 2016-02-05T03:20:38.686+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/mnt/mongodb/diagnostic.data' 2016-02-05T03:20:38.686+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker 2016-02-05T03:20:38.686+0000 I NETWORK [initandlisten] waiting for connections on port 27017 2016-02-05T03:20:38.690+0000 I REPL [ReplicationExecutor] New replica set config in use: { _id: "rsMesh2", version: 26, protocolVersion: 1, members: [ { _id: 2, host: "s2r3.db.meshfire.com:27020", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 6, host: "s2r2.db.meshfire.com:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 7, host: "s2r4.db.meshfire.com:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } 2016-02-05T03:20:38.690+0000 I REPL [ReplicationExecutor] This node is s2r4.db.meshfire.com:27017 in the config 2016-02-05T03:20:38.690+0000 I REPL [ReplicationExecutor] transition to STARTUP2 2016-02-05T03:20:38.690+0000 I REPL [ReplicationExecutor] Starting replication applier threads 2016-02-05T03:20:38.690+0000 I REPL [ReplicationExecutor] transition to RECOVERING 2016-02-05T03:20:38.692+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to s2r2.db.meshfire.com:27017 2016-02-05T03:20:38.692+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to s2r3.db.meshfire.com:27020 2016-02-05T03:20:38.692+0000 I REPL [ReplicationExecutor] Member s2r2.db.meshfire.com:27017 is now in state PRIMARY 2016-02-05T03:20:38.692+0000 I REPL [ReplicationExecutor] Member s2r3.db.meshfire.com:27020 is now in state ARBITER 2016-02-05T03:20:38.753+0000 I NETWORK [initandlisten] connection accepted from 10.237.230.185:58847 #1 (1 connection now open) 2016-02-05T03:20:38.806+0000 I NETWORK [initandlisten] connection accepted from 10.237.230.185:58849 #2 (2 connections now open) 2016-02-05T03:20:39.106+0000 I NETWORK [initandlisten] connection accepted from 10.229.23.118:55395 #3 (3 connections now open) 2016-02-05T03:20:39.357+0000 I NETWORK [initandlisten] connection accepted from 10.229.23.118:55396 #4 (4 connections now open) 2016-02-05T03:20:40.162+0000 I NETWORK [initandlisten] connection accepted from 10.63.216.53:39779 #5 (5 connections now open) 2016-02-05T03:20:41.360+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:54422 #6 (6 connections now open) 2016-02-05T03:20:42.702+0000 I NETWORK [initandlisten] connection accepted from 10.16.200.60:45730 #7 (7 connections now open) 2016-02-05T03:20:42.710+0000 I NETWORK [initandlisten] connection accepted from 10.216.208.20:33267 #8 (8 connections now open) 2016-02-05T03:20:43.019+0000 I NETWORK [initandlisten] connection accepted from 10.238.22.62:41563 #9 (9 connections now open) 2016-02-05T03:20:44.691+0000 I REPL [ReplicationExecutor] syncing from: s2r2.db.meshfire.com:27017 2016-02-05T03:20:44.693+0000 I REPL [SyncSourceFeedback] setting syncSourceFeedback to s2r2.db.meshfire.com:27017 2016-02-05T03:20:44.696+0000 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to s2r2.db.meshfire.com:27017 2016-02-05T03:20:44.697+0000 I REPL [rsBackgroundSync] starting rollback: OplogStartMissing our last op time fetched: (term: 12, timestamp: Feb 5 01:56:40:9). source's GTE: (term: 13, timestamp: Feb 5 01:56:43:1) hashes: (-6186595901470975452/13223962844377986) 2016-02-05T03:20:44.698+0000 F REPL [rsBackgroundSync] need to rollback, but in inconsistent state 2016-02-05T03:20:44.698+0000 I - [rsBackgroundSync] Fatal assertion 28723 UnrecoverableRollbackError need to rollback, but in inconsistent state. minvalid: (term: 13, timestamp: Feb 5 01:56:45:86) > our last optime: (term: 12, timestamp: Feb 5 01:56:40:9) @ 18750 2016-02-05T03:20:44.698+0000 I - [rsBackgroundSync] ***aborting after fassert() failure