2016-01-11T09:18:48.210+0000 W -        [initandlisten] Detected unclean shutdown - /mnt/mongodb/db/mongod.lock is not empty.
2016-01-11T09:18:48.210+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2016-01-11T09:18:48.211+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=14G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-01-11T09:18:54.497+0000 I STORAGE  [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
2016-01-11T09:18:54.552+0000 I CONTROL  [initandlisten] MongoDB starting : pid=14273 port=27017 dbpath=/mnt/mongodb/db 64-bit host=jermongo1-instance-1
2016-01-11T09:18:54.552+0000 I CONTROL  [initandlisten] db version v3.0.7
2016-01-11T09:18:54.552+0000 I CONTROL  [initandlisten] git version: 6ce7cbe8c6b899552dadd907604559806aa2e9bd
2016-01-11T09:18:54.552+0000 I CONTROL  [initandlisten] build info: Linux ip-10-183-78-195 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 BOOST_LIB_VERSION=1_49
2016-01-11T09:18:54.552+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2016-01-11T09:18:54.552+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, replication: { replSetName: "rs0" }, security: { authorization: "enabled", keyFile: "****" }, setParameter: { enableLocalhostAuthBypass: "true" }, storage: { dbPath: "/mnt/mongodb/db", directoryPerDB: true, engine: "wiredTiger", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/mnt/mongodb/log/mongodb.log" } }
2016-01-11T09:18:55.549+0000 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "rs0", version: 7, members: [ { _id: 0, host: "jermongo1:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.5, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "jermongo-arb:27017", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "jermongo2:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
2016-01-11T09:18:55.549+0000 I REPL     [ReplicationExecutor] This node is jermongo2:27017 in the config
2016-01-11T09:18:55.549+0000 I REPL     [ReplicationExecutor] transition to STARTUP2
2016-01-11T09:18:55.549+0000 I REPL     [ReplicationExecutor] Starting replication applier threads
2016-01-11T09:18:55.550+0000 I REPL     [ReplicationExecutor] transition to RECOVERING
2016-01-11T09:18:55.555+0000 I REPL     [ReplicationExecutor] transition to SECONDARY
2016-01-11T09:18:55.583+0000 I REPL     [ReplicationExecutor] Member jermongo-arb:27017 is now in state ARBITER
2016-01-11T09:18:55.584+0000 I REPL     [ReplicationExecutor] Standing for election
2016-01-11T09:18:55.602+0000 I REPL     [ReplicationExecutor] not electing self, we are not freshest
2016-01-11T09:18:55.602+0000 I REPL     [ReplicationExecutor] not electing self, we are not freshest
2016-01-11T09:18:56.481+0000 I REPL     [ReplicationExecutor] Member jermongo1:27017 is now in state PRIMARY
2016-01-11T09:18:59.552+0000 I REPL     [ReplicationExecutor] syncing from: jermongo1:27017
2016-01-11T09:19:01.044+0000 I REPL     [SyncSourceFeedback] replset setting syncSourceFeedback to jermongo1:27017
2016-01-11T09:19:03.396+0000 I REPL     [rsBackgroundSync] replSet our last op time fetched: Jan 11 09:01:32:1
2016-01-11T09:19:03.396+0000 I REPL     [rsBackgroundSync] replset source's GTE: Jan 11 09:03:35:1
2016-01-11T09:19:03.396+0000 I REPL     [rsBackgroundSync] beginning rollback
2016-01-11T09:19:03.396+0000 I REPL     [rsBackgroundSync] rollback 0
2016-01-11T09:19:03.396+0000 I REPL     [ReplicationExecutor] transition to ROLLBACK
2016-01-11T09:19:03.396+0000 I REPL     [rsBackgroundSync] rollback 1
2016-01-11T09:19:03.396+0000 I REPL     [rsBackgroundSync] rollback 2 FindCommonPoint
2016-01-11T09:19:03.755+0000 I REPL     [rsBackgroundSync] replSet info rollback our last optime:   Jan 11 09:01:32:1
2016-01-11T09:19:03.755+0000 I REPL     [rsBackgroundSync] replSet info rollback their last optime: Jan 11 09:23:06:1
2016-01-11T09:19:03.755+0000 I REPL     [rsBackgroundSync] replSet info rollback diff in end of log times: -1294 seconds
2016-01-11T09:19:11.840+0000 I REPL     [rsBackgroundSync] replSet rollback found matching events at Jan 11 08:58:45:2
2016-01-11T09:19:11.840+0000 I REPL     [rsBackgroundSync] replSet rollback findcommonpoint scanned : 2786
2016-01-11T09:19:11.840+0000 I REPL     [rsBackgroundSync] replSet rollback 3 fixup
2016-01-11T09:19:12.356+0000 I REPL     [rsBackgroundSync] rollback 3.5
2016-01-11T09:19:12.537+0000 I REPL     [rsBackgroundSync] rollback 4 n:1
2016-01-11T09:19:12.537+0000 I REPL     [rsBackgroundSync] replSet minvalid=Jan 11 09:23:20 56937488:1
2016-01-11T09:19:12.538+0000 I REPL     [rsBackgroundSync] rollback 4.6
2016-01-11T09:19:12.539+0000 I REPL     [rsBackgroundSync] rollback 4.7
2016-01-11T09:19:12.609+0000 I REPL     [rsBackgroundSync] rollback 5 d:0 u:1
2016-01-11T09:19:12.609+0000 I REPL     [rsBackgroundSync] rollback 6
2016-01-11T09:19:12.610+0000 I REPL     [rsBackgroundSync] rollback done
2016-01-11T09:19:12.610+0000 I REPL     [ReplicationExecutor] transition to RECOVERING
2016-01-11T09:19:12.610+0000 I REPL     [rsBackgroundSync] rollback finished
2016-01-11T09:19:12.611+0000 I REPL     [ReplicationExecutor] syncing from: jermongo1:27017
2016-01-11T09:19:14.088+0000 I REPL     [SyncSourceFeedback] replset setting syncSourceFeedback to jermongo1:27017
2016-01-11T09:19:16.413+0000 I REPL     [ReplicationExecutor] transition to SECONDARY
2016-01-11T09:19:23.749+0000 I REPL     [ReplicationExecutor] syncing from: jermongo1:27017
2016-01-11T09:19:25.240+0000 I REPL     [SyncSourceFeedback] replset setting syncSourceFeedback to jermongo1:27017
2016-01-11T09:20:34.435+0000 I REPL     [ReplicationExecutor] Member jermongo1:27017 is now in state SECONDARY
2016-01-11T09:20:34.435+0000 I REPL     [ReplicationExecutor] Standing for election
2016-01-11T09:20:34.436+0000 I REPL     [ReplicationExecutor] not electing self, jermongo-arb:27017 would veto with 'jermongo2:27017 is trying to elect itself but jermongo1:27017 is already primary and more up-to-date'
2016-01-11T09:20:34.436+0000 I REPL     [ReplicationExecutor] not electing self, we are not freshest
2016-01-11T09:20:34.437+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update: DBClientBase::findN: transport error: jermongo1:27017 ns: admin.$cmd query: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('565aeff5462fb7a9fc76aa3d'), optime: Timestamp 1452504283000|1, memberId: 2, cfgver: 7, config: { _id: 2, host: "jermongo2:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } } ] }
2016-01-11T09:20:34.437+0000 E REPL     [rsBackgroundSync] sync producer problem: 10278 dbclient error communicating with server: jermongo1:27017
2016-01-11T09:20:34.437+0000 I -        [rsBackgroundSync] caught exception (socket exception [FAILED_STATE] for jermongo1:27017 (5.29.3.150) failed) in destructor (kill)
2016-01-11T09:20:34.437+0000 I REPL     [ReplicationExecutor] could not find member to sync from
2016-01-11T09:20:35.437+0000 I REPL     [ReplicationExecutor] syncing from: jermongo1:27017
2016-01-11T09:20:35.707+0000 I REPL     [ReplicationExecutor] Standing for election
2016-01-11T09:20:35.883+0000 I REPL     [ReplicationExecutor] not electing self, we are not freshest
2016-01-11T09:20:35.884+0000 I REPL     [ReplicationExecutor] not electing self, we are not freshest
2016-01-11T09:20:36.608+0000 I REPL     [ReplicationExecutor] Standing for election
2016-01-11T09:20:36.609+0000 I REPL     [ReplicationExecutor] not electing self, we are not freshest
2016-01-11T09:20:36.609+0000 I REPL     [ReplicationExecutor] not electing self, we are not freshest
2016-01-11T09:20:36.940+0000 I REPL     [SyncSourceFeedback] replset setting syncSourceFeedback to jermongo1:27017
2016-01-11T09:20:37.708+0000 I REPL     [ReplicationExecutor] Standing for election
2016-01-11T09:20:37.882+0000 I REPL     [ReplicationExecutor] replSet possible election tie; sleeping 702ms until 2016-01-11T09:20:38.584+0000
2016-01-11T09:20:38.584+0000 I REPL     [ReplicationExecutor] Standing for election
2016-01-11T09:20:38.762+0000 I REPL     [ReplicationExecutor] replSet info electSelf
2016-01-11T09:20:38.763+0000 I REPL     [ReplicationExecutor] received vote: 1 votes from jermongo-arb:27017
2016-01-11T09:20:38.763+0000 I REPL     [ReplicationExecutor] replSet election succeeded, assuming primary role
2016-01-11T09:20:38.763+0000 I REPL     [ReplicationExecutor] transition to PRIMARY
2016-01-11T09:20:42.314+0000 I REPL     [rsSync] transition to primary complete; database writes are now permitted
2016-01-11T10:03:51.180+0000 I INDEX    [conn11847] build index on: JER_SIGNALS.test_ndfd_weather_forecast properties: { v: 1, unique: true, key: { pid: 1, ts: 1 }, name: "pid_ts_1_1", ns: "JER_SIGNALS.test_ndfd_weather_forecast" }
2016-01-11T10:03:51.180+0000 I INDEX    [conn11847]      building index using bulk method
2016-01-11T10:03:51.189+0000 I INDEX    [conn11847] build index on: JER_SIGNALS.test_ndfd_weather_forecast properties: { v: 1, key: { ts: 1 }, name: "ts_1", ns: "JER_SIGNALS.test_ndfd_weather_forecast" }
2016-01-11T10:03:51.189+0000 I INDEX    [conn11847]      building index using bulk method
2016-01-11T10:03:51.195+0000 I INDEX    [conn11847] build index done.  scanned 0 total records. 0 secs
2016-01-11T11:52:32.494+0000 I INDEX    [conn38709] build index on: JER_SIGNALS.test_ndfd_weather_forecast_ properties: { v: 1, unique: true, key: { pid: 1, ts: 1 }, name: "pid_ts_1_1", ns: "JER_SIGNALS.test_ndfd_weather_forecast_" }
2016-01-11T11:52:32.494+0000 I INDEX    [conn38709]      building index using bulk method
2016-01-11T11:52:32.504+0000 I INDEX    [conn38709] build index on: JER_SIGNALS.test_ndfd_weather_forecast_ properties: { v: 1, key: { ts: 1 }, name: "ts_1", ns: "JER_SIGNALS.test_ndfd_weather_forecast_" }
2016-01-11T11:52:32.504+0000 I INDEX    [conn38709]      building index using bulk method
2016-01-11T11:52:32.527+0000 I INDEX    [conn38709] build index done.  scanned 0 total records. 0 secs
2016-01-11T13:01:05.947+0000 I INDEX    [conn55074] build index on: JER_SIGNALS.test_ndfd_weather_forecast_ properties: { v: 1, unique: true, key: { pid: 1, ts: 1 }, name: "pid_ts_1_1", ns: "JER_SIGNALS.test_ndfd_weather_forecast_" }
2016-01-11T13:01:05.947+0000 I INDEX    [conn55074]      building index using bulk method
2016-01-11T13:01:05.957+0000 I INDEX    [conn55074] build index on: JER_SIGNALS.test_ndfd_weather_forecast_ properties: { v: 1, key: { ts: 1 }, name: "ts_1", ns: "JER_SIGNALS.test_ndfd_weather_forecast_" }
2016-01-11T13:01:05.957+0000 I INDEX    [conn55074]      building index using bulk method
2016-01-11T13:01:05.964+0000 I INDEX    [conn55074] build index done.  scanned 0 total records. 0 secs
2016-01-11T13:06:36.184+0000 I INDEX    [conn56163] build index on: JER_SIGNALS.test_ndfd_weather_forecast_ properties: { v: 1, unique: true, key: { pid: 1, ts: 1 }, name: "pid_ts_1_1", ns: "JER_SIGNALS.test_ndfd_weather_forecast_" }
2016-01-11T13:06:36.184+0000 I INDEX    [conn56163]      building index using bulk method
2016-01-11T13:06:36.193+0000 I INDEX    [conn56163] build index on: JER_SIGNALS.test_ndfd_weather_forecast_ properties: { v: 1, key: { ts: 1 }, name: "ts_1", ns: "JER_SIGNALS.test_ndfd_weather_forecast_" }
2016-01-11T13:06:36.193+0000 I INDEX    [conn56163]      building index using bulk method
2016-01-11T13:06:36.198+0000 I INDEX    [conn56163] build index done.  scanned 0 total records. 0 secs
2016-01-11T13:24:33.765+0000 I INDEX    [conn60096] build index on: JER_SIGNALS.test_ndfd_weather_forecast_ properties: { v: 1, unique: true, key: { pid: 1, ts: 1 }, name: "pid_ts_1_1", ns: "JER_SIGNALS.test_ndfd_weather_forecast_" }
2016-01-11T13:24:33.765+0000 I INDEX    [conn60096]      building index using bulk method
2016-01-11T13:24:33.774+0000 I INDEX    [conn60096] build index on: JER_SIGNALS.test_ndfd_weather_forecast_ properties: { v: 1, key: { ts: 1 }, name: "ts_1", ns: "JER_SIGNALS.test_ndfd_weather_forecast_" }
2016-01-11T13:24:33.774+0000 I INDEX    [conn60096]      building index using bulk method
2016-01-11T13:24:33.780+0000 I INDEX    [conn60096] build index done.  scanned 0 total records. 0 secs
2016-01-11T13:27:08.911+0000 I INDEX    [conn56770] build index on: JER_DB.Celery_log properties: { v: 1, key: { uuid: 1 }, name: "uuid_1", ns: "JER_DB.Celery_log" }
2016-01-11T13:27:08.911+0000 I INDEX    [conn56770]      building index using bulk method
2016-01-11T13:27:08.957+0000 I INDEX    [conn56770] build index done.  scanned 12694 total records. 0 secs
2016-01-11T13:32:33.822+0000 I INDEX    [conn56770] build index on: JER_DB.Celery_log properties: { v: 1, unique: true, key: { uuid: 1, state: 1 }, name: "uuid_state_1", ns: "JER_DB.Celery_log" }
2016-01-11T13:32:33.822+0000 I INDEX    [conn56770]      building index using bulk method
2016-01-11T13:32:33.877+0000 I INDEX    [conn56770] build index done.  scanned 13996 total records. 0 secs
2016-01-11T13:50:47.315+0000 I CONTROL  ***** SERVER RESTARTED *****
2016-01-11T13:50:47.493+0000 W -        [initandlisten] Detected unclean shutdown - /mnt/mongodb/db/mongod.lock is not empty.
2016-01-11T13:50:47.493+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2016-01-11T13:50:47.493+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=14G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-01-11T13:50:54.043+0000 I STORAGE  [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
2016-01-11T13:50:54.089+0000 I CONTROL  [initandlisten] MongoDB starting : pid=25564 port=27017 dbpath=/mnt/mongodb/db 64-bit host=jermongo1-instance-1
2016-01-11T13:50:54.090+0000 I CONTROL  [initandlisten] db version v3.0.7
2016-01-11T13:50:54.090+0000 I CONTROL  [initandlisten] git version: 6ce7cbe8c6b899552dadd907604559806aa2e9bd
2016-01-11T13:50:54.090+0000 I CONTROL  [initandlisten] build info: Linux ip-10-183-78-195 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 BOOST_LIB_VERSION=1_49
2016-01-11T13:50:54.090+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2016-01-11T13:50:54.090+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, replication: { replSetName: "rs0" }, security: { authorization: "enabled", keyFile: "****" }, setParameter: { enableLocalhostAuthBypass: "true" }, storage: { dbPath: "/mnt/mongodb/db", directoryPerDB: true, engine: "wiredTiger", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/mnt/mongodb/log/mongodb.log" } }
2016-01-11T13:50:55.178+0000 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "rs0", version: 7, members: [ { _id: 0, host: "jermongo1:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.5, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "jermongo-arb:27017", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "jermongo2:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
2016-01-11T13:50:55.178+0000 I REPL     [ReplicationExecutor] This node is jermongo2:27017 in the config
2016-01-11T13:50:55.178+0000 I REPL     [ReplicationExecutor] transition to STARTUP2
2016-01-11T13:50:55.178+0000 I REPL     [ReplicationExecutor] Starting replication applier threads
2016-01-11T13:50:55.194+0000 I REPL     [ReplicationExecutor] transition to RECOVERING
2016-01-11T13:50:55.196+0000 I REPL     [ReplicationExecutor] transition to SECONDARY
2016-01-11T13:50:55.247+0000 I REPL     [ReplicationExecutor] Member jermongo-arb:27017 is now in state ARBITER
2016-01-11T13:50:55.247+0000 I REPL     [ReplicationExecutor] Standing for election
2016-01-11T13:50:55.278+0000 I REPL     [ReplicationExecutor] not electing self, we are not freshest
2016-01-11T13:50:55.278+0000 I REPL     [ReplicationExecutor] not electing self, we are not freshest
2016-01-11T13:50:56.112+0000 I REPL     [ReplicationExecutor] Member jermongo1:27017 is now in state PRIMARY
2016-01-11T13:50:59.195+0000 I REPL     [ReplicationExecutor] syncing from: jermongo1:27017
2016-01-11T13:51:00.482+0000 I REPL     [SyncSourceFeedback] replset setting syncSourceFeedback to jermongo1:27017
2016-01-11T13:51:02.360+0000 I REPL     [rsBackgroundSync] replSet our last op time fetched: Jan 11 13:45:14:1
2016-01-11T13:51:02.361+0000 I REPL     [rsBackgroundSync] replset source's GTE: Jan 11 13:49:34:1
2016-01-11T13:51:02.361+0000 I REPL     [rsBackgroundSync] beginning rollback
2016-01-11T13:51:02.361+0000 I REPL     [rsBackgroundSync] rollback 0
2016-01-11T13:51:02.361+0000 I REPL     [ReplicationExecutor] transition to ROLLBACK
2016-01-11T13:51:02.361+0000 I REPL     [rsBackgroundSync] rollback 1
2016-01-11T13:51:02.361+0000 I REPL     [rsBackgroundSync] rollback 2 FindCommonPoint
2016-01-11T13:51:02.711+0000 I REPL     [rsBackgroundSync] replSet info rollback our last optime:   Jan 11 13:45:14:1
2016-01-11T13:51:02.711+0000 I REPL     [rsBackgroundSync] replSet info rollback their last optime: Jan 11 13:55:09:1
2016-01-11T13:51:02.711+0000 I REPL     [rsBackgroundSync] replSet info rollback diff in end of log times: -595 seconds
2016-01-11T13:51:16.555+0000 W REPL     [rsBackgroundSync] ignoring op with no _id during rollback. ns: JER_DB.system.indexes, document: { ts: Timestamp 1452519153000|2, h: -6306293468986696903, v: 2, op: "i", ns: "JER_DB.system.indexes", o: { name: "uuid_state_1", ns: "JER_DB.Celery_log", unique: true, key: { uuid: 1, state: 1 } } }
2016-01-11T13:51:16.676+0000 W REPL     [rsBackgroundSync] replSet info rollback of dropIndexes is slow in this version of mongod
2016-01-11T13:51:16.836+0000 W REPL     [rsBackgroundSync] ignoring op with no _id during rollback. ns: JER_DB.system.indexes, document: { ts: Timestamp 1452518828000|1, h: -4918113476754073791, v: 2, op: "i", ns: "JER_DB.system.indexes", o: { name: "uuid_1", ns: "JER_DB.Celery_log", key: { uuid: 1 } } }
2016-01-11T13:51:22.777+0000 W REPL     [rsBackgroundSync] ignoring op with no _id during rollback. ns: JER_SIGNALS.system.indexes, document: { ts: Timestamp 1452518673000|4, h: -8456178278876933663, v: 2, op: "i", ns: "JER_SIGNALS.system.indexes", o: { ns: "JER_SIGNALS.test_ndfd_weather_forecast_", name: "ts_1", key: { ts: 1 } } }
2016-01-11T13:51:22.777+0000 W REPL     [rsBackgroundSync] ignoring op with no _id during rollback. ns: JER_SIGNALS.system.indexes, document: { ts: Timestamp 1452518673000|3, h: 1746040611747149689, v: 2, op: "i", ns: "JER_SIGNALS.system.indexes", o: { ns: "JER_SIGNALS.test_ndfd_weather_forecast_", unique: true, name: "pid_ts_1_1", key: { pid: 1, ts: 1 } } }
2016-01-11T13:52:17.411+0000 I REPL     [rsBackgroundSync] replSet rollback found matching events at Jan 11 13:12:04:b0
2016-01-11T13:52:17.411+0000 I REPL     [rsBackgroundSync] replSet rollback findcommonpoint scanned : 94509
2016-01-11T13:52:17.481+0000 I REPL     [rsBackgroundSync] replSet rollback 3 fixup