2017-02-18T17:26:29.493+0800 I ACCESS [conn205] authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "CN=192.168.1.55,OU=889748ef-e4b4-4d6c-a19d-59df5372abf0,O=huawei,L=xian,ST=shanxi,C=CN" }
|
2017-02-18T17:26:29.642+0800 I NETWORK [conn202] end connection 192.168.1.55:23686 (39 connections now open)
|
2017-02-18T17:26:29.645+0800 I REPL [ReplicationExecutor] stepping down from primary, because a new term has begun: 9
|
2017-02-18T17:26:29.645+0800 I REPL [replExecDBWorker-2] transition to SECONDARY
|
2017-02-18T17:26:29.645+0800 I NETWORK [conn133] end connection 192.168.1.59:12214 (38 connections now open)
|
2017-02-18T17:26:29.645+0800 I NETWORK [conn136] end connection 192.168.1.59:12219 (38 connections now open)
|
2017-02-18T17:26:29.645+0800 I NETWORK [conn203] end connection 192.168.1.55:23687 (38 connections now open)
|
2017-02-18T17:26:29.645+0800 I NETWORK [conn205] end connection 192.168.1.55:23689 (38 connections now open)
|
2017-02-18T17:26:29.645+0800 I NETWORK [conn132] end connection 192.168.1.53:42855 (38 connections now open)
|
2017-02-18T17:26:29.645+0800 I NETWORK [conn158] end connection 192.168.1.62:53959 (38 connections now open)
|
2017-02-18T17:26:29.645+0800 I NETWORK [conn116] end connection 192.168.1.62:53826 (38 connections now open)
|
2017-02-18T17:26:29.645+0800 I NETWORK [conn159] end connection 192.168.1.62:53960 (38 connections now open)
|
2017-02-18T17:26:29.645+0800 I NETWORK [conn135] end connection 192.168.1.53:42857 (38 connections now open)
|
2017-02-18T17:26:29.645+0800 I NETWORK [conn115] end connection 192.168.1.59:12144 (38 connections now open)
|
2017-02-18T17:26:29.645+0800 I NETWORK [conn166] end connection 192.168.1.62:53964 (36 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn118] end connection 192.168.1.54:48174 (35 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn102] end connection 192.168.1.62:53818 (34 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn130] end connection 192.168.1.54:48239 (33 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn134] end connection 192.168.1.53:42856 (32 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn104] end connection 192.168.1.62:53820 (32 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn169] end connection 192.168.1.62:53967 (29 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn106] end connection 192.168.1.53:42750 (29 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn129] end connection 192.168.1.53:42854 (27 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn110] end connection 192.168.1.70:55705 (27 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn113] end connection 192.168.1.70:55708 (25 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn174] end connection 192.168.1.59:12330 (22 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn131] end connection 192.168.1.62:53935 (22 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn114] end connection 192.168.1.53:42751 (22 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn117] end connection 192.168.1.56:26951 (32 connections now open)
|
2017-02-18T17:26:29.647+0800 I NETWORK [conn204] end connection 192.168.1.55:23688 (19 connections now open)
|
2017-02-18T17:26:29.646+0800 I NETWORK [conn167] end connection 192.168.1.62:53965 (22 connections now open)
|
2017-02-18T17:26:29.647+0800 I NETWORK [initandlisten] connection accepted from 192.168.1.70:55982 #206 (19 connections now open)
|
2017-02-18T17:26:29.647+0800 I NETWORK [conn171] end connection 192.168.1.53:42882 (18 connections now open)
|
2017-02-18T17:26:29.647+0800 I NETWORK [conn160] end connection 192.168.1.62:53961 (15 connections now open)
|
2017-02-18T17:26:29.647+0800 I NETWORK [conn168] end connection 192.168.1.62:53966 (14 connections now open)
|
2017-02-18T17:26:29.647+0800 I NETWORK [conn161] end connection 192.168.1.54:48254 (10 connections now open)
|
2017-02-18T17:26:29.647+0800 I NETWORK [conn128] end connection 192.168.1.53:42846 (8 connections now open)
|
2017-02-18T17:26:29.647+0800 I NETWORK [conn140] end connection 192.168.1.70:55777 (7 connections now open)
|
2017-02-18T17:26:29.647+0800 I NETWORK [conn170] end connection 192.168.1.53:42881 (7 connections now open)
|
2017-02-18T17:26:29.648+0800 I NETWORK [conn172] end connection 192.168.1.54:48269 (5 connections now open)
|
2017-02-18T17:26:29.647+0800 I NETWORK [conn201] end connection 192.168.1.55:23685 (17 connections now open)
|
2017-02-18T17:26:29.648+0800 I NETWORK [conn137] end connection 192.168.1.54:48240 (5 connections now open)
|
2017-02-18T17:26:29.647+0800 I NETWORK [conn163] end connection 192.168.1.62:53962 (17 connections now open)
|
2017-02-18T17:26:29.648+0800 I NETWORK [conn120] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [192.168.1.70:55722]
|
2017-02-18T17:26:29.653+0800 I ACCESS [conn206] authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "CN=192.168.1.70,OU=889748ef-e4b4-4d6c-a19d-59df5372abf0,O=huawei,L=xian,ST=shanxi,C=CN" }
|
2017-02-18T17:26:29.672+0800 I NETWORK [initandlisten] connection accepted from 192.168.1.62:53998 #207 (2 connections now open)
|
2017-02-18T17:26:29.677+0800 I ACCESS [conn207] authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "CN=192.168.1.62,OU=889748ef-e4b4-4d6c-a19d-59df5372abf0,O=huawei,L=xian,ST=shanxi,C=CN" }
|
2017-02-18T17:26:29.681+0800 I NETWORK [initandlisten] connection accepted from 192.168.1.53:42911 #208 (3 connections now open)
|
2017-02-18T17:26:29.686+0800 I ACCESS [conn208] authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "CN=192.168.1.53,OU=889748ef-e4b4-4d6c-a19d-59df5372abf0,O=huawei,L=xian,ST=shanxi,C=CN" }
|
2017-02-18T17:26:30.492+0800 I REPL [ReplicationExecutor] Member 192.168.1.55:8637 is now in state PRIMARY
|
2017-02-18T17:26:30.681+0800 I NETWORK [initandlisten] connection accepted from 192.168.1.70:55987 #209 (4 connections now open)
|
2017-02-18T17:26:30.686+0800 I ACCESS [conn209] authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "CN=192.168.1.70,OU=889748ef-e4b4-4d6c-a19d-59df5372abf0,O=huawei,L=xian,ST=shanxi,C=CN" }
|
2017-02-18T17:26:30.687+0800 I NETWORK [conn209] end connection 192.168.1.70:55987 (3 connections now open)
|
2017-02-18T17:26:30.687+0800 I NETWORK [initandlisten] connection accepted from 192.168.1.70:55988 #210 (4 connections now open)
|
2017-02-18T17:26:30.687+0800 I NETWORK [initandlisten] connection accepted from 192.168.1.70:55989 #211 (5 connections now open)
|
2017-02-18T17:26:30.693+0800 I ACCESS [conn210] authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "CN=192.168.1.70,OU=889748ef-e4b4-4d6c-a19d-59df5372abf0,O=huawei,L=xian,ST=shanxi,C=CN" }
|
2017-02-18T17:26:30.693+0800 I ACCESS [conn211] authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "CN=192.168.1.70,OU=889748ef-e4b4-4d6c-a19d-59df5372abf0,O=huawei,L=xian,ST=shanxi,C=CN" }
|
2017-02-18T17:26:30.698+0800 I NETWORK [conn210] end connection 192.168.1.70:55988 (4 connections now open)
|
2017-02-18T17:26:31.189+0800 I REPL [ReplicationExecutor] syncing from: 192.168.1.55:8637
|
2017-02-18T17:26:31.197+0800 I REPL [SyncSourceFeedback] setting syncSourceFeedback to 192.168.1.55:8637
|
2017-02-18T17:26:31.203+0800 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to 192.168.1.55:8637
|
2017-02-18T17:26:31.204+0800 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 8, timestamp: Feb 18 17:26:29:a). source's GTE: (term: 9, timestamp: Feb 18 17:26:33:2) hashes: (-2724116090579412714/-4865368491621293515)
|
2017-02-18T17:26:31.205+0800 I REPL [rsBackgroundSync] beginning rollback
|
2017-02-18T17:26:31.205+0800 I REPL [rsBackgroundSync] rollback 0
|
2017-02-18T17:26:31.205+0800 I REPL [ReplicationExecutor] transition to ROLLBACK
|
2017-02-18T17:26:31.205+0800 I REPL [rsBackgroundSync] rollback 1
|
2017-02-18T17:26:31.205+0800 I NETWORK [conn207] end connection 192.168.1.62:53998 (3 connections now open)
|
2017-02-18T17:26:31.205+0800 I NETWORK [conn208] end connection 192.168.1.53:42911 (3 connections now open)
|
2017-02-18T17:26:31.205+0800 I NETWORK [conn206] end connection 192.168.1.70:55982 (3 connections now open)
|
2017-02-18T17:26:31.205+0800 I NETWORK [conn211] end connection 192.168.1.70:55989 (0 connections now open)
|
2017-02-18T17:26:31.211+0800 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint
|
2017-02-18T17:26:31.212+0800 I REPL [rsBackgroundSync] rollback our last optime: Feb 18 17:26:29:a
|
2017-02-18T17:26:31.212+0800 I REPL [rsBackgroundSync] rollback their last optime: Feb 18 17:26:35:9
|
2017-02-18T17:26:31.212+0800 I REPL [rsBackgroundSync] rollback diff in end of log times: -6 seconds
|
2017-02-18T17:26:31.212+0800 I REPL [rsBackgroundSync] rollback 3 fixup
|
2017-02-18T17:26:31.213+0800 I REPL [rsBackgroundSync] rollback 3.5
|
2017-02-18T17:26:31.213+0800 I REPL [rsBackgroundSync] rollback 4 n:1
|
2017-02-18T17:26:31.213+0800 I REPL [rsBackgroundSync] minvalid=(term: 9, timestamp: Feb 18 17:26:35:9)
|
2017-02-18T17:26:31.213+0800 I REPL [rsBackgroundSync] rollback 4.6
|
2017-02-18T17:26:31.213+0800 I REPL [rsBackgroundSync] rollback 4.7
|
2017-02-18T17:26:31.214+0800 I REPL [rsBackgroundSync] rollback 5 d:0 u:1
|
2017-02-18T17:26:31.214+0800 I REPL [rsBackgroundSync] rollback 6
|
2017-02-18T17:26:31.214+0800 I REPL [rsBackgroundSync] rollback done
|
2017-02-18T17:26:31.214+0800 I REPL [rsBackgroundSync] rollback finished
|
2017-02-18T17:26:31.214+0800 I NETWORK [SyncSourceFeedback] Socket send() errno:9 Bad file descriptor 192.168.1.55:8637
|
2017-02-18T17:26:31.214+0800 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update: socket exception [SEND_ERROR] for 192.168.1.55:8637
|
2017-02-18T17:26:31.214+0800 I REPL [SyncSourceFeedback] updateUpstream failed: Location9001: socket exception [SEND_ERROR] for 192.168.1.55:8637, will retry
|
2017-02-18T17:26:31.224+0800 I REPL [ReplicationExecutor] transition to RECOVERING
|
2017-02-18T17:26:31.224+0800 I REPL [ReplicationExecutor] syncing from: 192.168.1.70:8637
|
2017-02-18T17:26:31.231+0800 I REPL [SyncSourceFeedback] setting syncSourceFeedback to 192.168.1.70:8637
|
2017-02-18T17:26:31.237+0800 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to 192.168.1.70:8637
|
2017-02-18T17:26:31.240+0800 I REPL [ReplicationExecutor] could not find member to sync from
|
2017-02-18T17:26:31.241+0800 I ASIO [ReplicationExecutor] dropping unhealthy pooled connection to 192.168.1.70:8637
|
2017-02-18T17:26:31.241+0800 I ASIO [ReplicationExecutor] after drop, pool was empty, going to spawn some connections
|
2017-02-18T17:26:31.247+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to 192.168.1.70:8637
|
2017-02-18T17:26:31.247+0800 I REPL [ReplicationExecutor] Member 192.168.1.70:8637 is now in state RECOVERING
|
2017-02-18T17:26:31.494+0800 I NETWORK [initandlisten] connection accepted from 192.168.1.55:23700 #212 (1 connection now open)
|
2017-02-18T17:26:31.499+0800 I ACCESS [conn212] authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "CN=192.168.1.55,OU=889748ef-e4b4-4d6c-a19d-59df5372abf0,O=huawei,L=xian,ST=shanxi,C=CN" }
|
2017-02-18T17:26:32.241+0800 I REPL [ReplicationExecutor] syncing from: 192.168.1.55:8637
|
2017-02-18T17:26:32.248+0800 I REPL [SyncSourceFeedback] setting syncSourceFeedback to 192.168.1.55:8637
|
2017-02-18T17:26:32.249+0800 I REPL [rsBackgroundSync] Starting rollback due to OplogStartMissing: our last op time fetched: (term: 8, timestamp: Feb 18 17:26:29:a). source's GTE: (term: 9, timestamp: Feb 18 17:26:33:2) hashes: (-2724116090579412714/-4865368491621293515)
|
2017-02-18T17:26:32.249+0800 I - [rsBackgroundSync] Fatal assertion 18750 UnrecoverableRollbackError: need to rollback, but in inconsistent state. minvalid: (term: 9, timestamp: Feb 18 17:26:35:9) > our last optime: (term: 8, timestamp: Feb 18 17:26:29:a)
|
2017-02-18T17:26:32.249+0800 I - [rsBackgroundSync]
|
|
***aborting after fassert() failure
|