Fri Sep 28 14:09:04 [conn7492] command ump.$cmd command: { count: "entities.TaskReport", query: { status: "ERROR" } } ntoreturn:1 reslen:48 178ms Fri Sep 28 14:09:10 [conn8600] end connection 10.220.40.24:41826 Fri Sep 28 14:09:10 [initandlisten] connection accepted from 10.220.40.24:41830 #8603 Fri Sep 28 14:09:12 [conn8601] end connection 10.220.40.26:39137 Fri Sep 28 14:09:12 [initandlisten] connection accepted from 10.220.40.26:39140 #8604 Fri Sep 28 14:09:16 [conn6864] command ump.$cmd command: { count: "entities.TaskReport", query: { status: "ERROR" } } ntoreturn:1 reslen:48 166ms Fri Sep 28 14:09:22 [conn6938] query ump.entities.TaskReport query: { _tags: "f43e61-f43e61f67d9e" } nscanned:4 nreturned:4 reslen:1183 106ms Fri Sep 28 14:09:22 [conn6963] query ump.entities.TaskReport query: { _tags: "00D0D0-c864c76f7195" } nscanned:4 nreturned:4 reslen:1279 206ms Fri Sep 28 14:09:27 [conn7086] query ump.entities.TaskReport query: { _tags: "00D0D0-0022938BA4A5" } nscanned:4 nreturned:4 reslen:1181 107ms Fri Sep 28 14:09:29 [clientcursormon] mem (MB) res:11241 virt:157863 mapped:77853 Fri Sep 28 14:09:30 [conn8602] end connection 10.220.40.23:34183 Fri Sep 28 14:09:30 [initandlisten] connection accepted from 10.220.40.23:34187 #8605 Fri Sep 28 14:09:35 [conn7159] query datamodel.datamodel query: { _id: "C83A35-c83a351f6d48" } ntoreturn:1 idhack:1 reslen:199648 1937ms Fri Sep 28 14:09:35 [conn7492] command ump.$cmd command: { count: "entities.TaskReport", query: { status: "ERROR" } } ntoreturn:1 reslen:48 350ms Fri Sep 28 14:09:36 [rsHealthPoll] replSet info acs4:27017 is down (or slow to respond): replSet unexpected exception in ReplSetHealthPollTask Fri Sep 28 14:09:36 [rsHealthPoll] replSet member acs4:27017 is now in state DOWN Fri Sep 28 14:09:37 [conn7134] query ump.entities.Device query: { _id: "C83A35-c83a35310658" } ntoreturn:1 idhack:1 reslen:889 230ms Fri Sep 28 14:09:37 [conn7270] query ump.entities.TaskReport query: { _tags: "00D0D0-b075d547a471" } nscanned:4 nreturned:4 reslen:1181 197ms Fri Sep 28 14:09:38 [conn7076] query ump.entities.Device query: { _id: "00D0D0-c864c76fa9de" } ntoreturn:1 idhack:1 reslen:935 330ms Fri Sep 28 14:09:38 [conn6991] query datamodel.datamodel query: { _id: "00D0D0-B4B36255B9CA" } ntoreturn:1 idhack:1 reslen:86517 317ms Fri Sep 28 14:09:39 [conn7012] query datamodel.datamodel query: { _id: "00D0D0-002512C56A9C" } ntoreturn:1 idhack:1 reslen:85182 798ms Fri Sep 28 14:09:39 [conn6624] query datamodel.datamodel query: { _id: "00E0FC-C8D5FE1FD614" } ntoreturn:1 idhack:1 reslen:1962 899ms Fri Sep 28 14:09:40 [rsHealthPoll] replSet info acs2:27017 is down (or slow to respond): replSet unexpected exception in ReplSetHealthPollTask Fri Sep 28 14:09:40 [rsHealthPoll] replSet info acs1:27017 is down (or slow to respond): replSet unexpected exception in ReplSetHealthPollTask Fri Sep 28 14:09:40 [rsHealthPoll] replSet member acs2:27017 is now in state DOWN Fri Sep 28 14:09:40 [rsHealthPoll] replSet member acs1:27017 is now in state DOWN Fri Sep 28 14:09:40 [conn8603] end connection 10.220.40.24:41830 Fri Sep 28 14:09:40 [initandlisten] connection accepted from 10.220.40.24:41833 #8606 Fri Sep 28 14:09:42 [conn8604] end connection 10.220.40.26:39140 Fri Sep 28 14:09:42 [initandlisten] connection accepted from 10.220.40.26:39144 #8607 Fri Sep 28 14:09:51 [conn6631] query ump.entities.TaskReport query: { _tags: "f43e61-f43e61eb005e" } nscanned:4 nreturned:4 reslen:1181 213ms Fri Sep 28 14:09:54 [conn6627] query ump.entities.Device query: { _id: "00D0D0-c864c77ebb76" } ntoreturn:1 idhack:1 reslen:933 207ms Fri Sep 28 14:09:55 [conn6929] query ump.entities.TaskReport query: { _tags: "000ef4-000ef416dcc4" } nscanned:4 nreturned:4 reslen:1279 226ms Fri Sep 28 14:09:55 [conn6714] query datamodel.datamodel query: { _id: "00D0D0-0022938BEDA1" } ntoreturn:1 idhack:1 reslen:1959 163ms Fri Sep 28 14:09:56 [conn7710] query ump.entities.TaskReport query: { _tags: "00D0D0-B075D5274AC5" } nscanned:4 nreturned:4 reslen:1183 418ms Fri Sep 28 14:09:56 [conn6866] query ump.entities.TaskReport query: { _tags: "00D0D0-B075D5274ED3" } nscanned:4 nreturned:4 reslen:1282 164ms Fri Sep 28 14:09:57 [conn6965] query ump.entities.TaskReport query: { _tags: "00D0D0-B075D53930AD" } nscanned:4 nreturned:4 reslen:1271 319ms Fri Sep 28 14:09:57 [conn6646] query datamodel.datamodel query: { _id: "00D0D0-0026ED9A9FED" } ntoreturn:1 idhack:1 reslen:85177 696ms Fri Sep 28 14:09:57 [conn6792] query ump.entities.TaskReport query: { _tags: "C83A35-c83a352585b0" } nscanned:4 nreturned:4 reslen:382666 1623ms Fri Sep 28 14:09:57 [conn6895] query ump.entities.Device query: { _id: "C83A35-c83a35234260" } ntoreturn:1 idhack:1 reslen:891 206ms Fri Sep 28 14:09:59 [conn7125] query datamodel.datamodel query: { _id: "00E0FC-00E04DDA983A" } ntoreturn:1 idhack:1 reslen:57706 910ms Fri Sep 28 14:10:00 [rsHealthPoll] replSet member acs1:27017 is up Fri Sep 28 14:10:00 [rsHealthPoll] replSet member acs1:27017 is now in state SECONDARY Fri Sep 28 14:10:00 [conn8605] end connection 10.220.40.23:34187 Fri Sep 28 14:10:00 [initandlisten] connection accepted from 10.220.40.23:34191 #8608 Fri Sep 28 14:10:00 [rsHealthPoll] replSet member acs4:27017 is up Fri Sep 28 14:10:00 [rsHealthPoll] replSet member acs4:27017 is now in state SECONDARY Fri Sep 28 14:10:00 [rsMgr] replSet warning caught unexpected exception in electSelf() Fri Sep 28 14:10:00 Invalid access at address: 0x8 Fri Sep 28 14:10:00 Got signal: 11 (Segmentation fault). Fri Sep 28 14:10:00 Backtrace: 0xa9609a 0xa9678c 0x7fec59f93060 0x57405c 0x5763d4 0xaabdb0 0x7fec59f8aefc 0x7fec5952489d /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3aa) [0xa9609a] /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x22c) [0xa9678c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10060) [0x7fec59f93060] /usr/bin/mongod(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0xbc) [0x57405c] /usr/bin/mongod(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74) [0x5763d4] /usr/bin/mongod(thread_proxy+0x80) [0xaabdb0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7fec59f8aefc] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fec5952489d] Fri Sep 28 14:10:01 Invalid access at address: 0x4 Fri Sep 28 14:10:01 Got signal: 11 (Segmentation fault). Fri Sep 28 14:10:01 Invalid access at address: 0x4 Fri Sep 28 14:10:01 Got signal: 11 (Segmentation fault). Fri Sep 28 14:10:01 Invalid access at address: 0x4 Fri Sep 28 14:10:01 Got signal: 11 (Segmentation fault). Fri Sep 28 14:10:01 Invalid access at address: 0x4 Fri Sep 28 14:10:01 Got signal: 11 (Segmentation fault). Fri Sep 28 14:10:01 Invalid access at address: 0x4 Fri Sep 28 14:10:01 Got signal: 11 (Segmentation fault). Fri Sep 28 14:10:01 Invalid access at address: 0x4 Fri Sep 28 14:10:01 Got signal: 11 (Segmentation fault). Fri Sep 28 14:10:01 Invalid access at address: 0x4 Fri Sep 28 14:10:01 Got signal: 11 (Segmentation fault). Fri Sep 28 14:10:01 Invalid access at address: 0x4 Fri Sep 28 14:10:01 Got signal: 11 (Segmentation fault). Fri Sep 28 14:10:01 Invalid access at address: 0x4 Fri Sep 28 14:10:01 Got signal: 11 (Segmentation fault). Fri Sep 28 14:10:01 Invalid access at address: 0x4 Fri Sep 28 14:10:01 Got signal: 11 (Segmentation fault). Fri Sep 28 14:10:01 Backtrace: 0xa9609a 0xa9678c 0x7fec59f93060 0x54f952 0x55a864 0x55b554 0x8e07e5 0x9440a0 0x888fd7 0x88dbb9 0xaa0b38 0x638767 0x7fec59f8aefc 0x7fec5952489d /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3aa) [0xa9609a] /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x22c) [0xa9678c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10060) [0x7fec59f93060] /usr/bin/mongod(_ZN5mongo10FieldRangeC1ERKNS_11BSONElementEbbb+0x1a2) [0x54f952] /usr/bin/mongod(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BSONElementEb+0x84) [0x55a864] /usr/bin/mongod(_ZN5mongo13FieldRangeSetC1EPKcRKNS_7BSONObjEbb+0x194) [0x55b554] /usr/bin/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElementEbS5_S5_bb+0x1f5) [0x8e07e5] /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xbf0) [0x9440a0] /usr/bin/mongod() [0x888fd7] /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x559) [0x88dbb9] /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) [0xaa0b38] /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638767] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7fec59f8aefc] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fec5952489d] Fri Sep 28 14:10:01 Invalid access at address: 0xffffffffffffffe8 Fri Sep 28 14:10:01 Got signal: 11 (Segmentation fault). Fri Sep 28 14:10:01 Backtrace: 0xa9609a 0xa9678c 0x7fec59f93060 0x54f952 0x55a864 0x55b554 0x8e07e5 0x9440a0 0x888fd7 0x88dbb9 0xaa0b38 0x638767 0x7fec59f8aefc 0x7fec5952489d /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3aa) [0xa9609a] /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x22c) [0xa9678c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10060) [0x7fec59f93060] /usr/bin/mongod(_ZN5mongo10FieldRangeC1ERKNS_11BSONElementEbbb+0x1a2) [0x54f952] /usr/bin/mongod(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BSONElementEb+0x84) [0x55a864] /usr/bin/mongod(_ZN5mongo13FieldRangeSetC1EPKcRKNS_7BSONObjEbb+0x194) [0x55b554] /usr/bin/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElementEbS5_S5_bb+0x1f5) [0x8e07e5] /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xbf0) [0x9440a0] /usr/bin/mongod() [0x888fd7] /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x559) [0x88dbb9] /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) [0xaa0b38] /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638767] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7fec59f8aefc] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fec5952489d] Fri Sep 28 14:10:01 Backtrace: 0xa9609a 0xa9678c 0x7fec59f93060 0x54f952 0x55a864 0x55b554 0x8e07e5 0x9440a0 0x888fd7 0x88dbb9 0xaa0b38 0x638767 0x7fec59f8aefc 0x7fec5952489d /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3aa) [0xa9609a] /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x22c) [0xa9678c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10060) [0x7fec59f93060] /usr/bin/mongod(_ZN5mongo10FieldRangeC1ERKNS_11BSONElementEbbb+0x1a2) [0x54f952] /usr/bin/mongod(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BSONElementEb+0x84) [0x55a864] /usr/bin/mongod(_ZN5mongo13FieldRangeSetC1EPKcRKNS_7BSONObjEbb+0x194) [0x55b554] /usr/bin/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElementEbS5_S5_bb+0x1f5) [0x8e07e5] /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xbf0) [0x9440a0] /usr/bin/mongod() [0x888fd7] /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x559) [0x88dbb9] /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) [0xaa0b38] /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638767] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7fec59f8aefc] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fec5952489d] Fri Sep 28 14:10:01 Backtrace: 0xa9609a 0xa9678c 0x7fec59f93060 0x54f952 0x55a864 0x55b554 0x8e07e5 0x9440a0 0x888fd7 0x88dbb9 0xaa0b38 0x638767 0x7fec59f8aefc 0x7fec5952489d /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3aa) [0xa9609a] /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x22c) [0xa9678c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10060) [0x7fec59f93060] /usr/bin/mongod(_ZN5mongo10FieldRangeC1ERKNS_11BSONElementEbbb+0x1a2) [0x54f952] /usr/bin/mongod(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BSONElementEb+0x84) [0x55a864] /usr/bin/mongod(_ZN5mongo13FieldRangeSetC1EPKcRKNS_7BSONObjEbb+0x194) [0x55b554] /usr/bin/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElementEbS5_S5_bb+0x1f5) [0x8e07e5] /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xbf0) [0x9440a0] /usr/bin/mongod() [0x888fd7] /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x559) [0x88dbb9] /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) [0xaa0b38] /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638767] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7fec59f8aefc] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fec5952489d] Fri Sep 28 14:10:01 Invalid access at address: 0x4 Fri Sep 28 14:10:01 Got signal: 11 (Segmentation fault). *** glibc detected *** /usr/bin/mongod: corrupted double-linked list: 0x00007fec50022b50 *** ======= Backtrace: ========= Fri Sep 28 14:10:01 Backtrace: 0xa9609a 0xa9678c 0x7fec59f93060 0x54f952 0x55a864 0x55b554 0x8e07e5 0x9440a0 0x888fd7 0x88dbb9 0xaa0b38 0x638767 0x7fec59f8aefc 0x7fec5952489d /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3aa) [0xa9609a] /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x22c) [0xa9678c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10060) [0x7fec59f93060] /usr/bin/mongod(_ZN5mongo10FieldRangeC1ERKNS_11BSONElementEbbb+0x1a2) [0x54f952] /usr/bin/mongod(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BSONElementEb+0x84) [0x55a864] /usr/bin/mongod(_ZN5mongo13FieldRangeSetC1EPKcRKNS_7BSONObjEbb+0x194) [0x55b554] /usr/bin/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElementEbS5_S5_bb+0x1f5) [0x8e07e5] /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xbf0) [0x9440a0] /usr/bin/mongod() [0x888fd7] /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x559) [0x88dbb9] /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) [0xaa0b38] /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638767] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7fec59f8aefc] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fec5952489d] Logstream::get called in uninitialized state Fri Sep 28 14:10:01 Backtrace: 0xa9609a 0xa9678c 0x7fec59f93060 0x54f952 0x55a864 0x55b554 0x8e07e5 0x9440a0 0x888fd7 0x88dbb9 0xaa0b38 0x638767 0x7fec59f8aefc 0x7fec5952489d /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3aa) [0xa9609a] /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x22c) [0xa9678c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10060) [0x7fec59f93060] /usr/bin/mongod(_ZN5mongo10FieldRangeC1ERKNS_11BSONElementEbbb+0x1a2) [0x54f952] /usr/bin/mongod(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BSONElementEb+0x84) [0x55a864] /usr/bin/mongod(_ZN5mongo13FieldRangeSetC1EPKcRKNS_7BSONObjEbb+0x194) [0x55b554] /usr/bin/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElementEbS5_S5_bb+0x1f5) [0x8e07e5] /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xbf0) [0x9440a0] /usr/bin/mongod() [0x888fd7] /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x559) [0x88dbb9] /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) [0xaa0b38] /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638767] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7fec59f8aefc] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fec5952489d] ***** SERVER RESTARTED ***** Fri Sep 28 14:12:06 [initandlisten] MongoDB starting : pid=24038 port=27017 dbpath=/var/lib/mongodb-new 64-bit host=ACS-PROV3 Fri Sep 28 14:12:06 [initandlisten] Fri Sep 28 14:12:06 [initandlisten] ** WARNING: You are running on a NUMA machine. Fri Sep 28 14:12:06 [initandlisten] ** We suggest launching mongod like this to avoid performance problems: Fri Sep 28 14:12:06 [initandlisten] ** numactl --interleave=all mongod [other options] Fri Sep 28 14:12:06 [initandlisten] Fri Sep 28 14:12:06 [initandlisten] db version v2.0.7, pdfile version 4.5