2015-03-23T10:20:04.610+0000 I QUERY [conn31959] getmore scorr.normalized_profiles cursorid:25854140394 ntoreturn:0 keyUpdates:0 writeConflicts:0 numYields:308 nreturned:592 reslen:4196557 locks:{} 27700ms 2015-03-23T10:20:04.659+0000 I NETWORK [initandlisten] connection accepted from 10.80.144.170:38899 #32032 (31984 connections now open) 2015-03-23T10:20:04.787+0000 I QUERY [conn31818] getmore scorr.normalized_profiles cursorid:26802534337 ntoreturn:0 keyUpdates:0 writeConflicts:0 numYields:266 nreturned:550 reslen:4208679 locks:{} 21486ms 2015-03-23T10:20:04.792+0000 I NETWORK [initandlisten] connection accepted from 10.81.150.221:40599 #32033 (31985 connections now open) 2015-03-23T10:20:04.798+0000 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable 2015-03-23T10:20:04.798+0000 I NETWORK [initandlisten] can't create new thread, closing connection 2015-03-23T10:20:04.808+0000 I NETWORK [initandlisten] connection accepted from 10.80.144.170:38901 #32034 (31985 connections now open) 2015-03-23T10:20:04.808+0000 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable 2015-03-23T10:20:04.808+0000 I NETWORK [initandlisten] can't create new thread, closing connection 2015-03-23T10:20:04.822+0000 I QUERY [conn31514] getmore scorr.normalized_profiles cursorid:26600827105 ntoreturn:0 keyUpdates:0 writeConflicts:0 numYields:27 nreturned:36 reslen:334751 locks:{} 1787ms 2015-03-23T10:20:04.842+0000 I QUERY [conn32024] query scorr.normalized_profiles query: { $min: { _id: "vk158422961" }, $max: { _id: "vk158429480" }, $query: {} } planSummary: IXSCAN { _id: 1 } cursorid:27485609343 ntoreturn:0 ntoskip:0 nscanned:102 nscannedObjects:101 keyUpdates:0 writeConflicts:0 numYields:65 nreturned:101 reslen:796422 locks:{} 3020ms 2015-03-23T10:20:04.985+0000 I QUERY [conn31784] getmore scorr.normalized_profiles cursorid:27690416341 ntoreturn:0 keyUpdates:0 writeConflicts:0 numYields:259 nreturned:508 reslen:4196539 locks:{} 23379ms 2015-03-23T10:20:05.144+0000 I QUERY [conn31932] getmore scorr.normalized_profiles cursorid:26004341831 ntoreturn:0 keyUpdates:0 writeConflicts:0 numYields:281 nreturned:589 reslen:4199142 locks:{} 16491ms 2015-03-23T10:20:05.177+0000 I NETWORK [initandlisten] connection accepted from 10.80.144.170:38902 #32035 (31985 connections now open) 2015-03-23T10:20:05.220+0000 E STORAGE WiredTiger (11) [1427106005:218730][20485:0x7f65dea5c700], eviction-server: pthread_create: Resource temporarily unavailable 2015-03-23T10:20:05.220+0000 E STORAGE WiredTiger (11) [1427106005:220817][20485:0x7f65dea5c700], eviction-server: cache eviction server error: Resource temporarily unavailable 2015-03-23T10:20:05.220+0000 E STORAGE WiredTiger (-31804) [1427106005:220845][20485:0x7f65dea5c700], eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic 2015-03-23T10:20:05.221+0000 I - [conn31834] Fatal Assertion 28559 2015-03-23T10:20:05.221+0000 I - [conn31593] Fatal Assertion 28559 2015-03-23T10:20:05.221+0000 I - Fatal Assertion 28558 2015-03-23T10:20:05.221+0000 I - [conn31813] Fatal Assertion 28559 2015-03-23T10:20:05.221+0000 I - [conn31611] Fatal Assertion 28559 2015-03-23T10:20:05.221+0000 I - [conn31604] Fatal Assertion 28559 2015-03-23T10:20:05.221+0000 I - [conn31849] Fatal Assertion 28559 2015-03-23T10:20:05.221+0000 I - [conn31880] Fatal Assertion 28559 2015-03-23T10:20:05.222+0000 I - [conn31805] Fatal Assertion 28559 2015-03-23T10:20:05.222+0000 I - [conn31560] Fatal Assertion 28559 2015-03-23T10:20:05.222+0000 I - [conn31674] Fatal Assertion 28559 2015-03-23T10:20:05.223+0000 I - [conn31974] Fatal Assertion 28559 2015-03-23T10:20:05.223+0000 I - [conn31940] Fatal Assertion 28559 2015-03-23T10:20:05.223+0000 I - [conn31844] Fatal Assertion 28559 2015-03-23T10:20:05.224+0000 I - [conn31544] Fatal Assertion 28559 2015-03-23T10:20:05.224+0000 I - [conn31774] Fatal Assertion 28559 2015-03-23T10:20:05.224+0000 I - [conn31944] Fatal Assertion 28559 2015-03-23T10:20:05.225+0000 I - [conn31632] Fatal Assertion 28559 2015-03-23T10:20:05.225+0000 I - [conn31986] Fatal Assertion 28559 2015-03-23T10:20:05.225+0000 I - [conn31893] Fatal Assertion 28559 2015-03-23T10:20:05.226+0000 I - [conn31718] Fatal Assertion 28559 2015-03-23T10:20:05.226+0000 I - [conn31984] Fatal Assertion 28559 2015-03-23T10:20:05.226+0000 I - [conn31896] Fatal Assertion 28559 2015-03-23T10:20:05.227+0000 I - [conn31714] Fatal Assertion 28559 2015-03-23T10:20:05.227+0000 I - [conn31992] Fatal Assertion 28559 2015-03-23T10:20:05.227+0000 I - [conn31959] Fatal Assertion 28559 2015-03-23T10:20:05.228+0000 I - [conn31748] Fatal Assertion 28559 2015-03-23T10:20:05.228+0000 I - [conn31846] Fatal Assertion 28559 2015-03-23T10:20:05.228+0000 I - [conn31556] Fatal Assertion 28559 2015-03-23T10:20:05.229+0000 I - [conn31972] Fatal Assertion 28559 2015-03-23T10:20:05.229+0000 I - [conn31982] Fatal Assertion 28559 2015-03-23T10:20:05.229+0000 I - [conn31644] Fatal Assertion 28559 2015-03-23T10:20:05.230+0000 I - [conn31570] Fatal Assertion 28559 2015-03-23T10:20:05.230+0000 I - [conn31736] Fatal Assertion 28559 2015-03-23T10:20:05.230+0000 I - [conn31895] Fatal Assertion 28559 2015-03-23T10:20:05.231+0000 I - [conn31708] Fatal Assertion 28559 2015-03-23T10:20:05.231+0000 I - [conn31633] Fatal Assertion 28559 2015-03-23T10:20:05.231+0000 I - [conn31772] Fatal Assertion 28559 2015-03-23T10:20:05.231+0000 I - [conn31606] Fatal Assertion 28559 2015-03-23T10:20:05.232+0000 I - [conn31702] Fatal Assertion 28559 2015-03-23T10:20:05.232+0000 I - [conn31768] Fatal Assertion 28559 2015-03-23T10:20:05.233+0000 I - [conn31858] Fatal Assertion 28559 2015-03-23T10:20:05.233+0000 I - [conn31688] Fatal Assertion 28559 2015-03-23T10:20:05.233+0000 I - [conn31922] Fatal Assertion 28559 2015-03-23T10:20:05.233+0000 I - [conn31802] Fatal Assertion 28559 2015-03-23T10:20:05.234+0000 I - [conn31572] Fatal Assertion 28559 2015-03-23T10:20:05.234+0000 I - [conn31853] Fatal Assertion 28559 2015-03-23T10:20:05.234+0000 I - [conn31569] Fatal Assertion 28559 2015-03-23T10:20:05.235+0000 I - [conn31713] Fatal Assertion 28559 2015-03-23T10:20:05.235+0000 I - [conn31552] Fatal Assertion 28559 2015-03-23T10:20:05.235+0000 I - [conn31600] Fatal Assertion 28559 2015-03-23T10:20:05.236+0000 I - [conn31616] Fatal Assertion 28559 2015-03-23T10:20:05.236+0000 I - [conn31764] Fatal Assertion 28559 2015-03-23T10:20:05.236+0000 I - [conn31840] Fatal Assertion 28559 2015-03-23T10:20:05.237+0000 I - [conn31598] Fatal Assertion 28559 2015-03-23T10:20:05.237+0000 I - [conn31634] Fatal Assertion 28559 2015-03-23T10:20:05.237+0000 I - [conn31730] Fatal Assertion 28559 2015-03-23T10:20:05.238+0000 I - [conn31677] Fatal Assertion 28559 2015-03-23T10:20:05.238+0000 I - [conn31829] Fatal Assertion 28559 2015-03-23T10:20:05.238+0000 I - [conn31655] Fatal Assertion 28559 2015-03-23T10:20:05.239+0000 I - [conn31704] Fatal Assertion 28559 2015-03-23T10:20:05.239+0000 I - [conn31779] Fatal Assertion 28559 2015-03-23T10:20:05.239+0000 I - [conn31760] Fatal Assertion 28559 2015-03-23T10:20:05.240+0000 I - [conn31936] Fatal Assertion 28559 2015-03-23T10:20:05.240+0000 I - [conn31700] Fatal Assertion 28559 2015-03-23T10:20:05.240+0000 I - [conn31646] Fatal Assertion 28559 2015-03-23T10:20:05.241+0000 I - [conn32008] Fatal Assertion 28559 2015-03-23T10:20:05.241+0000 I - [conn31898] Fatal Assertion 28559 2015-03-23T10:20:05.241+0000 I - [conn32012] Fatal Assertion 28559 2015-03-23T10:20:05.242+0000 I - [conn31854] Fatal Assertion 28559 2015-03-23T10:20:05.242+0000 I - [conn32010] Fatal Assertion 28559 2015-03-23T10:20:05.243+0000 I - [conn31678] Fatal Assertion 28559 2015-03-23T10:20:05.243+0000 I - [conn31870] Fatal Assertion 28559 2015-03-23T10:20:05.243+0000 I - [conn31580] Fatal Assertion 28559 2015-03-23T10:20:05.243+0000 I - [conn31628] Fatal Assertion 28559 2015-03-23T10:20:05.244+0000 I - [conn31566] Fatal Assertion 28559 2015-03-23T10:20:05.244+0000 I - [conn31862] Fatal Assertion 28559 2015-03-23T10:20:05.244+0000 I - [conn31938] Fatal Assertion 28559 2015-03-23T10:20:05.244+0000 I NETWORK [initandlisten] connection accepted from 10.81.150.221:40600 #32036 (31986 connections now open) 2015-03-23T10:20:05.245+0000 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable 2015-03-23T10:20:05.245+0000 I NETWORK [initandlisten] can't create new thread, closing connection 2015-03-23T10:20:05.245+0000 I - [conn31690] Fatal Assertion 28559 2015-03-23T10:20:05.245+0000 I - [conn31785] Fatal Assertion 28559 2015-03-23T10:20:05.245+0000 I - [conn31537] Fatal Assertion 28559 2015-03-23T10:20:05.246+0000 I - [conn32016] Fatal Assertion 28559 2015-03-23T10:20:05.246+0000 I - [conn31732] Fatal Assertion 28559 2015-03-23T10:20:05.246+0000 I - [conn31716] Fatal Assertion 28559 2015-03-23T10:20:05.247+0000 I - [conn31554] Fatal Assertion 28559 2015-03-23T10:20:05.247+0000 I - [conn31911] Fatal Assertion 28559 2015-03-23T10:20:05.247+0000 I NETWORK [initandlisten] connection accepted from 10.81.150.221:40601 #32037 (31986 connections now open) 2015-03-23T10:20:05.247+0000 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable 2015-03-23T10:20:05.247+0000 I NETWORK [initandlisten] can't create new thread, closing connection 2015-03-23T10:20:05.248+0000 I - [conn31794] Fatal Assertion 28559 2015-03-23T10:20:05.248+0000 I - [conn31970] Fatal Assertion 28559 2015-03-23T10:20:05.248+0000 I - [conn31962] Fatal Assertion 28559 2015-03-23T10:20:05.248+0000 I - [conn31699] Fatal Assertion 28559 2015-03-23T10:20:05.249+0000 I - [conn31721] Fatal Assertion 28559 2015-03-23T10:20:05.249+0000 I - [conn31885] Fatal Assertion 28559 2015-03-23T10:20:05.250+0000 I - [conn31618] Fatal Assertion 28559 2015-03-23T10:20:05.250+0000 I - [conn31594] Fatal Assertion 28559 2015-03-23T10:20:05.250+0000 I - [conn31758] Fatal Assertion 28559 2015-03-23T10:20:05.250+0000 I - [conn31956] Fatal Assertion 28559 2015-03-23T10:20:05.251+0000 I - [conn31963] Fatal Assertion 28559 2015-03-23T10:20:05.251+0000 I - [conn31826] Fatal Assertion 28559 2015-03-23T10:20:05.251+0000 I - [conn31810] Fatal Assertion 28559 2015-03-23T10:20:05.251+0000 I - [conn31886] Fatal Assertion 28559 2015-03-23T10:20:05.252+0000 I - [conn32015] Fatal Assertion 28559 2015-03-23T10:20:05.252+0000 I - [conn31988] Fatal Assertion 28559 2015-03-23T10:20:05.252+0000 I - [conn31842] Fatal Assertion 28559 2015-03-23T10:20:05.253+0000 I - [conn31608] Fatal Assertion 28559 2015-03-23T10:20:05.253+0000 I - [conn31660] Fatal Assertion 28559 2015-03-23T10:20:05.253+0000 I - [conn31948] Fatal Assertion 28559 2015-03-23T10:20:05.254+0000 I - [conn31952] Fatal Assertion 28559 2015-03-23T10:20:05.254+0000 I - [conn31930] Fatal Assertion 28559 2015-03-23T10:20:05.254+0000 I - [conn31909] Fatal Assertion 28559 2015-03-23T10:20:05.255+0000 I - [conn31836] Fatal Assertion 28559 2015-03-23T10:20:05.255+0000 I - [conn31815] Fatal Assertion 28559 2015-03-23T10:20:05.255+0000 I - [conn31525] Fatal Assertion 28559 2015-03-23T10:20:05.256+0000 I - [conn31707] Fatal Assertion 28559 2015-03-23T10:20:05.256+0000 I - [conn31683] Fatal Assertion 28559 2015-03-23T10:20:05.256+0000 I - [conn31879] Fatal Assertion 28559 2015-03-23T10:20:05.257+0000 I - [conn31994] Fatal Assertion 28559 2015-03-23T10:20:05.257+0000 I - [conn31650] Fatal Assertion 28559 2015-03-23T10:20:05.257+0000 I - [conn31664] Fatal Assertion 28559 2015-03-23T10:20:05.258+0000 I - [conn31990] Fatal Assertion 28559 2015-03-23T10:20:05.258+0000 I - [conn31741] Fatal Assertion 28559 2015-03-23T10:20:05.259+0000 I - [conn31546] Fatal Assertion 28559 2015-03-23T10:20:05.259+0000 I - [conn31602] Fatal Assertion 28559 2015-03-23T10:20:05.259+0000 I - [conn31662] Fatal Assertion 28559 2015-03-23T10:20:05.260+0000 I - [conn31799] Fatal Assertion 28559 2015-03-23T10:20:05.260+0000 I - [conn31928] Fatal Assertion 28559 2015-03-23T10:20:05.260+0000 I - [conn31920] Fatal Assertion 28559 2015-03-23T10:20:05.261+0000 I - [conn31656] Fatal Assertion 28559 2015-03-23T10:20:05.261+0000 I - [conn31728] Fatal Assertion 28559 2015-03-23T10:20:05.262+0000 I - [conn32002] Fatal Assertion 28559 2015-03-23T10:20:05.277+0000 I CONTROL [conn31799] 0xf4d299 0xeeda71 0xed2661 0xd77aa0 0xd64e2f 0xd65416 0x911ef0 0xa02f3b 0xbd0264 0xbd0614 0xb9ca7a 0xab024e 0xab55dd 0x80e75d 0xf00e6b 0x7f65e0890182 0x7f65df35847d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"B4D299"},{"b":"400000","o":"AEDA71"},{"b":"400000","o":"AD2661"},{"b":"400000","o":"977AA0"},{"b":"400000","o":"964E2F"},{"b":"400000","o":"965416"},{"b":"400000","o":"511EF0"},{"b":"400000","o":"602F3B"},{"b":"400000","o":"7D0264"},{"b":"400000","o":"7D0614"},{"b":"400000","o":"79CA7A"},{"b":"400000","o":"6B024E"},{"b":"400000","o":"6B55DD"},{"b":"400000","o":"40E75D"},{"b":"400000","o":"B00E6B"},{"b":"7F65E0888000","o":"8182"},{"b":"7F65DF25E000","o":"FA47D"}],"processInfo":{ "mongodbVersion" : "3.0.1", "gitVersion" : "534b5a3f9d10f00cd27737fbcd951032248b5952", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "C35E766AD226FC0C16CB0C3885EC3B59E288A3F2" }, { "b" : "7FFF28EFE000", "elfType" : 3, "buildId" : "0074678E5FFFF79F46C476077E67057161772F37" }, { "b" : "7F65E0888000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7F65E062A000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "FF43D0947510134A8A494063A3C1CF3CEBB27791" }, { "b" : "7F65E024F000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "B927879B878D90DD9FF4B15B00E7799AA8E0272F" }, { "b" : "7F65E0047000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7F65DFE43000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7F65DFB3F000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "19EFDDAB11B3BF5C71570078C59F91CF6592CE9E" }, { "b" : "7F65DF839000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7F65DF623000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7F65DF25E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7F65E0AA6000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf4d299] mongod(_ZN5mongo10logContextEPKc+0xE1) [0xeeda71] mongod(_ZN5mongo13fassertFailedEi+0x61) [0xed2661] mongod(_ZN5mongo17wtRCToStatus_slowEiPKc+0x2D0) [0xd77aa0] mongod(_ZNK5mongo21WiredTigerRecordStore8_getDataERKNS_16WiredTigerCursorE+0x8F) [0xd64e2f] mongod(_ZNK5mongo21WiredTigerRecordStore7dataForEPNS_16OperationContextERKNS_8RecordIdE+0x86) [0xd65416] mongod(_ZNK5mongo10Collection6docForEPNS_16OperationContextERKNS_8RecordIdE+0x20) [0x911ef0] mongod(_ZN5mongo10FetchStage4workEPm+0x2DB) [0xa02f3b] mongod(_ZN5mongo12PlanExecutor18getNextSnapshottedEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0xA4) [0xbd0264] mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x34) [0xbd0614] mongod(_ZN5mongo7getMoreEPNS_16OperationContextEPKcixRNS_5CurOpEiRbPb+0x4BA) [0xb9ca7a] mongod(_ZN5mongo15receivedGetMoreEPNS_16OperationContextERNS_10DbResponseERNS_7MessageERNS_5CurOpE+0x35E) [0xab024e] mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x13CD) [0xab55dd] mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xDD) [0x80e75d] mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x34B) [0xf00e6b] libpthread.so.0(+0x8182) [0x7f65e0890182] libc.so.6(clone+0x6D) [0x7f65df35847d] ----- END BACKTRACE ----- 2015-03-23T10:20:05.280+0000 I - [conn31799] ***aborting after fassert() failure