Crash #1 2014-11-04T00:52:25.211+0000 [conn47651] getmore plover_development.socialMediaUserFollower cursorid:597738160359 ntoreturn:0 keyUpdates:0 numYields:13 locks(micros) r:17258 nreturned:1613 reslen:64540 566ms 2014-11-04T00:52:25.271+0000 [conn43137] getmore plover_development.latestInteraction cursorid:843041132555 ntoreturn:0 keyUpdates:0 numYields:34 locks(micros) r:48259 nreturned:675 reslen:38495 1003ms 2014-11-04T00:52:25.272+0000 [conn45726] getmore plover_development.latestInteraction cursorid:842903787214 ntoreturn:0 keyUpdates:0 numYields:38 locks(micros) r:56022 nreturned:675 reslen:38495 1038ms 2014-11-04T00:52:25.384+0000 [conn44163] command plover_development.$cmd command: aggregate { aggregate: "statusUpdate", pipeline: [ { $match: { missions: ObjectId('52bcb8cfe4b0a8e4b730bb35'), postTime: { $gte: new Date(1414975944605) }, streams: "home" } }, { $unwind: "$keywords" }, { $group: { _id: "$keywords", count: { $sum: { $const: 1 } }, newest: { $max: "$postTime" } } } ], fromRouter: true } ntoreturn:1 keyUpdates:0 numYields:34 locks(micros) r:226790 reslen:927733 779ms 2014-11-04T00:52:46.773+0000 [conn47831] end connection 10.234.8.93:41671 (811 connections now open) 2014-11-04T00:52:46.779+0000 [initandlisten] connection accepted from 10.234.8.93:41821 #47833 (812 connections now open) 2014-11-04T00:52:47.928+0000 [conn47832] end connection 10.31.102.139:40846 (811 connections now open) 2014-11-04T00:52:47.929+0000 [initandlisten] connection accepted from 10.31.102.139:40850 #47834 (812 connections now open) 2014-11-04T00:52:51.361+0000 [LockPinger] cluster config1.db.meshfire.com:27019,config2.db.meshfire.com:27020,config3.db.meshfire.com:27019 pinged successfully at Tue Nov 4 00:52:51 2014 by distributed lock pinger 'config1.db.meshfire.com:27019,config2.db.meshfire.com:27020,config3.db.meshfire.com:27019/ip-10-61-173-4:27017:1414496413:2103741112', sleeping for 30000ms 2014-11-04T00:53:00.775+0000 [conn43138] key seems to have moved in the index, refinding. 176:2d6d5000 2014-11-04T00:53:01.490+0000 [conn43087] key seems to have moved in the index, refinding. 176:2d7cb000 2014-11-04T00:53:02.109+0000 [conn47628] SEVERE: Invalid access at address: 0 2014-11-04T00:53:02.146+0000 [conn47628] SEVERE: Got signal: 11 (Segmentation fault). Backtrace:0x11e9b11 0x11e8eee 0x11e8fdf 0x7fd2528d75b0 0xaad3b1 0xd571bc 0xd57cdd 0xd5adde 0xc4bbc9 0xc57926 0xb9a935 0xba2f10 0x770d5f 0x119f93e 0x7fd2528cff18 0x7fd251be5b9d mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e9b11] mongod() [0x11e8eee] mongod() [0x11e8fdf] /lib64/libpthread.so.0(+0xf5b0) [0x7fd2528d75b0] mongod(_ZN5mongo18KeepMutationsStage4workEPm+0xc1) [0xaad3b1] mongod(_ZN5mongo15MultiPlanRunner12workAllPlansEPNS_7BSONObjEm+0x13c) [0xd571bc] mongod(_ZN5mongo15MultiPlanRunner12pickBestPlanEPmPNS_7BSONObjE+0xed) [0xd57cdd] mongod(_ZN5mongo15MultiPlanRunner7getNextEPNS_7BSONObjEPNS_7DiskLocE+0x34e) [0xd5adde] mongod(_ZN5mongo6updateERKNS_13UpdateRequestEPNS_7OpDebugEPNS_12UpdateDriverEPNS_14CanonicalQueryE+0x9a9) [0xc4bbc9] mongod(_ZN5mongo14UpdateExecutor7executeEv+0x66) [0xc57926] mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x6a5) [0xb9a935] mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xe60) [0xba2f10] mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x770d5f] mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4ee) [0x119f93e] /lib64/libpthread.so.0(+0x7f18) [0x7fd2528cff18] /lib64/libc.so.6(clone+0x6d) [0x7fd251be5b9d] Crash #2 2014-11-04T14:11:03.948+0000 [conn3182] CMD: drop plover_development.tmp.mr.socialMediaUserFollower_149 2014-11-04T14:11:13.027+0000 [conn3531] end connection 10.31.102.139:54536 (396 connections now open) 2014-11-04T14:11:13.028+0000 [initandlisten] connection accepted from 10.31.102.139:54540 #3533 (397 connections now open) 2014-11-04T14:11:13.905+0000 [conn2817] key seems to have moved in the index, refinding. 151:7cf49000 2014-11-04T14:11:13.919+0000 [conn1726] update plover_development.kanbanCard query: { _id: { $ne: ObjectId('54578d00e4b0668ee10b5162') }, mission: ObjectId('54578cf0e4b089ca8e88aee6'), archived: { $ne: true }, state: { $in: [ "READY", "IN_PROGRESS" ] }, priority: { $gt: 220 } } update: { $inc: { priority: -1 } } nscanned:95 nscannedObjects:93 nMatched:14 nModified:14 keyUpdates:2 numYields:57 locks(micros) w:14927 656ms 2014-11-04T14:11:16.527+0000 [conn1813] update plover_development.kanbanCard query: { _id: { $ne: ObjectId('54578d02e4b0e85dd0096ca2') }, mission: ObjectId('54578cf0e4b089ca8e88aee6'), archived: { $ne: true }, state: { $in: [ "READY", "IN_PROGRESS" ] }, priority: { $gt: 169 } } update: { $inc: { priority: -1 } } nscanned:63 nscannedObjects:61 nMatched:61 nModified:61 keyUpdates:2 numYields:28 locks(micros) w:12787 716ms 2014-11-04T14:11:17.012+0000 [conn2840] update plover_development.kanbanCard query: { _id: { $ne: ObjectId('54578d01e4b0aa2b90cd7fa0') }, mission: ObjectId('54578cf0e4b089ca8e88aee6'), archived: { $ne: true }, state: { $in: [ "READY", "IN_PROGRESS" ] }, priority: { $gt: 180 } } update: { $inc: { priority: -1 } } nscanned:38 nscannedObjects:36 nMatched:0 nModified:0 keyUpdates:0 numYields:33 locks(micros) w:7761 847ms 2014-11-04T14:11:17.159+0000 [conn2816] update plover_development.kanbanCard query: { _id: { $ne: ObjectId('54578d03e4b0e85dd0096cd4') }, mission: ObjectId('54578cf0e4b089ca8e88aee6'), archived: { $ne: true }, state: { $in: [ "READY", "IN_PROGRESS" ] }, priority: { $gt: 158 } } update: { $inc: { priority: -1 } } nscanned:46 nscannedObjects:44 nMatched:0 nModified:0 keyUpdates:0 numYields:41 locks(micros) w:8886 887ms 2014-11-04T14:11:17.181+0000 [conn1722] getmore plover_development.socialMediaUserFollower cursorid:598506130456 ntoreturn:0 keyUpdates:0 numYields:14 locks(micros) r:15165 nreturned:1721 reslen:68860 661ms 2014-11-04T14:11:17.224+0000 [conn1713] update plover_development.kanbanCard query: { _id: { $ne: ObjectId('54578d02e4b089ca8e88b173') }, mission: ObjectId('54578cf0e4b089ca8e88aee6'), archived: { $ne: true }, state: { $in: [ "READY", "IN_PROGRESS" ] }, priority: { $gt: 168 } } update: { $inc: { priority: -1 } } nscanned:64 nscannedObjects:62 nMatched:0 nModified:0 keyUpdates:0 numYields:57 locks(micros) w:13173 1411ms 2014-11-04T14:11:17.875+0000 [conn2136] getmore plover_development.socialMediaUserFollower cursorid:598133400552 ntoreturn:0 keyUpdates:0 numYields:74 locks(micros) r:75436 nreturned:9471 reslen:378860 3134ms 2014-11-04T14:11:17.914+0000 [conn2816] SEVERE: Invalid access at address: 0x3 2014-11-04T14:11:17.950+0000 [conn2816] SEVERE: Got signal: 11 (Segmentation fault). Backtrace:0x11e9b11 0x11e8eee 0x11e8fdf 0x7f45c115a5b0 0xaad3b1 0xd571bc 0xd57cdd 0xd5adde 0xc4bbc9 0xc57926 0xb9a935 0xba2f10 0x770d5f 0x119f93e 0x7f45c1152f18 0x7f45c0468b9d mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e9b11] mongod() [0x11e8eee] mongod() [0x11e8fdf] /lib64/libpthread.so.0(+0xf5b0) [0x7f45c115a5b0] mongod(_ZN5mongo18KeepMutationsStage4workEPm+0xc1) [0xaad3b1] mongod(_ZN5mongo15MultiPlanRunner12workAllPlansEPNS_7BSONObjEm+0x13c) [0xd571bc] mongod(_ZN5mongo15MultiPlanRunner12pickBestPlanEPmPNS_7BSONObjE+0xed) [0xd57cdd] mongod(_ZN5mongo15MultiPlanRunner7getNextEPNS_7BSONObjEPNS_7DiskLocE+0x34e) [0xd5adde] mongod(_ZN5mongo6updateERKNS_13UpdateRequestEPNS_7OpDebugEPNS_12UpdateDriverEPNS_14CanonicalQueryE+0x9a9) [0xc4bbc9] mongod(_ZN5mongo14UpdateExecutor7executeEv+0x66) [0xc57926] mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x6a5) [0xb9a935] mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xe60) [0xba2f10] mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x770d5f] mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4ee) [0x119f93e] /lib64/libpthread.so.0(+0x7f18) [0x7f45c1152f18] /lib64/libc.so.6(clone+0x6d) [0x7f45c0468b9d]