We have our mongo servers fail unpredictably, but consistent in that after a few days, the mongo server would fail. It typically is the primary mongo server in a replica set. According to a brief google search, it has to do with the SM limit for JS in spidermonkey.

The stacktrace is always of the form:

48197) } } bytes:175 nreturned:1 2ms
Sun Aug 14 05:05:35 [conn7714] insert weeklyad.system.indexes 0ms
Sun Aug 14 05:05:35 [conn7714] run command weeklyad.$cmd

{ getlasterror: 1 }

Sun Aug 14 05:05:35 [conn7714] query weeklyad.$cmd ntoreturn:1 command:

{ getlasterror: 1 }

reslen:99 0ms
Sun Aug 14 05:05:35 [conn17] getmore local.oplog.rs cid:6415614672211066276 getMore: { ts:

{ $gte: new Date(56397257525606 48197) }

} bytes:330 nreturned:2 2ms
Sun Aug 14 05:05:35 [initandlisten] connection accepted from 10.181.225.6:44071 #7715
Sun Aug 14 05:05:35 [conn7715] run command admin.$cmd

{ ismaster: 1 }

Sun Aug 14 05:05:35 [conn7715] query admin.$cmd ntoreturn:1 command:

{ ismaster: 1 }

reslen:216 0ms
Sun Aug 14 05:05:35 [conn7715] end connection 10.181.225.6:44071
Sun Aug 14 05:05:35 [initandlisten] connection accepted from 10.181.225.6:44074 #7716
Sun Aug 14 05:05:35 [conn7716] run command admin.$cmd

{ ismaster: 1 }

Sun Aug 14 05:05:35 [conn7716] query admin.$cmd ntoreturn:1 command:

{ ismaster: 1 }

reslen:216 0ms
Sun Aug 14 05:05:35 [conn4892] JS Error: out of memory nofile_a:0
Sun Aug 14 05:05:35 [conn4892] weeklyad.delayed_jobs Assertion failure JS_SetProperty( cx , obj , s.c_str() , &val ) scri
pting/engine_spidermonkey.cpp 1048
0x54f0ee 0x56073d 0x618c5d 0x9855b9 0x985196 0x9868f6 0x9668ae 0x956020 0x956469 0x913df2 0x62462c 0x5ee230 0x7402bc 0x66e
bac 0x78f0f6 0x7985c5 0x79be44 0x79f624 0x6534b1 0x65abfe
/usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54f0ee]
/usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfd) [0x56073d]
/usr/bin/mongod(ZN5mongo16resolveBSONFieldEP9JSContextP8JSObjectljPS3+0x3bd) [0x618c5d]
/usr/bin/mongod(js_LookupPropertyWithFlags+0x421) [0x9855b9]
/usr/bin/mongod(js_LookupProperty+0x49) [0x985196]
/usr/bin/mongod(js_GetProperty+0xff) [0x9868f6]
/usr/bin/mongod(js_Interpret+0xed54) [0x9668ae]
/usr/bin/mongod(js_Invoke+0xef2) [0x956020]
/usr/bin/mongod(js_InternalInvoke+0x189) [0x956469]
/usr/bin/mongod(JS_CallFunction+0x56) [0x913df2]
/usr/bin/mongod(_ZN5mongo7SMScope6invokeEP10JSFunctionRKNS_7BSONObjEib+0x2ec) [0x62462c]
/usr/bin/mongod(_ZN5mongo7Matcher7matchesERKNS_7BSONObjEPNS_12MatchDetailsE+0xf90) [0x5ee230]
/usr/bin/mongod(_ZN5mongo19CoveredIndexMatcher7matchesERKNS_7BSONObjERKNS_7DiskLocEPNS_12MatchDetailsEb+0xec) [0x7402bc]
/usr/bin/mongod(_ZN5mongo8UpdateOp4nextEv+0xbc) [0x66ebac]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) [0x78f0f6]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner3runEv+0x8c5) [0x7985c5]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x284) [0x79be44]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x64) [0x79f624]
-More-