[SERVER-2532] JS out of memory if collections are continuously created and dropped Created: 11/Feb/11  Updated: 12/Jul/16  Resolved: 12/Oct/11

Status: Closed
Project: Core Server
Component/s: JavaScript, MapReduce
Affects Version/s: 1.7.5
Fix Version/s: 2.1.0

Type: Bug Priority: Major - P3
Reporter: Reinaldo Giudici Assignee: Antoine Girbal
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Operating System: Linux
Participants:

 Description   

The collection that it references has about 2000 docs only.

We've seen this multiple times on 1.6.5, and we upgraded to 1.7.5 today and we still see it happening, this time happened 2 hours after the upgrade, after they happen they go for a long time until I step down the master

Here is part of the log with the error and the trace.

Thu Feb 10 20:02:36 [conn960] query ee895ca8156f4c7fa4104b1b4c9a8c38.$cmd ntoreturn:1 command: { group: { ns: "LEADEBOARD_cityofwonder-g-300_l", key:

{ value: true }

, cond: null, $reduce: "functi
on(doc,prev)

{ prev.numEntries ++ }", initial: { numEntries: 0 } } } reslen:28106 89617ms
Thu Feb 10 20:02:36 [conn809] query ee895ca8156f4c7fa4104b1b4c9a8c38.$cmd ntoreturn:1 command: { group: { ns: "LEADEBOARD_cityofwonder-g-300_l", key: { value: true }, cond: null, $reduce: "functi
on(doc,prev) { prev.numEntries ++ }

", initial:

{ numEntries: 0 }

} } reslen:28106 95479msThu Feb 10 20:02:36 [initandlisten] connection accepted from 10.36.115.8:41194 #3640
Thu Feb 10 20:02:37 [initandlisten] connection accepted from 10.36.115.6:45369 #3641
Thu Feb 10 20:02:37 [initandlisten] connection accepted from 10.36.115.6:45370 #3642
Thu Feb 10 20:02:37 [initandlisten] connection accepted from 10.36.115.8:41198 #3643
Thu Feb 10 20:02:37 [initandlisten] connection accepted from 10.36.105.84:28486 #3644
Thu Feb 10 20:02:37 [initandlisten] connection accepted from 10.36.115.6:45373 #3645
Thu Feb 10 20:02:37 [conn292] query ee895ca8156f4c7fa4104b1b4c9a8c38.$cmd ntoreturn:1 command: { group: { ns: "LEADEBOARD_cityofwonder-g-300_l", key:

{ value: true }

, cond: null, $reduce: "functi
on(doc,prev)

{ prev.numEntries ++ }", initial: { numEntries: 0 } } } reslen:28106 90503ms
Thu Feb 10 20:02:37 [conn277] getmore local.oplog.rs cid:3808073055396120208 getMore: { ts: { $gte: new Date(5572222262670524418) } } bytes:20 nreturned:0 97034ms
Thu Feb 10 20:02:37 [conn2346] killcursors: found 0 of 1
Thu Feb 10 20:02:37 [conn2346] killcursors: found 0 of 1
Thu Feb 10 20:02:41 [conn2354] JS Error: out of memory
Thu Feb 10 20:02:41 [conn2174] JS Error: out of memory
Thu Feb 10 20:02:41 [conn2174] Assertion: 10432:JS_NewObject failed for global
0x560819 0x61ddc7 0x6129c3 0x605f57 0x604b59 0x7f435b 0x7f6594 0x7d55e2 0x7d64e1 0x63fdc5 0x645e85 0x74dc0c 0x75399d 0x89a15e 0x8ad550 0x7f28d5e419ca 0x7f28d53f070d
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x129) [0x560819]
/usr/bin/mongod(_ZN5mongo7SMScopeC1Ev+0x247) [0x61ddc7]
/usr/bin/mongod(_ZN5mongo8SMEngine11createScopeEv+0x23) [0x6129c3]
/usr/bin/mongod(_ZN5mongo12ScriptEngine8newScopeEv+0x17) [0x605f57]
/usr/bin/mongod(_ZN5mongo12ScriptEngine14getPooledScopeERKSs+0x259) [0x604b59]
/usr/bin/mongod(_ZN5mongo12GroupCommand5groupESsRKSsRKNS_7BSONObjES3_SsSsPKcS3_SsRSsRNS_14BSONObjBuilderE+0x4b) [0x7f435b]
/usr/bin/mongod(_ZN5mongo12GroupCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0xe34) [0x7f6594]
/usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x9e2) [0x7d55e2]
/usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x851) [0x7d64e1]
/usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x63fdc5]
/usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x3005) [0x645e85]
/usr/bin/mongod() [0x74dc0c]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5ad) [0x75399d]
/usr/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x89a15e]
/usr/bin/mongod(thread_proxy+0x80) [0x8ad550]
/lib/libpthread.so.0(+0x69ca) [0x7f28d5e419ca]
/lib/libc.so.6(clone+0x6d) [0x7f28d53f070d]
Thu Feb 10 20:02:41 [conn2174] query ee895ca8156f4c7fa4104b1b4c9a8c38.$cmd ntoreturn:1 command: { group: { ns: "LEADEBOARD_cityofwonder-18-1-300_l", key: { value: true }, cond: null, $reduce: "function(doc,prev) { prev.numEntries ++ }

", initial:

{ numEntries: 0 }

} } reslen:117 52518ms
Thu Feb 10 20:02:41 [conn1563] JS Error: out of memory
Thu Feb 10 20:02:41 [conn1563] Assertion: 10432:JS_NewObject failed for global
0x560819 0x61ddc7 0x6129c3 0x605f57 0x604b59 0x7f435b 0x7f6594 0x7d55e2 0x7d64e1 0x63fdc5 0x645e85 0x74dc0c 0x75399d 0x89a15e 0x8ad550 0x7f28d5e419ca 0x7f28d53f070d
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x129) [0x560819]
/usr/bin/mongod(_ZN5mongo7SMScopeC1Ev+0x247) [0x61ddc7]
/usr/bin/mongod(_ZN5mongo8SMEngine11createScopeEv+0x23) [0x6129c3]
/usr/bin/mongod(_ZN5mongo12ScriptEngine8newScopeEv+0x17) [0x605f57]
/usr/bin/mongod(_ZN5mongo12ScriptEngine14getPooledScopeERKSs+0x259) [0x604b59]
/usr/bin/mongod(_ZN5mongo12GroupCommand5groupESsRKSsRKNS_7BSONObjES3_SsSsPKcS3_SsRSsRNS_14BSONObjBuilderE+0x4b) [0x7f435b]
/usr/bin/mongod(_ZN5mongo12GroupCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0xe34) [0x7f6594]
/usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x9e2) [0x7d55e2]
/usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x851) [0x7d64e1]
/usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x63fdc5]



 Comments   
Comment by Antoine Girbal [ 12/Oct/11 ]

This issue got fixed along with SERVER-3528

Comment by auto [ 12/Oct/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-2532: JS out of memory if collections are continuously created and dropped
Branch: master
https://github.com/mongodb/mongo/commit/0198a3c09726c3e85956e8cbd0bf363d05c74ac4

Comment by Antoine Girbal [ 12/Oct/11 ]

Reinaldo,
have you seen this error happen again?
You are saying that switching from DROP to REMOVE fixed the issue.
Where you using random collection names every time?

I think there is a possible memory issue if code keeps on querying different collections, since the collection objects are cached within the JS scope.
So over time this could reach the 8MB of memory limit.
I think SERVER-3528 indirectly fixed this problem since contexts are not cached forever anymore.
Will create test to verify it.

Comment by yuan tiger [ 25/Apr/11 ]

me too!
strange phenomenonmark.
mark

Comment by Reinaldo Giudici [ 15/Feb/11 ]

Our previous implementation had a a DROP collection on certain operations and relied on mongo to create collections, We needed to fix this on a production system, so upgrading to nightly was too risky.
We have changed the code now to do a remove() instead of drop() and this JS problem seems to have gone away. So I suspect some interaction between group operations, atomic increments and dropping collections on a concurrent environment that was leaking memory

Comment by Kristina Chodorow (Inactive) [ 11/Feb/11 ]

There have been a couple of fixes to MapReduce's memory usage in the last couple of days. Could you try the nightly?

Generated at Thu Feb 08 03:00:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.