[SERVER-9534] mongos v2.4.0 eats all memory, possible memory leak Created: 02/May/13  Updated: 10/May/13  Resolved: 10/May/13

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.4.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Yu Feng Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

linux


Attachments: File 31313.smaps    
Issue Links:
Related
is related to SERVER-8720 Memory leak in DBClientReplicaSet::sl... Closed
is related to SERVER-9139 cursor leak in mongos of unsharded cu... Closed
Operating System: ALL
Steps To Reproduce:

There is only one db which has two shards with each shard backed by 4 replica.
Aggregation jobs which uses the new aggregation framework runs continuously, several per minute. There are also normal upserts and finds with similar qps that of aggregations. Under this job load, it takes more than one month for the mongos to reach a RES memory of 14g+ with 30g+ of VIRT.
No authentication is used.

Participants:

 Description   

Mongos consumes 14g+ of the 16g physical memory, smaps file is attached.
The behavior is very similar to that of https://jira.mongodb.org/browse/SERVER-6354 and https://jira.mongodb.org/browse/SERVER-6785 , but both have been resolved before 2.4.0.

Detail of mongos version:
$mongos --version
MongoS version 2.4.0 starting: pid=31557 port=27017 64-bit host=xxxx (--help for usage)
git version: ce2d666c04b4a80af58e8bbb3388b0680e8cfeb6
build sys info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49

$ ldd mongos
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003f18800000)
librt.so.1 => /lib64/librt.so.1 (0x0000003f1c800000)
libstdc+.so.6 => /usr/lib64/libstdc+.so.6 (0x0000003f1bc00000)
libm.so.6 => /lib64/libm.so.6 (0x0000003f18400000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003f1b800000)
libc.so.6 => /lib64/libc.so.6 (0x0000003f17c00000)
/lib64/ld-linux-x86-64.so.2 (0x0000003f16c00000)



 Comments   
Comment by Judotens Budiarto [ 10/May/13 ]

i've same issue with Yu Feng and tried the 2.4.3 for last 4 days. it solved!

thanks

Comment by Tad Marshall [ 10/May/13 ]

Hi Yu,

Thanks for letting us know!

Tad

Comment by Yu Feng [ 10/May/13 ]

The memory of version 2.4.3 stayed stable after 3 days of running, this issue may be closed.

Comment by Yu Feng [ 03/May/13 ]

Thanks, Tad. I'll try the new versions and report the result here.

Comment by Christophe Spy [ 02/May/13 ]

ok I do that but, in 2.4.1 I have the same memory leak than Yu Feng, I will follow the 2 problem

Comment by Tad Marshall [ 02/May/13 ]

Hi Christophe,

Thanks for the stack trace!

This looks like a separate problem, and could be new. The stack trace shows that our V8 interface code is trying to allocate 128 MB for an object, which would never have worked in any version of MongoDB.

Can you open a separate ticket for this problem and include information on the operations that were going on when the problem happened? Steps to reproduce would be ideal, but any information on how to get mongod into this state would be very helpful.

Thank you!

Tad

Comment by Christophe Spy [ 02/May/13 ]

I have the same behavior : mongod consumme 100% of my RAM (32g) and then the server swap and oom killer kill mongod. I use mongo 2.4.1 (no shard, only replica set).
Previously I used 2.2.3 and didn't have any problem. I downgrade to 2.2.3 and server/RAM is ok

I try 2.4.3 but after some hours mongod crash with another error :

Thu May 2 13:46:03.265 [conn116] Assertion: 13548:BufBuilder attempted to grow() to 134217728 bytes, past the 64MB limit.
0xdcf361 0xd90a1b 0x6e6bc0 0xd70e9e 0xd7074d 0xd713ef 0xd700fc 0xd70a9e 0xd713ef 0xd7198e 0xd64831 0x869d7a 0x86bb00 0x8d236a 0x8d5535 0x8d6592 0xa7c97b 0xa80360 0x9f44d4 0x9f57e2
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcf361]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd90a1b]
/usr/bin/mongod(_ZN5mongo11_BufBuilderINS_16TrivialAllocatorEE15grow_reallocateEv+0xf0) [0x6e6bc0]
/usr/bin/mongod(_ZN5mongo7V8Scope16v8ToMongoElementERNS_14BSONObjBuilderERKSsN2v86HandleINS5_5ValueEEEiPNS_7BSONObjE+0xb3e) [0xd70e9e]
/usr/bin/mongod(_ZN5mongo7V8Scope16v8ToMongoElementERNS_14BSONObjBuilderERKSsN2v86HandleINS5_5ValueEEEiPNS_7BSONObjE+0x3ed) [0xd7074d]
/usr/bin/mongod(_ZN5mongo7V8Scope9v8ToMongoEN2v86HandleINS1_6ObjectEEEi+0x27f) [0xd713ef]
/usr/bin/mongod(_ZN5mongo7V8Scope15v8ToMongoObjectERNS_14BSONObjBuilderERKSsN2v86HandleINS5_5ValueEEEiPNS_7BSONObjE+0x2bc) [0xd700fc]
/usr/bin/mongod(_ZN5mongo7V8Scope16v8ToMongoElementERNS_14BSONObjBuilderERKSsN2v86HandleINS5_5ValueEEEiPNS_7BSONObjE+0x73e) [0xd70a9e]
/usr/bin/mongod(_ZN5mongo7V8Scope9v8ToMongoEN2v86HandleINS1_6ObjectEEEi+0x27f) [0xd713ef]
/usr/bin/mongod(_ZN5mongo7V8Scope9getObjectEPKc+0xbe) [0xd7198e]
/usr/bin/mongod(_ZN5mongo11PooledScope9getObjectEPKc+0x11) [0xd64831]
...

Comment by Tad Marshall [ 02/May/13 ]

There were several memory-related issues addressed in version 2.4.2, including SERVER-8720 and SERVER-9139, both of which could cause mongos to consume excessive memory.

Can you try version 2.4.3, which includes all of the 2.4.2 fixes and is the current 2.4 version, and see if your problem is resolved?

Generated at Thu Feb 08 03:20:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.