[SERVER-13743] Mongod killed by linux OOM under long running test load Created: 25/Apr/14  Updated: 10/Dec/14  Resolved: 13/May/14

Status: Closed
Project: Core Server
Component/s: Concurrency
Affects Version/s: 2.7.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Davide Italiano Assignee: Kaloian Manassiev
Resolution: Duplicate Votes: 0
Labels: 28qa, concurrency
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Source File MongoManyDBsStressTest.java     Text File memory_leak_before.txt     Text File memory_leak_latest.txt     PNG File server13743.png    
Issue Links:
Related
related to SERVER-13794 Unused snapshot history consuming sig... Closed
Operating System: Linux
Participants:

 Description   

This is probably due to a leak. Mongod runs out of memory and get kicked out from the OS. Memory (as reported by top(1)) looks growing linearly with time.
Scenario:
https://github.com/dcci/mongo-stress/blob/master/createdb.sh
It takes about 2 hours in my machine to reproduce.
You'll need the mongo C driver and libbsd (the code relies on arc4random()).



 Comments   
Comment by Kaloian Manassiev [ 13/May/14 ]

I am resolving this as duplicate of SERVER-13794. The snapshots code has been changed to only run if HTTP monitoring is enabled.

Comment by Davide Italiano [ 30/Apr/14 ]

I was able to isolate the commit that triggered the problem using git bisect.
https://github.com/mongodb/mongo/commit/1249034444dc146eb6177fbe4845e593b807f84d#diff-c44e3f2aa4926579a722790f6f3e1690R940
results in a memory usage growth until mongod crashes.
Reverting to

commit b3aaf2e5809e272367b0f60f66938d8bf712702c
Author: Siyuan Zhou <siyuan.zhou@mongodb.com>
Date:   Mon Apr 14 14:52:07 2014 -0400
 
    SERVER-13579 Wait for deletion after migration in feature2.js

makes the problem disappearing. The attached screenshot shows the memory usage.

Generated at Thu Feb 08 03:32:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.