[SERVER-5984] Memory problem on linux/debian server Created: 01/Jun/12  Updated: 04/Jan/13  Resolved: 04/Jan/13

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.0.3, 2.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Rainer Schmitz Assignee: Richard Kreuter (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux Debian
Linux L122SRV01 2.6.26-2-amd64 #1 SMP Sun Mar 4 21:48:06 UTC 2012 x86_64 GNU/Linux


Attachments: PDF File memory.pdf     PDF File swap.pdf    
Operating System: Linux
Participants:

 Description   

We collect on serveral server data from our production machines. In the past we use MySQL databases. Now we've changed to Mongodb and are happy the easy access, administration and performance. The data is usually collected every second. Up to four collection processes are running in parallel. The collection processes are PHP programs, started by cron from the CLI.

Beside the data collector processes a ZEND server is running on that machine. Occasionally our production clients query data from the mongodb via a php web interface. Nothing else is running on that machine. No mongo replication is in place.

In the begining the process was running forever. Unfortunately the process ran out of memory after some time. So now we stop the script every hour to free the complete memory. Nevertheless the monogo database consumes after some time all of the memory, then the swap space and finally the server crashes. Some server we reboot every day to free the memory but sometimes the server crashes within 24 hours.

I attache some graphs of one of the server with a higher stability that show the memory and swap consumption.

We played a lot with one persisting database connection during the script, persistend connection to the production device, cleaning up of all variables but could not solve the problem.

On every database we have one index for the the timestamp, as this is our primary key for the status and graph queries:

L143SRV01:/var/log/mongodb# mongo
MongoDB shell version: 2.0.4
connecting to: test
> use mes
switched to db mes
> show collections
preis
system.indexes
tenifer_1
tenifer_2
tenifer_3
tenifer_4
tenifer_5
tenifer_6
tenifer_7
> db.tenifer_1.getIndexes()
[
{
"v" : 1,
"key" :

{ "_id" : 1 }

,
"ns" : "mes.tenifer_1",
"name" : "id"
},
{
"v" : 1,
"key" :

{ "timestamp" : 1 }

,
"ns" : "mes.tenifer_1",
"name" : "timestamp_1"
}
]
>

The collector scripts are quite simple and all of the same structure:

echo (date ( 'c' ) . " INFO: starting\n");

require_once dirname ( _FILE_ ) . '/ModbusMasterTcp.php';

$ip = "192.168.143.100";

$channels = 14;

$time = microtime (true);
$runs = 0;

try {

while ( 1 ) {
$modbus = new ModbusMasterTcp ( $ip );

$data = $modbus->readMultipleRegisters ( 1, hexdec('f8c3'), 4*$channels );

$reply = "";

foreach ($data as $c)

{ $reply .= chr($c); }

$value = array();

for ($i = 1; $i <= $channels; $i++)

{ $bin = substr($reply, ($i-1)*8, 2).substr($reply, ($i-1)*8+2, 2); $value[$i] = str_float( str_hex($bin)); }

$mongo = new Mongo ();
$db = $mongo->mes;

$timestamp = new MongoDate ( time () );

for ($i = 0; $i < $channels/2; $i++) {
$line = array();
$line['actual'] = $value[$i*2+1];
$line['nominal'] = $value[$i*2+2];
$line['timestamp'] = $timestamp;
$collection = sprintf("tenifer_%1d", $i+1);
$db->{$collection}->insert($line);
}

unset($mongo);
unset($db);
unset($modbus);

$time += 15;
$sleep=$time - microtime(true);
if ($sleep > 0)

{ $sleep = $sleep*1000000; usleep($sleep); }

$runs++;
if ($runs >= 4 * 60)

{ system("/usr/local/mes/start_tenifer 1"); die ( date ( 'c' ) . " INFO: done\n" ); }

}
} catch ( Exception $e ) {
die ( date ( 'c' ) . " ERROR: " . $e->getMessage () . "\n" );
}

We have 10 server running at the momemt showing different behaviour. Some crash within 24 hours some run stable of days. Here the memory consumption of one server that runs now for several days:

top - 11:10:01 up 7 days, 21:23, 2 users, load average: 0.00, 0.00, 0.00
Tasks: 106 total, 2 running, 104 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.0%us, 1.0%sy, 0.0%ni, 95.0%id, 3.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 382288k total, 373264k used, 9024k free, 3844k buffers
Swap: 915664k total, 203928k used, 711736k free, 247668k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
3069 mongodb 20 0 30.1g 205m 193m S 0.0 55.1 11:58.86 29g mongod
23230 www-data 20 0 421m 1516 964 S 0.0 0.4 2:18.44 419m apache2
18470 www-data 20 0 422m 7456 2812 S 0.0 2.0 1:42.00 415m apache2
23234 www-data 20 0 421m 7256 2844 S 0.0 1.9 1:33.10 414m apache2
30292 www-data 20 0 412m 1508 964 S 0.0 0.4 2:05.24 410m apache2
23233 www-data 20 0 409m 1512 964 S 0.0 0.4 1:38.25 408m apache2

You can see that memory and swap is eaten up by mongo.

Here's the config file from mongo:

  1. mongodb.conf
  1. Where to store the data.
  1. Note: if you run mongodb as a non-root user (recommended) you may
  2. need to create and set permissions for this directory manually,
  3. e.g., if the parent directory isn't mutable by the mongodb user.
    dbpath=/var/lib/mongodb

#where to log
logpath=/var/log/mongodb/mongodb.log

logappend=true

#port = 27017

  1. Disables write-ahead journaling
  2. nojournal = true
  1. Enables periodic logging of CPU utilization and I/O wait
    #cpu = true
  1. Turn on/off security. Off is currently the default
    #noauth = true
    #auth = true
  1. Verbose logging output.
    #verbose = true
  1. Inspect all client data for validity on receipt (useful for
  2. developing drivers)
    #objcheck = true
  1. Enable db quota management
    #quota = true
  1. Set oplogging level where n is
  2. 0=off (default)
  3. 1=W
  4. 2=R
  5. 3=both
  6. 7=W+some reads
    #diaglog = 0
  1. Ignore query hints
    #nohints = true
  1. Disable the HTTP interface (Defaults to localhost:27018).
    #nohttpinterface = true
  1. Turns off server-side scripting. This will result in greatly limited
  2. functionality
    #noscripting = true
  1. Turns off table scans. Any query that would do a table scan fails.
    #notablescan = true
  1. Disable data file preallocation.
    #noprealloc = true
  1. Specify .ns file size for new databases.
  2. nssize = <size>
  1. Accout token for Mongo monitoring server.
    #mms-token = <token>
  1. Server name for Mongo monitoring server.
    #mms-name = <server-name>
  1. Ping interval for Mongo monitoring server.
    #mms-interval = <seconds>
  1. Replication Options
  1. in master/slave replicated mongo databases, specify here whether
  2. this is a slave or master
    #slave = true
    #source = master.example.com
  3. Slave only: specify a single database to replicate
    #only = master.example.com
  4. or
    #master = true
    #source = slave.example.com
  1. in replica set configuration, specify the name of the replica set
  2. replSet = setname

We already switched off the journaling.

Any idea what we can do to get the server stable?



 Comments   
Comment by Ian Whalen (Inactive) [ 04/Jan/13 ]

Hi Rainer, as discussed above I'm closing this ticket but please reopen if needed.

Comment by Richard Kreuter (Inactive) [ 20/Jun/12 ]

Rainer: thanks for the update. I'll check back in a few days, or please let us know when you're satisfied that this issue is okay and I'll resolve it.

Comment by Rainer Schmitz [ 18/Jun/12 ]

Hello Richard,

sorry for the delay. On the server we run additionally a ZEND Server with Apache and PHP. Apparantly the clients checking the status of the production machines created a lot of traffic on the apache server which caused a high memory consumption of the http process. As there are a lot of http processes running parallel they never showed up in the top list of processes ordered by memory consumption. So it took some time to find out that all http processe in total consumed too much memeory. We've reduced the number of client requests and now the server are much more stable. I suggest to monitor this for a while and then we can close the ticket.

Regards
Rainer

Comment by Richard Kreuter (Inactive) [ 11/Jun/12 ]

Rainer: are there any other kinds of programs that use the database? The insert method's code paths through mongod that your above PHP program exercises has never been seen to leak memory. How do you use the data once it's in the db?

Comment by Rainer Schmitz [ 11/Jun/12 ]

Hi Richard,

We had again several server crashes last week. Is there any progress on this ticket? Are you awating anything from me?

Rainer

Generated at Thu Feb 08 03:10:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.