We collect on serveral server data from our production machines. In the past we use MySQL databases. Now we've changed to Mongodb and are happy the easy access, administration and performance. The data is usually collected every second. Up to four collection processes are running in parallel. The collection processes are PHP programs, started by cron from the CLI.

Beside the data collector processes a ZEND server is running on that machine. Occasionally our production clients query data from the mongodb via a php web interface. Nothing else is running on that machine. No mongo replication is in place.

In the begining the process was running forever. Unfortunately the process ran out of memory after some time. So now we stop the script every hour to free the complete memory. Nevertheless the monogo database consumes after some time all of the memory, then the swap space and finally the server crashes. Some server we reboot every day to free the memory but sometimes the server crashes within 24 hours.

I attache some graphs of one of the server with a higher stability that show the memory and swap consumption.

We played a lot with one persisting database connection during the script, persistend connection to the production device, cleaning up of all variables but could not solve the problem.

On every database we have one index for the the timestamp, as this is our primary key for the status and graph queries:

L143SRV01:/var/log/mongodb# mongo
MongoDB shell version: 2.0.4
connecting to: test
> use mes
switched to db mes
> show collections
preis
system.indexes
tenifer_1
tenifer_2
tenifer_3
tenifer_4
tenifer_5
tenifer_6
tenifer_7
> db.tenifer_1.getIndexes()
[
{
"v" : 1,
"key" :

{ "_id" : 1 }

,
"ns" : "mes.tenifer_1",
"name" : "id"
},
{
"v" : 1,
"key" :

{ "timestamp" : 1 }

,
"ns" : "mes.tenifer_1",
"name" : "timestamp_1"
}
]
>

The collector scripts are quite simple and all of the same structure:

echo (date ( 'c' ) . " INFO: starting\n");

require_once dirname ( _FILE_ ) . '/ModbusMasterTcp.php';

$ip = "192.168.143.100";

$channels = 14;

$time = microtime (true);
$runs = 0;

try {

while ( 1 ) {
$modbus = new ModbusMasterTcp ( $ip );

$data = $modbus->readMultipleRegisters ( 1, hexdec('f8c3'), 4*$channels );

$reply = "";

foreach ($data as $c)

{ $reply .= chr($c); }

$value = array();

for ($i = 1; $i <= $channels; $i++)

{ $bin = substr($reply, ($i-1)*8, 2).substr($reply, ($i-1)*8+2, 2); $value[$i] = str_float( str_hex($bin)); }

$mongo = new Mongo ();
$db = $mongo->mes;

$timestamp = new MongoDate ( time () );

for ($i = 0; $i < $channels/2; $i++) {
$line = array();
$line['actual'] = $value[$i*2+1];
$line['nominal'] = $value[$i*2+2];
$line['timestamp'] = $timestamp;
$collection = sprintf("tenifer_%1d", $i+1);
$db->{$collection}->insert($line);
}

unset($mongo);
unset($db);
unset($modbus);

$time += 15;
$sleep=$time - microtime(true);
if ($sleep > 0)

{ $sleep = $sleep*1000000; usleep($sleep); }

$runs++;
if ($runs >= 4 * 60)

{ system("/usr/local/mes/start_tenifer 1"); die ( date ( 'c' ) . " INFO: done\n" ); }

}
} catch ( Exception $e ) {
die ( date ( 'c' ) . " ERROR: " . $e->getMessage () . "\n" );
}

We have 10 server running at the momemt showing different behaviour. Some crash within 24 hours some run stable of days. Here the memory consumption of one server that runs now for several days:

top - 11:10:01 up 7 days, 21:23, 2 users, load average: 0.00, 0.00, 0.00
Tasks: 106 total, 2 running, 104 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.0%us, 1.0%sy, 0.0%ni, 95.0%id, 3.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 382288k total, 373264k used, 9024k free, 3844k buffers
Swap: 915664k total, 203928k used, 711736k free, 247668k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
3069 mongodb 20 0 30.1g 205m 193m S 0.0 55.1 11:58.86 29g mongod
23230 www-data 20 0 421m 1516 964 S 0.0 0.4 2:18.44 419m apache2
18470 www-data 20 0 422m 7456 2812 S 0.0 2.0 1:42.00 415m apache2
23234 www-data 20 0 421m 7256 2844 S 0.0 1.9 1:33.10 414m apache2
30292 www-data 20 0 412m 1508 964 S 0.0 0.4 2:05.24 410m apache2
23233 www-data 20 0 409m 1512 964 S 0.0 0.4 1:38.25 408m apache2

You can see that memory and swap is eaten up by mongo.

Here's the config file from mongo:

1. mongodb.conf

1. Where to store the data.

1. Note: if you run mongodb as a non-root user (recommended) you may
2. need to create and set permissions for this directory manually,
3. e.g., if the parent directory isn't mutable by the mongodb user.
   dbpath=/var/lib/mongodb

#where to log
logpath=/var/log/mongodb/mongodb.log

logappend=true

#port = 27017

1. Disables write-ahead journaling
2. nojournal = true

1. Enables periodic logging of CPU utilization and I/O wait
   #cpu = true

1. Turn on/off security. Off is currently the default
   #noauth = true
   #auth = true

1. Verbose logging output.
   #verbose = true

1. Inspect all client data for validity on receipt (useful for
2. developing drivers)
   #objcheck = true

1. Enable db quota management
   #quota = true

1. Set oplogging level where n is
2. 0=off (default)
3. 1=W
4. 2=R
5. 3=both
6. 7=W+some reads
   #diaglog = 0

1. Ignore query hints
   #nohints = true

1. Disable the HTTP interface (Defaults to localhost:27018).
   #nohttpinterface = true

1. Turns off server-side scripting. This will result in greatly limited
2. functionality
   #noscripting = true

1. Turns off table scans. Any query that would do a table scan fails.
   #notablescan = true

1. Disable data file preallocation.
   #noprealloc = true

1. Specify .ns file size for new databases.
2. nssize = <size>

1. Accout token for Mongo monitoring server.
   #mms-token = <token>

1. Server name for Mongo monitoring server.
   #mms-name = <server-name>

1. Ping interval for Mongo monitoring server.
   #mms-interval = <seconds>

1. Replication Options

1. in master/slave replicated mongo databases, specify here whether
2. this is a slave or master
   #slave = true
   #source = master.example.com
3. Slave only: specify a single database to replicate
   #only = master.example.com
4. or
   #master = true
   #source = slave.example.com

1. in replica set configuration, specify the name of the replica set
2. replSet = setname

We already switched off the journaling.

Any idea what we can do to get the server stable?