[SERVER-5984] Memory problem on linux/debian server Created: 01/Jun/12 Updated: 04/Jan/13 Resolved: 04/Jan/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Stability |
| Affects Version/s: | 2.0.3, 2.0.4 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Rainer Schmitz | Assignee: | Richard Kreuter (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Linux Debian |
||
| Attachments: |
|
| Operating System: | Linux |
| Participants: |
| Description |
|
We collect on serveral server data from our production machines. In the past we use MySQL databases. Now we've changed to Mongodb and are happy the easy access, administration and performance. The data is usually collected every second. Up to four collection processes are running in parallel. The collection processes are PHP programs, started by cron from the CLI. Beside the data collector processes a ZEND server is running on that machine. Occasionally our production clients query data from the mongodb via a php web interface. Nothing else is running on that machine. No mongo replication is in place. In the begining the process was running forever. Unfortunately the process ran out of memory after some time. So now we stop the script every hour to free the complete memory. Nevertheless the monogo database consumes after some time all of the memory, then the swap space and finally the server crashes. Some server we reboot every day to free the memory but sometimes the server crashes within 24 hours. I attache some graphs of one of the server with a higher stability that show the memory and swap consumption. We played a lot with one persisting database connection during the script, persistend connection to the production device, cleaning up of all variables but could not solve the problem. On every database we have one index for the the timestamp, as this is our primary key for the status and graph queries: L143SRV01:/var/log/mongodb# mongo , , The collector scripts are quite simple and all of the same structure: echo (date ( 'c' ) . " INFO: starting\n"); require_once dirname ( _FILE_ ) . '/ModbusMasterTcp.php'; $ip = "192.168.143.100"; $channels = 14; $time = microtime (true); try { while ( 1 ) { $data = $modbus->readMultipleRegisters ( 1, hexdec('f8c3'), 4*$channels ); $reply = ""; foreach ($data as $c) { $reply .= chr($c); }$value = array(); for ($i = 1; $i <= $channels; $i++) { $bin = substr($reply, ($i-1)*8, 2).substr($reply, ($i-1)*8+2, 2); $value[$i] = str_float( str_hex($bin)); } $mongo = new Mongo (); $timestamp = new MongoDate ( time () ); for ($i = 0; $i < $channels/2; $i++) { unset($mongo); $time += 15;
$runs++; } We have 10 server running at the momemt showing different behaviour. Some crash within 24 hours some run stable of days. Here the memory consumption of one server that runs now for several days: top - 11:10:01 up 7 days, 21:23, 2 users, load average: 0.00, 0.00, 0.00 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND You can see that memory and swap is eaten up by mongo. Here's the config file from mongo:
#where to log logappend=true #port = 27017
We already switched off the journaling. Any idea what we can do to get the server stable? |
| Comments |
| Comment by Ian Whalen (Inactive) [ 04/Jan/13 ] |
|
Hi Rainer, as discussed above I'm closing this ticket but please reopen if needed. |
| Comment by Richard Kreuter (Inactive) [ 20/Jun/12 ] |
|
Rainer: thanks for the update. I'll check back in a few days, or please let us know when you're satisfied that this issue is okay and I'll resolve it. |
| Comment by Rainer Schmitz [ 18/Jun/12 ] |
|
Hello Richard, sorry for the delay. On the server we run additionally a ZEND Server with Apache and PHP. Apparantly the clients checking the status of the production machines created a lot of traffic on the apache server which caused a high memory consumption of the http process. As there are a lot of http processes running parallel they never showed up in the top list of processes ordered by memory consumption. So it took some time to find out that all http processe in total consumed too much memeory. We've reduced the number of client requests and now the server are much more stable. I suggest to monitor this for a while and then we can close the ticket. Regards |
| Comment by Richard Kreuter (Inactive) [ 11/Jun/12 ] |
|
Rainer: are there any other kinds of programs that use the database? The insert method's code paths through mongod that your above PHP program exercises has never been seen to leak memory. How do you use the data once it's in the db? |
| Comment by Rainer Schmitz [ 11/Jun/12 ] |
|
Hi Richard, We had again several server crashes last week. Is there any progress on this ticket? Are you awating anything from me? Rainer |