[SERVER-2793] Understanding memory/disk metrics Created: 18/Mar/11 Updated: 30/Mar/12 Resolved: 01/Apr/11 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Performance |
| Affects Version/s: | 1.6.5 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | charso | Assignee: | Gaetan Voyer-Perrault |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu 10.04 |
||
| Attachments: |
|
| Participants: |
| Description |
|
I'm trying to delve deeper into MongoDB memory/disk usage patterns to both optimize my configuration and to identify warning signs for impending performance problems. I have some high level questions first:
|
| Comments |
| Comment by Gaetan Voyer-Perrault [ 30/Mar/11 ] |
|
Any additional follow-up required or are we ready to close this issue? |
| Comment by Gaetan Voyer-Perrault [ 23/Mar/11 ] |
|
Here's the simplest definition I can find for the "mapped' value of /proc/meminfo: Mapped — The total amount of memory, in kilobytes, which have been used to map devices, files, or libraries using the mmap command. Looking at your charts, that's the gold line. > the resident and mapped memory footprint is leveling off at half of available RAM? Based on the attached images, you re-started the mongo process just before midnight. At this point the gold line plummets to 0. Then the line shoots up to over 30GBs and hovers around 30GBs. However, if you follow the gold line prior to the restart it was clearly above half RAM. So it's quite possible to "mmap" more than half of the memory. > I'm constantly adding data, so you'd think if it still had half of available RAM left that resident/mapped would keep growing... Well, if you're constantly adding data, then MongoDB only needs to "mmap" two big things: Notice how turning off MongoDB did not free all of the your memory (yellow stuff)? By your own charts you either have something else using up the RAM or the OS is doing some caching for you. At this point, I'm not seeing anything clearly "wrong" here. Is there a documented performance issue here? |
| Comment by charso [ 22/Mar/11 ] |
I found this explanation about resident memory: number of megabytes resident. It is typical over time, on a dedicated database server, for this number to approach the amount of physical ram on the box. However, this doesn't explain why, in my configuration which data and index exceeding RAM, the resident and mapped memory footprint is leveling off at half of available RAM. I'm constantly adding data, so you'd think if it still had half of available RAM left that resident/mapped would keep growing, but it isn't. I'm looking for thoughts around why it levels off at half of RAM. |
| Comment by charso [ 22/Mar/11 ] |
|
I became busy dealing with CS-415. I'm looking at this now and will follow-up soon. |
| Comment by Gaetan Voyer-Perrault [ 22/Mar/11 ] |
|
Is there any additional follow-up required? |
| Comment by Gaetan Voyer-Perrault [ 18/Mar/11 ] |
|
Lots of questions here, however, it looks like we have documents for much of this. > Any idea what causes resident memory size to grow vs pages just sticking in the OS file cache? http://www.mongodb.org/display/DOCS/Monitoring+and+Diagnostics > Any metrics/ratios to look out for in tools like iostat, sar, vmstat, etc.? http://www.mongodb.org/display/DOCS/iostat Is there something specific missing from those docs? |
| Comment by Gaetan Voyer-Perrault [ 18/Mar/11 ] |
|
So the first thing that jumps at me is that these are not equal shards. It looks like shard1 has two additional collections and a few extra indexes. Can you clarify the difference in collections? |
| Comment by charso [ 18/Mar/11 ] |
|
Attached is an example of the memory usage on two different shard primaries. This example illustrates some of my questions. Note that only the last few hours are relevant; previous to that there was a repair. Shard1 happens to be doing a lot of read IO, whereas Shard2 has very little IO. The only discrepancy I can see is the difference between inactive and mapped memory. Here are the associated db stats: Shard1 Shard2 |