[SERVER-4153] mongodump hung server Created: 26/Oct/11 Updated: 29/May/12 Resolved: 18/Nov/11 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Stability |
| Affects Version/s: | 1.8.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Chris Ferry | Assignee: | Brandon Diamond |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
CentOS 5.6 - EC2 m1.xlarge |
||
| Attachments: |
|
| Operating System: | Linux |
| Participants: |
| Description |
|
mongo server was completely unresponsive. Nothing in logs. mongodump was in progress at the time. |
| Comments |
| Comment by Brandon Diamond [ 10/Nov/11 ] |
|
Not sure about any special args; the only goal there is to find out where the process is hanging. |
| Comment by Chris Ferry [ 09/Nov/11 ] |
|
Do you have the gdb arguments you want me to use when attaching? |
| Comment by Brandon Diamond [ 01/Nov/11 ] |
|
Thanks for the clarification, Chris. MongoDB maps all data into virtual memory; as long RSS doesn't grow larger than physical memory, you shouldn't encounter any issues. Do you happen to have the mongod log files available from the time the issue was observed? If the problem occurs again, it'd also be extremely helpful if you could attach GDB to the process and see where the process is waiting ("where"). I also noticed that you're running on 1.8.3 – you should consider upgrading to the latest minor revision (1.8.4) for the latest patches and bugfixes. |
| Comment by Chris Ferry [ 01/Nov/11 ] |
|
Server that had the issue. |
| Comment by Chris Ferry [ 01/Nov/11 ] |
|
Primary MongoDB metrics for last day and last week. |
| Comment by Chris Ferry [ 01/Nov/11 ] |
|
Sorry I was on vacation. By unresponsive I mean all queries were timing out and the CLI was not connecting. Finally I tried a kill which failed to work until I kill nined it. We haven't had any lockups since, but I'm wondering what we can do to assist in troubleshooting if we were to have another. |
| Comment by Brandon Diamond [ 01/Nov/11 ] |
|
Haven't heard anything for awhile. Has anything changed? Otherwise, we'll close out this ticket tonight. |
| Comment by Brandon Diamond [ 28/Oct/11 ] |
|
One more thing – can you explain what you mean by "unresponsive"? Can you connect to the server with a separate mongoDB client? |
| Comment by Brandon Diamond [ 28/Oct/11 ] |
|
Thanks for all the info, Chris. What does your memory utilization look like over time? In other words, are you running the dump on a busy system with very little available memory? Or is the dump consuming most of the available memory on the system? This definitely looks like a lomem related issue. Any chance you could hook GDB into the process and find out where the tool is stalling? I'm having trouble reproducing on my end. |