[SERVER-712] Slave locks up Created: 07/Mar/10 Updated: 17/Mar/11 Resolved: 16/Jan/11 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 1.3.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Ianiv Schweber | Assignee: | Eliot Horowitz (Inactive) |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
AMD 2352 |
||
| Attachments: |
|
| Participants: |
| Description |
|
Slave database locks up. Pasting email thread so far. On Sun, Mar 7, 2010 at 8:45 PM, Ianiv Schweber <ischweber@nowpublic.com> wrote: top now shows I'm pretty sure status was S before running gdb. That scrolled out of screen's buffer so I can't double check anymore. Would you like me to open a new ticket and paste the conversation so far? Ianiv Schweber Public Key: http://www.blogaholics.ca/ianivpubkey.asc On 2010-03-07, at 5:26 PM, Eliot Horowitz wrote: Just added you to commercial support on jira. On Sun, Mar 7, 2010 at 8:16 PM, Ianiv Schweber <ischweber@nowpublic.com> wrote: I can't even start the JS console. It prints this and then just sits there. [root@mongo2 ~]# mongo scan_stats Likewise, the web console doesn't repsond. ctrl-c on gdb does nothing. The output of the gdb session is [root@mongo2 mongodb]# gdb bin/mongod --pid 13084 Ianiv Schweber Public Key: http://www.blogaholics.ca/ianivpubkey.asc On 2010-03-07, at 4:49 PM, Eliot Horowitz wrote: Can you connect with a shell and do db.currentOp() Can you view the web console? in gdb, what happens if you hit ctrl-c? do you have access to the commercial support section of jira yet? On Sun, Mar 7, 2010 at 7:26 PM, Ianiv Schweber <ischweber@nowpublic.com> wrote: Public Key: http://www.blogaholics.ca/ianivpubkey.asc hi, can you do the following with gdb: $ gdb mongod --pid=<slave process mongod pid> then also can you run mount and email the above? thanks On Sun, Mar 7, 2010 at 1:28 PM, Ianiv Schweber <ischweber@nowpublic.com> Hi Dwight, It looks like the slave locked up again, 1 hour 35 minutes ago. Last lines Sun Mar 7 17:05:36 repl: applied 269 operations Just like before. I'll keep it in this state for now in case it helps you figure out what is Thanks, Ianiv Schweber Public Key: http://www.blogaholics.ca/ianivpubkey.asc On 2010-03-04, at 1:26 PM, Narayan Newton wrote: Ianiv will chime in here soon, but: AMD 2352 On Thu, Mar 4, 2010 at 1:18 PM, Dwight Merriman <dwight@10gen.com> On Thu, Mar 4, 2010 at 4:16 PM, Narayan Newton Nice to meet you Dwight, We have had two issues recently, I'm currently investigating one (a We have "started over" with a new DB and the exceptions have not Thanks! -N |
| Comments |
| Comment by Ianiv Schweber [ 16/Mar/10 ] |
|
I'm watching the issue but for some reason I didn't get an email (and not in spam folder) so I just saw this. Tomorrow I'll recompile as you suggest. I'll see about getting you access to the server once it locks up again. |
| Comment by Eliot Horowitz (Inactive) [ 12/Mar/10 ] |
|
is there anyway we can get access to the server when this is happening? |
| Comment by Eliot Horowitz (Inactive) [ 12/Mar/10 ] |
|
This is very weird... Could you try building from source with --d (will get us debugging symbols when attach with gdb) |
| Comment by Ianiv Schweber [ 11/Mar/10 ] |
|
> db.currentOp() { "inprog" : [ ] }Then I tried > db.currentOp() , output from mongostat: insert/s query/s update/s delete/s getmore/s command/s mapped vsize res % locked % idx miss |
| Comment by Ianiv Schweber [ 11/Mar/10 ] |
|
output of slave's web console |
| Comment by Eliot Horowitz (Inactive) [ 11/Mar/10 ] |
|
Can you connect with the shell and db.currentOp() |
| Comment by Ianiv Schweber [ 11/Mar/10 ] |
|
Slave has locked up again. The last few log entries are: Thu Mar 11 17:20:24 repl: applied 376 operations Thu Mar 11 17:20:24 repl: applied 1 operations And the last entry repeats every minute. I'll keep it running so we can attempt to debug it again. |