[SERVER-4037] mongos: writeback failed because of stale config Created: 07/Oct/11 Updated: 11/Jul/16 Resolved: 08/Oct/11 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 2.0.0-rc2 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Y. Wayne Huang | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | mongos | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
ubuntu 10.04 x86_64 |
||
| Operating System: | Linux |
| Participants: |
| Description |
|
we are seeing mongos output about 300 errors/sec of the following: the attempts appear to be going to a shard which previously had a similar problem of outputting a ton of errors of the following: this is with mongos nightly as of oct 6. we are also seeing sporadic errors of "db assertion failed" when running map reduce jobs, similar to
|
| Comments |
| Comment by Y. Wayne Huang [ 21/Oct/11 ] |
|
it appears mongos does not spew log entries anymore but it certainly is getting into a tight loop of connecting and issuing queries for some reason |
| Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ] |
|
Can you open a new ticket with those logs, etc... |
| Comment by Y. Wayne Huang [ 20/Oct/11 ] |
|
Eliot, we upgraded to the nightly build the day of your comment and hadn't seen the problem in a while but it returned this morning. This time it didn't seem to generate log messages on mongos but it created upwards of 1-2k of command ops/sec on two of our shards. Bouncing mongos fixed the issue. Therefore I don't believe the fix works in all cases. Are there subsequent fixes in the 2.0.1 rc that are also related to this problem? If not, we should re-open this. |
| Comment by Eliot Horowitz (Inactive) [ 10/Oct/11 ] |
|
The fix was not in that version but is in the current nightly. |
| Comment by Y. Wayne Huang [ 10/Oct/11 ] |
|
Mon Oct 10 13:39:52 ./mongos db version v2.0.1-pre-, pdfile version 4.5 starting (--help for usage) |
| Comment by Eliot Horowitz (Inactive) [ 10/Oct/11 ] |
|
what git hash? |
| Comment by Y. Wayne Huang [ 10/Oct/11 ] |
|
hi Eliot – you mentioned these issues are fixed in 2.0.0 and 2.0 head, respectively. we are running 2.0 nightly (assume that comes from 2.0 head) and we still see this issue as of this morning. we can try a new mongos from last night but since you indicated the issues were fixed already and we have the nightly from friday, it seems it would not help. also, can you link the two issues you're referring to? |
| Comment by Eliot Horowitz (Inactive) [ 08/Oct/11 ] |
|
There were cases for this. |
| Comment by Y. Wayne Huang [ 07/Oct/11 ] |
|
restarting mongos stopped the infinite retry, which was causing 300-400 command ops/sec on one shard and increasing the write lock % from < 1% to 8%. this was effectively dos'ing one of our shards. |