Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major - P3
-
Resolution: Incomplete
-
Affects Version/s: 2.2.0
-
Fix Version/s: None
-
Component/s: Sharding
-
Labels:
-
Environment:linux 64 bit
-
Operating System:Linux
Description
We are running mongodb v2.2.0 Linux 64 Bit, 3 shards each having 3 nodes.
One node seemed to be dead, so we restarted it. In the logs we found thousands of "waiting till out of critical section", filling up our logs very fastly and making mongodb inaccessible.
What does this error mean?
Here are the logs from the restart, if it matters:
http://pastebin.com/raw.php?i=cZeRJ7NV
This happened to other nodes as well already to another time.
Could it be related to v2.2.0 or to authentification? We're using both since a few time and we encounter this error only since then. Should we downgrade or disable authentification?
Issue Links
- is related to
-
SERVER-7034
timeouts for all connections in migrate critical section
-
- Closed
-
-
SERVER-7472
Replication lag can cause cluster to hang in migration critical section
-
- Closed
-
-
SERVER-7493
Possible for read starvation to cause migration to get stuck in critical section
-
- Closed
-
Activity
Yes, ok, please follow up the support ticket, so this one can be closed.
SERVER-7034 is the primary ticket for dealing with this issue. Resolving this one as incomplete for now.
Thanks all for the answers, added 7034 to my watch list.
We don't use authentification and had the same issue after upgrading our whole cluter from 2.0.6 to 2.2.1.
Klebert, the issue in question, SERVER-7034, has not yet been resolved (and thus not backported.) Please add yourself as a watcher to SERVER-7034 so that you'll receive an update when it is backported.
Hi agahd,
This may be related to
SERVER-7034. The issue in that ticket is that we do not have a timeout (i.e. timeout is infinite) for some connections that are made to the config servers, and if this connection hangs this can block other activity, leading to the "waiting for critical section" messages and an unresponsive server.Can you post a full log to this ticket so that we can compare symptoms with the cases we have seen?
Are your servers in MMS? Can you post a link?
Tad