[JAVA-232] Getting node down error when all servers are up Created: 12/Dec/10 Updated: 25/Jun/13 Resolved: 24/Oct/11 |
|
| Status: | Closed |
| Project: | Java Driver |
| Component/s: | Cluster Management |
| Affects Version/s: | 2.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Joseph Wang | Assignee: | Antoine Girbal |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
[joseph.wang@stage1.vpc3 ~]$ uname -a |
||
| Attachments: |
|
| Description |
|
[joseph.wang@stage1.vpc3 ~]$ ps auxww | grep mongo > rs.status() , , { "_id" : 2, "name" : "stage1.vpc3.estalea.net:4103", "health" : 1, "state" : 2, "uptime" : 50789, "lastHeartbeat" : "Sun Dec 12 2010 05:07:16 GMT-0800 (PST)" } ], [22:55:05.073] node down: stage1.vpc3.estalea.net:4103 java.io.IOException: couldn't connect to [stage1.vpc3.estalea.net/10.9.0.5:4103] bc:java.net.ConnectException: Connection refused |
| Comments |
| Comment by Scott Hernandez (Inactive) [ 26/Aug/11 ] |
|
This code has been updated a lot, do you still see this with the latest driver? |
| Comment by Joseph Wang [ 12/Dec/10 ] |
|
latest version w/ getlasterror check |
| Comment by Joseph Wang [ 12/Dec/10 ] |
|
Still getting error. However, exception handling is not triggered. Cannot detech such condition to force reconnect. [15:27:36.227] at java.util.TimerThread.mainLoop(Timer.java:512) |
| Comment by Joseph Wang [ 12/Dec/10 ] |
|
Adding the following code to see if can work around the problem. DBObject dbObject = db.getLastError(); if (error_count >= error_count_threshold) { log.warn("DbCollectionVersionTableUpdater: Reset connection due to high error count " + error_count); MongoConnnection.getInstance().close(); MongoConnnection.getInstance().reconnect(); error_count = 0; }Looking at the resin log, the problem started 22:55 last night till this morning. [22:55:25.874] replica set mode, switching master ... Note that the primary has been running since Nov 30. |
| Comment by Joseph Wang [ 12/Dec/10 ] |
|
MongoConnection.java: connection pool singleton What is more troubling is that it stuck in this mode forever. If there is a problem with a slave, should the Java driver auto routes to another slave? |
| Comment by Joseph Wang [ 12/Dec/10 ] |
|
see sockets for all three mongo bash-3.2$ netstat -an | grep 4103 | grep -v CLOSE_WAIT | wc -l |