[JAVA-2074] Mongo client is leaking unclosed connection thread. Created: 05/Jan/16 Updated: 16/Feb/18 Resolved: 25/Jan/16 |
|
| Status: | Closed |
| Project: | Java Driver |
| Component/s: | Cluster Management |
| Affects Version/s: | 2.12.0 |
| Fix Version/s: | 3.2.2, 2.14.2 |
| Type: | Bug | Priority: | Minor - P4 |
| Reporter: | Bin Lan | Assignee: | Jeffrey Yemin |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Description |
|
On cloud manager, we have some cron job that creates new MongoClient to all the backing mongodb process to make sure that they are up and without startup warning. We observe that a lot of these ad-hoc clients are not being closed properly. On our side, we do guard the connection instance with a try/finally to make sure the close() method is called. An example thread dump line looks like this:
Note that the counter #817624 is relatively large and we do find multiple instance to the same host on the same thread dump. This points to a potential MongoClient leak. And the clients seem to be all from arbiter processes since they have all the higher counter number thread. |
| Comments |
| Comment by Githook User [ 16/Feb/18 ] | |||||||||||||
|
Author: {'email': 'jeff.yemin@10gen.com', 'name': 'Jeff Yemin', 'username': 'jyemin'}Message: | |||||||||||||
| Comment by Jeffrey Yemin [ 31/Mar/16 ] | |||||||||||||
|
2.14.2 has been released with a fix for this bug. | |||||||||||||
| Comment by Jeffrey Yemin [ 16/Feb/16 ] | |||||||||||||
|
Closed for 3.2.2 release | |||||||||||||
| Comment by Githook User [ 25/Jan/16 ] | |||||||||||||
|
Author: {u'username': u'jyemin', u'name': u'Jeff Yemin', u'email': u'jeff.yemin@10gen.com'}Message: | |||||||||||||
| Comment by Githook User [ 25/Jan/16 ] | |||||||||||||
|
Author: {u'username': u'jyemin', u'name': u'Jeff Yemin', u'email': u'jeff.yemin@10gen.com'}Message: | |||||||||||||
| Comment by Githook User [ 25/Jan/16 ] | |||||||||||||
|
Author: {u'username': u'jyemin', u'name': u'Jeff Yemin', u'email': u'jeff.yemin@10gen.com'}Message: | |||||||||||||
| Comment by Githook User [ 25/Jan/16 ] | |||||||||||||
|
Author: {u'username': u'jyemin', u'name': u'Jeff Yemin', u'email': u'jeff.yemin@10gen.com'}Message: | |||||||||||||
| Comment by Jeffrey Yemin [ 20/Jan/16 ] | |||||||||||||
|
Changing priority to minor, as this bug will only be exposed in the rare circumstance where a MongoClient is constructed but never actually used before it's closed. | |||||||||||||
| Comment by Jeffrey Yemin [ 05/Jan/16 ] | |||||||||||||
|
Yes, that was perfectly clear. | |||||||||||||
| Comment by Bin Lan [ 05/Jan/16 ] | |||||||||||||
|
Hi Jeff, I might be misleading on the language of the ticket. It is not the Mongo | |||||||||||||
| Comment by Jeffrey Yemin [ 05/Jan/16 ] | |||||||||||||
|
The Mongo.close() method interrupts the MongoCleaner thread and then calls join on it. Therefore, MongoClient.close() will not even return normally until that thread dies. So if you are also observing many alive MongoCleaner threads, it would suggest that there is a path where MongoClient instances are not being closed properly. | |||||||||||||
| Comment by Jeffrey Yemin [ 05/Jan/16 ] | |||||||||||||
|
I also failed to reproduce this with a direct connection to an arbiter. | |||||||||||||
| Comment by Jeffrey Yemin [ 05/Jan/16 ] | |||||||||||||
|
I'm not yet able to reproduce this. I created a replica set with three data bearing nodes and one arbiter, running on ports 27017-27020, and ran the following script:
Stepping through in the debugger, by the time I get to the last sleep an inspection of the running threads shows none that were created by the driver. To help debug this further, can you provide a code snippet showing the construction of one of the MongoClient instances that appear to be leaking threads? | |||||||||||||
| Comment by Jeffrey Yemin [ 05/Jan/16 ] | |||||||||||||
|
The number after "cluster" represents the cluster id, which is implemented as a static AtomicInteger. So that thread name also implies that the application has created at least 374,495 MongoClient instances in a single class loader context. |