[JAVA-494] Poor performance due to SimplePool._get() and SimplePool.done() Created: 21/Dec/11  Updated: 19/Oct/16  Resolved: 26/Nov/13

Status: Closed
Project: Java Driver
Component/s: Connection Management, Performance
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Arvind Tiwari Assignee: Jeffrey Yemin
Resolution: Done Votes: 2
Labels: connection
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified
Environment:

Linux 64 bit,
Mongo server 2.x ,
driver 2.7.1


Attachments: Text File MongoUpsertContentionTest.java     File profiler.bmp    
Issue Links:
Depends
Sub-Tasks:
Key
Summary
Type
Status
Assignee
JAVA-1006 3.0: Make connection pool non-blocking Sub-task Closed Jeffrey Yemin  
JAVA-1007 2.12: Make connection pool non-blocking Sub-task Closed Jeffrey Yemin  
Backwards Compatibility: Minor Change

 Description   

We are experiencing major bottleneck around following two calls on load, percentage time consumed by these two calls are given below.

com.mongodb.util.SimplePool._get() (42%)
com.mongodb.util.SimplePool.done() (37%)

Please let us know if this is a known issue and there exist a fix for the same?
Do we have any alternative to the SimplePool, which handles get and put connection more efficiently?



 Comments   
Comment by Jeffrey Yemin [ 10/Jul/13 ]

Before this change, the driver's internal connection pool made a best effort to provide thread affinity to the pooled connections. This behavior was not documented, nor was it reliable if the number of threads exceeded the number of pooled connections.

This commit removes thread affinity in the connection pool, so applications that rely on this thread affinity will not work as they used to. This is most likely to occur in applications that:

  1. use unacknowledged writes (see WriteConcern.UNACKNOWLEDGED)
  2. expect those writes to be visible to subsequent reads on the same thread
  3. do not surround these reads and writes with calls to DB.requestStart/requestDone.
Comment by Jeffrey Yemin [ 10/Jul/13 ]

Author:

{u'username': u'jyemin', u'name': u'Jeff Yemin', u'email': u'jeff.yemin@10gen.com'}

Branch: 3.0.x
https://github.com/mongodb/mongo-java-driver/commit/e2b285c

Comment by Jeffrey Yemin [ 25/Jun/13 ]

Planning to make the pool non-blocking for the 3.0 release.

Comment by Jeffrey Yemin [ 13/Jul/12 ]

Paul, can you describe a bit more what you're seeing, including:

  1. any connection pool sizing customizations in MongoOptions
  2. a series of full thread dumps over time
Comment by Paul Mokbel [ 13/Jul/12 ]

We're also experiencing mega contention here under load.

"http-80-156" daemon prio=10 tid=0x00007f5c58d8b000 nid=0x6189 waiting for monitor entry [0x00007f5c43d7c000]
  java.lang.Thread.State: BLOCKED (on object monitor)
       at com.mongodb.util.SimplePool.done(SimplePool.java:111)
       - waiting to lock <0x00007f5c7487c920> (a java.util.ArrayList)
       at com.mongodb.util.SimplePool.done(SimplePool.java:94)
       at com.mongodb.DBTCPConnector$MyPort.done(DBTCPConnector.java:344)
       at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:223)
       at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:296)
       at com.mongodb.DBCollection.findOne(DBCollection.java:558)
       at com.sitescout.dsp.capmanage.UserFrequency.getByKey(UserFrequency.java:37)
       at com.sitescout.dsp.capmanage.UserFrequency.getTimestamps(UserFrequency.java:42)

Comment by Jeffrey Yemin [ 22/May/12 ]

Attached MongoUpsertContentionTest, a benchmark for testing the affect of the connection pool on overall driver performance.

Comment by Jeffrey Yemin [ 27/Dec/11 ]

Let us know if you're still unable to create the new issue, and we'll walk you through it step by step.

Comment by Jeffrey Yemin [ 27/Dec/11 ]

Arvind,

You should see two drop-downs when you go to create an issue. You're looking at the second drop-down, labelled "Issue Type" The first drop-down, labelled "Project", is where you should see "Commercial Support" as an option. If you don't see that project name, use "Community Private"

Comment by Arvind Tiwari [ 27/Dec/11 ]

I am not seeing the option "project", following are the options I am getting

Bug
New Feature
Question
Task
Improvement

Comment by Scott Hernandez (Inactive) [ 27/Dec/11 ]

You create a Customer Support jira issue. It is a type (project) of issue you can choose when you create new issue. See the upper right hand corner of this web page.

Comment by Arvind Tiwari [ 27/Dec/11 ]

Please tell me how to create the "CS issue", I did not get any information on that.

Comment by Scott Hernandez (Inactive) [ 26/Dec/11 ]

No, anything is a cs issue is private.

Comment by Arvind Tiwari [ 26/Dec/11 ]

In that case my snapshot will not be publicly accessible?

Comment by Scott Hernandez (Inactive) [ 26/Dec/11 ]

Any CS issue can be linked to this. Just create a CS issue, attache the files, and click the "more actions" button to select the link action.

Comment by Arvind Tiwari [ 26/Dec/11 ]

Please let me know how to open ticket with commercial support queue (hope that wd not be public?) as I do not want to attach the snapshot in public forum.

Comment by Jeffrey Yemin [ 22/Dec/11 ]

Arvind,

Please attach a YourKit snapshot of the profiling run. Alternatively, you can open a ticket in the commercial support queue and attach it there.

Comment by Scott Hernandez (Inactive) [ 22/Dec/11 ]

It sounds like you are setting maxConnectionsPerHost=800, and have 300 concurrent web requests, is that correct? Can you post a bit more about the test config, including your MongoOptions, the app-server/host-env., the number of threads used for testing, and such?

You can try using db.requestStart/Done at the start/end of the request to keep the thread from needing to go to the connection pool for the request lifetime, but I suspect that there is something else going on here.

Comment by Jeffrey Yemin [ 22/Dec/11 ]

From the bmp it looks like you're using the YourKit profiler. Can you run again with Monitor Profiling enable to see if some of this is due to monitor contention?

Comment by Arvind Tiwari [ 21/Dec/11 ]

Attached the profiler hot spot page. Following is the test case

We are load testing a particular API
1. 800 connection configured
2. 1 request is composed of 8/10 DB calls, most of the calls are DB read, except 2 which writes to the DB (with WriteConcern none).
3. 300 concurrent user making the request.

Comment by Scott Hernandez (Inactive) [ 21/Dec/11 ]

What constitutes the rest of the time? What else was being profiled? Can you provide the full profiler results or test case?

Generated at Thu Feb 08 08:52:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.