[SERVER-6391] Yielding with one (or more) active writer and heavy read load results in severe performance degradation Created: 11/Jul/12 Updated: 11/Jul/16 Resolved: 25/Jul/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Concurrency |
| Affects Version/s: | 2.0.2 |
| Fix Version/s: | 2.0.7, 2.2.0-rc1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Ben Becker | Assignee: | Ben Becker |
| Resolution: | Done | Votes: | 0 |
| Labels: | concurrency | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu 12.04, Linux kernel v3.2.0-25 |
||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Operating System: | Linux | ||||||||
| Participants: | |||||||||
| Description |
|
Tests with the following command result in execution times of 20 seconds to 10 minutes:
The collection has 1000 identical documents:
With ~3944 concurrent readers and 10-15 concurrent writers:
*NOTE: stall/burst in this sense means minute-long intervals where only ~300K-500K of data is transmitted, followed by a burst of 1-5Mbps for up to a few seconds. 'Consistently' in this case meaning between 10-50Mbps (avg. ~35Mbps). Daemon and client are on the same VM. Having zero writers means yieldSuggest() always suggests not yielding. Under heavy load, this seems to indicate that the cost of actually yielding is tremendously expensive. It should be noted that recommendedYieldMicros() is expensive when lots of connections are running, taking an average of 1/3 of each threads execution time with 4k connections. That said, the impact of removing this call is relatively minimal in terms of test completion time: With the calls to recommendedYieldMicros() acquiring the clientsMutex lock and hard-coding writers to 0, the aforementioned test completes in 24 seconds, vs. 20 seconds when the lock/count is avoided. |
| Comments |
| Comment by auto [ 27/Jul/12 ] | |||||||||||||||
|
Author: {u'date': u'2012-07-15T19:22:03-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}Message: Conflicts: db/clientcursor.cpp | |||||||||||||||
| Comment by auto [ 27/Jul/12 ] | |||||||||||||||
|
Author: {u'date': u'2012-07-15T19:22:49-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}Message: | |||||||||||||||
| Comment by auto [ 27/Jul/12 ] | |||||||||||||||
|
Author: {u'date': u'2012-07-15T19:21:50-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}Message: | |||||||||||||||
| Comment by auto [ 25/Jul/12 ] | |||||||||||||||
|
Author: {u'date': u'2012-07-24T23:27:25-07:00', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}Message: | |||||||||||||||
| Comment by Ben Becker [ 16/Jul/12 ] | |||||||||||||||
|
yes, it just means that recommendedYieldMicros() only acquires clientsMutex and counts readers/writers once before every ~(numConnections/128) * 50 yields, instead of once before every yield. Essentially a hacky cache similar to capping the number of iterations, but (possibly) further reducing contention on clientsMutex (note these tests don't compare the 'capped iterations' implementation, but happy to run that test). | |||||||||||||||
| Comment by Andy Schwerin [ 16/Jul/12 ] | |||||||||||||||
|
Can you remind me what "cacheSuggest" is? | |||||||||||||||
| Comment by auto [ 16/Jul/12 ] | |||||||||||||||
|
Author: {u'date': u'2012-07-15T19:22:49-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}Message: | |||||||||||||||
| Comment by auto [ 16/Jul/12 ] | |||||||||||||||
|
Author: {u'date': u'2012-07-15T19:22:03-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}Message: | |||||||||||||||
| Comment by auto [ 16/Jul/12 ] | |||||||||||||||
|
Author: {u'date': u'2012-07-15T19:21:50-07:00', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}Message: | |||||||||||||||
| Comment by Ben Becker [ 11/Jul/12 ] | |||||||||||||||
|
Attaching simple test script. Requires the following modification to bench.cpp (based on r2.0.2):
Additional profiling changes are available at https://github.com/vrtx/mongo/commit/5b2c752477d113132c28dab129e54afd465af1f9. |