[SERVER-13574] performance/index problem on concurrent search/insert Created: 14/Apr/14 Updated: 10/Dec/14 Resolved: 20/May/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Concurrency, Performance, Querying |
| Affects Version/s: | 2.6.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Mike Pasieka | Assignee: | Thomas Rueckstiess |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Win 8.1 x64, i7, 16GB of RAM, 10GB available for Mongo. |
||
| Operating System: | Windows |
| Steps To Reproduce: | have 2 concurrent read/write processes, and fairly big collections (A is 60m, B is 4m records) |
| Participants: |
| Description |
|
I have a db and 2 processes that aggregate the data. They do the following: Issue was not present in 2.4, it's 2.6 only. Both collections have proper indices ( {s: 1, p: 1, d: 1}and {d:1, s:1 }), I query by s and p + sort by d. No sharding. When 1 worker is doing its job - it's ok. when i kill one worker and only one remains - everything works as usually, i mean fast. i'm running windows, data + indices are slightly bigger than RAM. console log, 2 workers are working, and then one is killed:
could you please explain that? is it fixable, or it's just the global write lock? |
| Comments |
| Comment by Thomas Rueckstiess [ 20/May/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Mike, I haven't heard back from you for some time, so I'll assume that you were able to resolve or work around the issue. I'll close the ticket now, but if you'd like to follow up feel free to re-open it. In that case, we'd still need the additional information requested in my previous reply. Kind Regards, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Thomas Rueckstiess [ 22/Apr/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Mike, From the output in the log file, I can see that the slow queries are using the wrong index:
This should be {s: 1.0, p: 1.0, d: 1.0}. That's why the query has to scan a large number of documents and also page them from disk if they do not reside in RAM already:
I've tried to reproduce this situation locally with the same indexes and queries as you describe, but in my test, it always picked the correct index. The different behavior could be explained by the data distribution. Mine was uniformly random, but perhaps you have certain values that appear a lot more than others. MongoDB's query planner is empirical, and occasionally runs short experiments to see which of the possible indexes is most effective. It's possible that it picked the wrong index based on the experiment, but without knowing your exact data distribution it's hard to tell. If you would like to investigate this further, I'd need to get an idea of the distribution of your data. If this is a non-production environment, you could run an aggregation task like this:
which would generate a new collection "datadistr" and count the number of documents per (s, p) pair. Once completed, you can mongodump this collection and attach it to the ticket. Note: If this is a production environment, I do not recommend running this aggregation task because it is computationally expensive and can have serious impact on the performance of your system! I'd also like to look at some example documents from this collection, so I can more accurately reproduce this here. Could you also attach the output of:
If you are looking for a short-term solution, as a workaround I recommend that you use a .hint() for these types of queries as you know exactly which index they should use. This will force the planner to use the correct index. Please let me know how you'd like to proceed. Thanks, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Mike Pasieka [ 15/Apr/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
hi! thanks for quick reply. 1. Both collections have same index structure, as listed below:
2. yes 3. I have a slow query treshold of 100ms. No slow entries in the log mean that all queries run under 100ms. I tried to grab a log of this with full profiling, but after restarting OS and MongoDB the issue is gone - maybe query plans got updated? Now it's fast enough. I'll try to reproduce this issue later today, with all the operations I was performing yesterday. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Thomas Rueckstiess [ 14/Apr/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Mike, Thanks for reporting this issue. I have some additional questions before we can determine what the problem is. From the log lines you attached it appears that the index being used to retrieve the products is {s: 1, d: 1}, instead of the more suitable index {s:1, p:1, d:1}, hence the scanning of documents.
Thanks, |