[SERVER-16758] Tons of ClientCursor::staticYield on a simple query Created: 07/Jan/15 Updated: 06/Apr/23 Resolved: 22/May/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Logging, Querying |
| Affects Version/s: | 2.6.6 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Alexander S. | Assignee: | Ramon Fernandez Marina |
| Resolution: | Duplicate | Votes: | 4 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu Linux x86_64, mongodb 2.6.6 |
||
| Issue Links: |
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
Getting tons of:
Had to configure log rotation to run hourly because log grows 10Gb per a few hours. Here's the ruby code (using mongoid gem) that causes the warning (it does not query same collection from multiple threads): http://pastebin.com/xVh9ex6r I think this should be considered as a high priority issue, since makes logging absolutely useless and we can't monitor real issues. And this warning is not descriptive at all, google is full of complaints but I didn't see real solutions. |
| Comments |
| Comment by Ramon Fernandez Marina [ 22/May/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
2.6.11 is tentatively scheduled for early July. Any updates to this schedule will be posted in the versions page for the SERVER project. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Volodymyr Gren [ 22/May/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
and when 2.6.11 will be released? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 22/May/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
All, while these messages are important as they warn users of operations that are taking too long to complete, it's probably not useful to have so many warnings in the logs – so the frequency has been reduced by two orders of magnitude as per The aforementioned change will be part of the upcoming 2.6.11. I'd also like to note that these messages are completely eliminated in v3.0, as findAndModify queries are now allowed to yield, so I'd encourage users affected by the flooding of the logs described in this ticket to upgrade to the latest v3.0 (3.0.3 at the time of this writing). Regards, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Volodymyr Gren [ 22/May/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Same here - on 2.4 it works well, while in staging environment on 2.6 dozens of Gbytes logs, please fix | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 19/May/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Write operations with poorly or no indexed queries may take a long time to run and not be able to yield, thus printing this message often. The recommended solution is to create a selective index that can help the query perform faster. For example, having an index on a field that has boolean values is not selective enough to speed up the query significantly. Note that there have been significant improvements in this area for 3.0 and some operations that were not able to yield before can now yield, so if upgrading is an option I'd recommend upgrading to 3.0.3 or older. Regards, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Jari Lamberg [ 12/Mar/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Mongodb 2.6.7 After getting disk full with this annoying nonsense warning I just had a look at the lg and found the same warning after a remove query: , | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Laurent Glayal [ 23/Jan/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Same problem, we also have such warnings during findAndModify(using multiple fields for query and with sort on _id), the explain() shows index is used on fields and around 470 documents are scanned, I found no way to easily get rid of such warnings. Such warnings are usefull to indicate a default somewhere but a possibility to turn it on/off should be available. For production these informaion could be turned on/off at runtime, like log verbosity, currently the I/O generated by such log information impair performance for system.We also have GB of information and need to logrotate every hour, after issuing a disk full... We are using mongodb 2.6.7. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Alexander S. [ 10/Jan/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Btw, we also use mongodb 2.6.6, but still see this warning in the logs. And yes, they are also all related to find_and_modify queries. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Alexander S. [ 10/Jan/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi, it seems it is indexed:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 08/Jan/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This could be related to
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Alexander S. [ 08/Jan/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
A few more details about the code. The warning above happens with ::Carmen::BackgroundRunner::Queue and this code:
Could insert new records to ::Carmen::BackgroundRunner::Queue. So we take one record from carmen_background_runner_queues, locking it with find_and_modify, and during processing this record in the application we can make one or more insertions to the carmen_background_runner_queues collection. I don't understand the nature of this warning, but could this be the cause? |