[SERVER-65214] MongoDB crashed with Invalid access at address + Got signal: 11 (Segmentation fault) Created: 04/Apr/22 Updated: 15/May/22 Resolved: 15/May/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.4.10 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Patanjali C | Assignee: | Dmitry Agranat |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | Bug | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||
| Issue Links: |
|
||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
We are running mongo as a replicaset of 3 nodes as docker containers, The running mongo version is 4.4.10. We are facing constant restarts since yesterday (2022-04-03) morning. This is causing issues with our production. The Backtrace is pointing to Resource Consumption on all nodes:
|
| Comments |
| Comment by Dmitry Agranat [ 15/May/22 ] | ||
|
As we have not heard back from you for some time now, I will go ahead and close this ticket. | ||
| Comment by Dmitry Agranat [ 11/Apr/22 ] | ||
|
Yes, either via adding to mongo.conf or by executing setParameter command. As always, please test this in your staging cluster before implementing in Production. | ||
| Comment by Patanjali C [ 07/Apr/22 ] | ||
|
Following up on my previous question can we set the parameters using the mongo.conf, and restarting the service will set the values?
| ||
| Comment by Patanjali C [ 06/Apr/22 ] | ||
|
Thank you for pointing out, We can put this parameter inside the mongo conf and on restarting mongo will set the value to 1000 or should we use the admin command to do set the parameter. | ||
| Comment by Dmitry Agranat [ 06/Apr/22 ] | ||
|
Here is the relevant documentation: https://www.mongodb.com/docs/manual/reference/command/setParameter/#setparameter To set this parameter via setParameter:
| ||
| Comment by Patanjali C [ 06/Apr/22 ] | ||
|
understood, can you somehow point me to the documentation that can help me set the internalPipelineLengthLimit or point to a method to set in the mongo.conf, This would additionally help in in fixing the config Thanks | ||
| Comment by Dmitry Agranat [ 06/Apr/22 ] | ||
|
The config is internalPipelineLengthLimit mentioned in my last comment | ||
| Comment by Patanjali C [ 06/Apr/22 ] | ||
|
so, you are saying that we are hitting the aggregate query which surpasses the default set limit for (internalPipelineLengthLimit) 2147483647 and reduce the limit from default to 1000, can i know which config value sets the pipeline limit. Help is much appreciated | ||
| Comment by Patanjali C [ 06/Apr/22 ] | ||
|
Thank you, For helping us with the issue. Will get back to you as soon as possible. | ||
| Comment by Dmitry Agranat [ 06/Apr/22 ] | ||
|
Thank you patanjali@wootcloud.com for providing the requested information. The issue is related to the pipeline length limit (internalPipelineLengthLimit) that you are hitting with your aggregation query. And since the default limit for this parameter was "2147483647" prior to MongoDB 5.0, I suggest lowering the limit to 1000 in order to avoid this situation. Please note that with the new limit of 1000, your operation will just fail instead of crashing the process so I would recommend fixing your aggregation query in any case. | ||
| Comment by Patanjali C [ 05/Apr/22 ] | ||
|
Uploading parameters.txt file | ||
| Comment by Dmitry Agranat [ 05/Apr/22 ] | ||
|
Thanks patanjali@wootcloud.com, I can see the full backtrace now. Could you execute this command to get all the current configurations in the system and send us back the output? You can save the output into some text file and attach it to this ticket.
| ||
| Comment by Patanjali C [ 05/Apr/22 ] | ||
|
I have re uploaded again, please verify. Adding the logs here too | ||
| Comment by Dmitry Agranat [ 05/Apr/22 ] | ||
|
I do not see any uploaded data in the provided secure folder. You can try to upload it again or cat the full backtrace into a text file, compress it and upload it directly into this ticket. | ||
| Comment by Patanjali C [ 05/Apr/22 ] | ||
|
shared the log file from 2022-04-03T00 to 2022-04-04T00 in a gzip format. please verify the logs, It has backtraces that also point to the same error. | ||
| Comment by Dmitry Agranat [ 05/Apr/22 ] | ||
|
patanjali@wootcloud.com, the log snippet you posted is still cut in half and does not include all the backtrace. Please archive and upload the full log into this secure uploader. | ||
| Comment by Patanjali C [ 04/Apr/22 ] | ||
|
Attaching some part of log from today, The mongo log is quite huge so cannot send it here
| ||
| Comment by Dmitry Agranat [ 04/Apr/22 ] | ||
|
patanjali@wootcloud.com, for a start, could you post the full backtrace or attach the relevant log containing the full backtrace + several operations before? |