[SERVER-65214] MongoDB crashed with Invalid access at address + Got signal: 11 (Segmentation fault) Created: 04/Apr/22  Updated: 15/May/22  Resolved: 15/May/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.4.10
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Patanjali C Assignee: Dmitry Agranat
Resolution: Duplicate Votes: 0
Labels: Bug
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongo-2022-04-03T00-2022-04-04T00.log.gz     Text File parameters.txt    
Issue Links:
Duplicate
is duplicated by SERVER-50547 Explore aggregation pipeline length l... Closed
Related
related to SERVER-50547 Explore aggregation pipeline length l... Closed
Operating System: ALL
Participants:

 Description   

We are running mongo as a replicaset of 3 nodes as docker containers, The running mongo version is 4.4.10. We are facing constant restarts since yesterday (2022-04-03) morning. This is causing issues with our production. The Backtrace is pointing to 
ZN5mongo12_GLOBAL_N_124abruptQuitWithAddrSignalEiP9siginfo_tPv. We are trying to take back control but mongo restarts have been happening on all 3 nodes.

Resource Consumption on all nodes:
CPU: Avg 3 to 4 %
Memory:  6%

{"t":{"$date":"2022-04-04T10:11:06.355+00:00"},"s":"F", "c":"CONTROL", "id":4757800, "ctx":"conn124","msg":"Writing fatal message","attr":{"message":"Invalid access at address: 0x7fc1411faff8"}}
{"t":{"$date":"2022-04-04T10:11:06.355+00:00"},"s":"F", "c":"CONTROL", "id":4757800, "ctx":"conn124","msg":"Writing fatal message","attr":{"message":"Got signal: 11 (Segmentation fault).\n"}}
{"t":{"$date":"2022-04-04T10:11:06.529+00:00"},"s":"I", "c":"CONTROL", "id":31431, "ctx":"conn124","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"55E53E657BEA","b":"55E53B87F000","o":"2DD8BEA","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"},{"a":"55E53E659679","b":"55E53B87F000","o":"2DDA679","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"55E53E656B4C","b":"55E53B87F000","o":"2DD7B4C","s":"_ZN5mongo12_GLOBAL__N_124abruptQuitWithAddrSignalEiP9siginfo_tPv","s+":"EC"}



 Comments   
Comment by Dmitry Agranat [ 15/May/22 ]

As we have not heard back from you for some time now, I will go ahead and close this ticket.

Comment by Dmitry Agranat [ 11/Apr/22 ]

Yes, either via adding to mongo.conf or by executing setParameter command. As always, please test this in your staging cluster before implementing in Production.

Comment by Patanjali C [ 07/Apr/22 ]

Following up on my previous question can we set the parameters using the mongo.conf, and restarting the service will set the values?

setParameter:
 internalPipelineLengthLimit : 1000

Comment by Patanjali C [ 06/Apr/22 ]

Thank you for pointing out, We can put this parameter inside the mongo conf and on restarting mongo will set the value to 1000 or should we use the admin command to do set the parameter.

Comment by Dmitry Agranat [ 06/Apr/22 ]

Here is the relevant documentation: https://www.mongodb.com/docs/manual/reference/command/setParameter/#setparameter

To set this parameter via setParameter:

db.adminCommand( { setParameter: 1, internalPipelineLengthLimit : 1000 } )

Comment by Patanjali C [ 06/Apr/22 ]

understood, can you somehow point me to the documentation that can help me set the internalPipelineLengthLimit  or point to a method to set in the mongo.conf, This would additionally help in in fixing the config

Thanks 

Comment by Dmitry Agranat [ 06/Apr/22 ]

The config is internalPipelineLengthLimit mentioned in my last comment

Comment by Patanjali C [ 06/Apr/22 ]

so, you are saying that we are hitting the aggregate query which surpasses the default set limit for (internalPipelineLengthLimit) 2147483647 and reduce the limit from default to 1000, can i know which config value sets the pipeline limit. Help is much appreciated
Thanks 

Comment by Patanjali C [ 06/Apr/22 ]

Thank  you, For helping us with the issue. Will get back to you as soon as possible.

Comment by Dmitry Agranat [ 06/Apr/22 ]

Thank you patanjali@wootcloud.com for providing the requested information. The issue is related to the pipeline length limit (internalPipelineLengthLimit) that you are hitting with your aggregation query. And since the default limit for this parameter was "2147483647" prior to MongoDB 5.0, I suggest lowering the limit to 1000 in order to avoid this situation. Please note that with the new limit of 1000, your operation will just fail instead of crashing the process so I would recommend fixing your aggregation query in any case.

Comment by Patanjali C [ 05/Apr/22 ]

Uploading parameters.txt file

parameters.txt

Comment by Dmitry Agranat [ 05/Apr/22 ]

Thanks patanjali@wootcloud.com, I can see the full backtrace now. Could you execute this command to get all the current configurations in the system and send us back the output? You can save the output into some text file and attach it to this ticket.

db.adminCommand( { getParameter : '*' } )

Comment by Patanjali C [ 05/Apr/22 ]

I have re uploaded again, please verify.

Adding the logs here too
mongo-2022-04-03T00-2022-04-04T00.log.gz

Comment by Dmitry Agranat [ 05/Apr/22 ]

I do not see any uploaded data in the provided secure folder. You can try to upload it again or cat the full backtrace into a text file, compress it and upload it directly into this ticket.

Comment by Patanjali C [ 05/Apr/22 ]

shared the log file from 2022-04-03T00 to 2022-04-04T00 in a gzip format. please verify the logs, It has backtraces that also point to the same error. 
Thanks

Comment by Dmitry Agranat [ 05/Apr/22 ]

patanjali@wootcloud.com, the log snippet you posted is still cut in half and does not include all the backtrace. Please archive and upload the full log into this secure uploader.

Comment by Patanjali C [ 04/Apr/22 ]

Attaching some part of log from today, The mongo log is quite huge so cannot send it here

 

{"t":{"$date":"2022-04-04T00:00:45.069+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:46812","connectionId":2312,"connectionCount":167}} {"t":{"$date":"2022-04-04T00:00:45.069+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2312","msg":"client metadata","attr":{"remote":"10.0.1.24:46812","client":"conn2312","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.102+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2311","msg":"Connection ended","attr":{"remote":"10.0.1.24:46570","connectionId":2311,"connectionCount":166}} {"t":{"$date":"2022-04-04T00:00:45.104+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:46928","connectionId":2313,"connectionCount":167}} {"t":{"$date":"2022-04-04T00:00:45.104+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2313","msg":"client metadata","attr":{"remote":"10.0.1.24:46928","client":"conn2313","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.151+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47128","connectionId":2314,"connectionCount":168}} {"t":{"$date":"2022-04-04T00:00:45.151+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2314","msg":"client metadata","attr":{"remote":"10.0.1.24:47128","client":"conn2314","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.160+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2312","msg":"Connection ended","attr":{"remote":"10.0.1.24:46812","connectionId":2312,"connectionCount":167}} {"t":{"$date":"2022-04-04T00:00:45.179+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47210","connectionId":2315,"connectionCount":168}} {"t":{"$date":"2022-04-04T00:00:45.179+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2315","msg":"client metadata","attr":{"remote":"10.0.1.24:47210","client":"conn2315","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.182+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2313","msg":"Connection ended","attr":{"remote":"10.0.1.24:46928","connectionId":2313,"connectionCount":167}} {"t":{"$date":"2022-04-04T00:00:45.188+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47238","connectionId":2316,"connectionCount":168}} {"t":{"$date":"2022-04-04T00:00:45.188+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2316","msg":"client metadata","attr":{"remote":"10.0.1.24:47238","client":"conn2316","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.227+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2314","msg":"Connection ended","attr":{"remote":"10.0.1.24:47128","connectionId":2314,"connectionCount":167}} {"t":{"$date":"2022-04-04T00:00:45.232+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2174","msg":"Connection ended","attr":{"remote":"10.0.1.24:59998","connectionId":2174,"connectionCount":166}} {"t":{"$date":"2022-04-04T00:00:45.250+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2315","msg":"Connection ended","attr":{"remote":"10.0.1.24:47210","connectionId":2315,"connectionCount":165}} {"t":{"$date":"2022-04-04T00:00:45.264+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2316","msg":"Connection ended","attr":{"remote":"10.0.1.24:47238","connectionId":2316,"connectionCount":164}} {"t":{"$date":"2022-04-04T00:00:45.364+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.142:60368","connectionId":2317,"connectionCount":165}} {"t":{"$date":"2022-04-04T00:00:45.364+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2317","msg":"client metadata","attr":{"remote":"10.0.1.142:60368","client":"conn2317","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.383+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47588","connectionId":2318,"connectionCount":166}} {"t":{"$date":"2022-04-04T00:00:45.383+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2318","msg":"client metadata","attr":{"remote":"10.0.1.24:47588","client":"conn2318","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.440+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47630","connectionId":2319,"connectionCount":167}} {"t":{"$date":"2022-04-04T00:00:45.440+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2319","msg":"client metadata","attr":{"remote":"10.0.1.24:47630","client":"conn2319","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.457+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2318","msg":"Connection ended","attr":{"remote":"10.0.1.24:47588","connectionId":2318,"connectionCount":166}} {"t":{"$date":"2022-04-04T00:00:45.465+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2317","msg":"Connection ended","attr":{"remote":"10.0.1.142:60368","connectionId":2317,"connectionCount":165}} {"t":{"$date":"2022-04-04T00:00:45.513+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2319","msg":"Connection ended","attr":{"remote":"10.0.1.24:47630","connectionId":2319,"connectionCount":164}} {"t":{"$date":"2022-04-04T00:00:45.561+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.141:43224","connectionId":2320,"connectionCount":165}} {"t":{"$date":"2022-04-04T00:00:45.562+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2320","msg":"client metadata","attr":{"remote":"10.0.1.141:43224","client":"conn2320","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.562+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.141:43240","connectionId":2321,"connectionCount":166}} {"t":{"$date":"2022-04-04T00:00:45.562+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2321","msg":"client metadata","attr":{"remote":"10.0.1.141:43240","client":"conn2321","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.562+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.141:43246","connectionId":2322,"connectionCount":167}} {"t":{"$date":"2022-04-04T00:00:45.563+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.141:43254","connectionId":2323,"connectionCount":168}} {"t":{"$date":"2022-04-04T00:00:45.563+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2323","msg":"Connection ended","attr":{"remote":"10.0.1.141:43254","connectionId":2323,"connectionCount":167}} {"t":{"$date":"2022-04-04T00:00:45.563+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2322","msg":"client metadata","attr":{"remote":"10.0.1.141:43246","client":"conn2322","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.563+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47654","connectionId":2324,"connectionCount":168}} {"t":{"$date":"2022-04-04T00:00:45.563+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2322","msg":"Connection ended","attr":{"remote":"10.0.1.141:43246","connectionId":2322,"connectionCount":167}} {"t":{"$date":"2022-04-04T00:00:45.563+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47656","connectionId":2325,"connectionCount":168}} {"t":{"$date":"2022-04-04T00:00:45.563+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2324","msg":"client metadata","attr":{"remote":"10.0.1.24:47654","client":"conn2324","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.563+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2325","msg":"client metadata","attr":{"remote":"10.0.1.24:47656","client":"conn2325","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47668","connectionId":2326,"connectionCount":169}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.142:60378","connectionId":2327,"connectionCount":170}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2326","msg":"Connection ended","attr":{"remote":"10.0.1.24:47668","connectionId":2326,"connectionCount":169}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.141:43258","connectionId":2328,"connectionCount":170}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2327","msg":"client metadata","attr":{"remote":"10.0.1.142:60378","client":"conn2327","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47674","connectionId":2329,"connectionCount":171}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2328","msg":"Connection ended","attr":{"remote":"10.0.1.141:43258","connectionId":2328,"connectionCount":170}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47672","connectionId":2330,"connectionCount":171}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2329","msg":"client metadata","attr":{"remote":"10.0.1.24:47674","client":"conn2329","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47670","connectionId":2331,"connectionCount":172}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2330","msg":"Connection ended","attr":{"remote":"10.0.1.24:47672","connectionId":2330,"connectionCount":171}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.141:43260","connectionId":2332,"connectionCount":172}} {"t":{"$date":"2022-04-04T00:00:45.564+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2331","msg":"Connection ended","attr":{"remote":"10.0.1.24:47670","connectionId":2331,"connectionCount":171}} {"t":{"$date":"2022-04-04T00:00:45.565+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.142:60382","connectionId":2333,"connectionCount":172}} {"t":{"$date":"2022-04-04T00:00:45.565+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2332","msg":"Connection ended","attr":{"remote":"10.0.1.141:43260","connectionId":2332,"connectionCount":171}} {"t":{"$date":"2022-04-04T00:00:45.565+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2329","msg":"Connection ended","attr":{"remote":"10.0.1.24:47674","connectionId":2329,"connectionCount":170}} {"t":{"$date":"2022-04-04T00:00:45.565+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47678","connectionId":2334,"connectionCount":171}} {"t":{"$date":"2022-04-04T00:00:45.565+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2333","msg":"client metadata","attr":{"remote":"10.0.1.142:60382","client":"conn2333","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:45.565+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:47680","connectionId":2335,"connectionCount":172}} {"t":{"$date":"2022-04-04T00:00:45.565+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2334","msg":"Connection ended","attr":{"remote":"10.0.1.24:47678","connectionId":2334,"connectionCount":171}} {"t":{"$date":"2022-04-04T00:00:45.565+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2335","msg":"Connection ended","attr":{"remote":"10.0.1.24:47680","connectionId":2335,"connectionCount":170}} {"t":{"$date":"2022-04-04T00:00:45.566+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2333","msg":"Connection ended","attr":{"remote":"10.0.1.142:60382","connectionId":2333,"connectionCount":169}} {"t":{"$date":"2022-04-04T00:00:47.300+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.30.2.39:55899","connectionId":2336,"connectionCount":170}} {"t":{"$date":"2022-04-04T00:00:47.303+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2336","msg":"client metadata","attr":{"remote":"10.30.2.39:55899","client":"conn2336","doc":{"driver":{"name":"PyMongo","version":"3.7.2"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.4.181-99.354.amzn2.x86_64"},"platform":"CPython 3.6.15.final.0"}}} {"t":{"$date":"2022-04-04T00:00:47.368+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn1901","msg":"Slow query","attr":{"type":"command","ns":"5f6d2f2287b4a2000130c92b.audit","command":{"aggregate":"audit","pipeline":[{"$match":{"device.managed":true}},{"$match":{"isNoise":false}},{"$match":{"device.services.id":{"$ne":{"$oid":"60cb78695c727ddfb1f72f76"}}}},{"$match":{"device.control":"user"}},{"$match":{"device.os":{"$in":["windows","macos","ios_macos"]}}},{"$group":{"_id":"$device_id","devices":{"$addToSet":"$device"}}},{"$project":{"_id":"$_id","device":{"$arrayElemAt":["$devices",0]}}},{"$count":"count"}],"cursor":{},"lsid":{"id":{"$uuid":"8c97b720-2832-4347-9a35-246d22e32748"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1649030442,"i":4401}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}},"$db":"5f6d2f2287b4a2000130c92b","$readPreference":{"mode":"secondaryPreferred"}},"planSummary":"IXSCAN { isNoise: 1 }","keysExamined":1785571,"docsExamined":1785571,"cursorExhausted":true,"numYields":1785,"nreturned":1,"queryHash":"8095C105","planCacheKey":"0F54D8D0","reslen":263,"locks":{"ReplicationStateTransition":{"acquireCount":{"w":1787}},"Global":{"acquireCount":{"r":1787}},"Database":{"acquireCount":{"r":1787}},"Collection":{"acquireCount":{"r":1787}},"Mutex":{"acquireCount":{"r":2}}},"storage":{},"protocol":"op_msg","durationMillis":4894}} {"t":{"$date":"2022-04-04T00:00:49.219+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.30.6.32:15407","connectionId":2337,"connectionCount":171}} {"t":{"$date":"2022-04-04T00:00:49.219+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2337","msg":"client metadata","attr":{"remote":"10.30.6.32:15407","client":"conn2337","doc":{"driver":{"name":"PyMongo|Motor","version":"3.12.3|2.5.1"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.4.172-90.336.amzn2.x86_64"},"platform":"CPython 3.7.12.final.0|asyncio"}}} {"t":{"$date":"2022-04-04T00:00:49.480+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn2337","msg":"Slow query","attr":{"type":"command","ns":"5a00b1759c7964880fa1c1a6.audit","command":{"aggregate":"audit","pipeline":[{"$match":{"device_id":{"$exists":true},"device.ownership":{"$exists":true,"$in":["corporate","corporate-unmanaged","employee-owned","undetermined","visiting"]}}},{"$match":{"$or":[{"lastseen":{"$gte":{"$date":"2022-04-03T23:55:49.072Z"},"$lte":{"$date":"2022-04-04T00:00:49.072Z"}}},{"last_updated":{"$gte":{"$date":"2022-04-03T23:55:49.072Z"},"$lte":{"$date":"2022-04-04T00:00:49.072Z"}}}]}},{"$group":{"_id":"$device_id"}}],"allowDiskUse":true,"cursor":{},"lsid":{"id":{"$uuid":"dd7decfd-c0bf-4abb-bcaa-863128f24143"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1649030443,"i":494}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}},"$db":"5a00b1759c7964880fa1c1a6","$readPreference":{"mode":"secondaryPreferred"}},"planSummary":"IXSCAN { device_id: 1 }","keysExamined":73132,"docsExamined":73132,"cursorExhausted":true,"numYields":73,"nreturned":24,"queryHash":"DBFA6FA9","planCacheKey":"458BE65A","reslen":1266,"locks":{"ReplicationStateTransition":{"acquireCount":{"w":75}},"Global":{"acquireCount":{"r":75}},"Database":{"acquireCount":{"r":75}},"Collection":{"acquireCount":{"r":75}},"Mutex":{"acquireCount":{"r":2}}},"storage":{"data":{"bytesRead":59603893,"timeReadingMicros":47303}},"protocol":"op_msg","durationMillis":260}} {"t":{"$date":"2022-04-04T00:00:49.810+00:00"},"s":"F", "c":"CONTROL", "id":4757800, "ctx":"conn2337","msg":"Writing fatal message","attr":{"message":"Invalid access at address: 0x7f91da4eaff8"}} {"t":{"$date":"2022-04-04T00:00:49.810+00:00"},"s":"F", "c":"CONTROL", "id":4757800, "ctx":"conn2337","msg":"Writing fatal message","attr":{"message":"Got signal: 11 (Segmentation fault).\n"}} {"t":{"$date":"2022-04-04T00:00:49.943+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.24:48294","connectionId":2338,"connectionCount":172}} {"t":{"$date":"2022-04-04T00:00:49.945+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2338","msg":"client metadata","attr":{"remote":"10.0.1.24:48294","client":"conn2338","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:49.974+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.141:43876","connectionId":2339,"connectionCount":173}} {"t":{"$date":"2022-04-04T00:00:49.974+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2339","msg":"client metadata","attr":{"remote":"10.0.1.141:43876","client":"conn2339","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:49.976+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.141:43882","connectionId":2340,"connectionCount":174}} {"t":{"$date":"2022-04-04T00:00:49.976+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2340","msg":"client metadata","attr":{"remote":"10.0.1.141:43882","client":"conn2340","doc":{"driver":{"name":"mongo-java-driver","version":"unknown"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"4.9.116-43.59.amzn1.x86_64"},"platform":"Java/Oracle Corporation/1.8.0_171-b10"}}} {"t":{"$date":"2022-04-04T00:00:49.979+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2338","msg":"Connection ended","attr":{"remote":"10.0.1.24:48294","connectionId":2338,"connectionCount":173}} {"t":{"$date":"2022-04-04T00:00:49.980+00:00"},"s":"I", "c":"CONTROL", "id":31431, "ctx":"conn2337","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"55C1EB859BEA","b":"55C1E8A81000","o":"2DD8BEA","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"},{"a":"55C1EB85B679","b":"55C1E8A81000","o":"2DDA679","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"55C1EB858B4C","b":"55C1E8A81000","o":"2DD7B4C","s":"_ZN5mongo12_GLOBAL__N_124abruptQuitWithAddrSignalEiP9siginfo_tPv","s+":"EC"},{"a":"7F920B48E3C0","b":"7F920B479000","o":"153C0","s":"funlockfile","s+":"60"},{"a":"55C1EA737FC2","b":"55C1E8A81000","o":"1CB6FC2","s":"_ZN5mongo42DocumentSourceSingleDocumentTransformation9doGetNextEv","s+":"22"},{"a":"55C1EA737FEC","b":"55C1E8A81000","o":"1CB6FEC","s":"_ZN5mongo42DocumentSourceSingleDocumentTransformation9doGetNextEv","s+":"4C"},{"a":"55C1EA737FEC","b":"55C1E8A81000","o":"1CB6FEC","s":"_ZN5mongo42DocumentSourceSingleDocumentTransformation9doGetNextEv","s+":"4C"},{"a":"55C1EA737FEC","b":"55C1E8A81000","o":"1CB6FEC","s":"_ZN5mongo42DocumentSourceSingleDocumentTransformation9doGetNextEv","s+":"4C"},{"a":"55C1EA737FEC","b":"55C1E8A81000","o":"1CB6FEC","s":"_ZN5mongo42DocumentSourceSingleDocumentTransformation9doGetNextEv","s+":"4C"},{"a":"55C1EA737FEC","b":"55C1E8A81000","o":"1CB6FEC","s":"_ZN5mongo42DocumentSourceSingleDocumentTransformation9doGetNextEv","s+":"4C"},{"a":"55C1EA737FEC","b":"55C1E8A81000","o":"1CB6FEC","s":"_ZN5mongo42DocumentSourceSingleDocumentTransformation9doGetNextEv","s+":"4C"},{"a":"55C1EA737FEC","b":"55C1E8A81000","o":"1CB6FEC","s":"_ZN5mongo42DocumentSourceSingleDocumentTransformation9doGetNextEv","s+":"4C"}, 

Comment by Dmitry Agranat [ 04/Apr/22 ]

patanjali@wootcloud.com, for a start, could you post the full backtrace or attach the relevant log containing the full backtrace + several operations before?

Generated at Thu Feb 08 06:02:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.