[SERVER-27485] mongos 3.2.4 use 100% CPU Created: 21/Dec/16  Updated: 17/Feb/17  Resolved: 17/Feb/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Akihiro Tatematsu Assignee: Kaloian Manassiev
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive 20161222_load.zip     PNG File mongo_load.png     PNG File mongos2.png    
Operating System: ALL
Participants:

 Description   

I'm using mongos 3.2.4 on CentOS 6.5.
Today, mongos use 100% CPU suddenly and calm down.
mongos log didn't have any errors.
I have attached load log.

Do you have any bug information?
And what should I do when this problem is occurred again?



 Comments   
Comment by Kaloian Manassiev [ 17/Feb/17 ]

Hi akihiro.tatematsu@rakuten.com,

Apologies for the delayed response. Like you observed, there are no errors or warnings in the mongos logs. I also looked at the number of connections established and I don't see anything abnormal either.

That being said, in version 3.2.12, which came out this month, we made numerous fixes which make the mongos to mongod communication more efficient. I would recommend that you upgrade to this release and see whether the issue still persists.

For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.

Best regards,
-Kal.

Comment by Akihiro Tatematsu [ 06/Jan/17 ]

Hello kaloian.manassiev, anonymous.user

We faced same trouble today.
I have attached load pics.
Please check it.

Thank you,
Akihiro

Comment by Akihiro Tatematsu [ 06/Jan/17 ]

Hi kaloian.manassiev, anonymous.user

Were you able to open the log of mongos??
If you checked it already, please share a current status of this ticket.

Thank you,
Akihiro

Comment by Akihiro Tatematsu [ 26/Dec/16 ]

Hi anonymous.user,

Thank you for sharing about a secure upload portal site.
I uploaded the log of mongos that was masked hostname and IP(1st-3rd octet).

Regards,
Akihiro

Comment by Kelsey Schubert [ 22/Dec/16 ]

Hi akihiro.tatematsu@rakuten.com,

I've created a secure upload portal for you to use. Files uploaded this portal are only visible to MongoDB employees and are routinely deleted after some time.

Thank you,
Thomas

Comment by Akihiro Tatematsu [ 22/Dec/16 ]

Hi Kaloian Manassiev,

> Can you please attach the log from mongos and the load from the same time interval from the shards?
We have 5 shards.
I have attached the load logs.

Can I send the log of mongos to your e-mail address for security?

Thank you,
Akihiro

Comment by Kaloian Manassiev [ 21/Dec/16 ]

Hi minsoo.jun@rakuten.com,

Is it possible that the specified mongos simply received increased load for the duration of the time when the problem occurred? Do you have a graph of whether there was increased load on the shards as well?

The number of established connections increasing might mean that this mongos received many incoming connections which triggered more load. Can you please attach the log from mongos and the load from the same time interval from the shards?

Best regards,
-Kal.

Comment by Minsoo Jun [ 21/Dec/16 ]

Additional information.
We have 2 mongos server. but only 1 mongos server happen problem.
The disturbing difference is number of "tcp ESTABLISHED"

Server one: 13387 <- this server have a errors
Server two: 2588

"check command"

netstat -an | grep ESTA | wc -l

Generated at Thu Feb 08 04:15:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.