[SERVER-47553] mongos crashes due to client disconnecting when signing keys being refreshed Created: 15/Apr/20  Updated: 29/Oct/23  Resolved: 30/Apr/20

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 4.2.0
Fix Version/s: 4.2.7, 4.4.0-rc4, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Alice classy Assignee: Gregory Noma
Resolution: Fixed Votes: 0
Labels: sharding-wfbf-day
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File BACKTRACE     File SERVER-47553_repro.diff     Zip Archive log.zip    
Issue Links:
Backports
Duplicate
is duplicated by SERVER-52918 Mongos router repeatedly crashes Closed
Problem/Incident
is caused by SERVER-39150 markKillOnClientDisconnect Closed
Related
related to SERVER-82027 Repeated Restarting of Mongos Process... Closed
is related to SERVER-47037 4.2. mongos crash Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.4, v4.2
Sprint: Sharding 2020-05-04, Sharding 2020-05-18
Participants:
Case:

 Description   
Issue Status as of May 1st, 2020

 ISSUE SUMMARY
When constructing a command response, mongos may need to refresh its set of signing keys if the key for the cluster time isn’t currently known. If the client disconnects before this refresh, the resulting error is not properly handled and causes mongos to crash.

USER IMPACT
mongos will crash with a message saying “operation was interrupted because a client disconnected. terminate() called.”

AFFECTED VERSIONS
This affects 4.2.6 and earlier versions of 4.2.

FIXED VERSIONS
The fix will be included in 4.4.0 and 4.2.7.

Original Summary

Unexpected Shutdown mongos version v4.2.0

Original Description 

Hello,

I'm facing a problem in a PROD server. 3 Mongos nodes was unexpectedly shutdown.
This is a message in the log file when the node down.

"[conn132492] operation was interrupted because a client disconnected
 [conn132492] terminate() called. No exception is active 0x5620c0615da1 0x5620c0615b58 0x5620c0720386 0x5620c07203c1 0x5620bf8a29ad 0x5620bfb1c5f9 0x5620bfb1cb72 0x5620bfa41f28 0x5620bfa6540c 0x5620bfa5f80f 0x5620bfa62a8c 0x5620bfdca2f2 0x5620bfa5d22d 0x5620bfa604c3 0x5620bfa5e8f7 0x5620bfa5f76b 0x5620bfa62a8c 0x5620bfdca75b 0x5620c04bcc64 0x7f4a7735cdd5 0x7f4a77085ead"

 

How do I know the root cause of this issue?
Could anyone help and suggest?



 Comments   
Comment by Jingcheng Li [ 05/Nov/20 ]

Hello,

Our multiple sharding instances also encounter this serious problem again, i think maybe the root issue is because of new signing keys not generated by the monitoring-keys-for-HMAC thread, which is the same issue as SERVER-48709, wish the root issue to be fixed, Thanks!

Comment by Mohammad Ali Tofighi [ 20/Aug/20 ]

Hi everyone.

Our production servers running MongoDB 4.2.1 faced this issue today. Since this issue imposed a 30-minute downtime on our business (and can cause the same to anyone else facing this issue), I would suggest to add this as a critical issue for people to alert them they need to upgrade their sharded cluster before such things happen to them.

Comment by Githook User [ 01/May/20 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-47553 Prevent appendRequiredFieldsToResponse() from throwing inside noexcept block

(cherry picked from commit 39a6dfded77db512ddb729e202b13562bba4c58f)
Branch: v4.2
https://github.com/mongodb/mongo/commit/b3311efddb9dfc7768bc59846ed1e88f17895d0e

Comment by Githook User [ 01/May/20 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-47553 Prevent appendRequiredFieldsToResponse() from throwing inside noexcept block

(cherry picked from commit 39a6dfded77db512ddb729e202b13562bba4c58f)
Branch: v4.4
https://github.com/mongodb/mongo/commit/0c0e00eb34e1da5ec0a8b07f5f71466e1945f14b

Comment by Jumphol Buntuek [ 01/May/20 ]

Dear Technical,

This case is fixed in 4.5.1 but I didn't found it available in mongoDB community repository. I understand that it is not release yet.

Would you mind if I would like to know roadmap of 4.5.1 will be launched?

 

Thank you for you kind support in advance.

Best regards,

Jeans.

 

Comment by Kullaya Maneekarn [ 01/May/20 ]

Dear Githook User,

Could you explain, what is the root cause? or is this a bug in version 4.2.0?

So, i have to upgrade to the version 4.5.1 right?

Thanks and Best Regards,

Comment by Githook User [ 30/Apr/20 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-47553 Prevent appendRequiredFieldsToResponse() from throwing inside noexcept block
Branch: master
https://github.com/mongodb/mongo/commit/39a6dfded77db512ddb729e202b13562bba4c58f

Comment by Gregory Noma [ 29/Apr/20 ]

We've been able to reproduce the issue and are working on a fix.

Comment by Alice classy [ 20/Apr/20 ]

Thank you so much carl.champain!

Best Regards,
Alice

Comment by Carl Champain (Inactive) [ 20/Apr/20 ]

Thank you wizard_club1230@hotmail.com!

I'm passing this ticket along to the appropriate team for additional investigation. Updates will be posted on this ticket as they happen.

Kind regards,
Carl

 

 

Comment by Alice classy [ 18/Apr/20 ]

Hi carl.champain,

Please find the log "BACKTRACE" from the attached file.

 

Thank you

Comment by Carl Champain (Inactive) [ 17/Apr/20 ]

Hi wizard_club1230@hotmail.com,

The logs don't cover when the backtrace happened. To help us investigate this issue, can you please provide the logs covering the backtrace?

Comment by Alice classy [ 15/Apr/20 ]

Hi carl.champain,

I just uploaded the log files, could you please find the log from the attached.

Thank you

Comment by Carl Champain (Inactive) [ 15/Apr/20 ]

Hi wizard_club1230@hotmail.com,

Can you please provide the logs for each node that went down (mongos.log or mongod.log)?

Generated at Thu Feb 08 05:14:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.