[SERVER-53334] Hitting unexpected invariant failure, causing server shutdown Created: 10/Dec/20  Updated: 29/Oct/23  Resolved: 04/Jan/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.9.0, 4.2.15, 4.4.7

Type: Bug Priority: Critical - P2
Reporter: Jeffrey Yemin Assignee: Benjamin Caimano (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File repro-SERVER-53334.py    
Issue Links:
Backports
Depends
Duplicate
is duplicated by SERVER-56935 "ClientMetadata was previously set" i... Closed
Related
related to SERVER-49336 Set client metadata if it is missing ... Closed
is related to PYTHON-2470 Investigate if mongo-orchestration ca... Closed
is related to SERVER-53400 Write tests for verifying client meta... Backlog
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4, v4.2
Sprint: Service arch 2020-12-28
Participants:
Linked BF Score: 175

 Description   

Testing latest server release, we are hitting an unexpected invariant:

ClientMetadata was previously set, it should be set precisely once","file":"src/mongo/rpc/metadata/client_metadata.cpp","line":545"

It's happening due to a combination of factors:

  1. We are testing against a server with requireApiVersion parameter set to true, which causes the isMaster handshake command to fail.
  2. For unrelated reasons, we are using a version of PyMongo with a bug that exposes the invariant (PYTHON-1434) by retrying the ismaster on the same connection.  It will be difficult to upgrade PyMongo in the short term.

What we expect to happen is that the server uasserts and perhaps closes the connection. What actually happens is that we hit the invariant and the server shuts down.

This behavior seems to have been introduced relatively recently, in scope of SERVER-49336. I discussed it briefly with the author of that commit, and it doesn't appear that this behavior is intended. Instead, we should first be hitting this uassert, but it seems that we are not.

Server logs are available here.



 Comments   
Comment by Githook User [ 20/May/21 ]

Author:

{'name': 'Ben Caimano', 'email': 'ben.caimano@10gen.com', 'username': 'bcaimano'}

Message: SERVER-53334 User assert instead of invariant in ClientMetadata

(cherry picked from commit 84cb1c85a16cd62a0fb8bec1958262adb002ae8d)
Branch: v4.2
https://github.com/mongodb/mongo/commit/641bdc9e26743d4c792765e3020ba1cd161cf3c2

Comment by Githook User [ 19/May/21 ]

Author:

{'name': 'Ben Caimano', 'email': 'ben.caimano@10gen.com', 'username': 'bcaimano'}

Message: SERVER-53334 User assert instead of invariant in ClientMetadata

(cherry picked from commit 84cb1c85a16cd62a0fb8bec1958262adb002ae8d)
Branch: v4.4
https://github.com/mongodb/mongo/commit/c2c51d75fe06f945dd815d270ea9bcc8822dd2c9

Comment by A. Jesse Jiryu Davis [ 19/May/21 ]

I backported SERVER-49336 to 4.4 and 4.2, now I'm requesting a backport of this ticket to fix the same crash in 4.4 and 4.2, exposed in embedded_sdk tests.

Comment by Benjamin Caimano (Inactive) [ 04/Jan/21 ]

If anyone thinks there is more work to be done, please feel free to reopen this ticket.

Comment by Githook User [ 04/Jan/21 ]

Author:

{'name': 'Ben Caimano', 'email': 'ben.caimano@10gen.com'}

Message: SERVER-53334 User assert instead of invariant in ClientMetadata
Branch: master
https://github.com/mongodb/mongo/commit/84cb1c85a16cd62a0fb8bec1958262adb002ae8d

Generated at Thu Feb 08 05:30:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.