[CSHARP-3196] Mongodb 4.4: Got MongoAuthenticationException : Server sent an invalid nonce. Created: 24/Aug/20  Updated: 28/Oct/23  Resolved: 10/Sep/20

Status: Closed
Project: C# Driver
Component/s: Connectivity
Affects Version/s: 2.11.0
Fix Version/s: 2.11.2

Type: Bug Priority: Critical - P2
Reporter: Gian Maria Ricci Assignee: Dmitry Lukyanov (Inactive)
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to CSHARP-3211 Add a multithreaded test in the exist... Backlog
is related to CSHARP-2358 Sporadic exception opening connection... Closed

 Description   

I'm working in a Windows environment, using mongo server version 4.4 (tried with local windows version and linux version in docker).

If you take open source project https://github.com/ProximoSrl/NStore and set environment variables NSTORE_MONGODB to a valid connection value mongodb://admin:mypassword@localhost/nstore?authSource=admin you can verify that some of the tests fails with

MongoAuthenticationException : Server sent an invalid nonce.

The error is reproducible, it always happen.

No error with 4.0 mongodb server.



 Comments   
Comment by Githook User [ 10/Sep/20 ]

Author:

{'name': 'DmitryLukyanov', 'email': 'dmitry.lukyanov@mongodb.com', 'username': 'DmitryLukyanov'}

Message: CSHARP-3196: Mongodb 4.4: Got MongoAuthenticationException : Server sent an invalid nonce.
Branch: v2.11.x
https://github.com/mongodb/mongo-csharp-driver/commit/5e0d8bc82d1d8d9de9aabca057ef4f912dfd71ae

Comment by Githook User [ 10/Sep/20 ]

Author:

{'name': 'DmitryLukyanov', 'email': 'dmitry.lukyanov@mongodb.com', 'username': 'DmitryLukyanov'}

Message: CSHARP-3196: Mongodb 4.4: Got MongoAuthenticationException : Server sent an invalid nonce.
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/8fafe00b5cfedf3690808c805e380d3043e02255

Comment by Mark Weaver [ 03/Sep/20 ]

I stuck a fix in:

https://github.com/mongodb/mongo-csharp-driver/pull/410

I'm not sure I like it much but it might be useful.

Comment by Mark Weaver [ 03/Sep/20 ]

The issue is essentially occurring becase DefaultAuthenticator._speculativeAuthenticator is shared across the concurrently connecting connections, and that holds the state that contains the nonce.  The flow goes something like:

ConnectionInitializer.InitializeConnectionAsync ->
ConnectionInitializer.CreateInitialIsMasterCommand ->
IsMasterHelper.CustomizeCommand ->
DefaultAuthenticator.CustomizeInitialIsMasterCommand ->

This sets DefaultAuthenticator._speculativeAuthenticator to a new ScramSha256Authenticator, and then calls ScramShaAuthenticator.CustomizeInitialIsMasterCommand which calls ScramShaAuthenticator.Initialize which generates and saves a nonce:
    _speculativeAuthenticator = new ScramSha256Authenticator(_credential, _randomStringGenerator);

    _speculativeAuthenticator.CustomizeInitialIsMasterCommand(isMasterCommand)

Note that DefaultAuthenticator is shared across all the connections.  When there are two concurrently connecting connections then the value of _speculativeAuthenticator depends on which connection last set it.

back in InitializeConnectionAsync, the IsMaster command is then executed:
   var isMasterCommand = CreateInitialIsMasterCommand(connection.Settings.Authenticators);
   var isMasterProtocol = IsMasterHelper.CreateProtocol(isMasterCommand);
   var isMasterResult = IsMasterHelper.GetResult(connection, isMasterProtocol, cancellationToken);

the isMaster command contains the per connection generated nonce / response with matching nonce.  InitializeConnectionAsync then proceeds with:
AuthenticationHelper.Authenticate(connection, description, cancellationToken);

which calls into
DefaultAuthenticator.AuthenticateAsync

which calls
DefaultAuthenticator.GetOrCreateAuthenticator – this uses the cached _speculativeAuthenticator (last value).  One of these connections now must have the wrong _speculativeAuthenticator.  Both call into SaslAuthenticator.AuthenticateAsync to process the IsMaster command result.  These then call Transition with _speculativeFirstStep, _speculativeFirstStep.Transition which ends up in ClientFirst.Transition where finally the mismatch is noted and the error gets thrown.

I'm not entirely clear what the correct fix is.

Comment by Jeffrey Yemin [ 03/Sep/20 ]

alkampfer thank you for bringing this to our attention. We are actively investigating it now.

mark.weaver@trelica.com thanks also for adding your repro into the mix.

Comment by Mark Weaver [ 03/Sep/20 ]

I can trivially reproduce this error using the test code at:

https://stackoverflow.com/questions/63223544/server-sent-an-invalid-nonce-when-making-multiple-rapid-connections-from-c-sharp

All on Windows 10 using mongo 4.4 community edition and the 2.11.1 c# driver.

I have tried debugging the c# driver and it appears what is happening is that when two connections are opened simultaneously two separate nonces are generated (with two different ScramShaAuthenticator classes) and then subsequently ScramShaAuthenticator.Transition is called with the wrong response, e.g. I captured:

ScramShaAuthenticator#1.rPrefix "MKS.RaW7GM5z.pJ-{s:Y" string
ScramShaAuthenticator#2.rPrefix "\"Zw/5(R.)*HK@8pcg^g" string

ScramShaAuthenticator#2.Transition map["r"] = "MKS.RaW7GM5z.pJ-{s:Y9Rcih99HWu94edIYZpq157r7VX7TgCDR" string

so the prefixes don't match and

var r = map['r'];
if (!r.StartsWith(_rPrefix, StringComparison.Ordinal))

{      throw new MongoAuthenticationException(conversation.ConnectionId, message: "Server sent an invalid nonce."); }

throws.

Also looking with tcpdump I see that the server reply includes the nonce that the client is sending so that it appears that the issue is with the client rather than the server.

I'm not sure where the mixing up is happening – some pointers would be helpful.

(I also pasted this comment in CSHARP-2358 which has other reports of the same problem but that's closed so I'm adding it to an active one as well, sorry if that's annoying)

Generated at Wed Feb 07 21:44:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.