-
Type:
Bug
-
Resolution: Cannot Reproduce
-
Priority:
Major - P3
-
None
-
Affects Version/s: 2.11.6
-
Component/s: Connectivity
-
Labels:None
-
Environment:.NET Framework 4.8 on Windows x64
MongoDB 4.2.6 on Linux x64
Hi,
The replica set member got restarted, became secondary, and then a batch of requests (with read preference that allowed secondary reads) fail with MongoCommandException (details below). I'm not sure why it happened but it looks like the server is rejecting cached credentials that are not valid after reconnection due to some (I'm speculating here) cache not being cleared after the connection is lost/broken. Unfortunately, this defeats the purpose of retryable writes since even a graceful failover can lead to a query failure with a non-retryable error. This happens almost every time when secondary reads are allowed and the primary node is restarted, becoming a secondary one.
This also could be a server-side issue (given what JIRA search shows up for related errors), but I doubt it.
MongoDB.Driver.MongoCommandException: Command find failed: Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1613369050, 99) } with id: 6893426160802201667.
TCommandResult MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol<TCommandResult>.ProcessResponse(ConnectionId connectionId, CommandMessage responseMessage)
TCommandResult MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol<TCommandResult>.Execute(IConnection connection, CancellationToken cancellationToken)
TCommandResult MongoDB.Driver.Core.WireProtocol.CommandWireProtocol<TCommandResult>.Execute(IConnection connection, CancellationToken cancellationToken)
TResult MongoDB.Driver.Core.Servers.Server+ServerChannel.ExecuteProtocol<TResult>(IWireProtocol<TResult> protocol, ICoreSession session, CancellationToken cancellationToken)
TResult MongoDB.Driver.Core.Servers.Server+ServerChannel.Command<TResult>(ICoreSession session, ReadPreference readPreference, DatabaseNamespace databaseNamespace, BsonDocument command, IEnumerable<Type1CommandMessageSection> commandPayloads, IElementNameValidator commandValidator, BsonDocument additionalOptions, Action<IMessageEncoderPostProcessor> postWriteAction, CommandResponseHandling responseHandling, IBsonSerializer<TResult> resultSerializer, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
TCommandResult MongoDB.Driver.Core.Operations.CommandOperationBase<TCommandResult>.ExecuteProtocol(IChannelHandle channel, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)
TCommandResult MongoDB.Driver.Core.Operations.ReadCommandOperation<TCommandResult>.ExecuteAttempt(RetryableReadContext context, int attempt, long? transactionNumber, CancellationToken cancellationToken)
TResult MongoDB.Driver.Core.Operations.RetryableReadOperationExecutor.Execute<TResult>(IRetryableReadOperation<TResult> operation, RetryableReadContext context, CancellationToken cancellationToken)
TCommandResult MongoDB.Driver.Core.Operations.ReadCommandOperation<TCommandResult>.Execute(RetryableReadContext context, CancellationToken cancellationToken)
IAsyncCursor<TDocument> MongoDB.Driver.Core.Operations.FindCommandOperation<TDocument>.Execute(RetryableReadContext context, CancellationToken cancellationToken)
IAsyncCursor<TDocument> MongoDB.Driver.Core.Operations.FindOperation<TDocument>.Execute(RetryableReadContext context, CancellationToken cancellationToken)
IAsyncCursor<TDocument> MongoDB.Driver.Core.Operations.FindOperation<TDocument>.Execute(IReadBinding binding, CancellationToken cancellationToken)
TResult MongoDB.Driver.OperationExecutor.ExecuteReadOperation<TResult>(IReadBinding binding, IReadOperation<TResult> operation, CancellationToken cancellationToken)
TResult MongoDB.Driver.MongoCollectionImpl<TDocument>.ExecuteReadOperation<TResult>(IClientSessionHandle session, IReadOperation<TResult> operation, ReadPreference readPreference, CancellationToken cancellationToken)
TResult MongoDB.Driver.MongoCollectionImpl<TDocument>.ExecuteReadOperation<TResult>(IClientSessionHandle session, IReadOperation<TResult> operation, CancellationToken cancellationToken)
IAsyncCursor<TProjection> MongoDB.Driver.MongoCollectionImpl<TDocument>.FindSync<TProjection>(IClientSessionHandle session, FilterDefinition<TDocument> filter, FindOptions<TDocument, TProjection> options, CancellationToken cancellationToken)
IAsyncCursor<TProjection> MongoDB.Driver.MongoCollectionImpl<TDocument>.FindSync<TProjection>(IClientSessionHandle session, FilterDefinition<TDocument> filter, FindOptions<TDocument, TProjection> options, CancellationToken cancellationToken)+(IClientSessionHandle session) => { }
TResult MongoDB.Driver.MongoCollectionImpl<TDocument>.UsingImplicitSession<TResult>(Func<IClientSessionHandle, TResult> func, CancellationToken cancellationToken)
IAsyncCursor<TProjection> MongoDB.Driver.MongoCollectionImpl<TDocument>.FindSync<TProjection>(FilterDefinition<TDocument> filter, FindOptions<TDocument, TProjection> options, CancellationToken cancellationToken)
IAsyncCursor<TProjection> Dnet.Mongo.Fx.MongoCollectionDecorator<TDocument>.FindSyncImpl<TProjection>(FilterDefinition<TDocument> filter, FindOptions<TDocument, TProjection> options, CancellationToken cancellationToken) in …/lib/Dnet.Mongo/Fx/Decorators/MongoCollectionDecorator`1.cs:line 893
IAsyncCursor<TProjection> Dnet.Mongo.Fx.MongoCollectionDecorator<TDocument>.FindSync<TProjection>(FilterDefinition<TDocument> filter, FindOptions<TDocument, TProjection> options, CancellationToken cancellationToken)+() => { } in …/lib/Dnet.Mongo/Fx/Decorators/MongoCollectionDecorator`1.cs:line 865
!Trace.Action
IAsyncCursor<TProjection> Dnet.Mongo.Fx.MongoCollectionDecorator<TDocument>.FindSync<TProjection>(FilterDefinition<TDocument> filter, FindOptions<TDocument, TProjection> options, CancellationToken cancellationToken) in …/lib/Dnet.Mongo/Fx/Decorators/MongoCollectionDecorator`1.cs:line 863
IAsyncCursor<TProjection> MongoDB.Driver.FindFluent<TDocument, TProjection>.ToCursor(CancellationToken cancellationToken)
Thanks for looking at this =)