Loading...

XML

Word

Printable

JSON

Type: Bug
Resolution: Gone away
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- external-user

Case:

Documentation Changes:
Not Needed

Hi guys,

We are facing an issue that happened several times a day. Our java driver version is 4.0.5 and Mongo DB version is 4.0.9. We have 2 routers and several nodes. And we are using reactive stream. This issue only happens in our production env. It usually first occurred while MongodbHealthIndicator checking the connections with mongo db. Please see the stacktrace as below:

org.springframework.dao.InvalidDataAccessResourceUsageException: The reply message length 1013281644 is greater than the maximum message length 48000000; nested exception is com.mongodb.MongoInternalException: The reply message length 1013281644 is greater than the maximum message length 48000000org.springframework.dao.InvalidDataAccessResourceUsageException: The reply message length 1013281644 is greater than the maximum message length 48000000; nested exception is com.mongodb.MongoInternalException: The reply message length 1013281644 is greater than the maximum message length 48000000 at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:92) Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s): |_ checkpoint ⇢ com.active.platform.modules.pos.service.config.ContextFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ HTTP GET "/agencies/f953b9de-2260-4833-bf32-d7be2a9ae945/catalogs" [ExceptionHandlingWebHandler]Stack trace: at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:92) at org.springframework.data.mongodb.core.ReactiveMongoTemplate.potentiallyConvertRuntimeException(ReactiveMongoTemplate.java:2814) at org.springframework.data.mongodb.core.ReactiveMongoTemplate.lambda$translateException$90(ReactiveMongoTemplate.java:2797) at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3286) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88) at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:185) at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:251) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165) at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87) at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87) at com.mongodb.reactivestreams.client.internal.AbstractSubscription.onError(AbstractSubscription.java:142) at com.mongodb.reactivestreams.client.internal.SingleResultCallbackSubscription.lambda$requestInitialData$0(SingleResultCallbackSubscription.java:41) at com.mongodb.internal.async.client.AsyncFindIterableImpl$1.onResult(AsyncFindIterableImpl.java:188) at com.mongodb.internal.async.client.AsyncFindIterableImpl$1.onResult(AsyncFindIterableImpl.java:184) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.async.client.OperationExecutorImpl$1$1$1.onResult(OperationExecutorImpl.java:92) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.operation.FindOperation$3.onResult(FindOperation.java:724) at com.mongodb.internal.operation.OperationHelper$ReferenceCountedReleasingWrappedCallback.onResult(OperationHelper.java:411) at com.mongodb.internal.operation.CommandOperationHelper$10.checkRetryableException(CommandOperationHelper.java:493) at com.mongodb.internal.operation.CommandOperationHelper$10.onResult(CommandOperationHelper.java:478) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor$2.onResult(DefaultServer.java:248) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.CommandProtocolImpl$1.onResult(CommandProtocolImpl.java:82) at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection$2.onResult(DefaultConnectionPool.java:517) at com.mongodb.internal.connection.UsageTrackingInternalConnection$2.onResult(UsageTrackingInternalConnection.java:111) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:380) at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:375) at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:635) at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:618) at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513) at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510) at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232) at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510) at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75) at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:375) at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:364) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:467) at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:464) at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:187) at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:181) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604) at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48) at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717) at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272) at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352) at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:414) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898) at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372) at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765) at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)Caused by: com.mongodb.MongoInternalException: The reply message length 1013281644 is greater than the maximum message length 48000000 at com.mongodb.internal.connection.MessageHeader.<init>(MessageHeader.java:41) at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:632) at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:618) at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513) at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510) at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232) at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510) at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75) at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:375) at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:364) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:467) at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:464) at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:187) at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:181) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604) at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48) at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717) at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272) at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352) at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:414) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898) at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372) at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765) at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

After about 20+ times of this error(the reply message length is different every time). The error will become


The responseTo (2342443) in the response does not match the requestId (2342522) in the request

*The responseTo is always smaller than the request id *

The full stack trace is

org.springframework.dao.InvalidDataAccessResourceUsageException: The responseTo (2342443) in the response does not match the requestId (2342522) in the request; nested exception is com.mongodb.MongoInternalException: The responseTo (2342443) in the response does not match the requestId (2342522) in the request
	at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:92)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ com.active.platform.modules.pos.service.config.ContextFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ HTTP POST "/agencies/3c91bc7d-2581-46b8-ac39-9b46cdbb7bc4/catalogs" [ExceptionHandlingWebHandler]
Stack trace:
		at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:92)
		at org.springframework.data.mongodb.core.ReactiveMongoTemplate.potentiallyConvertRuntimeException(ReactiveMongoTemplate.java:2814)
		at org.springframework.data.mongodb.core.ReactiveMongoTemplate.lambda$translateException$90(ReactiveMongoTemplate.java:2797)
		at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3286)
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88)
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:185)
		at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:251)
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165)
		at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87)
		at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87)
		at com.mongodb.reactivestreams.client.internal.AbstractSubscription.onError(AbstractSubscription.java:142)
		at com.mongodb.reactivestreams.client.internal.SingleResultCallbackSubscription.lambda$requestInitialData$0(SingleResultCallbackSubscription.java:41)
		at com.mongodb.internal.async.client.AsyncFindIterableImpl$1.onResult(AsyncFindIterableImpl.java:188)
		at com.mongodb.internal.async.client.AsyncFindIterableImpl$1.onResult(AsyncFindIterableImpl.java:184)
		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
		at com.mongodb.internal.async.client.OperationExecutorImpl$1$1$1.onResult(OperationExecutorImpl.java:92)
		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
		at com.mongodb.internal.operation.FindOperation$3.onResult(FindOperation.java:724)
		at com.mongodb.internal.operation.OperationHelper$ReferenceCountedReleasingWrappedCallback.onResult(OperationHelper.java:411)
		at com.mongodb.internal.operation.CommandOperationHelper$10.checkRetryableException(CommandOperationHelper.java:493)
		at com.mongodb.internal.operation.CommandOperationHelper$10.onResult(CommandOperationHelper.java:478)
		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
		at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor$2.onResult(DefaultServer.java:248)
		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
		at com.mongodb.internal.connection.CommandProtocolImpl$1.onResult(CommandProtocolImpl.java:82)
		at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection$2.onResult(DefaultConnectionPool.java:517)
		at com.mongodb.internal.connection.UsageTrackingInternalConnection$2.onResult(UsageTrackingInternalConnection.java:111)
		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
		at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:400)
		at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:375)
		at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:676)
		at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:643)
		at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513)
		at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)
		at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232)
		at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510)
		at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75)
		at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:633)
		at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:618)
		at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513)
		at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)
		at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232)
		at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510)
		at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75)
		at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:375)
		at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:364)
		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
		at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:467)
		at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:464)
		at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:187)
		at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:181)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
		at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
		at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
		at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
		at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
		at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717)
		at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272)
		at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352)
		at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:414)
		at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354)
		at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898)
		at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
		at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
		at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
		at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
		at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.lang.Thread.run(Thread.java:748)
Caused by: com.mongodb.MongoInternalException: The responseTo (2342443) in the response does not match the requestId (2342522) in the request
	at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:65)
	at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:43)
	at com.mongodb.internal.connection.ResponseBuffers.getResponseDocument(ResponseBuffers.java:47)
	at com.mongodb.internal.connection.LoggingCommandEventSender.sendSucceededEvent(LoggingCommandEventSender.java:141)
	at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:395)
	at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:375)
	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:676)
	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:643)
	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513)
	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)
	at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232)
	at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510)
	at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75)
	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:633)
	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:618)
	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513)
	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)
	at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232)
	at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510)
	at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75)
	at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:375)
	at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:364)
	at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
	at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:467)
	at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:464)
	at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:187)
	at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:181)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
	at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
	at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717)
	at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272)
	at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352)
	at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:414)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
	at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

I investigated the related source code, I guess that when the first error happened. It seems that mongo has replied an unknown message to the driver. When the driver decode the message follow the wire protocol, it found that the message length is too big(reading the first 16 bytes from the stream) and throw an exception then ignored the rest bytes. The following requests will keep doing the same until meeting the first correct reply message but the responseTo in the reply message will be smaller than the current request id.

I can't reproduce this on my laptop. I'm wondering why the driver needs to check the message size or the responseTo is matching the request id. Is there a concern regarding this? Why not check if the reply message is matching the wire protocol?
I guess this issue is more likely a network issue or protocol issue. Could you advise me on how to find the root cause?

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

service_2.log
14.02 MB
Nov 03 2021 07:02:36 AM UTC
service_1.log
2.05 MB
Nov 03 2021 06:40:05 AM UTC
POS-vin11c9a-eth0-2021_1020_0622_27.pcap.zip
16.27 MB
Nov 05 2021 02:18:22 AM UTC
image-2021-11-03-14-46-40-340.png
903 kB
Nov 03 2021 06:46:43 AM UTC

clones

JAVA-4210 MongoDb returns message length larger than the max message length

Closed

Assignee:: Valentin Kavalenka

Reporter:: 运军蒋

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: Nov 03 2021 05:49:34 AM UTC

Updated:: Oct 27 2023 07:48:31 PM UTC

Resolved:: Nov 30 2021 12:00:58 PM UTC

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates