Details
-
Improvement
-
Resolution: Won't Do
-
Unknown
-
None
-
None
-
None
-
None
Description
colored textSomewhat related to JAVA-4974
https://github.com/mongodb/mongo-java-driver/pull/1181
With thrown exceptions only showing one of errors, it is hard to tell whether the operations are being retried or not. Patching our version of the Mongo Java driver to log all the errors has shown which errors are being retried and which are not.
Here is an example of an error being retried, as you can see two different servers are being tried:
org.springframework.data.mongodb.UncategorizedMongoDbException: Connection pool for ServerId{clusterId=ClusterId{value='64de2b2fa0992c193b36462f', description='joom-ranker'}, address=10-0-61-13.backend-mongos-production-user-critical.backend-mongos.svc.cluster.local:27017} is paused because another operation failed; nested exception is com.mongodb.MongoConnectionPoolClearedException: Connection pool for ServerId{clusterId=ClusterId{value='64de2b2fa0992c193b36462f', description='joom-ranker'}, address=10-0-61-13.backend-mongos-production-user-critical.backend-mongos.svc.cluster.local:27017} is paused because another operation failed |
at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:133) |
at org.springframework.data.mongodb.core.ReactiveMongoTemplate.potentiallyConvertRuntimeException(ReactiveMongoTemplate.java:2814) |
at org.springframework.data.mongodb.core.ReactiveMongoTemplate.lambda$translateException$90(ReactiveMongoTemplate.java:2797) |
at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3325) |
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88) |
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:185) |
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:251) |
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165) |
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165) |
at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:126) |
at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87) |
at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87) |
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:185) |
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:251) |
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:251) |
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:190) |
at com.mongodb.reactivestreams.client.internal.MongoOperationPublisher.lambda$sinkToCallback$31(MongoOperationPublisher.java:570) |
at com.mongodb.reactivestreams.client.internal.OperationExecutorImpl.lambda$execute$2(OperationExecutorImpl.java:92) |
at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) |
at com.mongodb.internal.async.function.AsyncCallbackSupplier.lambda$whenComplete$1(AsyncCallbackSupplier.java:95) |
at com.mongodb.internal.async.function.RetryingAsyncCallbackSupplier$RetryingCallback.onResult(RetryingAsyncCallbackSupplier.java:109) |
at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) |
at com.mongodb.internal.async.function.AsyncCallbackSupplier.lambda$whenComplete$1(AsyncCallbackSupplier.java:95) |
at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) |
at com.mongodb.internal.operation.OperationHelper.lambda$withAsyncSuppliedResource$7(OperationHelper.java:640) |
at com.mongodb.internal.connection.DefaultServer$1.onResult(DefaultServer.java:130) |
at com.mongodb.internal.connection.DefaultServer$1.onResult(DefaultServer.java:122) |
at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) |
at com.mongodb.internal.connection.DefaultConnectionPool.lambda$getAsync$0(DefaultConnectionPool.java:199) |
at com.mongodb.internal.connection.DefaultConnectionPool.lambda$getAsync$1(DefaultConnectionPool.java:216) |
at com.mongodb.internal.connection.DefaultConnectionPool$Task.doComplete(DefaultConnectionPool.java:1382) |
at com.mongodb.internal.connection.DefaultConnectionPool$Task.execute(DefaultConnectionPool.java:1368) |
at com.mongodb.internal.connection.DefaultConnectionPool$AsyncWorkManager.workerRun(DefaultConnectionPool.java:1309) |
at com.mongodb.internal.connection.DefaultConnectionPool$AsyncWorkManager.runAndLogUncaught(DefaultConnectionPool.java:1339) |
at com.mongodb.internal.connection.DefaultConnectionPool$AsyncWorkManager.lambda$initUnlessClosed$0(DefaultConnectionPool.java:1274) |
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) |
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) |
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) |
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) |
at java.base/java.lang.Thread.run(Thread.java:829) |
Caused by: com.mongodb.MongoConnectionPoolClearedException: Connection pool for ServerId{clusterId=ClusterId{value='64de2b2fa0992c193b36462f', description='joom-ranker'}, address=10-0-61-13.backend-mongos-production-user-critical.backend-mongos.svc.cluster.local:27017} is paused because another operation failed |
at com.mongodb.internal.connection.DefaultConnectionPool$StateAndGeneration.lambda$pauseAndIncrementGeneration$0(DefaultConnectionPool.java:1491) |
at com.mongodb.internal.connection.ConcurrentPool$StateAndPermits.throwIfClosedOrPaused(ConcurrentPool.java:498) |
at com.mongodb.internal.connection.ConcurrentPool$StateAndPermits.acquirePermit(ConcurrentPool.java:401) |
at com.mongodb.internal.connection.ConcurrentPool.get(ConcurrentPool.java:152) |
at com.mongodb.internal.connection.DefaultConnectionPool.getPooledConnection(DefaultConnectionPool.java:304) |
at com.mongodb.internal.connection.DefaultConnectionPool.lambda$getAsync$1(DefaultConnectionPool.java:214) |
... 10 common frames omitted |
Suppressed: com.mongodb.MongoConnectionPoolClearedException: Connection pool for ServerId{clusterId=ClusterId{value='64de2b2fa0992c193b36462f', description='joom-ranker'}, address=10-0-53-94.backend-mongos-production-user-critical.backend-mongos.svc.cluster.local:27017} is paused because another operation failed |
... 16 common frames omitted |
Caused by: com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message |
at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.failed(AsynchronousChannelStream.java:263) |
at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.failed(AsynchronousChannelStream.java:233) |
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:129) |
at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219) |
at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) |
... 3 common frames omitted |
Caused by: java.nio.channels.InterruptedByTimeoutException: null |
at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:465) |
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) |
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) |
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) |
... 3 common frames omitted |
Caused by: com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message |
at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.failed(AsynchronousChannelStream.java:263) |
at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.failed(AsynchronousChannelStream.java:233) |
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:129) |
at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219) |
at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) |
... 3 common frames omitted |
Caused by: java.nio.channels.InterruptedByTimeoutException: null |
at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:465) |
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) |
at …
|
Here is another error which is not retried, though I believe that it should be:
org.springframework.data.mongodb.UncategorizedMongoDbException: The server at 10-0-48-200.backend-mongos-production-order.backend-mongos.svc.cluster.local:27017 is no longer available; nested exception is com.mongodb.MongoServerUnavailableException: The server at 10-0-48-200.backend-mongos-production-order.backend-mongos.svc.cluster.local:27017 is no longer available |
at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:133) |
at org.springframework.data.mongodb.core.ReactiveMongoTemplate.potentiallyConvertRuntimeException(ReactiveMongoTemplate.java:2814) |
at org.springframework.data.mongodb.core.ReactiveMongoTemplate.lambda$translateException$90(ReactiveMongoTemplate.java:2797) |
at reactor.core.publisher.Flux.lambda$onErrorMap$28(Flux.java:6568) |
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88) |
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onError(MonoFlatMapMany.java:247) |
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onError(FluxConcatMap.java:256) |
at reactor.core.publisher.FluxCreate$BaseSink.error(FluxCreate.java:452) |
at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:781) |
at reactor.core.publisher.FluxCreate$BufferAsyncSink.error(FluxCreate.java:726) |
at reactor.core.publisher.FluxCreate$SerializedSink.drainLoop(FluxCreate.java:229) |
at reactor.core.publisher.FluxCreate$SerializedSink.drain(FluxCreate.java:205) |
at reactor.core.publisher.FluxCreate$SerializedSink.error(FluxCreate.java:181) |
at reactor.core.publisher.LambdaMonoSubscriber.doError(LambdaMonoSubscriber.java:152) |
at reactor.core.publisher.LambdaMonoSubscriber.onError(LambdaMonoSubscriber.java:147) |
at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:126) |
at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87) |
at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87) |
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:185) |
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:251) |
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:251) |
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:190) |
at com.mongodb.reactivestreams.client.internal.MongoOperationPublisher.lambda$sinkToCallback$31(MongoOperationPublisher.java:570) |
at com.mongodb.reactivestreams.client.internal.OperationExecutorImpl.lambda$execute$2(OperationExecutorImpl.java:92) |
at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) |
at com.mongodb.internal.async.function.AsyncCallbackSupplier.lambda$whenComplete$1(AsyncCallbackSupplier.java:95) |
at com.mongodb.internal.async.function.RetryingAsyncCallbackSupplier$RetryingCallback.onResult(RetryingAsyncCallbackSupplier.java:109) |
at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) |
at com.mongodb.internal.async.function.AsyncCallbackSupplier.lambda$whenComplete$1(AsyncCallbackSupplier.java:95) |
at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) |
at com.mongodb.internal.operation.OperationHelper.lambda$withAsyncSuppliedResource$7(OperationHelper.java:640) |
at com.mongodb.internal.connection.DefaultServer$1.onResult(DefaultServer.java:130) |
at com.mongodb.internal.connection.DefaultServer$1.onResult(DefaultServer.java:122) |
at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) |
at com.mongodb.internal.connection.DefaultConnectionPool.lambda$getAsync$0(DefaultConnectionPool.java:199) |
at com.mongodb.internal.connection.DefaultConnectionPool.lambda$getAsync$1(DefaultConnectionPool.java:216) |
at com.mongodb.internal.connection.DefaultConnectionPool$Task.doComplete(DefaultConnectionPool.java:1382) |
at com.mongodb.internal.connection.DefaultConnectionPool$Task.execute(DefaultConnectionPool.java:1368) |
at com.mongodb.internal.connection.DefaultConnectionPool$AsyncWorkManager.workerRun(DefaultConnectionPool.java:1309) |
at com.mongodb.internal.connection.DefaultConnectionPool$AsyncWorkManager.runAndLogUncaught(DefaultConnectionPool.java:1339) |
at com.mongodb.internal.connection.DefaultConnectionPool$AsyncWorkManager.lambda$initUnlessClosed$0(DefaultConnectionPool.java:1274) |
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) |
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) |
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) |
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) |
at java.base/java.lang.Thread.run(Thread.java:829) |
Caused by: com.mongodb.MongoServerUnavailableException: The server at 10-0-48-200.backend-mongos-production-order.backend-mongos.svc.cluster.local:27017 is no longer available |
at com.mongodb.internal.connection.ConcurrentPool.poolClosedException(ConcurrentPool.java:297) |
at com.mongodb.internal.connection.ConcurrentPool$StateAndPermits.throwIfClosedOrPaused(ConcurrentPool.java:495) |
at com.mongodb.internal.connection.ConcurrentPool$StateAndPermits.acquirePermit(ConcurrentPool.java:401) |
at com.mongodb.internal.connection.ConcurrentPool.get(ConcurrentPool.java:152) |
at com.mongodb.internal.connection.DefaultConnectionPool.getPooledConnection(DefaultConnectionPool.java:304) |
at com.mongodb.internal.connection.DefaultConnectionPool.lambda$getAsync$1(DefaultConnectionPool.java:214) |
... 10 common frames omitted |
Attachments
Issue Links
- is related to
-
JAVA-4974 try another srv-record on MongoSocketException
-
- Closed
-