[JAVA-3852] InternalStreamConnection.openAsync leaks exception instead of passing it to callback Created: 07/Oct/20  Updated: 28/Oct/23  Resolved: 27/Nov/20

Status: Closed
Project: Java Driver
Component/s: Error Handling, Reactive Streams
Affects Version/s: 3.12.7
Fix Version/s: 4.2.0, 3.12.8

Type: Bug Priority: Major - P3
Reporter: Sahil Anand Assignee: Jeffrey Yemin
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File stackTrace    

 Description   

There seems to be a bug in  

 

com.mongodb.internal.connection.InternalStreamConnection.openAsync

 

// code placeholder
public void openAsync(final SingleResultCallback<Void> callback) {
    ..
    try {
        ..
    } catch (Throwable t) {
       ...
    }
    stream.openAsync(new AsyncCompletionHandler<Void>() {
      ..........
    });
}

 

stream.openAsync is not error handled properly and when this blows up there is no callback. This is causing unexpected error on our side which cannot be handled.

The error usually is 
com.mongodb.MongoSocketException: ......: System error
 

I have attached a stack trace I could provide. 



 Comments   
Comment by Yann Simon [ 13/Jan/21 ]

It'd be very kind if you could release the 3.12.8 version.

Thanks in advance.

Comment by Yann Simon [ 17/Dec/20 ]

For circuit breaker, I have the idea of continuously checking the state of the mongo servers (for example by fetching the availability of concurrency tickets) and opening the circuit breaker if the server is having issues. I just don't know if fetching the metrics of the server would put too much pressure on it. Something to check at some point...

Comment by Yann Simon [ 17/Dec/20 ]

Thanks a lot for the pointer to https://jira.mongodb.org/browse/JAVA-3909. This would be very helpful!

We are using mongo 4.0.21 and are planing to update.

Comment by Jeffrey Yemin [ 14/Dec/20 ]

yann.simon.fr@gmail.com

FYI we've scheduled JAVA-3909 which hopefully will help with explain. We don't plan to backport it to the 3.x branch so you will have to change some code as part of the upgrade. What version of MongoDB are you on?

Comment by Jeffrey Yemin [ 12/Dec/20 ]

Let me know if you need ideas on how to replace each of those in your application with something equivalent.

Comment by Yann Simon [ 11/Dec/20 ]

@jeff.yemin, a migration to the 4.x driver is something quite challenging for us. We are actually looking at it.

These are the deprecations that we are using right now:

If you want more info, I'd be glad to bring it.

Comment by Jeffrey Yemin [ 03/Dec/20 ]

Backported to the 3.12.x branch.

Is anything in particular preventing you from upgrading to the 4.x driver?

Comment by Jeffrey Yemin [ 03/Dec/20 ]

Author:

{'name': 'jyemin', 'email': 'jeff.yemin@mongodb.com', 'username': 'jyemin'}

Message: Ensure DNS errors bubble up as errors in reactive streams

Currently DNS lookups happen synchronously in the reactive streams
driver. This isn't good, but with this change at least DNS errors
will be properly reported.

JAVA-3852
Branch: 3.12.x
https://github.com/mongodb/mongo-java-driver/commit/0e556f1db850d47bd890d144f3a48064af554b5c

Comment by Yann Simon [ 03/Dec/20 ]

I see that the fix made to the 4.2.0 version.

As initially reported, we are using the version 3.12.7. We would also need a 3.x release to be able to use the fix.

Comment by Githook User [ 27/Nov/20 ]

Author:

{'name': 'jyemin', 'email': 'jeff.yemin@mongodb.com', 'username': 'jyemin'}

Message: Ensure DNS errors bubble up as errors in reactive streams

Currently DNS lookups happen synchronously in the reactive streams
driver. This isn't good, but with this change at least DNS errors
will be properly reported.

JAVA-3852
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/0eac1f09b9006899b2aed677dbcfdfe0ce94ab45

Comment by Jeffrey Yemin [ 07/Oct/20 ]

yann.simon.fr@gmail.com that's great to know about. Thanks for pointing it out.

Comment by Yann Simon [ 07/Oct/20 ]

For info, a possible extension would be to use Netty for asynchronous DNS resolution: https://netty.io/4.1/api/io/netty/resolver/dns/package-summary.html

I don't know how easy it would be.

Comment by Sahil Anand [ 07/Oct/20 ]

We can have a look how can we handle this at our end, until then.

Thank you

Comment by Jeffrey Yemin [ 07/Oct/20 ]

Yes, I see it now. All the DNS lookups are synchronous, which is unfortunate for async, but as a result also need to be protected.

In the meantime, can you work around this by fixing your DNS configuration?

Comment by Yann Simon [ 07/Oct/20 ]

Hi, colleague from Sahil here.

 

In the version 3.12.7, we can still see the following code:

@Override
public void openAsync(final SingleResultCallback<Void> callback) {
    isTrue("Open already called", stream == null, callback);
    try {
        stream = streamFactory.create(serverId.getAddress());
    } catch (Throwable t) {
        callback.onResult(null, t);
        return;
    }
    stream.openAsync(new AsyncCompletionHandler<Void>() {
[...]
    });
} 

Whenever an exception occurs in stream.openAsync, the callback is never called.

Ex of such an issue:

com.mongodb.MongoSocketException: <server DNS>: System error
	at com.mongodb.ServerAddress.getSocketAddresses(ServerAddress.java:211)
	at com.mongodb.connection.netty.NettyStream.openAsync(NettyStream.java:109)
	at com.mongodb.internal.connection.InternalStreamConnection.openAsync(InternalStreamConnection.java:152)
 

I think the simplest fix is to include stream.openAsync into the existing try.

Comment by Sahil Anand [ 07/Oct/20 ]

Hi Jeffery, 

I uploaded the stacktrac but I can copy it here again.

com.mongodb.MongoSocketException: <db-url> :System error
at com.mongodb.ServerAddress.getSocketAddresses(ServerAddress.java:211)
	at com.mongodb.connection.netty.NettyStream.openAsync(NettyStream.java:109)
	at com.mongodb.internal.connection.InternalStreamConnection.openAsync(InternalStreamConnection.java:152)
	at com.mongodb.internal.connection.UsageTrackingInternalConnection.openAsync(UsageTrackingInternalConnection.java:57)
	at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.openAsync(DefaultConnectionPool.java:404)
	at com.mongodb.internal.connection.DefaultConnectionPool.openAsync(DefaultConnectionPool.java:207)
	at com.mongodb.internal.connection.DefaultConnectionPool.getAsync(DefaultConnectionPool.java:158)
	at com.mongodb.internal.connection.DefaultServer.getConnectionAsync(DefaultServer.java:105)
	at com.mongodb.binding.AsyncClusterBinding$AsyncClusterBindingConnectionSource.getConnection(AsyncClusterBinding.java:146)
	at com.mongodb.async.client.ClientSessionBinding$SessionBindingAsyncConnectionSource.getConnection(ClientSessionBinding.java:140)
	at com.mongodb.operation.OperationHelper.withAsyncConnectionSource(OperationHelper.java:611)
	at com.mongodb.operation.OperationHelper.access$200(OperationHelper.java:63)
	at com.mongodb.operation.OperationHelper$AsyncCallableWithConnectionAndSourceCallback.onResult(OperationHelper.java:631)
	at com.mongodb.operation.OperationHelper$AsyncCallableWithConnectionAndSourceCallback.onResult(OperationHelper.java:619)
	at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:49)
	at com.mongodb.async.client.ClientSessionBinding$WrappingCallback.onResult(ClientSessionBinding.java:208)
	at com.mongodb.async.client.ClientSessionBinding$WrappingCallback.onResult(ClientSessionBinding.java:196)
	at com.mongodb.binding.AsyncClusterBinding$1.onResult(AsyncClusterBinding.java:120)
	at com.mongodb.binding.AsyncClusterBinding$1.onResult(AsyncClusterBinding.java:114)
	at com.mongodb.internal.connection.BaseCluster$ServerSelectionRequest.onResult(BaseCluster.java:440)
	at com.mongodb.internal.connection.BaseCluster.handleServerSelectionRequest(BaseCluster.java:304)
	at com.mongodb.internal.connection.BaseCluster.selectServerAsync(BaseCluster.java:160)
	at com.mongodb.internal.connection.AbstractMultiServerCluster.selectServerAsync(AbstractMultiServerCluster.java:54)
	at com.mongodb.binding.AsyncClusterBinding.getAsyncClusterBindingConnectionSource(AsyncClusterBinding.java:114)
	at com.mongodb.binding.AsyncClusterBinding.getReadConnectionSource(AsyncClusterBinding.java:99)
	at com.mongodb.async.client.ClientSessionBinding.getReadConnectionSource(ClientSessionBinding.java:58)
	at com.mongodb.operation.OperationHelper.withAsyncReadConnection(OperationHelper.java:558)
	at com.mongodb.operation.FindOperation.executeAsync(FindOperation.java:764)
	at com.mongodb.async.client.OperationExecutorImpl$1$1.onResult(OperationExecutorImpl.java:88)
	at com.mongodb.async.client.OperationExecutorImpl$1$1.onResult(OperationExecutorImpl.java:76)
	at com.mongodb.async.client.OperationExecutorImpl.getReadWriteBinding(OperationExecutorImpl.java:179)
	at com.mongodb.async.client.OperationExecutorImpl.access$200(OperationExecutorImpl.java:45)
	at com.mongodb.async.client.OperationExecutorImpl$1.onResult(OperationExecutorImpl.java:74)
	at com.mongodb.async.client.OperationExecutorImpl$1.onResult(OperationExecutorImpl.java:68)
	at com.mongodb.async.client.ClientSessionHelper.createClientSession(ClientSessionHelper.java:63)
	at com.mongodb.async.client.ClientSessionHelper.withClientSession(ClientSessionHelper.java:49)
	at com.mongodb.async.client.OperationExecutorImpl.execute(OperationExecutorImpl.java:68)
	at com.mongodb.async.client.MongoIterableImpl.batchCursor(MongoIterableImpl.java:167)
	at com.mongodb.async.client.MongoIterableSubscription.requestInitialData(MongoIterableSubscription.java:46)
	at com.mongodb.async.client.AbstractSubscription.tryRequestInitialData(AbstractSubscription.java:167)
	at com.mongodb.async.client.AbstractSubscription.request(AbstractSubscription.java:90)
	at org.mongodb.scala.ObservableImplicits$BoxedSubscription.request(ObservableImplicits.scala:494)
	at org.mongodb.scala.internal.FoldLeftObservable$$anon$1$$anon$2.request(FoldLeftObservable.scala:42)
	at org.mongodb.scala.ObservableImplicits$ScalaObservable$$anon$2.onSubscribe(ObservableImplicits.scala:367)
	at org.mongodb.scala.ObservableImplicits$ToSingleObservable$$anon$3.onSubscribe(ObservableImplicits.scala:445)
	at org.mongodb.scala.internal.SubscriptionCheckingObserver.onSubscribe(SubscriptionCheckingObserver.scala:28)
	at org.mongodb.scala.internal.MapObservable$$anon$1.onSubscribe(MapObservable.scala:28)
	at org.mongodb.scala.internal.SubscriptionCheckingObserver.onSubscribe(SubscriptionCheckingObserver.scala:28)
	at org.mongodb.scala.internal.FoldLeftObservable$$anon$1.onSubscribe(FoldLeftObservable.scala:47)
	at org.mongodb.scala.internal.SubscriptionCheckingObserver.onSubscribe(SubscriptionCheckingObserver.scala:28)
	at org.mongodb.scala.Observer.onSubscribe(Observer.scala:85)
	at org.mongodb.scala.Observer.onSubscribe$(Observer.scala:85)
	at org.mongodb.scala.internal.SubscriptionCheckingObserver.onSubscribe(SubscriptionCheckingObserver.scala:21)
	at com.mongodb.async.client.MongoIterableSubscription.<init>(MongoIterableSubscription.java:40)
	at com.mongodb.async.client.Observables$1.subscribe(Observables.java:47)
	at org.mongodb.scala.ObservableImplicits$BoxedObservable.subscribe(ObservableImplicits.scala:478)
	at org.mongodb.scala.FindObservable.subscribe(FindObservable.scala:344)
	at org.mongodb.scala.internal.FoldLeftObservable.subscribe(FoldLeftObservable.scala:27)
	at org.mongodb.scala.internal.MapObservable.subscribe(MapObservable.scala:25)
	at org.mongodb.scala.ObservableImplicits$ToSingleObservable.subscribe(ObservableImplicits.scala:437)
	at org.mongodb.scala.ObservableImplicits$ScalaObservable.headOption(ObservableImplicits.scala:359)
	at org.mongodb.scala.ObservableImplicits$ScalaObservable.head(ObservableImplicits.scala:345)
	at org.mongodb.scala.ObservableImplicits$ScalaObservable.toFuture(ObservableImplicits.scala:337)

 

 

 

 

 

 

 

Comment by Jeffrey Yemin [ 07/Oct/20 ]

Hi sahilanand1025@gmail.com

Thanks for reporting the issue. I thought it might be a duplicate of JAVA-3680, but that was fixed in 3.12.3.

You mentioned an attached stack trace, but I don't see one. Can you provide anything more?

Comment by Sahil Anand [ 07/Oct/20 ]

Sorry for the wrong version, what we are using is the driver version is 3.12.7.

Generated at Thu Feb 08 09:00:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.