[JAVA-3973] Regression in reactive streams implementation memory consumption Created: 28/Jan/21  Updated: 28/Oct/23  Resolved: 11/Feb/21

Status: Closed
Project: Java Driver
Component/s: Reactive Streams, Scala
Affects Version/s: 4.2.0
Fix Version/s: 4.2.1

Type: Bug Priority: Major - P3
Reporter: Fred Roth Assignee: Ross Lawley
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

linux, OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.9.1+1, mixed mode), atlas


Attachments: PNG File image-2021-02-02-12-26-11-680.png     PNG File image-2021-02-02-12-27-01-888.png    
Issue Links:
Problem/Incident
causes JAVA-4044 Exception when dropping a collection ... Closed

 Description   

An upgrade from mongo-java-driver (and mongo-java-driver) from 4.1.1 to 4.2.0 lead to increased memory consumption and decreased performance. (Jobs taking massively longer than before)

We solved the issues by rolling back to 4.1.1.

We don't have minimal reproducing examples but it is very visible for usecases where we stream a huge amount of documents reading from the database using reactive streams.

The changelog states that you exchanged the internal reactive-streams implementation so maybe it has something to do with this.



 Comments   
Comment by Githook User [ 05/Apr/21 ]

Author:

{'name': 'Valentin Kovalenko', 'email': 'valentin.kovalenko@mongodb.com', 'username': 'stIncMale'}

Message: Backport to 4.2.x JAVA-4044/PR#689 and the changes it depends on (JAVA-3938 & JAVA-3907 / PR#661) (#691)

  • Regression test for change stream cancellation (#661)

Ensures that all sessions are returned to the pool

JAVA-3938 JAVA-3907

  • Guarantee that ChangeStreamPublisher for a collection completes after dropping the collection (#689)

Before the changes made within JAVA-3973,
ChangeStreamPublisher had been terminating with onError.
After the changes in JAVA-3973 neither onError nor onComplete is called,
but those changes allow us to terminated it with onComplete.
I could have specified only assertTerminalEvent()
without specifying assertNoErrors(), thus accepting either onComplete or onError
(the old behavior), but terminating with onComplete is nicer.

The approach with using startAtOperationTime to ensure that
a change stream is guaranteed to observe collection.drop()
works only if there is no leader re-election that results in
rolling back the delete operation from which the operationTime
was extracted. While such rollback can be prevented by using
the "majority" write concern, the common approach in driver tests
is to not use it for efficiency and tolerate a tiny chance of
experiencing a rollback.

JAVA-4044

Co-authored-by: Ross Lawley <ross.lawley@gmail.com>
Branch: 4.2.x
https://github.com/mongodb/mongo-java-driver/commit/77c8795eec482f909cfb658b46771b51c2ded7e0

Comment by Githook User [ 24/Mar/21 ]

Author:

{'name': 'Valentin Kovalenko', 'email': 'valentin.kovalenko@mongodb.com', 'username': 'stIncMale'}

Message: Guarantee that ChangeStreamPublisher for a collection completes after dropping the collection (#689)

Before the changes made within JAVA-3973,
ChangeStreamPublisher had been terminating with onError.
After the changes in JAVA-3973 neither onError nor onComplete is called,
but those changes allow us to terminated it with onComplete.
I could have specified only assertTerminalEvent()
without specifying assertNoErrors(), thus accepting either onComplete or onError
(the old behavior), but terminating with onComplete is nicer.

The approach with using startAtOperationTime to ensure that
a change stream is guaranteed to observe collection.drop()
works only if there is no leader re-election that results in
rolling back the delete operation from which the operationTime
was extracted. While such rollback can be prevented by using
the "majority" write concern, the common approach in driver tests
is to not use it for efficiency and tolerate a tiny chance of
experiencing a rollback.

JAVA-4044
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/b3698be5d218af6c3ee94aca2a5cccbf7a853e3a

Comment by Jeffrey Yemin [ 16/Feb/21 ]

joao@nezasa.com by the end of this week.

Comment by João Ferreira [ 11/Feb/21 ]

Also fixes our problem. Any ETA for 4.2.1?

Comment by Ross Lawley [ 11/Feb/21 ]

Thanks frederick.roth@breuninger.de for testing.

Comment by Fred Roth [ 11/Feb/21 ]

Hi Ross,

I just ran our workload with 4.1.1, 4.2.0, and 4.2.1-SNAPSHOT. The performance and memory footprint of the new 4.2.1-SNAPSHOT version is very similar to 4.1.1. And therefore way better than 4.2.0.

So from my point of view it seems to be fixed.

Thanks for your effort

Fred

Comment by Ross Lawley [ 11/Feb/21 ]

Hi frederick.roth@breuninger.de & jtjeferreira@gmail.com,

Just to let you know that the 4.2.1-SNAPSHOT has been released with an improvement in the BatchCursorPublisher.

If possible it would be great if you could test this fix and provide feedback before the release.

Ross

Comment by Githook User [ 03/Feb/21 ]

Author:

{'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}

Message: Fix reactive streams batch cursor regression. (#652)

The BatchCursorPublisher now waits for demand to be signalled before
creating the cursor.

The batch cursor publisher also internally respects the batch size.
Its either the configured batch size if set or the amount requested.

JAVA-3973
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/4af9e9f2a541b8a89eadd7f91124b324015866eb

Comment by Ross Lawley [ 02/Feb/21 ]

HI jtjeferreira@gmail.com,

That's great, internally we still use the BufferAsyncSink however, we have identified a change in how demand is signalled and processed, which can lead to excessive buffering.

Ross

Comment by João Ferreira [ 02/Feb/21 ]

Hi

We also see something similar when upgrading to 4.2. Rolling back to 4.1.1 seems to fix it. I dont have a minimal reproducer, but I have a an heapdump which I can share some screenshots:

Overview

List object with outgoing references

Comment by Ross Lawley [ 29/Jan/21 ]

PR: https://github.com/mongodb/mongo-java-driver/pull/652/

Comment by Ross Lawley [ 28/Jan/21 ]

Hi frederick.roth@breuninger.de,

No problems, I inferred that you were using Scala but as the main change has been the introduction of Project Reactor I'll base my investigations there to begin with.

Ross

Comment by Fred Roth [ 28/Jan/21 ]

My colleague spotted a typo in my description: I wanted to write "mongo-java-driver (mongo-scala-driver)" meaning: we use the scala driver using the underlying reactive streams java driver.

Comment by Ross Lawley [ 28/Jan/21 ]

Firstly, thanks for the report frederick.roth@breuninger.de and bring this to our attention. Secondly, apologies for causing a performance regression.

We'll investigate and will report back.

Ross

Generated at Thu Feb 08 09:00:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.