[JAVA-4075] DocumentationSamples.testWatch may cause failures of other tests Created: 17/Mar/21  Updated: 28/Oct/23  Resolved: 18/Mar/21

Status: Closed
Project: Java Driver
Component/s: Test Failure
Affects Version/s: 4.3.0
Fix Version/s: 4.2.3

Type: Improvement Priority: Major - P3
Reporter: Valentin Kavalenka Assignee: Valentin Kavalenka
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Epic Link: Investigate flaky tests
Backwards Compatibility: Fully Compatible
Documentation Changes: Not Needed

 Description   

The test documentation.DocumentationSamples.testQueryingArrayValues failed in Evergreen for the build variant tests-jdk8-unsecure__version~4.4_os~linux_topology~replicaset_auth~noauth_ssl~nossl_jdk~jdk8: https://spruce.mongodb.com/task/mongo_java_driver_tests_jdk8_unsecure__version~4.4_os~linux_topology~replicaset_auth~noauth_ssl~nossl_jdk~jdk8_test_patch_d57dfa8af686180239058a06eca838e18bfe5cdd_605277f93e8e863987251298_21_03_17_21_43_22/tests?execution=0&page=0&sortBy=STATUS&sortDir=ASC with the following assertion error

FAILURE: java.lang.AssertionError: expected:<5> but was:<7> (java.lang.AssertionError)
java.lang.AssertionError: expected:<5> but was:<7>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at documentation.DocumentationSamples.testQueryingArrayValues(DocumentationSamples.java:277)

This test is for the collection.insertMany code sample on the page https://docs.mongodb.com/manual/tutorial/query-arrays/.

The error happened despite documentation.DocumentationSamples doing collection.drop() synchronously in the @After tearDown method. The test class does the following:

...//some modifications to the collection
collection.drop();//in @After tearDown
collection.insertMany(asList(
                Document.parse("{ item: 'journal', qty: 25, tags: ['blank', 'red'], dim_cm: [ 14, 21 ] }"),
                Document.parse("{ item: 'notebook', qty: 50, tags: ['red', 'blank'], dim_cm: [ 14, 21 ] }"),
                Document.parse("{ item: 'paper', qty: 100, tags: ['red', 'blank', 'plain'], dim_cm: [ 14, 21 ] }"),
                Document.parse("{ item: 'planner', qty: 75, tags: ['blank', 'red'], dim_cm: [ 22.85, 30 ] }"),
                Document.parse("{ item: 'postcard', qty: 45, tags: ['blue'], dim_cm: [ 10, 15.25 ] }")
        ));
        //End Example 20
 
assertEquals(5, collection.countDocuments());

The assertion relies on the following:

  1. drop is ordered before insertMany (monotonic writes)
  2. countDocuments observes insertMany (read your writes).

According to https://docs.mongodb.com/manual/core/causal-consistency-read-write-concerns/, RYW is guaranteed only if the majority write and read concerns are used. Thus, the test must use them. However, the lack of RYW in the test cannot explain observing 7 elements in the collection instead of 5, which violates the monotonic writes consistency. The problem here is that "MongoDB provides monotonic write guarantees, by default, for standalone mongod instances and replica set.", and the test failed for a replica set. I do not know how to explain the failure, but I think it is worth thinking about and discussing.



 Comments   
Comment by Githook User [ 19/Mar/21 ]

Author:

{'name': 'Valentin Kovalenko', 'email': 'valentin.kovalenko@mongodb.com', 'username': 'stIncMale'}

Message: Join the thread in DocumentationSamples.testWatch (#686)

JAVA-4075
Branch: 4.2.x
https://github.com/mongodb/mongo-java-driver/commit/1471dd258cf4ca65ebe298c36979cf910ca2478f

Comment by Githook User [ 18/Mar/21 ]

Author:

{'name': 'Valentin Kovalenko', 'email': 'valentin.kovalenko@mongodb.com', 'username': 'stIncMale'}

Message: Join the thread in DocumentationSamples.testWatch (#686)

JAVA-4075
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/b1de7e57cf1dfd12fb80d215f36fecd1c4c5ae8a

Comment by Valentin Kavalenka [ 18/Mar/21 ]

The explanation of the behavior that I interpreted as a monotonic writes consistency violation turned out to be caused by a thread started in a different test (thanks jeff_y for spotting this!): https://github.com/stIncMale/mongo-java-driver/blob/master/driver-sync/src/examples/documentation/DocumentationSamples.java#L715

All we need to fix the issue is to make sure that the death of that thread happens before the return from the method testWatch in which the thread is started.

As for the RYW - we have many tests that do not use the majority read/write concerns and yet rely on RYW. This is done for the following reasons:

  1. having cluster nodes on the same physical machine communicating over the loopback interface reduces the chances of a client observing RYW being violated; we can tolerate this possibility as a tradeoff for the next reason
  2. we want the running time of our test suit to be reasonably small, and not using majority write/read concern definitely helps reducing the time.
Generated at Thu Feb 08 09:01:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.