Hey folks,

It looks like I found a violation of linearizability in MongoDB 3.6.4 - it returns stale data after a network partition is healed.

The docker scripts with repro steps: https://github.com/rystsov/consistency-mongodb
Logs: https://github.com/rystsov/consistency-mongodb/tree/master/experiments/may-17

Structure of the test:

• Four nodes in the system
• Three host MongoDB's replica set
• Same nodes host a thin Node.js application with talks with the DB and exposes http interface to read, create and update a key/value pair
• Fourth node is a client working with http interface of each node and detecting consistency violations

The client has three async/await coroutines to sequentially update value of its key using an increasing sequence. Each coroutine works with its key and node. Updates to key1 go to node1, key2 to node2 and key3 to node3. Also there are 9 reading coroutines: three coroutines per node to read "key1", "key2" and "key3" keys.

All reads were done with "linearizable" read concern and "majority" write concern.

It looks like the bug may be in the replication layer of the database or the MongoDB driver for Node.js (3.1.0-beta4).