[SERVER-37948] Linearizable read concern is not satisfied by getMores on a cursor Created: 06/Nov/18  Updated: 29/Oct/23  Resolved: 14/Feb/19

Status: Closed
Project: Core Server
Component/s: Querying, Replication
Affects Version/s: 4.1.4
Fix Version/s: 4.1.9

Type: Bug Priority: Major - P3
Reporter: William Schultz (Inactive) Assignee: Jason Chan
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-39364 Audit uses of setLastOpToSystemLastOp... Closed
is related to SERVER-72896 Linearizable readConcern timeout for ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

/*
 * Test linearizability of getMore on a cursor.
 */
(function() {
    'use strict';
 
    var num_nodes = 3;
    var name = 'linearizable_read_concern';
    var replTest = new ReplSetTest({name: name, nodes: num_nodes, useBridge: true});
    var config = replTest.getReplSetConfig();
 
    // Increased election timeout to avoid having the primary step down while we are
    // testing linearizable functionality on an isolated primary.
    config.settings = {electionTimeoutMillis: 20000};
 
    replTest.startSet();
    replTest.initiate(config);
    replTest.awaitReplication();
    var primary = replTest.getPrimary();
    var secondaries = replTest.getSecondaries();
 
    // Do a write on the original primary.
    assert.writeOK(
        primary.getDB("test").foo.insert({_id: 0, x: 0}, {"writeConcern": {"w": "majority"}}));
    primary = replTest.getPrimary();
 
    // Open a cursor on the original primary at 'linearizable' read concern.
    let result = primary.getDB("test").runCommand(
        {"find": "foo", filter: {_id: 0}, "readConcern": {level: "linearizable"}, batchSize: 0});
    assert.commandWorked(result);
    let cursorId = result.cursor.id;
 
    jsTestLog(
        "Setting up partitions such that the primary is isolated: [Secondary-Secondary] [Primary]");
    secondaries[0].disconnect(primary);
    secondaries[1].disconnect(primary);
 
    jsTestLog("Step up a new primary, and wait until it can accept writes.");
    assert.commandWorked(secondaries[0].adminCommand({replSetStepUp: 1}));
    assert.soonNoExcept(function() {
        return secondaries[0].adminCommand({isMaster: 1}).ismaster;
    });
    let newPrimary = secondaries[0];
    jsTestLog("New node " + newPrimary + " should now be primary");
 
    jsTestLog("Do a majority write to the new primary.");
    assert.writeOK(newPrimary.getDB("test").foo.update(
        {_id: 0}, {$set: {x: 1}}, {"writeConcern": {"w": "majority"}}));
 
    jsTestLog("Do a linearizable read on the new primary.");
    result = newPrimary.getDB("test").runCommand(
        {"find": "foo", filter: {_id: 0}, "readConcern": {level: "linearizable"}, batchSize: 1});
    assert.commandWorked(result);
    // A linearizable read should return the effects of the most recent majority committed write.
    assert.docEq([{_id: 0, x: 1}], result.cursor.firstBatch);
 
    jsTestLog("Do a linearizable getMore read on the old primary.");
    result =
        primary.getDB("test").runCommand({"getMore": cursorId, collection: "foo", batchSize: 1});
    assert.commandWorked(result);
    // A linearizable read should return the effects of the most recent majority committed write
    // (this should fail if getMores do not uphold linearizability guarantee correctly).
    assert.docEq([{_id: 0, x: 1}], result.cursor.nextBatch);
 
    // Re-connect and shut down.
    secondaries[0].reconnect(primary);
    secondaries[1].reconnect(primary);
    replTest.stopSet();
}());

Sprint: Repl 2019-01-14, Repl 2019-02-11, Repl 2019-02-25
Participants:

 Description   

When a cursor is opened with "linearizable" read concern, we guarantee that any data returned will reflect all successful majority-acknowledged writes that completed prior to the start of the read operation. If we do subsequent getMore operations on this cursor, however, we do not satisfy this linearizability guarantee.

I believe this bug is caused by the fact that when we wait for linearizable read concern here, we check the read concern arguments on the OperationContext of the running command. getMore commands, however, do not include a read concern directly. Their read concern is stored on the cursor object they are associated with. Since we only check the read concern from the OperationContext, we will presumably just return local read concern, bypassing the logic to satisfy the linearizability guarantee.



 Comments   
Comment by Githook User [ 14/Feb/19 ]

Author:

{'name': 'Jason Chan', 'email': 'jason.chan@10gen.com'}

Message: SERVER-37948 Satisfy linearizable read concern on getmore cursors.
Branch: master
https://github.com/mongodb/mongo/commit/22e1ef5134181dd9f58e8408e04744f205d7b41d

Comment by Githook User [ 14/Feb/19 ]

Author:

{'name': 'Louis Williams', 'email': 'louis.williams@mongodb.com', 'username': 'louiswilliams'}

Message: Revert "SERVER-37948 Satisfy linearizable read concern on getMores"

This reverts commit 0cdb86f3cf2bbabe448669598c32297f5ec8214f.
Branch: master
https://github.com/mongodb/mongo/commit/83336cb56b269195110253918d226cbba4377a03

Comment by Githook User [ 13/Feb/19 ]

Author:

{'name': 'Jason Chan', 'email': 'jason.chan@10gen.com'}

Message: SERVER-37948 Satisfy linearizable read concern on getMores
Branch: master
https://github.com/mongodb/mongo/commit/0cdb86f3cf2bbabe448669598c32297f5ec8214f

Comment by Charlie Swanson [ 03/Jan/19 ]

Just a little query perspective weighing in here - I think a getMore should be thought of as an implementation detail to get around the 16MB limit really. If you're doing a find, followed by something else, then followed by a getMore, you generally cannot expect to see that something else in your getMore. I don't think linearizable reads should be any exception. Requiring intervening writes to be visible in a subsequent getMore prevents the server from implementations which batch results or eagerly request more, both of which are things we sometimes do on mongos.

Comment by Gregory McKeon (Inactive) [ 12/Nov/18 ]

We don't think we can ban getMores on cursors with "linearizable" set or queries with batchSize of 0 because of scatter-gather queries. We could do the write on the getMore. We should fix this because sharded queries are broken with this.

Comment by Judah Schvimer [ 07/Nov/18 ]

I personally think that getMores should error if "linearizable" is set on the cursor. Linearizable cursors should probably also be required to contain a 'limit: 1', but that is already a documented requirement for safe linearizability.

Comment by Charlie Swanson [ 07/Nov/18 ]

Also note some code we have on mongos to set up the read preference on the OpCtx from the cursor object. Perhaps we should do this for read concern as well, and also on mongod? The motivation for the read preference was to make it so that a change stream's "updateLookup" operations from mongos will obey the read preference.

Comment by Max Hirschhorn [ 06/Nov/18 ]

Does Andy's argument for the getMore command not accepting a read concern in SERVER-31015 change whether this was a guarantee around linearizable that was intended to be promised?

getmore operations don't need read concern because they inherit the concern from the cursor. All we promised was that the cursor contents are from after the time sent on the find or aggregate.

This is normally fine because getmore is an implement detail, invisible below the driver. Maybe get more should reject rather than ignore read concern.

I'd welcome argument on why tailable cursors are special.

https://jira.mongodb.org/browse/SERVER-31015?focusedCommentId=1669939&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1669939

Generated at Thu Feb 08 04:47:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.