[SERVER-44805] [v3.6] Resuming a change stream can fail with "resume token not found" if a shard's collection sharding state is stale Created: 22/Nov/19  Updated: 27/Oct/23  Resolved: 21/Jul/22

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Nicholas Zolnierz Assignee: Backlog - Query Optimization
Resolution: Gone away Votes: 0
Labels: qopt-team
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Assigned Teams:
Query Optimization
Operating System: ALL
Steps To Reproduce:

(function() {
    "use strict";
 
    var st = new ShardingTest({
        shards: 2,
        rs: {nodes: 2, setParameter: {periodicNoopIntervalSecs: 1, writePeriodicNoops: true}}
    });
 
    var testDB = st.s.getDB('test');
    testDB.dropDatabase();
 
    assert.commandWorked(testDB.adminCommand({enableSharding: 'test'}));
    st.ensurePrimaryShard('test', st.shard0.shardName);
    st.shardColl(testDB.foo, {x: 1}, {x: 10}, {x: 10});
 
    let stream = testDB.foo.aggregate([{$changeStream: {}}], {$readPreference: {mode: "secondary"}});
    assert.writeOK(testDB.foo.insert({x: 0}, {writeConcern: {w: "majority"}}));
 
    // This will trigger a refresh on the secondaries and they will now be aware that the collection
    // is sharded.
    assert.commandWorked(testDB.foo.runCommand({
        count: "foo",
        query: {},
        $readPreference: {mode: "secondary"},
        readConcern: {"level": "local"}
    }));
 
    assert.soon(()=> stream.hasNext());
 
    let token = stream.next()._id;
 
    // Restart the secondary, resetting it's sharding state to be unsharded.
    st.rs0.restart(st.rs0.getSecondary());
 
    let resumeStream = testDB.foo.aggregate([{$changeStream: {resumeAfter: token}}], 
                             {$readPreference: {mode: "secondary"}});
 
    assert.writeOK(testDB.foo.insert({x: 1}));    // THIS WILL FAIL TO RESUME
    assert.soon(()=> resumeStream.hasNext());
 
    st.stop();
}());

Participants:
Case:

 Description   

When generating a resume token (either to return to the client or to compare against a given token), the change stream stage will consult the CollectionShardingState to determine the document key fields to extract. In most cases, this is perfectly fine and the document key will either be '_id' for an unsharded collection or the shard key + '_id' for sharded collections. However, this assumption breaks down for any of the following reasons:

1. The node has not received any versioned commands, this is especially obvious for secondaries.
2. A node is restarted, clearing its internal sharding state (this is very common in atlas maintenance, for instance). The node now assumes that the collection is unsharded. This is tracked by SERVER-32198.
3. The collection becomes sharded in between the time that the token is generated and the resume is attempted. The document key in the token will only have _id but the resume logic will see the shard key and incorrectly fail.

For the second case above, there are (at least) two possibilities for resume failures if the collection is sharded. Either 1) we're attempting to resume against a node which incorrectly thinks the collection is unsharded or 2) we're attempting to resume on a fresh node but the token was generated from a node with stale metadata. In both situations we will assert since the document key fields do not match.



 Comments   
Comment by James Wahlin [ 21/Jul/22 ]

Closing as the versions of MongoDB exposed to this issue having fallen out of support.

Comment by Nicholas Zolnierz [ 22/Nov/19 ]

Note that this only affects v3.6 and has been fixed somewhat indirectly by SERVER-34705 and SERVER-34090 in 4.0+. Filing this ticket for tracking purposes as it doesn't make sense to backport all of SERVER-34705. I'm also thinking that this will ticket will help support to track down issues in case we see this more often in the field.

CC david.storch bernard.gorman charlie.swanson@mongodb.com

Generated at Thu Feb 08 05:07:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.