Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-45002

Addition of "comment" field to getMore breaks agg commands with comment during upgrade of sharded cluster

XMLWordPrintableJSON

    • Fully Compatible
    • ALL
    • Hide

      The following script reproduces the issue:

      // Show that the "comment" to the aggregate command can result in spurious failed queries in a
// sharded cluster during the 4.2 = > 4.4 upgrade.
(function() {
"use strict";

load("jstests/multiVersion/libs/multi_rs.js");  // upgradeSet

// Start with a 4.2 cluster with two shards.
const st = new ShardingTest({
    shards: 2,
    mongos: 1,
    rs: {nodes: 3},
    other: {
        mongosOptions: {binVersion: "4.2"},
        configOptions: {binVersion: "4.2"},
        rsOptions: {binVersion: "4.2"},
    }
});

jsTest.log("upgrading shard rs1 to 4.4");
st.rs1.upgradeSet({binVersion: "latest"});

const testDb = st.s.getDB("testDb");
assert.commandWorked(testDb.source.insert({_id: -1}));
assert.commandWorked(testDb.source.insert({_id: 1}));

// Shard a collection and ensure that there are chunks on both shards.
st.ensurePrimaryShard("testDb", st.shard1.shardName);
st.shardColl("source", {_id: 1}, {_id: 0}, {_id: -1}, "testDb", true);

testDb.printShardingStatus();

// Run a $merge which will use the 4.4 node as the merger, and add a comment. This should succeed,
// but it will fail when the 4.4 shard sends a getMore containing the comment field to the 4.2
// shard.
assert.eq(
    0,
    testDb.source
        .aggregate(
            [
                {$_internalInhibitOptimization: {}},
                {$merge: {into: "destination", whenMatched: "replace", whenNotMatched: "insert"}}
            ],
            {comment: "my comment"})
        .itcount());
assert.eq(2, testDb.destination.find().itcount());

st.stop();
})();
      Show
      The following script reproduces the issue: // Show that the "comment" to the aggregate command can result in spurious failed queries in a // sharded cluster during the 4.2 = > 4.4 upgrade. (function() { "use strict" ; load( "jstests/multiVersion/libs/multi_rs.js" ); // upgradeSet // Start with a 4.2 cluster with two shards. const st = new ShardingTest({ shards: 2, mongos: 1, rs: {nodes: 3}, other: { mongosOptions: {binVersion: "4.2" }, configOptions: {binVersion: "4.2" }, rsOptions: {binVersion: "4.2" }, } }); jsTest.log( "upgrading shard rs1 to 4.4" ); st.rs1.upgradeSet({binVersion: "latest" }); const testDb = st.s.getDB( "testDb" ); assert .commandWorked(testDb.source.insert({_id: -1})); assert .commandWorked(testDb.source.insert({_id: 1})); // Shard a collection and ensure that there are chunks on both shards. st.ensurePrimaryShard( "testDb" , st.shard1.shardName); st.shardColl( "source" , {_id: 1}, {_id: 0}, {_id: -1}, "testDb" , true ); testDb.printShardingStatus(); // Run a $merge which will use the 4.4 node as the merger, and add a comment. This should succeed, // but it will fail when the 4.4 shard sends a getMore containing the comment field to the 4.2 // shard. assert .eq( 0, testDb.source .aggregate( [ {$_internalInhibitOptimization: {}}, {$merge: {into: "destination" , whenMatched: "replace" , whenNotMatched: "insert" }} ], {comment: "my comment" }) .itcount()); assert .eq(2, testDb.destination.find().itcount()); st.stop(); })();
    • Query 2019-12-30, Query 2020-01-13, Query 2020-01-27, Query 2020-02-10
    • 49

      In SERVER-29794, we added comment support to many commands, including getMore. When a user runs an aggregate command with a comment, this comment will be added to any getMore commands that the system has to run between nodes in sharded cluster in order to answer the query. This includes both mongos running a getMore against a cursor on a shard, and a shard running a getMore against a cursor on another shard. The latter can happen if a shardsvr node is chosen to execute the merging part of the pipeline.

      Suppose an agg command is sent to a cluster with both 4.4 and 4.2 shards during the 4.2 <=> 4.4 upgrade/downgrade. If a 4.4 node needs to send a getMore to a 4.2 node during execution of the query, it will append the comment. However, the changes for SERVER-29794 are not present in the 4.2 branch, and they are not planned for backport. Consequently, the 4.2 node will throw a parsing error, complaining that the comment parameter to the getMore command is unrecognized.

      In order to fix this, we need to make sure that a 4.4 mongod does not append the comment to any getMore commands until the upgrade to 4.4 is complete and the featureCompatibilityVersion is fully upgraded to "4.4". Since the shards must be upgraded before mongos, no changes are necessary to the logic that dispatches getMores from mongos to the shards.

            Assignee:
            david.storch@mongodb.com David Storch
            Reporter:
            david.storch@mongodb.com David Storch
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: