[SERVER-45002] Addition of "comment" field to getMore breaks agg commands with comment during upgrade of sharded cluster Created: 06/Dec/19  Updated: 29/Oct/23  Resolved: 27/Jan/20

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, Sharding
Affects Version/s: None
Fix Version/s: 4.3.4

Type: Bug Priority: Major - P3
Reporter: David Storch Assignee: David Storch
Resolution: Fixed Votes: 0
Labels: qexec-team
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-45762 Take collection lock before calling C... Closed
Related
related to SERVER-45579 Remove FCV check guarding addition of... Closed
is related to SERVER-29794 Adding a comment to all commands Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

The following script reproduces the issue:

// Show that the "comment" to the aggregate command can result in spurious failed queries in a
// sharded cluster during the 4.2 = > 4.4 upgrade.
(function() {
"use strict";
 
load("jstests/multiVersion/libs/multi_rs.js");  // upgradeSet
 
// Start with a 4.2 cluster with two shards.
const st = new ShardingTest({
    shards: 2,
    mongos: 1,
    rs: {nodes: 3},
    other: {
        mongosOptions: {binVersion: "4.2"},
        configOptions: {binVersion: "4.2"},
        rsOptions: {binVersion: "4.2"},
    }
});
 
jsTest.log("upgrading shard rs1 to 4.4");
st.rs1.upgradeSet({binVersion: "latest"});
 
const testDb = st.s.getDB("testDb");
assert.commandWorked(testDb.source.insert({_id: -1}));
assert.commandWorked(testDb.source.insert({_id: 1}));
 
// Shard a collection and ensure that there are chunks on both shards.
st.ensurePrimaryShard("testDb", st.shard1.shardName);
st.shardColl("source", {_id: 1}, {_id: 0}, {_id: -1}, "testDb", true);
 
testDb.printShardingStatus();
 
// Run a $merge which will use the 4.4 node as the merger, and add a comment. This should succeed,
// but it will fail when the 4.4 shard sends a getMore containing the comment field to the 4.2
// shard.
assert.eq(
    0,
    testDb.source
        .aggregate(
            [
                {$_internalInhibitOptimization: {}},
                {$merge: {into: "destination", whenMatched: "replace", whenNotMatched: "insert"}}
            ],
            {comment: "my comment"})
        .itcount());
assert.eq(2, testDb.destination.find().itcount());
 
st.stop();
})();

Sprint: Query 2019-12-30, Query 2020-01-13, Query 2020-01-27, Query 2020-02-10
Participants:
Linked BF Score: 49

 Description   

In SERVER-29794, we added comment support to many commands, including getMore. When a user runs an aggregate command with a comment, this comment will be added to any getMore commands that the system has to run between nodes in sharded cluster in order to answer the query. This includes both mongos running a getMore against a cursor on a shard, and a shard running a getMore against a cursor on another shard. The latter can happen if a shardsvr node is chosen to execute the merging part of the pipeline.

Suppose an agg command is sent to a cluster with both 4.4 and 4.2 shards during the 4.2 <=> 4.4 upgrade/downgrade. If a 4.4 node needs to send a getMore to a 4.2 node during execution of the query, it will append the comment. However, the changes for SERVER-29794 are not present in the 4.2 branch, and they are not planned for backport. Consequently, the 4.2 node will throw a parsing error, complaining that the comment parameter to the getMore command is unrecognized.

In order to fix this, we need to make sure that a 4.4 mongod does not append the comment to any getMore commands until the upgrade to 4.4 is complete and the featureCompatibilityVersion is fully upgraded to "4.4". Since the shards must be upgraded before mongos, no changes are necessary to the logic that dispatches getMores from mongos to the shards.



 Comments   
Comment by Githook User [ 27/Jan/20 ]

Author:

{'username': 'dstorch', 'name': 'David Storch', 'email': 'david.storch@10gen.com'}

Message: SERVER-45002 Only propagate 'comment' field between nodes when upgrade to 4.4 is complete.

create mode 100644 jstests/multiVersion/agg_with_comment_during_upgrade.js
Branch: master
https://github.com/mongodb/mongo/commit/666877a5da9a6b4c532df6c0c087bcf45123eed0

Comment by Kaloian Manassiev [ 26/Jan/20 ]

I believe the dependent SERVER-45762 has been fixed, just FYI.

Comment by David Storch [ 24/Jan/20 ]

This is currently blocked. An early version of this patch had a C++ global initialization-order fiasco bug, so I need to get a green build on the ASAN builder before merging. However, db_s_collection_sharding_runtime_test appears to be still broken in master: see https://evergreen.mongodb.com/task/mongodb_mongo_master_ubuntu1804_debug_aubsan_lite_unittests_patch_3c77b4ab9e633b3e36688ef1866c29380d9267eb_5e2b534357e85a6fffa2edd1_20_01_24_20_29_37##%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%25223c77b4ab9e633b3e36688ef1866c29380d9267eb%2522%257D%255D%257D. I'll need to re-run a patch build once the ASAN builder gets fixed.

Generated at Thu Feb 08 05:07:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.