[SERVER-53433] Map reduce is versioned on direct connection to shards Created: 18/Dec/20  Updated: 29/Oct/23  Resolved: 13/Apr/21

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 4.4.2
Fix Version/s: 4.4.7, 5.0.0-rc0

Type: Bug Priority: Major - P3
Reporter: Marcos José Grillo Ramirez Assignee: Eric Cox (Inactive)
Resolution: Fixed Votes: 0
Labels: qexec-team
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
related to SERVER-59924 Error executing aggregate with $out w... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Steps To Reproduce:

1. Set up a replica set
2. Add some data
3. Restart the replica set with --shardsvr
4. Add the replica set to a sharded cluster
5. Run a mapreduce with output on the primary directly

Participants:
Linked BF Score: 24

 Description   

After SERVER-48128 the requests sent directly to a shard should not be versioned, so, there shouldn't be any cache refresh on the node that a direct client can't retry, however, this is not the case, there is a cache refresh that's being called when executing a mapReduceWrite. Running a jstest will show that during the direct client check, there is a refresh while doing a mapReduceWithWrite.



 Comments   
Comment by Githook User [ 10/May/21 ]

Author:

{'name': 'Eric Cox', 'email': 'eric.cox@mongodb.com', 'username': 'ericox'}

Message: SERVER-53433 Map reduce is versioned on direct connection to shards

(cherry picked from commit 513af98cb33e736d885a2a1b159685c335492cf0)
Branch: v4.4
https://github.com/mongodb/mongo/commit/1595be5ccc30782bb8482938c84e57cfaa09c5b3

Comment by Githook User [ 13/Apr/21 ]

Author:

{'name': 'Eric Cox', 'email': 'eric.cox@mongodb.com', 'username': 'ericox'}

Message: SERVER-53433 Map reduce is versioned on direct connection to shards
Branch: master
https://github.com/mongodb/mongo/commit/513af98cb33e736d885a2a1b159685c335492cf0

Comment by Eric Cox (Inactive) [ 12/Feb/21 ]

bernard.gorman I think I was able to reproduce this using these steps, does this make sense?

1.) Start a single node replSet for the shard nodes,

./build/install/bin/mongod --replSet rs0 --port 27018 --bind_ip localhost --dbpath /data/db

connect to this node with the mongo shell, and run rs.initiate()

2.) Start a single node replSet for a config server,

./build/install/bin/mongod --configsvr --replSet cfg0 --dbpath /data/dbcfg --bind_ip localhost --port 27019

connect to this node with the mongo shell, and run rs.initiate()

3.) start a mongos connected to the config server replSet cfg0

./build/install/bin/mongos --configdb cfg0/localhost:27019 --bind_ip localhost --port 27017

4.) connect to mongos with the mongo shell and shard a database, and shard some collection "mrfoo".

sh.addShard("rs0/localhost:27018")
sh.enableSharding("db")
sh.shardCollection("db.mrfoo", {a: 1})

5.) insert data through mongos

db.mrfoo.insert([{a: 1}, {b: 2}, ...])

6.) Stop mongo shell and directly connect to the shard server mongod at localhost:27018.

Run a map reduce job,

var mapper = function() { emit(this._id, this.a) };
var reduce = function(k, v) { return Array.sum(v) };
db.mrfoo.mapReduce(mapper, reduce, {out: "mr_out"})

This should run fine.

Now to reproduce the db refresh error, we will: 

  • teardown the config server, nuke it's state on disk under /data/dbcfg
  • kill mongos
  • bring up the cfg server and mongos as in step 2 and 3
  • Now don't connect to mongos at all, but rather directly connect to the mongod, and run the same mapReduce job as before.

The error that I'm seeing is:

rs0:PRIMARY> db.mrfoo.mapReduce(mapper, reduce, {out: "mr_out"})rs0:PRIMARY> db.mrfoo.mapReduce(mapper, reduce, {out: "mr_out"})uncaught exception: Error: map reduce failed:{ "ok" : 0, "errmsg" : "MapReduce internal error :: caused by :: database test not found", "code" : 26, "codeName" : "NamespaceNotFound", "$gleStats" : { "lastOpTime" : Timestamp(0, 0), "electionId" : ObjectId("7fffffff0000000000000001") }, "lastCommittedOpTime" : Timestamp(1613161903, 1), "$configServerState" : { "opTime" : { "ts" : Timestamp(1613161912, 2), "t" : NumberLong(-1) } }, "$clusterTime" : { "clusterTime" : Timestamp(1613161912, 2), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1613161903, 1)} :

Comment by Eric Cox (Inactive) [ 04/Feb/21 ]

I took another look at this. The main problem that was pointed out above is that a mapReduce command is versioned via a direct client connection which triggers a refresh here.  The call to getCollectionOptions() during the mapReduce with an out parameter is invoked in DocumentSourceOut.

Unfortunately, I still haven't been able to reproduce the failure linked in BF-19541 I essentially did the following to try to reproduce. 

  • ran mongo --nodb
  • Executed commands one-at-a-time in the jstest file that showed the failure, but I move the mapreduceWithWrite in the CRUDCommands objects to the very first position so the shard version begins with UNKNOWN.

After speaking with marcos.grillo and bernard.gorman, who discussed the changes in SERVER-48128 which was focused on avoiding adding versioning info to internal commands if the parent op is unversioned. It seems like the best course of action is to avoid a cache refresh is to go through the command flow for a map reduce and make sure that the command is unversioned under a direct client connection.

Speaking to Bernard we discussed one possible way to do this is to rework the ShardServerProcessInterface methods to avoid using the cache completely if !_opIsVersioned  his might entail some changes to _versionCommandIfAppropriate. Another idea may be to call a superclass method like CommonMongoProcessInterface::<methodName> if the parent op isn't versioned.

Generated at Thu Feb 08 05:30:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.