[SERVER-57403] Serialization error in let variables in a sharded cluster Created: 03/Jun/21  Updated: 29/Oct/23  Resolved: 14/Jun/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.0.0-rc2, 5.1.0-rc0

Type: Bug Priority: Major - P3
Reporter: Charlie Swanson Assignee: Katherine Wu (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by DRIVERS-776 Support 'let' option for aggregate c... Closed
Related
related to SERVER-69504 Leave index build phase intact upon e... Closed
is related to SERVER-46707 Support 'let' variables in aggregate ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0
Sprint: Query Optimization 2021-06-28
Participants:

 Description   

The following command should work, but doesn't on a mongos:

{
  aggregate: 'coll0',
  pipeline: [
    {
      '$match': { '$expr': { '$eq': [ '$_id', 2] } }
    },
    { '$project': { _id: 0, y: '$$y'} }
  ],
  let: { y: { '$literal': '$bar' } },
  cursor: {}
}

If you look at the profiler in the shards, it looks like we just need to preserve a $const or $literal in the serialization. That is dropped and we just end up with y: "$bar":

{
	"op" : "command",
	"ns" : "test.coll0",
	"command" : {
		"aggregate" : "coll0",
		"pipeline" : [
			{
				"$match" : {
					"$expr" : {
						"$eq" : [
							"$_id",
							2
						]
					}
				}
			},
			{
				"$project" : {
					"_id" : 0,
					"y" : "$$y"
				}
			}
		],
		"cursor" : {
			"batchSize" : NumberLong(101)
		},
		"let" : {
			"y" : "$bar"
		},
		"fromMongos" : false,
		"readConcern" : {
			"level" : "local",
			"provenance" : "implicitDefault"
		},
		"writeConcern" : {
			"w" : "majority",
			"wtimeout" : 0,
			"provenance" : "customDefault"
		},
		"shardVersion" : [
			Timestamp(0, 0),
			ObjectId("000000000000000000000000")
		],
		"databaseVersion" : {
			"uuid" : UUID("d40d3f32-02c0-42aa-939d-2d5f640d43b6"),
			"timestamp" : Timestamp(1622740896, 32),
			"lastMod" : 1
		},
		"lsid" : {
			"id" : UUID("bd190341-9e6d-4458-b8df-35a7dd5a5072"),
			"uid" : BinData(0,"47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=")
		},
		"$clusterTime" : {
			"clusterTime" : Timestamp(1622741302, 25),
			"signature" : {
				"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
				"keyId" : NumberLong(0)
			}
		},
		"$configTime" : Timestamp(1622741302, 25),
		"$topologyTime" : Timestamp(1622740776, 3),
		"$client" : {
			"application" : {
				"name" : "MongoDB Shell"
			},
			"driver" : {
				"name" : "MongoDB Internal Client",
				"version" : "5.0.0-rc0"
			},
			"os" : {
				"type" : "Darwin",
				"name" : "Mac OS X",
				"architecture" : "x86_64",
				"version" : "19.4.0"
			},
			"mongos" : {
				"host" : "charlie-macbook-pro:20003",
				"client" : "127.0.0.1:54610",
				"version" : "5.0.0-rc0"
			}
		},
		"$configServerState" : {
			"opTime" : {
				"ts" : Timestamp(1622741302, 25),
				"t" : NumberLong(-1)
			}
		},
		"$db" : "test"
	},
	"numYield" : 0,
	"locks" : {
		"Global" : {
			"acquireCount" : {
				"r" : NumberLong(1)
			}
		},
		"Mutex" : {
			"acquireCount" : {
				"r" : NumberLong(2)
			}
		}
	},
	"flowControl" : {
 
	},
	"readConcern" : {
		"level" : "local",
		"provenance" : "implicitDefault"
	},
	"writeConcern" : {
		"w" : "majority",
		"wtimeout" : 0,
		"provenance" : "customDefault"
	},
	"ok" : 0,
	"errMsg" : "Command let Expression tried to access a field, but this is not allowed because Command let Expressions run before the query examines any documents.",
	"errName" : "Location4890500",
	"errCode" : 4890500,
	"responseLength" : 578,
	"protocol" : "op_msg",
	"millis" : 0,
	"ts" : ISODate("2021-06-03T17:28:30.794Z"),
	"client" : "127.0.0.1",
	"appName" : "MongoDB Shell",
	"allUsers" : [ ],
	"user" : ""
}



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 16/Jun/21 ]

Author:

{'name': 'Katherine Wu', 'email': 'katherine.wu@mongodb.com', 'username': 'kaywux'}

Message: SERVER-57403 Serialize 'let' variables by wrapping with $literal
Branch: SERVER-34632
https://github.com/mongodb/mongo/commit/19572bcc17bf07c391a2248e0dd52bc08d207fcd

Comment by Katherine Wu (Inactive) [ 15/Jun/21 ]

jmikola This has now been backported to 5.0 and should be in the rc2 preview release. I've checked the "Downstream Team Attention" for drivers on this ticket since the backport ticket didn't have that metadata field available. Let me know if there are any other issues regarding this!

Comment by Githook User [ 15/Jun/21 ]

Author:

{'name': 'Katherine Wu', 'email': 'katherine.wu@mongodb.com', 'username': 'kaywux'}

Message: SERVER-57403 Serialize 'let' variables by wrapping with $literal

(cherry picked from commit 365905c56bb39c82e729df1a4b77de797606ba5e)
Branch: v5.0
https://github.com/mongodb/mongo/commit/b065ffbb33bb891b87620fb24cd6652527df6912

Comment by Jeremy Mikola [ 15/Jun/21 ]

katherine.wu: Ah, I didn't realize backports use their own tickets. Relying on that to track the downstream change for drivers works for me. Thanks!

Comment by Katherine Wu (Inactive) [ 15/Jun/21 ]

jmikola this patch is in master (currently 5.1.0) and is not yet in any 5.0 release. I filed BACKPORT-9664 which tracks the work to backport it into a 5.0 preview release (specifically rc2 since that is targeted for the 16th). I was planning to use the "Downstream Team Attention" for drivers for that ticket only to avoid duplication, but can also mention that here.

Comment by Jeremy Mikola [ 14/Jun/21 ]

katherine.wu: Should "Downstream Team Attention" at least impact drivers since we have spec tests depending on this fix?

Comment by Githook User [ 14/Jun/21 ]

Author:

{'name': 'Katherine Wu', 'email': 'katherine.wu@mongodb.com', 'username': 'kaywux'}

Message: SERVER-57403 Serialize 'let' variables by wrapping with $literal
Branch: master
https://github.com/mongodb/mongo/commit/19572bcc17bf07c391a2248e0dd52bc08d207fcd

Comment by Katherine Wu (Inactive) [ 08/Jun/21 ]

Got it, thanks for the context! From the release dashboard it looks like 5.0.0-rc1 is targeted for tomorrow, and there is no rc2 planned so unfortunately this fix will not make it into a preview release. I'll start work on this soon, and once it's complete can see about the possibility of backporting it into rc1 so drivers can verify before the final release.

Comment by Jeremy Mikola [ 07/Jun/21 ]

In my last comment, "5.0 final" was referring to the 5.0.0 stable release. A preview release would be anything before that (e.g. rc0, rc1, rc2). Ideally, a fix should be available in some release before the final so that we (drivers) can verify and test that it has been fixed.

Comment by Jeffrey Yemin [ 07/Jun/21 ]

Here's another example, which charlie.swanson says has the same root cause. I can repro this on a standalone. This command:

              {
                  "aggregate": "let-test",
                  "cursor": {},
                  "let": {
                    "y": {
                      "$literal": "$bar"
                    }                  },
                  "pipeline": [
                    {
                      "$project": {
                        "_id": 0,
                        "y": "$$y",
                      }
                    }
                    ,
                    {
                      "$merge": {
                        "into": "let-test-other"
                      }
                    }
                  ]
                }

fails with

{
   "ok": 0.0, 
   "errmsg": "PlanExecutor error during aggregation :: caused by :: Command let Expression tried to access a field, but this is not allowed because Command let Expressions run before the query examines any documents.", 
   "code": 4890500, 
   "codeName": "Location4890500"}

Comment by Githook User [ 07/Jun/21 ]

Author:

{'name': 'Jeremy Mikola', 'email': 'jmikola@gmail.com', 'username': 'jmikola'}

Message: DRIVERS-776: Split tests to work around SERVER-57403 (#1013)
Branch: master
https://github.com/mongodb/specifications/commit/9465cda31270e41d3d87fe9b9b14ad601c3861c6

Comment by Jeremy Mikola [ 07/Jun/21 ]

Note: this issue caused a failure in the spec tests for DRIVERS-776. I just created a PR (mongodb/specifications#1013) to split the original test in two and work around the mongos issue, but would appreciate it if we could have an ETA on when this might be addressed (e.g. 5.0 preview release, 5.0 final release, subsequent patch release).

Comment by Jeremy Mikola [ 04/Jun/21 ]

Cross-referencing with SERVER-46707, since this appears to be specific to mongos.

Generated at Thu Feb 08 05:41:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.