[SERVER-39019] $elemMatch $ne serialization is incorrect, doesn't roundtrip Created: 15/Jan/19  Updated: 29/Oct/23  Resolved: 19/Feb/19

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: 3.6.15, 4.0.7, 4.1.9, 3.4.24

Type: Bug Priority: Major - P3
Reporter: Brigitte Lamarche (Inactive) Assignee: James Wahlin
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
depends on SERVER-13779 Allow $not to be applied to $regex (c... Closed
is depended on by SERVER-38544 Serialize optimized $match stage inst... Closed
Related
is related to SERVER-1454 Use $not as a top-level logical op Backlog
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0, v3.6, v3.4, v3.2
Sprint: Query 2019-01-28, Query 2019-02-11, Query 2019-02-25
Participants:
Case:

 Description   

To reproduce: 

./mongo

> pipeline = [\{ $match: { 'a': { $elemMatch: {$ne:400 }} }} ]

> db.hello.explain().aggregate(pipeline)

Result looks like: 

 {
	"stages" : [
		{
			"$cursor" : {
				"query" : {
					"a" : {
						"$elemMatch" : {
							"$ne" : 400
						}
					}
				},
				"queryPlanner" : {
					"plannerVersion" : 1,
					"namespace" : "test.hello",
					"indexFilterSet" : false,
					"parsedQuery" : {
						"a" : {
							"$elemMatch" : {
								"0" : {
									"" : {
										"$eq" : 400
									}
								}
							}
						}
					},
					"winningPlan" : {
						"stage" : "EOF"
					},
					"rejectedPlans" : [ ]
				}
			}
		}
	],
	"ok" : 1
}

 

 Note that parsedQuery does not make sense. Namely, it has a key value which is an empty string.



 Comments   
Comment by Githook User [ 24/Sep/19 ]

Author:

{'name': 'James Wahlin', 'username': 'jameswahlin', 'email': 'james.wahlin@mongodb.com'}

Message: SERVER-39019 $elemMatch $ne serialization is incorrect, doesn't roundtrip

(cherry picked from commit 3db787da9dc453192adcf7706991d2d09f0ff0c7)
Branch: v3.4
https://github.com/mongodb/mongo/commit/523f08103d6317dff1700828d939b09d11c2fd2d

Comment by James Wahlin [ 16/Sep/19 ]

Thanks for reporting this zhihuifan@163.com. I have opened SERVER-43349 to investigate and fix. Please add yourself as a watcher to that ticket to track.

Comment by zhihui fan [ 16/Sep/19 ]

looks I found another corner case for this issue,  just FYI.

try: 

{ { "a" : { "$elemMatch" : { "$not" :

{ "$ne" : 400 }

} } }

 

 

Comment by Githook User [ 11/Sep/19 ]

Author:

{'name': 'James Wahlin', 'username': 'jameswahlin', 'email': 'james.wahlin@mongodb.com'}

Message: SERVER-39019 $elemMatch $ne serialization is incorrect, doesn't roundtrip
Branch: v3.6
https://github.com/mongodb/mongo/commit/3db787da9dc453192adcf7706991d2d09f0ff0c7

Comment by David Storch [ 05/Sep/19 ]

bduncan@visualmining.com, we are continuing to work on the custom fix for this issue on the 3.6 branch, but at the moment it has not been merged yet. The "fixVersion" field will be updated accordingly when the fix is merged. You should also see a comment on this ticket linking to the github commit when this is merged to the 3.6 branch. Our apologies for the delay! We found a correctness issue with the originally proposed backport and are working through it.

Comment by Bruce Duncan [ 05/Sep/19 ]

@david.storch Was this fix merged and present in v3.6.14? If not, when will it get pushed out?

Comment by David Storch [ 03/Jul/19 ]

brucehappy, apologies for the delay. We've been discussing the best route forward internally, since the fix implemented for 4.0 and 4.2 does not backport cleanly to 3.6 or older branches. Attempting to backport the complete fix to older stable branches seems too risky. After some discussion, we think we have identified an appropriate custom fix that would be safe for backport to the 3.6 branch. I've scheduled the custom fix to be worked on by the Server Query team in our upcoming development iteration ending on July 29th. Once the fix is merged, it will be released in the next 3.6.x version.

Comment by Bruce Duncan [ 01/Jul/19 ]

Could I please get an ETA on backporting this to 3.6 @james.wahlin? Its been 4 months since it was requested.

Comment by Githook User [ 19/Feb/19 ]

Author:

{'name': 'James Wahlin', 'email': 'james@mongodb.com', 'username': 'jameswahlin'}

Message: SERVER-39019 Fix incorrect $elemMatch $ne serialization

(cherry picked from commit 4677879eb75934ee3cd76c9e9807f6892ac17400)

SERVER-13779 Allow $not to be applied to $regex

(cherry picked from commit a2c0f15d6dc9fcda389b18b54287c4fcb5be44cd)
Branch: v4.0
https://github.com/mongodb/mongo/commit/a973d4f651a4e0f5a19c2edb2de2e6e53ee5e212

Comment by Githook User [ 19/Feb/19 ]

Author:

{'name': 'James Wahlin', 'username': 'jameswahlin', 'email': 'james@mongodb.com'}

Message: SERVER-39019 Fix incorrect $elemMatch $ne serialization
Branch: master
https://github.com/mongodb/mongo/commit/f74085d9935953ccfef6547ab5fdb2db48096828

Comment by James Wahlin [ 17/Jan/19 ]

When a MatchExpression tree representing the query {x: {$ne: 1}} is serialized, we serialize to {$nor: [{$eq: 1}]} rather than {x: {$not: {$eq: 1}}} which are equivalent statements. We run in to trouble however when the $ne is embedded in a $elemMatch. In this case we attempt to serialize {x: {$elemMatch: {$ne: 1}}} to {x: {$elemMatch: {$nor: [{$eq: 1}]}}}, which is not valid as $nor expects children that are top-level expressions.   

Comment by Brigitte Lamarche (Inactive) [ 15/Jan/19 ]

SERVER-38544 needs to serialize _expression (instead of _predicate) in DocumentSourceMatch's getQuery. However, this expression will serialize incorrectly for  the bug presented in SERVER-39019.

Generated at Thu Feb 08 04:50:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.