[SERVER-67446] Ensure consistent wildcardProjection and columnstoreProjection specs in catalog Created: 22/Jun/22  Updated: 29/Oct/23  Resolved: 19/Jan/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.0.15, 6.3.0-rc0, 6.0.5

Type: Improvement Priority: Major - P3
Reporter: Bernard Gorman Assignee: Kevin Cherkauer
Resolution: Fixed Votes: 0
Labels: query-offsite
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-57987 Factor index projection normalization... Closed
Documented
is documented by DOCS-15829 [Server] Ensure consistent wildcardPr... Closed
Related
related to SERVER-73302 Changed output format in createIndexes Closed
is related to SERVER-25023 no way to index the same fields with ... Closed
is related to SERVER-47659 Add 'wildcardProjection' parameter to... Closed
is related to SERVER-57987 Factor index projection normalization... Closed
Assigned Teams:
Query Execution
Backwards Compatibility: Minor Change
Backport Requested:
v6.1, v6.0, v5.3, v5.0
Participants:

 Description   

In SERVER-25023 we started normalizing the wildcardProjection field of index specs that we persist in the catalog, in preparation for SERVER-47659 where we added wildcardProjection to the index signature. The normalization is intended to obviate any differences in how the user may specify the same projection, e.g. {"a.b": 1} and {a: {b: 1} } will both be transformed to the latter form. This allows us to easily compare the BSONObj specs when the user attempts to build a new index, to see whether an identical index already exists.

However, it has recently been observed that building a wildcard index on a sharded cluster may permute the ordering of fields in the normalized spec across shards; for instance, we may end up with {name: 0, type: 0, _id: 1} on one shard and {type: 0, name: 0, _id: 1} on another. This appears to be because the projection tree holds its nodes as an unordered map and may therefore produce a field-order-nondeterministic normalization of the spec.

While the Server itself will regard these specs as being equivalent and will behave correctly if the user submits multiple createIndex requests, this complicates the logic required by downstream products which read the set of index specs from the catalog, notably C2C.

We should address this by doing one of the following (in likely order of desirability):

  • Persist the user's requested wildcardProjection verbatim and defer normalization until the point where we need to compare index specs.
  • Persist the user's requested wildcardProjection verbatim and add an operator== to the Projection tree to facilitate direct comparison
  • Store the projection nodes in a deterministic order so that the persisted normalized forms are also deterministic in their field ordering.


 Comments   
Comment by Githook User [ 03/Feb/23 ]

Author:

{'name': 'Kevin Cherkauer', 'email': 'kevin.cherkauer@mongodb.com', 'username': 'kevin-cherkauer'}

Message: SERVER-67446 Ensure consistent wildcardProjection specs in catalog

(cherry picked from commit d0c617dbf2dfbd15e30dc33102cdfde5113e44bb)
Branch: v5.0
https://github.com/mongodb/mongo/commit/04da63f65f00c17daca95c1e0c68f95853bdbdbb

Comment by Githook User [ 27/Jan/23 ]

Author:

{'name': 'Kevin Cherkauer', 'email': 'kevin.cherkauer@mongodb.com', 'username': 'kevin-cherkauer'}

Message: SERVER-67446 Ensure consistent wildcardProjection specs in catalog
Branch: v6.0
https://github.com/mongodb/mongo/commit/d0c617dbf2dfbd15e30dc33102cdfde5113e44bb

Comment by Githook User [ 27/Jan/23 ]

Author:

{'name': 'Kevin Cherkauer', 'email': 'kevin.cherkauer@mongodb.com', 'username': 'kevin-cherkauer'}

Message: SERVER-67446 Ensure consistent wildcardProjection specs in catalog
Branch: v6.0
https://github.com/10gen/mongo-enterprise-modules/commit/2646486497b564e1dace34ff64821df448eea311

Comment by Githook User [ 19/Jan/23 ]

Author:

{'name': 'Kevin Cherkauer', 'email': 'kevin.cherkauer@mongodb.com', 'username': 'kevin-cherkauer'}

Message: SERVER-67446 Ensure consistent wildcardProjection specs in catalog
Branch: master
https://github.com/mongodb/mongo/commit/210f9cbb2a8759db1d9161ee21cf3b596d02d028

Comment by Githook User [ 19/Jan/23 ]

Author:

{'name': 'Kevin Cherkauer', 'email': 'kevin.cherkauer@mongodb.com', 'username': 'kevin-cherkauer'}

Message: SERVER-67446 Ensure consistent wildcardProjection specs in catalog
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/cd5d83f28bd37869e6e8cc8f56ecacb450c5edd7

Comment by Kevin Cherkauer [ 27/Dec/22 ]

The fix for this ticket also fixes SERVER-57987.

Comment by Brenda Rodriguez [ 12/Jul/22 ]

Because lingzhi.deng@mongodb.com said this is not a C2C blocker, throwing this onto the backlog.

Generated at Thu Feb 08 06:08:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.