[SERVER-76875] Exclude fields containing dots from indexes Created: 05/May/23  Updated: 10/Nov/23  Resolved: 10/May/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.1.0-rc0, 7.0.0-rc1, 6.0.7

Type: Bug Priority: Minor - P4
Reporter: Alberto Massari Assignee: Alberto Massari
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Documented
is documented by DOCS-16097 [SERVER] Document better the limitati... Backlog
Duplicate
duplicates SERVER-75505 Difference in dotted path handling be... Closed
Problem/Incident
causes SERVER-76542 Geo 2d index update validation differ... Closed
causes SERVER-83105 Disable index key validation in multi... Closed
Related
related to SERVER-77113 Exclude fields containing dots from t... Closed
is related to SERVER-78238 Text index indexes content of fields ... Closed
Assigned Teams:
Query Execution
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v7.0, v6.3, v6.0
Sprint: QE 2023-05-15
Participants:
Linked BF Score: 120

 Description   

When an index is defined on a path "obj.nested.field", it's supposed to index a document 

{ obj: {nested: {field: "value"}}}

However, it also indexes 

{obj: {"nested.field": "value"}}

as it looks evident in this testcase 

 

db.geo.createIndex({"obj.inner.geoLegacy": "2d"})
db.geo.insert([ {_id: 17922, "obj": {_id: 17923, "str": "random string"} } ])
db.geo.runCommand( {"update": "geo", "updates": [{"q": {}, "u": [{$set: {"obj": {$setField: {field: "inner.geoLegacy", input: "$obj", value: "$obj"}},}}, ], "upsert": false, "multi": false, }], "ordered": true, })
db.geo.validate({full: true})

where the created document

 

{
  "_id": 17922,
  "obj": {
    "_id": 17923,
    "str": "random string",
    "inner.geoLegacy": {
      "_id": 17923,
      "str": "random string",
    }
  }
}

is using the content of the "inner.geoLegacy" field as a valid match for the indexed path "obj.inner.geoLegacy", and fails to treat it as a 2D legacy point.

 

{
...
        "valid" : false,
        "repaired" : false,
...
        "errors" : [
                "exception during collection validation: Location13026: geo values must be 'legacy coordinate pairs' for 2d indexes :: caused by :: { _id: 17922.0, obj: { _id: 17923.0, str: \"random string\", inner.geoLegacy: { _id: 17923.0, str: \"random string\" } } }"
        ],
...
        "advice" : "A corrupt namespace has been detected. See http://dochub.mongodb.org/core/data-recovery for recovery steps.",
        "ok" : 1
}

 

 

Given that we document that fields whose name is starting with "$" or containing "." are not included in indexes or encrypted, we should skip them rather than including them and yield wrong results.

 

A maybe related issue is that the insertion of a dotted path matching the index key makes a document be excluded when the index is used, compared with when a collection scan is performed:

db.x.insert([{a:1, b:{c:{x: 2, y:2}}}, {a:2, b:{c:{x:2, y:2}}}])
db.x.update({a:2}, [{$replaceWith: {$setField: {field: "b.c", input: "$$ROOT", value: {x: 20, y:20}}}}])
db.x.find()
{ "_id" : ObjectId("6454c87a0f747333893c632f"), "a" : 1, "b" : { "c" : { "x" : 2, "y" : 2 } } }
{ "_id" : ObjectId("6454c87a0f747333893c6330"), "a" : 2, "b" : { "c" : { "x" : 2, "y" : 2 } }, "b.c" : { "x" : 20, "y" : 20 } }
 
db.x.find({"b.c": {$geoWithin: {$center: [ [2,2], 10]}}})
{ "_id" : ObjectId("6454c87a0f747333893c632f"), "a" : 1, "b" : { "c" : { "x" : 2, "y" : 2 } } }
{ "_id" : ObjectId("6454c87a0f747333893c6330"), "a" : 2, "b" : { "c" : { "x" : 2, "y" : 2 } }, "b.c" : { "x" : 20, "y" : 20 } }
 
db.x.createIndex({"b.c":"2d"})
 
db.x.find({"b.c": {$geoWithin: {$center: [ [2,2], 10]}}})
{ "_id" : ObjectId("6454c87a0f747333893c632f"), "a" : 1, "b" : { "c" : { "x" : 2, "y" : 2 } } }

 

 



 Comments   
Comment by Githook User [ 12/May/23 ]

Author:

{'name': 'Alberto Massari', 'email': 'alberto.massari@mongodb.com', 'username': 'albymassari'}

Message: SERVER-76875 Exclude fields with embedded dots from 2D and FTS indexes
Branch: v7.0
https://github.com/mongodb/mongo/commit/a73a398bc0f57a71954ca266ad01c262a0a7e92a

Comment by Githook User [ 12/May/23 ]

Author:

{'name': 'Alberto Massari', 'email': 'alberto.massari@mongodb.com', 'username': 'albymassari'}

Message: SERVER-76875 Exclude fields with embedded dots from 2D and FTS indexes
Branch: v6.0
https://github.com/mongodb/mongo/commit/9ade16a356a725a00baec30bea7a6e43f7267473

Comment by Githook User [ 10/May/23 ]

Author:

{'name': 'Alberto Massari', 'email': 'alberto.massari@mongodb.com', 'username': 'albymassari'}

Message: SERVER-76875 Exclude fields with embedded dots from 2D and FTS indexes
Branch: master
https://github.com/mongodb/mongo/commit/406e69f6f5dee8b698c4e4308de2e9e5cef6c12c

Generated at Thu Feb 08 06:33:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.