[SERVER-75449] Query using _id index doesn't populate indexKey metadata field Created: 29/Mar/23  Updated: 13/Apr/23  Resolved: 13/Apr/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ben Shteinfeld Assignee: Backlog - Query Optimization
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Query Optimization
Operating System: ALL
Participants:

 Description   

Steps to reproduce:

 

> db.c.insert({_id: 1, a: "foo"})
> db.c.createIndex({a: 1})
> db.c.find({a: "foo"}, {key: {$meta: "indexKey"}})
[ { _id: 1, a: 'foo', key: { a: 'foo' } } ]
> db.c.find({_id: 1}, {key: {$meta: "indexKey"}})
[ { _id: 1, a: 'foo' } ]

I would have expected:

 

> db.c.find({_id: 1}, {key: {$meta: "indexKey"}})
[ { _id: 1, a: 'foo', key: { _id: 1 } } ] 

> db.c.find({_id: 1}, {key: {$meta: "indexKey"}}).explain().queryPlanner
{
  namespace: 'test.c',
  indexFilterSet: false,
  parsedQuery: { _id: { '$eq': 1 } },
  queryHash: '58F0F49D',
  planCacheKey: '0028A5F4',
  maxIndexedOrSolutionsReached: false,
  maxIndexedAndSolutionsReached: false,
  maxScansToExplodeReached: false,
  winningPlan: {
    stage: 'PROJECTION_DEFAULT',
    transformBy: { key: { '$meta': 'indexKey' } },
    inputStage: { stage: 'IDHACK' }
  },
  rejectedPlans: []
} 

One theory is that IDHACK is not setting the metadata field properly, but this requires further investigation.

 

 



 Comments   
Comment by Nicholas Zolnierz [ 13/Apr/23 ]

While this is likely a bug, $meta indexKey/returnKey is internal for debugging and we don't plan to fix bugs in this area.

Comment by Sarah Olson [ 31/Mar/23 ]

Thanks for the additional detail ben.shteinfeld@mongodb.com. My recommendation is that we shouldn't document this then, on the grounds that this issue does not have customer impact for the significant majority of our customers. 

Comment by Ben Shteinfeld [ 30/Mar/23 ]

> MongoDB bypasses the normal query plan path for the query shape, perhaps to improve speed. Is that correct?

Yes that's right. It is a special plan which uses the _id index. The documented semantics of {$meta: "indexKey"} are to emit the indexKey when a plan uses an index scan stage.

> Do users expect to be able to bypass the normal query plan path for {}{$meta: "indexKey"}{}? Or will disqualifying {$meta: "indexKey"} from being able to bypass the normal query plan path be brand new behavior? (It sounds to me like the former, but thought I should check.)

If we were to fix this in the server, then I expect the fix to be to disqualify a plan with a meta projection from using IDHACK.

> How would this change impact users?

It is highly unlikely any user has run into this. The meta projection is documented as a debug-only expression. Additionally, if you are using the _id index, then getting the indexKey is not useful to you as you already are using it in the query.

Comment by Sarah Olson [ 30/Mar/23 ]

Great question, ben.shteinfeld@mongodb.com! Thanks for asking.
Usually, it's a better idea to fix the underlying issue than to try to document our way out of software bugs. I might be misreading the ticket / potential documentation request, though, so a few follow-on questions: * We don't document IDHACK anywhere. Googling around, it seems to mean that MongoDB bypasses the normal query plan path for the query shape, perhaps to improve speed. Is that correct?

  • Do users expect to be able to bypass the normal query plan path for {}{$meta: "indexKey"}{}? Or will disqualifying {$meta: "indexKey"} from being able to bypass the normal query plan path be brand new behavior? (It sounds to me like the former, but thought I should check.)
  • How would this change impact users?

Thanks a ton for the additional detail!

Comment by Ben Shteinfeld [ 30/Mar/23 ]

We discussed this during QO triage and are unsure if this should be a server bug or could be fixed with a documentation update? Specifically, we could change the documented semantics of {$meta: "indexKey"} to explicitly disqualify IDHACK plans. ashley.brown@mongodb.com sarah.olson@mongodb.com do you think this is a reasonable path forward?

Generated at Thu Feb 08 06:30:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.