[SERVER-55158] Missing objects in request rersponse. Created: 11/Mar/21  Updated: 04/May/21  Resolved: 04/May/21

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 3.6.22
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Ivan Maireaux Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File documents_content-1.txt     Text File query_ok_explain-1.txt     Text File wrong_results_query_explain-1.txt    
Participants:

 Description   

Hello,

 

I have an issue when querying a collection using this criteria : 

{deleted_at: null, event_campaign_optin: true, event_id: ObjectId('5ea92da4a5522a080614d29f'), status: "registered", "label_ids": [ObjectId('6048a3c4b3d10c003c14b22e')]}

=> It returns only one Object (instead of 5)

 

So I tried to add a filter on the ids of expected objects :

_id: {"$in": [ObjectId('60377e2172fcdc005b32e9ff'), ObjectId('5efdaeafa62e5400233657a0'), ObjectId('5efca695f820c7144e46ffda'), ObjectId('5ee0cebdb28593001c8bfaf2'), ObjectId('5ed65cd396f090255d089b77')]}

 

The complete query : 

{deleted_at: null, event_campaign_optin: true, event_id: ObjectId('5ea92da4a5522a080614d29f'), status: "registered", "label_ids": [ObjectId('6048a3c4b3d10c003c14b22e')], _id: {"$in": [ObjectId('60377e2172fcdc005b32e9ff'), ObjectId('5efdaeafa62e5400233657a0'), ObjectId('5efca695f820c7144e46ffda'), ObjectId('5ee0cebdb28593001c8bfaf2'), ObjectId('5ed65cd396f090255d089b77')]}}

=> This one returns 5 objects

 

I do not understand how adding a AND filter on 5 ids could return more results ...

I executed this requests on the Atlas interface, same issue form mongoid ODM form the ruby console.

 

Any idea of what happen ? I have no clue.

I thought that the index used for the request could change the result, one request using a sparse index maybe ... so I forced the index to use with :

.extras(hint: {deleted_at: 1, event_id: 1, created_at: -1})

(from the ruby console this time, not atlas)

And still the same results.

 

Any clue is welcome. Thanks for your help.



 Comments   
Comment by Eric Sedor [ 04/May/21 ]

Thanks ivan.maireaux@eventmaker.io, I'm going to close this ticket but feel free to reach back out if needed.

Comment by Ivan Maireaux [ 03/May/21 ]

Hi Eric,

 

As the 3.6 reached end of life, we upgraded recently to 4.4 and effectively I can't reproduce anymore.

 

Thank you very much for your help !

Comment by Eric Sedor [ 27/Apr/21 ]

Hi ivan.maireaux@eventmaker.io and thanks for your patience.

I don't have specific information about other possibilities. It was simply my intention to not rule anything out.

I've had a chance to review the latest output you provided. If the document with id: ObjectId('605b147862f1bc005ca7bacd') can be found in the _id index but not in the event_id_1_thematics_quartile_1 index, it does suggest the event_id_1_thematics_quartile_1 index doesn't have an entry for the document.

I believe that in this case the problem could be related to the size of the thematics_quartile field. In not_returned_document_that_should_havve_been.txt, this looks to be a subdocument with about 43 subfields, like:

  :thematics_quartile => {
        "6006b0f86cfc425be86917dd" => 3,
        "6006b0f96cfc425be86917e9" => 1,
        "6006b0f96cfc425be86917ed" => 3,
        "6006b0f96cfc425be86917ef" => 1,
        "6006b0d76cfc425be8691673" => 1,
        "6006b0f96cfc425be86917eb" => 1,
        "6006b0f96cfc425be86917ee" => 1,
        "6006b0f96cfc425be86917f6" => 0,
        "6006b0fa6cfc425be8691800" => 0,
        "6006b0fa6cfc425be8691806" => 0,
        "6006b0fa6cfc425be8691807" => 0,
        "6006b0f96cfc425be86917ec" => 0,
        "6006b0f96cfc425be86917f1" => 0,
        "6006b0f96cfc425be86917f3" => 0,
        "6006b0f96cfc425be86917f4" => 4,
        "6006b0f96cfc425be86917f5" => 0,
        "6006b0f96cfc425be86917f7" => 0,
        "6006b0fa6cfc425be86917f9" => 0,
        "6006b0fa6cfc425be86917fb" => 0,
        "6006b0fa6cfc425be86917fc" => 0,
        "6006b0fa6cfc425be86917fd" => 0,
        "6006b0fa6cfc425be86917ff" => 0,
        "6006b0fa6cfc425be8691801" => 0,
        "6006b0fa6cfc425be8691804" => 0,
        "6006b0fa6cfc425be8691808" => 3,
        "6006b0fa6cfc425be8691809" => 0,
        "6006b0fa6cfc425be869180b" => 0,
        "6006b0fa6cfc425be869180c" => 0,
        "6006b0fb6cfc425be869180d" => 0,
        "6006b0fb6cfc425be869180e" => 4,
        "605cc8002959a70486ab404d" => 0,
        "605cc8012959a70486ab4051" => 0,
        "605cc8012959a70486ab4054" => 0,
        "605cc8012959a70486ab4055" => 0,
        "605cc8012959a70486ab4056" => 0,
        "605cc8012959a70486ab405b" => 0,
        "605cc8012959a70486ab405e" => 0,
        "605cc8012959a70486ab4060" => 0,
        "605cc8012959a70486ab4065" => 0,
        "605cc8022959a70486ab406d" => 0,
        "605ccc09e5fb72057d04fb80" => 0,
        "6006b0fa6cfc425be8691805" => 0,
        "606c7fc246c331003ec0af37" => 0
    }

This looks like it would be subject to the index key limitation. Can you provide the output of validate on the collection and also compare the other documents you expect to match the query? Do they also have large thematics_quartile subdocuments?

Importantly, since MongoDB 3.6 is reaching end of life at the end of this month (April 2021), it will help if you can reproduce the issue on MongoDB 4.0 and rule out the possibility that the data and indexes were created on 3.6 or an earlier version of MongoDB. Is this feasible?

Comment by Ivan Maireaux [ 07/Apr/21 ]

Hi Eric,

As the queries do not work, our customer has updated to use some other fields and launch his campaign so I can't reproduce the initial issue.

 

But we had an other similar case today.

I uploaded 4 files using the provided link (https://10gen-httpsupload.s3.amazonaws.com/upload_forms/77adcadf-ab72-4d61-9f06-12662007c3b5.html) 

  • The explain('executionStats') of the query that do not return all matches documents (4 instead of 59)
  • A full document returned by the previous query
  • The explain of the same query using a selector with an additional  condition on an id of document that the previous query do not return (but should have)
  • A full document that should be returned but is not.

 

In your first answer you wrote : "checking results against different indexes does sound like a good first step. But, there may be other possibilities"

Could you tell me more about these other possibilities ?

Comment by Eric Sedor [ 25/Mar/21 ]

HI ivan.maireaux@eventmaker.io;

Apologies, I should have asked for explain("executionStats") as it will include more information to help us infer what's being seen in the indexes. Could you re-run those?

Totally understood about the sensitivity documents in question. We've a couple of options. Could you either:

  • Upload full documents to this support uploader link. Files here are only visible to MongoDB employees and are deleted after a time; OR
  • Reproduce this issue in the MongoDB shell (using your projection strategy to project only what is in the query filter), and provide that output?
Comment by Ivan Maireaux [ 16/Mar/21 ]

Hi Eric,

I cannot provide a full content of a document as it contains personal data about people.

So I displayed fields that are present into the criteria selector => documents_content-1.txt

Query with wrong results explain => wrong_results_query_explain-1.txt

Query with filter on expected ids (results ok) explain => query_ok_explain-1.txt

 

Last week 1 guest was returned instead of 5, today I have 0 objects returned.

No update on any of these documents since 2021-03-10 12:26:35 UTC 

 

 

Comment by Eric Sedor [ 15/Mar/21 ]

Hi ivan.maireaux@eventmaker.io, checking results against different indexes does sound like a good first step. But, there may be other possibilities, so I'd like to ask for more information. Can you provide:

  • the full contents of a document you expect to match the query, that is not being returned?
  • the output of explain for both query forms? This will provide explicit information about what indexes are in use.
Generated at Thu Feb 08 05:35:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.