[SERVER-37653] data inconsistent Created: 17/Oct/18  Updated: 07/Apr/23  Resolved: 23/Oct/18

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 3.2.20
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: chensi Assignee: Kelsey Schubert
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

know no way to reproduce

Participants:

 Description   

data can be found by some query on primary:

shard01:PRIMARY> db.XXXXXXXXXX.find({"task_ids" : [ NumberLong("6566381934111359492") ]})

{ "_id" : ObjectId("5b20be13e1803d1f2529dbe0"), "count" : 1, "task_ids" : [ NumberLong("6566381934111359492") ],....}

 

but cannot found by _id:

shard01:PRIMARY> db.XXXXXXXXXX.find({"_id":ObjectId("5b20be13e1803d1f2529dbe0")}).count()
0

the other secondaries are ok.

 



 Comments   
Comment by Kelsey Schubert [ 23/Oct/18 ]

Hi chensi_04@126.com,

Thanks for the additional information. Unfortunately, in cases like this where the issue could have occurred some time ago without any hints to reproduce the issue, it's very challenging to investigate. As MongoDB 3.2 is no longer supported, I'm going to resolve this ticket.

Since your secondaries are consistent, I would recommend clearing the $dbpath of the affected node and performing an initial sync. From the information provided, it appears that the _id index on this node is likely corrupt. While the index could be rebuilt, an initial sync would ensure that that the data is consistent across the replica set.

Kind regards,
Kelsey

Comment by chensi [ 22/Oct/18 ]

we don't use the momgos, just query against mongod.

the following are explain results:

A. data not found

{{ "queryPlanner" : { "plannerVersion" : 1, "namespace" : "XXX", "indexFilterSet" : false, "parsedQuery" : { "_id" :

{ "$eq" : ObjectId("5b20be13e1803d1f2529dbe0") }

}, "winningPlan" : { "stage" : "IDHACK" }, "rejectedPlans" : [ ] }, "executionStats" : { "executionSuccess" : true, "nReturned" : 0, "executionTimeMillis" : 0, "totalKeysExamined" : 0, "totalDocsExamined" : 0, "executionStages" :

{ "stage" : "IDHACK", "nReturned" : 0, "executionTimeMillisEstimate" : 0, "works" : 1, "advanced" : 0, "needTime" : 0, "needYield" : 0, "saveState" : 0, "restoreState" : 0, "isEOF" : 1, "invalidates" : 0, "keysExamined" : 0, "docsExamined" : 0 }

},

 

B. data founded

{{ "queryPlanner" : { "plannerVersion" : 1, "namespace" : "ops_aggdata.tcs_project_blindverify_not_same_hourly", "indexFilterSet" : false, "parsedQuery" : { "task_ids" :

{ "$eq" : [ NumberLong("6566381934111359492") ] }

}, "winningPlan" : { "stage" : "COLLSCAN", "filter" : { "task_ids" :

{ "$eq" : [ NumberLong("6566381934111359492") ] }

}, "direction" : "forward" }, "rejectedPlans" : [ ] }, "executionStats" : { "executionSuccess" : true, "nReturned" : 1, "executionTimeMillis" : 4031, "totalKeysExamined" : 0, "totalDocsExamined" : 1156773, "executionStages" : { "stage" : "COLLSCAN", "filter" : { "task_ids" :

{ "$eq" : [ NumberLong("6566381934111359492") ] }

}, "nReturned" : 1, "executionTimeMillisEstimate" : 3911, "works" : 1156775, "advanced" : 1, "needTime" : 1156773, "needYield" : 0, "saveState" : 9037, "restoreState" : 9037, "isEOF" : 1, "invalidates" : 0, "direction" : "forward", "docsExamined" : 1156773 } },

 

BTW, data on the secondaries are consistent, we are moving to the 3.6 version

Comment by Kelsey Schubert [ 19/Oct/18 ]

Hi chensi_04@126.com,

Thanks for reporting this issue. Please note that MongoDB 3.2 is no longer supported. I would recommend upgrading to a more recent version of MongoDB.

Would you please clarify whether these queries reliably produce inconsistent results? If so, would you please provide the
.explain("executionStats") for both? When you query from the mongos rather than directly against the mongod, what do you see?

Thank you,
Kelsey

Generated at Thu Feb 08 04:46:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.