[SERVER-38309] "No query solutions" WriteConcernException on simple delete query with index Created: 29/Nov/18  Updated: 08/Feb/19  Resolved: 08/Feb/19

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 3.4.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Daniel Breitlauch Assignee: Eric Sedor
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

It does not happen in all sharded collections. So i dont think this is reproducible.

Although i can reproduce it consistently on this collection.

Participants:

 Description   

I have a sharded collection with 5 repsets consisting of 3 mongod instances respectively. On the shards I have a collection sharded by

{ t: hashed}

.

I enabled the no-table-scan parameter via db query to all mongod instances.
A while after that, delete and findAndModify queries showed up with no query solution.
The weird situation is that the queries should have a solution since indices for it are present.

There are these indices on the collection on all mongod instances:

[
    {
        "v" : 1,
        "key" : {
            "t" : "hashed"
        },
        "name" : "t_hashed",
    },
    {
        "v" : 1,
        "unique" : true,
        "key" : {
            "t" : 1,
            "ty" : 1,
            "s" : 1
        },
        "name" : "t_1_ty_1_s_1",
        "background" : true
    },
    {
        "v" : 1,
        "key" : {
            "s" : 1,
            "ty" : 1
        },
        "name" : "s_1_ty_1",
        "background" : true
    },
    {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
    }
]

When I query with: db.getCollection('x').find(

{s : 327079355}

)
I get an answer (although empty).
if I do a remove query: db.getCollection('x').remove(

{s : 327079355}

)
I get:

multiple errors for op : error processing query: ns=dbname.x: s == 327079355.0
Sort: {}
Proj: {}
 No query solutions :: and :: error processing query: ns=dbname.xTree: s == 327079355.0
Sort: {}
Proj: {}
 No query solutions ...

i also did a short test and created a document and used that in the delete query.
Weirdly the remove query deleted the document but returned the error above too.
 
The output of db.getCollection('x').find(

{s : 327079355}

).explain():

/* 1 */
{
    "queryPlanner" : {
        "mongosPlannerVersion" : 1,
        "winningPlan" : {
            "stage" : "SHARD_MERGE",
            "shards" : [ 
                {
                    "shardName" : "repset11",
                    "serverInfo" : {
                        "version" : "3.4.9",
                        "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
                    },
                    "plannerVersion" : 1,
                    "namespace" : "dbname.x",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                        "s" : {
                            "$eq" : 327079355.0
                        }
                    },
                    "winningPlan" : {
                        "stage" : "FETCH",
                        "inputStage" : {
                            "stage" : "IXSCAN",
                            "keyPattern" : {
                                "s" : 1,
                                "ty" : 1
                            },
                            "indexName" : "s_1_ty_1",
                            "isMultiKey" : false,
                            "multiKeyPaths" : {
                                "s" : [],
                                "ty" : []
                            },
                            "isUnique" : false,
                            "isSparse" : false,
                            "isPartial" : false,
                            "indexVersion" : 1,
                            "direction" : "forward",
                            "indexBounds" : {
                                "s" : [ 
                                    "[327079355.0, 327079355.0]"
                                ],
                                "ty" : [ 
                                    "[MinKey, MaxKey]"
                                ]
                            }
                        }
                    },
                    "rejectedPlans" : []
                }, 
                {
                    "shardName" : "repset12",
                    "serverInfo" : {
                        "version" : "3.4.9",
                        "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
                    },
                    "plannerVersion" : 1,
                    "namespace" : "dbname.x",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                        "s" : {
                            "$eq" : 327079355.0
                        }
                    },
                    "winningPlan" : {
                        "stage" : "FETCH",
                        "inputStage" : {
                            "stage" : "IXSCAN",
                            "keyPattern" : {
                                "s" : 1,
                                "ty" : 1
                            },
                            "indexName" : "s_1_ty_1",
                            "isMultiKey" : false,
                            "multiKeyPaths" : {
                                "s" : [],
                                "ty" : []
                            },
                            "isUnique" : false,
                            "isSparse" : false,
                            "isPartial" : false,
                            "indexVersion" : 1,
                            "direction" : "forward",
                            "indexBounds" : {
                                "s" : [ 
                                    "[327079355.0, 327079355.0]"
                                ],
                                "ty" : [ 
                                    "[MinKey, MaxKey]"
                                ]
                            }
                        }
                    },
                    "rejectedPlans" : []
                }, 
                {
                    "shardName" : "repset13",
                    "serverInfo" : {
                        "version" : "3.4.9",
                        "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
                    },
                    "plannerVersion" : 1,
                    "namespace" : "dbname.x",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                        "s" : {
                            "$eq" : 327079355.0
                        }
                    },
                    "winningPlan" : {
                        "stage" : "FETCH",
                        "inputStage" : {
                            "stage" : "IXSCAN",
                            "keyPattern" : {
                                "s" : 1,
                                "ty" : 1
                            },
                            "indexName" : "s_1_ty_1",
                            "isMultiKey" : false,
                            "multiKeyPaths" : {
                                "s" : [],
                                "ty" : []
                            },
                            "isUnique" : false,
                            "isSparse" : false,
                            "isPartial" : false,
                            "indexVersion" : 1,
                            "direction" : "forward",
                            "indexBounds" : {
                                "s" : [ 
                                    "[327079355.0, 327079355.0]"
                                ],
                                "ty" : [ 
                                    "[MinKey, MaxKey]"
                                ]
                            }
                        }
                    },
                    "rejectedPlans" : []
                }, 
                {
                    "shardName" : "repset14",
                    "serverInfo" : {
                        "version" : "3.4.9",
                        "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
                    },
                    "plannerVersion" : 1,
                    "namespace" : "dbname.x",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                        "s" : {
                            "$eq" : 327079355.0
                        }
                    },
                    "winningPlan" : {
                        "stage" : "FETCH",
                        "inputStage" : {
                            "stage" : "IXSCAN",
                            "keyPattern" : {
                                "s" : 1,
                                "ty" : 1
                            },
                            "indexName" : "s_1_ty_1",
                            "isMultiKey" : false,
                            "multiKeyPaths" : {
                                "s" : [],
                                "ty" : []
                            },
                            "isUnique" : false,
                            "isSparse" : false,
                            "isPartial" : false,
                            "indexVersion" : 1,
                            "direction" : "forward",
                            "indexBounds" : {
                                "s" : [ 
                                    "[327079355.0, 327079355.0]"
                                ],
                                "ty" : [ 
                                    "[MinKey, MaxKey]"
                                ]
                            }
                        }
                    },
                    "rejectedPlans" : []
                }, 
                {
                    "shardName" : "repset17",
                    "serverInfo" : {
                        "version" : "3.4.9",
                        "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
                    },
                    "plannerVersion" : 1,
                    "namespace" : "dbname.x",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                        "s" : {
                            "$eq" : 327079355.0
                        }
                    },
                    "winningPlan" : {
                        "stage" : "FETCH",
                        "inputStage" : {
                            "stage" : "IXSCAN",
                            "keyPattern" : {
                                "s" : 1,
                                "ty" : 1
                            },
                            "indexName" : "s_1_ty_1",
                            "isMultiKey" : false,
                            "multiKeyPaths" : {
                                "s" : [],
                                "ty" : []
                            },
                            "isUnique" : false,
                            "isSparse" : false,
                            "isPartial" : false,
                            "indexVersion" : 1,
                            "direction" : "forward",
                            "indexBounds" : {
                                "s" : [ 
                                    "[327079355.0, 327079355.0]"
                                ],
                                "ty" : [ 
                                    "[MinKey, MaxKey]"
                                ]
                            }
                        }
                    },
                    "rejectedPlans" : []
                }
            ]
        }
    },
    "ok" : 1.0
}

The output of db.getCollection('x').explain().remove(

{s : 327079355}

):

/* 1 */
{
    "queryPlanner" : {
        "mongosPlannerVersion" : 1,
        "winningPlan" : {
            "stage" : "SHARD_WRITE",
            "shards" : [ 
                {
                    "shardName" : "repset11",
                    "serverInfo" : {
                        "version" : "3.4.9",
                        "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
                    },
                    "plannerVersion" : 1,
                    "namespace" : "dbname.x",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                        "s" : {
                            "$eq" : 327079355.0
                        }
                    },
                    "winningPlan" : {
                        "stage" : "DELETE",
                        "inputStage" : {
                            "stage" : "FETCH",
                            "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                    "s" : 1,
                                    "ty" : 1
                                },
                                "indexName" : "s_1_ty_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                    "s" : [],
                                    "ty" : []
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 1,
                                "direction" : "forward",
                                "indexBounds" : {
                                    "s" : [ 
                                        "[327079355.0, 327079355.0]"
                                    ],
                                    "ty" : [ 
                                        "[MinKey, MaxKey]"
                                    ]
                                }
                            }
                        }
                    },
                    "rejectedPlans" : []
                }, 
                {
                    "shardName" : "repset12",
                    "serverInfo" : {
                        "version" : "3.4.9",
                        "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
                    },
                    "plannerVersion" : 1,
                    "namespace" : "dbname.x",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                        "s" : {
                            "$eq" : 327079355.0
                        }
                    },
                    "winningPlan" : {
                        "stage" : "DELETE",
                        "inputStage" : {
                            "stage" : "FETCH",
                            "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                    "s" : 1,
                                    "ty" : 1
                                },
                                "indexName" : "s_1_ty_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                    "s" : [],
                                    "ty" : []
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 1,
                                "direction" : "forward",
                                "indexBounds" : {
                                    "s" : [ 
                                        "[327079355.0, 327079355.0]"
                                    ],
                                    "ty" : [ 
                                        "[MinKey, MaxKey]"
                                    ]
                                }
                            }
                        }
                    },
                    "rejectedPlans" : []
                }, 
                {
                    "shardName" : "repset13",
                    "serverInfo" : {
                        "version" : "3.4.9",
                        "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
                    },
                    "plannerVersion" : 1,
                    "namespace" : "dbname.x",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                        "s" : {
                            "$eq" : 327079355.0
                        }
                    },
                    "winningPlan" : {
                        "stage" : "DELETE",
                        "inputStage" : {
                            "stage" : "FETCH",
                            "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                    "s" : 1,
                                    "ty" : 1
                                },
                                "indexName" : "s_1_ty_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                    "s" : [],
                                    "ty" : []
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 1,
                                "direction" : "forward",
                                "indexBounds" : {
                                    "s" : [ 
                                        "[327079355.0, 327079355.0]"
                                    ],
                                    "ty" : [ 
                                        "[MinKey, MaxKey]"
                                    ]
                                }
                            }
                        }
                    },
                    "rejectedPlans" : []
                }, 
                {
                    "shardName" : "repset14",
                    "serverInfo" : {
                        "version" : "3.4.9",
                        "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
                    },
                    "plannerVersion" : 1,
                    "namespace" : "dbname.x",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                        "s" : {
                            "$eq" : 327079355.0
                        }
                    },
                    "winningPlan" : {
                        "stage" : "DELETE",
                        "inputStage" : {
                            "stage" : "FETCH",
                            "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                    "s" : 1,
                                    "ty" : 1
                                },
                                "indexName" : "s_1_ty_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                    "s" : [],
                                    "ty" : []
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 1,
                                "direction" : "forward",
                                "indexBounds" : {
                                    "s" : [ 
                                        "[327079355.0, 327079355.0]"
                                    ],
                                    "ty" : [ 
                                        "[MinKey, MaxKey]"
                                    ]
                                }
                            }
                        }
                    },
                    "rejectedPlans" : []
                }, 
                {
                    "shardName" : "repset17",
                    "serverInfo" : {
                        "version" : "3.4.9",
                        "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e"
                    },
                    "plannerVersion" : 1,
                    "namespace" : "dbname.x",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                        "s" : {
                            "$eq" : 327079355.0
                        }
                    },
                    "winningPlan" : {
                        "stage" : "DELETE",
                        "inputStage" : {
                            "stage" : "FETCH",
                            "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                    "s" : 1,
                                    "ty" : 1
                                },
                                "indexName" : "s_1_ty_1",
                                "isMultiKey" : false,
                                "multiKeyPaths" : {
                                    "s" : [],
                                    "ty" : []
                                },
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 1,
                                "direction" : "forward",
                                "indexBounds" : {
                                    "s" : [ 
                                        "[327079355.0, 327079355.0]"
                                    ],
                                    "ty" : [ 
                                        "[MinKey, MaxKey]"
                                    ]
                                }
                            }
                        }
                    },
                    "rejectedPlans" : []
                }
            ]
        }
    },
    "ok" : 1.0
}



 Comments   
Comment by Eric Sedor [ 08/Feb/19 ]

Sorry Daniel, we aren't sure what would have caused this bad state. We're glad you were able to determine a workaround but without the ability to reproduce that state we're not able to investigate much further. Ideally we would want to look at a problem like this immediately after it starts on a supported version.

Comment by Daniel Breitlauch [ 07/Feb/19 ]

Hi Eric,

I had a look at SERVER-17397. I am quite sure that we never dropped and recreated the collection. I think the shard tag was not directly applied and the db and collection was created on all repsets. And I think all the repsets had the initial chunks. We then added the shard tag and moved the chunks to the repsets according to shard tag.

Somehow (years later) the mongo routers are still sending requests to repsets other than shards with chunks just because there the collection still exists although without chunks.

I can confirm the problem went away after deleting the collection on repsets without chunks.

Any idea why?

Best regards,

Daniel

Comment by Eric Sedor [ 31/Jan/19 ]

Hi Daniel this is Eric stepping in to clarify. Shards without chunks should not be receiving queries. That suggests SERVER-17397 or another issue if you can rule out that known cause of this behavior.

Comment by Daniel Breitlauch [ 31/Jan/19 ]

Hi  Kelsey,

I can confirm that all queries now work normally after deleting all collections and databases from repsets that are not part of the collection (have no chunks of the collection).

I am not sure if this collection was dropped once and recreated.

For me the question still stands why the mongo router would forward the query to repsets that hold no chunks of the collection according to config.chunks? It was doing that only because there was the db and collection still on the repset.

Is that intended behaviour?

Many thanks for clearing that up,

Daniel

Comment by Kelsey Schubert [ 29/Jan/19 ]

Hi breitlauch,

Sorry for the delay getting back to you. Before MongoDB 4.0.3, the only way to have tags defined for an unsharded collection would would be if the collection was previously sharded, tags were defined, and then the collection was subsequently dropped. As a result, I suspect you're likely encountering SERVER-17397, which is leading to the behavior you're now observing.

To clarify, under normal operation, query execution is optimized to just considering shards that hold chunks of that collection. I suspect that as a result of SERVER-17397 the queries are not being correctly targeted resulting in inconsistencies between whether the query is issued on the mongos or directly against each shard.

Has this collection's namespace ever been dropped and recreated? If so, were the steps described in SERVER-17397 followed?

Kind regards,
Kelsey

Comment by Daniel Breitlauch [ 04/Jan/19 ]

Hi Kelsey,

I think I found the issue!

The issue has 3 components that play together.

  1. the mongo behaviour when creating new sharded collections (probably of an old mongo version)
  2. scatter/getter queries in the mongo router
  3. Index creation queries from the mongo router

Additional knowledge about our system: There are many more shards for other collections.

1.Part:

When creating a new sharded collection mongodb creates the collection on all shards despite that it has a tagRange associated. Only after the balancer run, chunks will be put to the right shards.

At least that is what happened in older versions of mongodb and the collection in question is quite old. I don't know if this is still happening.

2. Part:

The query in question is untargeted since it has no shard key in the query. The router pushes the query to all shards not only the ones that hold chunks of the collection.

Not sure why this is done, couldn't the query execution be optimized by just considering shards that hold chunks of that collection?

3. Part:

Index creation over the mongo router is targeted only to shards that hold chunks of the collection.

 

Conclusion:

Finally the no query solutions error is produced because:

  1. Some unrelated shards still have the collection despite that they hold no chunks
  2. Newly created indices only go to the shards that hold chunks.
  3. Scatter queries go to all shards

Hence we have unrelated shards with no chunks being queries without having the indices. They report no query solution. Shards that hold chunks of the collection will just work normally.

Sorry for the lengthy report ;D

Could you elaborate why this is done this way? Especially having untargeted queries go to all shards?

Kind regards,

Daniel Breitlauch

Comment by Daniel Breitlauch [ 02/Jan/19 ]

Hi Kelsey,

executing the query on the shards produced some interesting insights.
No primary complained about the delete query.

1. I created a test document
2. I executed the remove on the mongos and got the 'No query solutions' error.
3. I checked, the document was deleted despite the error
4. I reinserted the document and executed the remove on all shards.
4.1 the shard that has the document just deleted it: WriteResult({ "nRemoved" : 1 })
4.2. all other shards did also not produce any errors: WriteResult({ "nRemoved" : 0 })

Only going through mongoses produces the error!
I checked with multiple mongoses now and all produce this error:

db.getCollection('relations').remove({s: -1})
WriteResult({
	"nRemoved" : 0,
	"writeError" : {
		"code" : 65,
		"errmsg" : "multiple errors for op : error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions :: and :: error processing query: ns=refindPublicationRelations001.relationsTree: s == -1.0\nSort: {}\nProj: {}\n No query solutions"
	}
})

The sharding status:

MongoDB shell version v3.4.9
MongoDB server version: 3.4.9
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"minCompatibleVersion" : 5,
	"currentVersion" : 6,
	"clusterId" : ObjectId("5194c7d2de640d0598cf0566")
}
  shards:
	{  "_id" : "repset11",  "host" : "repset11/mongs-25:27018,mongs-26:27018",  "tags" : [ "x" ],  "state" : 1 }
	{  "_id" : "repset12",  "host" : "repset12/mongs-29:27018,mongs-30:27018",  "tags" : [ "x" ],  "state" : 1 }
	{  "_id" : "repset13",  "host" : "repset13/mongs-31:27018,mongs-32:27018",  "tags" : [ "x" ],  "state" : 1 }
	{  "_id" : "repset14",  "host" : "repset14/mongs-34:27018,mongs-36:27018",  "tags" : [ "x" ],  "state" : 1 }
	{  "_id" : "repset17",  "host" : "repset17/mongs-33:27018,mongs-35:27018",  "tags" : [ "x" ],  "state" : 1 }
  active mongoses:
	{  "_id" : "master...:27017",  "ping" : ISODate("2019-01-02T14:23:29.420Z"),  "up" : NumberLong(12971962),  "waiting" : true,  "mongoVersion" : "3.4.9" }
 autosplit:
	Currently enabled: yes
  balancer:
	Currently enabled:  no
	Currently running:  no
		Balancer lock taken at Mon Sep 24 2018 09:46:59 GMT+0000 (UTC) by ConfigServer:Balancer
	Failed balancer rounds in last 5 attempts:  5
	Last reported error:  could not get updated shard list from config server due to interrupted at shutdown
	Time of Reported error:  Mon Jun 12 2017 13:27:58 GMT+0000 (UTC)
	Migration Results for the last 24 hours: 
		No recent migrations
  databases:
	...
	{  "_id" : "dbName",  "partitioned" : true,  "primary" : "repset11" }
		dbName.x
			shard key: { "t" : "hashed" }
			unique: false
			balancing: true
			chunks:
				repset11	884
				repset12	942
				repset13	907
				repset14	891
				repset17	912

I omitted all the chunks lmk if you want to see them too.

Any idea what could be happening on the mongos side?
Thanks for helping here,
Daniel

Comment by Kelsey Schubert [ 11/Dec/18 ]

Hi breitlauch,

Thanks for your report. I'm also struggling to reproduce this issue, but I hope that some additional information will help our investigation. It'd be great if you could address the following:

  1. Would you please provide the output of sh.status(true)?
  2. Would you please connect directly to the primary of each shard to run the problematic command and note which shards produce this error?

This information should give us a better sense of how the data is distributed and which shards are affected.

Thank for your help,
Kelsey

Comment by Eric Sedor [ 05/Dec/18 ]

Hello and thanks for your patience so far. I did want to let you know that we are investigating your report!

Eric

Generated at Thu Feb 08 04:48:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.