[SERVER-38309] "No query solutions" WriteConcernException on simple delete query with index Created: 29/Nov/18 Updated: 08/Feb/19 Resolved: 08/Feb/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance |
| Affects Version/s: | 3.4.9 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Daniel Breitlauch | Assignee: | Eric Sedor |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Steps To Reproduce: | It does not happen in all sharded collections. So i dont think this is reproducible. Although i can reproduce it consistently on this collection. |
| Participants: |
| Description |
|
I have a sharded collection with 5 repsets consisting of 3 mongod instances respectively. On the shards I have a collection sharded by { t: hashed}. I enabled the no-table-scan parameter via db query to all mongod instances. There are these indices on the collection on all mongod instances:
When I query with: db.getCollection('x').find( {s : 327079355}) )
i also did a short test and created a document and used that in the delete query. ).explain():
The output of db.getCollection('x').explain().remove( {s : 327079355}):
|
| Comments |
| Comment by Eric Sedor [ 08/Feb/19 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Sorry Daniel, we aren't sure what would have caused this bad state. We're glad you were able to determine a workaround but without the ability to reproduce that state we're not able to investigate much further. Ideally we would want to look at a problem like this immediately after it starts on a supported version. | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Breitlauch [ 07/Feb/19 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Eric, I had a look at Somehow (years later) the mongo routers are still sending requests to repsets other than shards with chunks just because there the collection still exists although without chunks. I can confirm the problem went away after deleting the collection on repsets without chunks. Any idea why? Best regards, Daniel | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Eric Sedor [ 31/Jan/19 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Daniel this is Eric stepping in to clarify. Shards without chunks should not be receiving queries. That suggests | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Breitlauch [ 31/Jan/19 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Kelsey, I can confirm that all queries now work normally after deleting all collections and databases from repsets that are not part of the collection (have no chunks of the collection). I am not sure if this collection was dropped once and recreated. For me the question still stands why the mongo router would forward the query to repsets that hold no chunks of the collection according to config.chunks? It was doing that only because there was the db and collection still on the repset. Is that intended behaviour? Many thanks for clearing that up, Daniel | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 29/Jan/19 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi breitlauch, Sorry for the delay getting back to you. Before MongoDB 4.0.3, the only way to have tags defined for an unsharded collection would would be if the collection was previously sharded, tags were defined, and then the collection was subsequently dropped. As a result, I suspect you're likely encountering To clarify, under normal operation, query execution is optimized to just considering shards that hold chunks of that collection. I suspect that as a result of Has this collection's namespace ever been dropped and recreated? If so, were the steps described in Kind regards, | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Breitlauch [ 04/Jan/19 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Kelsey, I think I found the issue! The issue has 3 components that play together.
Additional knowledge about our system: There are many more shards for other collections. 1.Part: When creating a new sharded collection mongodb creates the collection on all shards despite that it has a tagRange associated. Only after the balancer run, chunks will be put to the right shards. At least that is what happened in older versions of mongodb and the collection in question is quite old. I don't know if this is still happening. 2. Part: The query in question is untargeted since it has no shard key in the query. The router pushes the query to all shards not only the ones that hold chunks of the collection. Not sure why this is done, couldn't the query execution be optimized by just considering shards that hold chunks of that collection? 3. Part: Index creation over the mongo router is targeted only to shards that hold chunks of the collection.
Conclusion: Finally the no query solutions error is produced because:
Hence we have unrelated shards with no chunks being queries without having the indices. They report no query solution. Shards that hold chunks of the collection will just work normally. Sorry for the lengthy report ;D Could you elaborate why this is done this way? Especially having untargeted queries go to all shards? Kind regards, Daniel Breitlauch | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Breitlauch [ 02/Jan/19 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Kelsey, executing the query on the shards produced some interesting insights. 1. I created a test document Only going through mongoses produces the error!
The sharding status:
I omitted all the chunks lmk if you want to see them too. Any idea what could be happening on the mongos side? | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 11/Dec/18 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi breitlauch, Thanks for your report. I'm also struggling to reproduce this issue, but I hope that some additional information will help our investigation. It'd be great if you could address the following:
This information should give us a better sense of how the data is distributed and which shards are affected. Thank for your help, | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Eric Sedor [ 05/Dec/18 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello and thanks for your patience so far. I did want to let you know that we are investigating your report! Eric |