[SERVER-27701] Race in AsyncResultsMerger error handling may trigger mongos invariant Created: 17/Jan/17 Updated: 07/Sep/17 Resolved: 06/Feb/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 3.4.1 |
| Fix Version/s: | 3.2.14, 3.4.4, 3.5.3 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Remi Forest | Assignee: | David Storch |
| Resolution: | Done | Votes: | 0 |
| Labels: | bkp | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Backport Requested: |
v3.4, v3.2
|
||||||||||||||||
| Steps To Reproduce: | I was not able to reproduce so far. |
||||||||||||||||
| Sprint: | Query 2017-02-13 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||
| Linked BF Score: | 0 | ||||||||||||||||
| Description |
|
We were running a stress test on a distributed sharded cluster (8 shards, 7 replicas, on 5 AWS regions), checking behaviour when killing different parts of the cluster. We setup 5 application servers (1 per region) with mongos on the same server. One of the mongos from where we were running the tests crashed after we killed some processes (including some config servers). The message when crashing was : Attached is the mongos log. Tests started at 15:47 after sharding the collection "renault" |
| Comments |
| Comment by Githook User [ 26/May/17 ] | |||||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}Message: (cherry picked from commit 5a35a6c425b815ecf7795d8e2706b9f3e82c7ce6) | |||||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 31/Mar/17 ] | |||||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}Message: (cherry picked from commit 5a35a6c425b815ecf7795d8e2706b9f3e82c7ce6) | |||||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 06/Feb/17 ] | |||||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}Message: | |||||||||||||||||||||||||||||||||||||||||||
| Comment by David Storch [ 03/Feb/17 ] | |||||||||||||||||||||||||||||||||||||||||||
|
Hi remi.forest, Thanks for reporting this issue. I've tracked down the root cause. It's a race condition in the AsyncResultsMerger, the component that the mongos read path uses to merge the streams of results from the shards into a single stream to return to the client driver. The AsyncResultsMerger is used by the mongos query path's merge stage, RouterStageMerge: https://github.com/mongodb/mongo/blob/r3.4.1/src/mongo/s/query/router_stage_merge.cpp#L43-L56 When RouterStageMerge is asked to retrieve another query result, it delegates to the AsyncResultsMerger, first calling ARM::ready(). This method call asks whether either there is either an error or a result document available to return to the client. If neither an error nor a document is available, it calls ARM::nextEvent(), which will schedule commands to run against the shards in order to retrieve more results. However, these two function calls happen in two steps, and each will separately acquire and release the AsyncResultsMerger's mutex. This means that an error response from the shards can be processed after ARM::ready() but before ARM::nextEvent()! The failed invariant incorrectly assumes that it is impossible to have an outstanding error in ARM::nextEvent(), since the caller is expected to have checked for an error using ARM::ready(). I was able to reproduce the invariant by applying the following patch to mongos:
This patch mocks the problem scenario by injecting an error into the ARM in between the calls to ARM::next() and ARM::ready(). It reliably triggers the invariant when I run a find with no options against a sharded collection. This seems pretty unlikely to happen in practice under normal operation, but it is definitely something which our team will work on fixing. Best, | |||||||||||||||||||||||||||||||||||||||||||
| Comment by Remi Forest [ 17/Jan/17 ] | |||||||||||||||||||||||||||||||||||||||||||
|
Yes, but the same type of error happened at 16:27 and 15:59 but without crash (though on different TaskExecutorPool) | |||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 17/Jan/17 ] | |||||||||||||||||||||||||||||||||||||||||||
|
Looks like there were some connectivity problems before the assertion failure.
|