[SERVER-26182] 3.2 node syncing from a 3.0 node can crash due to too-large BSON during upconversion to find command reply Created: 20/Sep/16 Updated: 11/Feb/17 Resolved: 08/Nov/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Querying, Replication |
| Affects Version/s: | None |
| Fix Version/s: | 3.2.11 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Avraham Kalvo | Assignee: | Charlie Swanson |
| Resolution: | Done | Votes: | 0 |
| Labels: | code-and-test | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: | Upgrade member to 3.2 |
||||||||
| Sprint: | Query 2016-10-10, Query 2016-11-21 | ||||||||
| Participants: | |||||||||
| Case: | (copied to CRM) | ||||||||
| Description |
|
When a 3.2 replica set member is syncing from a 3.0 node, it downconverts find command requests to legacy OP_QUERY find requests and upconverts legacy OP_QUERY find replies to find command replies. This upconversion process can create a document that is larger than the 16 MB maximum, causing the secondary to trip a fatal assertion. See comment below for more details. This issue should only affect users whose data files contain documents 12MB or larger. A backtrace such as the following should appear in the logs when this issue occurs:
Original descriptionWe've recently started upgrading our Mongo replica sets to 3.2, 2016-09-19T13:07:35.977+0000 I NETWORK [initandlisten] connection accepted from 10.14.0.51:45381 #140 (136 connections now open) This appears to have to do with a specific collection that at least one document that is bigger than 16MB, thing is we've issued db.upgradeCheckAllDBs() prior to the upgrade for this member and it completed perfectly fine. How can we spot the root cause for this and eliminate it please so we can move on with the 3.2 upgrade for this replica set as well as all of our other replica sets? Many thanks, |
| Comments |
| Comment by Githook User [ 16/Nov/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'charlie.swanson@mongodb.com'}Message: | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 08/Nov/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'charlie.swanson@mongodb.com'}Message: | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Avraham Kalvo [ 09/Oct/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Many thanks, | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by David Storch [ 26/Sep/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
avrahamk, a mixed 3.0.12 / 3.2 cluster should be affected by this issue as well. Without further information, my best guess is that you simply happened not to hit the issue. This requires a particular sequence of variously sized BSON objects to be returned in a query response, which is why the issue does not affect many 3.0 => 3.2 rolling upgrades. | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Avraham Kalvo [ 25/Sep/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks a lot David, Thomas, for that elaborated response. I couldn't find where to add a comment on the ticket, Thanks, On Sat, Sep 24, 2016 at 1:17 AM, David Storch (JIRA) <jira@mongodb.org> | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by David Storch [ 23/Sep/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
I've reproduced the reported stack trace with the following script:
Moving this ticket into "Needs Triage" so that it will be evaluated by the query team. | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by David Storch [ 22/Sep/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
I reviewed this with anonymous.user, and we came up with a hypothesis of what we think is happening. Thomas will develop a repro script to confirm. In 3.2, we introduced a new protocol for performing queries called find command. The syntax for the find command is documented here: https://docs.mongodb.com/manual/reference/command/find/#dbcmd.find. This replaces an older protocol typically referred to as OP_QUERY legacy find. When a version 3.2 member of a replica set syncs from another 3.2, it uses the new find command protocol for fetching oplog entries from the sync source. If the sync source is 3.0, however, then it must use legacy OP_QUERY find for compatibility. In order to deal with this, the replication system downconverts find command requests to OP_QUERY find and then upconverts OP_QUERY find replies to find command replies. The upconversion happens here: Not all OP_QUERY find replies, however, can be represented as find command replies! The find command reply format allows for at most 16MB of user data. The OP_QUERY reply format, however, can theoretically contain 20MB of user data. This rarely happens in practice, however, since the batching rules in 3.0 are to stop adding documents when the size of the reply buffer exceeds 4MB. But imagine the following problematic scenario:
The find command itself is not affected by a similar problem; when it retrieves a document from the query system that will put the batch size over 16MB, it queues the document for later and returns the batch without it. One possible fix, assuming that we have correctly identified the problem, would be to backport code to 3.0 that adds similar behavior to the legacy OP_QUERY find path. Another approach would be to put a fix into the 3.2 branch to make the replication upconversion avoid creating a too-large document. | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 21/Sep/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi avrahamk, Thanks for reporting this issue. We are investigating it and will update you when we know more. Kind regards, | |||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Avraham Kalvo [ 21/Sep/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||
|
We are discussing upgrade to version 3.2.9 from probably version 3.0.9 (we didn't notice its exact minor version before having setup the 3.2.9 binaries) This happens over Debian GNU/Linux 7 Thanks, |