[SERVER-27243] Upgrade to 3.4 'create collection' fails in Backup Created: 30/Nov/16 Updated: 05/Apr/17 Resolved: 13/Dec/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | 3.4.0 |
| Fix Version/s: | 3.4.2, 3.5.1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Isabel Peters | Assignee: | Charlie Swanson |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Backport Requested: |
v3.4
|
||||||||
| Sprint: | Query 2016-12-12, Query 2017-01-23 | ||||||||
| Participants: | |||||||||
| Description |
|
After upgrading to 3.4. backup for a deployment stopped working. Investigation into the mongod logs shows following error while trying to apply oplogs.
The version for the first backup snapshot is 3.0.7 (2015). The deployment dates back to 2013 (when collection oauthnonces was first created). Find the `oauthnonces` dataset attached. CC: steve.briskin, john.morales |
| Comments |
| Comment by Githook User [ 22/Dec/16 ] | |
|
Author: {u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'charlie.swanson@mongodb.com'}Message: It is possible for older versions of MongoDB to save invalid collection (cherry picked from commit 68908f8e03ee1a4c0400dae26cc20b0c304b913b) | |
| Comment by Githook User [ 13/Dec/16 ] | |
|
Author: {u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'charlie.swanson@mongodb.com'}Message: It is possible for older versions of MongoDB to save invalid collection (cherry picked from commit 6fba074768fad5f47611de38682257257005a1a6) | |
| Comment by Charlie Swanson [ 06/Dec/16 ] | |
|
steve.briskin as discussed in person, our best guess is that the order of the collection options are getting jumbled somewhere between reading them from the oplog and being stored on the backup daemon's mongod. I have done some testing and also read the code for a while. I don't see any place where the order could get swapped server side. Would it be possible for you to do some testing and log what the command looks like at various points in the process (e.g. when interpreted in go driver, when sent to the backup agent, when sent as applyOps command from the backup agent)? If you can't do this, is there somewhere you could point me to to set up the backup daemon myself and also point me where to add the logging (if you happen to know)? On a separate note, I have also tested (at least against 2.4, I can't remember if I've tested on newer versions) what the server does if it receives an applyOps with the op being in the wrong order (e.g. {capped: true, create: "oauthnonces"} instead of {create: "oauthnonces", capped: true}). It looks like the server does not respond with an error, but neither does it actually create a collection with those invalid options. So I still have no clear picture of how this happened... | |
| Comment by Asya Kamsky [ 05/Dec/16 ] | |
|
Python won't maintain order entered of a dictionary (so when order is important we recommend using SON class). It also used to allow passing in options dictionary (deprecated in pymongo 2.2) to create_collection command and in 2.4 server it would have just stored all the options passed in. | |
| Comment by Tess Avitabile (Inactive) [ 01/Dec/16 ] | |
|
I wonder whether the driver is alphabetizing the options. | |
| Comment by James Wahlin [ 01/Dec/16 ] | |
|
The 3.4.0 code base rejects unknown options. There is an exception for the "create" field which was generated in versions 2.4 and earlier but we expect it to be the first field in the options metadata (see this comment). It appears to be in the second position here:
|